A real-time screen watermarking method and system for tracing screen shooting leaks

Abstract

The present invention discloses a real-time screen watermarking method and system for tracing screen shooting leaks, which can be used to obtain evidence from secretly photographed screen photos. The method uses a preset watermark template composed of an information watermark and a synchronization watermark, and encodes key information such as a user ID and an IP address as watermark information into a corresponding watermark template through a QR code encoding method and an inverse Fourier transform. The least noticeable difference model is adopted to directly embed the pre-generated watermark template into the back buffer of the desktop window manager exchange chain, thereby ensuring the imperceptibility of the watermark and realizing the real-time refresh of the watermark. Automatic perspective correction is realized through a synchronous response indexing method, and the information watermark template embedding area is located. Finally, the information watermark is extracted from the discrete Fourier transform domain of the noise component. The method can effectively resist various screen secretly photographing methods, and realize the high-frequency refresh of the watermark and the robustness of watermark evidence without affecting the normal display of the screen.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and in particular relates to a real-time screen watermarking method and system for tracing screen shooting leaks based on a preset watermark template. Background Art

[0002] With the popularity of digital office and smart phones, the acquisition and transmission of information has become very convenient, but new security issues have also emerged. Unauthorized use of smart phones to shoot sensitive content on the screen has become a common means of information leakage. Traditional information protection technology cannot prevent such screenshot leakage. To solve this problem, screen watermarking technology is proposed to track the source of leakage. Visible watermarks affect the user experience and are easily erased. Therefore, imperceptible screen watermarking technology is more practical.

[0003] Existing imperceptible image watermarking techniques are mainly designed for conventional image processing attacks, but not for screen capture attacks. The distortion caused by screen capture is different from print scanning or print camera attacks. Many screen capture robust watermarking methods have been proposed, which can be divided into video and static image methods according to the different watermark carriers. Video watermarking methods are used to track illegal camera recordings, but are not suitable for scenarios where screen content is captured.

[0004] Static image methods can be divided into restricted and unrestricted methods based on how perspective distortion is handled. Restricted methods require capturing the entire host image and performing manual perspective correction to extract the watermark. These methods are limited in scope and require the shape of the host image to be known in advance. Unrestricted methods can automatically extract watermarks under perspective distortion, but the algorithm complexity is high, not suitable for real-time applications, and can only protect data that is pre-embedded with the watermark. Summary of the invention

[0005] Purpose of the invention: In view of the information leakage security issues brought about by the popularity of digital office and smart phones in modern office environments, as well as the problem of unauthorized capture of sensitive content on the screen, the purpose of the present invention is to propose a real-time screen watermarking method and system for tracing screen shooting leaks, aiming to achieve the protection of screen content and tracing the source of leakage.

[0006] Technical solution: To achieve the above-mentioned purpose, the present invention provides a real-time screen watermarking method for tracing screen shooting leaks, comprising the following steps:

[0007] Generate a watermark template, including an information watermark template, a first synchronization watermark template and a second synchronization watermark template; the information watermark template is generated by QR encoding of user information including user ID, IP address and operation time through inverse Fourier transform; the first synchronization watermark template and the second synchronization watermark template are generated by inverse Fourier transform from preset synchronization watermark information, wherein the information of the first and second synchronization watermark templates are respectively embedded in the second and fourth quadrants and the first and third quadrants of the Fourier transform domain amplitude spectrum at preset radii and angles;

[0008] According to the screen size, the information watermark template and the synchronization watermark template are adaptively combined to generate a screen watermark template;

[0009] Get the function address of the swap chain swap function in the desktop window manager process, get the back buffer address of the desktop window manager swap chain, use a simplified just noticeable difference model, calculate the JND threshold of the back buffer data, and embed the pre-generated screen watermark template directly into the back buffer according to the threshold;

[0010] When a photo image of the shooting screen is obtained, a multi-scale synchronous watermark template is detected to obtain the positions of the first synchronous watermark template and the second synchronous watermark template, and automatic perspective correction is implemented based on the positions of the synchronous watermark templates, and the information watermark template embedding area is located to obtain the embedded user information.

[0011] Furthermore, the watermark template generation includes the following steps:

[0012] The user information is used to generate a QR code using a QR code algorithm, and the coding elements in the QR code coding area are recorded line by line as a binary watermark information sequence W1;

[0013] According to the screen size M×N, determine the information watermark block size L1×L1 and the synchronization watermark block size L2×L2, L1>L2;

[0014] The L1×L1 zero matrix is ​​regarded as the Fourier transform domain amplitude map of the information watermark block. Determine the location of embedded watermark information based on Embed watermark information and generate information watermark template through iDFT; where W1(j) represents the jth element in W1, l1 represents the length of W1, R represents the intermediate frequency radius, M1(x, y) represents the amplitude coefficient after embedding the watermark, and k1 is a constant representing the watermark embedding strength;

[0015] The L2×L2 zero matrix is ​​regarded as the Fourier transform domain amplitude map of the synchronization watermark block. At least 15 coefficients are set as the carriers of the synchronization watermark information at at least 3 preset radii and 5 equally spaced angles in the second and fourth quadrants. The watermark information is embedded based on M2(x, y)=k2. The first synchronization watermark template a is generated through iDFT. At least 15 coefficients are set as the carriers of the synchronization watermark information at at least 3 preset radii and 5 equally spaced angles in the first and third quadrants. The watermark information is embedded based on M2(x, y)=k2. The second synchronization watermark template b is generated through iDFT; wherein M2(x, y) represents the amplitude coefficient after the watermark is embedded, and k2 is a constant representing the watermark embedding strength.

[0016] Furthermore, the adaptive combination of the information watermark template and the synchronization watermark template to generate the screen watermark template is specifically:

[0017] Assume the screen size is M×N, according to the formula Determine the information watermark template position, where and They represent the number of rows and columns that can be accommodated, L1 represents the side length of the information watermark template; m1 and n1 represent the indexes of the rows and columns in all c1×c2 information watermarks, respectively;

[0018] Based on the formula Determine the position of the first synchronization watermark template a; wherein m2, n2 represent the index of the row and column in the synchronization watermark, and L2 represents the side length of the synchronization watermark template;

[0019] Between every two first synchronization watermark templates a, three second synchronization watermark templates b are uniformly embedded.

[0020] Furthermore, the method for obtaining the back buffer address of the desktop window manager swap chain is as follows: obtaining the addresses of the Present function and the GetFrameStatisticInternal function in the dynamic link library to be injected by continuous matching of machine codes; hooking the Present function by an inline hook method; calculating the swap chain address, the swap chain base address is directly obtained by the offset of the Present function address, the swap chain constant is obtained by the offset of the GetFrameStatisticInternal function address, the swap chain pointer address is obtained according to the swap chain constant offset, and then the back buffer address to be displayed is obtained.

[0021] Furthermore, when the simplified just noticeable difference model calculates the JND threshold, the following Sobel operator is used to respectively calculate the horizontal gradient G1 and the vertical gradient G2 of the two-dimensional image in the buffer, thereby obtaining a contrast masking threshold based on the gradient and the background brightness;

[0022]

[0023] Finally, based on the formula JND(x,y)=P L (x,y)+P c (x,y)-a*min(P L (x,y),P c (x,y)) calculates the JND threshold, where P L (x,y),P c (x, y) represents the brightness masking threshold and contrast masking threshold at the position (x, y), respectively, and a is the coefficient.

[0024] Further, the two-dimensional image object I written into the back buffer is calculated based on the following formula: b :

[0025] I b (x,y)=Bg(x,y)+M w (x,y)*JND(x,y)*η

[0026] Among them B g represents the average background brightness, M w represents the binary watermark template, and η represents the watermark embedding strength.

[0027] Furthermore, the detection of the multi-scale synchronous watermark template comprises the following steps:

[0028] Convert the captured image I into a brightness image I a ;

[0029] Image I a Scale to multiple scales and compute image I at each scale a The noise component I n ;

[0030] On each scale image, a window of preset pixel size is used to identify I n Perform slider detection one by one;

[0031] In the Fourier transform domain amplitude diagram of each slider i, the maximum specified number of amplitude values ​​within the preset angle range of the first quadrant, the preset angle range of the second quadrant, and the preset radius range are recorded, and the amplitude values ​​located in the second quadrant or the first quadrant are recorded as V respectively. l (i) and V r (i), based on the formula Calculate the slider window synchronization response value, where R s,i represents the synchronous response index of the detection window corresponding to the i-th slider on scale s, N l,1 and N r,1 Respectively represent V l (i) and Vr (i) The number of recorded amplitude values;

[0032] Determine the optimal scale according to the synchronous response index of each scale, and determine the synchronous response point of each window in the optimal scale;

[0033] Retrieve all synchronous response points, find all point sets that can be used to correct the information watermark position, each point set contains four points that can form a quasi-rectangle, cut out the area formed by the point set, and perform perspective transformation;

[0034] In the clipped area, a synchronization response index is calculated using a sliding window to determine a synchronization response point of the second synchronization watermark template.

[0035] Furthermore, the automatic perspective correction based on the synchronized watermark template position comprises the following steps:

[0036] For the synchronization response points representing the first and second synchronization watermark templates around the message watermark template, in the area centered on them, the sliding window interval is reduced to perform synchronization response detection; for the synchronization response points of the first synchronization watermark template, the maximum preset number of amplitude values ​​within the preset radius and angle range of the second quadrant is recorded; for the synchronization response points of the second synchronization watermark template, the maximum preset number of amplitude values ​​within the preset radius and angle range of the first quadrant is recorded; the precision correction synchronization response index of each coordinate is calculated based on the following formula:

[0037] R a,i =sum(V a (i))

[0038] R b,i =sum(V b (i))

[0039] Where V a (i) and R a,i are the amplitude value set of the detection window corresponding to the i-th slider of the first synchronization watermark template and the precise correction synchronization response index, V b (i) and R b,i are respectively the amplitude value set of the detection window corresponding to the i-th slider of the second synchronization watermark template and the precise correction synchronization response index;

[0040] The coordinates of the regional maximum fine-calibrated synchronous response index are perspective transformed, and the transformed coordinates are used as the result of fine calibration.

[0041] Furthermore, the method for obtaining the embedded user information is:

[0042] Extract the information block, use the noise component of its brightness image, and convert it to the DFT domain; record the maximum amplitude value V within the 3×3 range centered at each watermark embedding positionmax (j), where j represents the jth watermark bit; based on the formula Determine the extracted watermark information bit W′(j), where and σ T They represent the mean and standard deviation of all amplitude coefficient values ​​within the embedding range respectively, and k3 is a constant. Perform QR decoding on W′(j) to obtain the user watermark information.

[0043] The present invention also provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the real-time screen watermarking method for tracing screen shooting leaks are implemented.

[0044] Beneficial effects: Features and technical advantages of the method of the present invention: (1) The method is designed for real-time screen capture scenarios. By means of the HOOK exchange chain exchange function, the preset watermark template is directly embedded into the back buffer of the desktop window manager exchange chain, which meets the refresh rate requirements of the screen watermark and ensures that the human eye perceives continuous screen content; (2) The proposed watermark technology enhances the imperceptibility of the watermark through the minimum perceptible model, and will not have a significant impact on the user's visual experience. Compared with visible watermarks, it is more difficult to be deleted or modified by leakers; (3) By adopting an adaptive embedding method, the watermark can remain stable under various distortion conditions and has good anti-attack performance; (4) The method is suitable for screen capture scenarios of various static images and videos. Whether it is a picture or a video, no matter what content is displayed on the screen, it can be protected by the watermark provided by this method; (5) The specific position of the synchronization watermark is calculated by the synchronization response index method, and the precise position of the two synchronization watermarks is obtained by synchronization response detection. Based on the two synchronization watermarks, a total of 16 position coordinate algorithms can achieve a more accurate watermark extraction method without manual perspective correction, which can be automatically extracted under perspective distortion without complicated operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is an overall flow chart of the method of the embodiment of the present invention;

[0046] Figure 2 is a flow chart of watermark template generation according to an embodiment of the present invention;

[0047] Figure 3 is a watermark embedding flow chart of an embodiment of the present invention;

[0048] Figure 4 is a flow chart of watermark information extraction and detection according to an embodiment of the present invention;

[0049] Figure 5 This is an example diagram of a watermark information QR code according to an embodiment of the present invention;

[0050] Figure 6 It is an information watermark and a synchronous watermark template diagram using an embodiment of the present invention;

[0051] Figure 7 is a schematic diagram of a watermark template using an embodiment of the present invention;

[0052] Figure 8 is a diagram of an example of a watermark template using an embodiment of the present invention;

[0053] Fig. 9 is a screen watermark image using an embodiment of the present invention;

[0054] Fig.10 is a captured image obtained using an embodiment of the present invention;

[0055] Fig.11 is a synchronous response heat map using an embodiment of the present invention;

[0056] Fig.12 It is a screenshot of the watermark information extraction area using an embodiment of the present invention;

[0057] Fig.13 This is an example diagram of an information watermark QR code extracted and generated using an embodiment of the present invention;

[0058] Fig.14 This is a screenshot of the watermark information extracted using the embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0060] The embodiment of the present invention discloses a real-time screen watermark method for tracing screen shooting leaks based on a preset template, a simplified JND algorithm, a HOOK function and an adaptive embedding method. The method mainly includes watermark template generation, watermark embedding, and watermark extraction and detection. The specific process is as follows:

[0061] 1) Watermark template generation, the watermark template includes an information watermark template, a first synchronous watermark template a and a second synchronous watermark template b; according to the screen size, the information watermark template and the synchronous watermark template are adaptively combined to generate a screen watermark template. The information watermark template is generated by QR encoding of user information including user ID, IP address and operation time through inverse Fourier transform; the first synchronous watermark template and the second synchronous watermark template are generated by inverse Fourier transform from preset synchronous watermark information, wherein the information of the first and second synchronous watermark templates are respectively embedded in the preset radius and angle of the second and fourth quadrants and the first and third quadrants of the Fourier transform domain amplitude spectrum. The specific steps are as follows Figure 2 As shown, including:

[0062] Step 11: Get the input watermark information and record it as W0;

[0063] Step 12: Based on W0, a QR code is generated using a QR code algorithm, and the coding elements in the QR code coding area are recorded line by line as a binary watermark information sequence W1;

[0064] Step 13: The screen size is M×N, and the information watermark block size is L1×L1 based on formula (1);

[0065] L1=floor(min(M,N)*0.47) (1)

[0066] Step 14: Consider the zero matrix of size L1×L1 as the Fourier transform domain amplitude map of the information watermark block, determine the location of the embedded watermark information based on formula (2), embed the watermark information based on formula (3), and generate the information watermark template using iDFT; where L1 represents the side length, W1(j) represents the jth element in W1, l1 represents the length of W1, R represents the intermediate frequency radius, which in this example is (132, 145, 159, 172), M1(x, y) represents the amplitude coefficient after embedding the watermark, and k1 represents a constant, indicating the watermark embedding strength;

[0067]

[0068]

[0069] Step 15: The screen size is M×N, and the synchronization watermark block size L2×L2 is obtained based on formula (4);

[0070] L2=floor(min(M,N)*0.06) (4)

[0071] The above-mentioned proportional coefficients for generating L1 and L2 are set based on the template layout and the size ratio between the information watermark and the synchronization watermark, and can be fine-tuned.

[0072] Step 16: Consider the zero matrix of size L2×L2 as the Fourier transform domain amplitude map of the synchronization watermark block, select the radius 20, 24 and 28 in the first quadrant of the DFT domain amplitude spectrum, the angle 35° to 55°, with an interval of 5°, a total of 15 coefficients, take 15 coefficients at the corresponding position in the third quadrant, embed the watermark information based on formula (5), and iDFT generates a synchronization watermark template b; select the radius 20, 24 and 28 in the second quadrant of the DFT domain amplitude spectrum, the angle 125° to 145°, with an interval of 5°, a total of 15 coefficients, take 15 coefficients at the corresponding position in the fourth quadrant, embed the watermark information based on formula (5), and iDFT generates different synchronization watermark templates a; where M2(x, y) represents the amplitude coefficient after embedding the watermark, and k2 represents a constant, which represents the watermark embedding strength;

[0073] M2(x,y)=k2 (5)

[0074] Step 17: The screen size is M×N. The information watermark position is determined according to formulas (6) and (7), where c1 and c2 represent the number of rows and columns that can be accommodated, respectively. The screen image can accommodate a total of c1×c2 information watermarks; m1 and n1 represent the row and column indexes of all c1×c2 information watermarks, respectively. The remaining uncovered area of ​​the screen watermark template is filled with an incomplete information watermark.

[0075]

[0076]

[0077] Step 18: Determine the position of the synchronization watermark a based on formula (8); where m2 and n2 represent the row and column indexes in the synchronization watermark.

[0078]

[0079] Step 19: Evenly embed three synchronization watermarks b between every two synchronization watermarks a.

[0080] 2) Watermark embedding: Get the function address of the swap chain swap function in the desktop window manager process, get the back buffer address of the desktop window manager swap chain, use a simplified just noticeable difference model, calculate the JND threshold of the back buffer data, and embed the pre-generated screen watermark template directly into the back buffer according to the threshold. The specific steps are as follows: Figure 3 As shown, including:

[0081] Step 21: Obtain the addresses of the Present function and the GetFrameStatisticInternal function in the dynamic link library to be injected by continuous matching of machine codes;

[0082] Step 22: The Present function is hooked by inline hooking;

[0083] Step 23: Calculate the swap chain address. The swap chain base address can be directly obtained by the Present function address offset. The swap chain constant is obtained by the GetFrameStatisticInternal function address offset. The swap chain pointer address is obtained according to the swap chain constant offset, and then the back buffer address to be displayed is obtained.

[0084] Step 24: Based on the back buffer address, copy the back buffer texture to the temporary buffer:

[0085] Step 25: Bind the OpenCV object to the OpenCL buffer object, and then use OpenCL's texture data reading method to write the texture data to the 2D image object;

[0086] Step 26: Use OpenCV and calculate the average background brightness Bg of the two-dimensional image object bg based on formula (9), where (x, y) represents the position of the pixel, h(n, m) represents a weighted 5×5 low-pass filter, and the weight coefficient is 32, as shown in formula (10);

[0087]

[0088]

[0089] Step 27: Calculate the brightness masking threshold P based on the average background brightness and formula (11) L ;

[0090]

[0091] Step 28: Based on the Sobel operators Sob1 and Sob2 of the 5*5 template shown in formula (12) and formulas (13) and (14), calculate the horizontal gradient G1 of the image and the vertical gradient G2 of the image;

[0092]

[0093]

[0094]

[0095] Step 29: Calculate the image gradient G based on formula (15);

[0096]

[0097] Step 210: Calculate the function α(x, y) and k(x, y) related to the background brightness Bg based on formula (16) and formula (17);

[0098] α(x,y)=0.0001*Bg(x,y)+0.115 (16)

[0099]

[0100] Step 211: Calculate the contrast masking threshold P of the screen image bg based on formula (18): c , where γ represents a parameter used to weigh the objective quality and perceptual characteristics of the image;

[0101] P c(x,y)=G(x,y)*γ*α(x,y)+k(x,y) (18)

[0102] Step 212: Based on formula (19), the brightness masking threshold P L and contrast masking threshold P c Combined, the JND threshold of the image is calculated.

[0103] JND(x,y)=P L (x,y)+P c (x,y)-0.3*min(P L (x,y),P c (x,y)) (19)

[0104] Step 213: Calculate the two-dimensional image object I based on formula (20) b , where M w represents the binary watermark template, JND represents the just noticeable difference, η represents the watermark embedding strength, and finally the two-dimensional image object is written to the bound buffer;

[0105] I b (x,y)=Bg(x,y)+M w (x,y)*JND(x,y)*η (20)

[0106] Step 214: write the processed data from the OpenCL buffer object back to the back buffer texture to achieve the update after the watermark is drawn;

[0107] Step 215: Write the dynamic link library into the desktop window manager process by injection, so that the screen watermark embedding process runs in the desktop window manager.

[0108] 3) Watermark extraction and detection: When the photo image of the shooting screen is obtained, the multi-scale synchronous watermark template is detected to obtain the position of the first synchronous watermark template and the second synchronous watermark template, and automatic perspective correction is implemented based on the position of the synchronous watermark template, and the information watermark template embedding area is located to obtain the embedded user information. The specific steps are as follows: Figure 4 As shown, including:

[0109] Step 31: Obtain the captured image I and convert it into a brightness image I a ;

[0110] Step 32: Image I a Scale to 100%, 90%, 80%, 70%, and 60% for a total of 5 scales, and calculate the screen brightness image I at each scale based on formula (21) i The noise component I n , where Hw Represents a 5×5 spatial domain Wiener filter;

[0111] I n =I a -H w I a (twenty one)

[0112] Step 33: On each scale image, use a 86×86 pixel window with a 25 pixel interval to n Perform slider detection one by one. When each window is detected, I n Convert to DFT domain;

[0113] Step 34: In the Fourier transform domain amplitude diagram of each slider i, record the 15 largest amplitude values ​​in the first quadrant 30° to 60°, the second quadrant 120° to 150°, and the radius range 18 to 30. According to whether these 15 values ​​are in the second quadrant or the first quadrant, record them as V l (i) and V r (i) Calculate the slider window synchronization response value R based on formula (22) s,i , where R s,i represents the synchronization response index of the i-th detection window on the s scale, V l (i) and V r The number of recorded amplitude values ​​in (i) is denoted as N l,1 and N r,1 ;

[0114]

[0115] Step 35: Calculate the synchronous response index of each scale based on formula (23), where R s represents the synchronous response value of scale s (100%, 90%, 80%, 70% and 60%), σ s Indicates the scale R s,i The standard deviation of

[0116] R s =σ s *s 2 (twenty three)

[0117] Step 36: Based on formula (24), calculate R s Maximum R m The corresponding scale s m .

[0118] R m =max(R s ) (twenty four)

[0119] Step 37: Based on formula (25), select R m,i The area is greater than the set threshold T m The peak point is taken as the synchronous response point P m,i ,in Represents R m,i The mean of

[0120]

[0121] Step 38: Retrieve all synchronous response points P m,i , find all the point sets that can be used to correct the information watermark position according to the following method, each point set contains four points that can form a quasi-rectangle. m,i Select points P in descending order of their synchronous response values a In P m,i Among the remaining points in b , so that The angle between the X axis is less than 35 degrees and has a minimum Similarly, find a point P c ,make The angle with the Y axis is less than 35 degrees and has a minimum value Find a point P d , so that Between and between, Greater than and But less than and If four points are found successfully, then in the set S a The nth point set is recorded in n represents the number of points found. Continue this process for the next point until all points are searched. a The maximum synchronous response value of the four points in each point set is calculated in descending order. a Sort and output the obtained S a ;

[0122] Step 39: Cut out the point set A slightly larger area with four points and a perspective transformation;

[0123] Step 310: In this region, use a window of the same size of 86×86 pixels with a sliding interval of 25 pixels and select R according to equations (22) and (25). m,i Less than the specified threshold -T m The peak point P b,i As the synchronization response point of the synchronization watermark b.

[0124] Step 311: For the 16 points representing synchronization watermarks a and b around the message watermark, a window of 86×86 pixels and a sliding interval of 2 pixels are used to perform synchronization response detection in a 150×150 area centered on them. For the synchronization response point of a, the largest 15 amplitude values ​​V in the second quadrant radius of 18 to 30 and angle of 120° to 150° are recorded. a (i) For the synchronous response point b, record the 15 largest amplitude values ​​V in the first quadrant with a radius of 18 to 30 and an angle of 30° to 60°. b (i) Based on formula (26), calculate the precise correction synchronization response index of each coordinate;

[0125] R i,i =sum(V a (i))

[0126] R b,i =sum(V b (i)) (26)

[0127] Step 312: taking the coordinates of the maximum precise correction synchronous response indexes of the 16 regions as the precise correction results, and recording them as a new point set S′;

[0128] Step 313: Perform perspective correction on the 16 points in S′ and extract the information watermark area.

[0129] Step 314: Extract the information block and convert it to the DFT domain using the noise component of its brightness image. Record the maximum amplitude value V within the 3×3 range centered at each watermark embedding position. max (j), where j represents the jth watermark bit; based on formula (27), the extracted watermark information bit W′(j) is determined, where and σ T They represent the mean and standard deviation of all amplitude coefficient values ​​in the range of [130,175] respectively, and k3 represents a constant parameter.

[0130]

[0131] Step 315: Perform QR decoding on W′(j) to obtain watermark information.

[0132] A screen image is selected as experimental data to illustrate the entire process of watermark template generation, embedding and detection. In this embodiment, a screen image with 2560×1440 pixels is selected as experimental data.

[0133] (1) Watermark template generation

[0134] Step 1: Read the watermark information W0: '0101341391040363291748202305201605' consisting of key information such as user ID, IP address, and time, and generate a QR code image based on the QR code algorithm (such as Figure 5 ), record the coding elements of the QR code coding area line by line as a binary watermark information sequence W1[0,0,0,1,0,0,1,…,1,1,1,0,1,0,1,0];

[0135] Step 2: Calculate the information watermark block size to be 676*676;

[0136] Step 3: In the Fourier transform domain amplitude diagram of the information watermark block, the information watermark embedding radius is set to 132, 145, 159, 172, the watermark embedding strength k1 = 10000, and the amplitude coefficient M1 (x, y) after embedding the watermark is obtained according to the formula. iDFT generates the information watermark template (such as Figure 6 (a));

[0137] Step 4: Calculate the synchronization watermark block size to be 86*86;

[0138] Step 5: In the Fourier transform domain amplitude diagram of the synchronization watermark block, the synchronization watermark embedding radius is set to 20, 24 and 28, the watermark embedding strength k2 = 20000, and the amplitude coefficient M2 (x, y) after embedding the watermark is obtained according to the formula. iDFT generates the synchronization watermark template (such as Figure 6 (b));

[0139] Step 6: According to the screen size of 2560*1440, combine the information watermark block and the synchronization watermark block to generate the watermark template T w (like Figure 7 );

[0140] (2) Watermark template embedding

[0141] Step 1: Get the addresses of the Present function and the GetFrameStatisticInternal function in the dynamic link library to be injected by matching the machine code continuously;

[0142] Step 2: Hook present function

[0143] Step 3: Calculate the swap chain address. The swap chain base address can be directly obtained by the Present function address offset. The swap chain constant is obtained by the GetFrameStatisticInternal function address offset. The swap chain pointer address is obtained according to the swap chain constant offset, and then the back buffer address to be displayed is obtained.

[0144] Step 4: Calculate the JND threshold [[3.68019723 3.57055375 3.40016583...3.37252749 3.521383043.64024025] [3.53609403 3.35260418 3.0422761 ... 3.11346151 3.36736762 3.5419132] [3.45709342 3.16986167 2.35125309 ... 2.28881704 3.11300685 3.411417] ...;

[0149] Step 5: Write the texture data back to the back buffer;

[0150] Step 6: Write the dynamic link library into the desktop window manager process by injection, so that the screen watermark embedding process runs in the desktop window manager; (such as Fig. 9 )

[0151] (3) Watermark extraction and detection

[0152] Step 1: Based on the captured image (such as Fig.10 ), obtain 5 noise components of different scales I n ;

[0153] Step 2: For each scale I n Perform synchronous watermark a detection; (such as Fig.11 ) [[28601.2521 34568.5541...3346.3155 2254.5559] [11642.8847 7852.4426...41553.3786 38642.5164] [7768.4298 5568.3495...2256.8843 3486.3315]

[0157] …;

[0158] Step 3: Calculate R for each scale s (45376.5512, 33846.8859, 29764.3546, 30346.8721, 41067.3492), select the best scale s m ;

[0159] Step 4: At the optimal scale s m Under this condition, calculate the synchronous response threshold T m(55894.6721), get the synchronous response point P m,i ;

[0160] Step 5: Retrieve all P m,i , construct the set S a ;

[0161] Step 6: Based on point set Perform perspective transformation; (such as Fig.12 (a))

[0162] Step 7: According to the synchronization response threshold - T m , get the synchronization watermark b response point.

[0163] Step 8: Get the response value V of the synchronized watermarks a and b a (i) and V b (i) Obtain the precise coordinates of a and b based on the precise correction synchronization response index; [[54521.2611 45887.5656...60752.4385 62334.7168] [26842.7042 19842.4832...50273.4618 47615.3419] [18035.4765 8642.5437...9641.4795 13476.7349]

[0167] Step 9: Perspective transformation based on precise coordinates (such as Fig.12 (b)), extract the watermark area and obtain the watermark information W′(j)[0,0,0,1,0,0,1,…,1,1,1,0,1,0,1,0];

[0168] Step 10: QR decoding of W′(j) (e.g. Fig.13 ), get the watermark information. (such as Fig.14 )

[0169] A computer system disclosed in an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the above-mentioned real-time screen watermarking method for tracing screen shooting leaks are implemented.

Claims

1. A real-time screen watermarking method for tracing screen shooting leaks, characterized in that: The steps include: Generate a watermark template, including an information watermark template, a first synchronization watermark template and a second synchronization watermark template; the information watermark template is generated by QR encoding of user information including user ID, IP address and operation time through inverse Fourier transform; the first synchronization watermark template and the second synchronization watermark template are generated by inverse Fourier transform from preset synchronization watermark information, wherein the information of the first and second synchronization watermark templates are respectively embedded in the second and fourth quadrants and the first and third quadrants of the Fourier transform domain amplitude spectrum at preset radii and angles; According to the screen size, the information watermark template and the synchronization watermark template are adaptively combined to generate a screen watermark template; Obtaining the function address of the swap chain swap function in the desktop window manager process, obtaining the back buffer address of the desktop window manager swap chain, using the just noticeable difference model to calculate the JND threshold of the back buffer data, and directly embedding the pre-generated screen watermark template into the back buffer according to the threshold; when the just noticeable difference model calculates the JND threshold, the following Sobel operator is used to respectively calculate the horizontal gradient G1 and the vertical gradient G2 of the two-dimensional image in the buffer, so as to obtain the contrast masking threshold based on the gradient and the background brightness; Finally, based on the formula JND(x,y)=P L (x,y)+P c (x,y)-a*min(P L (x,y),P c (x,y)) calculates the JND threshold, where P L (x,y),P c (x, y) represents the brightness masking threshold and contrast masking threshold at the position (x, y), respectively, and a is the coefficient; When a photo image of the shooting screen is obtained, a multi-scale synchronous watermark template is detected to obtain the positions of the first synchronous watermark template and the second synchronous watermark template, and automatic perspective correction is implemented based on the positions of the synchronous watermark templates, and the information watermark template embedding area is located to obtain the embedded user information.

2. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The watermark template generation includes the following steps: The user information is used to generate a QR code using a QR code algorithm, and the coding elements in the QR code coding area are recorded line by line as a binary watermark information sequence W1; According to the screen size M×N, determine the information watermark block size L1×L1 and the synchronization watermark block size L2×L2, L1>L2; The L1×L1 zero matrix is ​​regarded as the Fourier transform domain amplitude map of the information watermark block. Determine the location of embedded watermark information based on Embed watermark information and generate information watermark template through iDFT; where W1(j) represents the jth element in W1, l1 represents the length of W1, R represents the intermediate frequency radius, M1(x, y) represents the amplitude coefficient after embedding the watermark, and k1 is a constant representing the watermark embedding strength; The L2×L2 zero matrix is ​​regarded as the Fourier transform domain amplitude map of the synchronization watermark block. At least 15 coefficients are set as the carriers of the synchronization watermark information at at least 3 preset radii and 5 equally spaced angles in the second and fourth quadrants. The watermark information is embedded based on M2(x, y)=k2. The first synchronization watermark template a is generated through iDFT. At least 15 coefficients are set as the carriers of the synchronization watermark information at at least 3 preset radii and 5 equally spaced angles in the first and third quadrants. The watermark information is embedded based on M2(x, y)=k2. The second synchronization watermark template b is generated through iDFT; wherein M2(x, y) represents the amplitude coefficient after the watermark is embedded, and k2 is a constant representing the watermark embedding strength.

3. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The adaptive combination of the information watermark template and the synchronization watermark template to generate the screen watermark template is specifically: Assume the screen size is M×N, according to the formula Determine the information watermark template position, where and They represent the number of rows and columns that can be accommodated, L1 represents the side length of the information watermark template; m1 and n1 represent the indexes of the rows and columns in all c1×c2 information watermarks, respectively; Based on the formula Determine the position of the first synchronization watermark template a; wherein m2, n2 represent the index of the row and column in the synchronization watermark, and L2 represents the side length of the synchronization watermark template; Between every two first synchronization watermark templates a, three second synchronization watermark templates b are uniformly embedded.

4. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The method for obtaining the back buffer address of the desktop window manager swap chain is as follows: obtaining the addresses of the Present function and the GetFrameStatisticInternal function in the dynamic link library to be injected by continuous matching of machine codes; hooking the Present function by an inline hook method; calculating the swap chain address, the swap chain base address is directly obtained by offsetting the Present function address, the swap chain constant is obtained by offsetting the GetFrameStatisticInternal function address, the swap chain pointer address is obtained according to the swap chain constant offset, and then the back buffer address to be displayed is obtained.

5. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The two-dimensional image object I in the buffer after writing is calculated based on the following formula b : I b (x,y)=Bg(x,y)+M w (x,y)*JND(x,y)*η Among them B g represents the average background brightness, M w represents the binary watermark template, and η represents the watermark embedding strength.

6. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The detection of the multi-scale synchronous watermark template comprises the following steps: Convert the captured image I into a brightness image I a ; Image I a Scale to multiple scales and compute image I at each scale a The noise component I n ; On each scale image, a window of preset pixel size is used to identify I n Perform slider detection one by one; In the Fourier transform domain amplitude diagram of each slider i, the maximum specified number of amplitude values ​​within the preset angle range of the first quadrant, the preset angle range of the second quadrant, and the preset radius range are recorded, and the amplitude values ​​located in the second quadrant or the first quadrant are recorded as V respectively. l (i) and V r (i), based on the formula Calculate the slider window synchronization response value, where R s,i represents the synchronous response index of the detection window corresponding to the i-th slider on scale s, N l,1 and N r,1 Respectively represent V l (i) and V r (i) The number of recorded amplitude values; Determine the optimal scale according to the synchronous response index of each scale, and determine the synchronous response point of each window in the optimal scale; Retrieve all synchronous response points, find all point sets that can be used to correct the information watermark position, each point set contains four points that form a quasi-rectangle, cut out the area formed by the point set, and perform perspective transformation; In the clipped area, a synchronization response index is calculated using a sliding window to determine a synchronization response point of the second synchronization watermark template.

7. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The method of realizing automatic perspective correction based on the synchronous watermark template position comprises the following steps: For the synchronization response points representing the first and second synchronization watermark templates around the message watermark template, in the area centered on them, the sliding window interval is reduced to perform synchronization response detection; for the synchronization response points of the first synchronization watermark template, the maximum preset number of amplitude values ​​within the preset radius and angle range of the second quadrant is recorded; for the synchronization response points of the second synchronization watermark template, the maximum preset number of amplitude values ​​within the preset radius and angle range of the first quadrant is recorded; the precision correction synchronization response index of each coordinate is calculated based on the following formula: R a,i =sum(V a (i)) R b,i =sum(V b (i)) Where V a (i) and R a,i are the amplitude value set of the detection window corresponding to the i-th slider of the first synchronization watermark template and the precise correction synchronization response index, V b (i) and R b,i are respectively the amplitude value set of the detection window corresponding to the i-th slider of the second synchronization watermark template and the precise correction synchronization response index; The coordinates of the regional maximum fine-calibrated synchronous response index are perspective transformed, and the transformed coordinates are used as the result of fine calibration.

8. The real-time screen watermarking method for tracing screen shooting leaks according to claim 1 is characterized in that: The method for obtaining embedded user information is: Extract the information block, use the noise component of its brightness image, and convert it to the DFT domain; record the maximum amplitude value V within the 3×3 range centered at each watermark embedding position max (j), where j represents the jth watermark bit; based on the formula Determine the extracted watermark information bit W′(j), where and σ T They represent the mean and standard deviation of all amplitude coefficient values ​​within the embedding range respectively, and k3 is a constant. Perform QR decoding on W′(j) to obtain the user watermark information.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into a processor, the steps of the real-time screen watermarking method for tracing screen shot leaks according to any one of claims 1 to 8 are implemented.

Images