A watermark updating method suitable for multi-stage transmission process

By using a stream-based codec (FED) and analog noise layer processing, combined with reversible neural blocks and loss function optimization, the problems of watermark update and source tracing accuracy during multi-level transmission are solved, achieving efficient watermark extraction and image restoration during screen capture.

CN118741149BActive Publication Date: 2024-11-12UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411123468.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-12
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

Existing digital watermarking schemes cannot update watermarks during multi-level transmission, leading to a decrease in the accuracy of traceability. Furthermore, traditional methods lack robustness during screen capture and cannot effectively resist complex distortions.

Method used

A stream-based codec (FED) is employed, which combines forward encoding and backward decoding with analog noise layer processing to update and restore watermark information. Reversible neural blocks and loss functions are used to optimize the watermark extraction process, ensuring robustness and source tracing accuracy.

Benefits of technology

The watermark is effectively updated during multi-level transmission, improving the accuracy of source tracing, resisting complex distortions during screen capture, and ensuring the accuracy of watermark information extraction and image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118741149B_ABST
    Figure CN118741149B_ABST
Patent Text Reader

Abstract

The present invention discloses a watermark update method suitable for a multi-level transmission process. First, the watermark information M is input into a stream-based codec FED together with the original image, and the watermarked image is output after forward encoding; the watermarked image is subjected to a simulated noise layer to obtain a noise image; the noise image and the all-zero matrix are input into a stream-based codec FED, and the restored image and watermark information are output after backward decoding, and the restored image is used as a carrier for watermark update. The above method solves the watermark update problem that cannot be achieved by previous methods in the multi-level data flow process, and improves the traceability accuracy while ensuring robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of digital watermarking, and in particular to a watermark updating method suitable for a multi-stage transmission process. Background Art

[0002] The so-called digital watermark is to embed copyright protection information, i.e. watermark, into the original data to ensure the ownership of the data information. The embedded watermark can be a piece of text, logo, serial number, image and other digital signals. Moreover, this watermark is usually invisible. It is closely integrated with the original data and hidden in it. It can be preserved after undergoing some operations that do not destroy the use value or commercial value of the original data. When necessary, the watermark can be extracted through certain technical detection methods as evidence for determining the ownership of media copyright and tracking and prosecuting illegal infringement. Digital watermark provides an effective technical means for multimedia data security in terms of authentication, anti-counterfeiting, anti-tampering, and ensuring data security and integrity. Especially in places with high risk of leakage such as public networks, digital watermarks can track and trace the leaked data in a timely manner.

[0003] Traditional digital watermarking schemes often design algorithms based on distortion analysis. For example, in order to resist JPEG distortion, some algorithms choose to embed watermarks in the DCT domain. However, with the development of digital devices, screen capture has become a major source of data leakage. The screen capture process is extremely complex, and it is difficult to quantitatively analyze the distortion produced in it. Therefore, it is a challenge to design an algorithm to ensure robustness under this distortion. For traditional screen capture elastic watermarking algorithms, they only qualitatively analyze the distortion that may occur during the screenshot process and use some general watermark embedding methods to ensure robustness. However, such methods often produce severe visual distortion because the embedding method is non-adaptive and cannot fully utilize the characteristics of the carrier image.

[0004] The above watermarking schemes only focus on the watermark information finally extracted, and do not care about the image after the extraction process is completed. However, in the multi-level flow process, data is usually transmitted along one or more transmission chains composed of multiple users. In order to locate the source of the leakage more accurately, the embedded watermark should contain the information of the sender and receiver in the current level transmission process, that is, the watermark information embedded in different transmission processes is different, and these different watermarks will interfere with each other during extraction. If different watermarks are embedded in the same image multiple times, the extraction accuracy will be greatly reduced, which will lead to traceability failure. Summary of the invention

[0005] The purpose of the present invention is to provide a watermark updating method suitable for a multi-level transmission process, which solves the watermark updating problem that cannot be achieved by previous methods during the multi-level flow of data, and improves the traceability accuracy while ensuring robustness.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A watermark updating method applicable to a multi-stage transmission process, the method comprising:

[0008] Step 1: Combine the watermark information M with the original image Input the stream-based codec FED together, forward encode and output the watermarked image ;

[0009] Step 2: Watermarked Image After simulating the noise layer, the noise image is obtained ;

[0010] Step 3: Noise image and all-zero matrix input stream-based codec FED, output the restored image after backward decoding and watermark information , and the restored image As a carrier for watermark updates.

[0011] It can be seen from the technical solution provided by the present invention that the above method solves the watermark update problem that cannot be achieved by previous methods during the multi-level flow of data, and improves the traceability accuracy while ensuring robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic flow chart of a watermark updating method applicable to a multi-stage transmission process provided by an embodiment of the present invention;

[0014] Figure 2 is a schematic structural diagram of the i-th reversible neural block according to an embodiment of the present invention;

[0015] Figure 3 A schematic diagram of a network implementation framework of the method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] like Figure 1 FIG. 1 is a flow chart of a watermark updating method applicable to a multi-stage transmission process provided by an embodiment of the present invention, wherein the method comprises:

[0018] Step 1: Combine the watermark information M with the original image Input the stream-based codec FED together, forward encode and output the watermarked image ;

[0019] In this step, the flow-based model consists of two parts: the forward encoding function and have the same parameters The inverse function of ; In the specific implementation, during the forward encoding process, the watermark information matrix is ​​received and the original image As input, output watermarked image and the redundancy matrix ; In the backward decoding process, the all-zero matrix and noisy image Input to the stream-based codec FED to extract the watermark information And get the restored image ;in and is the height and width of the message tensor, and they satisfy the relationship , is the length of the watermark message; and are the height and width of the image respectively;

[0020] The stream-based codec FED consists of n reversible neural blocks, such as Figure 2 FIG. 1 is a schematic diagram of the structure of the i-th reversible neural block according to an embodiment of the present invention, which consists of an upper subnetwork and two sub-networks , , i=1, 2...n; where, The role of watermark information Upsample to the original image Same size; and The role of Downsample to Same size; is the output of the i-th reversible neural block in the forward encoding process. Its size is consistent with the image size and can be regarded as the intermediate result of encoding;

[0021] For the i-th reversible neural block in the forward encoding process, the input is and , the output is and , which is expressed as follows:

[0022] ;

[0023] in Represents the dot product operation;

[0024] After n reversible neural blocks, we get and , corresponding to the watermarked images and the redundancy matrix .

[0025] Step 2: Watermarked Image After simulating the noise layer, the noise image is obtained ;

[0026] In this step, since the distortion of the screen photography process is extremely complex and difficult to simulate or quantitatively analyze, only the distortions with the greatest impact in this process are considered, and they are mixed as combined noise for training, which will bring stronger robustness. After investigating the screen photography process, this application summarizes the following three most influential distortions: perspective distortion, illumination distortion, and moiré distortion. The above three distortions can all be represented in a differentiable way, and Gaussian noise is used as an approximation for other distortions in the process, specifically:

[0027] The simulated noise layer includes perspective distortion, illumination distortion, moiré distortion and Gaussian noise, wherein:

[0028] Perspective distortion comes from the change of shooting angle. Different shooting angles will cause different deformations of the captured pictures. The effect of perspective distortion is simulated by randomly generating homography mapping. The four vertices of the original image are represented as , , , ; The image vertices after perspective distortion are represented as , , , ; The perspective distortion is simulated by randomly perturbing the four corners (up to ±8 pixels). The four corners before the perturbation and the four corners after the perturbation correspond to each other in the following homography:

[0029] ;

[0030] Substituting the coordinates of the above 8 points into the equation, we can obtain the parameters of the homography mapping function used to simulate transmission distortion, namely: ;

[0031] When all parameters are confirmed, the remaining points in the image will also be mapped to obtain perspective distorted points; input image with watermark After that, each pixel is bilinearly sampled into the perspective distorted image to obtain the perspective distorted image. ;

[0032] In the process of screen photography, due to the influence of ambient light and screen photography, different shooting conditions will lead to different light distribution in the captured image. Point diffusion distribution and line diffusion distribution are used to simulate two lighting conditions: point light source and line light source respectively; specifically:

[0033] For the distortion caused by point light sources, the following matrix is ​​generated , expressed as:

[0034]

[0035] in are the coordinates of the weight matrix; are the coordinates of a point light source randomly selected in the image area; Indicates the maximum distance from the point light source to the four vertices of the image; is the minimum illumination change rate; is the maximum illumination change rate, which can be uniformly sampled in [0.7, 0.9] and [1.1, 1.3] respectively in the specific implementation.

[0036] For the distortion caused by the line light source, the following matrix is ​​generated , expressed as:

[0037] in:

[0038] ;

[0039] Respectively indicate that The matrix obtained after rotating 90°, 180°, and 270°; is from Randomly selected from is the height of the image; the final lighting distortion is from Randomly selected from

[0040] Moiré distortion refers to the irregular texture that appears in the captured image due to the difference in sampling frequency between the screen and the camera. It is the most special distortion in the screen shooting process and is simulated using the following mathematical formula:

[0041] ; ; ;

[0042] in is the coordinate of the pixel in the image; , are points calculated using different functions The molar distortion component of is the smaller component selected from the two components; The final calculated point The value of moiré distortion;

[0043] For calculation function For example, we first randomly select a coordinate in the image. Then, the cosine function is used to calculate the moiré distortion component with this point as the center, and randomly Select Parameters Perform calculations;

[0044] The final noisy image obtained after the simulated noise layer It is expressed by the following formula:

[0045]

[0046] in represents Gaussian noise ( ); and Indicates the degree of distortion of the corresponding distortion, the default settings are 0.85 and 0.15; For lighting distortion; It is perspective distortion; Moiré distortion.

[0047] Step 3: Noise image and all-zero matrix input stream-based codec FED, output the restored image after backward decoding and watermark information , and the restored image As a carrier for watermark updates.

[0048] In this step, the noise image and all-zero matrix input stream-based codec FED, output the restored image after backward decoding and watermark information , the specific process is:

[0049] For the backward decoding process, information flows from the i+1th reversible neural block to the ith reversible neural block. For the ith reversible neural block, the input of the stream-based codec FED is and , the output is and , which is expressed as follows:

[0050] ;

[0051] in, Corresponding to the all-zero matrix ; Corresponding noise image ;

[0052] After passing through all neural blocks, the final output and , which is the watermark information and the restored image .

[0053] Furthermore, the restored image The process of updating watermark as the carrier of watermark update is as follows:

[0054] User A receives a picture from the previous user. Then decode it and extract the watermark information and the restored image ; Used to verify the identity of the parent user and ensure the source of the image is correct; As watermark information Embedded vector, get the picture And transmit it to the next level user B; It is a new watermark encoded according to the identity information of users A and B. So far, a round of watermark update has been completed;

[0055] When using FED to implement watermark update, since the system is completely reversible, the input noise-free watermark image With redundant information You can output watermark information With the original image ; Although redundant information It plays a great role in the watermark update process, but its storage and transmission are difficult in practice, which also leads to The complete restoration of the image becomes difficult, but if only copyright protection is considered, the restored image can be restored without affecting the watermark update. Constrain the quality to make it as close to the original image as possible ;

[0056] Leak tracing and watermark updating can be regarded as two different tasks. Due to their different purposes, the corresponding loss function compositions are also different. In the process of leak tracing, the purpose is to ensure high-precision extraction of watermark information when taking pictures of the screen, and at the same time, the watermark needs to be invisible. The loss function composition is as follows:

[0057] ; ;

[0058] in, is the total loss function, which is composed of message loss And image loss Composition; message loss Using mean square error To calculate; image loss By mean square error And the perceptual loss function Joint calculation; , , is the weight of each loss; is the watermark information of the original image; To restore the watermark information of the image;

[0059] During the watermark update process, attention should be paid to the restored image The quality of The quality constraints are as follows:

[0060] ; in, To restore the image The quality constraints Through the perception function to calculate; is the weight of the quality constraint.

[0061] Different from traditional pixel-level evaluation indicators (such as PSNR and SSIM), the perceptual loss LPIPS uses a convolutional neural network to extract features from images and calculate the differences between images in the feature space, which can better capture the similarities perceived by the human visual system. For , it is mainly to use the perceptual loss to focus on higher-level features, which helps to restore the clean image we want; for For the watermark, it uses its similar imitation of human eye perception to improve the invisibility of the watermark.

[0062] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0063] The effect of the method of the present invention is described below with specific examples. Figure 3 The diagram shows a schematic diagram of a network implementation framework of the method according to an embodiment of the present invention, in which the rightward arrow indicates the forward encoding process, and the leftward arrow indicates the reverse decoding process.

[0064] This example embeds a watermark message of length L=64 in a picture of size 128*128, then tries to update the watermark 7-12 times, and extracts the watermark under unbiased shooting conditions at a distance of d=50cm to test its robustness. The following evaluation indicators are mainly used to evaluate the performance of the method:

[0065] Visual quality: Peak signal-to-noise ratio (PSNR), the higher the value, the better the visual quality;

[0066] Robustness: Extraction accuracy (ACC), larger values ​​indicate better robustness.

[0067] The results of 7 watermark updates are recorded in Table 1 below:

[0068] Table 1 Watermark update results

[0069] Round PSNR ACC (without screen-shooting) ACC (under screen-shooting) 1 40.747 99.8% 93.1% 2 40.494 100% 96.2% 3 40.200 100% 97.0% 4 40.353 100% 93.8% 5 40.248 100% 95.6% 6 40.007 100% 97.1% 7 40.441 100% 97.2%

[0070] Judging from the test results in Table 1, the watermark extraction accuracy in the screen shooting environment remains above 93%, successfully resisting the distortion caused by screen photography; on the other hand, the successful extraction of the watermark also shows that the existence of the old watermark does not interfere with the embedding and extraction of the new watermark, and the watermark update is successfully performed.

[0071] In summary, the method described in the embodiment of the present invention can use the same network to effectively implement forward encoding and backward decoding, so the encoder and decoder share the same parameters, ensuring the tight coupling between the encoder and the decoder; and because the entire architecture is reversible, while extracting the watermark, a picture similar to the original picture can be restored as a carrier for watermark update.

[0072] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0073] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A watermark updating method suitable for a multi-stage transmission process, characterized in that: The method comprises: Step 1: Combine the watermark information M with the original image Input the stream-based codec FED together, and output the watermarked image after forward encoding ; In step 1, the flow-based model consists of two parts: the forward encoding function and have the same parameters The inverse function of ; The stream-based codec FED consists of n reversible neural blocks, and the i-th reversible neural block includes an upper subnetwork and two sub-networks , , i=1, 2...n; where, The role of watermark information Upsample to the original image Same size; and The role of Downsample to Same size; is the output of the i-th reversible neural block in the forward encoding process; and is the height and width of the message tensor, and they satisfy the relationship , is the length of the watermark message; and are the height and width of the image respectively; For the i-th reversible neural block in the forward encoding process, the input is and , the output is and , which is expressed as follows: ; in Represents the dot product operation; After n reversible neural blocks, we get and , corresponding to the watermarked images and the redundancy matrix ; Step 2: Watermarked Image After simulating the noise layer, the noise image is obtained ; Step 3: Noise image and all-zero matrix input stream-based codec FED, output the restored image after backward decoding and watermark information , and the restored image As a carrier for watermark updates; In step 3, the noisy image and all-zero matrix input stream-based codec FED, output the restored image after backward decoding and watermark information , the specific process is: For the backward decoding process, information flows from the i+1th reversible neural block to the ith reversible neural block. For the ith reversible neural block, the input of the stream-based codec FED is and , the output is and , which is expressed as follows: ; in, Corresponding to the all-zero matrix ; Corresponding noise image ; After passing through all neural blocks, the final output and , which is the watermark information and the restored image ; In step 3, the restored image The process of updating watermark as the carrier of watermark update is as follows: User A receives a picture from the previous user. Then decode it and extract the watermark information and the restored image ; Used to verify the identity of the parent user and ensure the source of the image is correct; As watermark information Embedded vector, get the picture And transmit it to the next level user B; It is a new watermark encoded according to the identity information of users A and B. So far, a round of watermark update has been completed; In the process of leak tracing, the goal is to ensure high-precision extraction of watermark information when taking pictures of the screen, and at the same time, the watermark needs to be made invisible. The loss function is composed as follows: ; ; in, is the total loss function, which is composed of message loss And image loss Composition; message loss Using mean square error To calculate; image loss By mean square error And the perceptual loss function Joint calculation; , , is the weight of each loss; is the watermark information of the original image; To restore the watermark information of the image; During the watermark update process, attention should be paid to the restored image The quality of The quality constraints are as follows: ; in, To restore the image The quality constraints Through the perception function to calculate; is the weight of the quality constraint.

2. The watermark updating method suitable for a multi-stage transmission process according to claim 1, characterized in that: In step 2, the simulated noise layer includes perspective distortion, illumination distortion, moiré distortion and Gaussian noise, wherein: Perspective distortion comes from the change of shooting angle. Different shooting angles will cause different deformations of the captured pictures. The effect of perspective distortion is simulated by randomly generating homography mapping. The four vertices of the original image are represented as , , , ; The image vertices after perspective distortion are represented as , , , ; The perspective distortion is simulated by randomly perturbing the four corners. The four corners before the perturbation and the four corners after the perturbation correspond to each other in the following homography mapping relationship: ; Substituting the coordinates of the above 8 points into the equation, we can obtain the parameters of the homography mapping function used to simulate transmission distortion, namely: ; When all parameters are confirmed, the remaining points in the image will also be mapped to obtain perspective distorted points; input image with watermark After that, each pixel is bilinearly sampled into the perspective distorted image to obtain the perspective distorted image. ; When taking photos on the screen, different shooting conditions will result in different light distributions in the captured images. Point diffusion distribution and line diffusion distribution are used to simulate two lighting conditions: point light source and line light source. Specifically: For the distortion caused by point light sources, the following matrix is ​​generated , expressed as: in are the coordinates of the weight matrix; are the coordinates of a point light source randomly selected in the image area; Indicates the maximum distance from the point light source to the four vertices of the image; is the minimum illumination change rate; is the maximum illumination change rate; For the distortion caused by the line light source, the following matrix is ​​generated , expressed as: in: ; Respectively indicate The matrix obtained after rotating 90°, 180°, and 270°; is from Randomly selected from is the height of the image; the final lighting distortion is from Randomly selected from Moiré distortion refers to the irregular texture that appears in the captured image due to the difference in sampling frequency between the screen and the camera. It is simulated using the following mathematical formula: ; ; ; in is the coordinate of the pixel in the image; , are points calculated using different functions The molar distortion component of is the smaller component selected from the two components; The final calculated point The value of moiré distortion; For calculation function For example, we first randomly select a coordinate in the image. Then, the cosine function is used to calculate the moiré distortion component with this point as the center, and randomly Select Parameters Perform calculations; The final noisy image obtained after the simulated noise layer It is expressed by the following formula: in represents Gaussian noise; and Indicates the degree of distortion of the corresponding distortion, the default settings are 0.85 and 0.15; For lighting distortion; It is perspective distortion; Moiré distortion.