Active defense method for deeply counterfeited face by using physical countermeasure watermark

Abstract

The invention discloses a deeply-forged face active defense method for resisting watermarks by using physics, which comprises the following steps of: 1, acquiring an image to be protected and a watermark resisting watermark, and assigning a local area mask; step 2, face key points are extracted from the to-be-protected image by using a dlib library; step 3, based on the extracted face key points and the local area mask, aligning an adversarial watermark to a target area of the to-be-protected image, and obtaining a protected image through pixel-by-pixel multiplication; 4, forming a sample pair by the to-be-protected image and the protected image, and extracting potential spatial features of the sample pair by using an encoder of a counterfeit model; 5, performing multi-constraint optimization on the watermark; and step 6, generating an actual physical watermark through a physical means according to the target area aligned with the countermeasure watermark, and implementing the method. According to the method, data does not need to be protected one by one, and robustness can still be kept after network propagation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an active defense method for deep forgery of human faces using physical adversarial watermarks. Background Art

[0002] In recent years, with the rapid development of deep learning technology, the number of malicious deep fake incidents has continued to rise. Deep fake technology can achieve operations such as editing the attributes of facial images, tampering with identity information, and controlling facial expressions. Malicious attackers use these technologies to spread false information, which not only threatens personal privacy but also undermines social credibility. Although existing detection methods can reliably identify the generated forged images, these methods are passive defenses and cannot prevent forged content before it is generated. Once forged content is generated, its negative impact has begun to appear, and it is even more difficult to control its consequences after it spreads through social networks. To address this problem, some studies have proposed active defense methods that destroy the generation effect of deep fakes by embedding digital adversarial watermarks in images. However, existing active defense methods generally have limitations. The most prominent problem is that digital adversarial watermarks need to be embedded separately for each image to be protected, which makes it difficult to effectively meet the protection needs of massive data in social networks.

[0003] Among them, adversarial samples are carefully designed perturbations imposed on the input image, aiming to interfere with the reasoning process of the deep learning model and cause deviations in its output results. The core principle is that despite the excellent performance of deep learning models, due to their highly linear characteristics, they are vulnerable when processing specific inputs. This vulnerability can be exploited through specially designed input perturbations, causing the model to make incorrect judgments that are inconsistent with expectations. Usually, in digital adversarial images, the perturbation directly acts on the pixel values ​​of the input image, and the digital adversarial perturbation is generated through algorithm optimization to accurately control the interference effect on the model. In physical adversarial, the perturbation exists in the form of the real world, such as printing on the surface of an object or embedding it into a scene, and then re-transmitted to the model through shooting or sensing devices, thereby achieving interference with the model. In the field of active face defense, these adversarial perturbations are called adversarial watermarks. Current research mainly focuses on the form of digital adversarial, that is, by embedding invisible tiny digital adversarial watermarks on the image to be protected. Carefully designed digital adversarial watermarks can effectively interfere with the feature extraction and generation mechanism in the generation process of deep fake models, thereby preventing unauthorized users from generating fake images and achieving active defense effects. However, how to extend adversarial watermarking from digital space to physical space is still an important direction worth exploring.

[0004] Generative adversarial networks are a deep learning framework that excels at generating samples that are similar to real data. Its network structure consists of a generator and a discriminator. The principle of generative adversarial networks is based on the confrontation between the generator and the discriminator. Specifically, during the training process of the network, the generator generates realistic samples as much as possible so that the discriminator cannot distinguish them, while the discriminator tries to distinguish between real data and samples output by the generator as much as possible. In the field of face forgery, generative adversarial networks can learn rich semantic information in the latent feature space. Manipulating this semantic information means that the attributes of the face can be tampered with, thereby achieving the purpose of forgery.

[0005] It can be seen that the current active defense method requires embedding a digital adversarial watermark for each image to be protected. This individualized processing method not only consumes a lot of time and resources when protecting a large amount of data in social networks, but also has difficulty in achieving comprehensive coverage. In addition, the dynamic update of social network content further increases the complexity and difficulty of embedding watermarks one by one. Although some studies have attempted to design more generalized watermarks to shorten training time and support embedding a single watermark into multiple images, these improvements are still difficult to fully meet the needs of practical applications.

[0006] Current digital adversarial watermarks have a problem of insufficient robustness during transmission on social networks. Specifically, when images are uploaded or shared, the interference effect of the watermark is often weakened due to automatic compression or other post-processing operations applied by the platform. In addition, malicious attackers can also use simple image processing techniques to specifically weaken or remove the defense capabilities of the watermark, thereby circumventing the active defense mechanism. This problem of insufficient robustness limits the effectiveness of existing methods in defending against malicious forgeries in actual scenarios. Summary of the invention

[0007] In view of the shortcomings of the existing technology, the present invention first proposes an active defense method for deep fake faces using physical adversarial watermarks. It does not need to protect data one by one, and the method can still maintain robustness after transmission over the network, providing a wider range of application scenarios and more efficient solutions for deep fake defense.

[0008] In order to solve the above technical problems, the technical solution of the present invention is:

[0009] A method for actively defending against deep fake faces using physical adversarial watermarks, comprising the following steps:

[0010] Step 1: Obtain the image to be protected and the initial watermark, and specify the area in the image to be protected that does not affect the key facial features as a local area mask;

[0011] Step 2: Use the face detection and key point extraction model of the dlib library to extract facial key points from the image to be protected;

[0012] Step 3: Based on the extracted facial key points and local area masks, the initial watermark is aligned to the target area of ​​the image to be protected, and the protected image is obtained by pixel-by-pixel multiplication.

[0013] Step 4: The image to be protected and the protected image form a sample pair, and an encoder of a forged model is used to extract the latent space features of the sample pair;

[0014] Step 5: Multiple constraints to optimize watermark;

[0015] Step 6: Generate an actual physical watermark through physical means according to the target area aligned with the adversarial watermark, and implement it.

[0016] Preferably, in step 3, the method for obtaining the protection image is:

[0017]

[0018] in, For the image to be protected, To protect the image, is the local area mask, is the initial watermark, It is a pixel-by-pixel multiplication operation.

[0019] Preferably, the forgery model adopts a GAN model for editing and forging.

[0020] Preferably, the plurality of constrained optimization methods include optimizing watermark parameters of the watermark by means of a feature distance loss function, minimizing the difference between the color in the adversarial watermark and the color in the reference set, and optimizing the edge and texture of the watermark by means of a smoothness loss function.

[0021] Preferably, in step 5, the watermark parameters of the initial watermark are optimized by a feature distance loss function, and the expression is as follows:

[0022] ;

[0023] in, is the distance loss function, To extract features from images using the encoding of the forged model.

[0024] Preferably, in step 5, the method for obtaining the counter watermark through the initial watermark optimized by the feature distance loss function is:

[0025] , ;

[0026] in Statement After rounds of optimization, the adversarial watermark based on The watermark is clipped by the threshold value to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. is the weight coefficient, Gradient of the previous loss with respect to the watermark.

[0027] Preferably, in step 5, the method for minimizing the difference between the color in the adversarial watermark and the color in the reference set is: first, a color reference set is constructed by printing a color palette with multiple colors and collecting the colors. ; Next, each pixel in the adversarial watermark is compared with the closest color in the reference set, thereby minimizing the difference between the color in the adversarial watermark and the color in the reference set.

[0028] Preferably, the method for evaluating the difference between the color in the adversarial watermark and the color in the reference set is:

[0029] The evaluation is performed by calculating the distance between each pixel and the reference color. The evaluation method is as follows:

[0030] , ;

[0031] in, is the color reproduction loss function, Indicates that the horizontal and vertical coordinates of the watermark are and Pixels, Indicates the color from the reference set that is closest to the pixel value.

[0032] Preferably, in step 5, the texture and edge are optimized by a smoothness loss function, and the expression is as follows:

[0033] ;

[0034] in, is the smoothness loss function. Indicates that the horizontal and vertical coordinates of the watermark are and Pixels.

[0035] Preferably, in step 5, the watermark parameters of the adversarial watermark are optimized for smoothness and color reproduction, so that the adversarial watermark can still have an adversarial effect in the physical world, and the expression is as follows:

[0036] , ;

[0037] in Statement After rounds of optimization, the adversarial watermark based on The watermark is clipped by the threshold value to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. is the weight coefficient, Gradient of the previous loss with respect to the watermark.

[0038] The present invention implements active defense from the image acquisition stage by deploying adversarial watermarks in the physical world, without processing each digital image one by one, thereby effectively solving the time and resource consumption problems of traditional active defense methods in large-scale data protection. The present invention specially designs digital adversarial watermarks so that they can be smoothly migrated to the physical world while retaining their defense performance. In addition, a step-by-step constrained loss update strategy is adopted to ensure that the trained watermarks can be effectively implemented and deployed in the physical world. As a defense entity, the physical adversarial watermark has strong robustness and can effectively resist common compression and post-processing operations in social networks, further enhancing its effectiveness and adaptability in actual scenarios. In general, the present invention avoids the large amount of overhead of processing digital images one by one, while having effective defense capabilities and showing good robustness in social network scenarios.

[0039] The present invention has the following characteristics and beneficial effects:

[0040] The present invention proposes an active deep counterfeit defense method that can be deployed in the physical world. It protects information from the source of the data without the need to operate each data separately, greatly improving the practicality and efficiency of the defense.

[0041] Different from the traditional digital watermark form, the present invention designs a local, visible and colorful digital watermark, so that the watermark has the ability to be transferred to the physical world.

[0042] This paper proposes an optimization strategy consisting of three carefully designed loss functions to ensure that the watermark still has the ability to defend against deep forgeries in the physical world.

[0043] The present invention can not only effectively combat deep fakes in the physical world, but also ensure that the protected data still has reliable defense performance and robustness after being transmitted through social networks.

[0044] This method is not only applicable to digital environments, but can also be deployed in the physical world, realizing cross-scenario applications that traditional digital active defense methods cannot achieve.

[0045] This method protects information from the source of data without having to operate the data one by one, which greatly reduces the consumption of time and resources and improves defense efficiency.

[0046] This method demonstrates reliable defense capabilities in both the digital and physical worlds. At the same time, the protected data can still maintain good robust performance after being transmitted through social networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0049] Figure 2 The figure is a schematic diagram of the process of watermark optimization in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0051] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0052] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0053] The present invention provides an active defense method for deep forgery of human face using physical countermeasure watermarks, such as Figure 1 As shown in the figure, the overall framework consists of two parts: digital watermark design and physical watermark implementation. The digital watermark design part aims to develop a watermark with active defense performance and meet the conditions for migration to the physical world; the physical watermark implementation part ensures that the watermark can be successfully migrated to the physical world and realized in practical application. The specific steps are as follows:

[0054] 1. Digital watermark design:

[0055] 1-1. Protected image acquisition: Physical adversarial watermarks need to have a significant and visible form of expression, and should not interfere with the identity characteristics of the protected object. To meet this requirement, the present invention specifies the key areas in the image to be protected that do not affect the facial features as local area masks. In this embodiment, the key areas refer to the facial features. The local area mask is used to standardize the shape of the watermark, ensure that the watermark deployment meets the design requirements, and minimize interference with the identity characteristics.

[0056] 1-2. Facial key point extraction: In order to accurately and automatically deploy the watermark to the face during the training process, we use facial key point positioning technology. The face detection and key point extraction model of the library extracts facial key points from the image to be protected to ensure the precise alignment of the watermark with the image to be protected, providing a reliable basis for subsequent operations.

[0057] Specifically, dlib performs face detection through get_frontal_face_detector() and uses shape_predictor() to extract 68 facial key points to obtain the precise locations of eyes, nose, mouth, and facial contours. For example, taking the adversarial watermark in the shape of glasses as an example, the watermark has a local mask of the corresponding shape, and the center of the eyebrows (number 27) can be specified as the mask center to embed the adversarial watermark.

[0058] 1-3. Watermark alignment and initialization: Based on the extracted facial key points and local area mask, the watermark is accurately aligned to the target area of ​​the image to be protected, that is, the specified facial key point coordinates are aligned with the local mask center coordinates, and the protected image is obtained by pixel-by-pixel multiplication: The mathematical formula is as follows:

[0059] ;

[0060] in, For the image to be protected, To protect the image, is the local area mask, is the initial watermark, It is a pixel-by-pixel multiplication operation; in order to achieve the effect of simulating physical wearing in the subsequent watermark optimization process.

[0061] This embodiment is different from the digital watermark in the traditional active defense field. It relies on global and small disturbances and innovatively designs a local, visible and colored watermark form. This design not only ensures the visual significance of the watermark, but also makes it more suitable for defense scenarios in the physical world. The watermark is initially set to gray and optimized and adjusted as the default color.

[0062] 1-4. Sample feature pair extraction: The input sample pair is composed of the image to be protected and the protected image, and the encoder of the forgery model is used to extract the latent space features of the sample pair. Before generating the forged image, the generator of the forgery model inverts the input image to the feature space through the encoder. The feature space contains rich semantic information, and a variety of forgery effects can be achieved by editing the feature space. Therefore, using these features for watermark training is helpful to optimize the watermark defense performance.

[0063] The sample feature pair is the two feature digital vectors obtained by inputting the image to be protected and the protection image through the encoder of the forgery model.

[0064] 2. Multiple constraints optimize watermarks. Through multiple loss constraints, feature distance loss, color reproduction loss and smoothness loss are alternately optimized to ensure that the watermark has a good defense effect.

[0065] 2-1. Calculation of feature distance loss function: The present invention introduces a feature distance loss function, optimizes the watermark parameters of the initial watermark through the feature distance loss function, measures the interference changes of the forged model output with or without the addition of the watermark, optimizes the watermark parameters, thereby weakening the generation ability of the forged model and improving the defense effect of the watermark. The mathematical formula is as follows:

[0066]

[0067] in, is the distance loss function, To extract features from images using the encoding of the forged model.

[0068] Then, the method of obtaining the adversarial watermark by optimizing the initial watermark with the feature distance loss function is:

[0069] , ;

[0070] in Statement After rounds of optimization, the adversarial watermark based on The watermark is clipped by the threshold value to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. is the weight coefficient, is the gradient of the loss with respect to the watermark.

[0071] 2-2. Calculation of color reproduction loss function: To ensure that the color of the adversarial watermark can be accurately reproduced after physical printing, a color reproduction loss function is introduced. Specifically, a color reference set is constructed by first printing a palette with multiple colors and collecting these colors. , which represents the reproducible colors of the device. Then, each pixel in the adversarial watermark is compared with the closest color in the reference set to minimize the difference between the watermark color and the reference set color. This difference is evaluated by calculating the distance between each pixel and the reference color, and the mathematical formula is as follows:

[0072] , ;

[0073] in, is the color reproduction loss function, Indicates that the horizontal and vertical coordinates of the watermark are and Pixels, Indicates the color from the reference set that is closest to the pixel value.

[0074] 2-3. Calculation of smoothness loss function: The present invention optimizes the texture and edge of the adversarial watermark by introducing a smoothness loss function to ensure its continuity and visual consistency in the physical world. Physical imaging devices can usually only capture continuous image areas, but for fragmented, irregular textures or random noise, detail information is easily lost. Therefore, the watermark needs to appear as continuous as possible in the physical world to avoid the impact of the loss of details on the defense effect. By optimizing the smoothness loss, the adversarial watermark can better adapt to the acquisition environment of the physical world, and can also be more natural and harmonious visually, while maintaining its ability to interfere with counterfeit models. The mathematical formula is as follows:

[0075]

[0076] in, is the smoothness loss function. Indicates that the horizontal and vertical coordinates of the watermark are and Pixels

[0077] 2-4. Optimize the watermark parameters of smoothness and color reproduction of the adversarial watermark so that the adversarial watermark can still have an adversarial effect in the physical world. That is, optimize the watermark parameters through the smoothness loss function and the color reproduction loss function so that the digital adversarial watermark can still maintain effective adversarial performance in the physical world after being reproduced as a physical adversarial watermark by a physical device. Specifically, the expression is as follows:

[0078] , ;

[0079] in Describe the adversarial watermark after the first round of optimization, The watermark is clipped based on the threshold to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. It is the weight coefficient. is the gradient of the loss with respect to the watermark.

[0080] 3. Implementation and application of physical adversarial watermarks: The implementation and application of this invention aims to transform the optimized adversarial watermarks into practical physical watermarks through physical printing and other means. The generated physical watermarks are precisely deployed at the expected positions designed during watermark alignment, thereby effectively helping the protected objects resist tampering and attacks by deep fake models.

[0081] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. An active defense method for deep fake faces using physical adversarial watermarks, characterized in that: The steps include: Step 1: Obtain the image to be protected and the initial watermark, and specify the area in the image to be protected that does not affect the key facial features as a local area mask; Step 2: Use the face detection and key point extraction model of the dlib library to extract facial key points from the image to be protected; Step 3: Based on the extracted facial key points and local area masks, the initial watermark is aligned to the target area of ​​the image to be protected, and the protected image is obtained by pixel-by-pixel multiplication. Step 4: The image to be protected and the protected image form a sample pair, and an encoder of a forged model is applied to extract the latent space features of the sample pair; Step 5: Multiple constraints to optimize watermark; Step 6: Generate an actual physical watermark through physical means according to the target area aligned with the adversarial watermark, and implement it.

2. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 1, characterized in that: In step 3, the method for obtaining the protected image is: ； in, For the image to be protected, To protect the image, is the local area mask, is the initial watermark, It is a pixel-by-pixel multiplication operation.

3. According to claim 1, a method for actively defending against deep fake faces using physical adversarial watermarks, characterized in that: The forgery model adopts a GAN model for editing and forging.

4. According to claim 1, a method for actively defending against deep fake faces using physical adversarial watermarks, characterized in that: The multiple constrained optimization methods include watermark parameter optimization, minimizing the difference between the color in the adversarial watermark and the color of the reference set, and optimizing the edge and texture of the watermark through a smoothness loss function.

5. According to claim 3, a method for actively defending against deep fake faces using physical adversarial watermarks, characterized in that: In step 5, the watermark parameters of the initial watermark are optimized by the feature distance loss function, and the expression is as follows: ； in, is the distance loss function, To extract features from images using the encoding of the forged model.

6. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 4, characterized in that: In step 5, the method of obtaining the countermeasure watermark after the initial watermark is optimized by the feature distance loss function is: ， ； in Statement After rounds of optimization, the adversarial watermark based on The watermark is clipped by the threshold value to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. is the weight coefficient, is the gradient of the loss with respect to the watermark.

7. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 5, characterized in that: In step 5, the method of minimizing the difference between the color in the adversarial watermark and the color in the reference set is: first, a color reference set is constructed by printing a color palette with multiple colors and collecting the colors. ; Next, each pixel in the adversarial watermark is compared with the closest color in the reference set, thereby minimizing the difference between the color in the adversarial watermark and the color in the reference set.

8. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 7, characterized in that: The method for evaluating the difference between the color in the adversarial watermark and the color in the reference set is: The evaluation is performed by calculating the distance between each pixel and the reference color. The evaluation method is as follows: ， ； in, is the color reproduction loss function, Indicates that the horizontal and vertical coordinates of the watermark are and Pixels, Indicates the color from the reference set that is closest to the pixel value.

9. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 8, characterized in that: In step 5, the texture and edge are optimized by the smoothness loss function, and the expression is as follows: ； in, is the smoothness loss function, Indicates that the horizontal and vertical coordinates of the watermark are and Pixels.

10. The method for actively defending against deep fake faces using physical adversarial watermarks according to claim 9, characterized in that: In step 5, the watermark parameters of the adversarial watermark are optimized for smoothness and color reproduction, so that the adversarial watermark can still have an adversarial effect in the physical world. The expression is as follows: ， ； in Describe the adversarial watermark after the first round of optimization, The watermark is clipped based on the threshold to prevent the pixel value of the watermark from exceeding the pixel color range during the optimization process. It is the weight coefficient. is the gradient of the loss with respect to the watermark.

Images