Active Deepfake traceability method based on face identity information
By embedding robust invisible watermarks in the face image and extracting watermark information using the Unet network and GAN module, the problem of untraceable Deepfake images in the prior art is solved, and the balance between robustness and inesisibility of watermarks is achieved. Reduces the harm of Deepfake.
Patent Information
- Application Number
- CN202411401404.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-12-31
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art cannot effectively conduct active Deepfake traceability of face images, and cannot realize traceability of face exchange images while maintaining the robustness of watermarks on the basis of ensuring high unperception of watermark information.
Adopting the active Deepfake traceability method based on facial identity information, the feature vector is extracted through the pre-trained face recognition network and embedded a robust invisible watermark. The Unet network and GAN module are used to insert watermarks into the image, and a hybrid operation module is designed to simulate attacks. To extract watermark information, ensure that the watermark can still be extracted after various attacks.
It achieves a good balance between the robustness and inesisibility of watermark information, and can resist various conventional attacks and Deepfake attacks while maintaining high visual quality, reducing the harm caused by Deepfake.
Smart Images
Figure CN119229544A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an active Deepfake tracing method based on facial identity information. Background Art
[0002] Face swapping technology can seamlessly transfer a person's identity to another image while maintaining the authenticity of the facial appearance. It is difficult for the human eye to tell whether an image or video has been manipulated. Although face swapping is popular in the entertainment and film industries, it is also controversial due to its potential unethical uses and possible threats to politics, economy and personal privacy. In some countries, there are laws restricting the scope of application of face swapping technology.
[0003] In order to prevent Deepfake from infringing reputation and legitimate rights, Deepfake detection has become an important topic that has attracted much attention. Currently, most existing methods mainly improve the accuracy of distinguishing Deepfake from original video or image frames through binary classification. In addition, there are also some methods that try to identify fake images by checking the pre-embedded watermarks in the original image. However, these techniques can only determine whether the video or image has been forged, and cannot provide further information about the original source.
[0004] At present, the detection of Deepfake face swapping mainly relies on the design of passive detectors, which detect by analyzing the traces of suspicious images in various feature domains. However, as the quality of synthesis continues to improve, the challenges are also increasing. Therefore, the active defense method is no longer to look for synthetic artifacts, but to protect the original image by inserting imperceptible watermarks before the potential operation occurs. These watermarks can be semi-fragile or robust. Semi-fragile watermarks will show vulnerability in the face of Deepfakes and robustness in the face of conventional attacks, so they are mostly used for Deepfake detection; robust watermarks can show robustness in the face of any attack, that is, the watermark information can be completely extracted, so as to trace back to the source of the image identity information, which puts higher requirements on how to balance the imperceptibility and robustness of the watermark. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide an active Deepfake tracing method based on facial identity information. By embedding a robust invisible watermark into a partial area of the facial image, the face-swapped image can be traced while ensuring the high imperceptibility of the image watermark information, and the imperceptibility and robustness of the watermark information can be well guaranteed at the same time.
[0006] The present invention adopts the following technical solutions to achieve the invention objectives: An active Deepfake tracing method based on face identity information, characterized by comprising the following steps: S1: For an input face image First, extract the feature vector of the last fully connected layer through the pre-trained face recognition network ArcFace and flatten it into a binary form to obtain the identity information sequence ; S2: For the obtained identity information sequence , assuming The length is , first through a linear layer to expand the length to , and then reshaped into the same size as the current encoder feature map by bilinear interpolation, and concatenated with the feature map vector after passing through the convolution layer, so that the identity information sequence Embed into cropped image middle; S3: Sequence the identity information Embed into cropped image In the imperceptible pixel area; Specifically, we first use the Dlib library to analyze the face image. Extract and retain the background image. At the same time, select the largest square area in the background image as the area to embed the watermark, so as to obtain the cropped image that needs to be embedded with the watermark. ; S4: The image to be watermarked The image to be watermarked is sent to an end-to-end framework for training. The framework consists of an encoder and a decoder. The Unet network is used, and the SE block is added as the residual information connection. After passing through the encoder, you will get a watermarked image , in order to ensure that the watermarked image The visual quality of the generated image is constrained by an MSE loss to the original image; (1); S5: Add a GAN module to distinguish whether the current image contains a watermark. The prediction loss can be expressed as: (2); in: It represents the prediction result of the GAN module for the image. The purpose of this application is to make the image generated by the encoder very similar to the original image in visual effect, so that the GAN module is difficult to distinguish and form an adversarial process. Therefore, the GAN loss of the encoder is: (3); S6: After obtaining the watermarked image After that, with the whole face image If you stitch the areas without watermark, you will get a complete image with watermark. ; S7: Design a hybrid operation module to simulate attacks so that the decoder can be fully trained to extract the complete watermark information; The image after the attack is represented as , can be obtained ,in Indicates various attack methods in the hybrid module; S8: After the attack is completed, the image after the attack The cropping is performed according to the method in S3 to obtain the cropped image and represent it as , then the decoder is used to extract the cropped image The watermark information is extracted from the decoder. Specifically, the decoder uses the same architecture as the encoder. The cropped image Finally, after the multi-channel tensor conversion, it will pass through a convolution layer and reshape into , and finally pass through a linear layer to get the same length as the embedded message Consistent face information prediction results , and use an MSE loss to optimize the decoder: (4); The total loss of the entire end-to-end framework can then be expressed as: (5) in: and is a hyperparameter that controls the weight of the loss function.
[0007] As a further limitation of this technical solution, the identity information sequence It is expressed as: ; in: Indicates the use of ArcFace for the input face image After processing, it is converted into a binary face identity information sequence.
[0008] As a further limitation of the present technical solution, the cropped image It is expressed as: ; in: Indicates cropping the selected area; It means using the Dlib library to extract the face from the face image and take the largest square frame in the background area.
[0009] As a further limitation of the present technical solution, the GAN module is essentially an adversarial discriminator, which is composed of multiple ConvReLU blocks and finally obtains the final prediction result through a linear layer.
[0010] As a further limitation of the present technical solution, the specific structure of the hybrid operation module is a combination of conventional attacks and Deepfake attacks. Conventional attacks include JPEG compression, Gaussian blur and salt and pepper noise, and Deepfake attacks include FaceSwap, SimSwap and InfoSwap.
[0011] As a further limitation of this technical solution, at each iteration A random attack method will be selected to attack the complete watermarked image to attack.
[0012] Compared with the prior art, the advantages and positive effects of the present invention are as follows: the present invention proposes a novel active Deepfake tracing method. By using the background area of the face image with less attention to embed the watermark, the watermark information can be extracted by the decoder after various attacks. The present invention achieves a good balance between the robustness and imperceptibility of the watermark, that is, while maintaining high visual quality, the watermark can maintain sufficient robustness and resist various conventional attacks and Deepfake attacks.
[0013] Existing active Deepfake methods mainly focus on embedding watermark information for Deepfake detection or post-Deepfake distortion. There are few technologies that use watermark information for tracing the source. The reason is that tracing the source requires the watermark to be strong enough, but this undoubtedly loses the visual quality of the image. From the perspective of active Deepfake tracing, the present invention designs a solution that can well maintain robustness and watermark imperceptibility, and can perform source detection on the forged image, thereby greatly reducing the serious harm caused by Deepfake. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a scheme framework diagram of the present invention. DETAILED DESCRIPTION
[0015] A specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation.
[0016] The present invention comprises the following steps: S1: for an input face image First, extract the feature vector of the last fully connected layer through the pre-trained face recognition network ArcFace and flatten it into a binary form to obtain the identity information sequence ; ; in: Indicates the use of ArcFace for the input face image After processing, it is converted into a binary face identity information sequence.
[0017] S2: For the obtained identity information sequence , assuming The length is , first through a linear layer to expand the length to , and then reshaped into the same size as the current encoder feature map by bilinear interpolation, and concatenated with the feature map vector after passing through the convolution layer, so that the identity information sequence Embed into cropped image middle.
[0018] S3: At the same time, in order to ensure the high imperceptibility of the embedded watermark, the identity information sequence Embed into cropped image In the imperceptible pixel area; Specifically, we first use the Dlib library to analyze the face image. Extract and retain the background image. At the same time, select the largest square area in the background image as the area to embed the watermark, so as to obtain the cropped image that needs to be embedded with the watermark. ;
[0019] ; in: Indicates cropping the selected area; It means using the Dlib library to extract the face from the face image and take the largest square frame in the background area.
[0020] S4: The image to be watermarked The image to be watermarked is sent to an end-to-end framework for training. The framework consists of an encoder and a decoder. The Unet network is used, and the SE block is added as the residual information connection. After passing through the encoder, you will get a watermarked image , in order to ensure that the watermarked image The visual quality of the generated image is constrained by an MSE loss to the original image; (1); S5: Following other robust watermarking methods, a GAN module is added to distinguish whether the current image contains a watermark. The prediction loss can be expressed as: (2); in: It represents the prediction result of the GAN module for the image. The purpose of this application is to make the image generated by the encoder very similar to the original image in visual effect, so that the GAN module is difficult to distinguish and form an adversarial process. Therefore, the GAN loss of the encoder is: (3); The GAN module is essentially an adversarial discriminator, which consists of multiple ConvReLU blocks and finally passes through a linear layer to obtain the final prediction result.
[0021] S6: After obtaining the watermarked image After that, with the whole face image If you stitch the areas without watermark, you will get a complete image with watermark. ; S7: In order to cope with various attacks in the real world, such as compression and blurring when spreading on social media, and Deepfake attacks by malicious tamperers, a hybrid operation module is designed to simulate attacks so that the decoder can be fully trained to extract the complete watermark information; The image after the attack is represented as , can be obtained ,in Represents various attack methods in the hybrid module.
[0022] The specific structure of the hybrid operation module is a combination of conventional attacks and Deepfake attacks. Conventional attacks include JPEG compression, Gaussian blur and salt and pepper noise, while Deepfake attacks include FaceSwap, SimSwap and InfoSwap.
[0023] At each iteration A random attack method will be selected to attack the complete watermarked image to attack.
[0024] S8: After the attack is completed, the image after the attack The cropping is performed according to the method in S3 to obtain the cropped image and represent it as , then the decoder is used to extract the cropped image The watermark information is extracted from the decoder. Specifically, the decoder uses the same architecture as the encoder. The cropped image Finally, after the multi-channel tensor conversion, it will pass through a convolution layer and reshape into , and finally pass through a linear layer to get the same length as the embedded message Consistent face information prediction results , and use an MSE loss to optimize the decoder: (4); The total loss of the entire end-to-end framework can then be expressed as: (5) in: and To control the hyperparameters of the loss function weight, in the present invention, it is set to , .
[0025] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. An active Deepfake tracing method based on face identity information, characterized in that: The following steps are involved: S1: For an input face image First, extract the feature vector of the last fully connected layer through the pre-trained face recognition network ArcFace and flatten it into a binary form to obtain the identity information sequence ; S2: For the obtained identity information sequence , assuming The length is , first through a linear layer to expand the length to , and then reshaped into the same size as the current encoder feature map by bilinear interpolation, and concatenated with the feature map vector after passing through the convolution layer, so that the identity information sequence Embed into cropped image middle; S3: Sequence the identity information Embed into cropped image In the imperceptible pixel area; Specifically, we first use the Dlib library to analyze the face image. Extract and retain the background image. At the same time, select the largest square area in the background image as the area to embed the watermark, so as to obtain the cropped image that needs to be embedded with the watermark. ; S4: The image to be watermarked The image to be watermarked is sent to an end-to-end framework for training. The framework consists of an encoder and a decoder. The Unet network is used, and the SE block is added as the residual information connection. After passing through the encoder, you will get a watermarked image , in order to ensure that the watermarked image The visual quality of the generated image is constrained by an MSE loss to the original image; （1）； S5: Add a GAN module to distinguish whether the current image contains a watermark. The prediction loss can be expressed as: （2）； in: It represents the prediction result of the GAN module for the image. The purpose of this application is to make the image generated by the encoder very similar to the original image in visual effect, so that the GAN module is difficult to distinguish and form an adversarial process. Therefore, the GAN loss of the encoder is: （3）； S6: After obtaining the watermarked image After that, with the whole face image If you stitch the areas without watermark, you will get a complete image with watermark. ; S7: Design a hybrid operation module to simulate attacks so that the decoder can be fully trained to extract the complete watermark information; The image after the attack is represented as , can be obtained ,in Indicates various attack methods in the hybrid module; S8: After the attack is completed, the image after the attack The cropping is performed according to the method in S3 to obtain the cropped image and represent it as , then the decoder is used to extract the cropped image The watermark information is extracted from the decoder. Specifically, the decoder uses the same architecture as the encoder. The cropped image Finally, after the multi-channel tensor conversion, it will pass through a convolution layer and reshape into , and finally pass through a linear layer to get the same length as the embedded message Consistent face information prediction results , and use an MSE loss to optimize the decoder: （4）； The total loss of the entire end-to-end framework can then be expressed as: （5） in: and is a hyperparameter that controls the weight of the loss function.
2. The active Deepfake tracing method based on face identity information according to claim 1, characterized in that: The identity information sequence It is expressed as: ； in: Indicates the use of ArcFace for the input face image After processing, it is converted into a binary face identity information sequence.
3. The active Deepfake tracing method based on face identity information according to claim 2 is characterized by: The cropped image It is expressed as: ； in: Indicates cropping the selected area; It means using the Dlib library to extract the face from the face image and take the largest square frame in the background area.
4. The active Deepfake tracing method based on face identity information according to claim 1, characterized in that: The GAN module is essentially an adversarial discriminator, which consists of multiple ConvReLU blocks and finally passes through a linear layer to obtain the final prediction result.
5. The active Deepfake tracing method based on face identity information according to claim 1, characterized in that: The specific structure of the hybrid operation module is a combination of conventional attacks and Deepfake attacks. Conventional attacks include JPEG compression, Gaussian blur and salt and pepper noise, while Deepfake attacks include FaceSwap, SimSwap and InfoSwap.
6. The active Deepfake tracing method based on face identity information according to claim 5 is characterized by: At each iteration A random attack method will be selected to attack the complete watermarked image to attack.