Active detection method and detection system for deep counterfeiting of human face

Abstract

The invention belongs to an image processing technology, and particularly relates to an active detection method and detection system for face depth forgery, and the method comprises the steps: enabling learnable protection information to be embedded into an original image through a depth image fusion network, and obtaining a protected image; the depth image fusion network performs reverse processing on the protected image to obtain secret protection information, and calculates a residual error between the secret protection information and the learnable protection information of the embedded image; and inputting the obtained residual error into a joint training decision device to judge whether the to-be-detected image is tampered or not. According to the method, on one hand, good detection performance can be achieved for various types of deep forging methods, and on the other hand, the embedded protection information can be prevented from being subjected to benign image transformation in image propagation.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a method and system for actively detecting deep fake faces. Background Art

[0002] As one of the main applications of AI-generated models, deep fake technology has become a powerful tool that can create fake facial images or videos by modifying the identity, attributes or expressions of the original face. On the one hand, deep fake technology has been widely used in the film industry, entertainment applications and advertising. At the same time, some criminals use deep fake technology to create fake news, online fraud, etc., which greatly endangers social stability. Although relevant AI-generated content regulations are gradually maturing, at the technical level, we still need an effective means to ensure accurate identification of whether an image has been maliciously forged.

[0003] In the research field, existing face deep fake detection methods can be summarized into two categories:

[0004] 1) Passive detection technology for deep fake faces.

[0005] The defects of deep fake face technology often cause the generated images to have various unrealistic artifacts, so passive detection technology of deepfake has become one of the main directions of existing deep fake face defense. These detection methods usually rely on identifying abnormal features or artifacts in fake images, and judge the authenticity of images by analyzing clues such as the boundaries, texture details, and reconstruction errors of the face. As images undergo multiple compressions on the Internet, traditional spatial domain detection methods face challenges because the forgery traces become more difficult to detect after compression. To address this problem, researchers began to explore the extraction of detection clues from the frequency domain perspective, trying to capture artifact traces from different frequency bands or multi-scale features. However, although these passive detection methods show strong detection capabilities for some forgery technologies, the detection effect may drop significantly when faced with complex forgery datasets or new forgery models, and the generalization ability is insufficient. Therefore, the current passive detection methods still need to be further improved to cope with the new challenges brought by the evolving forgery technology.

[0006] 2) Active facial deepfake detection technology.

[0007] Active forensics-based defense methods do not require the destruction of the generated results of the deep fake model, but focus on tracing and identity authentication. These methods include embedding steganography or watermarks into images to identify and extract embedded information in forged images and achieve forgery tracing. Some methods also use embedded protection signals to locate deep fake areas, or use robust watermarking technology to effectively identify and restore original information after the image has undergone transformations such as tampering or compression. Although these methods have shown some effectiveness in preventing forgery, they usually focus on specific types of deep fake technology and may not be widely applicable to all forgery scenarios. In addition, such methods also face some challenges in practical applications, such as the embedded watermark or protection signal may greatly affect the quality of the original image, or the protection effect may be weakened under high-intensity attacks. Summary of the invention

[0008] In view of the problem that the image quality and authenticity detection rate after embedding protection information in the current mainstream face deep fake active detection are not ideal, the present invention proposes a deep fake active detection method, which embeds learnable protection information into the original image through a deep image fusion network to obtain a protected image. The process of detecting whether the protected image after unknown image transformation has been tampered with includes the following steps:

[0009] The deep image fusion network performs inverse processing to obtain the secret protection information, and calculates the residual between the secret protection information and the learnable protection information embedded in the image;

[0010] The obtained residual is input into the jointly trained decision maker to determine whether the image to be detected has been tampered with.

[0011] Furthermore, the deep image fusion network includes four cascaded reversible blocks, each of which includes three affine transformation modules, each of which includes two inputs and two outputs, and the forward transformation process includes:

[0012] The learnable protection information is used as a first input end of the first reversible block, and the original image is used as a second input end of the first reversible block;

[0013] After the first input end is transformed by the first affine transformation block, it is fused with the second input end by adding and used as the second output end, and the second output end is used as the second input end of the next-level reversible block;

[0014] The protected image is transformed by the second affine transformation block and then fused with the learnable protection information by multiplication to obtain a first intermediate image;

[0015] The protected image is transformed by the third affine transformation block and then fused with the first intermediate image by addition as a first output end, and the first output end is used as a first input end of the next-level reversible block;

[0016] The first output terminal of the last-stage reversible block outputs redundant protection information, and the second output terminal outputs a protected image.

[0017] Furthermore, when the deep image fusion network performs inverse transformation, a random Gaussian noise is input to the first output end of the last-level reversible block, and a protected image is input to the second output end, and the secret protection information is obtained at the first input end after inverse transformation.

[0018] The present invention also proposes an active face deep fake detection system, which is used to implement an active face deep fake detection method, including a jointly trained deep image fusion network and a judge, the forward transformation process of the deep image fusion network is to embed learnable protection information into the original image, and the inverse transformation process of the deep image fusion network is to extract a secret protection information from the protected image, and the judge is used to predict whether the protected image has been tampered with based on the residual between the secret protection information and the learnable protection information.

[0019] The present invention constructs a deep image fusion network through a reversible deep neural network, and by fusing and embedding the original image with the learnable protection information multiple times, it ensures that the generated image is similar to the original input image to be protected, effectively solving the problem of the availability of the protected image. The specific beneficial effects of the present invention include:

[0020] 1) The present invention has strong generalization, that is, in terms of deep fake detection, the present invention can have good detection performance for various types of deep fake methods.

[0021] 2) The present invention has strong robustness, that is, the protection information embedded in the present invention can be unaffected by benign image transformations in image propagation, such as image compression, image blurring and other operations.

[0022] 3) The present invention has strong fidelity, that is, the protected image after embedding the information is similar to the original image and does not affect the normal use of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a method for active face deep fake detection according to the present invention;

[0024] Figure 2 A schematic diagram of the composition of the reversible neural network in the present invention;

[0025] Figure 3 It is the overall network schematic diagram of the present invention;

[0026] Figure 4 This is a schematic diagram of the effect after the processing is implemented by the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The present invention proposes a deep forgery active detection method, which embeds learnable protection information into the original image through a deep image fusion network to obtain a protected image. The process of detecting whether the protected image after the unknown image transformation has been tampered with includes the following steps:

[0029] The deep image fusion network performs inverse processing to obtain the secret protection information, and calculates the residual between the secret protection information and the learnable protection information embedded in the image;

[0030] The obtained residual is input into the jointly trained decision maker to determine whether the image to be detected has been tampered with.

[0031] In this embodiment, if Figure 1 The present invention is a training process of a deep image fusion network in a deep face forgery active detection method, which specifically includes the following steps:

[0032] S1. For an input face image to be protected, it is first sent together with the learnable protection information into a deep image fusion network trained based on a deep fake detection model. After being processed by the fusion network, the final protection image is output. The protection image is visually similar to the original face image to be protected, but the learnable protection information is hidden.

[0033] S2. As a preferred implementation, the protected image is preprocessed by an image transformation module, and the preprocessing means include but are not limited to image blurring, image noise addition, image compression, face swapping and other operations.

[0034] S3. Use the inverse operation of the above-mentioned deep image fusion network to extract the original embedded image from the preprocessed protected image.

[0035] S4. Send the residual between the extracted protection information and the original protection information to the decision device, and judge whether the input image has been tampered with according to the integrity of the input image.

[0036] The construction process of the deep image fusion network adopted in this embodiment includes:

[0037] Step 1) construct a reversible module. The reversible block in the present invention can be constructed by any affine transformation module in the prior art. Multiple affine transformation modules are fused through various operations such as addition and multiplication. The present invention requires that the constructed reversible block has two inputs and two outputs.

[0038] Step 2) A flow-based reversible neural network is constructed by sequentially cascading four reversible blocks. The forward process of the reversible neural network can embed an image into another image, and the reverse process can extract the embedded information in the carrier image.

[0039] This embodiment proposes a reversible block structure obtained by fusing three affine transformation blocks through multiplication and addition operations. In this embodiment, the reversible block includes three nonlinear affine transformation blocks composed of a dense product network, such as Figure 2 , the connection relationship between the three affine transformation modules and the process of forward transformation of the image in the deep image fusion network include:

[0040] The learnable protection information is used as a first input end of the first reversible block, and the original image is used as a second input end of the first reversible block;

[0041] After the first input end is transformed by the affine transformation block 1, it is fused with the second input end by adding and used as the second output end, and the second output end is used as the second input end of the next-level reversible block;

[0042] The protected image is transformed by the affine transformation block 2 and then fused with the learnable protection information by multiplication to obtain a first intermediate image;

[0043] The protected image is transformed by the affine transformation block 3 and then fused with the first intermediate image by addition as a first output terminal, and the first output terminal is used as a first input terminal of the next-level reversible block;

[0044] The first output terminal of the last-stage reversible block outputs redundant protection information, and the second output terminal outputs a protected image.

[0045] Preferably, when the deep image fusion network performs inverse transformation, a random Gaussian noise is input to the first output end of the last-level reversible block, and a protected image is input to the second output end, and the secret protection information is obtained at the first input end after inverse transformation.

[0046] The deep image fusion network used in this embodiment is a reversible neural network. The reversible neural networks used in the embedding and extraction processes share network parameters. The redundant information generated in the embedding process is discarded. During the restoration process, noise is randomly sampled from the Gaussian space as auxiliary information to help the restoration. The reversible block is constructed by three identical nonlinear affine transformation modules. This densely connected transformation module consists of 5 two-dimensional convolution operations and activation functions. This embodiment gives the following example: Figure 2A specific implementation of a deep image fusion network is shown, which performs fusion by adding or multiplying the outputs of three affine transformation modules in a reversible block to obtain a protection image and redundant information after embedding protection information, and the protection image is similar to the original image.

[0047] Other networks may also be selected in the art to fuse two images, and the present invention does not require the specific structure of the network.

[0048] The active face deep fake detection framework in this embodiment is as follows: Figure 3 The training shown is subject to four loss functions, namely, image embedding loss function and restoration loss function, learnable protection information loss function, and discriminator prediction loss function. The image embedding function is used to ensure that the protection image after embedding the protection information is similar to the original image in visual sense, the image extraction loss is used to ensure that the protection information extracted from the untampered protection image is similar to the original protection information, the learnable protection information loss function is used to ensure that the embedded information is as similar as possible to the original face to improve the embedding quality, and the discriminator prediction loss function is used to ensure that the predicted authenticity type is consistent with the actual type. The total loss function includes:

[0049]

[0050] in, represents the total loss function of the original image, the protection information, and the extracted information; L Steg Indicates the image quality loss caused by steganographic protection information in the original image; L Rec Indicates the loss of restored information and original protection information; L Sec Represents the loss of original protection information and input face image; L Det represents the loss between the discriminator’s prediction of the input information type and the actual type; λ1, λ2, λ3, and λ4 are L Steg , L Rec , L Sec , L Det The weight of .

[0051] As an optional implementation, the image quality loss L between the protection image and the original image Steg It is expressed as:

[0052] L Steg =||It(I,S)||2

[0053] Where I represents the original image, t(I,S) represents the protected image obtained by embedding the learnable protection information S into the original image I through the forward transformation of the deep image fusion network; ||·||2 represents the calculation of the L2 distance, that is, the calculation of the mean square error between the two images.

[0054] As an optional implementation, the loss of the restored information and the original protection information L Rec It is expressed as:

[0055]

[0056] in, represents the original image after a benign image transformation, Represents the inverse transformation from image to image through the deep image fusion network The secret protection information is extracted from , Z′ represents the noise randomly selected from the Gaussian space, S represents the learnable protection information; ||·||2 represents the calculation of the L2 distance, that is, the calculation of the mean square error between the two images.

[0057] As an optional implementation, the loss L between the original protection information and the input face image Sec It is expressed as:

[0058]

[0059] Among them, DWT LL (·) indicates the frequency domain information of the image, S indicates the learnable protection information, I indicates the original image, and ||·||2 indicates the calculation of the L2 distance, that is, the calculation of the mean square error between the low-frequency features of the two images.

[0060] As an optional implementation, the loss L between the discriminator predicting the input information type and the actual type Det It is expressed as:

[0061]

[0062] Wherein, y represents the true category of the input image; f(ΔS) represents the predicted category obtained by the discriminator according to the residual ΔS between the secret protection information extracted by the inverse transform of the deep image fusion network and the learnable protection information. In this embodiment, the discriminator is a deep counterfeit prediction model based on the Patch discriminator, which is used to determine whether the information extracted from the protection image has changed compared to the original protection information, thereby realizing deep counterfeit detection.

[0063] Existing deep fake active detection methods rely on specific forgery methods during training, while the proxy forgery model contained in the image preprocessing module of the present invention is a self-mixing method. The forged samples obtained by this non-traditional forgery method are richer, making the trained forgery detection more generalizable and able to adapt to more different forgery methods.

[0064] The present invention also proposes a human face deep forgery active detection system, which is used to implement a human face deep forgery active detection method, including a jointly trained deep image fusion network and a discriminator, the process of the deep image fusion network forward transformation is to embed learnable protection information into the original image, the process of the deep image fusion network reverse transformation is to extract a secret protection information from the protected image, and the discriminator is used to predict whether the protected image has been tampered with based on the residual between the secret protection information and the learnable protection information. First, the trained protection information is embedded in the original face image. The protection image containing the protection information may be subject to ordinary image transformation or malicious face tampering during network transmission. When face forgery detection is required, the protection image is input into the reversible neural network to extract the protection information, and the residual of the restored protection information and the original protection information is input into the discriminator for authenticity prediction.

[0065] This embodiment also provides a specific training process for the active face deep fake detection model, which specifically includes the following steps:

[0066] 1) Dataset and preprocessing

[0067] FFHQ dataset: mainly used for training, divided into three parts: 30,000 images for training, 3,000 for verification, and 3,000 for testing. The model training in this embodiment mainly relies on this dataset.

[0068] VGGFace2 dataset: used in testing to simulate cross-dataset detection scenarios. We randomly extracted 3,000 images from this dataset. The VGGFace2 dataset contains 3.31 million images from 9,131 identities, with an average of 362.6 images per person; about 59.7% of the images are male; each image is also annotated with a face frame, 5 key points, and information such as age and posture.

[0069] CelebA-HQ dataset: used in the test to simulate cross-dataset detection scenarios. We randomly selected 3,000 images from this dataset. CelebA-HQ is a high-quality version of the CelebA dataset, containing 202,599 face images of 10,177 identities, and annotated with about 40 facial attributes, such as whether wearing glasses, whether smiling, etc. In the experiment, all images were scaled or cropped, and the resolution was uniformly set to 256×256.

[0070] Image benign transformation operations: During training, for JPEG compression, the compression quality factor varies between 50 and 95. For Gaussian blur, we fix the Gaussian kernel size to 3 and randomly vary the standard deviation uniformly between 1 and 2. For Gaussian noise, we randomly sample its standard deviation uniformly between 0 and 0.05. For image rescaling, we downsample the image according to a scaling factor uniformly sampled between 0.5 and 1.0, and then resample it to the original size.

[0071] 2) Network training

[0072] The training process was optimized using the Adam optimizer with a learning rate of 0.0001.

[0073] This example gives Figure 4 The effect diagram shown is as follows: the first column is the original face image to be protected, the second column is the protected image after embedding the protection information, the third column is the residual between the protected image and the original image, the fourth column is the forged protected image, the fifth column is the protection information extracted from the unforged protected image, the sixth column is the protection information extracted from the maliciously forged protected image, the seventh column is the residual between the fifth column and the original protection information, and the eighth column is the residual between the sixth column and the original protection information. Figure 4 As shown, the protected image is almost identical to the original image in terms of visual perception, the protection information extracted from the untampered protected image is consistent with the original protection information, and the protection information extracted from the tampered protected image is quite different from the original image.

[0074] This embodiment also quantitatively verifies the proposed face deep fake active detection model through simulation experiments, the purpose of which is to verify the usability of the protected image and the deep fake detection performance. Some experiments compare with the methods proposed by SepMark, FaceSigns, and PDDIW and effectively prove the superiority of the method of the present invention.

[0075] In terms of active protection performance, the visual difference between the original image and the protected image is measured by objective methods. PSNR, LPIPS and SSIM are used to measure the difference between the protected image and the original image on different datasets. Tables 1 to 3 show the objective indicators of information embedding performance of this method on FFHQ, CelebA-HQ, and VGGFace2 datasets. The results show that the information embedding ability of this method is much higher than that of the comparison method, and the capacity of embedded information is also higher than that of the comparison method.

[0076] Table 1 Objective indicators of information embedding performance of the proposed method on the FFHQ dataset

[0077]

[0078]

[0079] Table 2 Objective indicators of information embedding performance of the proposed method on the CelebA-HQ dataset

[0080]

[0081] Table 3 Objective indicators of information embedding performance of the proposed method on the VGGFace2 dataset

[0082]

[0083] In terms of deep fake detection rate, deep fake detection experiments were conducted on the FFHQ face image dataset, using the forgery recognition rate of the residual difference between the restored protection information and the original protection information as an indicator. Tables 4 to 6 show the face forgery detection rate of the method of the present invention compared with SepMark, FaceSigns, and PDDIW in different common image transformation methods and cross-forgery method scenarios. The results show that the protection information is extracted from the protection image obtained by the method of the present invention, and its integrity is judged, which can maintain a high forgery detection rate and can exceed the comparison method in most cases.

[0084] Table 4 Comparison of the authenticity detection rate of the proposed method with SepMark, FaceSigns, and PDDIW for different counterfeiting methods on the FFHQ dataset

[0085]

[0086]

[0087] Table 5 Comparison of the authenticity detection rate of the proposed method with SepMark, FaceSigns, and PDDIW for different counterfeiting methods on the CelebA-HQ dataset

[0088]

[0089] Table 6 Comparison of the authenticity detection rate of the proposed method with SepMark, FaceSigns, and PDDIW for different counterfeiting methods on the VGGFace2 dataset

[0090]

[0091] In the above table, the PDDIW method comes from "Proactive DeepfakeDefence via Identity Watermarking" published by Yuan Zhao et al.; the SepMark method comes from "SepMark: Deep Separable Watermarking for Unified Source Tracing and DeepfakeDetection" published by Xiaoshuai Wu et al.; the FaceSigns method comes from "FaceSigns: Semi-FragileNeural Watermarks for Media Authentication and Countering Deepfakes" published by Paarth Neekhara et al.; the SimSwap method comes from "SimSwap: An Efficient Framework For HighFidelity Face Swapping" published by Renwang Chen et al.; FaceShifter comes from "FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping" published by Lingzhi Li et al.; FaceSwap comes from the face-changing method contributed by Wu Huikai et al. in the GitHub open source community; MFaceSwap comes from "MobileFaceSwap: A Lightweight Framework for Video Face Swapping" published by Zhiliang Xu et al. Swapping”; StarGAN2 comes from “StarGAN v2: Diverse Image Synthesis for Multiple Domains” published by Yunjey Choi et al.; TTEdit comes from “Talk-To-Edit: Fine-Grained FacialEditing via Dialog” published by Yuming Jiang et al.; FIT comes from “Password-conditioned Anonymization and Deanonymization with Face Identity Transformers” published by Xiuye ​​Gu et al.

[0092] To sum up, the example of the present invention has verified the feasibility of the scheme in the present embodiment through the above-mentioned simulation experiments. The example of the present invention has ensured the availability of the generated protection image while effectively solving the problem of low performance of deep fake face detection through a universal active detection method for deep fake face.

[0093] The present invention also proposes a computer device, which includes a processor and a memory, wherein the processor is used to run a computer program stored in the memory to implement the aforementioned active detection method for deep fake faces.

[0094] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for actively detecting deep fake faces, characterized in that: The learnable protection information is embedded into the original image through a deep image fusion network to obtain a protected image. The process of detecting whether the protected image after the unknown image transformation has been tampered with includes the following steps: The deep image fusion network performs inverse processing to obtain the secret protection information, and calculates the residual between the secret protection information and the learnable protection information embedded in the image; The obtained residual is input into the joint training decision maker to determine whether the image to be detected has been tampered with.

2. According to claim 1, a method for actively detecting deep fake faces is characterized in that: The deep image fusion network consists of four cascaded reversible blocks, each of which consists of three affine transformation modules. Each affine transformation module envelops two inputs and two outputs. The forward transformation process includes: The learnable protection information is used as a first input end of the first reversible block, and the original image is used as a second input end of the first reversible block; After the first input end is transformed by the first affine transformation block, it is fused with the second input end by adding and used as the second output end, and the second output end is used as the second input end of the next-level reversible block; The protected image is transformed by the second affine transformation block and then fused with the learnable protection information by multiplication to obtain a first intermediate image; The protected image is transformed by the third affine transformation block and then fused with the first intermediate image by addition as a first output end, and the first output end is used as a first input end of the next-level reversible block; The first output terminal of the last-stage reversible block outputs redundant protection information, and the second output terminal outputs a protected image.

3. The method for actively detecting deep fake faces according to claim 2, characterized in that: When the deep image fusion network performs inverse transformation, a random Gaussian noise is input to the first output end of the last-level reversible block and a protected image is input to the second output end. After inverse transformation, the secret protection information is obtained at the first input end.

4. A method for actively detecting deep fake faces according to claim 1 or 2, characterized in that: The loss function used when training the deep image fusion network and the discriminator is expressed as: in, represents the total loss function of the original image, the protection information, and the extracted information; L Steg Indicates the image quality loss caused by steganographic protection information in the original image; L Rec Indicates the loss of restored information and original protection information; L Sec represents the loss of the original protection information and the input face image; L Det represents the loss between the discriminator’s prediction of the input information type and the actual type; λ1, λ2, λ3, and λ4 are L Steg , L Rec , L Sec , L Det The weight of .

5. The method for actively detecting deep fake faces according to claim 4, characterized in that: Image quality loss between the protected image and the original image L Steg It is expressed as: L Steg =||I-t(I,S)||2 Where I represents the original image, t(I,S) represents the protected image obtained by embedding the learnable protection information S into the original image I through the forward transformation of the deep image fusion network; ||·||2 represents the calculation of the L2 distance, that is, the calculation of the mean square error between the two images.

6. The method for actively detecting deep fake faces according to claim 4, characterized in that: The loss of restored information and original protected information L Rec It is expressed as: in, represents the original image after a benign image transformation, Represents the inverse transformation from image to image through the deep image fusion network The secret protection information is extracted from , Z′ represents the noise randomly selected from the Gaussian space, S represents the learnable protection information; ||·||2 represents the calculation of the L2 distance, that is, the calculation of the mean square error between the two images.

7. The method for actively detecting deep fake faces according to claim 6, characterized in that: Benign image transformation is to perform image compression operation, image blur operation, image Gaussian noise operation or image size downsampling operation on the image.

8. The method for actively detecting deep fake faces according to claim 4, characterized in that: The loss of original protection information and input face image L Sec It is expressed as: Where DWTLL(·) means to find the low-frequency domain information of the image, S means the learnable protection information, I means the original image, and ||·||2 means to calculate the L2 distance, that is, to calculate the mean square error of the low-frequency features between two images.

9. The method for actively detecting deep fake faces according to claim 4, characterized in that: The loss LDet of the discriminator's prediction of the input information type and the actual type is expressed as: Among them, y represents the true category of the input image; f(ΔS) represents the predicted category obtained by the decision maker based on the residual ΔS between the secret protection information extracted by the inverse transform of the deep image fusion network and the learnable protection information.

10. A facial deepfake active detection system, characterized in that: A method for actively detecting deep fake faces of a human face used to implement the method described in claim 1, comprising a jointly trained deep image fusion network and a judge, wherein the process of forward transformation of the deep image fusion network is to embed learnable protection information into the original image, and the process of inverse transformation of the deep image fusion network is to extract a secret protection information from the protected image, and the judge is used to predict whether the protected image has been tampered with based on the residual between the secret protection information and the learnable protection information.

Images