A face forgery detection method based on local region regularization

Abstract

The application discloses a face forgery detection method based on local area regularization, and belongs to the technical field of artificial intelligence. The arrangement order of image blocks formed after scaling and blocking a face area image is randomly disturbed to generate a new image, and the arrangement order is recorded. Then, features of the new image generated by disturbing the order are extracted. In the training process, the features of the new image generated by disturbing the order are input into a position reconstruction branch to obtain the arrangement order of the image blocks, so as to encourage the model to model the correlation of the image blocks. The features of the new image generated by disturbing the order are input into a classifier to obtain the probability that the final image is a forged image. The application realizes local area regularization by disturbing and reconstructing the image at the block level, overcomes overfitting caused by distribution deviation of a training set and a test set, encourages the model to focus on local areas of the image and model the correlation of the image blocks, enhances the detection and generalization ability of the network, and improves the forgery detection performance on damaged data.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a face forgery detection method based on local area regularization. Background Art

[0002] With the rapid development of deep learning technology, deep learning-based face forgery has made significant progress. Researchers generally categorize face replacement, facial attribute editing, and face synthesis based on the intended purpose. Deepfakes, a representative example of face replacement technology, can alter the identity of a face in a photo to create a specific fake image. The application of technologies such as autoencoders and generative adversarial networks has significantly reduced the cost of forgery. Deepfakes encode different faces using an encoder, then train two decoders to reconstruct the faces. Exchanging the decoders allows face replacement. The widespread dissemination of these fake images on the internet creates security risks, which in turn promotes the development of deepfake detection technology.

[0003] Most current face-swapping forgery detection techniques treat it as a binary classification problem, using data-driven training of deep convolutional networks to extract features from the spatial domain and frequency domain information to detect forged faces. Nguyen et al. (Nguyen, Huy Hoang, Junichi Yamagishi and Isao Echizen. Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos[C] / / ICASSP 2019 -2019 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2019, pp. 2307-2311.) proposed a detection method based on capsule networks. Li et al. (Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping artifacts[C] / / 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)) proposed using artifacts that appear after affine transformation of the face as unique features for forgery detection and achieved state-of-the-art performance based on SSPNet. Li et al. (Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and B. Guo. Face x-ray for more general face forgery detection[C] / / 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp 5000-5009.) proposed Face X-ray, which focuses on the forgery mixing step and achieves state-of-the-art transferable performance. However, it cannot be used for fully synthesized images, and its performance drops sharply on low-resolution images.Durallet et al. (Durall, Ricard, Margret Keuper, Franz-Josef Pfreundt and Janis Keuper. Unmasking DeepFakes with simple Features[C] / / ArXivabs / 1911.00686, 2019.) first proposed using discrete Fourier transform (DFT) to mine abnormal information in forged face images. Luo et al. (Luo, Yucheng, Yong Zhang, Junchi Yan and Wei Liu. Generalizing Face Forgery Detection with High-frequency Features[C] / / 2021 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), 2021, pp 16312-16321.) proposed using high-frequency noise for face forgery detection. Qian et al. (Qian, Yuyang, Guojun Yin, LuSheng, Zixuan Chen and Jing Shao. Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues[C] / / ECCV, 2020.) used the DFT to extract frequency-aware image decomposition and local frequency statistics to improve forgery detection performance in highly compressed videos. However, performance decreased significantly across datasets. Because the differences between real and fake images are primarily subtle and localized, researchers sought solutions from local regions. Du et al. (Du, Mengnan, Shiva K. Pentyala, Yuening Li and Xia Hu. Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder[C].\Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020.) focused on local spatial features and used an autoencoder to learn internal representations of local forgery regions to bridge the generalization gap.Zhao et al. (Zhao, Hanqing, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang and Nenghai Yu. Multi-attentional DeepfakeDetection[C] / / 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) ,2021,pp 2185-2194.) proposed a multi-attention network to focus on different local parts and subtle artifacts. Wang et al. (Wang, Chengrui and Weihong Deng. RepresentativeForgery Mining for Fake Face Detection[C] / / 2021 IEEE / CVF Conference onComputer Vision and Pattern Recognition (CVPR) ,2021,pp 14918-14927.) proposed an attention-based data augmentation framework that encourages the network to mine different features by masking sensitive areas. Wang et al. (Wang, Junke, Zuxuan Wu, Jingjing Chen and Yu-Gang Jiang. M2TR: Multi-modalMulti-scale Transformers for Deepfake Detection[C] / / Proceedings of the 2022International Conference on Multimedia Retrieval, 2022) proposed a multimodal multi-scale transformer to capture subtle artifacts at different scales. However, images often undergo varying degrees of degradation (such as resizing and blurring) during transmission, which poses a challenge to the generalization ability of detection methods. In severe cases, deep models may perform well on the training data but experience a significant drop in accuracy when predicting new data.

[0004] Patent CN113269167A discloses a face forgery detection method based on image block scrambling. This method divides the image into blocks and scrambles the blocks to enhance forgery detection capabilities. However, the scrambling itself introduces noise that misleads the detection model, resulting in limited gains in generalization and robustness, and poor performance at low image quality. Patent CN114445891A discloses a face forgery detection and localization method based on SSIM difference maps. However, the backbone network used is relatively complex, the computation time is long, and the hardware environment requirements are relatively high. The method is still dependent on data and requires the inclusion of multiple data enhancement modules to assist in training.

[0005] In summary, current face forgery detection methods rely on data, resulting in limited generalization capabilities of detection models. Furthermore, most images and videos transmitted online experience a certain degree of quality degradation, which leads to unstable local features in forged images. This causes existing forgery detection models to experience significant performance degradation when faced with compromised data. Furthermore, these models often require high network complexity and computational complexity, making them difficult to train and apply. Summary of the Invention

[0006] Technical problem solved by the present invention: The purpose of the present invention is to provide a face forgery detection method based on image block scrambling and recovery, which encourages a model to model the correlation of image blocks based on the scrambling and recovery between image blocks and obtains accurate face image forgery detection results.

[0007] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting face forgeries based on local region regularization includes: randomly shuffling the order of image blocks formed after scaling and segmenting a face region image to generate a new image, and recording the order; then extracting features from the shuffled new image; during training, inputting the extracted features of the shuffled new image into a position reconstruction branch to obtain the order of image blocks, thereby encouraging the model to model the correlation of image blocks; and inputting the extracted features of the shuffled new image into a classifier to obtain the probability that the final image is a forged image. The present invention overcomes overfitting caused by distribution deviations between training and test sets by implementing local region regularization through block-level shuffling and reconstruction of the image, encouraging the model to focus on the correlation between local image regions and modeled image blocks, enhancing the network's detection and generalization capabilities, and improving forgery detection performance on compromised data.

[0009] Furthermore, the arrangement order of the image blocks formed after scaling and segmenting the face region image is randomly disrupted to generate a new image, and the arrangement order step is recorded. For the image divided into image blocks of size 32×32, p(i) For the imagei image blocks, i ϵ{1,2,…,49}, the new image block after shuffling p'(i) Expressed as:

[0010] p'(i) = p(α i ) (3)

[0011] Where, α Generated by randomly disrupting the order of elements in the vector [1,2,…,49], α i Represents a vector α The i elements; a new image generated by shuffling the order I * , for each image, α All will be regenerated and recorded, which is called the image arrangement order M.

[0012] Furthermore, in the feature extraction step of the new image generated in the disordered order, the five convolutional blocks of the backbone network Xception are used to extract the new image generated in the disordered order. I * Features F.

[0013] Furthermore, the position reconstruction branch R consists of a PixleShuffle sampling layer, a 1×1 convolution layer, and a HardTanh activation layer;

[0014] During the training process, the feature F of the new image generated by the shuffled order is first input into the PixleShuffle sampling layer for pixel reorganization; then it is input into the 1×1 convolution layer to obtain a feature with a size of 7×7 and a number of channels of 2; the feature is input into the HardTanh activation layer to obtain a restored order with a size of 7×7 and a number of channels of 2; the restored order and the permutation order are input into the Smooth L1 loss function to calculate the loss and position reconstruction loss. Expressed as:

[0015] (4)

[0016] in, C represents the backbone network, Represents the backbone network C The parameters of the 1 to 5 convolution blocks, R Indicates the location reconstruction branch, Indicates location reconstruction branch R Parameters, I * Represents a new image generated in a disrupted order, MIndicates the order in which images are arranged.

[0017] Furthermore, the classifier consists of an average pooling layer and a fully connected layer; the new images generated in a random order I * The size of the feature F is 7×7 and the number of channels is 2048. The feature is input into the average pooling layer to obtain a feature with a size of 1×1 and the number of channels is 2028. After passing through a fully connected layer, a probability is output to indicate the probability that the image is a forged image. The cross entropy loss of the classification is expressed as:

[0018] (5)

[0019] in, Represents the backbone network C Parameters, y Indicates the authenticity label of the image.

[0020] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0021] The face forgery detection method based on local area regularization of the present invention is

[0022] (1) The image block shuffling method is used to destroy the global structure of the image, indirectly achieve equal probability sampling of the blocks, encourage the model to focus on the local area of ​​the image, prevent the forgery detection task from degenerating into the face recognition task, enhance the translation invariance of the network, and enhance the model's ability to detect deep forgeries;

[0023] (2) Using the inter-block restoration method to restore the original arrangement of image blocks, model the semantic relevance of image blocks, and enhance the model's ability to detect deep fake images;

[0024] (3) By disrupting and restoring the order of image blocks, regularization is achieved for local areas, overcoming the overfitting caused by the distribution deviation of the training set and test set caused by image distortion, and enhancing the generalization ability of the model for distorted images;

[0025] (4) Maintaining the detection performance on the original dataset, and being able to obtain accurate face image forgery detection results. This effectively improves the accuracy of face forgery detection, enhances the generalization ability of the face forgery detection model, and improves the situation where the performance of the detection task is severely degraded when facing corrupted data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure is a flow chart of the face forgery detection method based on image block scrambling and restoration according to the present invention.

[0027] Figure 2 A schematic diagram of the image block shuffling method used in the present invention.

[0028] Figure 3 Schematic diagram of the position reconstruction branch used in the present invention.

[0029] Figure 4 This is a schematic diagram of the effects of the present invention on disrupting and restoring facial images using different forgery methods.

[0030] Figure 5 This is a statistical graph of position differences after shuffling and reconstruction on the FaceForensics++ dataset by the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further illustrated below with reference to specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0032] The face forgery detection method based on local area regularization of this embodiment uses a face detection model to perform face detection on the input image and crops the face area to obtain a face image. I c ; The face image I c After scaling to a fixed size, the image is divided into blocks, and the blocks are randomly shuffled and the order of arrangement is recorded. The new image generated by the shuffled order is input into the convolutional neural network to extract image features. During model training, the extracted features of the new image generated by the shuffled order are input into the position reconstruction branch to obtain the order of the image blocks, thereby encouraging the model to model the correlation of the image blocks. The extracted features of the new image generated by the shuffled order are then input into the classifier to obtain the probability that the final image is a forged image. The specific steps include:

[0033] Step 1: Use the face detection model RetinaFace to detect the face of the input image and crop the face area; the details are as follows:

[0034] Use the face detection model RetinaFace to detect the input image I Perform face detection to obtain the area where the face is located in the image I f :

[0035] I f = I(x f ,y f ,w f ,h f ) (1)

[0036] Where,x f represents the horizontal coordinate of the center of the detected face, y f represents the vertical coordinate of the center of the detected face, w f represents the width of the detected face, h f represents the height of the detected face; I(x f ,y f ,w f ,h f ) represents the region in the image I centered at x f with a width of ,y f and a height of w f . h f

[0037] the face cropping region of the image I I c is:

[0038] I c = I(x f ,y f ,1.2×w f ,1.2×h f ) (2)

[0039] The above formula represents that the width and height of the face cropping region I c are 1.2 times the region I f where the face is located.

[0040] Step 2, scale the cropped image to a fixed size and block the image; specifically as follows:

[0041] Crop the face region of the image I c to a width and height of 224 pixels by using the bicubic interpolation method, so as to facilitate the blocking operation of the image and the input into the backbone network to extract features. The scaled image is divided into 49 blocks with a side length of 32 pixels.

[0042] ​Step 3: Randomly shuffle the order of these image blocks to generate a new image, and record the order of arrangement; the details are as follows:

[0043] The image is divided into image blocks of size 32×32. p(i) For the image i image blocks, i ϵ{1,2,…,49}, the new image block after shuffling p'(i) Expressed as:

[0044] p'(i) = p(α i ) (3)

[0045] Where, α Generated by randomly disrupting the order of elements in the vector [1,2,…,49], α i Represents a vector α The i elements; a new image generated by shuffling the order I * , for each image, α All will be regenerated and recorded, which is called the image arrangement order M. Figure 2 The figure shows the scrambling process of dividing an image into 7×7 image blocks. The image blocks in the image are scrambled according to the randomly generated α.

[0046] Inter-block shuffling destroys the global structure of the image, indirectly achieves equal-probability sampling of blocks, encourages the model to focus on local areas of the image, prevents the forgery detection task from degenerating into a face recognition task, enhances the translation invariance of the network, and enhances the model's ability to detect deep fake images.

[0047] Step 4: Use the backbone network Xception to extract the features of the new image generated by the shuffled order in step 3; the details are as follows:

[0048] For the new image generated in step 3, the five convolutional blocks of the original Xception network C are used to extract the image features F. The input image size is 224×224, the image features F has a size of 7×7, and the number of channels is 2048. The Xception network effectively reduces the number of parameters by performing separate convolutions on the feature channels.

[0049] Step 5: During training, the features extracted in step 4 are input into the position reconstruction branch to obtain the order of the image blocks, thereby encouraging the model to model the correlation of the image blocks; the details are as follows:

[0050] Position reconstruction branch such as Figure 3As shown, the position reconstruction branch R consists of a PixleShuffle sampling layer, a 1×1 convolution layer, and a HardTanh activation layer;

[0051] During the training process, first the new images generated by the shuffled order obtained in step 4 are I * The feature F is input to the PixleShuffle sampling layer for pixel reorganization; then it is input to the 1×1 convolution layer to obtain a feature with a size of 7×7 and a number of channels of 2; the feature is input to the HardTanh activation layer to obtain a restored order with a size of 7×7 and a number of channels of 2; the restored order and the permutation order are input to the Smooth L1 loss function to calculate the loss and position reconstruction loss Expressed as:

[0052] (4)

[0053] in, C represents the backbone network, Represents the backbone network C The parameters of the 1 to 5 convolution blocks, R Indicates the location reconstruction branch, Indicates location reconstruction branch R Parameters, I * Represents a new image generated in a disrupted order, M Represents the order of image arrangement recorded in step 3. By restoring the position of image blocks, the semantic association between local regions can be modeled, thus enhancing the detection capability of the network.

[0054] Step 6: Disorganize the order of the new image obtained in step 4 I * The feature F is input into the classifier to obtain the probability that the final image is a forged image, as follows:

[0055] The classifier consists of an average pooling layer and a fully connected layer; the new image extracted in step 4 I * The size of the feature F is 7×7 and the number of channels is 2048. The feature is input into the average pooling layer to obtain a feature with a size of 1×1 and the number of channels is 2028. After passing through a fully connected layer, a probability is output to indicate the probability that the image is a forged image. The cross entropy loss of the classification is expressed as:

[0056] (5)

[0057] in, Represents the backbone network C Parameters, y Indicates the authenticity label of the image.

[0058] Table 1 Comparison of the present invention on the FaceForensics++ dataset

[0059]

[0060] Table 1 compares the results of the proposed method on the FaceForensics++ dataset. The evaluation was conducted on FaceForensics++'s C23 and C40 video quality data. The model was trained on the C23 data and then tested on both video quality levels, with detection accuracy as the metric. It can be seen that the proposed method performs well when tested with C40 video quality, which did not appear in the training set, and also surpasses the original method in detection accuracy for C23 video quality. This demonstrates that the proposed method can effectively enhance the generalization capability of face forgery detection network models.

[0061] Figure 4 The results of scrambling and reconstructing real data from the FaceForensics++ dataset and data forged using different methods are shown. The first row shows the original image, the second row shows the scrambled image, and the third row shows the reconstructed image. As can be seen, our method restores the original scrambled face image to its original correct arrangement. Figure 5 The distance statistics between the original image blocks and the reconstructed image blocks on the FaceForensics++ dataset are shown. It can be seen that more than 80% of the image blocks are restored to a position with a distance less than or equal to 1 from the original position, which proves the effectiveness of the reconstruction branch in the method of the present invention.

[0062] This paper proposes a block-shuffled recovery method to improve the generalization capabilities of forgery detection models. This method encourages the model to focus on local block-level regions and model the correlations between image blocks. This effectively addresses the limited generalization capabilities of detection methods, which can significantly degrade performance when exposed to corrupted data, with minimal performance overhead.

[0063] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A face forgery detection method based on local region regularization, characterized in that: include: The arrangement order of the image blocks formed after scaling and segmenting the face area image is randomly disrupted to generate a new image, and the arrangement order is recorded; then the features of the new image generated by the disrupted order are extracted; During training, the extracted features of the new image generated by shuffling the order are input into the position reconstruction branch to obtain the arrangement order of the image blocks, thereby encouraging the model to model the correlation of the image blocks. The extracted features of the new image generated by shuffling the order are input into the classifier to obtain the probability that the final image is a forged image. Among them, the position reconstruction branch R consists of a PixleShuffle sampling layer, a 1×1 convolution layer, and a HardTanh activation layer; During training, the features F of the new image generated by shuffling the order are first input into the PixleShuffle sampling layer for pixel reorganization; then they are input into the 1×1 convolution layer to obtain features of size 7×7 and number of channels 2; the features are input into the HardTanh activation layer to obtain the restored order of size 7×7 and number of channels 2; the restored order and the permutation order are input into the loss function to calculate the loss; Input the restored order and permutation order into the Smooth L1 loss function to calculate the loss and position reconstruction loss Expressed as: (4) Among them, C represents the backbone network, represents the parameters of the 1 to 5 convolution blocks of the backbone network C, R represents the position reconstruction branch, I represents the parameters of the position reconstruction branch R, * Represents a new image generated by disrupting the order, and M represents the order of the images.

2. The method for detecting forged faces based on local region regularization according to claim 1, characterized in that: The order of the image blocks formed after scaling and blocking the face area image is randomly shuffled to generate a new image, and the order of arrangement is recorded. In the step where the image is divided into image blocks of size 32×32, p(i) is the i-th image block of the image, iϵ{1,2,…,49}, and the new image block p'(i) after shuffling is expressed as: p'(i) = p(a i ) (3) In the formula, α is generated by randomly disrupting the order of elements in the vector [1,2,…,49], i Represents the i-th element in the vector α; the new image I generated by shuffling the order * , for each image, α will be regenerated and recorded, which is called the image arrangement order M.

3. The face forgery detection method based on local region regularization according to claim 1, characterized in that: In the feature extraction step of the new image generated by the disordered order, the five convolutional blocks of the backbone network Xception are used to extract the new image I generated by the disordered order. * Features F.

4. The method for detecting forged faces based on local region regularization according to claim 1, wherein: The classifier consists of an average pooling layer and a fully connected layer; the new image I generated by shuffling the order * The size of the feature F is 7×7 and the number of channels is 2048. The feature is input into the average pooling layer to obtain a feature with a size of 1×1 and the number of channels is 2028. After passing through a fully connected layer, a probability is output, indicating the probability that the image is a forged image.

5. The method for detecting forged faces based on local region regularization according to claim 4, characterized in that: The cross entropy loss for classification is expressed as: (5) in, represents the parameters of the backbone network C, and y represents the true or false label of the image.

Images