Deepfake detection method and device based on image diversification features

Abstract

The application discloses a kind of based on image diversification feature's deep fake detection method and device, comprising: obtaining the color face image to be identified, it is changed into gray image, texture feature is extracted by the SRNet neural network added with channel attention mechanism, and first feature map is obtained;The semantic feature capable of representing the overall mode of original image is extracted from the color face image to be identified by the XceptionNet neural network added with spatial attention mechanism, and second feature map is obtained;First, second feature map is placed into bidirectional GRU neural network added with timing attention mechanism respectively, and the timing feature capable of representing frame level difference is extracted, and third, fourth feature map is obtained;Third, fourth feature map is fused by the splicing of channel dimension, and fifth feature map is obtained;Fifth feature map is sent into true-false classifier, and the detection result of face image is obtained.The application can effectively improve the detection ability to deep fake sample.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision and deep learning, and in particular to a method and apparatus for detecting deep forgeries based on diverse image features. Background Technology

[0002] In recent years, technologies that generate fake videos and images through digital manipulation, such as DeepFake, Face2Face, and FaceSwap, have come to the forefront of public attention.

[0003] While deep learning-based methods have achieved remarkable results in deepfake detection, most existing methods neglect the performance improvements brought by feature diversity and fail to fully utilize the advantages of different features. Furthermore, the enhancements to temporal networks brought by attention mechanisms are easily overlooked by researchers, as forged videos require processing all frames in a coherent manner. Low-level artifacts caused by face manipulation are expected to further manifest as temporal artifacts with frame-level inconsistencies, necessitating the allocation of different weights to different time series during detection. Therefore, designing deepfake detection methods based on diverse image features has become one of the urgent technical challenges in the development of face authentication. Summary of the Invention

[0004] The primary objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an effective and scientifically sound method for deepfake detection based on diverse image features. This method can fully capture and utilize various features, thereby enhancing the detection performance and robustness of the detection network.

[0005] A second objective of this invention is to provide a deepfake detection device based on diverse image features.

[0006] The first objective of this invention is achieved through the following technical solution: a deepfake detection method based on diverse image features, comprising the following steps:

[0007] S1. Obtain the color face image to be identified, convert it into a grayscale image, and extract the texture information that can represent the surface properties of the image through the SRNet neural network with added channel attention mechanism to obtain the first feature map.

[0008] S2. Extract semantic features that can represent the overall pattern of the original image from the color face image obtained in step S1 using the XceptionNet neural network with added spatial attention mechanism to obtain the second feature map;

[0009] S3. The first feature map and the second feature map are respectively fed into a bidirectional GRU neural network with a temporal attention mechanism to extract temporal information that can represent frame-level differences, and the third feature map and the fourth feature map are obtained.

[0010] S4. By concatenating the channel dimensions, the third feature map obtained in step S3 is fused with the fourth feature map to obtain the fifth feature map;

[0011] S5. The fifth feature map is fed into the real / fake classifier for classification processing to obtain the detection result of the color face image to be identified, that is, whether the color face image to be identified is a real face image or a fake face image that has been tampered with.

[0012] Furthermore, in step S1, the color face image is a single frame of real face image or a fake face image modified by a deepfake algorithm; the face region in the video frame is detected using the Dlib database, and then the face region image in the color face image to be identified is cropped out, and the cropped face region image is scaled to M×M, where M represents the number of pixels in the width and height directions of the image;

[0013] An SRNet neural network is selected as the backbone network for extracting texture information. The color face image to be recognized is converted into a single-channel grayscale image. The texture information of the color face image is extracted through the input block of the untrained SRNet neural network, where the feature map includes multiple local block features. Each local block feature in the feature map is converted into a channel-coded feature. The channel weight information of each local block is set according to the channel-coded feature, resulting in a matrix of size C×1×1. Finally, the values ​​in the matrix are normalized to the 0-1 interval using the Sigmoid function to obtain the channel attention weight map W. c , where C is the number of feature channels;

[0014] Using channel attention weight map W c The input features are weighted to obtain a weighted grayscale feature map. The calculation formula is as follows:

[0015]

[0016] In the formula, I represents the input feature, and F n This represents the weighted grayscale feature map, and ⊙ represents broadcast multiplication. This represents the addition of elements one by one;

[0017] The weighted grayscale feature map is passed through the intermediate block of the SRNet neural network to perform texture feature analysis on the color face image, resulting in the first feature map.

[0018] Further, in step S2, an XceptionNet neural network is selected as the backbone network for extracting semantic information; through the input blocks of the XceptionNet neural network pre-trained on ImageNet, features are extracted from the color face image to be recognized to obtain an RGB feature map; the grayscale feature map in step S1 is downsampled and converted into a feature map with the same width and height as the RGB feature map; each local block feature in the feature map is converted into a positional encoding feature, and the positional weight information of each local block is set according to the positional encoding feature to obtain a matrix of size 1×H×W, where H and W are the height and width of the feature, respectively; finally, the values ​​in the matrix are normalized to the 0-1 interval using the Sigmoid function to obtain the spatial attention weight map w. s The calculation formula is:

[0019] F g =down(F n )

[0020] w s =σ(f 7*7 ([AvgPool(F g ),MaxPool(F g )]))

[0021] In the formula, down(·) represents downsampling, σ represents the Sigmoid function, and f 7*7 F represents a convolutional neural network with a filter size of 7×7. n Represents the grayscale feature map, F g A feature map with the same width and height as the RGB feature map; using a spatial attention weight map w s The RGB feature maps are weighted to obtain the attention-added features. The calculation formula is as follows:

[0022]

[0023] In the formula, F r This represents the RGB feature map, F' represents the final feature map after attention is added, and ⊙ represents broadcast multiplication. This represents the addition of elements one by one;

[0024] The attention-added features are passed through the intermediate block of the XceptionNet neural network to perform semantic feature analysis on the color face image, resulting in a second feature map.

[0025] Furthermore, in step S3, temporal information representing frame-level differences is extracted, and attention enhancement processing is performed on the video frame sequence based on the corresponding temporal weight information, as detailed below:

[0026] A bidirectional GRU neural network was selected as the temporal network for extracting temporal information;

[0027] The first and second feature maps are respectively passed through an untrained bidirectional GRU neural network to extract the hidden layer states S of the feature maps. t The hidden layer state is derived from the forward hidden layer state and the reverse hidden layer state;

[0028] The hidden layer state S t The network is activated by the tanh function to enhance its nonlinear transformation capability, and then normalized to the 0-1 interval by the softmax function to obtain the temporal attention weight map w. t The calculation formula is:

[0029]

[0030] In the formula, tanh(·) represents the tanh activation function, u v Represents a randomly initialized attention matrix, ∑ t This represents the cumulative summation over the time interval 0-t; using a temporal attention weight graph w. t The temporal features are weighted to obtain the weighted third and fourth feature maps.

[0031] Furthermore, in step S4, the different temporal features obtained in step S3 are fused by concatenating the channel dimensions to obtain the fifth feature map, as detailed below:

[0032] The fusion method selected is feature concatenation at the channel dimension. The third feature map and the fourth feature map are concatenated to obtain the feature concatenation matrix.

[0033] The feature concatenation matrix is ​​transformed into a pre-defined size for the classifier using a convolutional neural network to obtain the fifth feature map.

[0034] The second objective of this invention is achieved through the following technical solution: a deepfake detection device based on diverse image features, used to implement the aforementioned deepfake detection method based on diverse image features, comprising:

[0035] The data acquisition module is used to acquire a color face image to be identified, wherein the color face image is a single frame of real face image or a fake face image modified by a deepfake algorithm;

[0036] The data preprocessing module is used to preprocess the color face images acquired by the data acquisition module. In the preprocessing process, the face region in the image is first detected using the Dlib database, the face region image is cropped and scaled to a preset size.

[0037] The data processing module is used to input the preprocessed face image into an SRNet neural network with channel attention mechanism, an XceptionNet neural network with spatial attention mechanism, and a bidirectional GRU neural network with temporal attention mechanism to extract semantic features, texture features, and temporal features of the image. The output result is sent to a real / fake classifier to determine whether the color face image to be identified is a real face image or a forged face image that has been tampered with.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] 1. This invention captures semantic information, texture information, and temporal information of images, and has more diverse features that can effectively help the final classifier distinguish between genuine and fake images.

[0040] 2. This invention fully utilizes the advantages of different features, and uses the spatial attention map generated by image texture information to perform attention enhancement processing on the spatial feature map, which effectively strengthens the attention to the fake regions in the face region.

[0041] 3. This invention employs channel attention mechanism and temporal attention mechanism to perform attention enhancement processing on corresponding local blocks, effectively strengthening the network's attention to key areas. Attached Figure Description

[0042] Figure 1 This is a flowchart of the deepfake detection method disclosed in Example 1.

[0043] Figure 2 This is an architecture diagram of deepfake detection disclosed in Example 1.

[0044] Figure 3 This is a detailed diagram of the attention enhancement processing disclosed in Example 1.

[0045] Figure 4 This is a schematic diagram of the deepfake detection device disclosed in Example 2. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment discloses a deepfake detection method based on diverse image features, including the following steps:

[0049] S1. Obtain the color face image to be identified, convert it into a grayscale image, and extract the texture information that can represent the surface properties of the image through the SRNet neural network with added channel attention mechanism to obtain the first feature map.

[0050] The color face image is either a single frame of a real face image or a fake face image modified by a deepfake algorithm. The face region in the video frame is detected using the Dlib database, and then the face region image in the color face image to be identified is cropped out. The cropped face region image is then scaled to an M×M size, where M represents the number of pixels in the width and height directions of the image.

[0051] This invention trains a network model based on the FaceForensics++ dataset, which contains 1,000 real videos from the Internet. Each real video corresponds to a deepfake video generated by four different operation methods: Deepfakes (DF), NeuralTextures (NT), FaceSwap (FS), and Face2Face (F2F).

[0052] The Dlib database, which uses 68 points to mark important facial features, is used to obtain the x-coordinate of the lower left corner, y-coordinate of the lower left corner, height h, and width w of the face region. Then, the face region is cropped according to (x, y, h, w) and scaled to a preset size of 299×299.

[0053] like Figure 2 As shown, the SRNet neural network is selected as the backbone network to extract texture information. The color face image to be identified is converted into a single-channel grayscale image. The texture information of the color face image is extracted through the input block of the untrained SRNet neural network. The feature map includes multiple local block features.

[0054] Each local block feature in the feature map is converted into a channel-encoded feature. Channel weights for each local block are then set based on these channel-encoded features, resulting in a C×1×1 matrix. Finally, the values ​​in this matrix are normalized to the 0-1 range using the Sigmoid function, yielding the channel attention weight map W. c , where C is the number of feature channels.

[0055] Using channel attention weight map W c The input features are weighted to obtain a weighted grayscale feature map. The calculation formula is as follows:

[0056]

[0057] In the formula, I represents the input feature, and F n This represents the weighted grayscale feature map, and ⊙ represents broadcast multiplication. This represents the addition of elements one by one.

[0058] The weighted grayscale feature map is passed through the intermediate block of the SRNet neural network to perform texture feature analysis on the color face image, resulting in the first feature map.

[0059] S2. Extract semantic features that represent the overall pattern of the original image from the color face image obtained in step S1 using the XceptionNet neural network with added spatial attention mechanism to obtain the second feature map;

[0060] like Figure 2 As shown, the XceptionNet neural network is selected as the backbone network for extracting semantic information. The XceptionNet neural network input block, which is pre-trained on ImageNet, is used to extract features from the color face image to be recognized, and the RGB feature map is obtained.

[0061] like Figure 3 As shown, the grayscale feature map in step S1 is downsampled and converted into a feature map with the same width and height as the RGB feature map.

[0062] Each local block feature in the feature map is converted into a position-encoded feature. Based on these position-encoded features, positional weights are set for each local block, resulting in a 1×H×W matrix, where H and W are the height and width of the feature, respectively. Finally, the values ​​in this matrix are normalized to the 0-1 range using the Sigmoid function, yielding the spatial attention weight map w. s The calculation formula is:

[0063] F g =down(F n )

[0064] w s =σ(f 7*7 ([AvgPool(F g ),MaxPool(F g )]))

[0065] In the formula, down(·) represents downsampling, σ represents the Sigmoid function, and f 7*7 F represents a convolutional neural network with a filter size of 7×7. n Represents the grayscale feature map, F g A feature map with the same width and height as the RGB feature map; using a spatial attention weight map w s The RGB feature maps are weighted to obtain the attention-added features. The calculation formula is as follows:

[0066]

[0067] In the formula, F r This represents the RGB feature map, F' represents the final feature map after attention is added, and ⊙ represents broadcast multiplication. This represents the addition of elements one by one.

[0068] The attention-added features are passed through the intermediate block of the XceptionNet neural network to perform semantic feature analysis on the color face image, resulting in a second feature map.

[0069] S3. The first feature map and the second feature map are respectively fed into a bidirectional GRU neural network with a temporal attention mechanism to extract temporal information that can represent frame-level differences, and the third feature map and the fourth feature map are obtained.

[0070] Specifically, a bidirectional GRU neural network is selected as the temporal network for extracting temporal information. The first feature map and the second feature map are respectively passed through the untrained bidirectional GRU neural network to extract the hidden layer state S of the feature maps. t The hidden layer state is derived from the forward hidden layer state and the reverse hidden layer state.

[0071] The hidden layer state S t The network is activated by the tanh function to enhance its nonlinear transformation capability, and then normalized to the 0-1 interval by the softmax function to obtain the temporal attention weight map w. t The calculation formula is:

[0072]

[0073] In the formula, tanh(·) represents the tanh activation function, u v Represents a randomly initialized attention matrix, ∑ t This represents the cumulative summation over the time interval 0-t; using a temporal attention weight graph w. t The time-series features are weighted to obtain the weighted third and fourth feature maps.

[0074] S4. By concatenating the channel dimensions, the third feature map obtained in step S3 is fused with the fourth feature map to obtain the fifth feature map, as follows:

[0075] The fusion method selected is feature concatenation at the channel dimension. The third feature map and the fourth feature map are concatenated to obtain the feature concatenation matrix.

[0076] The feature concatenation matrix is ​​transformed into a pre-defined size for the classifier using a convolutional neural network to obtain the fifth feature map.

[0077] S5. The fifth feature map is fed into the real / fake classifier for classification processing to obtain the detection result of the color face image to be identified, that is, whether the color face image to be identified is a real face image or a fake face image that has been tampered with.

[0078] To verify the detection performance of the aforementioned deepfake detection method based on diverse image features, the FaceForensics++ and Celeb-DF datasets were used for performance testing. The FaceForensics++ dataset was divided into two compression levels: High Quality (HQ) and Low Quality (LQ). To evaluate the classification results obtained from the experiments, two evaluation metrics were introduced based on practical applications: accuracy and the area under the ROC curve (AUC), as shown in the following formula:

[0079]

[0080]

[0081] Among them, True Positive (TP) means that the predicted value of the sample matches the true value and both are positive; True Negative (TN) means that the predicted value of the sample matches the true value and both are negative; False Positive (FP) means that the predicted value of the sample is positive while the true value is negative; False Negative (FN) means that the predicted value of the sample is negative while the true value is positive. Let represent the index of the i-th sample (arranged in ascending order of probability scores, at the rank position), and let represent the sum of the indices of only positive samples. M′ and N are the number of positive samples and the number of negative samples, respectively.

[0082] Accuracy is widely used to determine whether a predicted value matches the true value, i.e., true or false. AUC is an evaluation metric for binary classification models, representing the probability that a predicted positive sample is ranked ahead of a negative sample. It is easy to determine the impact of any threshold on the network's generalization performance.

[0083] Table 1. Test results of the network at different compression ratios on the FaceForensics++ dataset.

[0084]

[0085] Table 1 shows the test results for different compression ratios on the FaceForensics++ dataset. This invention flexibly captures diverse features and allows different features to interact fully, achieving the best accuracy in detecting FaceForensics++ datasets with different compression ratios.

[0086] The present invention further tests the generalization performance of the network by training the network on FaceForensics++ and testing it on Celeb-DF. The results are shown in Table 2. The test results show that the deep forgery detection method based on diverse image features provided by the present invention has good generalization performance.

[0087] Table 2. AUC values ​​of the network trained on FaceForensics++ and tested on Celeb-DF.

[0088] MesoNet 84.70 54.80 SRNet 97.91 68.12 XceptionNet 99.70 65.30 This invention 99.80 76.41

[0089] Example 2

[0090] This embodiment discloses a deepfake detection device based on diverse image features, used to implement the deepfake detection method based on diverse image features described in Embodiment 1, such as... Figure 4 As shown, the system includes the following functional modules:

[0091] The data acquisition module is used to acquire a color face image to be identified, wherein the color face image is a single frame of real face image or a fake face image modified by a deepfake algorithm;

[0092] The data preprocessing module is used to preprocess the color face images acquired by the data acquisition module. In the preprocessing process, the face region in the image is first detected using the Dlib database, the face region image is cropped and scaled to a preset size.

[0093] The data processing module is used to input the preprocessed face image into an SRNet neural network with channel attention mechanism, an XceptionNet neural network with spatial attention mechanism, and a bidirectional GRU neural network with temporal attention mechanism to extract semantic features, texture features, and temporal features of the image. The output result is sent to a real / fake classifier to determine whether the color face image to be identified is a real face image or a tampered face image.

[0094] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A deepfake detection method based on diverse image features, characterized in that, Includes the following steps: S1. Obtain the color face image to be identified, convert it into a grayscale image, and extract the texture information that can represent the surface properties of the image through the SRNet neural network with added channel attention mechanism to obtain the first feature map. The color face image is either a single frame of real face image or a fake face image modified by a deepfake algorithm; the face region in the video frame is detected using the Dlib database, and then the face region image in the color face image to be identified is cropped out, and the cropped face region image is scaled to M×M, where M represents the number of pixels in the width and height directions of the image; An SRNet neural network is selected as the backbone network for extracting texture information; the color face image to be identified is converted into a single-channel grayscale image, and the texture information of the color face image is extracted through the input block of the untrained SRNet neural network, wherein the feature map includes multiple local block features. Each local block feature in the feature map is converted into a channel-encoded feature. Channel weights for each local block are then set based on these channel-encoded features, resulting in a C×1×1 matrix. Finally, the values ​​in this matrix are normalized to the 0-1 range using the Sigmoid function to obtain the channel attention weight map. , where C is the number of feature channels; Using channel attention weight map The input features are weighted to obtain a weighted grayscale feature map. The calculation formula is as follows: ； In the formula, Represents input features, This represents the weighted grayscale feature map. Represents broadcast multiplication. This represents the addition of elements one by one; The weighted grayscale feature map is passed through the intermediate block of the SRNet neural network to perform texture feature analysis on the color face image, thereby obtaining the first feature map; S2. Extract semantic features that can represent the overall pattern of the original image from the color face image obtained in step S1 using the XceptionNet neural network with added spatial attention mechanism to obtain the second feature map; An XceptionNet neural network is selected as the backbone network for extracting semantic information. Features are extracted from the color face image to be recognized using input blocks of the XceptionNet neural network pre-trained on ImageNet, resulting in an RGB feature map. The grayscale feature map from step S1 is downsampled and converted into a feature map with the same width and height as the RGB feature map. Each local block feature in the feature map is converted into a position-encoded feature, and the position weight information of each local block is set according to the position-encoded feature, resulting in a matrix of size 1×H×W, where H and W are the height and width of the feature, respectively. Finally, the values ​​in this matrix are normalized to the 0-1 interval using the Sigmoid function to obtain the spatial attention weight map. The calculation formula is: ； ； In the formula, This represents downsampling. Represents the Sigmoid function. This represents a convolutional neural network with a filter size of 7×7. Representative grayscale feature map, A feature map with the same width and height as the RGB feature map; using spatial attention weight maps. The RGB feature maps are weighted to obtain the attention-added features. The calculation formula is as follows: ； In the formula, Represents the RGB feature map. This represents the final features after attention has been added. Represents broadcast multiplication. This represents the addition of elements one by one; The attention-added features are passed through the intermediate block of the XceptionNet neural network to perform semantic feature analysis on the color face image, resulting in a second feature map. S3. The first feature map and the second feature map are respectively fed into a bidirectional GRU neural network with a temporal attention mechanism to extract temporal information that can represent frame-level differences, and the third feature map and the fourth feature map are obtained. Temporal information representing frame-level differences is extracted, and attention enhancement processing is performed on the video frame sequence based on the corresponding temporal weight information, as detailed below: A bidirectional GRU neural network was selected as the temporal network for extracting temporal information; The first and second feature maps are respectively passed through an untrained bidirectional GRU neural network to extract the hidden layer states of the feature maps. The hidden layer state is derived from the forward hidden layer state and the reverse hidden layer state; Hidden layer state The network's nonlinear transformation capability is enhanced by applying the tanh activation function, and finally normalized to the 0-1 range using the Softmax function to obtain the temporal attention weight map. The calculation formula is: ； In the formula, Represents the tanh activation function. This represents a randomly initialized attention matrix. This represents the cumulative summation over the time interval 0-t; using a temporal attention weight graph. The temporal features are weighted to obtain the weighted third and fourth feature maps; S4. By concatenating the channel dimensions, the third feature map obtained in step S3 is fused with the fourth feature map to obtain the fifth feature map; S5. The fifth feature map is fed into the real / fake classifier for classification processing to obtain the detection result of the color face image to be identified, that is, whether the color face image to be identified is a real face image or a fake face image that has been tampered with.

2. The deepfake detection method based on diverse image features according to claim 1, characterized in that, In step S4, the different temporal features obtained in step S3 are fused by concatenating the channels to obtain the fifth feature map, as detailed below: The fusion method selected is feature concatenation at the channel dimension. The third feature map and the fourth feature map are concatenated to obtain the feature concatenation matrix. The feature concatenation matrix is ​​transformed into a pre-defined size for the classifier using a convolutional neural network to obtain the fifth feature map.

3. A deepfake detection device based on diverse image features, characterized in that, To implement the deepfake detection method based on diverse image features as described in claim 1 or 2, the method comprises: The data acquisition module is used to acquire a color face image to be identified, wherein the color face image is a single frame of real face image or a fake face image modified by a deepfake algorithm; The data preprocessing module is used to preprocess the color face images acquired by the data acquisition module. In the preprocessing process, the face region in the image is first detected using the Dlib database, the face region image is cropped and scaled to a preset size. The data processing module is used to input the preprocessed face image into an SRNet neural network with channel attention mechanism, an XceptionNet neural network with spatial attention mechanism, and a bidirectional GRU neural network with temporal attention mechanism to extract semantic features, texture features, and temporal features of the image. The output result is sent to a real / fake classifier to determine whether the color face image to be identified is a real face image or a forged face image that has been tampered with.

Images