A deep fake detection method based on cross-domain feature fusion and separable watermarking
Patent Information
- Application Number
- CN202411027846.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2044-07-30
AI Technical Summary
然而，SepMark模型却不能实现对未嵌入水印或擦除水印伪造图像的检测
[0042]本发明相较于SepMark模型，补充了对于未嵌入水印或擦除水印的伪造图像的被动检测功能，构造针对深度伪造图像具有主动取证和被动取证的全防御网络模型。特别是采用对ResNet18网络进行改进的BasicBlock模块，在空域位置增加了通道和空间注意力模块，可以自动学习区分通道或空间位置信息的主次，从而有选择性地强化空域特征和频域特征重要信息，抑制不重要信息，实现空域和频域特征的交叉融合；利用融合后的特征，可以准确识别细微的伪造特征，从而判断图像的真伪。由于引入反映图像的全局信息和纹理信息的频域特征，可以提高深度伪造检测的精度、在面对各种攻击和干扰时具有更强的鲁棒性以及更高泛化能力。
Smart Images
Figure CN119068321B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a deepfake detection method, and more particularly to a deepfake detection method based on cross-domain feature fusion and separable watermarking. Background Technology
[0002] Currently, deepfake detection methods are divided into two types: passive evidence collection and active evidence collection.
[0003] Passive forensics relies on traces left over from the forgery process or differences in the statistical distribution between forged and genuine images to determine authenticity. The main methods are as follows:
[0004] Visual feature-based methods rely on analyzing visual anomalies in images or videos, such as unnatural facial expressions and abnormal blinking frequencies. For example, research on the FaceForensics++ dataset found that analyzing blinking frequency can effectively detect early deepfake videos, as the blinking frequency of people in these videos is significantly lower than the normal range of 15-30 times per minute. However, with the advancement of forgery techniques, these visual features are becoming increasingly difficult to identify.
[0005] CNN-based spatial domain methods: These methods primarily utilize convolutional neural networks (CNNs) to extract image features in the spatial domain, which are then used by a classifier for identification. The advantage of this approach is its ability to automatically learn effective features and its generally good performance on large datasets. However, its disadvantages include the need for extensive labeled training data and potential poor performance against novel or unseen forgery techniques.
[0006] Frequency domain analysis-based methods detect forgery traces by analyzing the frequency domain features of images or videos. The FouSeg method is a good example; it uses the Discrete Fourier Transform (DFT) to analyze the frequency domain features of an image, detecting specific patterns exhibited in the high-frequency domain by segmenting the image's spectrum. These methods are effective against certain forgery techniques but struggle against novel forgery methods.
[0007] Biometric-based methods utilize physiological characteristics such as heart rate and blood flow to detect forgeries. The DeepVit method extracts heart rate information by analyzing subtle color changes in the facial skin of a person in a video, thus distinguishing between real and fake videos. However, it is difficult to implement and is easily affected by environmental factors.
[0008] In summary, although existing passive forensics detection methods have achieved certain results in specific scenarios, they still have problems such as insufficient generalization ability, insufficient robustness, poor interpretability, insufficient real-time performance, and poor cross-domain adaptability. In particular, passive forensics has the problem of being unable to immediately stop the spread of forged images.
[0009] The existing SepMark model is one of the active forensics methods. The SepMark model consists of an encoder and two separable decoders: a robust decoder and a semi-robust decoder. The two separable decoders have the same network structure but different weights, so the decoders will extract the watermark twice from noisy images. During training, the robust decoder aims to extract the pre-embedded watermark under any noise attack, while the semi-robust decoder aims to extract the watermark under distortion attacks but not under deepfake attacks. The encoder embeds a watermark (such as the publisher's identifier and timestamp) into the protected facial image, while the two separable decoders extract the robust watermark and the semi-robust watermark, respectively. The SepMark model does not detect which images are forged, but rather focuses on ensuring which images are genuine and identifying their origin. Furthermore, even without knowing the content of the original watermark, it is generally possible to compare the two extracted watermarks to determine whether the tagged facial image has been maliciously modified after publication and to immediately prevent the further propagation of suspicious images that do not match the watermark. However, the SepMark model cannot detect forged images without embedded watermarks or with erased watermarks.
[0010] To date, there have been no reports on deepfake detection methods based on cross-domain feature fusion and separable watermarking. Summary of the Invention
[0011] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing a deep forgery detection method based on cross-domain feature fusion and separable watermarking.
[0012] The technical solution of this invention is: a deepfake detection method based on cross-domain fusion and separable watermarking, which is carried out in the following steps:
[0013] Step 1. Image encoding determination
[0014] An adversarial coding discriminator is used to determine whether the image contains a SepMark watermark. If yes, proceed to step 2; otherwise, proceed to step 3.
[0015] Step 2. Input the image into the two separable decoders of the SepMark module. The two decoders decode the robust watermark and the semi-robust watermark respectively. Compare whether the two watermarks have significant differences to detect whether the image has been tampered with. End.
[0016] Step 3. Two-domain transformation
[0017] Step 3.1 Read the image tensor T iThe system uses 7×7 convolutional layers, followed by batch normalization and activation function processing, and finally max pooling to extract features, resulting in 64-channel low-dimensional spatial features, which are used as the original spatial tensor T. S ;
[0018] Step 3.2 Read the image tensor T i Multiply by 255 to revert to an RGB channel image; convert the RGB image to a YCrCb image, then convert it to a tensor type; perform a DCT transform on the YCrCb image tensor to obtain the frequency domain tensor T. f ;
[0019] Step 4: Convert the spatial tensor T S and frequency domain tensor T f The input is fed into the BasicBlock module for further spatial and frequency domain feature extraction and local feature fusion.
[0020] Step 4.1 Spatial Tensor T f The spatial features f are obtained through a convolutional layer Conv1_1 and a batch normalization layer Bn1-1. s Then, after passing through the channel attention module and the spatial attention module, the spatial features f with the added attention mechanism are obtained. s ′;Frequency domain tensor T f The frequency domain features f are obtained through a convolutional layer Conv2_1 and a batch normalization layer Bn2-1. f ;
[0021] Step 4.2 will add the attention mechanism to the spatial features f s ′ and frequency domain features f f The input is fed into the local feature fusion module for feature fusion.
[0022] Step 4.2.1 Let W s and W f These are the spatial features f after the attention mechanism has been added. s ′ and frequency domain features f f The absolute value of the batch normalized weights, θ w The threshold used to classify importance;
[0023] W s ≥θ w The corresponding feature channels are called important spatial feature channels, and W s <θ w The corresponding feature channels are called spatial secondary feature channels;
[0024] W f ≥θ w The corresponding characteristic channels are called important characteristic channels in the frequency domain, and W f <θw The corresponding feature channel is called the frequency domain secondary feature channel;
[0025] Step 4.2.2 will W s W f Sort each in descending order to obtain the sorted W. s index n s And sorted W f index n f ;
[0026] Step 4.2.3 Initialize shape and f s The same zero tensor Z s and shape with f f The same zero tensor Z f ;
[0027] Step 4.2.4 Preserve the spatial features f after adding the attention mechanism. s The important feature channels in ' are sorted according to the index n. f Using frequency domain features f f Important feature channels in the spatial feature f after adding an attention mechanism s The secondary feature channels in ' are used to obtain the spatial feature Z after cross-domain fusion. s ;
[0028] Step 4.2.5 Retain the important feature channels in the frequency domain features, based on the sorted index n s Using spatial features f with added attention mechanism s The important feature channel in ' replaces the frequency domain feature f f The secondary feature channels in the data are used to obtain the spatial feature Z after cross-domain fusion. f ;
[0029] Step 4.2.6 merging the spatial features Z from the cross-domain fusion s The ReLU activation function is applied, followed by a convolutional layer Conv1_2 and a batch normalization layer Bn1-2 to obtain the spatial features Z after cross-domain fusion of the activated features. s ′;
[0030] Step 4.2.7 merging the frequency domain features Z from the cross-domain fusion f The ReLU activation function is applied, followed by a convolutional layer Conv2_2 and a batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after cross-domain fusion of the activated features. f ;
[0031] Step 4.2.8 fuses the activated cross-domain spatial features Z s ′ and spatial tensor T SThe addition completes the residual connection. The result after residual connection is activated by the ReLU activation function to obtain the locally fused spatial features F. s ;
[0032] Step 4.2.9 merging the activated cross-domain frequency domain features Z′ f With frequency domain tensor T f The residual connection is completed by progressive addition. The result after residual connection is activated by the ReLU activation function to obtain the locally fused frequency domain features F. f ;
[0033] Step 5. Using the obtained locally fused spatial features F respectively s Frequency domain features F of local fusion f As spatial and frequency domain tensors, repeat step 4 until the processed spatial feature F is obtained. s ′ and frequency domain features F f Proceed to step 6;
[0034] Step 6. Global Feature Fusion
[0035] Step 6.1 Process the spatial features F s ′ and frequency domain features F f Perform average pooling;
[0036] Step 6.2: Feed the result after average pooling into the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial features. s Judgment result d of local fusion frequency domain features f ;
[0037] Step 7. Based on the judgment result vector, perform authenticity verification on the image.
[0038] Step 7.1 will d s ,d f The final evidence result y of the model is obtained by linearly combining the following formulas.
[0039] y=αd s +βd f
[0040] Where α and β are the parameters used in training the follow-up detection model;
[0041] Step 7.2 compares the final evidence result y with the labels of the training dataset to obtain the true / false image identification result, and then ends.
[0042] Compared to the SepMark model, this invention supplements the passive detection function for forged images without embedded watermarks or with erased watermarks, constructing a comprehensive defense network model for deepfake images with both active and passive forensic capabilities. Specifically, it employs a BasicBlock module, an improvement on the ResNet18 network, adding channel and spatial attention modules in the spatial domain. This allows for the automatic learning and differentiation of the importance of channel or spatial location information, selectively strengthening important spatial and frequency domain features while suppressing unimportant information, achieving cross-fusion of spatial and frequency domain features. Using the fused features, subtle forgery characteristics can be accurately identified, thus determining the authenticity of the image. By introducing frequency domain features that reflect the global and texture information of the image, the accuracy of deepfake detection is improved, along with stronger robustness and higher generalization ability against various attacks and interference. Attached Figure Description
[0043] Figure 1 This is a flowchart of an embodiment of the present invention.
[0044] Figure 2 This is a schematic diagram of the BasicBlock module in an embodiment of the present invention. Detailed Implementation
[0045] This invention provides a deepfake detection method based on cross-domain fusion and separable watermarking, such as... Figure 1 The steps shown are as follows:
[0046] Step 1. Image encoding determination
[0047] An adversarial coding discriminator is used to determine whether the image contains a SepMark watermark. If yes, proceed to step 2; otherwise, proceed to step 3.
[0048] The SepMark watermark is a watermark embedded in an image by the image publisher using an existing SepMark model encoder;
[0049] Step 2. Input the image into the two separable decoders of the SepMark module. According to the existing method, the two SepMark decoders decode the robust watermark and the semi-robust watermark respectively. Compare whether the two watermarks have significant differences to detect whether the image has been tampered with. End.
[0050] Step 3. Two-domain transformation
[0051] Step 3.1 Read the image tensor T i The process involves a 7×7 convolutional layer, followed by batch normalization and activation function processing, and finally max pooling for feature extraction (FE), resulting in 64-channel low-dimensional spatial features, which are used as the original spatial tensor T. S ;
[0052] Step 3.2 Read the image tensor T i Multiply by 255 to revert to an RGB channel image; convert the RGB image to a YCrCb image, then convert it to a tensor type; perform a DCT transform on the YCrCb image tensor to obtain the frequency domain tensor T. f ;
[0053] Step 4: Convert the spatial tensor T S and frequency domain tensor T f Input to such Figure 2 The BasicBlock module shown performs further spatial and frequency domain feature extraction and local feature fusion.
[0054] Step 4.1 Spatial Tensor T f The spatial features f are obtained through a convolutional layer Conv1_1 and a batch normalization layer Bn1-1. s Then, after passing through the channel attention module and the spatial attention module, the spatial features f with the added attention mechanism are obtained. s ′;Frequency domain tensor T f The frequency domain features f are obtained through a convolutional layer Conv2_1 and a batch normalization layer Bn2-1. f ;
[0055] f s =BN(Conv(T) s ));
[0056] f f =BN(Conv(T) f ));
[0057] f s =Attention(f) s );
[0058] Step 4.2 will add the attention mechanism to the spatial features f s ′ and frequency domain features f f The input is fed into the local feature fusion module for feature fusion.
[0059] Step 4.2.1 Let W s and W f These are the spatial features f after the attention mechanism has been added. s ′ and frequency domain features f f The absolute value of the batch normalized weights, θ w The threshold used to classify importance;
[0060] The feature channels and the weights of the normalization layer are in one-to-one correspondence. Assuming the feature tensor shape is [1, N, 512, 512], the weight parameters of the corresponding normalization layer are a 1×N one-dimensional array, that is, one feature channel corresponds to one weight. Therefore, the importance of the feature channels can be divided according to the weights of the normalization layer and the set threshold.
[0061] W s ≥θ w The corresponding feature channels are called important spatial feature channels, and W s <θ w The corresponding feature channels are called spatial secondary feature channels;
[0062] W f ≥θ w The corresponding characteristic channels are called important characteristic channels in the frequency domain, and W f <θ w The corresponding feature channel is called the frequency domain secondary feature channel;
[0063] Step 4.2.2 will W s W f Sort each in descending order to obtain the sorted W. s index n s And sorted W f index n f ;
[0064] Step 4.2.3 Initialize shape and f s The same zero tensor Z s and shape with f f The same zero tensor Z f ;
[0065] Step 4.2.4 Preserve the spatial features f after adding the attention mechanism. s The important feature channels in ' are sorted according to the index n. f Using frequency domain features f f Important feature channels in the spatial feature f after adding an attention mechanism s The secondary feature channels in ' are used to obtain the spatial feature Z after cross-domain fusion. s ;
[0066] Step 4.2.5 Retain the important feature channels in the frequency domain features, based on the sorted index n s Using spatial features f with added attention mechanism s The important feature channel in ' replaces the frequency domain feature f f The secondary feature channels in the data are used to obtain the spatial feature Z after cross-domain fusion. f ;
[0067] Step 4.2.6 merging the spatial features Z from the cross-domain fusion s The ReLU activation function is applied, followed by a convolutional layer Conv1_2 and a batch normalization layer Bn1-2 to obtain the spatial features Z after cross-domain fusion of the activated features. s ′;
[0068] Z s =BN(Conv(Relu(Z)) s )));
[0069] Step 4.2.7 merging the frequency domain features Z from the cross-domain fusion f The ReLU activation function is applied, followed by a convolutional layer Conv2_2 and a batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after cross-domain fusion of the activated features. f ;
[0070] Zf′=BN(Conv(Relu(Zf)));
[0071] Step 4.2.8 fuses the activated cross-domain spatial features Z s ′ and spatial tensor T S The addition completes the residual connection. The result after residual connection is activated by the ReLU activation function to obtain the locally fused spatial features F. s ;
[0072] F s =Relu(Z) s ′+T s );
[0073] Step 4.2.9 merging the activated cross-domain frequency domain features Z′ f With frequency domain tensor T f The residual connection is completed by progressive addition. The result after residual connection is activated by the ReLU activation function to obtain the locally fused frequency domain features F. f ;
[0074] F f =Relu(Z) f ′+T f );
[0075] Step 5. Using the obtained locally fused spatial features F respectively s Frequency domain features F of local fusion f As spatial and frequency domain tensors, repeat step 4 until the processed spatial feature F is obtained. s ′ and frequency domain features F f Proceed to step 6;
[0076] Step 6. Global Feature Fusion
[0077] Step 6.1 Process the spatial features F s ′ and frequency domain features F f Perform average pooling;
[0078] Step 6.2: Feed the result after average pooling into the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial features. s Judgment result d of local fusion frequency domain features f ;
[0079] d s =FC(AvePool(F S ′));
[0080] d f =FC(AvePool(F f ′));
[0081] Step 7. Based on the judgment result vector, perform authenticity verification on the image.
[0082] Step 7.1 will d s ,d f The final evidence result y of the model is obtained by linearly combining the following formulas.
[0083] y=αd s +βd f
[0084] Where α and β are the parameters used in training the follow-up detection model;
[0085] Step 7.2 compares the final evidence result y with the labels of the training dataset to obtain the true / false image identification result, and then ends.
[0086] experiment:
[0087] 1. The embodiments of the present invention were tested in-library on three different versions of FF++, and the results are shown in Table 1.
[0088] Table 1. Test results of different methods on the FF++ dataset (%)
[0089]
[0090] Note: "-" indicates that there is no corresponding experimental data.
[0091] Table 1 shows that the present invention achieves the best results on the original dataset. On both compressed datasets, the passive detection method of the present invention demonstrates strong resistance to compression. In particular, the passive detection method of the present invention detects key tampering traces by combining frequency features, and even when using the lighter ResNet18, the method still exhibits good forensic accuracy on the heavily compressed version.
[0092] 2. For the four forgery methods in FF++, this invention was tested on the original FF++ dataset and the accuracy (ACC) was displayed. The test results are shown in Table 2.
[0093] Table 2 shows the test results for different forged data types in the FF++ (RAW) dataset.
[0094]
[0095] In Table 2, the passive detection method of this invention outperforms MTD-Net by 0.33% and 0.83% in ACC on F2F and NT forgery types of data, respectively, and is basically on par with the methods compared in the table on the other two forgery types of datasets. Meanwhile, although the passive detection method of this invention does not surpass DeepFake-Adapter, its ACC is not significantly different from that of DeepFake-Adapter, and it exhibits good resistance to compression.
[0096] 3. Remove the frequency domain fusion module and retrain and test the network. The test results are shown in Table 3.
[0097] Table 3. Impact of fused frequency domain characteristics on network performance
[0098]
[0099] The results show that compared with networks that only use spatial features, networks that incorporate frequency domain features improve the accuracy on C23 and C40 by 19.8% and 16.3%, respectively, with an average improvement of 18%.
[0100] 4. To demonstrate the necessity and effectiveness of adding the attention mechanism, this invention trained a network without the attention mechanism for performance testing and comparison, and the results are shown in Table 4.
[0101] Table 4. The impact of attention mechanism on network performance
[0102]
[0103] The results show that the network with the added attention mechanism improved the accuracy of C23 and C40 by 4.07% and 3.47%, respectively, with an average improvement of 3.77%.
Claims
1. A deepfake detection method based on cross-domain fusion and separable watermarking, characterized in that... Follow these steps in sequence: Step 1. Determining Image Encoding An adversarial coding discriminator is used to determine whether the image contains a SepMark watermark. If yes, proceed to step 2; otherwise, proceed to step 3. Step 2. Input the image into the two separable decoders of the SepMark module. The two decoders decode the robust watermark and the semi-robust watermark respectively. Compare whether the two watermarks have significant differences to detect whether the image has been tampered with. End. Step 3. Two-domain transformation Step 3.1 Read the image tensor T i The system uses 7×7 convolutional layers, followed by batch normalization and activation function processing, and finally max pooling to extract features, resulting in 64-channel low-dimensional spatial features, which are used as the original spatial tensor T. S ; Step 3.2 Read the image tensor T i Multiply by 255 to revert to an RGB channel image; convert the RGB image to a YCrCb image, then convert it to a tensor type; perform a DCT transform on the YCrCb image tensor to obtain the frequency domain tensor T. f ; Step 4: Convert the spatial tensor T S and frequency domain tensor T f The input is fed into the BasicBlock module for further spatial and frequency domain feature extraction and local feature fusion. Step 4.1 Spatial Tensor T s The spatial features f are obtained through a convolutional layer Conv1_1 and a batch normalization layer Bn1-1. s Then, after passing through the channel attention module and the spatial attention module, the spatial features f with the added attention mechanism are obtained. s ′;Frequency domain tensor T f The frequency domain features f are obtained through a convolutional layer Conv2_1 and a batch normalization layer Bn2-1. f ; Step 4.2 will add the attention mechanism to the spatial features f s ′ and frequency domain features f f The input is fed into the local feature fusion module for feature fusion. Step 4.2.1 Let W s and W f These are the spatial features f after the attention mechanism has been added. s ′ and frequency domain features f f The absolute value of the batch normalized weights, θ w The threshold used to classify importance; W s ≥θ w The corresponding feature channels are called important spatial feature channels, and W s <θ w The corresponding feature channels are called spatial secondary feature channels; W f ≥θ w The corresponding characteristic channels are called important characteristic channels in the frequency domain, and W f <θ w The corresponding feature channel is called the frequency domain secondary feature channel; Step 4.2.2 will W s W f Sort each in descending order to obtain the sorted W. s index n s And sorted W f index n f ; Step 4.2.3 Initialize shape and f s The same zero tensor Z s and shape with f f The same zero tensor Z f ; Step 4.2.4 Preserve the spatial features f after adding the attention mechanism. s The important feature channels in ' are sorted according to the index n. f Using frequency domain features f f Important feature channels in the spatial feature f after adding an attention mechanism s The secondary feature channels in ' are used to obtain the spatial feature Z after cross-domain fusion. s ; Step 4.2.5 Retain the important feature channels in the frequency domain features, based on the sorted index n s Using spatial features f with added attention mechanism s The important feature channel in ' replaces the frequency domain feature f f The secondary feature channels in the data are used to obtain the spatial feature Z after cross-domain fusion. f ; Step 4.2.6 merging the spatial features Z from the cross-domain fusion s The ReLU activation function is applied, followed by a convolutional layer Conv1_2 and a batch normalization layer Bn1-2 to obtain the spatial features Z after cross-domain fusion of the activated features. s ′; Step 4.2.7 merging the frequency domain features Z from the cross-domain fusion f The ReLU activation function is applied, followed by a convolutional layer Conv2_2 and a batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after cross-domain fusion of the activated features. f ; Step 4.2.8 fuses the activated cross-domain spatial features Z s ′ and spatial tensor T S The addition completes the residual connection. The result after residual connection is activated by the ReLU activation function to obtain the locally fused spatial features F. s ; Step 4.2.9 merging the activated cross-domain frequency domain features Z′ f With frequency domain tensor T f The residual connection is completed by progressive addition. The result after residual connection is activated by the ReLU activation function to obtain the locally fused frequency domain features F. f ; Step 5. Using the obtained locally fused spatial features F respectively s Frequency domain features F of local fusion f As spatial and frequency domain tensors, repeat step 4 until the processed spatial feature F is obtained. s ′ and frequency domain features F f Proceed to step 6; Step 6. Global Feature Fusion Step 6.1 Process the spatial features F s ′ and frequency domain features F f Perform average pooling; Step 6.2: Feed the result after average pooling into the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial features. s Judgment result d of local fusion frequency domain features f ; Step 7. Based on the judgment result vector, perform authenticity verification on the image. Step 7.1 will d s ,d f The final evidence result y of the model is obtained by linearly combining the following formulas. y=αd s +βd f Where α and β are the parameters used in training the follow-up detection model; Step 7.2 compares the final evidence result y with the labels of the training dataset to obtain the true / false image identification result, and then ends.