Deep forgery detection method based on cross domain feature fusion and separable watermark
Through the deep forgery detection method of cross-domain feature fusion and separable watermarks, combined with the BasicBlock module and attention mechanism, the problem of insufficient generalization ability and robustness of deep forgery detection in the existing technology is solved, and the watermarks are not embedded is realized Or erasing the detection of watermark forged images improves detection accuracy and robustness.
Patent Information
- Application Number
- CN202411027846.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The existing deep forgery detection methods have insufficient generalization ability, insufficient robustness, poor interpretability, insufficient real-timeness and poor cross-domain adaptability. It is particularly difficult to immediately prevent the propagation of forgery images, and it is impossible to detect forgery without embedding or erasing watermarks. image.
Depth forgery detection method based on cross-domain feature fusion and separable watermarks are used to determine whether the image contains SepMark watermarks through an adversarial encoding discriminator, and the airspace and frequency domain feature extraction and fusion are extracted and fusion with the BasicBlock module, and the attention mechanism is used to automatically learn to distinguish channels. Or spatial location information, realize cross-domain fusion, and authenticity identification is carried out through global feature fusion.
Improve the accuracy and robustness of deep forgery detection, enhance the resistance to various attacks and interferences, realize the passive detection function of fake images forged without embedding watermarks or erasing watermarks, and provide full range of active forensics and passive forensics. Defense network model.
Smart Images

Figure CN119068321A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep forgery detection method, in particular to a deep forgery detection method based on cross-domain feature fusion and separable watermark. Background Art
[0002] Currently, deep fake detection methods are divided into passive forensics and active forensics.
[0003] Passive forensics is based on the traces left by the forgery process or the difference in statistical distribution between the forged image and the real image to determine the authenticity. The main methods are as follows:
[0004] Methods based on visual features: They rely on analyzing visual anomalies in images or videos, such as unnatural facial expressions, abnormal blinking frequency, etc. For example, research on the FaceForensics++ dataset found that early deep fake videos can be effectively detected by analyzing blinking frequency, because the characters in these videos blink significantly less frequently than the normal range of 15-30 times per minute. However, with the advancement of forgery technology, these visual features have become increasingly difficult to identify.
[0005] CNN-based spatial domain methods: This type of method mainly uses convolutional neural networks (CNN) to extract image features in the spatial domain and then makes judgments through classifiers. The advantage of this method is that it can automatically learn effective features and its performance on large data sets is usually good. However, its disadvantage is that it requires a large amount of labeled data for training and may not work well for new or unseen forgery techniques.
[0006] Methods based on frequency domain analysis: Detect forgery traces by analyzing the frequency domain features of images or videos. The FouSeg method is a good example, which uses discrete Fourier transform (DFT) to analyze the frequency domain features of images and detects specific patterns in the high frequency domain of deep forgery images by segmenting the spectrum of the image. This type of method is effective for certain specific forgery techniques, but it is difficult to deal with new forgery methods.
[0007] Biometric-based methods: Use human physiological characteristics, such as heartbeat and blood flow, to detect forgeries. The DeepVit method extracts heart rate information by analyzing the slight color changes of the facial skin in the video to distinguish between real and forged videos. However, it is difficult to implement and is easily affected by environmental factors.
[0008] In summary, although existing passive forensics detection methods have achieved certain results in specific scenarios, they still have problems such as insufficient generalization ability, insufficient robustness, poor interpretability, insufficient real-time performance, and poor cross-domain adaptability. In particular, passive forensics has the problem of difficulty in immediately preventing the spread of forged images.
[0009] The existing SepMark model is one of the active forensics methods. The SepMark model includes an encoder and two separable decoders, namely a robust decoder and a semi-robust decoder. The network structure of the two separable decoders is the same but the weights are different, so the decoders will extract the watermark twice for the noisy image. During the training process, the goal of the robust decoder is to extract the pre-embedded watermark under any noise attack, while the goal of the semi-robust decoder is to extract the watermark under distortion attack and fail to extract the watermark under deep fake attack. The encoder embeds a watermark (such as the publisher's logo and timestamp) into the protected facial image, and the two separable decoders extract the robust watermark and the semi-robust watermark respectively. The SepMark model does not detect which images are forged, but is committed to ensuring which images are authentic and identifying their sources. In addition, even if the content of the original watermark is unknown, it is generally possible to compare the two extracted watermarks to determine whether the marked face image has been maliciously modified after publication, and immediately prevent the further spread of suspicious images that do not match the watermark. However, the SepMark model cannot detect forged images without embedded watermarks or erased watermarks.
[0010] So far, there are no reports on deep fake detection methods based on cross-domain feature fusion and separable watermarks. Summary of the invention
[0011] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a deep counterfeit detection method based on cross-domain feature fusion and separable watermark.
[0012] The technical solution of the present invention is: a deep forgery detection method based on cross-domain fusion and separable watermark, which is carried out in the following steps:
[0013] Step 1. Image coding determination
[0014] Use the adversarial coding discriminator to determine whether the image contains the SepMark watermark. If yes, proceed to step 2; if no, proceed to step 3.
[0015] Step 2. Input the image into the two separable decoders of the SepMark module. The two decoders decode the robust watermark and the semi-robust watermark respectively. Compare whether the two watermarks have a large difference to detect whether the image has been tampered with. End;
[0016] Step 3. Dual domain transformation
[0017] Step 3.1 Read the image tensor T iThrough the 7×7 convolution layer, batch normalization and activation function processing, and finally the maximum pooling operation, feature extraction is performed to obtain the 64-channel low-dimensional spatial domain features as the original spatial domain tensor T S ;
[0018] Step 3.2 Read the image tensor T i Multiply by 255 to convert back to RGB channel image; convert the RGB image to YCrCb image, and then convert it into tensor type; perform DCT transformation on the YCrCb image tensor as frequency domain tensor T f ;
[0019] Step 4: Transform the spatial tensor T S and the frequency domain tensor T f Input to the BasicBlock module for further spatial and frequency domain feature extraction and local feature fusion
[0020] Step 4.1 Spatial tensor T f The spatial domain feature f is obtained through the convolution layer Conv1_1 and the batch normalization layer Bn1-1 s , and then through the channel attention module and the spatial attention module to obtain the spatial feature f after adding the attention mechanism s ′; frequency domain tensor T f The frequency domain feature f is obtained through the convolution layer Conv2_1 and the batch normalization layer Bn2-1 f ;
[0021] Step 4.2 adds the spatial feature f after the attention mechanism s ′ and frequency domain features f f Input to the local feature fusion module for feature fusion;
[0022] Step 4.2.1 Set W s and W f They are the spatial features f after adding the attention mechanism s ′ and frequency domain features f f The absolute value of the batch normalization weight, θ w is the threshold used to classify importance;
[0023] W s ≥θ w The corresponding feature channel is called the important feature channel in the spatial domain. s <θ w The corresponding feature channel is called the spatial secondary feature channel;
[0024] W f ≥θ w The corresponding feature channel is called the important feature channel in the frequency domain. f <θw The corresponding feature channel is called the secondary feature channel in the frequency domain;
[0025] Step 4.2.2: W s , W f Sort them in descending order respectively, and get the sorted W s The index n s And after sorting W f The index n f ;
[0026] Step 4.2.3 Initialize shape and f s ′The same zero tensor Z s and shape with f f The same zero tensor Z f ;
[0027] Step 4.2.4 retains the spatial features f after adding the attention mechanism s ′, according to the sorted index n f , using the frequency domain feature f f The important feature channels in the image replace the spatial features f after adding the attention mechanism s ′, and obtain the spatial feature Z after cross-domain fusion s ;
[0028] Step 4.2.5 retains the important feature channels in the frequency domain features, according to the sorted index n s , using the spatial feature f after adding the attention mechanism s The important feature channels in ′ replace the frequency domain features f f The secondary feature channel in the cross-domain fusion is used to obtain the spatial feature Z f ;
[0029] Step 4.2.6 The spatial feature Z after cross-domain fusion s Bring in the Relu activation function for activation, and then pass through the convolution layer Conv1_2 and the batch normalization layer Bn1-2 to obtain the activated cross-domain fused spatial domain feature Z s ′;
[0030] Step 4.2.7 The frequency domain feature Z after cross-domain fusion f Bring in the Relu activation function for activation, and then pass through the convolution layer Conv2_2 and the batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after the activated cross-domain fusion. f ;
[0031] Step 4.2.8: The spatial feature Z after the activated cross-domain fusion s ′ and the spatial tensor T SThe residual connection is completed by adding them together. The result after the residual connection is activated by the Relu activation function to obtain the local fusion spatial domain feature F s ;
[0032] Step 4.2.9: The frequency domain feature Z′ after the activated cross-domain fusion f With the frequency domain tensor T f The residual connection is completed by adding, and the result after residual connection is activated by Relu activation function to obtain the local fusion frequency domain feature F f ;
[0033] Step 5. Use the obtained local fusion spatial domain features F s And the local fusion frequency domain feature F f As the spatial domain tensor and frequency domain tensor, repeat step 4 until the processed spatial domain feature F is obtained s ′ and frequency domain feature F f ', proceed to step 6;
[0034] Step 6. Global feature fusion
[0035] Step 6.1: Process the spatial features F s ′ and frequency domain feature F f 'Perform average pooling;
[0036] Step 6.2: Send the average pooling result to the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial domain feature. s The judgment result of local fusion frequency domain features d f ;
[0037] Step 7. Authenticate the image based on the judgment result vector
[0038] Step 7.1: s ,d f Perform linear combination according to the following formula to obtain the final forensic result y of the model;
[0039] y=αd s +βd f
[0040] Among them, α and β are the parameters for following the detection model training;
[0041] Step 7.2 compares the final forensic result y with the label of the training data set to obtain the authenticity image identification result and ends.
[0042] Compared with the SepMark model, the present invention supplements the passive detection function for forged images without embedded watermarks or erased watermarks, and constructs a full defense network model with active and passive forensics for deep forged images. In particular, the BasicBlock module that improves the ResNet18 network is adopted, and the channel and spatial attention modules are added in the spatial position, which can automatically learn to distinguish the primary and secondary information of the channel or spatial position information, so as to selectively strengthen the important information of the spatial and frequency domain features, suppress unimportant information, and realize the cross-fusion of spatial and frequency domain features; using the fused features, subtle forged features can be accurately identified, so as to judge the authenticity of the image. Due to the introduction of frequency domain features that reflect the global information and texture information of the image, the accuracy of deep forgery detection can be improved, and it has stronger robustness and higher generalization ability in the face of various attacks and interferences. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 is a flow chart of an embodiment of the present invention.
[0044] Figure 2 It is a schematic diagram of the structure of the BasicBlock module of an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The present invention provides a deep forgery detection method based on cross-domain fusion and separable watermark, such as Figure 1 Follow the steps below:
[0046] Step 1. Image coding determination
[0047] Use the adversarial coding discriminator to determine whether the image contains the SepMark watermark. If yes, proceed to step 2; if no, proceed to step 3.
[0048] The SepMark watermark is a watermark embedded in the image by the image publisher using an existing SepMark model encoder;
[0049] Step 2. Input the image into the two separable decoders of the SepMark module. According to the existing method, the two SepMark decoders decode the robust watermark and the semi-robust watermark respectively, and compare whether the two watermarks have a large difference to detect whether the image has been tampered with. End;
[0050] Step 3. Dual domain transformation
[0051] Step 3.1 Read the image tensor T i Through a 7×7 convolution layer, batch normalization and activation function processing, and finally a maximum pooling operation, feature extraction (FE) is performed to obtain a 64-channel low-dimensional spatial domain feature as the original spatial domain tensor T S ;
[0052] Step 3.2 Read the image tensor T i Multiply by 255 to convert back to RGB channel image; convert the RGB image to YCrCb image, and then convert it into tensor type; perform DCT transformation on the YCrCb image tensor as frequency domain tensor T f ;
[0053] Step 4: Transform the spatial tensor T S and the frequency domain tensor T f Input Figure 2 The BasicBlock module shown in the figure performs further spatial and frequency domain feature extraction and local feature fusion
[0054] Step 4.1 Spatial tensor T f The spatial domain feature f is obtained through the convolution layer Conv1_1 and the batch normalization layer Bn1-1 s , and then through the channel attention module and the spatial attention module to obtain the spatial feature f after adding the attention mechanism s ′; frequency domain tensor T f The frequency domain feature f is obtained through the convolution layer Conv2_1 and the batch normalization layer Bn2-1 f ;
[0055] f s =BN(Conv(T s ));
[0056] f f =BN(Conv(T f ));
[0057] f s ′=Attention(f s );
[0058] Step 4.2 adds the spatial feature f after the attention mechanism s ′ and frequency domain features f f Input to the local feature fusion module for feature fusion;
[0059] Step 4.2.1 Set W s and W f They are the spatial features f after adding the attention mechanism s ′ and frequency domain features f f The absolute value of the batch normalization weight, θ w is the threshold used to classify importance;
[0060] The feature channels and the weights of the normalization layer are one-to-one corresponding. Assuming that the shape of the feature tensor is [1, N, 512, 512], the weight parameter corresponding to the normalization layer is a 1×N one-dimensional array, that is, one feature channel corresponds to one weight. Therefore, the importance of the feature channels can be divided according to the normalization layer weight and the set threshold.
[0061] W s ≥θ w The corresponding feature channel is called the important feature channel in the spatial domain. s <θ w The corresponding feature channel is called the spatial secondary feature channel;
[0062] W f ≥θ w The corresponding feature channel is called the important feature channel in the frequency domain. f <θ w The corresponding feature channel is called the secondary feature channel in the frequency domain;
[0063] Step 4.2.2: W s , W f Sort them in descending order respectively, and get the sorted W s The index n s And after sorting W f The index n f ;
[0064] Step 4.2.3 Initialize shape and f s ′The same zero tensor Z s and shape with f f The same zero tensor Z f ;
[0065] Step 4.2.4 retains the spatial features f after adding the attention mechanism s ′, according to the sorted index n f , using the frequency domain feature f f The important feature channels in the image replace the spatial features f after adding the attention mechanism s ′, and obtain the spatial feature Z after cross-domain fusion s ;
[0066] Step 4.2.5 retains the important feature channels in the frequency domain features, according to the sorted index n s , using the spatial feature f after adding the attention mechanism s The important feature channels in ′ replace the frequency domain features f f The secondary feature channel in the cross-domain fusion is used to obtain the spatial feature Z f ;
[0067] Step 4.2.6 The spatial feature Z after cross-domain fusion s Bring in the Relu activation function for activation, and then pass through the convolution layer Conv1_2 and the batch normalization layer Bn1-2 to obtain the activated cross-domain fused spatial domain feature Z s ′;
[0068] Z s ′=BN(Conv(Relu(Z s )));
[0069] Step 4.2.7 The frequency domain feature Z after cross-domain fusion f Bring in the Relu activation function for activation, and then pass through the convolution layer Conv2_2 and the batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after the activated cross-domain fusion. f ;
[0070] Zf′=BN(Conv(Relu(Zf)));
[0071] Step 4.2.8: The spatial feature Z after the activated cross-domain fusion s ′ and the spatial tensor T S The residual connection is completed by adding them together. The result after the residual connection is activated by the Relu activation function to obtain the local fusion spatial domain feature F s ;
[0072] F s =Relu(Z s ′+T s );
[0073] Step 4.2.9: The frequency domain feature Z′ after the activated cross-domain fusion f With the frequency domain tensor T f The residual connection is completed by adding, and the result after residual connection is activated by Relu activation function to obtain the local fusion frequency domain feature F f ;
[0074] F f =Relu(Z f ′+T f );
[0075] Step 5. Use the obtained local fusion spatial domain features F s And the local fusion frequency domain feature F f As the spatial domain tensor and frequency domain tensor, repeat step 4 until the processed spatial domain feature F is obtained s ′ and frequency domain feature F f ', proceed to step 6;
[0076] Step 6. Global feature fusion
[0077] Step 6.1: Process the spatial features F s ′ and frequency domain feature F f 'Perform average pooling;
[0078] Step 6.2: Send the average pooling result to the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial domain feature. s The judgment result of local fusion frequency domain features d f ;
[0079] d s =FC(AvePool(F S ′));
[0080] d f =FC(AvePool(F f ′));
[0081] Step 7. Authenticate the image based on the judgment result vector
[0082] Step 7.1: s ,d f Perform linear combination according to the following formula to obtain the final forensic result y of the model;
[0083] y=αd s +βd f
[0084] Among them, α and β are the parameters for following the detection model training;
[0085] Step 7.2 compares the final forensic result y with the label of the training data set to obtain the authenticity image identification result and ends.
[0086] experiment:
[0087] 1. The embodiments of the present invention were tested on three different versions of FF++. The results are shown in Table 1.
[0088] Table 1 Test results of different methods on FF++ dataset%
[0089]
[0090] Note: “-” indicates no corresponding experimental data.
[0091] The results in Table 1 show that the present invention achieves the best results on the original version of the dataset. On the two compressed versions of the dataset, the passive detection method of the present invention demonstrates strong compression resistance. In particular, the passive detection method of the present invention detects key tampering traces by combining frequency features, and even when using the lighter ResNet18, the method still shows good forensic accuracy on the heavily compressed version.
[0092] 2. For the four counterfeiting methods in FF++, the present invention is tested on the FF++ original dataset and demonstrates the accuracy (ACC). The test results are shown in Table 2.
[0093] Table 2 Test results of different forged data types in the FF++ (RAW) dataset
[0094]
[0095] In Table 2, the passive detection method of the present invention surpasses MTD-Net by 0.33% and 0.83% in ACC index on the two types of forged data, F2F and NT, respectively, and is basically on par with the comparison methods in the table on the other two types of forged data sets. At the same time, although the passive detection method of the present invention did not surpass DeepFake-Adapter, its ACC index is not much different from that of the method, and it shows good anti-compression performance.
[0096] 3. Remove the frequency domain fusion module and retrain and test the network. The test results are shown in Table 3.
[0097] Table 3 Impact of fused frequency domain features on network performance
[0098]
[0099] The results show that compared with the network that only uses spatial domain features, the network that integrates frequency domain features has improved the accuracy of C23 and C40 by 19.8% and 16.3% respectively, with an average improvement of 18%.
[0100] 4. In order to prove the necessity and effectiveness of adding the attention mechanism, the present invention trained the network without the attention mechanism for performance testing and comparison. The results are shown in Table 4.
[0101] Table 4 Impact of attention mechanism on network performance
[0102]
[0103] The results show that the network with the added attention mechanism has improved its accuracy by 4.07% and 3.47% on C23 and C40 respectively, with an average improvement of 3.77%.
Claims
1. A deep forgery detection method based on cross-domain fusion and separable watermark, characterized by Follow the steps below: Step 1. Image coding determination Use the adversarial coding discriminator to determine whether the image contains the SepMark watermark. If yes, proceed to step 2; if no, proceed to step 3. Step 2. Input the image into the two separable decoders of the SepMark module. The two decoders decode the robust watermark and the semi-robust watermark respectively. Compare whether the two watermarks have a large difference to detect whether the image has been tampered with. End; Step 3. Dual domain transformation Step 3.1 Read the image tensor T i Through the 7×7 convolution layer, batch normalization and activation function processing, and finally the maximum pooling operation, feature extraction is performed to obtain the 64-channel low-dimensional spatial domain features as the original spatial domain tensor T S ; Step 3.2 Read the image tensor T i Multiply by 255 to convert back to RGB channel image; convert the RGB image to YCrCb image, and then convert it into tensor type; perform DCT transformation on the YCrCb image tensor as frequency domain tensor T f ; Step 4: Transform the spatial tensor T S and the frequency domain tensor T f Input to the BasicBlock module for further spatial and frequency domain feature extraction and local feature fusion Step 4.1 Spatial tensor T f The spatial domain feature f is obtained through the convolution layer Conv1_1 and the batch normalization layer Bn1-1 s , and then through the channel attention module and the spatial attention module to obtain the spatial feature f after adding the attention mechanism s ′; frequency domain tensor T f The frequency domain feature f is obtained through the convolution layer Conv2_1 and the batch normalization layer Bn2-1 f ; Step 4.2 adds the spatial feature f after the attention mechanism s ′ and frequency domain features f f Input to the local feature fusion module for feature fusion; Step 4.2.1 Set W s and W f They are the spatial features f after adding the attention mechanism s ′ and frequency domain features f f The absolute value of the batch normalization weight, θ w is the threshold used to classify importance; W s ≥θ w The corresponding feature channel is called the important feature channel in the spatial domain. s <θ w The corresponding feature channel is called the spatial secondary feature channel; W f ≥θ w The corresponding feature channel is called the important feature channel in the frequency domain. f <θ w The corresponding feature channel is called the secondary feature channel in the frequency domain; Step 4.2.2: W s , W f Sort them in descending order respectively, and get the sorted W s The index n s And after sorting W f The index n f ; Step 4.2.3 Initialize shape and f s ′The same zero tensor Z s and shape with f f The same zero tensor Z f ; Step 4.2.4 retains the spatial features f after adding the attention mechanism s ′, according to the sorted index n f , using the frequency domain feature f f The important feature channels in the image replace the spatial features f after adding the attention mechanism s ′, and obtain the spatial feature Z after cross-domain fusion s ; Step 4.2.5 retains the important feature channels in the frequency domain features, according to the sorted index n s , using the spatial feature f after adding the attention mechanism s The important feature channels in ′ replace the frequency domain features f f The secondary feature channel in the cross-domain fusion is used to obtain the spatial feature Z f ; Step 4.2.6 The spatial feature Z after cross-domain fusion s Bring in the Relu activation function for activation, and then pass through the convolution layer Conv1_2 and the batch normalization layer Bn1-2 to obtain the activated cross-domain fused spatial domain feature Z s ′; Step 4.2.7 The frequency domain feature Z after cross-domain fusion f Bring in the Relu activation function for activation, and then pass through the convolution layer Conv2_2 and the batch normalization layer Bn2-2 to obtain the frequency domain feature Z′ after the activated cross-domain fusion. f ; Step 4.2.8: The spatial feature Z after the activated cross-domain fusion s ′ and the spatial tensor T S The residual connection is completed by adding them together. The result after the residual connection is activated by the Relu activation function to obtain the local fusion spatial domain feature F s ; Step 4.2.9: The frequency domain feature Z′ after the activated cross-domain fusion f With the frequency domain tensor T f The residual connection is completed by adding, and the result after residual connection is activated by Relu activation function to obtain the local fusion frequency domain feature F f ; Step 5. Use the obtained local fusion spatial domain features F s And the local fusion frequency domain feature F f As the spatial domain tensor and frequency domain tensor, repeat step 4 until the processed spatial domain feature F is obtained s ′ and frequency domain feature F f ', proceed to step 6; Step 6. Global feature fusion Step 6.1: Process the spatial features F s ′ and frequency domain feature F f 'Perform average pooling; Step 6.2: Send the average pooling result to the fully connected layer to obtain the preliminary judgment result vector, which is the judgment result d of the local fusion spatial domain feature. s The judgment result of local fusion frequency domain features d f ; Step 7. Authenticate the image based on the judgment result vector Step 7.1: s ,d f Perform linear combination according to the following formula to obtain the final forensic result y of the model; y=αd s +βd f Among them, α and β are the parameters for following the detection model training; Step 7.2 compares the final forensic result y with the label of the training data set to obtain the authenticity image identification result and ends.
Citation Information
Patent Citations
False face detection method based on frequency attention feature fusion, medium and equipment
CN116434351A
Image depth region tampering active evidence obtaining method based on model key
CN117876204A
Face deep counterfeiting evidence obtaining method, system and device and storage medium
CN118279995A
Robust hologram watermarking system based on artificial intelligence
US20240214648A1
Cited By
Dual face counterfeiting active defense method based on frequency limitation
CN119274241A
Face deep counterfeiting evidence obtaining method, system and device and storage medium
CN119723689A
A method and device for extracting watermark of three-dimensional model
CN119784568A
