Deep forgery detection and implicit identity tracing method based on explicit identity

Abstract

The invention discloses an explicit identity-based deep forgery detection and implicit identity traceability method, and the deep forgery detection method comprises the steps: introducing a double-flow ViT network based on an explicit identity (face), and effectively integrating spatial domain and frequency domain features through a shallow-layer and deep-layer feature fusion mechanism of a double-flow ViT architecture, thereby achieving the deep forgery detection and implicit identity traceability. The detection robustness is improved by utilizing a simplified linear attention mechanism and cross attention, so that the model has stronger generalization ability when facing novel and complex forged samples; meanwhile, the implicit identity information is extracted by cutting the background area in the image, implicit identity tracing is carried out by utilizing background features, and the limitation of an existing method on forged image tracing is broken through. According to the invention, the source information of the forged image can be traced on the basis of detecting the authenticity of the image, and a new direction is provided for the treatment of the forged content.

Description

Technical Field

[0001] The present invention relates to a deep fake detection and tracing method, and in particular to an explicit identity-based deep fake detection and implicit identity tracing method. Background Art

[0002] DeepFake technology usually uses deep learning models such as generative adversarial networks (GANs), autoregressive models, and diffusion models to create highly realistic fake images and videos by changing the content of images or videos. It has attracted much attention in recent years for its amazing visual effects. Although deepfake technology has a wide range of applications in legal fields such as virtual entertainment and movie special effects, it has also been used by some lawless people for information tampering and privacy infringement, which seriously threatens social credibility, judicial justice, and even national security. Therefore, developing deepfake detection and tracing methods in the new generation of social media and establishing a social trust mechanism are in line with the national security protection development strategy and promote the healthy development of national network security.

[0003] The most common deep fake image technology is AI face-changing technology, in addition to attribute modification (collage attack) and artificial intelligence generated content (Artificial Intelligence Generated Content, AIGC). AI face-changing forged images involve two important identity concepts: explicit identity and implicit identity. Specifically, explicit identity refers to the identity of the fake face, that is, the identity of the target face presented by AI face-changing technology. In contrast, implicit identity refers to the original identity information that may be hidden in the fake face. Although the fake face visually appears to be the target face, it may still contain features from the source real face, especially the background information is often not changed.

[0004] In the field of deep fake detection, many methods have tried to solve the problem of detecting fake images from different angles. Early methods often rely on finding some visual abnormal features in fake images, such as whether the facial expression is natural, whether the eye blinking is consistent with the normal frequency of humans, etc. Although these methods can effectively distinguish fake images from real images in the initial stage, with the advancement of generation technology, fake images are becoming more and more realistic, and it is difficult to achieve accurate detection based on these obvious visual features alone. Frequency domain feature analysis is another method used for fake detection. This method does not directly observe the visual features of the image, but identifies anomalies by analyzing the frequency information of the image. For example, the discrete Fourier transform (DFT) or discrete cosine transform (DCT) is used to extract the high-frequency components in the image, and the authenticity of the image is judged by detecting the fake features therein. Frequency domain analysis shows good results in certain specific types of fake scenes, but overall, it still lacks generalization and robustness when facing a variety of new types of fakes.

[0005] With the development of deep learning, convolutional neural networks (CNNs) have become an important tool in deepfake detection. The power of CNN lies in its ability to automatically learn features from data, especially in forgery detection, where it can extract tiny differences from a large number of real and forged images. However, a major challenge of CNN models is that they are not robust enough to new and never-before-seen types of forgeries, which means that when faced with different forgeries, the model may not be able to make effective judgments. In addition, training CNN models usually requires a large amount of labeled data, which increases the cost and difficulty of practical applications.

[0006] Most of the existing generated image traceability is based on watermarks, which are mainly divided into traceability without watermark embedding, traceability with pre-watermark embedding and traceability with post-watermark embedding. However, the above traceability is based on AIGC and ignores the traceability of AI face-changing. Summary of the invention

[0007] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provide a deep fake detection and implicit identity tracing method based on explicit identity.

[0008] The technical solution of the present invention is: a method for deep fake detection and implicit identity tracing based on explicit identity, which is carried out in the following steps:

[0009] Step 1. Image segmentation and preprocessing

[0010] The input query image is subjected to face detection through the YOLO algorithm, and the face part and the background part are cropped out. The cropped face part is copied, and the main body is retained in the RGB format as the spatial domain image. The copied face is subjected to DCT transformation as the frequency domain image. The dual domain image and the background image are normalized respectively, and the image size is standardized and converted into a tensor format to obtain the spatial domain tensor and the frequency domain tensor of the face image.

[0011] Step 2. Explicit identity feature extraction and feature fusion

[0012] Step 2.1 Shallow feature extraction

[0013] The spatial domain tensor and frequency domain tensor of the face image are input into the ViT encoder for preliminary feature extraction, and then a simplified linear attention mechanism is added to the extracted spatial domain features and frequency domain features to further focus on important features, thus obtaining spatial domain features and frequency domain features.

[0014] Step 2.2 Shallow feature fusion

[0015] Step 2.2.1: Slice the spatial domain features and the frequency domain features to separate the classification labels and feature labels of the spatial domain features and the frequency domain features;

[0016] Step 2.2.2 connects the classification mark of the spatial domain feature and the feature mark of the frequency domain feature; then connects the classification mark of the frequency domain feature and the feature mark of the spatial domain feature; applies cross attention to the connected features respectively, performs feature fusion, and obtains the fused spatial domain feature and frequency domain feature;

[0017] Step 2.3 Deep feature extraction

[0018] The fused spatial and frequency domain features are input into the ViT encoder to further extract features and obtain deep spatial and frequency domain features.

[0019] Step 2.4 Deep feature fusion

[0020] Step 2.4.1: Slice the deep spatial domain features and frequency domain features to separate the classification mark and feature mark of the spatial domain features; separate the classification mark and feature mark of the frequency domain features;

[0021] Step 2.4.2 connects the classification mark of the spatial domain feature and the feature mark of the frequency domain feature; then connects the classification mark of the frequency domain feature and the feature mark of the spatial domain feature; applies cross attention to the connected features again, performs feature fusion to obtain the spatial domain features and frequency domain features after deep fusion;

[0022] Step 3. Explicit identity forgery detection

[0023] The deep fusion features are output through the normalization layer and the classification head, and the mean of the two is calculated. The output shows the forgery detection result of the identity. If the identity is not forged, it ends directly, otherwise it goes to step 4;

[0024] Step 4. Implicit identity tracing

[0025] Step 4.1 Tracing module training

[0026] Step 4.1.1 collect background images with known source identities, use images with the same background as positive sample pairs, and images with different backgrounds as negative sample pairs;

[0027] Step 4.1.2 uses a separate ViT module to extract features from positive sample pairs and negative sample pairs;

[0028] Step 4.1.3 calculates the Euclidean distance between the features of the positive sample pairs and the Euclidean distance between the features of the negative sample pairs;

[0029] Step 4.1.4 calculates the contrast loss based on the preset distance threshold m, back-propagates the contrast loss and updates the model parameters until the ViT module completes training;

[0030] Step 4.2 Feature library construction

[0031] Step 4.2.1 Use the trained ViT module to extract features of the background image with known source identity again to obtain feature vectors;

[0032] Step 4.2.2 constructs a feature library using the extracted feature vectors, and stores the features and corresponding identity tags in the feature library;

[0033] Step 4.3 Tracing implicit identity

[0034] Step 4.3.1 uses the trained ViT model to receive the background tensor of the query image and perform feature extraction to obtain the query feature;

[0035] Step 4.3.2 calculates the similarity between the query feature and each feature in the feature library to obtain a similarity score;

[0036] Step 4.3.3 sorts the similarity scores, selects the matching result with the highest similarity, completes the source identity tracing through the mapping relationship between the constructed library index feature vector and the query image source identity, obtains the tracing result, and ends.

[0037] The deep fake detection method of the present invention introduces a two-stream ViT network based on explicit identity (face), effectively integrates spatial and frequency domain features through the shallow and deep feature fusion mechanism of the two-stream ViT architecture, and uses simplified linear attention mechanism and cross attention to improve the robustness of detection, so that the model has stronger generalization ability when facing new and complex fake samples; at the same time, the present invention extracts implicit identity information by cropping the background area in the image, and uses background features to trace the implicit identity, breaking through the limitations of existing methods in tracing the source of fake images. The present invention can trace the source information of fake images on the basis of detecting the authenticity of the image, providing a new direction for the governance of fake content. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of an embodiment of the present invention.

[0039] Figure 2 2 is a visualization experiment result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention provides a method for deep forgery detection and implicit identity tracing based on explicit identity, such as Figure 1 Follow the steps below:

[0041] Step 1. Image segmentation and preprocessing

[0042] The input query image I is subjected to face detection through the YOLO algorithm, the face part and the background part are cropped, the cropped face part is copied, and the main body retains the RGB format as the spatial domain image x face , copy the face and perform DCT transformation as the frequency domain image, and compare the dual domain image and the background image x background Normalize the image size and convert it into tensor format to get the face image spatial domain tensor x space and the frequency domain tensor x freq , as shown in the following formula;

[0043] x face =YOLO(I) (1)

[0044] x background =YOLO(I) (2)

[0045] x space =ToTensor(Norm(x face )) (3)

[0046] x freq =ToTensor(Norm(DCT(x face ))) (4)

[0047] Step 2. Explicit identity feature extraction and feature fusion

[0048] Step 2.1 Shallow feature extraction

[0049] The face image spatial tensor x space and the frequency domain tensor x freq Input ViT encoder Encoder for preliminary feature extraction, and then add simplified linear attention mechanism to the extracted spatial and frequency domain features to further focus on important features, and obtain spatial features x′ space and frequency domain feature x′ freq , as shown in formula (5,6);

[0050] x′ space =SLAB(Encoder(x space )) (5)

[0051] x′ freq =SLAB(Encoder(x freq )) (6)

[0052] Step 2.2 Shallow feature fusion

[0053] Step 2.2.1 For the spatial feature x′ space and frequency domain feature x′ freqPerform slicing operations to separate the classification labels Cls and feature labels Tokens of spatial domain features and frequency domain features, as shown in formulas (7, 8);

[0054] Cls space ,Tokens space = Slice(x′ space ) (7)

[0055] Cls freq ,Tokens freq = Slice(x′ freq ) (8)

[0056] Step 2.2.2 Connect the classification labels Cls of the spatial features space Tokens of frequency domain features freq , as shown in formula (9); then connect the classification mark Cls of the frequency domain feature freq Tokens of spatial features space , as shown in formula (10); cross attention is applied to the connected features respectively to perform feature fusion, and the fused spatial domain features are obtained and frequency domain characteristics As shown in formula (11,12);

[0057]

[0058] Step 2.3 Deep feature extraction

[0059] The fused spatial features and frequency domain characteristics Input ViT encoder Encoder to further extract features and obtain deep features x″ space and x″ freq As shown in formula (13,14);

[0060]

[0061] Step 2.4 Deep feature fusion

[0062] Step 2.4.1: deep spatial feature x″ space and frequency domain features x″ freq Perform slicing operation to separate the spatial features x″ space Classification mark Cls″ space and Tokens″ space , as shown in formula (15); separate frequency domain features x″ freq Classification mark Cls″ freq and Tokens″ freq, as shown in formula (16);

[0063] Cls″ space ,Tokens″ space =Slice(x″) space ) (15)

[0064] Cls″ freq ,Tokens″ freq =Slice(x″) freq ) (16)

[0065] Step 2.4.2 Connect the classification labels Cls″ of the spatial features space and frequency domain feature tokens freq , as shown in formula (17); then connect the classification mark Cls″ of the frequency domain feature freq Tokens of spatial features s ″ pace , as shown in formula (18); cross attention is applied to the connected features again to perform feature fusion to obtain the deep fused spatial features and frequency domain characteristics As shown in formula (19,20);

[0066]

[0067] Step 3. Explicit identity forgery detection

[0068] The deep fusion features The output is calculated through the normalization layer and the classification head, and the mean of the two is calculated. The output shows the forgery detection result of the identity, as shown in formula (21). If the identity is not forged, it ends directly, otherwise it goes to step 4;

[0069]

[0070] Step 4. Implicit identity tracing

[0071] Step 4.1 Tracing module training

[0072] Step 4.1.1 collect background images with known source identities, use images with the same background as positive sample pairs I1, and images with different backgrounds as negative sample pairs I2;

[0073] Step 4.1.2 uses a separate ViT module to extract features from the positive sample pair I1 and the negative sample pair I2, as shown in formula (22):

[0074] f1,f2=F ViT (I1,I2) (22)

[0075] Step 4.1.3 calculates the Euclidean distance between the features of the positive sample pairs and the Euclidean distance between the features of the negative sample pairs;

[0076] Step 4.1.4 calculates the contrast loss based on the preset distance threshold m (m=2.5 in this embodiment), back-propagates the contrast loss and updates the model parameters until the ViT module completes the training; the loss function is shown in formula (23), when the input is a positive sample y i =1, otherwise y i =0,D i is the Euclidean distance between features, m is the preset distance threshold, and N is the number of samples;

[0077]

[0078] Step 4.2 Feature library construction

[0079] Step 4.2.1 Use the trained ViT module to extract features of the background image with known source identity again to obtain feature vectors;

[0080] Step 4.2.2 constructs a feature library using the extracted feature vectors, as shown in formula (24), and stores the features and corresponding identity tags in F database ;

[0081]

[0082] Step 4.3 Tracing implicit identity

[0083] Step 4.3.1 Use the trained ViT model to receive the background tensor of the query image I and perform feature extraction, as shown in formula (25), to obtain the query feature f query ;

[0084] f query =F ViT (I query ) (25)

[0085] Step 4.3.2: Query feature f query With each feature f in the feature library j Perform similarity calculation, as shown in formula (26), to obtain a similarity score;

[0086]

[0087] Step 4.3.3 sorts the similarity scores, selects the matching result with the highest similarity, and completes the source identity tracing through the mapping relationship between the constructed library index feature vector and the query image source identity, and obtains the tracing result Result shown in formula (27), and ends;

[0088]

[0089] experiment:

[0090] 1. The embodiments of the present invention and different methods were tested in the library on three different versions of FF++, and the results are shown in Table 1. As can be seen from Table 1, the deep fake detection method of the present invention shows high accuracy on the three versions of the data set. In particular, the detection method of the present invention shows good forensic accuracy on the heavily compressed version C40 by combining frequency domain features to detect key tampering traces.

[0091] Table 1 Test results of different methods on FF++ dataset%

[0092]

[0093] Note: “-” indicates no corresponding experimental data.

[0094] 2. For the four forgery methods in FF++, Table 2 shows the accuracy (ACC) of the present invention and different methods on the FF++ original dataset. The data in Table 2 show that the detection performance of the two-stream ViT network model proposed in the present invention is particularly superior in this dataset, with an accuracy of DF: 99.88%, FS: 99.94%, F2F: 99.65% and NT: 97.38%. These results show that the two-stream ViT network model demonstrates significant effectiveness in forgery detection tasks, surpassing the performance of existing methods, highlighting its application potential in related research.

[0095] Table 2 Test results of different forged data types in the FF++(RAW) dataset

[0096]

[0097] 3. Remove the frequency domain fusion module and retrain and test the network. The test results are shown in Table 3.

[0098] Table 3 Impact of fused frequency domain features on network performance

[0099]

[0100] The results show that compared with the network that only uses spatial domain features, the network that integrates frequency domain features improves the accuracy of C23 and C40 by 12.73% and 22.55% respectively, with an average improvement of 17.43%.

[0101] 4. In order to prove the necessity and effectiveness of adding the attention mechanism, the present invention trained the network without the attention mechanism for performance testing and comparison. The results are shown in Table 4.

[0102] Table 4 Impact of attention mechanism on network performance

[0103]

[0104] The results show that the network with the added attention mechanism has improved its accuracy by 14.73% and 19.92% on C23 and C40 respectively, with an average improvement of 17.27%.

[0105] 5. In order to prove the effectiveness of the ViT model trained by the present invention using background images and the feasibility of implicit identity tracing, the present invention conducted visualization experiments on the background features extracted by the ViT model, similarity matching features, feature differences, and tracing results. The results are as follows: Figure 2 shown.

[0106] Figure 2 From left to right are the query image, query image features, best matching features, feature difference graph, and traceability structure graph. The calculated feature similarity is 0.9997, which shows the effectiveness of the contrast loss applied in the present invention for ViT model training and the rationality of implicit identity tracing through background areas.

Claims

1. A method for deep fake detection and implicit identity tracing based on explicit identity, characterized in that Follow the steps below: Step 1. Image segmentation and preprocessing The input query image is subjected to face detection through the YOLO algorithm, and the face part and the background part are cropped out. The cropped face part is copied, and the main body is retained in the RGB format as the spatial domain image. The copied face is subjected to DCT transformation as the frequency domain image. The dual domain image and the background image are normalized respectively, and the image size is standardized and converted into a tensor format to obtain the spatial domain tensor and the frequency domain tensor of the face image. Step 2. Explicit identity feature extraction and feature fusion Step 2.1 Shallow feature extraction The spatial domain tensor and frequency domain tensor of the face image are input into the ViT encoder for preliminary feature extraction, and then a simplified linear attention mechanism is added to the extracted spatial domain features and frequency domain features to further focus on important features, thus obtaining spatial domain features and frequency domain features. Step 2.2 Shallow feature fusion Step 2.2.1: Slice the spatial domain features and the frequency domain features to separate the classification labels and feature labels of the spatial domain features and the frequency domain features; Step 2.2.2 connects the classification mark of the spatial domain feature and the feature mark of the frequency domain feature; Then connect the classification mark of the frequency domain feature and the feature mark of the spatial domain feature; Apply cross attention to the connected features respectively, perform feature fusion, and obtain the fused spatial domain features and frequency domain features; Step 2.3 Deep feature extraction The fused spatial and frequency domain features are input into the ViT encoder to further extract features and obtain deep spatial and frequency domain features. Step 2.4 Deep feature fusion Step 2.4.1: Slice the deep spatial domain features and frequency domain features to separate the classification mark and feature mark of the spatial domain features; separate the classification mark and feature mark of the frequency domain features; Step 2.4.2 connects the classification mark of the spatial domain feature and the feature mark of the frequency domain feature; Then connect the classification mark of the frequency domain feature and the feature mark of the spatial domain feature; Apply cross attention to the connected features again, perform feature fusion to obtain deep fused spatial and frequency domain features; Step 3. Explicit identity forgery detection The deep fusion features are output through the normalization layer and the classification head, and the mean of the two is calculated. The output shows the identity forgery detection result. If the identity is not forged, it ends directly, otherwise it goes to step 4; Step 4. Implicit identity tracing Step 4.1 Tracing module training Step 4.1.1 collect background images with known source identities, use images with the same background as positive sample pairs, and images with different backgrounds as negative sample pairs; Step 4.1.2 uses a separate ViT module to extract features from positive sample pairs and negative sample pairs; Step 4.1.3 calculates the Euclidean distance between the features of the positive sample pairs and the Euclidean distance between the features of the negative sample pairs; Step 4.1.4 calculates the contrast loss based on the preset distance threshold m, back-propagates the contrast loss and updates the model parameters until the ViT module completes training; Step 4.2 Feature library construction Step 4.2.1 Use the trained ViT module to extract features of the background image with known source identity again to obtain feature vectors; Step 4.2.2 constructs a feature library using the extracted feature vectors, and stores the features and corresponding identity tags in the feature library; Step 4.3 Tracing implicit identity Step 4.3.1 uses the trained ViT model to receive the background tensor of the query image and perform feature extraction to obtain the query feature; Step 4.3.2 calculates the similarity between the query feature and each feature in the feature library to obtain a similarity score; Step 4.3.3 sorts the similarity scores, selects the matching result with the highest similarity, completes the source identity tracing through the mapping relationship between the constructed library index feature vector and the query image source identity, obtains the tracing result, and ends.

Images