Deep forgery detection method based on balanced contrast learning
By using the combination of balanced contrast learning and multi-scale attention interaction modules in deep forgery detection, the problem of overfitting and detection difficulty in the existing technology is solved, and accurate identification and efficient detection of deep forgery content is achieved.
Patent Information
- Application Number
- CN202410917979.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-10
AI Technical Summary
The existing deep forgery detection methods have overfitting problems, making it difficult to generalize to unseen data sets or different manipulation methods. With the advancement of generative models, the content of new deep forgery is difficult to distinguish, and the detection methods need to be continuously updated. to improve sensitivity and accuracy.
Using a deep forgery detection method based on balanced contrast learning, combined with unsupervised contrast learning and supervised contrast learning strategies, features are extracted through data enhancement and multi-scale attention interaction modules, and the classification effect is enhanced by jointly enhancing the supervised contrast loss and unsupervised contrast loss to construct End-to-end deep-fake detection model.
This method effectively distinguishes between fake faces and real faces, and shows excellent detection performance, especially in the data set that has strong generalization ability, which can accurately identify fake content, and improves the security and performance of detection.
Smart Images

Figure CN119068319A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence related technologies, and specifically to a deep fake detection method based on balanced contrastive learning. Background Art
[0002] The detection of deep fakes can be viewed as a binary classification task. Some past studies have focused on detecting features of fake images or videos, such as blinking features, heart rate features, texture features, frequency features, etc. Other studies have focused on exploring network models suitable for deep fake detection tasks, such as Xception, EfficientNet, VisionTransformer, etc. Although existing deep fake detection methods perform well, there are also some problems. Many deep fake detection methods have the problem of overfitting the training data set, and their prediction results tend to be more inclined to the probability of data distribution rather than forgery traces. Such models that overfit the data distribution have poor generalization ability for unseen data sets or different manipulation methods.
[0003] In addition, as deepfake technology continues to develop and evolve, detection methods also need to be constantly updated and improved to meet new challenges. For example, with the continuous advancement of generative models, new deepfake content may be more realistic and difficult to distinguish, which requires detection methods to have higher sensitivity and accuracy. In addition, deepfake technology may use adversarial training to circumvent detection methods, making traditional detection algorithms ineffective. Therefore, researchers need to continuously explore new technologies and algorithms to cope with the continuous evolution of deepfake technology.
[0004] In view of this, we conducted in-depth research on the above issues, which led to the present case. Summary of the invention
[0005] The purpose of the present invention is to provide a deep fake detection method based on balanced contrastive learning to solve the problem that the deep fake detection model proposed in the above background technology is overfitting and the detection method needs to be continuously updated and improved.
[0006] To achieve the above objectives, the present invention provides the following technical solutions: a deep fake detection method based on balanced contrast learning;
[0007] The deep fake detection model is a deep fake detection model of balanced contrastive learning strategy, which combines unsupervised contrastive learning strategy and supervised contrastive learning strategy, uses data enhancement method to expand data, and jointly enhances deep fake classification effect through supervised contrast loss and unsupervised contrast loss;
[0008] The model consists of a twin network encoder, two multi-scale attention interaction modules MAI, two representation modules Projection and a prediction head Prediction. Its steps are:
[0009] Step 1: For the input video, first split it into video frames, and randomly extract the video frames to crop the face area;
[0010] Step 2: For the input image, crop the face area;
[0011] Step 3: For the cropped face area, data enhancement is used to obtain an enhanced face area image;
[0012] Step 4: The face area of the original image and the face area of the enhanced image are respectively subjected to the twin network encoder to obtain the multi-scale features of the original image and the multi-scale features of the enhanced image;
[0013] Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module respectively to obtain the multi-scale attention fusion features of the face area of the original image and the face area of the enhanced image;
[0014] Step 6: The fused features are passed through a representation layer respectively and input into a prediction head through splicing to obtain the results of deep fake detection.
[0015] Preferably, in step 1, for the input video, first use the opencv tool to cut the video frame, and randomly select a frame of the video to crop the face area; obtain the face area through dlib, enlarge the face area by 1.2 times, and then crop the image to obtain the face area to be detected.
[0016] Preferably, in step 2, for the input image, dlib is used to obtain the face area, and the face area is enlarged by 1.2 times and then the image is cropped to obtain the face area to be detected.
[0017] Preferably, in step 3, data enhancement adopts random flipping (RF), random resizing and cropping (RRC) and random erasing (RE); the probability of the RF is set to 0.5; the RRC adopts a scaling factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); the RE adopts a scaling factor (0.02, 0.2), an aspect ratio (0.5, 2) and a probability (0.5).
[0018] Preferably, in step 4, for the face area x of the input original image i and the face region x of the enhanced image i(j) , respectively, through the encoder to obtain the multi-scale features of the original face area (Ei , M i , O i ) and (E j(i) , M j(i) , O j(i) ), whose formula is as follows:
[0019] E i ,M i ,O i =Encoder(x i );
[0020] E j(i) ,M j(i) ,O j(i) =Encoder(x j(i) );
[0021] Among them, the encoder Encoder is the Xception network, E represents the feature vector of the first stage EntryFlow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
[0022] Preferably, in step 5, the multi-scale attention interaction module MAI first uses the attention mechanism to extract the attention of multiple scale features, and obtains the direct relevant attention of different features by multiplying them two by two, and finally performs feature fusion by adding features.
[0023] Preferably, in the multi-scale attention interaction module, each input (E i , M i , O i ) are processed by a SEAttention module, and then the adaptive average pooling layer is used to reduce the size of the feature map to 7×7 pixels; then, a 1x1 Conv2D layer adjusts the dimension of its features to 2048, and then batch normalization is performed to obtain the attention features with a dimension of 7×7×2048; in this structure, the input E is obtained i , M i , O i The attention feature E′ i , M′ i , O′ i ; Then a pairwise method is used to calculate their cross attention; E′ i and M′ i Multiply pixel by pixel to obtain the cross-attention feature A1; similarly, calculate (E′ i , O′ i ) and (M′ i , O′i ) to obtain A2 and A3; through the above steps, A1, A2 and A3 have comprehensive information of different scales respectively; A1, A2 and A3 are connected together, and then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain the multi-scale attention feature F i , and as the output of the multi-scale attention interaction module, the formula can be expressed as:
[0024] F i =MAI(E i ,M i ,O i ).
[0025] For the input (E j(i) , M j(i) , O j(i) ), and also obtain the multi-scale attention feature F i(j) , the formula can be expressed as:
[0026] F j(i) =MAI(E j(i) ,M j(i) ,O j(i) ).
[0027] Preferably, in step 6, the F obtained in step 5 i and F j(i) Input a representation layer Projection respectively. The Projection of the representation layer is a simple multi-layer perceptron structure. Through the representation layer, the representation features V of the face area of the original image are obtained respectively. i And enhance the representation feature V of the face area of the image j(i) ; By characterizing the vector V i and V j(i) Splice and input a prediction head Prediction to obtain the true or false prediction result P of the input image i The whole process can be expressed by the following formula:
[0028] V i =Projection(F i );
[0029] V j(i) =Projection(F j(i) );
[0030] W = Concatenation (V i ,V j(i) );
[0031] P i=Prediction(W);.
[0032] Preferably, the model has three losses, namely, unsupervised contrastive learning loss L cl , supervised contrastive learning loss L scl and classification loss L ce ;
[0033] The unsupervised contrastive learning loss L cl It can be defined as follows: for a small batch containing N original images, i∈I≡{1,2,...2N} represents the index of all images (original images and enhanced images); if index i is an original image, then index j(i) is the data augmented image of index i, and vice versa; for unsupervised contrastive learning, the sample corresponding to index i is called the anchor point, and the sample corresponding to index j(i) is called the positive sample; other indexes A in the batch (i) The sample with index i is called a negative sample, and the number of negative samples is 2N-2; the representative feature of the sample with index i is recorded as V i ; Therefore, the contrastive learning loss L cl It can be expressed as:
[0034]
[0035] in represents the inner product, T is a temperature parameter;
[0036] The supervised contrastive learning loss L scl It can be defined as: all samples of the same category in the batch (including: original images and enhanced images) are defined as positive samples, denoted as B(i), and |B(i)| is its cardinality; similarly, samples from different categories are regarded as negative samples, denoted as C(i); therefore, the supervised contrastive learning loss L scl It can be expressed as:
[0037]
[0038] For all original images in a mini-batch, k∈K≡1,2,...N represents the index of the original image; for the input with index k, its label is Y k , the predicted probability is P k ; Cross entropy is used to quantify the difference between the predicted value and the label; cross entropy loss L ce It can be defined as:
[0039]
[0040] The total loss of the model is L total It can be defined as:
[0041] L total=100L cl +90L scl +10L ce ..
[0042] Compared with the prior art, the beneficial effects of the present invention are: the deep fake detection method based on balanced contrast learning has the following advantages:
[0043] Firstly, a deep fake detection model with balanced contrastive learning strategy is proposed. The proposed deep fake detection model is an end-to-end model that extracts facial features from images and videos and uses contrastive learning to effectively distinguish fake faces from real faces. In particular, the model still shows excellent detection performance when dealing with unseen deep fake images and videos.
[0044] Secondly, a feature fusion method based on multi-scale attention interaction is proposed; the proposed balanced contrastive learning strategy of supervised contrastive learning and unsupervised contrastive learning strategies has a simple structure and good performance; the strategy combines the advantages of supervised contrastive learning and unsupervised contrastive learning, so that the model can learn the subtle differences between real and forged content more comprehensively during the training process. This method achieves accurate identification of forged content by extracting features of images and videos at different scales and using the attention mechanism for feature fusion.
[0045] In general, the present invention performs well in terms of security, detection performance, and ease of use when facing deep fake content. With the continuous advancement of artificial intelligence technology, these technical solutions can play a vital role in the future field of computer vision. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 This is a flowchart of the deep fake detection model framework based on balanced contrast learning of the present invention;
[0047] Figure 2 This is a framework flow chart of the multi-scale attention interaction module of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] See also Figure 1-2 ,The present invention provides the following technical solutions: a deep fake detection method based on balanced contrast learning;
[0050] The deep fake detection model is a deep fake detection model with a balanced contrastive learning strategy. This model combines unsupervised contrastive learning strategy and supervised contrastive learning strategy, uses data augmentation method to expand data, and jointly enhances the deep fake classification effect through supervised contrast loss and unsupervised contrast loss.
[0051] The model consists of a twin network encoder, two multi-scale attention interaction modules MAI, two representation modules Projection and a prediction head Prediction, as shown in Figure 1 The steps are:
[0052] Step 1: For the input video, first split it into video frames, and randomly extract the video frames to crop the face area;
[0053] Step 2: For the input image, crop the face area;
[0054] Step 3: For the cropped face area, data enhancement is used to obtain an enhanced face area image;
[0055] Step 4: The face area of the original image and the face area of the enhanced image are respectively subjected to the twin network encoder to obtain the multi-scale features of the original image and the multi-scale features of the enhanced image;
[0056] Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module respectively to obtain the multi-scale attention fusion features of the face area of the original image and the face area of the enhanced image, such as Figure 2 As shown;
[0057] Step 6: The fused features are passed through a representation layer respectively and input into a prediction head through splicing to obtain the results of deep fake detection.
[0058] In step 1, for the input video, first use the opencv tool to cut the video frame, and randomly select a frame of the video to crop the face area; obtain the face area through dlib, enlarge the face area by 1.2 times, and then crop the image to obtain the face area to be detected.
[0059] In step 2, for the input image, dlib is used to obtain the face area, and the face area is enlarged by 1.2 times and then the image is cropped to obtain the face area to be detected.
[0060] In step 3, data augmentation uses random flipping (RF), random resizing and cropping (RRC), and random erasing (RE); the probability of RF is set to 0.5; RRC uses a scale factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); RE uses a scale factor (0.02, 0.2), an aspect ratio (0.5, 2), and a probability (0.5).
[0061] In step 4, for the face area x of the input original image i and the face region x of the enhanced image i(j) , respectively, through the encoder to obtain the multi-scale features of the original face area (E i , M i , O i ) and (E j(i) , M j(i) , O j(i) ), whose formula is as follows:
[0062] E i ,M i ,O i =Encoder(x i ); (1)
[0063] E j(i) ,M j(i) ,O j(i) =Encoder(x j(i) ); (2)
[0064] Among them, the encoder Encoder is the Xception network, E represents the feature vector of the first stage EntryFlow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
[0065] In step 5, the multi-scale attention interaction module MAI first uses the attention mechanism to extract the attention of multiple scale features, and obtains the direct relevant attention of different features by multiplying them two by two, and finally performs feature fusion by adding features.
[0066] In the multi-scale attention interaction module, each input (E i , M i , O i) are processed by an SE Attention module, and then the size of the feature map is reduced to 7×7 pixels after passing through an adaptive average pooling layer; then, a 1x1 Conv2D layer adjusts the dimension of its feature to 2048, and then batch normalization is performed to obtain an attention feature with a dimension of 7×7×2048; in this structure, the input E is obtained i , M i , O i The attention feature E′ i , M′ i , O′ i ; Then a pairwise method is used to calculate their cross attention; E′ i and M′ i Multiply pixel by pixel to obtain the cross-attention feature A1; similarly, calculate (E′ i , O′ i ) and (M′ i , O′ i ) to obtain A2 and A3; through the above steps, A1, A2 and A3 have comprehensive information of different scales respectively; A1, A2 and A3 are connected together, and then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain the multi-scale attention feature F i , and as the output of the multi-scale attention interaction module, the formula can be expressed as:
[0067] F i =MAI(E i ,M i ,O i ). (3)
[0068] For the input (E j(i) , M j(i) , O j(i) ), and also obtain the multi-scale attention feature F i(j) , the formula can be expressed as:
[0069] F j(i) =MAI(E j(i) ,M j(i) ,O j(i) ). (4)
[0070] In step 6, the F obtained in step 5 is i and F j(i) Input a representation layer Projection respectively. The Projection of the representation layer is a simple multi-layer perceptron structure. Through the representation layer, the representation features V of the face area of the original image are obtained respectively. i And enhance the representation feature V of the face area of the image j(i) ; By characterizing the vector Vi and V j(i) Splice and input a prediction head Prediction to obtain the true or false prediction result P of the input image i The whole process can be expressed by the following formula:
[0071] V i =Projection(F i ); (5)
[0072] V j(i) =Projection(F j(i) ); (6)
[0073] W = Concatenation (V i ,V j(i) ); (7)
[0074] P i =Prediction(W); (8)
[0075] There are three losses in the model, namely, unsupervised contrastive learning loss L cl , supervised contrastive learning loss L scl and classification loss L ce ;
[0076] The unsupervised contrastive learning loss L cl It can be defined as follows: for a small batch containing N original images, i∈I≡{1,2,...2N} represents the index of all images (original images and enhanced images); if index i is an original image, then index j(i) is the data augmented image of index i, and vice versa; for unsupervised contrastive learning, the sample corresponding to index i is called the anchor point, and the sample corresponding to index j(i) is called the positive sample; other indexes A in the batch (i) The sample with index i is called a negative sample, and the number of negative samples is 2N-2; the representative feature of the sample with index i is recorded as V i ; Therefore, the contrastive learning loss L cl It can be expressed as:
[0077]
[0078] in represents the inner product, T is a temperature parameter;
[0079] The supervised contrastive learning loss L sclIt can be defined as: all samples of the same category in the batch (including: original images and enhanced images) are defined as positive samples, denoted as B(i), and |B(i)| is its cardinality; similarly, samples from different categories are regarded as negative samples, denoted as C(i); therefore, the supervised contrastive learning loss L scl It can be expressed as:
[0080]
[0081] For all original images in a mini-batch, k∈K≡1,2,...N represents the index of the original image; for the input with index k, its label is Y k , the predicted probability is P k ; Cross entropy is used to quantify the difference between the predicted value and the label; cross entropy loss L ce It can be defined as:
[0082]
[0083] The total loss of the model is L total It can be defined as:
[0084] L total =100L cl +90L scl +10L ce ..
[0085] The specific implementation cases are as follows:
[0086] The development language is Python, the development environment system is Ubuntu, the deep learning framework is PyTorch, and the datasets are Faceforensics++ and Celeb-DF-V2. The Facef orensics++ dataset contains 5,000 videos, including 1,000 real videos and 4,000 fake videos. The 4,000 fake videos are NeuralTextures, DeepFakes, Face2Face, and FaceSwap, with 1,000 fake videos each. The Celeb-DF-V2 dataset contains 6,529 videos, including 890 real videos and 5,639 fake videos.
[0087] The specific steps are as follows:
[0088] Step 1: Data Input
[0089] The input of the data input is a video or a picture, and the output is a face area picture of the extracted frame. If the input is a video, first convert the video into frames. Use Opencv to cut the video into frames, and randomly select 30 frames of pictures for each video to crop the face area. Use dib to obtain the face key area frame of the frame or picture, and expand the face area by 1.2 times to crop the image to obtain the face area picture. Divide the face area pictures into training set, validation set and test set in a ratio of 8:1:1.
[0090] Step 2: Face Representation
[0091] The input of step 2 is a face image, and the output is an image representation vector of the face image. The size of the input image is adjusted to 299×299×3, and the augmented image is obtained through data augmentation. The original image and the enhanced image distribution are input into the Xception network to obtain the three-level features of each image, as shown in formulas (1) and (2). By inputting the three-level feature distribution of the encoder of the original image and the enhanced image into the multi-scale attention interaction module MAI, as shown in formulas (3) and (4), the multi-scale attention features of the image are obtained. The multi-scale attention features are represented by the same representation layer Projection to obtain the representation vectors of the original face and the enhanced face, as shown in formulas (5) and (6).
[0092] Step 3: Deep Fake Image Prediction
[0093] Step 3 inputs the image representation vector of the face image, and outputs the deepfake probability value of the original image. The face representation vectors of the original image and the enhanced image are concatenated, and a prediction head is used to obtain the probability value of the deepfake prediction of the group of images, as shown in formulas (7) and (8). The range of the probability value is: [0, 1]; a probability value greater than or equal to 0.5 indicates a real image, and a probability value less than 0.5 indicates a deepfake image.
[0094] Step 4: Model training
[0095] Use the pytorch framework to follow steps one, two and three Figure 1 The model was trained for 30 epochs, the batch size was set to 64, the Adam optimizer was used, and the initial learning rate was set to 2e -4 In the first 4 epochs, the learning rate is warmed up using Warmup. During the training process, the learning rate is adjusted to half of the original rate every 5 epochs. The model uses AUC as the evaluation indicator, and records the AUC indicator of the validation set and the parameters of the current model for each epoch.
[0096] Step 5: Model testing
[0097] The model with the largest AUC in step 4 is used as the test model. In order to test the robustness of the model, the model trained in Faceforensics++ is tested on the Celeb-DF-V2 dataset. The range of the predicted probability value for each image is: [0, 1]; where the probability value greater than or equal to 0.5 indicates a real image, and the probability value less than 0.5 indicates a deep fake image.
[0098] Contents not described in detail in this specification belong to the prior art known to professional and technical personnel in the field. Although the embodiments of the present invention have been shown and described, it is understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the attached claims and their equivalents.
Claims
1. A deep fake detection method based on balanced contrastive learning, characterized by: The deep fake detection model is a deep fake detection model of balanced contrastive learning strategy, which combines unsupervised contrastive learning strategy and supervised contrastive learning strategy, uses data enhancement method to expand data, and jointly enhances deep fake classification effect through supervised contrast loss and unsupervised contrast loss; The model consists of a twin network encoder, two multi-scale attention interaction modules MAI, two representation modules Projection and a prediction head Prediction. Its steps are: Step 1: For the input video, first split it into video frames, and randomly extract the video frames to crop the face area; Step 2: For the input image, crop the face area; Step 3: For the cropped face area, data enhancement is used to obtain an enhanced face area image; Step 4: The face area of the original image and the face area of the enhanced image are respectively subjected to the twin network encoder to obtain the multi-scale features of the original image and the multi-scale features of the enhanced image; Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module respectively to obtain the multi-scale attention fusion features of the face area of the original image and the face area of the enhanced image; Step 6: The fused features are passed through a representation layer respectively and input into a prediction head through splicing to obtain the results of deep fake detection.
2. The deep fake detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 1, for the input video, first use the opencv tool to cut the video frame, and randomly select a frame of the video to crop the face area; obtain the face area through dlib, enlarge the face area by 1.2 times, and then crop the image to obtain the face area to be detected.
3. The deep fake detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 2, for the input image, dlib is used to obtain the face area, and the face area is enlarged by 1.2 times and then the image is cropped to obtain the face area to be detected.
4. The deep fake detection method based on balanced contrastive learning according to claim 1, characterized in that: In the step 3, data enhancement uses random flipping (RF), random resizing and cropping (RRC), and random erasing (RE); the probability of the RF is set to 0.5; the RRC uses a scaling factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); the RE uses a scaling factor (0.02, 0.2), an aspect ratio (0.5, 2), and a probability (0.5).
5. The deep fake detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 4, for the face area x of the input original image i and the face region x of the enhanced image i(j) , respectively, through the encoder to obtain the multi-scale features of the original face area (E i , M i , O i ) and (E j(i) , M j(i) , O j(i) ), whose formula is as follows: E i ,M i ,O i =Encoder(x i ); E j(i) ,M j(i) ,O j(i) =Encoder(x j(i) ); Among them, the encoder Encoder is the Xception network, E represents the feature vector of the first stage Entry Flow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
6. The deep fake detection method based on balanced contrastive learning according to claim 5, characterized in that: In step 5, the multi-scale attention interaction module MAI first uses the attention mechanism to extract the attention of multiple scale features, and obtains the direct relevant attention of different features by multiplying them two by two, and finally performs feature fusion by adding features.
7. The deep fake detection method based on balanced contrastive learning according to claim 6, characterized in that: In the multi-scale attention interaction module, each input (E i , M i , O i ) are processed by an SE Attention module, and then go through an adaptive average pooling layer to reduce the size of the feature map to 7×7 pixels; then, a 1x1 Conv2D layer resizes its feature dimension to 2048, and then batch normalizes it to obtain an attention feature with a dimension of 7×7×2048; In this structure, the input E is obtained i , M i , O i The attention feature E′ i , M′ i , O′ i ; Then a pairwise method is used to calculate their cross attention; E′ i and M′ i Multiply pixel by pixel to obtain the cross-attention feature A1; similarly, calculate (E′ i , O′ i ) and (M′ i , O′ i ) to obtain A2 and A3; through the above steps, A1, A2 and A3 have comprehensive information of different scales respectively; A1, A2 and A3 are connected together, and then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain the multi-scale attention feature F i , and as the output of the multi-scale attention interaction module, the formula can be expressed as: F i =MAI(E i ,M i ,O i ). For the input (E j(i) , M j(i) , O j(i) ), and also obtain the multi-scale attention feature F i(j) , the formula can be expressed as: F j(i) =MAI(E j(i) ,M j(i) ,O j(i) ).。 8. The deep fake detection method based on balanced contrastive learning according to claim 7, characterized in that: In step 6, the F obtained in step 5 is i and F j(i) Input a representation layer Projection respectively. The Projection of the representation layer is a simple multi-layer perceptron structure. Through the representation layer, the representation features V of the face area of the original image are obtained respectively. i And enhance the representation feature V of the face area of the image j(i) ; By characterizing the vector V i and V j(i) Splice and input a prediction head Prediction to obtain the true or false prediction result P of the input image i ; The whole process can be expressed by the following formula: V i =Projection(F i ); V j(i) =Projection(F j(i) ); W=Concatenation(V i ,V j(i) ); P i =Prediction(W);。 9. The deep fake detection method based on balanced contrastive learning according to claim 8, characterized in that: The model has three losses, namely, unsupervised contrastive learning loss L cl , supervised contrastive learning loss L scl and classification loss L ce ; The unsupervised contrastive learning loss L cl It can be defined as follows: for a small batch containing N original images, i∈I≡{1,2,...2N} represents the index of all images (original images and enhanced images); if index i is an original image, then index j(i) is the data augmented image of index i, and vice versa; for unsupervised contrastive learning, the sample corresponding to index i is called the anchor point, and the sample corresponding to index j(i) is called the positive sample; other indexes A in the batch (i) The sample with index i is called a negative sample, and the number of negative samples is 2N-2; the representative feature of the sample with index i is recorded as V i ; Therefore, the contrastive learning loss L cl It can be expressed as: in represents the inner product, T is a temperature parameter; The supervised contrastive learning loss L scl It can be defined as: all samples of the same category in the batch (including original images and enhanced images) are defined as positive samples, denoted as B(i), and |B(i)| is its cardinality; Similarly, samples from different categories are regarded as negative samples, denoted as C(i); therefore, the supervised contrastive learning loss L scl It can be expressed as: For all original images in a mini-batch, k∈K≡1,2,...N represents the index of the original image; for the input with index k, its label is Y k , the predicted probability is P k ; Cross entropy is used to quantify the difference between the predicted value and the label; cross entropy loss L ce It can be defined as: The total loss of the model is L total It can be defined as: L total =100L cl +90L scl +10L ce .。
Citation Information
Patent Citations
Image tampering detection method based on multi-scale interaction and cross-feature contrast learning
CN115578631A
Face forgery detection method based on double-layer twin network and sustainable learning
CN116206375A
KR20240103346A
Cited By
Rapid nondestructive testing system based on fusion of near infrared spectrum and Raman spectrum
CN120121571A
Rapid nondestructive testing system based on the fusion of near-infrared spectroscopy and Raman spectroscopy
CN120121571B
Depth forgery detection method and system capable of resisting depth image watermark interference
CN120852975A
