Face forgery detection method based on cross-domain consistency learning
Through cross-domain consistency learning and design of cross-domain learning modules and loss functions, the problem of the reduction in accuracy of the face forgery detection model in the existing technology in the unknown forgery methods is solved, and the common feature learning of different forgery methods is improved, and the realization of the realization of the Higher generalization performance and detection accuracy.
Patent Information
- Application Number
- CN202410899329.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-05
AI Technical Summary
When faced with completely unknown forgery methods, the detection accuracy of existing face forgery detection methods is reduced, and there is a lack of learning about commonalities between different forgery methods, resulting in insufficient generalization in real scenarios.
Using a cross-domain consistency learning method, learn multiple learning by co-training the model by combining the data generated by specific forgery technologies (source domain) and different forgery technologies (target domains), and designing cross-domain learning modules and cross-domain consistency center losses. A forgery method generates cross-domain consistency characteristics of images to improve the accuracy of detection.
Improve the generalization performance of the model in the face of various forged images, enhance the detection accuracy of unknown forged data methods, extract the deep fine-grained features through cross-domain consistency center loss and diversion center differential attention, and increase the inter-class Distance reduces the intra-class distance and improves the accuracy of face forgery detection.
Smart Images

Figure CN119027995A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a face forgery detection method based on cross-domain consistency learning. Background Art
[0002] With the popularity of deep generative models, face forgery technology has made rapid progress, producing a large number of images that are difficult for the human eye to distinguish. These forged media may be abused for unethical and malicious purposes, posing a huge threat to security. Therefore, researchers from various countries have joined the research on "face anti-counterfeiting".
[0003] Face forgery detection is a binary classification task. Existing face forgery detection methods mainly detect suspected forgery artifacts left over from the manipulation process. Early works mainly used hand-crafted features to identify the differences between real and forged images, such as lack of blinking or inconsistent head poses. Although these methods are effective in some cases, they usually lack sufficient generalization ability because hand-crafted features often require in-depth knowledge and experience of experts to design, and the applicability of these features may be greatly reduced for different forgery techniques.
[0004] With the continuous development of deep learning, a large number of deep learning-based methods learn texture-related or spectrum-related deep features in the spatial domain and frequency domain. Representative methods include: FDFL (Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection) and SFDG (Dynamic Graph Learning with Content-guided Spatial-Frequency Relation Reasoning for Deepfake Detection). This type of method mainly improves the recognition accuracy of forged content by extracting deep features. Later, researchers also improved the generalization of face forgery detection methods from the aspects of data and learning strategies. Representative methods include: EFNB4+SBIs (Detecting Deepfakes with Self-Blended Images) and SLAE (Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection). The core idea of this type of method is to enrich the training face image set and use unsupervised learning methods to force the network model not to overfit to a certain forgery method, so that it can perform well in the face of different forgery methods.
[0005] Although these methods have made significant progress in face forgery detection, they still cause the accuracy of model detection to decrease when facing completely unknown forgery methods. That is to say, when new types of operations appear, their performance will be adversely affected even if they are semantically similar. This is because these methods all start from the perspective of binary classification, analyzing the feature differences between images generated by the same forgery method and the original images, but ignoring the commonalities between different forgery methods. Therefore, it is not robust to train only by training a data set of forged faces to solve the generalization problem of face forgery detection models in real scenarios. Therefore, how to make the model learn the commonalities between different forged data and achieve highly generalized face forgery detection is still a challenging problem. In order to solve the above problems, the present invention proposes a face forgery detection method based on cross-domain consistency learning.
[0006] After searching, the application publication number CN117292442B is a cross-modal and cross-domain universal face forgery positioning network. The steps of the present invention are as follows: 1. Preprocessing of experimental data; 2. Constructing a face forgery positioning network ForgeryNet; 3. Training the ForgeryNet network model; 4. Using the trained ForgeryNet network model to apply on multiple forgery positioning data sets, and evaluating the positioning results. However, the patent still only trains the model to learn the interactive features of the spatial domain and the frequency domain under the data set of the same forgery method, and cannot well explore the cross-domain consistency features between multiple forgery methods, and the generalization problem of face forgery detection has not been well solved. Therefore, this patent uses a set of data generated by a specific forgery technology (source domain) and different forgery technologies (target domain) to jointly train the model. At the same time, a cross-domain learning module is designed to learn the cross-domain consistency features of images generated by multiple forgery methods. In addition, a cross-domain consistency center loss is further designed to expand the inter-class distance and reduce the intra-class distance, thereby optimizing the accuracy of face forgery detection. Summary of the invention
[0007] The present invention aims to solve the above problems of the prior art. A face forgery detection method based on cross-domain consistency learning is proposed. The technical solution of the present invention is as follows:
[0008] A face forgery detection method based on cross-domain consistency learning comprises the following steps:
[0009] Step 1: Input a set of labeled source data and labeled target data into the detection network, extract the training sample features of the source domain and the target domain through the basic network, and feed them into the differential attention of the diversion center of each branch respectively;
[0010] Step 2: The split center differential attention obtains the local fine-grained features of the forged image by capturing the pixel intensity information and pixel gradient information, and injects the heterogeneous receptive field size into the tag, and extracts deep fine-grained features by using different sizes of keys and values for different attention heads;
[0011] Step 3: The cross-domain learning module captures the cross-domain consistency features of the source domain and the target domain by sharing the query Query in the source domain split center differential attention and the key Key and value Value in the target domain data;
[0012] Step 4: Input the final features of the source domain and the target domain into the classifier to judge the authenticity of the image, and design the cross-domain consistency center loss for supervision; use the distillation loss for the final features of the target domain and the cross-domain consistency features of the cross-domain learning module to narrow the difference between the two, so that the target domain branch can obtain the knowledge of the source domain branch as much as possible; at the same time, input the final features of the source domain and target domain pseudo-images into the discriminator for domain judgment, and further learn the deep cross-domain consistency representation by minimizing the discrimination loss.
[0013] Furthermore, the step 1 specifically includes the following steps:
[0014] A1. Detect the facial key points of the forged face images in the source domain and the target domain through the face detection and alignment network MTCNN, align the forged face images, and crop them into an input image I of size 224×224. s ,I t ;
[0015] A2. I s ,I t Input into the basic network to extract the primary features of the image, respectively using x s 、x t express; x s 、x t Feed to the split center differential attention module; C, H, W represent the number of image channels, height and width respectively.
[0016] Furthermore, the step 2 specifically includes:
[0017] B1. Input low-level features x s 、x t Use 3×3 convolution to generate query feature X q , the formula is:
[0018] Xq(p0)=∑ pn∈R W(pn)·X(p0+pn) (1)
[0019] Where W(pn) represents the weight, p0 represents the current position on the input and output feature maps, and pn enumerates the position in the receptive field area R; X q Flattened into a sequence of tokens Treat each pixel as a token;
[0020] B2. In X q Local central difference convolution is used to generate fine-grained invariant features It has the shared weight W(pn) of the above 3×3 convolution, and its formula is:
[0021] X kv(p0) = Xq(p0) - ∑ pn∈R W(pn)·X(p0) (2)
[0022] B3. In order to obtain deeper fine-grained features, a diversion system is adopted to divide multiple attention heads into several groups, and each group uses different sizes of key tokens and value tokens; specifically, for different attention heads, K and V are downsampled to different sizes; the specific formula is as follows:
[0023]
[0024] Where MAT() is the multi-scale token aggregation layer, i is the head index, ri is the downsampling rate, and W i K , W i V are the parameters of the linear projection; and value token
[0025] B4. Finally, in order to obtain the final fine-grained features, the attention score between the inputs is calculated by the dot product between the Query and the Key, and the softmax function is applied to the score to obtain the probability distribution of positive values; the Value is weighted and summed using the attention weight to obtain the final output; the calculation formula is as follows:
[0026]
[0027] Among them, d h Indicates the dimension.
[0028] Furthermore, the step 3 specifically includes:
[0029] C1, low-level features of input x s 、x t By splitting the center differential attention, we can get the Q corresponding to the source domain. s , K s 、V s Q corresponding to the target domain t , K t 、V t ; Use the source domain’s Q in the cross-domain learning module s and K in the target domain t 、V t To learn the forged features that make the target domain image similar to the source domain;
[0030] C2. Through Q s and K t The attention score between the inputs is calculated by taking the dot product between them and applying the softmax function to the score to get a probability distribution of positive values.t Perform weighted summation to obtain the final output. The calculation formula is as follows:
[0031]
[0032] Among them, d h Indicates the dimension.
[0033] Furthermore, in step 4, the final features of the source domain and the target domain are input into the classifier to judge the authenticity of the image, and a cross-domain consistency center loss is designed for supervision, which specifically includes the following steps:
[0034] D1. Center loss maintains a center vector matrix during training, with one center vector for each category. In each training iteration, the loss calculates the Euclidean distance between the feature vector of the current sample and its corresponding center vector, and updates the center vector of the category to which the sample belongs through back propagation, so as to better cluster similar samples. The formula is as follows:
[0035]
[0036] Where N is the total number of training samples, c is the category center;
[0037] D2. Design cross-domain consistency center loss function L ce , this function adds a maximum category center constraint to expand the distance between categories;
[0038]
[0039] Where N is the number of samples in a mini-batch, x i is the feature of the i-th sample, c is the category center, c s 、c t are the true and false category centers respectively, and m is the interval boundary between category centers;
[0040] D3. Introduce softmax loss with global information to guide the update of the center point. The formula of softmax loss function is as follows:
[0041]
[0042] where x i represents the characteristics of the i-th sample belonging to the j-th class, represents the weight matrix of the i-th sample, represents the weight matrix of the jth class, b j represents bias, and N represents the number of categories.
[0043] Furthermore, step 4 uses distillation loss for the final features of the target domain and the cross-domain consistency features of the cross-domain learning module, specifically including the following steps:
[0044] E1. The features obtained by the cross-domain learning module are fed into the classifier to obtain a probability distribution. The probability distribution is regarded as a soft label to further supervise the target domain branch through distillation loss. The formula is as follows:
[0045]
[0046] Among them, q k and p k are the probabilities of category k obtained from the cross-domain learning branch and the target domain branch, respectively.
[0047] Furthermore, the step 4 inputs the final features of the source domain and the target domain pseudo images into the discriminator for domain judgment, and further learns the deep cross-domain consistency representation by minimizing the discrimination loss, which specifically includes the following steps:
[0048] F1. The entire feature extraction process is regarded as a feature extractor G, and the entire feature extractor G is trained to deceive pseudo faces so that the domain labels of pseudo faces cannot be distinguished; the entire feature extractor G and the discriminator D are trained in an adversarial manner to reduce domain differences and learn deep domain-invariant features of pseudo faces across different related domains;
[0049] F2. During the training process, the discriminator is optimized by minimizing the loss of domain classification and the feature extractor is optimized by maximizing the same loss. The specific formula is as follows:
[0050]
[0051] Where M represents the total number of pseudo training samples, x f is a forged image sample, z i is the domain label, E(.) represents the feature extractor, and D(.) is the discriminator.
[0052] Furthermore, the four losses are added together to obtain the final image forgery detection loss L total , including:
[0053] By integrating the cross-domain learning module and the diversion center differential attention into a unified network framework, a face forgery detection framework based on cross-domain consistency learning is basically formed, and its parameters can be obtained by L total To optimize:
[0054] L total =λ1L dtl +λ2L adv +λ3L ce +LSoftmax (11)
[0055] Among them, λ1, λ2 and λ3 represent the hyperparameters in the overall loss function.
[0056] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for detecting face forgery based on cross-domain consistency learning as described in any one of the above is implemented.
[0057] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting face forgery based on cross-domain consistency learning as described in any one of the above is implemented.
[0058] The advantages and beneficial effects of the present invention are as follows:
[0059] The present invention mainly aims at the problem that the existing face forgery detection methods have poor accuracy in face forgery detection when facing an endless stream of forgery methods and changing reality factors. A face forgery detection method based on cross-domain consistency learning is designed. The core of the method is to learn the commonalities of multiple forged images by learning the consistent forgery features of images generated by different forgery methods, so as to improve the generalization performance of existing methods in the face of various forged images. At the same time, the diversion center differential attention designed by the present invention prompts the model to pay more attention to the deep and fine-grained features of the forged images, so that the cross-domain learning module can learn deeper cross-domain consistency features. Finally, the present invention designs a cross-domain consistency center loss to further improve the recognition accuracy.
[0060] The main advantages and beneficial effects are as follows:
[0061] 1. This paper designs a split central differential attention, which extracts features by capturing pixel intensity information and pixel gradient information while injecting heterogeneous receptive field sizes into the label, effectively capturing the deep and fine-grained features of the source domain and the target domain, and improving the detection accuracy of the model on known forged data methods.
[0062] 2. The present invention designs a cross-domain learning module, which learns the cross-domain consistency features of the source domain and the target domain by sharing the query (Query) in the source domain diversion center differential attention and the key (Key) and value (Value) in the target domain data, thereby improving the detection accuracy of the model on unknown forged data methods.
[0063] 3. The present invention designs a cross-domain consistency center loss to effectively reduce the intra-class distance between images, increase the inter-class distance, and further improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 It is a schematic diagram of the overall network model structure of a preferred embodiment provided by the present invention;
[0065] Figure 2 A schematic diagram of a split center differential attention structure according to an embodiment of the present invention;
[0066] Figure 3 Schematic diagram of the cross-domain learning module structure of an embodiment of the present invention. DETAILED DESCRIPTION
[0067] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.
[0068] The technical solution of the present invention to solve the above technical problems is:
[0069] The embodiment of the present invention is based on the vision transformer feature extraction network as the basic framework. For details, see the literature Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [J]. arXiv preprint arXiv: 2010.11929, 2020.
[0070] First, the primary features of the source domain and the target domain are extracted through the basic network, and are sent to the split center differential attention network proposed in the present invention to learn their respective fine-grained forgery features. At the same time, the features of the source domain and the target domain are sent to the cross-domain learning module, and the key-value pairs K and V generated from the query Q generated from the image and the target domain features are used to learn the cross-domain consistency features. The source domain features, cross-domain consistency features and target domain features are sent to the label classifier, and the standard cross entropy loss is replaced by Softmax loss, distillation loss, cross-domain consistency center loss and discrimination loss. The above four losses are used to constitute the total loss to supervise the learning process of the entire face forgery detection model.
[0071] Figure 1 The overall network model structure diagram of an embodiment of the present invention includes the following steps:
[0072] 1. As attached Figure 1 As shown in the figure, a set of labeled source data and labeled target data are input into the detection network, and the training sample features of the source domain and the target domain are extracted through the basic network, and are fed into the differential attention of the diversion center of each branch respectively; the specific steps include:
[0073] A1. Detect the facial key points of the forged face images in the source domain and the target domain through the face detection and alignment network MTCNN, align the forged face images, and crop them into an input image I of size 224×224. s ,I t ;
[0074] A2. I s ,I t Input into the basic network to extract the primary features of the image, respectively using x s 、x t express. x s 、x t Feed to the split center differential attention module. C, H, W represent the number of image channels, height and width respectively.
[0075] 2. As attached Figure 2 The schematic diagram of the structure of the split center differential attention of an embodiment of the present invention is as follows. The split center differential attention obtains the local fine-grained features of the forged image by capturing the pixel intensity information and the pixel gradient information, and injects the heterogeneous receptive field size into the mark, and extracts deep fine-grained features by using different sizes of keys (Key) and values (Value) for different attention heads; specifically, the following steps are included:
[0076] B1. Input low-level features x s 、x t Use 3×3 convolution to generate query feature X q , the formula is:
[0077] Xq(p0)=∑ pn∈R W(pn)·X(p0+pn)(1)
[0078] Where W(pn) represents the weight, p0 represents the current position on the input and output feature maps, and pn enumerates the position in the receptive field area R. q Flattened into a sequence of tokens Treat each pixel as a token.
[0079] B2. In X q Local central difference convolution is used to generate fine-grained invariant features It has the shared weight W(pn) of the above 3×3 convolution, and its formula is:
[0080] X kv (p0) = Xq(p0) - ∑ pn∈R W(pn)·X(p0) (2)
[0081] B3. In order to obtain deeper fine-grained features, a diversion system is adopted to divide multiple attention heads into several groups, and each group uses different sizes of key tokens and value tokens. Specifically, for different attention heads, K and V are downsampled to different sizes. The specific formula is as follows:
[0082] k i 、v i =MAT(X k , r i )W i K , MAT(X v , r i )W i V (3)
[0083] Where MAT() is the multi-scale token aggregation layer, i is the head index, ri is the downsampling rate, and W i K , W i V are the parameters of the linear projection. and value token
[0084] B4. Finally, in order to obtain the final fine-grained features, the attention score between the inputs is calculated by the dot product between the Query and the Key, and the softmax function is applied to the score to obtain the probability distribution of positive values. The Value is weighted and summed using the attention weight to obtain the final output. The calculation formula is as follows:
[0085]
[0086] Among them, d h Indicates the dimension.
[0087] 3. If Figure 3 The cross-domain learning module structure diagram of an embodiment of the present invention captures the cross-domain consistency features of the source domain and the target domain by sharing the query (Query) in the source domain shunting center differential attention and the key (Key) and value (Value) in the target domain data; specifically, the following steps are included:
[0088] C1, low-level features of input x s 、x t By splitting the center differential attention, we can get the Q corresponding to the source domain. s , K s 、V s Q corresponding to the target domain t , K t 、V t . Use the source domain’s Q in the cross-domain learning modules and K in the target domain t 、V t To learn forged features that make the target domain images similar to the source domain.
[0089] C2. Through Q s and K t The attention score between the inputs is calculated by taking the dot product between them and applying the softmax function to the score to get a probability distribution of positive values. t Perform weighted summation to get the final output. The calculation formula is as follows:
[0090]
[0091] Among them, d h Representation Dimension
[0092] 4. Further, Figure 1 As shown in the figure, the final features of the source domain and the target domain are input into the classifier to judge the authenticity of the image, and the cross-domain consistency center loss is designed for supervision. The specific steps include:
[0093] D1. Center loss maintains a center vector matrix during training, with one center vector for each category. In each training iteration, the loss calculates the Euclidean distance between the feature vector of the current sample and its corresponding center vector, and updates the center vector of the category to which the sample belongs through back propagation to better cluster similar samples. The formula is as follows:
[0094]
[0095] Among them, N is the total number of training samples and c is the category center.
[0096] D2. Designed a cross-domain consistency center loss function L ce , this function adds a maximum category center constraint to expand the distance between categories;
[0097]
[0098] Where N is the number of samples in a mini-batch, x i is the feature of the i-th sample, c is the category center, c s 、c t are the true and false category centers respectively, and m is the interval boundary between category centers.
[0099] D3. Introduce softmax loss with global information to guide the update of the center point. The formula of softmax loss function is as follows:
[0100]
[0101] where x i represents the characteristics of the i-th sample belonging to the j-th class, represents the weight matrix of the i-th sample, W j T represents the weight matrix of the jth class, b j represents bias, N represents the number of classifications;
[0102] 5. Further, Figure 1 As shown in Figure 1, distillation loss is used for the final features of the target domain and the cross-domain consistency features of the cross-domain learning module. The specific steps include:
[0103] E1. Send the features obtained by the cross-domain learning module into the classifier to obtain a probability distribution, which is regarded as a soft label to further supervise the target domain branch through distillation loss. The formula is as follows:
[0104]
[0105] Among them, q k and p k are the probabilities of category k obtained from the cross-domain learning branch and the target domain branch, respectively.
[0106] 6. Further, Figure 1 As shown in the figure, although the use of distillation loss can make the target domain learn the forged features of the source domain image, it is easy to make the target domain over-focus on the deep fine-grained information of the source domain rather than the deep domain consistency information of the source and target domains. Therefore, a discriminator is used to further learn cross-domain consistency features. The final features of the source and target domain pseudo images are input into the discriminator for domain judgment, and the deep cross-domain consistency representation is further learned by minimizing the discrimination loss. Specifically, the following steps are included:
[0107] F1. The entire feature extraction process is considered as a feature extractor G. The entire feature extractor G is trained to fool the fake faces so that the domain labels of the fake faces cannot be distinguished. In other words, the entire feature extractor G and the discriminator D are trained in an adversarial manner to reduce the domain differences and learn deep domain-invariant features of fake faces across different related domains.
[0108] F2. During the training process, the discriminator is optimized by minimizing the loss of domain classification and the feature extractor is optimized by maximizing the same loss. The specific formula is as follows:
[0109]
[0110] Where M represents the total number of pseudo training samples, x f is a forged image sample, z iis the domain label, E(.) represents the feature extractor, and D(.) is the discriminator.
[0111] 7. Further, the above four losses are added together to obtain the final image forgery detection loss L total , including:
[0112] By integrating the cross-domain learning module and the diversion center differential attention into a unified network framework, a face forgery detection framework based on cross-domain consistency learning is basically formed, and its parameters can be obtained by L total To optimize:
[0113] L total =λ1L dtl +λ2L adv +λ3L ce +L Softmax (11)
[0114] Among them, λ1, λ2 and λ3 represent the hyperparameters in the overall loss function.
[0115] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.
[0116] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0118] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.
Claims
1. A face forgery detection method based on cross-domain consistency learning, It is characterized in that The following steps are involved: Step 1: Input a set of labeled source data and labeled target data into the detection network, extract the training sample features of the source domain and the target domain through the basic network, and feed them into the differential attention of the diversion center of each branch respectively; Step 2: The split center differential attention obtains the local fine-grained features of the forged image by capturing the pixel intensity information and pixel gradient information, and injects the heterogeneous receptive field size into the tag, and extracts deep fine-grained features by using different sizes of keys and values for different attention heads; Step 3: The cross-domain learning module captures the cross-domain consistency features of the source domain and the target domain by sharing the query Query in the source domain split center differential attention and the key Key and value Value in the target domain data; Step 4: Input the final features of the source domain and the target domain into the classifier to judge the authenticity of the image, and design the cross-domain consistency center loss for supervision; use the distillation loss for the final features of the target domain and the cross-domain consistency features of the cross-domain learning module to narrow the difference between the two, so that the target domain branch can obtain the knowledge of the source domain branch as much as possible; at the same time, input the final features of the source domain and target domain pseudo-images into the discriminator for domain judgment, and further learn the deep cross-domain consistency representation by minimizing the discrimination loss.
2. According to the method for face forgery detection based on cross-domain consistency learning according to claim 1, It is characterized in that The step 1 specifically comprises the following steps: A1. Detect the facial key points of the forged face images in the source domain and the target domain through the face detection and alignment network MTCNN, align the forged face images, and crop them into an input image I of size 224×224. s ,I t ; A2. I s ,I t Input into the basic network to extract the primary features of the image, respectively using x s 、x t express; x s 、x t Feed to the split center differential attention module; C, H, W represent the number of image channels, height and width respectively.
3. According to the method for face forgery detection based on cross-domain consistency learning according to claim 2, It is characterized in that The step 2 specifically includes: B1. Input low-level features x s 、x t Use 3×3 convolution to generate query feature X q , the formula is: Xq(p0)=∑ pn∈R W(pn)·X(p0+pn) (1) Where W(pn) represents the weight, p0 represents the current position on the input and output feature maps, and pn enumerates the position in the receptive field area R; X q Flattened into a sequence of tokens Treat each pixel as a token; B2. In X q Local center difference convolution is used to generate fine-grained invariant features It has the shared weight W(pn) of the above 3×3 convolution, and its formula is: X kv (p0)=Xq(p0)-∑ pn∈R W(pn)·X(p0) (2) B3. In order to obtain deeper fine-grained features, a diversion system is adopted to divide multiple attention heads into several groups, and each group uses different sizes of key tokens and value tokens; specifically, for different attention heads, K and V are downsampled to different sizes; the specific formula is as follows: Where MAT() is the multi-scale token aggregation layer, i is the head index, ri is the downsampling rate, are the parameters of the linear projection; and value token B4. Finally, in order to obtain the final fine-grained features, the attention score between the inputs is calculated by the dot product between the Query and the Key, and the softmax function is applied to the score to obtain the probability distribution of positive values; the Value is weighted and summed using the attention weight to obtain the final output; the calculation formula is as follows: Among them, d h Indicates the dimension.
4. According to the method for detecting face forgery based on cross-domain consistency learning according to claim 3, It is characterized in that The step 3 specifically includes: C1, low-level features of input x s 、x t By splitting the center differential attention, we can get the Q corresponding to the source domain. s , K S 、V S Q corresponding to the target domain t , K t 、V t ; Use the source domain’s Q in the cross-domain learning module s and K in the target domain t 、V t To learn the forged features that make the target domain image similar to the source domain; C2. Through Q s and K t The attention score between the inputs is calculated by taking the dot product between them and applying the softmax function to the score to get a probability distribution of positive values. t Perform weighted summation to obtain the final output. The calculation formula is as follows: Among them, d h Indicates the dimension.
5. According to the method for detecting face forgery based on cross-domain consistency learning according to claim 4, It is characterized in that The step 4 inputs the final features of the source domain and the target domain into the classifier to judge the authenticity of the image, and designs a cross-domain consistency center loss for supervision, which specifically includes the following steps: D1. Center loss maintains a center vector matrix during training, with one center vector for each category. In each training iteration, the loss calculates the Euclidean distance between the feature vector of the current sample and its corresponding center vector, and updates the center vector of the category to which the sample belongs through back propagation, so as to better cluster similar samples. The formula is as follows: Where N is the total number of training samples, c is the category center; D2. Design cross-domain consistency center loss function L ce , this function adds a maximum category center constraint to expand the distance between categories; Where N is the number of samples in a mini-batch, x i is the feature of the i-th sample, c is the category center, c s 、c t are the true and false category centers respectively, and m is the interval boundary between category centers; D3. Introduce softmax loss with global information to guide the update of the center point. The formula of softmax loss function is as follows: where x i represents the characteristics of the i-th sample belonging to the j-th class, represents the weight matrix of the i-th sample, represents the weight matrix of the jth class, b j represents bias, and N represents the number of categories.
6. According to the method for detecting face forgery based on cross-domain consistency learning according to claim 5, It is characterized in that The step 4 uses distillation loss for the final features of the target domain and the cross-domain consistency features of the cross-domain learning module, and specifically includes the following steps: E1. The features obtained by the cross-domain learning module are fed into the classifier to obtain a probability distribution. The probability distribution is regarded as a soft label to further supervise the target domain branch through distillation loss. The formula is as follows: Among them, q k and p k are the probabilities of category k obtained from the cross-domain learning branch and the target domain branch, respectively.
7. According to claim 6, a face forgery detection method based on cross-domain consistency learning, It is characterized in that The step 4 inputs the final features of the source domain and the target domain pseudo images into the discriminator for domain judgment, and further learns the deep cross-domain consistency representation by minimizing the discrimination loss, which specifically includes the following steps: F1. The entire feature extraction process is regarded as a feature extractor G, and the entire feature extractor G is trained to deceive pseudo faces so that the domain labels of pseudo faces cannot be distinguished; the entire feature extractor G and the discriminator D are trained in an adversarial manner to reduce domain differences and learn deep domain-invariant features of pseudo faces across different related domains; F2. During the training process, the discriminator is optimized by minimizing the loss of domain classification and the feature extractor is optimized by maximizing the same loss. The specific formula is as follows: Where M represents the total number of pseudo training samples, x f is a forged image sample, z i is the domain label, E(.) represents the feature extractor, and D(.) is the discriminator.
8. According to the method for detecting face forgery based on cross-domain consistency learning according to claim 7, It is characterized in that The four losses are added together to obtain the final image forgery detection loss L total , specifically including: By integrating the cross-domain learning module and the diversion center differential attention into a unified network framework, a face forgery detection framework based on cross-domain consistency learning is basically formed, and its parameters can be obtained by L total To optimize: L total =λ 1 L dtl +λ 2 L adv + λ 3 L ce + L Softmax (11) Where λ 1 ,λ 2 and λ 3 Represents the hyperparameters in the overall loss function.
9. An electronic device, It is characterized in that The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting face forgery based on cross-domain consistency learning as claimed in any one of claims 1 to 8 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for detecting face forgery based on cross-domain consistency learning as described in any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
A cross-modal and cross-domain universal face forgery detection method
CN117292442B
Deep forgery detection algorithm based on generalization framework in facial semantic content decomposition field
CN116343279A
Transform-based cross-domain double-branch adversarial domain adaptive image classification method
CN116740434A
Deep fake face detection positioning method capable of learning local difference
CN117496583A
Cross-domain face forgery detection method based on double-branch collaborative learning
CN118097759A
Cited By
Image tampering detection method based on magnified macroscopic and microscopic inconsistent learning
CN120726460A
Semi-supervised domain adaptive deep forgery detection method
CN121482581A
