Fair face counterfeiting detection method, device and equipment based on double-domain feature fusion

Abstract

The invention provides a fair face forgery detection method, device and equipment based on double-domain feature fusion, and belongs to the field of face forgery detection. The method comprises the following steps: acquiring a target face image; and inputting the target face image into a face forgery detection model to obtain an output result of the face forgery detection model, the output result being used for indicating whether the target face image is a forgery image, the face forgery detection model comprises an airspace feature extraction module for extracting general forgery features of the target face image, and a high-frequency feature extraction module for extracting high-frequency forgery features of the target face image; and the feature fusion module is used for fusing the general counterfeit features and the high-frequency counterfeit features to obtain target output features, and the output result is determined based on the target output features. Therefore, the fairness and accuracy of model detection can be improved, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present application belongs to the field of face forgery detection, and specifically relates to a fair face forgery detection method, device and equipment based on dual-domain feature fusion. Background Art

[0002] With advances in image generation and processing, facial forgery technology has significantly evolved. These technologies make it possible to manipulate highly realistic faces, raising serious concerns about social security, such as privacy violations and financial fraud. Therefore, developing robust face detection technology is crucial. Traditional face forgery detection methods exhibit inconsistent accuracy when evaluating faces of different genders and groups. For example, the accuracy of assessments for males is higher than that for females, and the accuracy of assessments for faces with darker skin is higher than that for faces with lighter skin. Consequently, current face forgery detection methods suffer from issues of fairness and generalizability, resulting in low accuracy. Summary of the Invention

[0003] The present application provides a fair face forgery detection method, apparatus and device based on dual-domain feature fusion, which is used to improve the fairness and accuracy of face forgery detection and improve the generalization of face forgery detection models.

[0004] This application provides a fair face forgery detection method based on dual-domain feature fusion, including: Get the target face image; The target facial image is respectively input into a facial forgery detection model to obtain an output result of the facial forgery detection model, wherein the output result is used to indicate whether the target facial image is a forged image. The facial forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain a target output feature. The output result is determined based on the target output feature.

[0005] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, the target output feature is obtained in the following manner: respectively calculating the mean and standard deviation of the general forgery feature and the high-frequency forgery feature; preliminarily fusing the general forgery feature and the high-frequency forgery feature according to the mean and the standard deviation to obtain a preliminary fusion feature; determining the channel attention weight based on the preliminary fusion feature; and generating the target output feature based on the preliminary fusion feature and the channel attention weight.

[0006] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, the face forgery detection model is trained based on the following method: obtaining a training sample pair, wherein the training sample pair includes a real face image and a forged face image; inputting the training sample pair into the spatial domain feature extraction module of the pre-trained face forgery detection model to obtain a reference feature of each training sample, wherein the reference feature includes the demographic feature of the training sample; determining a disentanglement loss function based on the reference feature of each training sample; inputting the training sample pair into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain a high-frequency forgery feature of each training sample; A frequency domain perception loss function is determined based on the high-frequency forgery features of each training sample and the true label of each training sample in the high-frequency forgery feature category; the intersection of the demographic categories to which each training sample belongs is determined based on the demographic label corresponding to each training sample; a dual-domain feature fusion loss function is determined based on the intersection of the demographic categories to which each training sample belongs and the demographic features; a target loss function is determined based on the disentanglement loss function, the frequency domain perception loss function and the dual-domain feature fusion loss function; and the pre-trained face forgery detection model is trained based on the target loss function to obtain a trained face forgery detection model.

[0007] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, the disentanglement loss function is determined based on the reference features of each training sample, including: disentangling the reference features into forgery features, the demographic features and background features, the forgery features including the general forgery features and the specific forgery features; respectively determining the classification results of the general forgery features, the specific forgery features and the demographic features in their respective classification categories; determining the classification loss function based on the classification results; obtaining the first Euclidean distance of the training sample with its similar image and the second Euclidean distance with its dissimilar image; and calculating the classification loss function based on the first Euclidean distance. and the second Euclidean distance to determine a contrast loss function; generate a first reconstructed image according to the forged features of the real face image in each training sample pair, the demographic features of the real face image and the background features of the real face image; generate a second reconstructed image according to the forged features of the forged face image in each training sample pair, the demographic features of the real face image and the background features of the real face image; determine a reconstruction loss function according to the real face image, the first reconstructed image and the second reconstructed image; determine the disentanglement loss function according to the classification loss function, the contrast loss function and the reconstruction loss function.

[0008] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, the training sample pairs are input into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain the high-frequency forgery features of each training sample, including: constructing a two-dimensional Gaussian kernel based on the coordinates of each pixel in the face image relative to the center of the filter kernel and the standard deviation of the Gaussian distribution; assigning a value to each position in the convolution kernel to obtain a unit pulse kernel, the value of the center position of the convolution is 1, and the values ​​of the remaining positions are 0; constructing a high-pass filter of the high-frequency feature extraction module based on the two-dimensional Gaussian kernel and the unit pulse kernel; inputting the training sample pairs into the high-pass filter to obtain a high-frequency image of each training sample output by the high-pass filter; and inputting the high-frequency image into the backbone network of the high-frequency feature extraction module to obtain the high-frequency forgery features of each training sample.

[0009] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, the dual-domain feature fusion loss function is determined based on the intersection of the demographic categories to which each training sample belongs and the demographic features, including: determining a feature classification result based on the demographic features; determining an initial cross-entropy loss function based on the feature classification result and the intersection of the demographic categories to which the training data belongs; determining an internal loss function based on the cross-entropy loss function and a local weight parameter; obtaining the optimal weight of each demographic category intersection; performing weighted summation on the internal loss function according to the optimal weight, and performing global smoothing processing based on a smoothing function to obtain a dual-domain feature fusion loss function.

[0010] According to the fair face forgery detection method based on dual-domain feature fusion provided by the present application, before inputting the training sample pairs into the spatial domain feature extraction module of the pre-trained face forgery detection model, the method also includes: determining the current training round; when the current training round is greater than the preset round, determining the output result of the previous round of the pre-trained face forgery detection model; determining the classification difficulty based on the output result of the previous round; determining the data enhancement amplitude based on the classification difficulty; and performing data enhancement processing on the training sample pairs of the current training round according to the data enhancement amplitude.

[0011] This application also provides a fair face forgery detection device based on dual-domain feature fusion, comprising: An acquisition unit, used for acquiring a target face image; A detection unit is used to input the target facial image into a facial forgery detection model to obtain an output result of the facial forgery detection model, wherein the output result is used to indicate whether the target facial image is a forged image. The facial forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain target output features. The output result is determined based on the target output features.

[0012] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-mentioned fair face forgery detection methods based on dual-domain feature fusion.

[0013] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned fair face forgery detection methods based on dual-domain feature fusion.

[0014] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned fair face forgery detection methods based on dual-domain feature fusion.

[0015] The present application provides a fair face forgery detection method, apparatus, and device based on dual-domain feature fusion. First, a target face image is acquired. Then, the target face image is input into a face forgery detection model to obtain an output result of the face forgery detection model. The output result is used to indicate whether the target face image is a forged image. The face forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target face image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target face image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain target output features. The output result is determined based on the target output features. This solution extracts forgery features from both the spatial and frequency domain perspectives, and then fuses the dual-domain features based on the feature fusion module to obtain more general and fair forgery features, thereby improving the accuracy of face forgery detection and enhancing the generalization and fairness of the face forgery detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flowchart of a fair face forgery detection method based on dual-domain feature fusion provided by this application.

[0018] Figure 2 This is one of the structural diagrams of the face forgery detection model provided in this application.

[0019] Figure 3 This is the second structural diagram of the face forgery detection model provided by this application.

[0020] Figure 4 This is a schematic diagram of the model training process provided by this application.

[0021] Figure 5 This is one of the experimental result comparison charts provided in this application.

[0022] Figure 6 This is the second comparison chart of experimental results provided by this application.

[0023] Figure 7 This is a block diagram of the functional units of a fair face forgery detection device based on dual-domain feature fusion provided by this application.

[0024] Figure 8 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0026] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0027] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] Current face forgery detection methods have problems with fairness and lack of generalization, resulting in low accuracy of face forgery detection.

[0029] To address the above issues, this application provides a fair face forgery detection method, device and equipment based on dual-domain feature fusion. Figure 1 ,The fair face forgery detection method based on dual domain feature fusion includes the following steps.

[0030] S101, obtaining a target face image.

[0031] The target face image is the face image to be detected. After the face image to be detected is obtained, it can be preprocessed such as image size cropping to obtain the target face image.

[0032] S102: Input the target face image into a face forgery detection model to obtain an output result of the face forgery detection model.

[0033] The output result indicates whether the target facial image is a forged image. That is, the face forgery detection model detects whether the input facial image is a real or forged facial image. The face forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and high-frequency forgery features to obtain target output features. The output result is determined based on the target output features.

[0034] In a specific implementation, after a target facial image is input into the face forgery detection model, the spatial feature extraction module first extracts general forgery features from the target facial image. Simultaneously, the high-frequency feature extraction module extracts high-frequency features. These high-frequency features and general forgery features are then input into a feature fusion module to generate fused target output features. Based on these target output features, the target facial image is then determined to be authentic or forged. This feature fusion module can be an attention feature fusion module.

[0035] It can be seen that in this embodiment, forgery features are extracted from both the spatial domain and the frequency domain, and then the dual-domain features are fused based on the feature fusion module to obtain more general and fair forgery features, thereby improving the accuracy of face forgery detection and improving the generalization and fairness of the face forgery detection model.

[0036] In a possible embodiment, the target output feature is obtained by: respectively calculating the mean and standard deviation of the general forgery feature and the high-frequency forgery feature; preliminarily fusing the general forgery feature and the high-frequency forgery feature according to the mean and the standard deviation to obtain a preliminary fusion feature; determining the channel attention weight according to the preliminary fusion feature; and generating the target output feature according to the preliminary fusion feature and the channel attention weight.

[0037] During feature fusion, a channel attention mechanism is introduced into the feature fusion module to further adjust the channel weights of the fused features. It automatically learns the importance of each channel and assigns a weight to each channel. This weight reflects the importance of the channel to the final task. A channel with a larger weight means it contains more critical information, that is, information related to forgery. A channel with a smaller weight may contain noise or redundant information. Figure 2 , Figure 2 The attention fusion module is the feature fusion module in this solution. The feature fusion steps include normalizing the general forged features (x) and the high-frequency forged features (y) by calculating the mean and standard deviation of the two features respectively. Through Adaptive Instance Normalization (AdaIN), the general forged features (x) are normalized by their mean and standard deviation, and then the mean and standard deviation of the high-frequency forged features (y) are used to scale and translate the normalized general forged features (x) to obtain the preliminary fusion features. , specifically calculated by the following formula: in, is the standard deviation of the general forged features, is the mean of general forged features, is the standard deviation of high-frequency forged features, is the mean of high-frequency forged features.

[0038] After obtaining the preliminary fusion features, the preliminary fusion features are input into the channel attention module CA of the feature fusion module to calculate the attention weight ω: Then, the preliminary fusion features are weighted based on the attention weights and the final output features fuse_fea are generated: It can be seen that in this embodiment, forgery features are extracted from both the spatial domain and the frequency domain, and then the dual-domain features are fused based on the feature fusion module to obtain more general and fair forgery features, thereby improving the accuracy of face forgery detection and improving the generalization and fairness of the face forgery detection model.

[0039] In a possible embodiment, the face forgery detection model is trained based on the following method: obtaining a training sample pair, wherein the training sample pair includes a real face image and a forged face image; inputting the training sample pair into a spatial feature extraction module of a pre-trained face forgery detection model to obtain a reference feature of each training sample, wherein the reference feature includes a demographic feature of the training sample; determining a disentanglement loss function based on the reference feature of each training sample; inputting the training sample pair into a high-frequency feature extraction module of the pre-trained face forgery detection model to obtain a high-frequency forgery feature of each training sample; and determining a disentanglement loss function based on the reference feature of each training sample. The frequency domain perception loss function is determined based on the high-frequency forged features of the samples and the true label of each training sample in the high-frequency forged feature category; the intersection of the demographic categories to which each training sample belongs is determined based on the demographic label corresponding to each training sample; the dual-domain feature fusion loss function is determined based on the intersection of the demographic categories to which each training sample belongs and the demographic features; the target loss function is determined based on the disentanglement loss function, the frequency domain perception loss function and the dual-domain feature fusion loss function; the pre-trained face forgery detection model is trained based on the target loss function to obtain a trained face forgery detection model.

[0040] Among them, such as Figure 3 As shown, the face forgery detection model includes a spatial feature extraction module (i.e. Figure 3 The spatial decoupling branch shown in (a) in Figure 2), the high-frequency feature extraction module (i.e. Figure 3(b) shows the backbone network of the high-frequency feature extraction module and the feature fusion module. Therefore, when obtaining the target loss function, we can obtain the disentanglement loss function based on the spatial feature extraction module, the frequency-domain perception loss function based on the high-frequency feature extraction module, and the dual-domain feature fusion loss function based on the feature fusion module.

[0041] As can be seen, in this embodiment, loss functions are obtained based on multiple modules respectively, and a final loss function is obtained, which can make the finally trained face forgery detection model more fair and have higher generalization and accuracy.

[0042] In a possible embodiment, the determining of the disentanglement loss function based on the reference features of each training sample includes: disentangling the reference features into forged features, the demographic features and background features, the forged features including the general forged features and the specific forged features; respectively determining the classification results of the general forged features, the specific forged features and the demographic features in their respective classification categories; determining the classification loss function based on the classification results; obtaining the first Euclidean distance of the training sample and its similar image and the second Euclidean distance of its dissimilar image; determining the contrast loss function based on the first Euclidean distance and the second Euclidean distance; generating a first reconstructed image based on the forged features of the real face image, the demographic features of the real face image and the background features of the real face image in each training sample pair; generating a second reconstructed image based on the forged features of the forged face image, the demographic features of the real face image and the background features of the real face image in each training sample pair; determining the reconstruction loss function based on the real face image, the first reconstructed image and the second reconstructed image; and determining the disentanglement loss function based on the classification loss function, the contrast loss function and the reconstruction loss function.

[0043] Among them, the spatial feature extraction module can obtain facial features through the encoder, and then disentangle the facial features into general forged features f c , specific forgery features f s , background features c and demographic features d. At the same time, for general forgery features f c , specific forgery features f s And demographic features d can set three classification heads respectively, and add three classification losses to supervise and realize the classification of the above three features. c The real and fake classification is performed. For a specific fake feature f sThe classification is performed into Deepfakes, Face2Face, FaceShifter, FaceSwap, NeuralTextures, and Real. For the demographic feature d, the classification is based on gender and group. Based on the classification results of each category, the classification loss function L can be obtained. cls : Among them, C is the standard cross entropy loss, M is the population-aware margin loss, , , are the general forged features of the i-th training sample , specific forgery features , and demographic characteristics The corresponding classification head, Y i , S i , D i are their corresponding true labels respectively.

[0044] In order to better separate general forged features from specific forged features, contrast loss can be added to constrain the feature. The contrast loss function L con for: Among them, b is a boundary hyperparameter, this method minimizes the anchor image (i.e. training sample) x A and its similar image x P Maximize the Euclidean distance between the anchor image x A and its dissimilar image x N The Euclidean distance between .

[0045] In order to ensure the integrity of the extracted features and the consistency between the reconstructed image generated by the decoder and the original image, a reconstruction loss L is also designed for pixel-level images. rec : Among them, X i is the real face image in the training sample pair, c i is the background feature of the real face image, f i is the forged feature of the real face image, d i is the demographic characteristics of real face images, f i ' is the forged feature of the forged face image in the training sample pair. That is, the reconstruction loss L recThe left term is the self-reconstruction loss, which leverages the latent features of the input image to minimize the reconstruction error. The right term is the cross-reconstruction loss, which penalizes reconstruction errors by combining spurious features from paired images. These two losses, when combined, can significantly improve the performance of feature disentanglement.

[0046] Therefore, the disentanglement loss function determined based on the spatial feature extraction module is L dis : Among them, λ1 and λ2 are hyperparameters.

[0047] It can be seen that in this embodiment, the disentanglement loss function is obtained based on classification loss, contrast loss and reconstruction loss, which can improve the fairness of the trained face forgery detection model and improve the generalization performance and accuracy of the model.

[0048] In a possible embodiment, the inputting the training sample pairs into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain the high-frequency forgery features of each training sample includes: constructing a two-dimensional Gaussian kernel based on the coordinates of each pixel in the face image relative to the center of the filter kernel and the standard deviation of the Gaussian distribution; assigning a value to each position in the convolution kernel to obtain a unit pulse kernel, the value of the center position of the convolution is 1, and the values ​​of the remaining positions are 0; constructing a high-pass filter of the high-frequency feature extraction module based on the two-dimensional Gaussian kernel and the unit pulse kernel; inputting the training sample pairs into the high-pass filter to obtain a high-frequency image of each training sample output by the high-pass filter; and inputting the high-frequency image into the backbone network of the high-frequency feature extraction module to obtain the high-frequency forgery features of each training sample.

[0049] Among them, such as Figure 3 As shown in Figure 1, after obtaining a training sample pair (consisting of two face images, the forged x0 and the real x1), a high-pass filter is used to obtain the high-frequency image. This high-frequency image is then fed into the encoder of the high-frequency feature extraction module, which is the backbone network of the high-frequency feature extraction module to extract high-frequency features. Therefore, a high-pass filter must be constructed first. The essence of the high-pass kernel of this high-pass filter can be obtained by subtracting the Gaussian sum from the original unit pulse kernel. The design of the two-dimensional Gaussian kernel is as follows: Among them, x and y are the coordinates of each pixel in the face image relative to the center of the filter kernel (nucleus), σ is the standard deviation of the Gaussian distribution, and G is the two-dimensional Gaussian distribution function.

[0050] The unit pulse kernel is a unit pulse matrix, which is as follows: Among them, x and y represent the position in the convolution kernel. Only the value of the center position is 1, and the values ​​of the other positions are 0.

[0051] So the formula for the high-pass filter kernel based on the Gaussian filter is as follows: Finally, the generated high-pass filter kernel is normalized to ensure that its center value is -1, and the single-channel high-pass filter kernel is expanded to multiple channels to accommodate multi-channel images. In this way, each channel shares the same high-pass filter kernel.

[0052] In the specific implementation, the backbone network of this high-frequency feature extraction module is Xception, which is an architecture based on deep convolutional neural network (CNN). The core idea of ​​Xception is depthwise separable convolution, which extracts more efficient features by disassembling the standard convolution. The Xception structure can be divided into entry flow (i.e. Figure 3 The Xception network consists of three parts: Xception Blocks 1-3, the middle flow, and the exit flow. In practical applications, after obtaining a high-frequency image using a high-pass filter, it is processed through two 3x3x32 convolution kernels, batch normalization, and a Reinforced Luminance (ReLU) activation function before extracting high-frequency forged features using the Xception network.

[0053] In the specific implementation, after extracting the high-frequency forgery feature, it is necessary to supervise the feature to ensure that it can distinguish between true and false images. Therefore, the frequency domain perception loss is designed, which is essentially based on the cross entropy loss. The frequency domain perception loss function L fre as follows: where h fre High-frequency forgery features Corresponding to the classification head, Gi is the true label of the training sample in the high-frequency forged feature category.

[0054] It can be seen that in this embodiment, the high-frequency image is obtained by using a high-pass filter, and the high-frequency forgery features are extracted based on the high-frequency image, which can improve the efficiency of extracting the high-frequency forgery features.

[0055] In a possible embodiment, determining the dual-domain feature fusion loss function based on the intersection of the demographic categories to which each training sample belongs and the demographic characteristics includes: determining a feature classification result based on the demographic characteristics; determining an initial cross-entropy loss function based on the feature classification result and the intersection of the demographic categories to which the training data belongs; determining an internal loss function based on the cross-entropy loss function and a local weight parameter; obtaining the optimal weight of each demographic category intersection; performing weighted summation on the internal loss function based on the optimal weight, and performing global smoothing based on a smoothing function to obtain a dual-domain feature fusion loss function.

[0056] Among them, the final fusion loss obtained based on the feature fusion module can be achieved by fusing multiple category intersection losses through dynamic weighting and smoothing. First, obtain the demographic categories to which all training samples in the training set belong, such as the gender and group to which the training sample belongs. The same training sample can correspond to multiple demographic categories, and then determine the intersection of all demographic categories corresponding to the training sample. For example, intersection 1 corresponds to the gender of the training sample being male and the group to which it belongs being group A. For each category intersection i, calculate its internal loss Loss inner,i : Loss inner,i =weighted_loss(L entropy [intersec_label==i],λ i ) Among them, λi is the local weight parameter obtained by optimization, intersec_label is the demographic label, L entropy is the initial cross entropy loss and weighted_loss is the function used to calculate the local weighted loss.

[0057] The internal losses of all categories are weighted summed and globally smoothed using the smoothing function to obtain the final fusion loss L fuse : Here λ i is the optimal weight for each category intersection, I is the set of all category intersections, and smooth is the function used for global smooth loss.

[0058] Therefore, the target loss function can be expressed as the disentanglement loss function L dis , frequency domain perception loss function L fre And the dual-domain feature fusion loss function L fuse Composition, that is, the objective loss function L is: L=L dis +λ3Lfre +λ4L fuse Among them, λ3 and λ4 are hyperparameters.

[0059] It can be seen that in this embodiment, the fairness and accuracy of the trained face forgery detection model can be improved by fusing multiple category intersection losses and determining the dual-domain feature fusion loss function through dynamic weighting and smoothing processing.

[0060] In a possible embodiment, before inputting the training sample pairs into the spatial feature extraction module of the pre-trained face forgery detection model, the method further includes: determining a current training round; when the current training round is greater than a preset round, determining the output result of the previous round of the pre-trained face forgery detection model; determining the classification difficulty based on the output result of the previous round; determining the data enhancement amplitude based on the classification difficulty; and performing data enhancement processing on the training sample pairs of the current training round according to the data enhancement amplitude.

[0061] The annotations added to the four forgery detection datasets (FF++, Celeb-DF, DFD, and DFDC) provide race and gender labels for these datasets. Dlib is then used to extract and align faces, and the cropped faces are resized to 256×256 for training and testing. After obtaining frames from the training dataset (FF++), the face images are preprocessed.

[0062] In the specific implementation, an entropy-driven facial data enhancement scheme is designed during preprocessing. A parameter called warm_up can be set, and its default value is 3. That is, when the number of training rounds is less than 3, only traditional normalization is performed on the images. When the number of training rounds is greater than 3, entropy-driven facial data enhancement is first performed on the images. Enhancement methods include common horizontal flipping, random scaling, color dithering, and brightness and sharpness transformations. The enhancement magnitude is calculated based on the information entropy of the samples in the previous round.

[0063] Specifically, for our face forgery detection model f θ , x is the input sample image, f θ (x) is the output of the face forgery detection model. Based on the output results of the previous round, we can obtain the probability distribution g(x): g(x)=softmax(f θ (x)) The output of the softmax function is used as information entropy. g(x) is a probability distribution, g(x)1+g(x)2=1, meaning that g(x)1 can be the probability that the output result is correct, and g(x)2 can be the probability that the output result is incorrect.

[0064] The classification difficulty of a sample is defined as: The enhancement of samples is inversely proportional to the difficulty of classification. The higher the classification difficulty, the lower the enhancement; the lower the classification difficulty, the higher the enhancement.

[0065] The enhancement amplitude is defined as: After the samples are augmented, they are normalized to ensure that the input image data meets the needs of the neural network. After preprocessing, the images input to the model are paired, one real and one forged, and each image includes four attributes: image data, authenticity label, forgery type label, and demographic label.

[0066] As can be seen, in this embodiment, entropy-driven face data augmentation is performed, with the magnitude of the augmentation applied based on the sample's classification difficulty. For samples with high classification difficulty, weak augmentation is applied to improve the model's learning of these samples; whereas for samples with low classification difficulty, strong augmentation is applied to increase sample diversity and improve the model's generalization ability.

[0067] See also Figure 4 The overall logic of the face forgery detection model training process is described. First, original frames (i.e., the aforementioned real face images) and forged images from the training dataset are obtained. The data is preprocessed (perhaps using the entropy-driven data augmentation method described above) and cropped into training face images of a predetermined size. The preprocessed real and forged images are then fed into the spatial disentanglement branch (i.e., the aforementioned spatial feature extraction module) and the frequency branch (i.e., the aforementioned high-frequency feature extraction module) for training. The general forgery features extracted from the spatial disentanglement branch and the high-frequency forgery features extracted from the frequency branch are then fused via AFFM (i.e., the aforementioned feature fusion module) to train the final face forgery detection model. Finally, frames from the test set are obtained, cropped to a predetermined size, and fed into the face forgery detection model to output the true and false detection results. Furthermore, since the classification thresholds for different demographic categories may vary, this solution uses an adaptive approach to adjust the thresholds for different categories to achieve better fairness.

[0068] To better demonstrate the fairness and generalization of our proposed face forgery detection model, we trained our model on the widely used benchmark, FaceForensics++, and tested it on FF++, DeepfakeDetection (DFD), the Deepfake Detection Challenge (DFDC), and Celeb-DF. The forged images we used in FF++ were generated by five face manipulation algorithms: DeepFakes (DF), Face2Face (F2F), FaceSwap (FS), NerualTexture (NT), and FaceShifter (FST). Since the original dataset lacked demographic information for each video or image, we followed previous research on data processing, data annotation, and sensitive attribute combinations (intersections). Therefore, the intersection groups were based on gender-group representations.

[0069] In terms of detection comparison, we use the area under the curve (AUC) to measure the performance of our method compared with previous work, which is consistent with the detection evaluation method used in previous studies. Regarding fairness, we use four different fairness indicators to evaluate the effectiveness of our proposed method. Specifically, we report the equal false alarm rate (F FPR ), maximum equilibrium opportunity ( FMEO ), demographic fairness (F DP ) and overall accuracy fairness (F OAE We compare our method with the state-of-the-art fairness methods in deepfake detection: DAW-FDD (WACV2023), UCF (ICCV2023), and Fairness (CVPR2024), all built on the Xception backbone.

[0070] All experiments are based on PyTorch. We set the batch size to 8, the training rounds to 100, and the optimizer to use the sharpness-aware minimization (SAM) optimizer. For the loss parameters, ρ1 and ρ2 are both 0.1, λ1 and λ2 are 0.05 and 0.3 respectively, and λ3 and λ4 are 0.1 and 1 respectively.

[0071] based on Figure 5 and Figure 6 It can be seen that compared with other deep fake detection methods, the face fake detection model provided by this solution has a higher performance in F FPR 、 FMEO 、F DP and F OAE, AUC and other indicators are the best.

[0072] The following describes a fair face forgery detection device based on dual-domain feature fusion provided by the present application. The fair face forgery detection device based on dual-domain feature fusion described below corresponds to the fair face forgery detection method based on dual-domain feature fusion described above.

[0073] See also Figure 7 A fair face forgery detection device 700 based on dual-domain feature fusion includes: an acquisition unit 701, used to acquire a target face image; a detection unit 702, used to input the target face image into a face forgery detection model to obtain an output result of the face forgery detection model, wherein the output result is used to indicate whether the target face image is a forged image, the face forgery detection model includes a spatial domain feature extraction module for extracting general forgery features of the target face image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target face image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain a target output feature, and the output result is determined based on the target output feature.

[0074] In a possible embodiment, in the target output feature acquisition unit, the detection unit 702 is specifically used to: calculate the mean and standard deviation of the general forgery feature and the high-frequency forgery feature, respectively; perform a preliminary fusion of the general forgery feature and the high-frequency forgery feature according to the mean and the standard deviation to obtain a preliminary fusion feature; determine the channel attention weight according to the preliminary fusion feature; and generate the target output feature according to the preliminary fusion feature and the channel attention weight.

[0075] In a possible embodiment, the fair face forgery detection device 700 based on dual-domain feature fusion further includes a training unit. In terms of the face forgery detection model, the training unit is specifically used to: obtain a training sample pair, wherein the training sample pair includes a real face image and a forged face image; input the training sample pair into the spatial domain feature extraction module of the pre-trained face forgery detection model to obtain a reference feature of each training sample, wherein the reference feature includes the demographic feature of the training sample; determine a disentanglement loss function based on the reference feature of each training sample; input the training sample pair into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain a reference feature of each training sample; high-frequency forgery features of this; determining the frequency domain perception loss function according to the high-frequency forgery features of each training sample and the true label of each training sample in the high-frequency forgery feature category; determining the intersection of the demographic categories to which each training sample belongs according to the demographic label corresponding to each training sample; determining the dual-domain feature fusion loss function according to the intersection of the demographic categories to which each training sample belongs and the demographic features; determining the target loss function according to the disentanglement loss function, the frequency domain perception loss function and the dual-domain feature fusion loss function; training the pre-trained face forgery detection model according to the target loss function to obtain a trained face forgery detection model.

[0076] In one possible embodiment, in terms of determining the disentanglement loss function based on the reference features of each training sample, the training unit is specifically used to: disentangle the reference features into forged features, the demographic features and background features, the forged features including the general forged features and the specific forged features; respectively determine the classification results of the general forged features, the specific forged features and the demographic features in their respective classification categories; determine the classification loss function based on the classification results; obtain the first Euclidean distance of the training sample with its similar image and the second Euclidean distance with its dissimilar image; determine the contrast loss function based on the first Euclidean distance and the second Euclidean distance; generate a first reconstructed image based on the forged features of the real face image, the demographic features of the real face image and the background features of the real face image in each training sample pair; generate a second reconstructed image based on the forged features of the forged face image, the demographic features of the real face image and the background features of the real face image in each training sample pair; determine the reconstruction loss function based on the real face image, the first reconstructed image and the second reconstructed image; determine the disentanglement loss function based on the classification loss function, the contrast loss function and the reconstruction loss function.

[0077] In a possible embodiment, in terms of inputting the training sample pairs into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain the high-frequency forgery features of each training sample, the training unit is specifically used to: construct a two-dimensional Gaussian kernel based on the coordinates of each pixel in the face image relative to the center of the filter kernel and the standard deviation of the Gaussian distribution; assign a value to each position in the convolution kernel to obtain a unit pulse kernel, the value of the center position of the convolution is 1, and the values ​​of the remaining positions are 0; construct a high-pass filter of the high-frequency feature extraction module based on the two-dimensional Gaussian kernel and the unit pulse kernel; input the training sample pairs into the high-pass filter to obtain a high-frequency image of each training sample output by the high-pass filter; input the high-frequency image into the backbone network of the high-frequency feature extraction module to obtain the high-frequency forgery features of each training sample.

[0078] In one possible embodiment, in terms of determining the dual-domain feature fusion loss function based on the intersection of the demographic categories to which each training sample belongs and the demographic characteristics, the training unit is specifically used to: determine a feature classification result based on the demographic characteristics; determine an initial cross-entropy loss function based on the feature classification result and the intersection of the demographic categories to which the training data belongs; determine an internal loss function based on the cross-entropy loss function and a local weight parameter; obtain the optimal weight of each demographic category intersection; perform weighted summation on the internal loss function according to the optimal weight, and perform global smoothing based on a smoothing function to obtain a dual-domain feature fusion loss function.

[0079] In a possible embodiment, before inputting the training sample pairs into the spatial feature extraction module of the pre-trained face forgery detection model, the training unit is further used to: determine the current training round; if the current training round is greater than a preset round, determine the output result of the previous round of the pre-trained face forgery detection model; determine the classification difficulty based on the output result of the previous round; determine the data enhancement amplitude based on the classification difficulty; and perform data enhancement processing on the training sample pairs of the current training round according to the data enhancement amplitude.

[0080] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of the electronic device provided by this application. Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communications bus 840. The processor 810 may invoke logic instructions in the memory 830 to execute a fair face forgery detection method based on dual-domain feature fusion. The method includes: obtaining a target facial image; inputting the target facial image into a face forgery detection model to obtain an output result of the face forgery detection model, wherein the output result indicates whether the target facial image is a forgery. The face forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain target output features. The output result is determined based on the target output features.

[0081] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fair face forgery detection method based on dual-domain feature fusion provided by the above-mentioned methods, the method comprising: obtaining a target face image; inputting the target face image into a face forgery detection model to obtain an output result of the face forgery detection model, the output result being used to indicate whether the target face image is a forged image, the face forgery detection model comprising a spatial domain feature extraction module for extracting general forgery features of the target face image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target face image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain a target output feature, the output result being determined based on the target output feature.

[0083] On the other hand, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned fair face forgery detection methods based on dual-domain feature fusion, the method including: obtaining a target face image; inputting the target face image into a face forgery detection model to obtain an output result of the face forgery detection model, the output result being used to indicate whether the target face image is a forged image, the face forgery detection model including a spatial feature extraction module for extracting general forgery features of the target face image, and a high-frequency feature extraction module for extracting high-frequency forgery features of the target face image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain a target output feature, the output result being determined based on the target output feature.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0085] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A fair face forgery detection method based on dual-domain feature fusion, characterized by: include: Get the target face image; The target facial image is input into a facial forgery detection model to obtain an output result of the facial forgery detection model, wherein the output result is used to indicate whether the target facial image is a forged image. The facial forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain target output features. The output result is determined based on the target output features.

2. The method according to claim 1, characterized in that The target output features are obtained as follows: Calculating the mean and standard deviation of the general forgery feature and the high-frequency forgery feature respectively; Performing a preliminary fusion of the general forgery feature and the high-frequency forgery feature according to the mean and the standard deviation to obtain a preliminary fusion feature; Determining a channel attention weight based on the preliminary fusion features; The target output feature is generated according to the preliminary fusion feature and the channel attention weight.

3. The method according to claim 1, characterized in that The face forgery detection model is trained based on the following method: Acquire a training sample pair, wherein the training sample pair includes a real face image and a forged face image; Inputting the training sample pairs into a spatial feature extraction module of a pre-trained face forgery detection model to obtain reference features for each training sample, wherein the reference features include demographic features of the training sample; Determining a disentanglement loss function according to the reference features of each training sample; Inputting the training sample pairs into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain the high-frequency forgery features of each training sample; Determine a frequency domain perception loss function according to the high-frequency forged features of each training sample and the true label of each training sample in the high-frequency forged feature category; Determining the intersection of demographic categories to which each training sample belongs according to the demographic label corresponding to each training sample; Determining a dual-domain feature fusion loss function according to the intersection of the demographic categories to which each training sample belongs and the demographic features; Determine a target loss function according to the disentanglement loss function, the frequency domain perception loss function and the dual-domain feature fusion loss function; The pre-trained face forgery detection model is trained according to the target loss function to obtain a trained face forgery detection model.

4. The method according to claim 3, characterized in that The determining of the disentanglement loss function according to the reference feature of each training sample includes: untangling the reference features into forged features, the demographic features, and background features, the forged features comprising the general forged features and specific forged features; respectively determining classification results of the general forgery feature, the specific forgery feature, and the demographic feature in their respective classification categories; Determine a classification loss function according to the classification result; Obtaining a first Euclidean distance between the training sample and its similar image and a second Euclidean distance between the training sample and its dissimilar image; Determine a contrast loss function based on the first Euclidean distance and the second Euclidean distance; generating a first reconstructed image based on the forged features of the real face image, the demographic features of the real face image, and the background features of the real face image in each training sample pair; generating a second reconstructed image based on the forged features of the forged facial image, the demographic features of the real facial image, and the background features of the real facial image in each training sample pair; Determine a reconstruction loss function according to the real face image, the first reconstructed image and the second reconstructed image; The disentanglement loss function is determined according to the classification loss function, the contrast loss function and the reconstruction loss function.

5. The method according to claim 3, characterized in that Inputting the training sample pairs into the high-frequency feature extraction module of the pre-trained face forgery detection model to obtain the high-frequency forgery features of each training sample includes: Construct a two-dimensional Gaussian kernel based on the coordinates of each pixel in the face image relative to the center of the filter kernel and the standard deviation of the Gaussian distribution; Assign a value to each position in the convolution kernel to obtain a unit pulse kernel, where the value of the center position of the convolution is 1 and the values ​​of the other positions are 0; Constructing a high-pass filter of the high-frequency feature extraction module according to the two-dimensional Gaussian kernel and the unit pulse kernel; Inputting the training sample pairs into the high-pass filter to obtain a high-frequency image of each training sample output by the high-pass filter; The high-frequency image is input into the backbone network of the high-frequency feature extraction module to obtain the high-frequency forged features of each training sample.

6. The method according to claim 3, characterized in that Determining the dual-domain feature fusion loss function according to the intersection of the demographic categories to which each training sample belongs and the demographic features includes: determining a characteristic classification result based on the demographic characteristics; Determining an initial cross entropy loss function based on the intersection of the feature classification results and the demographic categories to which the training data belongs; Determine an internal loss function based on the cross entropy loss function and local weight parameters; Obtain the optimal weight for the intersection of each demographic category; The internal loss function is weighted and summed according to the optimal weight, and a global smoothing process is performed based on a smoothing function to obtain a dual-domain feature fusion loss function.

7. The method according to any one of claims 3 to 6, characterized in that: Before inputting the training sample pair into the spatial feature extraction module of the pre-trained face forgery detection model, the method further includes: Determine the current training round; If the current training round is greater than a preset round, determining an output result of a previous round of the pre-trained face forgery detection model; Determining the classification difficulty based on the output results of the previous round; Determining the extent of data augmentation based on the classification difficulty; Perform data enhancement processing on the training sample pairs of the current training round according to the data enhancement amplitude.

8. A fair face forgery detection device based on dual-domain feature fusion, characterized in that: include: An acquisition unit, used for acquiring a target face image; A detection unit is used to input the target facial image into a facial forgery detection model to obtain an output result of the facial forgery detection model, wherein the output result is used to indicate whether the target facial image is a forged image. The facial forgery detection model includes a spatial feature extraction module for extracting general forgery features of the target facial image, a high-frequency feature extraction module for extracting high-frequency forgery features of the target facial image, and a feature fusion module for fusing the general forgery features and the high-frequency forgery features to obtain target output features. The output result is determined based on the target output features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the fair face forgery detection method based on dual-domain feature fusion as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fair face forgery detection method based on dual-domain feature fusion as described in any one of claims 1 to 7 is implemented.

Images