Face forgery detection method based on multi-modal difference prior and regional feature comparative learning

Abstract

The invention provides a face forgery detection method based on multi-modal difference prior and regional feature comparative learning, and aims to solve the problems of lack of fine positioning labels and insufficient generalization of models to unknown attacks in face forgery detection. The method comprises the following two main parts: (1) a multi-modal difference prior module: aligning true and false image pairs through dense key points, fusing multi-modal difference analysis of a spatial domain (pixel), a structural domain (SSIM) and a frequency domain (FFT), and automatically clustering to generate a high-precision counterfeited area anchor frame as prior; and (2) RFCL-Net taking YOLOv11 as a trunk: innovatively designing an artifact detection branch, guiding BBox-Mask pooling by using anchor frame prior, and accurately extracting features of a tampered region and a context background region. And through a region comparison learning mechanism, a network is forced to learn essential differences of true and false faces in region consistency, and finally efficient and robust face forgery dichotomy and tampered region accurate positioning are realized.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and deep learning image processing technologies, specifically to a face forgery detection method based on multimodal difference priors and region feature contrast learning. This invention is applicable to scenarios such as digital media forensics, network content security auditing, and identity authentication protection, enabling robust binary classification and identification of various deepfake images, as well as automated and precise localization of tampered regions. Background Technology

[0002] With the rapid advancements in deep learning and computer vision technologies, face spoofing techniques, exemplified by Deepfake, have become increasingly sophisticated, capable of generating highly realistic, indistinguishable fake facial images and videos to the naked eye. If these technologies are maliciously exploited for spreading false information, infringing on individual portrait rights, or engaging in fraudulent activities, they will seriously threaten information security, social trust, and even national security. Therefore, developing efficient, robust, and precisely targeted face spoofing detection technologies has become an urgent need in the field of information security.

[0003] Current face forgery detection techniques primarily rely on deep convolutional neural networks (CNNs) to extract image features for binary classification of real and fake faces. Early research focused on specific physiological or physical anomalies, such as unnatural blinking frequencies or facial distortion artifacts, but these methods struggle to adapt to the rapidly improving quality of new forgery techniques. Although current mainstream deep learning methods demonstrate high detection accuracy on specific datasets, their practical applications are still limited by the following key technical bottlenecks.

[0004] First, it relies heavily on high-quality region-level labeled data, which is difficult to obtain. Most existing methods are only supervised training based on image-level "real / fake" labels, lacking fine-grained annotation signals (such as pixel-level masks or anchor boxes) for specific tampered regions (such as replacement areas for eyes or mouths). Since manually obtaining large-scale, high-quality region-level forgery annotations is extremely costly, the model struggles to obtain sufficient supervision information to focus on subtle local tampering traces, thus limiting its detection accuracy and localization capabilities.

[0005] Secondly, the models lack generalization ability and robustness. Many detection methods tend to overfit to single-modal traces left by specific forgery algorithms in the training data, such as specific frequency domain artifacts or color inconsistencies. When faced with unknown attacks using entirely new generation mechanisms, or forged images that have undergone post-processing such as compression and noise, the model's performance often degrades significantly. This is because these models fail to deeply understand the inherent biological and statistical consistency between real facial regions, and the fundamental disruption of this consistency caused by the forgery process.

[0006] To address the core pain points of existing technologies, such as the lack of automated high-precision region annotation methods, poor model generalization robustness to unknown attacks, and difficulty in accurately locating tampered regions, this invention proposes a face forgery detection method based on multimodal difference priors and region feature comparison learning. Summary of the Invention

[0007] The purpose of this invention is to address the limitations of existing deepfake detection technologies, such as the lack of automated high-precision region annotation methods, insufficient robustness of models to generalize against unknown forgery attacks, and difficulty in accurately locating tampered regions. This invention proposes a face forgery detection method based on multimodal difference priors and region feature comparison learning.

[0008] The present invention adopts the following technical solution:

[0009] A face forgery detection method based on multimodal difference prior and region feature contrast learning, characterized by the following main steps:

[0010] S1: Multimodal difference prior generation stage: A specially designed face forgery anchor box generation module (FFA-Gen) is used to process pairs of real face images and forged face images. This module first performs high-precision affine alignment on the image pairs based on dense key points, then calculates and fuses the three modal difference maps in the spatial domain, structural domain, and frequency domain respectively, and finally automatically generates high-precision forged region bounding boxes based on the fused difference maps using a clustering algorithm, which are used as forged anchor box priors; a training dataset containing the original images and their corresponding anchor box prior annotations is constructed.

[0011] S2: Region Feature Contrast Learning Network Construction and Training Stage: Construct a face forgery detection network (RFCL-Net) that integrates a region contrast learning mechanism; the RFCL-Net uses a pre-trained convolutional neural network as a shared backbone network to extract multi-scale feature maps, including a standard localization branch for locating standard face regions and a forgery detection branch for identifying forgery traces; the forgery detection branch uses the forgery anchor box prior generated in step S1 to guide the design of the BBox-Mask region pooling module, accurately extracting tampered region features and context background region features on the feature map, and introducing a region feature contrast loss function to force the network to learn the inconsistency between the two; the network is jointly trained end-to-end by combining localization loss, real / fake classification loss and region feature contrast loss.

[0012] The specific processing flow of the face forgery anchor box generation module (FFA-Gen) in step S1 includes:

[0013] S1.1 Face Detection and Key Point Extraction: The face detection model is used to locate the face region in the input image pair and extract dense face key points; a face convex hull mask is generated based on the key points to limit the effective area for subsequent difference calculation and eliminate background interference;

[0014] S1.2 High-precision affine alignment: Based on the extracted dense key points, the optimal affine transformation matrix is ​​calculated to accurately align the real face image to the coordinate system of the fake face image, eliminating pose and scale differences;

[0015] S1.3 Multimodal difference map calculation: Under the constraint of the face convex hull mask, the pixel-level absolute difference map between the aligned real image and the fake image is calculated to represent the spatial domain difference, the structural similarity (SSIM) difference map is calculated to represent the structural domain difference, and the amplitude spectrum difference map is calculated after performing a fast Fourier transform (FFT) on the image to represent the frequency domain difference.

[0016] S1.4 Difference Map Fusion and Anchor Box Generation: The calculated spatial, structural, and frequency domain difference maps are weighted and fused to obtain a combined difference map. The density-based spatial clustering algorithm (DBSCAN) is used to perform cluster analysis on the highly significant regions in the combined difference map to extract several high-response regions. Boundary box fitting and screening are performed on the clustered regions to finally determine the set of boundary boxes that accurately surround the forgery traces, which serves as the forgery anchor box prior GT of the forged image.

[0017] S1.5 Dataset Construction: The original real images are labeled as negative samples, the original fake images are labeled as positive samples, and the generated fake anchor box prior GT and the corresponding face overall bounding box GT are stored in the annotation file to construct a standard dataset for subsequent network training.

[0018] The specific architectural features of the face forgery detection network (RFCL-Net) that integrates the region contrast learning mechanism in step S2 are as follows:

[0019] S2.1 Shared Backbone Network: A pre-trained YOLOv11 backbone network and neck network are used as feature extractors to perform multi-layer convolution and downsampling operations on the input image, and output a multi-scale intermediate layer feature map sequence containing rich semantic and spatial information.

[0020] S2.2 Standard Localization Branch: This branch connects to the output of the shared backbone network and adopts the standard YOLO detection head structure. This detection head contains several convolutional layers for further processing of multi-scale feature maps, predicting the bounding box coordinates and confidence scores of face targets in the image, and providing location information of the overall face region for subsequent steps.

[0021] S2.3 Forgery Detection Branch: This branch is also connected to the output of the shared backbone network and runs in parallel with the standard localization branch. This branch first includes a Spatial and Channel Frequency Attention (SFA) module, which is used to enhance the feature responses of specific frequencies and spatial locations related to forgery traces in the input feature map and suppress background noise. Then it is connected to a BBox-Mask region pooling core module, which is a key component for realizing region contrastive learning. Finally, it is connected to the feature fusion module and the True / False Binary Classification Head (MLP) to output the final true / false category prediction value.

[0022] The Spatial and Channel Frequency Attention (SFA) module in step S2.3 is specifically configured to refine the input intermediate layer feature map. Its processing includes: setting up a channel attention submodule and a spatial attention submodule in parallel; the channel attention submodule first performs global average pooling on the input feature map to obtain statistical information about the channel dimensions, then learns the nonlinear interaction relationship between channels through a multilayer perceptron (MLP) containing dimensionality reduction and expansion layers, generating a channel attention weight map, and using this weight map to reweight each channel of the input feature map; the spatial attention submodule first performs max pooling and average pooling operations on the input feature map along the channel axis, concatenates the two resulting two-dimensional feature maps, then generates a two-dimensional spatial attention weight map through a standard convolutional layer, and uses this weight map to reweight the spatial location of the feature map to emphasize regions containing high-frequency forgery traces; finally, the feature maps enhanced by channel and spatial attention are fused element-wise as the output of the SFA module.

[0023] The BBox-Mask region pooling core module in step S2.3 is crucial for implementing region contrastive learning, and its operation process is as follows:

[0024] (1) Coordinate mapping: Receive the feature map of the current level, as well as the face bounding box GT and the fake anchor box prior GT read from the training data; according to the downsampling ratio of the current feature map relative to the original image, map the coordinates of the face bounding box GT and the fake anchor box prior GT from the image space to the current feature space;

[0025] (2) Region mask generation: Based on the mapped overall bounding box coordinates of the face, a binary face region mask is constructed on the feature map plane, where the pixel value within the face region is 1 and the rest is 0; based on the mapped prior coordinates of the fake anchor box, a binary tamper mask is constructed; the difference between the face region mask and the tamper mask is calculated through logical operations to generate a context mask representing the untampered area inside the face.

[0026] (3) Masked feature pooling: Using the tampered region mask, the features in the corresponding region of the input feature map are subjected to masked mean pooling or region of interest alignment (ROI Align) operation to extract the tampered region feature vector of fixed dimension; similarly, using the context background region mask, the features in the corresponding region of the input feature map are subjected to the same pooling operation to extract the context background region feature vector of fixed dimension.

[0027] The feature fusion module, the processing of the true / false binary classification head (MLP), and the loss function for joint network training in step S2.3 are defined as follows:

[0028] (1) Feature fusion: The tampered region feature vector extracted in the previous steps and the context background region feature vector are input into the feature fusion module; the feature fusion module uses one or more of the following methods: feature splicing, calculating the absolute value of feature difference, or calculating the element-wise product of features to generate the fused region comparison feature vector;

[0029] (2) Classification prediction: The fused region contrast feature vector is input into the real vs. fake binary classification head (MLP); the binary classification head contains several fully connected layers and non-linear activation functions, which are used to map high-dimensional features into classification prediction Logits values ​​that indicate whether the image is real or fake;

[0030] (3) Definition of joint loss: The total loss function of the RFCL-Net network is defined as the weighted sum of localization loss, real / fake classification loss and region feature contrast loss; wherein, the localization loss is used to supervise the bounding box regression accuracy of the standard localization branch; the real / fake classification loss is used to supervise the classification accuracy of the artifact detection branch; the region feature contrast loss adopts the form of a contrastive loss function, which is used to constrain the network to make the similarity between the extracted tampered region features and the context background region features approach the maximum value when processing real face samples, and to make the similarity between the two approach the minimum value when processing fake face samples, thereby forcing the network to learn the essential difference between real and fake images in terms of region consistency.

[0031] The specific execution process of step S3, the inference and result visualization stage, includes: inputting the single image to be detected into the trained RFCL-Net network; after the network shares the backbone to extract features, the standard localization branch outputs the predicted bounding box coordinates of the face target. Simultaneously, the artifact detection branch outputs a probability score indicating that the image is a fake face. The final real / fake category of the image is determined based on the comparison result of the probability score and a preset threshold. If determined to be a real face, the face bounding box output by the standard localization branch is drawn on the original image using a first striking color (e.g., green) and labeled "Real". If determined to be a fake face, the face bounding box output by the standard localization branch is drawn on the original image using a second striking color (e.g., red) different from the first color, to accurately define the range of the fake region detected by the model, and labeled "Tampered". Finally, a detection result image with visual annotations is output.

[0032] In step S2.3, the joint loss definition uses a cosine embedding loss function for calculating the region feature comparison loss. This loss function receives the extracted tampered region feature vector and the context background region feature vector as input pairs, and is constrained by the sample's real label: when the input sample is a real face, the optimization objective of the loss function is to maximize the cosine similarity between the two feature vectors, making it approach 1, to represent the consistency between regions; when the input sample is a fake face, the optimization objective of the loss function is to minimize the cosine similarity between the two feature vectors, making it lower than a preset boundary threshold (margin), to represent the inconsistency between regions.

[0033] This invention aims to provide a face forgery detection method based on multimodal difference prior and region feature contrast learning, which has the following beneficial effects:

[0034] (1) Automated high-precision annotation: The FFA-Gen module can automatically generate fine forged anchor box priors that integrate multimodal information without the need for manual labor, effectively solving the bottleneck problem of scarce regional annotation data.

[0035] (2) Strong robustness and generalization ability: Through the region feature contrast learning mechanism in RFCL-Net, the model is forced to focus on the destruction of face region consistency rather than specific fake textures, thereby significantly enhancing the generalization robustness against unknown attack types and post-processing operations.

[0036] (3) Precise positioning capability: It can not only achieve high-precision binary classification of true and false, but also accurately locate the tampered area in the image during the inference stage based on the features learned by the anchor box prior, thereby improving the interpretability and practical value of the model.

[0037] (4) High-efficiency end-to-end detection: The end-to-end network built on the YOLOv11 backbone integrates target localization and forgery detection functions, ensuring detection speed and accuracy, and is suitable for deployment in practical application scenarios. Attached Figure Description

[0038] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention.

[0039] Figure 1 This is a schematic diagram of the overall process framework of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the processing flow of the face forgery anchor frame generation module (FFA-Gen) in this invention;

[0041] Figure 3 This is a schematic diagram illustrating the anchor frame generation principle based on multimodal difference fusion and DBSCAN clustering in this invention;

[0042] Figure 4 This is a schematic diagram of the overall model architecture of the face forgery detection network (RFCL-Net) that integrates region contrast learning in this invention;

[0043] Figure 5 This is a schematic diagram of the SFA spatial and channel frequency attention module in this invention;

[0044] Figure 6 This is a schematic diagram illustrating the principle and process of the BBox-Mask region pooling core module in this invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0047] Example 1:

[0048] This embodiment provides a face forgery detection method based on multimodal difference prior and region feature contrast learning. This method aims to address the problems of weak model localization ability and poor generalization caused by the lack of fine-grained region annotation in existing technologies.

[0049] Overall methodology and process: such as Figure 1 As shown, in one embodiment of this application, a face forgery detection method based on multimodal difference prior and regional feature contrast learning is provided. The overall process of this method includes three main stages: multimodal difference prior generation stage (S1), regional feature contrast learning network construction and training stage (S2), and reasoning and result visualization stage (S3).

[0050] S1: Multimodal differential prior generation stage

[0051] This stage forms the foundation of our method. Its core task is to utilize a specially designed Face Forgery Anchor Box Generation (FFA-Gen) module to perform in-depth analysis on pairs of real and forged face images. By calculating and fusing multimodal difference information, high-precision forged region bounding boxes are automatically generated and used as prior anchors. Finally, a training dataset containing the original images and their corresponding prior anchor box annotations is constructed, providing high-quality supervision signals for subsequent refined training of the model.

[0052] S2: Construction and Training Phase of Region Feature Comparison Learning Network

[0053] This stage is the core of our method. We construct a face forgery detection network (RFCL-Net) that integrates a region contrastive learning mechanism. This network is jointly trained end-to-end using a dataset with anchor box priors generated in stage S1. Through a specially designed mechanism, the network is forced to focus on the disruption of internal consistency within real face regions, rather than relying solely on specific forgery textures, thereby learning to distinguish between real and fake faces and extracting robust forgery trace features.

[0054] S3: Reasoning and Results Visualization Stage

[0055] This stage is the application objective of this method. The unknown image to be detected is input into the trained RFCL-Net network. The network will output the final binary classification result of true and false, and visually mark the detected face regions and precisely located forgery trace regions on the image, providing intuitive detection basis.

[0056] As the core of phase S1, the FFA-Gen module aims to automatically generate high-quality fake region annotations without human intervention. For example... Figure 2 As shown, the specific processing flow is as follows:

[0057] S1.1: Face Detection and Key Point Extraction

[0058] First, a pre-trained high-precision face detection model (such as RetinaFace) is used to locate the input image pair (real image). and fake images The face region is then identified. Next, dense facial keypoints (e.g., 468 keypoints) are extracted to capture the fine structure of the face. Based on the extracted keypoint set, its minimum and maximum values ​​in the horizontal and vertical directions are calculated to determine the initial bounding box of the face. To ensure the complete face region is included, a certain margin (e.g., 8%) is added to the initial bounding box to obtain the final face detection box (BBox). Simultaneously, based on these 468 keypoints, a convex hull mask that closely fits the face contour is constructed to strictly limit the effective area for difference calculations in subsequent steps, effectively eliminating interference from background noise and non-face regions.

[0059] S1.2: High-precision affine alignment

[0060] To accurately calculate subtle differences between two images, inconsistencies in pose and scale must be eliminated. Using keypoints extracted from both the real and fake images in step S1.1, several highly stable keypoints (such as the corners of the eyes, nose, and mouth) are selected as alignment reference points. The optimal affine transformation matrix is ​​calculated using robust estimation methods such as least squares or LMEDS. This matrix is ​​used to extract real human face images. The region of interest (ROI) is precisely transformed and aligned to the fake face image. In the ROI coordinate system, the aligned real image is obtained. .

[0061] S1.3: Multimodal difference map calculation: Under the constraint of the face convex hull mask, calculate the ROI of the aligned ground image separately. ROI with forged images Difference maps in three different modalities to capture different types of forgery traces:

[0062] (1) Spatial Difference: The absolute difference between two ROI images at the pixel level is calculated, and binarization is performed using an adaptive thresholding method to obtain the spatial difference map. This difference map primarily captures intuitive visual anomalies such as color, brightness, and contrast.

[0063] (2) Structural Difference: Calculate the SSIM index map of two ROI images using the Structural Similarity (SSIM) algorithm. Convert the SSIM index map into a difference map (e.g., through...). (or reverse thresholding) to obtain the domain difference map This difference map primarily captures disruptions in texture, edges, and structural consistency.

[0064] (3) Frequency Difference: Perform a two-dimensional Fast Fourier Transform (FFT) on each of the two ROI images to convert the image signals to the frequency domain. Calculate the amplitude spectrum of each image and obtain the absolute difference between them. To highlight high-frequency artifacts, perform high-pass filtering or center suppression on the difference spectrum, and then perform an inverse Fourier Transform to return it to the spatial domain to obtain the frequency difference map. This difference map primarily captures high-frequency anomalous signals such as periodic artifacts and checkerboard effects generated by the generative model.

[0065] S1.4: Difference Map Fusion and Anchor Frame Generation

[0066] like Figure 3 As shown, in order to comprehensively utilize the complementary information of different modes, the calculated spatial difference map is... Domain Difference Diagram Frequency domain difference diagram Normalization is performed. Then, a weighted fusion or logical operation (such as bitwise AND or OR) is used to combine the three into a comprehensive combined difference diagram. Next, the density-based spatial clustering algorithm (DBSCAN) was used to analyze the combined difference map. Cluster analysis is performed on highly significant response regions in the image. The DBSCAN algorithm can automatically discover forged regions of arbitrary shape without pre-specifying the number of clusters. For each high-response region obtained from clustering, its minimum bounding rectangle is calculated to generate an initial set of bounding boxes. Finally, a series of post-processing steps are used for filtering and refinement, including: removing noisy regions with excessively small areas, merging regions with high overlap, and ranking regions based on their response intensity and morphological characteristics. Ultimately, a set of bounding boxes that precisely enclose the forgery traces is determined, and these bounding boxes are mapped from the ROI coordinate system back to the original image's global coordinate system, serving as the forged anchor boxes (GT) prior for this forged image.

[0067] S1.5: Dataset Construction

[0068] Based on the above processing results, a standard dataset for subsequent network training is constructed. For each record in the dataset, the original real image is labeled as a negative sample (Label=0), and its overall face bounding box ground truth (GT) is recorded; the original fake image is labeled as a positive sample (Label=1), and its overall face bounding box GT and the fake anchor box prior GT generated by the FFA-Gen module are recorded. All annotation information is stored in standard XML format, corresponding one-to-one with the corresponding image file, thus completing the preparation of high-quality training data.

[0069] This embodiment innovatively achieves automated generation of high-precision regional annotations without manual costs by designing the FFA-Gen module and combining advanced facial landmark technology with multimodal difference analysis. This effectively solves the long-standing data shortage bottleneck in the field of deepfake detection and lays a solid foundation for building high-performance and robust detection models.

[0070] Example 2: Construction and Training of Region Feature Contrast Learning Network (RFCL-Net)

[0071] This embodiment should be understood to include all the features of any of the foregoing embodiments, and to further improve upon them, with a focus on explaining the network architecture, data loading and alignment strategies, and model training and verification details in step S2.

[0072] like Figure 4 As shown, the specific architectural features of the face forgery detection network (RFCL-Net) with fusion region contrast learning mechanism constructed in step S2 are as follows:

[0073] 1. Shared backbone network and feature extraction

[0074] RFCL-Net uses a pre-trained YOLOv11 backbone and neck network as shared feature extractors. For the input image to be detected... The backbone network performs multi-layer convolution and downsampling operations to output a multi-scale feature map sequence. In this embodiment, a high-level semantic feature map is extracted by registering a forward hook at a specific layer of the backbone network (e.g., the layer before the Detect head). Its downsampling step size relative to the original image is denoted as This feature map will be used by both the standard localization branch and the artifact detection branch.

[0075] 2. Standard positioning branch

[0076] This branch uses the standard detection head structure of YOLOv11 and is connected to the output of the shared backbone network. It is responsible for processing multi-scale feature maps and predicting the bounding box coordinates and confidence scores of face targets in the image. During the training phase, the loss function of this branch... It consists of bounding box regression (CIoU Loss), object classification loss (BCE Loss), and distribution focus loss (DFL Loss), and is used to supervise the accuracy of face region localization.

[0077] 3. The Artifact Detection Branch aims to distinguish between genuine and fake artifacts and learn forgery traces. It is connected in parallel to the extracted feature maps. The specific implementation logic is as follows:

[0078] 3.1 SFA Spatial and Channel Frequency Attention Module

[0079] like Figure 5 As shown, feature map First, the input is fed into a lightweight Spatial and Channel Frequency Attention (SFA) module. Channel descriptors are obtained through Global Average Pooling (GAP), followed by a two-layer MLP that includes dimensionality reduction and dimensionality increase (the intermediate layer uses SiLU activation, denoted as ). The nonlinear interaction between learning channels is activated by the Sigmoid activation function. Generate channel attention weights

[0080]

[0081] in This is the weight matrix of the MLP.

[0082] Then, mean pooling is performed on the input feature map along the channel axis. and max pooling The two obtained two-dimensional feature maps are concatenated (denoted as Concat). ), through a The convolutional layer is activated by Sigmoid to obtain spatial weights.

[0083]

[0084] Final enhanced feature map By element-wise multiplication get:

[0085]

[0086] Subsequently Through a Convolutional layers reduce the number of channels to a minimum. (e.g., 128) to obtain the projected feature map .

[0087] 3.2 BBox-Mask Region Pooling Core Module

[0088] like Figure 6 As shown, this module uses the prior annotation GT in the training data to accurately extract region features on the feature map.

[0089] (1) Coordinate mapping and mask generation: During training, the overall face boundary GT of the current image is read. ) and falsified anchor frame prior GT ( Divide these coordinates by the step size. Mapping to the feature map coordinate system. Based on the mapped coordinates, construct a binary mask on the two-dimensional plane. If the current training sample is a forged image: based on... Generate face mask ;based on Generate tamper-proof region mask (Take the intersection with the face mask). Generate a context mask through logical operations: &(~ ).

[0090] (2) Masked Mean Pooling: Using the generated binary mask (Right now or ), for feature maps Perform pooling to extract region feature vectors The calculation formula is as follows:

[0091]

[0092] in To prevent the use of tiny constants that divide by zero, this formula is used to obtain the feature vectors of the tampered regions. and contextual background region feature vectors .

[0093] 3.3 Feature Fusion and Classification Prediction

[0094] To capture the consistent differences between regions, and Perform interactive fusion to construct combined feature vectors :

[0095]

[0096] Will Input is fed into an MLP classification head containing multiple fully connected layers and Dropout, and the output is the true / false classification logits without Sigmoid activation. .

[0097] 4. Definition and Training of Joint Loss Function

[0098] RFCL-Net uses a multi-task joint loss function for end-to-end training:

[0099]

[0100] in , The hyperparameters for balancing the weights.

[0101] (1) True / False Classification Loss The prediction of the classification head is supervised using binary cross-entropy loss (BCEWithLogitsLoss). With image real label GT Consistency:

[0102]

[0103] (2) Regional feature contrast loss Cosine Embedding Loss is used to constrain the consistency of region features. For the input feature pairs... and the corresponding target tags (Real image is 1, fake image is -1):

[0104]

[0105] in The cosine similarity is used, and the margin is the boundary threshold (e.g., 0.0). This loss forces the network to bring the region feature distance closer for real samples and push the region feature distance further away for fake samples during training, thereby grasping the essential difference in region consistency between real and fake images.

[0106] Training strategy: Load pre-trained YOLOv11x weights into the backbone and randomly initialize artifact branches. Employ the AdamW optimizer, setting a high learning rate for the artifact branches. Combine learning rate warm-up and cosine annealing strategies for iterative optimization over 70 epochs.

[0107] During training, a custom validator is used to evaluate the model. In addition to the standard detection metric mAP, image-level forgery detection metrics are calculated. The predicted probabilities and ground truth labels of all images on the validation set are aggregated, and the area under the ROC curve (AUC) and accuracy (ACC) are calculated to comprehensively and quantitatively evaluate the model's performance.

[0108] Example 3: Visualization of Reasoning and Results

[0109] This embodiment should be understood to include all the features of any of the foregoing embodiments, and to further improve upon them, focusing on explaining the inference process of the RFCL-Net network in step S3 and the method for determining and visualizing the final detection results. Figure 1 As shown, after completing the prior generation in stage S1 and the network construction and training in stage S2, the parameter-optimized RFCL-Net model is obtained. This embodiment utilizes this trained model to perform real-time face forgery detection and localization on unknown images to be detected. The specific steps are as follows:

[0110] S3.1 Image Input and Preprocessing: Input the single RGB image to be detected.

[0111] The image is input to the inference module. First, necessary preprocessing operations are performed on the image, including uniformly adjusting the image size (e.g., resizing). ), pixel normalization (mapping pixel values ​​to) The inputs are processed using intervals and standardization (using the same mean and standard deviation as during training) to meet the input requirements of the RFCL-Net network.

[0112] S3.2 Network Forward Propagation and Two-Branch Output

[0113] The preprocessed image is input into the trained RFCL-Net network.

[0114] S3.2.1 Feature Extraction: The shared YOLOv11 backbone network and neck network perform multi-scale feature extraction on the input image, generating a sequence of intermediate layer feature maps containing rich semantic and spatial information.

[0115] S3.2.2 Parallel Branch Processing: The extracted feature maps are input into two functional branches in parallel.

[0116] (1) Standard localization branch: After processing the feature map, this branch outputs the predicted coordinates of the face bounding boxes of all face targets in the image. (Including center point coordinates, width and height) and their corresponding confidence scores. Then, the non-maximum suppression (NMS) algorithm is used to remove redundant detection boxes and retain high-confidence face detection results.

[0117] (2) Artifact Detection Branch: This branch uses the trained SFA attention module, BBox-Mask region pooling module, and MLP classification head to process the feature map. The final output is an unactivated Logits value. This is used to indicate the likelihood that the image is a fake face.

[0118] S3.3 True / False Category Determination

[0119] Logits value output by the artifact detection branch Perform Sigmoid activation function processing and map it to Interval probability score :

[0120]

[0121] Set a preset judgment threshold. (Typically, an empirical value of 0.5 is used). Compare the probability score with the threshold: If If so, the input image is determined to be a fake face image; If the input image is a real human face image, then it is determined that the input image is a real human face image.

[0122] 3.4 Results Visualization and Annotation Output Based on

[0123] The judgment result of S3.3 and the face bounding box information output by the standard localization branch of S3.2 are used to generate the final detection result image with visual annotations:

[0124] (1) Real face annotation: If the face is determined to be a real face, the face bounding box output by the standard localization branch is drawn on the original input image using the first striking color (green), and a "Real" label is marked above or near the box, along with the determination probability ( ).

[0125] (2) Forged Face Labeling: If a face is identified as forged, a second, more conspicuous color (red) different from the first color is used to draw the face bounding box output by the standard localization branch on the original input image. This bounding box not only indicates the face location but also more accurately defines the area containing forgery traces detected by the model. Similarly, a "Tampered" label is added above or near the bounding box, along with the probability of forgery. .

[0126] Through the above process, this embodiment not only achieves high-precision image-level binary classification and identification of true and false images, but also endows the model with region attention capabilities during the training phase based on anchor box priors, and accurately locates tampered regions in the image during the inference phase using the output of the standard localization branch. This visualized output intuitively demonstrates the model's detection basis, significantly improving the model's interpretability and practical value in scenarios such as content security auditing and identity authentication.

[0127] To fully verify the effectiveness of the face forgery detection method (RFCL-Net) based on multimodal difference prior and region feature contrast learning proposed in this invention, and to quantitatively evaluate the technical contributions of each key innovation module, we conducted comparative experiments on the benchmark dataset FaceForensics++ (FF++).

[0128] The experiments used a high-quality compressed version (c23) from the FF++ dataset, which is the most common quality level in internet videos and is highly representative of practical applications. The test set covers video frame images generated by four mainstream face spoofing methods: Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT), to comprehensively evaluate the model's generalization ability.

[0129] As shown in Table 1, the comparative experimental results show that in the image-level real / fake identification dimension, the complete model proposed in this paper (experiment d: RFCL-Net) achieved the highest values ​​in all comparative experiments in the three key indicators of AUC, ACC and F1-Score, which is a good improvement over YOLOv11-Baseline.

[0130] Specifically, its AUC reaches 0.984, ACC reaches 0.978, and F1-Score reaches 0.979. Compared with the baseline YOLOv11 model (Experiment a), the improvement is significant. The introduction of FFA-Gen prior guidance (Experiment b) brought initial improvement; while the introduction of the BBox-Mask region pooling module (Experiment c) resulted in a qualitative leap in all indicators, indicating that the extraction of region-level features is crucial for distinguishing between genuine and fake features; the addition of the SFA attention module (Experiment d) further fine-tuned and optimized the features, achieving an even better level. In the dimension of fake region localization, the baseline methods (Experiments a and b) cannot output accurate fake region localization results due to the lack of region modeling fake region localization capabilities. The introduction of the BBox-Mask module (Experiment c) gave the model the ability to perform region-level fine-grained analysis for the first time, with its mAP50 index reaching 0.952. On this basis, after adding the SFA module and the complete RFCL-Net joint training strategy (Experiment d), the localization accuracy was further improved to 0.967, proving that the model can accurately define the fake regions.

[0131] Table 1 Comparison Experiment of Network Models

[0132] method AUC ACC F1-Score mAP50 mAP50-95 Experiment a: YOLOv11-Baseline 0.957 0.951 0.955 / / Experiment b: YOLOv11 + FFA-Gen 0.962 0.958 0.959 / / Experiment c: YOLOv11 + FFA-Gen + BBox-Mask 0.973 0.967 0.97 0.952 0.832 Experiment d: YOLOv11 + FFA-Gen + RFCL-Net (Ours) 0.984 0.978 0.979 0.967 0.846

[0133] Parameter description:

[0134] For image-level real / fake identification, AUC, ACC, and F1-Score are used as evaluation metrics:

[0135] (1) AUC (Area Under Curve): The area under the receiver operating characteristic curve is a comprehensive indicator for evaluating the performance of a binary classification model. It has good robustness to imbalanced sample distribution. The closer the value is to 1, the better the model performance.

[0136] (2) ACC (Accuracy): Accuracy, which represents the proportion of samples correctly classified by the model out of the total samples, and intuitively reflects the overall judgment ability of the model;

[0137] (3) F1-Score: The harmonic mean of precision and recall, used to comprehensively measure the balance of the model’s recognition performance on positive samples (fake images) and negative samples (real images).

[0138] For the regional-level spoofed location dimension, mAP50 and mAP50-95 are used as evaluation indicators:

[0139] (4) mAP50 (mean Average Precision at IoU=0.50): A standard metric in the field of object detection. When the intersection-union ratio (IoU) between the predicted fake region bounding box and the real labeled box is greater than 0.50, it is considered a correct detection. This metric is used to quantitatively characterize the basic accuracy of the model in locating fake regions;

[0140] (5) mAP50-95 (mean Average Precision at IoU=0.50:0.95): The mean of the average precision is calculated within the range of IoU thresholds from 0.50 to 0.95 (with a step size of 0.05). Compared to mAP50 with a single threshold, mAP50-95 is a more rigorous and comprehensive indicator that can reflect the overall performance of the model under high-precision positioning requirements.

[0141] This embodiment introduces the FFA-Gen module to automatically generate high-precision region priors, and combines it with the BBox-Mask region pooling core module to achieve accurate separation and feature extraction of tampered regions from the context region on the feature map. Furthermore, it incorporates the SFA spatial and channel frequency attention modules to enhance subtle trace features, and forces the model to learn the essential differences in consistency between regions through the joint training strategy of RFCL-Net. This design not only greatly improves the model's accuracy in identifying various face forgeries, but also endows the model with the ability to accurately locate forged regions, achieving a comprehensive improvement in detection performance.

[0142] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A face forgery detection method based on multimodal difference prior and region feature contrast learning, characterized in that, The method includes the following main steps: S1: Multimodal difference prior generation stage: The real face image and fake face image are processed by the face fake anchor box generation module (FFA-Gen); this module performs high-precision alignment of image pairs based on dense key points, calculates and fuses multimodal difference maps in the spatial domain, structural domain and frequency domain respectively, and uses a clustering algorithm to automatically generate high-precision fake region bounding boxes based on the difference maps as fake anchor box priors; a training dataset containing the original image and its corresponding anchor box prior annotations is constructed; S2: Region Feature Contrast Learning Network Construction and Training Stage: Construct a face forgery detection network (RFCL-Net) that integrates a region contrast learning mechanism; the RFCL-Net uses a pre-trained convolutional neural network as a shared backbone network to extract multi-scale feature maps, including a standard localization branch for face region localization and an artifact detection branch for identifying forgery traces; the artifact detection branch uses the forgery anchor box prior generated in step S1, and accurately extracts the tampered region features and context background region features on the feature map through a specially designed region pooling module, and introduces a region feature contrast loss function to force the network to learn the inconsistency between the two; the network is jointly trained end-to-end by combining localization loss, real / fake classification loss and region feature contrast loss; S3: Reasoning and Result Visualization Stage: Input the image to be detected into the trained RFCL-Net; the standard localization branch outputs the predicted face bounding box; the artifact detection branch outputs the true and false classification results of the image; combining the bounding box information of the localization branch and the classification results of the artifact detection branch, the final binary classification judgment is output, and the detected face region and the forgery trace region are visually marked on the image.

2. The method as described in claim 1, characterized in that, The specific process of the FFA-Gen module in step S1 includes: S1.1 Face detection and key point extraction: The model is used to locate the face and extract dense key points, and a face convex hull mask is generated to limit the effective area; S1.2 High-precision affine alignment: Calculate the optimal affine transformation matrix based on dense key points to align the real face image to the coordinate system of the fake face image; S1.3 Multimodal difference map calculation: Under mask constraints, the pixel-level absolute difference map (spatial domain difference), structural similarity difference map (structural domain difference), and fast Fourier transform amplitude spectrum difference map (frequency domain difference) of the aligned image pairs are calculated respectively. S1.4 Difference Map Fusion and Anchor Box Generation: The three sets of difference maps are weighted and fused to obtain a combined difference map; the density-based spatial clustering algorithm (DBSCAN) is used to cluster highly significant regions, and the set of bounding boxes that accurately surround the forgery traces is fitted and selected as the forgery anchor box prior GT; S1.5 Dataset Construction: Label real images as negative samples and fake images as positive samples. Store the fake anchor box prior GT and the overall face bounding box GT in the annotation file to construct the standard dataset.

3. The method as described in claim 2, characterized in that, The specific architecture of RFCL-Net in step S2 is as follows: S2.1 Shared Backbone Network: A pre-trained YOLOv11 backbone and neck network are used as feature extractors to output a multi-scale intermediate layer feature map sequence; S2.2 Standard Localization Branch: Connected to the shared backbone output, it uses the standard YOLO detection head structure to process feature maps and predict the bounding box coordinates and confidence scores of face targets; S2.3 Forgery Detection Branch: Connected in parallel to the shared backbone output; First, it includes a Spatial and Channel Frequency Attention (SFA) module to enhance the feature response of forgery traces at specific frequencies and spatial locations; then it connects to the BBox-Mask region pooling core module to achieve region contrast learning; finally, it connects to the feature fusion module and the True / False Binary Classification Head (MLP) to output true / false category prediction values.

4. The method as described in claim 3, characterized in that, The SFA module in step S2.3 includes a parallel channel attention submodule and a spatial attention submodule; the channel attention submodule generates a channel attention weight map through global average pooling and multilayer perceptron, and uses the weight map to reweight the channels of the input feature map; The spatial attention submodule generates a spatial attention weight map by pooling and convolution along the channel axis using the maximum and average values, and uses this weight map to reweight the spatial location of the input feature map. Finally, the feature maps are enhanced by fusing the two sub-modules.

5. The face forgery detection method based on multimodal difference prior and region feature contrast learning as described in claim 3, characterized in that, The BBox-Mask region pooling core module in step S2.3 is key to realizing region contrast learning, and its operation process is as follows: (1) Coordinate mapping: Receive the feature map of the current level, as well as the face bounding box GT and the fake anchor box prior GT read from the training data; according to the downsampling ratio of the current feature map relative to the original image, map the coordinates of the face bounding box GT and the fake anchor box prior GT from the image space to the current feature space; (2) Region mask generation: Based on the mapped overall bounding box coordinates of the face, a binary face region mask is constructed on the feature map plane, where the pixel value within the face region is 1 and the rest is 0; based on the mapped prior coordinates of the fake anchor box, a binary tamper mask is constructed; the difference between the face region mask and the tamper mask is calculated through logical operations to generate a context mask representing the untampered area inside the face. (3) Masked feature pooling: Using the tampered region mask, the features in the corresponding region of the input feature map are subjected to masked mean pooling or region of interest alignment (ROI Align) operation to extract the tampered region feature vector of fixed dimension; similarly, using the context background region mask, the features in the corresponding region of the input feature map are subjected to the same pooling operation to extract the context background region feature vector of fixed dimension.

6. The method as described in claim 5, characterized in that, The feature fusion, classification prediction, and loss function definitions in step S2.3 are as follows: (1) Feature fusion: Input the feature vectors of the tampered region and the context region into the fusion module, and generate the fused region comparison feature vector by splicing, absolute value of difference or element-wise multiplication. (2) Classification prediction: The fused feature vector is input into the true / false binary classification head (MLP) and mapped to the classification prediction Logits value; (3) Definition of joint loss: The total loss function is the weighted sum of localization loss, real / fake classification loss and region feature comparison loss; the region feature comparison loss is used to constrain the network to maximize the similarity between the tampered and context region features when processing real samples, and to minimize the similarity when processing fake samples.

7. The method as described in claim 1, characterized in that, The specific process of step S3 includes: inputting the image to be detected into the trained RFCL-Net; the standard localization branch outputs the face prediction bounding box; the artifact detection branch outputs the forgery probability score; determining the final real / fake category based on the score; if it is real, drawing the face bounding box with the first striking color (green) and labeling it "Real"; if it is fake, drawing the face bounding box with the second striking color (red) to define the fake area and labeling it "Tampered", and outputting the visualization result image.

8. The method as described in claim 6, characterized in that, In step S2.3, the joint loss definition uses a cosine embedding loss function to calculate the region feature comparison loss. This loss function receives feature vector pairs of the tampered region and the context region, combined with real label constraints: when the input is a real face, the optimization objective is to make the cosine similarity between feature vectors approach 1; when the input is a fake face, the optimization objective is to make the cosine similarity lower than a preset boundary threshold.

9. The method as described in claim 1, characterized in that, The network joint training in step S2 adopts the following strategy: a custom data loader is built to synchronously read images and annotation files, ensuring spatial alignment and performing data augmentation and normalization; a transfer learning strategy is adopted, initializing the shared backbone parameters as pre-trained weights and randomly initializing the artifact detection branch parameters; the AdamW or SGD optimizer is selected, and the learning rate is dynamically adjusted by combining learning rate warm-up and cosine annealing strategies.

Images