Face forgery detection model training method and face forgery identification method based on multi-dimensional feature fusion and contrast learning

Abstract

A training method and a face forgery detection method of a face forgery detection model based on multi-dimensional feature fusion and contrast learning are disclosed. The training method trains a model comprising a preprocessing, feature extraction, multi-dimensional feature fusion and classification module. First, a video sample set is obtained, and the face regions of consecutive frames are spliced into a space-time planar image. Then, initial space-time features that fuse time correlation and local spatial details are extracted based on the image, and frequency domain feature enhancement is performed on the initial space-time features to obtain frequency enhanced features. A prediction result is output based on the frequency enhanced features, and a classification loss is constructed. The training method also performs feature interpolation data enhancement on the initial space-time features of the forgery samples to generate enhanced forgery features, and constructs a supervised contrast loss in combination with the features of the real samples. The classification loss and the contrast loss are weighted and fused to obtain a total loss, and the learnable parameters of the model are updated accordingly. The detection accuracy and robustness are improved through multi-dimensional feature fusion and forgery enhancement contrast learning.

Description

Technical Field

[0001] This disclosure relates to the fields of deep learning and computer vision technology, specifically to a face forgery detection method, face authentication method, and electronic device based on multi-dimensional feature interaction and forgery enhancement contrast learning, which can be used to detect forged face video content generated based on deepfake and other technologies. Background Technology

[0002] With the development of artificial intelligence technology, especially the widespread application of the internet and digital products, the acquisition, editing, and transmission of visual media materials such as videos and images have become more convenient. However, the resulting security problems, such as identity forgery, are also becoming increasingly serious. In particular, the level of simulation of fake facial images and videos generated using artificial intelligence technologies such as deep learning is becoming increasingly sophisticated, making it difficult for ordinary people to distinguish between genuine and fake information, thus posing a significant threat to social trust and security. Therefore, developing effective facial forgery detection technology to address these risks is particularly important.

[0003] In forgery detection, the core challenge lies in identifying the subtle clues that distinguish synthetic images from real ones. Since most forgery algorithms typically operate frame-by-frame, the resulting videos often exhibit spatial and temporal traces of forgery. For example, facial features may display unnatural details or inconsistencies between different frames.

[0004] Existing detection methods often focus on extracting spatial features from a single frame, while neglecting the continuity and dynamic changes within the sequence.

[0005] To address this issue, some studies have introduced spatiotemporal neural networks to detect temporal inconsistencies in videos. While these methods have made some progress in modeling temporal inconsistencies, they still overemphasize the extraction of spatial features and fail to fully capture temporal dependencies during local feature learning, thus limiting the full utilization of multidimensional feature representations.

[0006] Recently, the rise of the Mamba model has spurred the emergence of corresponding face forgery detection methods. This model extracts local features by partitioning a single frame image and scanning each partition using a specific structured state-space sequence model. However, the Mamba model does not explicitly consider temporal relationships during local feature extraction, instead relying on subsequent global temporal modeling to handle temporal information. This approach ignores the temporal dependencies between local features, potentially leading to insufficient detection of subtle inconsistencies in forged videos. Summary of the Invention

[0007] To address the shortcomings of existing face forgery detection models, such as insufficient capture of subtle forgery traces, poor generalization ability, and failure to fully utilize multi-dimensional spatiotemporal and frequency domain information, this disclosure proposes a method based on multi-dimensional feature interaction and forgery enhancement contrastive learning. Specifically, MambaVision is used for initial spatiotemporal feature extraction. Based on this, spatial structure and frequency features are jointly modeled to capture complex correlations across time, space, and channel dimensions, improving sensitivity to subtle manipulations. Furthermore, this disclosure introduces a forgery enhancement contrastive learning strategy, using spherical interpolation to enhance forgery features and contrastive learning to emphasize fine-grained forgery traces and temporal differences, further improving robustness.

[0008] This disclosure provides a training method for a face forgery detection model. The face forgery detection model includes a preprocessing module, a feature extraction module, a multi-dimensional feature fusion module, and a classification module. The training method includes: acquiring a video sample set, which includes real samples and forged samples with category labels; the preprocessing module is configured to: for each sample in the video sample set, select a predetermined number of consecutive frames, extract a face region from each of the multiple consecutive frames, and spatially stitch the face regions in the multiple consecutive frames in a predetermined order to obtain a spatiotemporal plane image; the feature extraction module is configured to: extract initial spatiotemporal features based on the spatiotemporal plane image, wherein the initial spatiotemporal features are obtained by combining a first feature used to characterize the correlation of face regions in the spatiotemporal plane image in the time dimension and a feature used to characterize the spatiotemporal plane image. The second feature of local spatial details of the image is obtained by fusing; the multi-dimensional feature fusion module is configured to: perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features; the classification module is configured to: perform video sample classification prediction based on the frequency-enhanced features and output the prediction result; construct a classification loss based on the prediction result and the category label; perform feature interpolation data enhancement on the initial spatiotemporal features of the fake sample to generate enhanced fake features, and construct a contrastive loss based on the category label and the feature set composed of the enhanced fake features and the initial spatiotemporal features of the real sample using a supervised contrastive learning method; weightedly fuse the classification loss and the contrastive loss to obtain a total loss, and update the learnable parameters in the feature extraction module, the multi-dimensional feature fusion module and the classification module based on the total loss.

[0009] According to one or more embodiments of this disclosure, the preprocessing module is configured to perform the following steps to obtain the spatiotemporal plane image, including: dividing video samples in the video sample set into video segments of predetermined length in chronological order, randomly selecting a start frame from each video segment, and selecting a predetermined number of consecutive frames starting from the start frame; for each of the multiple consecutive frames, performing face bounding box detection, selecting the face bounding box with the largest area from the frame, and cropping the frame based on the face bounding box with the largest area to obtain the face region; scaling the face region to a predetermined size, and stitching the face regions corresponding to the multiple consecutive frames into a two-dimensional grid layout in chronological order to construct the spatiotemporal plane image.

[0010] According to one or more embodiments of this disclosure, the feature extraction module is configured to perform the following steps to extract initial spatiotemporal features based on the spatiotemporal plane image, including: inputting the spatiotemporal plane image into a first branch and a second branch respectively; in the first branch, sequentially performing a linear transformation for channel dimensionality reduction, convolution operation, and nonlinear activation on the spatiotemporal plane image, and performing a sequential scan on the nonlinearly activated features through a scan operator in the state space model to obtain the first feature; in the second branch, sequentially performing a linear transformation for channel dimensionality reduction, convolution operation, and nonlinear activation on the spatiotemporal plane image to obtain the second feature; concatenating the first feature and the second feature through channels, and projecting the concatenation result through a linear layer to obtain the initial spatiotemporal features; wherein the learnable parameters in the feature extraction module include at least: weight parameters for linear transformation and convolution kernel parameters for convolution operation in the first branch and the second branch, parameters of the state space model for implementing sequential scanning in the first branch, and weight parameters of the linear layer for projecting the concatenation result.

[0011] According to one or more embodiments of this disclosure, the training method further includes a step of normalizing the initial spatiotemporal features before performing feature interpolation data augmentation. The normalization step includes: performing average pooling on the initial spatiotemporal features of the real samples extracted by the feature extraction module and performing L2 normalization to obtain a normalized feature set mapped to a unit hypersphere, the normalized feature set including normalized real features and normalized fake features; and wherein the feature interpolation data augmentation step includes: constructing a first augmented fake feature set as the augmented fake features based on the normalized fake features. The first and second enhanced forgery feature sets are constructed by combining the following steps: using each feature in the normalized forgery features as the current feature, randomly selecting another feature different from the current feature from the normalized forgery features as a pairing feature, and randomly sampling interpolation coefficients from a preset distribution; performing spherical linear interpolation on the unit hypersphere based on the current feature, the pairing feature, and the interpolation coefficients to generate corresponding enhanced forgery features; the number of enhanced forgery features in both the first and second enhanced forgery feature sets is consistent with the number of forgery samples.

[0012] According to one or more embodiments of this disclosure, the step of constructing the contrastive loss includes: constructing a first linear projection head and a second linear projection head that are independent of each other, both of which have learnable parameters; combining the first enhanced fake feature set with the normalized real features to construct a first combined feature set, and combining the second enhanced fake feature set with the normalized real features to construct a second combined feature set; inputting the first combined feature set into the first linear projection head to obtain a first embedded feature set, and inputting the second combined feature set into the second linear projection head to obtain a second embedded feature set; calculating the cosine similarity between the features of the i-th sample in the first embedded feature set and the features of the j-th sample in the second embedded feature set to construct a similarity matrix between samples; constructing a supervision mask matrix based on the class labels, wherein when the class labels of the i-th sample and the j-th sample are the same, the elements in the supervision mask matrix are... The value is 1 if the total loss is 1 otherwise 0; the contrast loss is constructed based on the difference between the similarity matrix and the supervision mask matrix, and the training method further includes: updating the learnable parameters of the first linear projection head and the second linear projection head based on the total loss.

[0013] According to one or more embodiments of this disclosure, the multidimensional feature fusion module is configured to perform the following steps to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features, including: performing a two-dimensional fast Fourier transform on each channel of the initial spatiotemporal features along the spatial dimension to obtain real and imaginary features of the frequency domain features; concatenating the real and imaginary features and sequentially performing pointwise convolution, batch normalization, and nonlinear activation processing to obtain feature-modulated frequency domain features; performing a two-dimensional inverse fast Fourier transform on the feature-modulated frequency domain features to obtain reconstructed spatial domain features, and adding the reconstructed spatial domain features to the initial spatiotemporal features by residual addition to obtain the frequency-enhanced features.

[0014] According to one or more embodiments of this disclosure, the classification module is configured to perform the following steps to perform video sample classification prediction based on the frequency enhancement features and output a prediction result, including: inputting the frequency enhancement features into a convolutional layer to compress the channel dimension to obtain intermediate features; performing average pooling on the intermediate features to obtain downsampled features with reduced spatial resolution; flattening the intermediate features and the downsampled features respectively and inputting them into a multi-head attention mechanism, wherein the intermediate features serve as a query and the downsampled features serve as a key and value, to obtain attention enhancement features; inputting the attention enhancement features into a pointwise convolutional layer to restore the channel dimension to obtain output features; and performing global average pooling on the output features and inputting the result into a classification layer to obtain the prediction result.

[0015] According to one or more embodiments of this disclosure, constructing a classification loss based on the prediction result and the category label includes: comparing the prediction result output by the classification module with the corresponding category label, and calculating the classification loss using a cross-entropy loss function.

[0016] This disclosure also provides a face spoofing detection method, which includes: acquiring a video sample to be detected; inputting the video sample to be detected into a face spoofing detection model trained by the training method of the face spoofing detection model described in any of the above embodiments of this disclosure, wherein the face spoofing detection model includes a preprocessing module, a feature extraction module, a multi-dimensional feature fusion module, and a classification module, wherein the preprocessing module is configured to: select a predetermined number of consecutive frames from the video sample to be detected, extract face regions from each of the multiple consecutive frames, and sort the face regions in the multiple consecutive frames in a predetermined order. Spatial stitching is performed to obtain a spatiotemporal plane image; the feature extraction module is configured to extract initial spatiotemporal features based on the spatiotemporal plane image, the initial spatiotemporal features being obtained by fusing a first feature characterizing the correlation of face regions in the spatiotemporal plane image in the temporal dimension and a second feature characterizing the local spatial details of the spatiotemporal plane image; the multidimensional feature fusion module is configured to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features; the classification module is configured to perform video sample classification prediction based on the frequency-enhanced features and output the prediction results.

[0017] This disclosure also provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the training method for the face forgery detection model as described above, or to implement the face forgery authentication method as described above.

[0018] This application employs MambaVision as the backbone network for feature extraction and combines it with forgery enhancement contrastive learning and a multi-dimensional feature interaction strategy, achieving significant beneficial effects. First, the forgery enhancement contrastive learning method, through a carefully designed contrastive loss function, effectively widens the distance between real and forged face features, thereby enhancing inter-class separability and intra-class compactness. Furthermore, contrastive learning emphasizes fine-grained forgery traces and temporal differences, significantly improving the model's robustness. Second, the multi-dimensional feature interaction strategy, by jointly modeling complementary information in the frequency and spatial domains, captures complex correlations across temporal, spatial, and channel dimensions, enabling the extraction of deep feature representations and improving sensitivity to subtle forgery operations. In summary, this invention achieves high-precision classification and detection of forged videos (such as deepfake face images and videos) through the full utilization and enhancement of multi-dimensional features. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0020] Figure 1 This is a schematic diagram of a training method for a face forgery detection model according to an embodiment of the present disclosure.

[0021] Figure 2 This is a schematic diagram of the structure of a face forgery detection model according to an embodiment of the present disclosure.

[0022] Figure 3 This is a schematic diagram of the spatiotemporal planar layout constructed according to an embodiment of the present disclosure.

[0023] Figure 4 This is a schematic diagram of the backbone of MambaVision according to an embodiment of the present disclosure.

[0024] Figure 5 This is a schematic diagram of a multi-dimensional feature interaction strategy according to an embodiment of the present disclosure.

[0025] Figure 6 This is a schematic diagram of spoofing-enhanced contrastive learning according to an embodiment of the present disclosure.

[0026] Figure 7 This is a block diagram of a computing device according to embodiments of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0028] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0029] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0030] Figure 1 This is a schematic diagram of the training method for the face forgery detection model provided in the embodiments of this disclosure. Figure 2 This is a schematic diagram of the network model structure disclosed herein.

[0031] like Figure 1 As shown, the training method mainly includes steps S100 to S800. It should be noted that the method described in this embodiment is used to train a face forgery detection model, which mainly includes a preprocessing module, a feature extraction module, a multi-dimensional feature fusion module, and a classification module.

[0032] In step S100, a video sample set is obtained, which includes real samples and fake samples with category labels. The video sample set (input video) is used as input data for the model. Each sample in the video sample set is labeled with a category label indicating whether it is a real face or a fake face (real category label).

[0033] In step S200, the preprocessing module is configured to: select a predetermined number of consecutive frames for each sample in the video sample set, extract face regions from each of the multiple consecutive frames, and spatially stitch the face regions in the multiple consecutive frames in a predetermined order to obtain a spatiotemporal plane image.

[0034] The preprocessing module prepares the data. Specifically, it arranges and combines facial regions extracted from multiple consecutive frames of a video clip in a specific way to construct a spatiotemporal plane image (a specific spatiotemporal plane layout / thumbnail layout) containing time-series information. The specific process of selecting consecutive frames, extracting faces, and constructing the spatiotemporal plane image in step S200 will be explained later. Figure 3 A more detailed description follows.

[0035] In step S300, the feature extraction module is configured to extract initial spatiotemporal features based on the spatiotemporal plane image. These initial spatiotemporal features are obtained by fusing a first feature characterizing the temporal correlation of face regions in the spatiotemporal plane image with a second feature characterizing local spatial details of the spatiotemporal plane image. In this embodiment, MambaVision can be used to process the input data. This module extracts features through a dual-branch structure: one branch uses a state-space model to extract the first feature characterizing temporal correlation (including branch features from the scanning process), and the other branch uses convolution operations to extract the second feature characterizing local spatial details (excluding branch features from the scanning process). The two are then fused to obtain the initial spatiotemporal features (preliminary spatiotemporal features / multidimensional feature representation fusing local convolution and state scanning information). The internal structure of the feature extraction module and the specific process of feature fusion in step S300 will be discussed later. Figure 4 A more detailed description follows.

[0036] In step S400, the multi-dimensional feature fusion module is configured to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features. In this embodiment, to capture complex correlations across channels and the frequency domain, frequency domain feature enhancement processing (the frequency domain processing part of the multi-dimensional feature interaction strategy) is performed on the initial spatiotemporal features obtained in the previous step. This includes projecting the features into the frequency domain (e.g., using the FFT-2D method), performing nonlinear modulation, restoring them to the spatial domain, and performing residual concatenation with the original features to generate frequency-enhanced features. The specific steps for frequency domain feature enhancement in step S400 will be discussed later. Figure 5 A more detailed description follows.

[0037] In step S500, the classification module is configured to perform video sample classification prediction based on frequency enhancement features and output the prediction result. In this embodiment, the classification module further performs feature compression and global attention modeling (global region feature compression and global region attention modeling) on ​​the frequency enhancement features, and finally outputs the prediction result representing the probability of the sample being true or false. The specific structure of the classification module will be discussed later. Figure 5 A more detailed description follows.

[0038] In step S600, a classification loss is constructed based on the prediction results and category labels. In this embodiment, the prediction results output by the classification module are compared with the corresponding category labels to calculate the classification loss (cross-entropy loss). This loss function is used to measure the accuracy of the model's predictions. (See below for more details.) Figure 5 Step S600 will be described in further detail.

[0039] In step S700, feature interpolation data augmentation is performed on the initial spatiotemporal features of the fake sample to generate enhanced fake features. A supervised contrastive learning method is then used to construct a contrastive loss based on the category label and the feature set consisting of the enhanced fake features and the initial spatiotemporal features of the real sample. In this embodiment, a specific feature interpolation data augmentation method (fake enhancement contrastive learning method) is introduced to improve the robustness of the model. It should be noted that, in this process, the original fake sample features are replaced by the newly generated enhanced fake features (interpolated enhanced features). and The features, along with the initial spatiotemporal features of the real samples (normalized real features), constitute the feature set used for contrastive learning. By calculating the similarity between samples in this feature set and combining it with a supervised mask, a contrastive loss (spoofing enhancement contrastive loss) is constructed. The specific process for feature interpolation, linear projection head setup, and contrast loss construction in step S700 will be discussed later. Figure 6 A more detailed description follows.

[0040] In step S800, the classification loss and contrastive loss are weighted and fused to obtain the total loss, and the learnable parameters in the feature extraction module, the multi-dimensional feature fusion module, and the classification module are updated based on the total loss. In this step, the classification loss and contrastive loss are combined by weighted fusion to calculate the total loss ( ). See the following formula for details:

[0041]

[0042] in, This is a tradeoff coefficient. It is used to balance the relative influence of the classification supervision term and the contrast regularization term. By dynamically adjusting this hyperparameter, an optimal balance can be achieved between accuracy and robustness. The default value is... The value is set to 0.005. Based on this total loss, the learnable parameters in the model are updated using the backpropagation algorithm. These learnable parameters include not only the parameters from the feature extraction module, the multidimensional feature fusion module, and the classification module, but also the parameters of the linear projection head used for contrastive learning in step S700. This step ensures that the model can learn accurate classification capabilities during training, while also enhancing its sensitivity to fine-grained forgery traces through contrastive learning.

[0043] Figure 3 This is a schematic diagram of the spatiotemporal planar layout constructed according to an embodiment of the present disclosure.

[0044] Referring to Figure 3, the process of "obtaining a spatiotemporal plane image (a specific spatiotemporal plane layout)" described in step S200 may include sub-steps S210 to S230.

[0045] In step S210, the preprocessing module processes the video samples (input video) from the video sample set. ,in It's the frame rate of the video. It is the number of channels. and The video segments (those consisting of the height and width of the frame) are divided into video segments of predetermined lengths in chronological order. The preprocessing module randomly selects a start frame from each video segment and then selects the predetermined number of segments starting from that start frame. Multiple consecutive frames (continuous) (Frame / Clip).

[0046] As an example, the video can be divided into... (The default setting is 8) equal video clips, each with a length of [length missing]. This facilitates subsequent processing. Within each video segment, consecutive segments can be randomly selected. Frames (4 frames by default) are used to create multiple consecutive frames (a clip) to capture the temporal sequence information of the video.

[0047] In step S220, for each frame of each clip, face bounding box detection (face detection) is performed, the largest face bounding box (maximum area face bounding box) is selected from the frame, and the frame is cropped based on the largest face bounding box to obtain the face region.

[0048] In a specific implementation, the MTCNN algorithm can be used to perform face bounding box detection (face detection) on each frame. The specific method is to extract only the face bounding box with the largest area (maximum area face bounding box) and add 30% boundary space on each side, thereby removing background interference and focusing on the face region (facial features).

[0049] In step S230, the face region is scaled to a predetermined size, and the face regions corresponding to the multiple consecutive frames (a clip) are stitched together in chronological order into a two-dimensional grid layout to construct the spatiotemporal plane image (a specific spatiotemporal plane layout / initial input).

[0050] In a specific embodiment, the continuous data obtained from each clip can be... The frames are scaled and arranged in left-to-right order. of Thumbnail layout. Each sub-image (i.e., the scaled-up face area) is sized as follows: This ensures that the spatiotemporal information of frames is preserved in a single layout.

[0051] Figure 4 This is a schematic diagram of the backbone of MambaVision according to an embodiment of the present disclosure.

[0052] Reference Figure 4 The process of extracting initial spatiotemporal features based on the spatiotemporal plane image described in step S300 can be implemented by a feature extraction module (e.g., MambaVision). This module adopts a dual-branch structure to take into account both long-range dependencies and local features, and may include sub-steps S310 to S340.

[0053] In step S310, the feature extraction module extracts the spatiotemporal plane image (the input feature) Enter the first branch and the second branch respectively.

[0054] In step S320, in the first branch, the spatiotemporal plane image (input features) is processed. ) sequentially perform the linear transformation (Linear() for channel dimensionality reduction) Convolution operation Conv1D ( and nonlinear activation (e.g., ReLU), and through the scan operator SSM in the state-space model ( The first feature is obtained by sequentially scanning the features after nonlinear activation. .

[0055] In a specific embodiment, the processing of the first branch can be represented by the following equation (1):

[0056]

[0057] in, This indicates that the number of input channels will be reduced from... Mapped to The linear transformation is described above. It is worth noting that, to adapt to visual tasks, regular convolutions are used here instead of the causal convolutions in the original Mamba mixer, because causal convolutions restrict the influence to one direction, which is unnecessary and limiting for visual tasks. The SSM is a scan operator in the state-space model, specifically designed to capture long-range dependencies in sequences.

[0058] Meanwhile, in step S330, in the second branch, the spatiotemporal plane image (input features) is processed. ) sequentially perform the linear transformation (Linear() for channel dimensionality reduction) Convolution operation Conv1D ( and nonlinear activation (e.g., ReLU) to obtain the second feature. In a specific embodiment, the second branch does not include the SSM scanning process, and its processing can be represented by the following equation (2):

[0059]

[0060] The purpose of this branch is to preserve local spatial features (fine-grained local features) to compensate for the content that may be lost in the first branch due to the sequential constraints of the state-space model.

[0061] In step S340, the final step of feature fusion and projection is to combine the first feature... Second feature Perform channel splicing ( The splicing result is then projected through a linear layer (e.g., a linear upsampling layer) to obtain the initial spatiotemporal features (such as...). Figure 4 The multidimensional feature representation shown is a fusion of local convolution and state scan information. In a specific embodiment, this combination ensures that the final feature representation integrates sequence information and spatial information, while taking advantage of the advantages of both branches, as shown in equation (3) below:

[0062]

[0063] Furthermore, in the above process, the learnable parameters in the feature extraction module include at least: weight parameters for linear transformation and convolution kernel parameters for convolution operations in the first and second branches, parameters for the state space model used to implement sequence scanning in the first branch, and weight parameters for the linear layer used to project the concatenation result. These parameters will be updated based on the total loss in subsequent training steps (S800).

[0064] Figure 5 This is a schematic diagram of a multi-dimensional feature interaction strategy according to an embodiment of the present disclosure.

[0065] Please refer to Figure 5 This strategy mainly covers the frequency domain feature enhancement processing in step S400 (steps S410 to S430) and the video sample classification and prediction process in step S500 (steps S510 to S530). This process aims to jointly model spatial structure and frequency features to capture complex cross-dimensional correlations.

[0066] Reference Figure 1 and Figure 5 In step S410, the multidimensional feature fusion module processes the initial spatiotemporal features (input features) Each channel of the frequency domain is subjected to a two-dimensional fast Fourier transform along the spatial dimension to obtain the real and imaginary parts of the frequency domain features (real part). With the imaginary part ).

[0067] In a specific embodiment, the process can be represented by the following equation (4):

[0068]

[0069] in, This indicates a two-dimensional real-valued fast Fourier transform (FFT-2D) operation performed independently for each channel.

[0070] In step S420, the real part feature is... and the characteristics of the imaginary part The data is then concatenated (e.g., channel concatenation) and sequentially processed through pointwise convolution PConv, batch normalization (BN), and nonlinear activation ReLU to obtain the frequency domain features modulated by the features. ).

[0071] In a specific embodiment, the specific calculation of the frequency domain features after feature modulation is shown in equation (5):

[0072]

[0073] in, This indicates a channel splicing operation. This represents the ReLU activation function.

[0074] In step S430, the frequency domain features modulated by the feature are ( and Performing a two-dimensional inverse fast Fourier transform (two-dimensional real-valued inverse Fourier transform / inverse FFT operation Inv FFT-2D) yields the reconstructed spatial domain features. ), and the reconstructed spatial domain features ( ) and initial spatiotemporal characteristics Perform residual summation (residual concatenation) to obtain the frequency enhancement feature. .

[0075] In a specific embodiment, this step preserves the spatial structure while incorporating frequency domain forgery clues, as shown in equation (6):

[0076]

[0077] Then, continue to refer to Figure 1 and Figure 5 In step S510, the classification module will enhance the frequency features. Input convolutional layers compress the channel dimensions to obtain intermediate features. Meanwhile, for intermediate features Performing average pooling yields downsampled features with reduced spatial resolution. .

[0078] In a specific embodiment, let the kernel size be... Step size is To reduce channel dimensions and generate intermediate features Then, an average pooling operation is performed, with both the pooling kernel size and stride being [value missing]. Thus, the downsampling features are obtained. .

[0079] In step S520, the intermediate features are... and downsampling features Each feature is flattened and input into a multi-head attention mechanism (MHA). The intermediate features are used as the query (…). ), downsampled features as keys and values ​​( To obtain attention-enhancing features .

[0080] In a specific embodiment, this step aims to establish long-range dependencies (global interactions) across regions between high-resolution and low-resolution features, as shown in equation (7):

[0081]

[0082] In step S530, attention enhancement features are... Inputting a pointwise convolutional layer to restore the channel dimension (original channel dimension) yields the output features. ; and perform global average pooling (GAP) on the output features, and input the result into the classification layer (not shown) to obtain the prediction result.

[0083] In a specific embodiment, the obtained prediction results will be used to subsequently calculate the classification loss. (i.e., step S600), and combined with the aforementioned contrastive loss to optimize the model. For example, the prediction results output by the classification module are compared with the corresponding class labels, and the classification loss is calculated using the cross-entropy loss function.

[0084] Figure 6 This is a schematic diagram of spoofing-enhanced contrastive learning according to an embodiment of the present disclosure.

[0085] Please refer to Figure 6. This part mainly corresponds to the aforementioned step S700, which aims to improve the robustness of the model through feature interpolation data augmentation and supervised contrastive learning. It may include sub-steps S710 to S770.

[0086] In step S710, the training method includes a step of normalizing the initial spatiotemporal features before performing feature interpolation data augmentation. Specifically, the initial spatiotemporal features of the real samples extracted by the feature extraction module and the initial spatiotemporal features of the fake samples are subjected to average pooling and L2 normalization to obtain a normalized feature set mapped to a unit hypersphere.

[0087] In a specific embodiment, the normalized feature set includes normalized real features and normalized fake features (feature representations on a unit sphere). This prepares for subsequent spherical interpolation.

[0088] In step S720, a feature interpolation data augmentation step is performed. This step constructs a first set of enhanced forgery features as the enhanced forgery features based on the normalized forgery features. Second enhanced forgery feature set The process of constructing both sets includes: using normalized forged features. Each feature in the data is used as the current feature. Randomly select another feature different from the current feature from the normalized forged features as the paired feature. Interpolation coefficients are randomly sampled from a preset distribution. Based on current features Pairing features and interpolation coefficients Spherical linear interpolation is performed on the unit hypersphere to generate the corresponding enhanced forgery features.

[0089] In a specific embodiment, the interpolation coefficients t are uniformly distributed from the interval. Sampling. The specific calculation of spherical linear interpolation (Slerp) is shown in equation (8):

[0090]

[0091] Among them, angle Because the features are L2 normalized, the generated enhanced features are located on the spherical arc between the two fake samples, effectively filling the gaps in the fake feature space.

[0092] It should be noted that the original forged samples will be replaced by these enhanced samples. That is, the number of enhanced forged features in the first and second enhanced forged feature sets is the same as the number of forged samples.

[0093] In step S730, an independent first linear projection head is constructed ( ) and second linear projection head ( Both have learnable parameters. The first enhanced forgery feature set... Combined with normalized true features to construct the first combined feature set The second enhanced forgery feature set Combined with normalized true features to construct a second combined feature set .

[0094] The first combined feature set is input into the first linear projection head to obtain the first embedded feature set. The second combined feature set is then input into the second linear projection head to obtain the second embedded feature set. .

[0095] In a specific embodiment, the projection process is as shown in equation (9):

[0096]

[0097] in, It refers to a combination of real features and enhanced forgery features.

[0098] In step S740, the first embedded feature set is calculated. The features of the sample and the second embedded feature set in the second set of features The cosine similarity between features of each sample is used to construct a similarity matrix between samples. ).

[0099] In a specific embodiment, the similarity matrix The calculation is shown in equation (10):

[0100]

[0101] in, It is a square matrix with a size equal to the number of samples, and its values ​​range from -1 (i.e., perfectly negative correlation) to 1 (i.e., perfectly positive correlation).

[0102] In step S750, a supervision mask matrix M is constructed based on the category labels. Wherein, when the... The sample and the first When all samples have the same class label, the elements in the supervision mask matrix M... It is 1 if it is true, otherwise it is 0.

[0103] In a specific embodiment, the supervision mask is defined as shown in equation (11):

[0104]

[0105] Finally, a contrastive loss is constructed based on the difference between the similarity matrix C and the supervision mask matrix M. .

[0106] In a specific embodiment, the fake enhancement contrast loss function is defined as shown in equation (12):

[0107]

[0108] This loss function aims to constrain the feature similarity of samples of the same class to approach 1, while suppressing the similarity of samples of different classes. Based on this loss, the learnable parameters of the first and second linear projectors can be updated (e.g., through backpropagation). For example, these parameters will be updated based on the total loss in subsequent training steps (S800).

[0109] Figure 7 This is a block diagram of a computing device according to embodiments of the present disclosure.

[0110] Reference Figure 7 The computing device 700 according to embodiments of the present disclosure may include a processor 710 and a memory 720. The processor 710 may include (but is not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a microprocessor, an application-specific integrated circuit (ASIC), etc. The memory 720 may store computer programs to be executed by the processor 710. The memory 720 includes high-speed random access memory and / or a non-volatile computer-readable storage medium. When the processor 710 executes the computer program stored in the memory 720, the training method for the face forgery detection model or the face authentication method described above can be implemented.

[0111] Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and to provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. In one example, the computer programs and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0112] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A training method for a face forgery detection model based on multi-dimensional feature fusion and contrastive learning, characterized in that, The face forgery detection model includes a preprocessing module, a feature extraction module, a multi-dimensional feature fusion module, and a classification module. The training method includes: Obtain a video sample set, which includes real samples and fake samples with category labels; The preprocessing module is configured to: select a predetermined number of consecutive frames for each sample in the video sample set, extract face regions from each of the multiple consecutive frames, and stitch the face regions in the multiple consecutive frames into a two-dimensional grid layout in chronological order to obtain a spatiotemporal plane image; The feature extraction module is configured to: extract initial spatiotemporal features based on the spatiotemporal plane image, wherein the initial spatiotemporal features are obtained by fusing a first feature used to characterize the correlation of the face region in the spatiotemporal plane image in the time dimension and a second feature used to characterize the local spatial details of the spatiotemporal plane image; The multidimensional feature fusion module is configured to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features; The classification module is configured to: perform video sample classification prediction based on the frequency enhancement features and output the prediction results; A classification loss is constructed based on the prediction results and the category labels; Feature interpolation data augmentation is performed on the initial spatiotemporal features of the fake sample to generate enhanced fake features, and a contrastive loss is constructed based on the category label and the feature set consisting of the enhanced fake features and the initial spatiotemporal features of the real sample using a supervised contrastive learning method. The classification loss and the contrastive loss are weighted and fused to obtain the total loss. Based on this total loss, at least the learnable parameters in the feature extraction module, the multi-dimensional feature fusion module, and the classification module are updated. The feature extraction module is configured to perform the following steps to extract initial spatiotemporal features based on the spatiotemporal plane image, including: inputting the spatiotemporal plane image into a first branch and a second branch respectively; in the first branch, sequentially performing a linear transformation for channel dimensionality reduction, a regular convolution operation, and a nonlinear activation on the spatiotemporal plane image, and performing a sequential scan on the nonlinearly activated features through a scan operator in the state space model to obtain the first feature; in the second branch, sequentially performing a linear transformation for channel dimensionality reduction, a regular convolution operation, and a nonlinear activation on the spatiotemporal plane image to obtain the second feature; concatenating the first feature and the second feature through channels, and projecting the concatenation result through a linear layer to obtain the initial spatiotemporal features; wherein the learnable parameters in the feature extraction module include at least: weight parameters for linear transformation and convolution kernel parameters for regular convolution operation in the first and second branches, parameters of the state space model for implementing sequential scanning in the first branch, and weight parameters of the linear layer for projecting the concatenation result.

2. The training method according to claim 1, characterized in that, The preprocessing module is configured to perform the following steps to obtain the spatiotemporal plane image, including: The video samples in the video sample set are divided into video segments of a predetermined length in chronological order, and a starting frame is randomly selected from each video segment, and a predetermined number of consecutive frames starting from the starting frame are selected. For each of the multiple consecutive frames, face bounding box detection is performed, the largest face bounding box is selected from the frame, and the frame is cropped based on the largest face bounding box to obtain the face region. The face region is scaled to a predetermined size, and the face regions corresponding to the multiple consecutive frames are stitched together in chronological order to form the two-dimensional grid layout, thereby constructing the spatiotemporal plane image.

3. The training method according to claim 1, characterized in that, The training method further includes a step of normalizing the initial spatiotemporal features before performing feature interpolation data augmentation. The normalization step includes: performing average pooling and L2 normalization on the initial spatiotemporal features of the real samples and the fake samples extracted by the feature extraction module to obtain a normalized feature set mapped to a unit hypersphere. The normalized feature set includes normalized real features and normalized fake features. The feature interpolation data enhancement step includes: constructing a first enhanced forgery feature set and a second enhanced forgery feature set as the enhanced forgery features based on the normalized forgery features. The process of constructing both the first and second enhanced forgery feature sets includes: using each feature in the normalized forgery features as the current feature; randomly selecting another feature different from the current feature from the normalized forgery features as a paired feature; and randomly sampling interpolation coefficients from a preset distribution; performing spherical linear interpolation on the unit hypersphere based on the current feature, the paired feature, and the interpolation coefficients to generate the corresponding enhanced forgery features; the number of enhanced forgery features in both the first and second enhanced forgery feature sets is consistent with the number of forgery samples.

4. The training method according to claim 3, characterized in that, The steps for constructing the contrastive loss include: Construct a first linear projection head and a second linear projection head that are independent of each other, both of which have learnable parameters; The first enhanced forgery feature set is combined with the normalized real features to construct a first combined feature set, and the second enhanced forgery feature set is combined with the normalized real features to construct a second combined feature set; The first combined feature set is input into the first linear projection head to obtain the first embedded feature set, and the second combined feature set is input into the second linear projection head to obtain the second embedded feature set; Calculate the cosine similarity between the features of the i-th sample in the first embedded feature set and the features of the j-th sample in the second embedded feature set to construct a similarity matrix between samples; A supervision mask matrix is ​​constructed based on the category labels, wherein when the category labels of the i-th sample and the j-th sample are the same, the elements in the supervision mask matrix are... It is 1 if it is true, otherwise it is 0; The contrastive loss is constructed based on the difference between the similarity matrix and the supervision mask matrix, and The training method further includes updating the learnable parameters of the first linear projection head and the second linear projection head based on the total loss.

5. The training method according to claim 1, characterized in that, The multidimensional feature fusion module is configured to perform the following steps to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features, including: Perform a two-dimensional fast Fourier transform along the spatial dimension on each channel of the initial spatiotemporal features to obtain the real and imaginary parts of the frequency domain features; The real part features and the imaginary part features are concatenated and then sequentially processed by pointwise convolution, batch normalization and nonlinear activation to obtain the frequency domain features modulated by the features. A two-dimensional inverse fast Fourier transform is performed on the frequency domain features modulated by the features to obtain the reconstructed spatial domain features. The reconstructed spatial domain features are then added to the initial spatiotemporal features by residual addition to obtain the frequency enhancement features.

6. The training method according to claim 5, characterized in that, The classification module is configured to perform the following steps to perform video sample classification prediction based on the frequency enhancement features and output the prediction results, including: The frequency enhancement features are input into the convolutional layer, and the channel dimension is compressed to obtain intermediate features; Perform average pooling on the intermediate features to obtain downsampled features with reduced spatial resolution; The intermediate features and the downsampled features are flattened and input into a multi-head attention mechanism, wherein the intermediate features are used as queries and the downsampled features are used as keys and values ​​to obtain attention-enhanced features; The attention-enhanced features are input into a pointwise convolutional layer to restore the channel dimension, resulting in the output features; and Global average pooling is performed on the output features, and the result is input into the classification layer to obtain the prediction result.

7. The training method according to claim 6, characterized in that, A classification loss is constructed based on the prediction results and the category labels, including: The prediction results output by the classification module are compared with the corresponding category labels, and the classification loss is calculated using the cross-entropy loss function.

8. A method for face authentication, characterized in that, The face authentication method includes: Obtain the video sample to be tested; The video sample to be detected is input into a face forgery detection model trained using the training method for a face forgery detection model based on multidimensional feature fusion and contrastive learning according to any one of claims 1 to 7. The face forgery detection model includes a preprocessing module, a feature extraction module, a multidimensional feature fusion module, and a classification module. The preprocessing module is configured to: select a predetermined number of consecutive frames from the video sample to be detected, extract face regions from each of the multiple consecutive frames, and spatially stitch the face regions in the multiple consecutive frames in a predetermined order to obtain a spatiotemporal plane image. The feature extraction module is configured to: extract initial spatiotemporal features based on the spatiotemporal plane image, wherein the initial spatiotemporal features are obtained by fusing a first feature used to characterize the correlation of the face region in the spatiotemporal plane image in the time dimension and a second feature used to characterize the local spatial details of the spatiotemporal plane image; The multidimensional feature fusion module is configured to perform frequency domain feature enhancement processing on the initial spatiotemporal features to obtain frequency-enhanced features; The classification module is configured to perform video sample classification prediction based on the frequency enhancement features and output the prediction results.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the face forgery detection model based on multidimensional feature fusion and contrastive learning as described in any one of claims 1 to 7, or to implement the face forgery detection method as described in claim 8.

Images