Deep fake face image detection method based on double-flow CNN and ViT hybrid model

Abstract

The invention relates to the technical field of computer vision and pattern recognition, in particular to a deep-forged face image detection method based on a double-flow CNN and ViT hybrid model. The method comprises the following steps: preprocessing a real video and a forged video to obtain an original face image; obtaining a high-frequency noise residual error feature image of the original face image based on an improved high-frequency noise residual error feature extraction module; performing first feature enhancement operation on the original face image and the high-frequency noise residual feature image based on a double-flow CNN; carrying out secondary feature enhancement operation on the original face image and the high-frequency noise residual feature image which are subjected to the primary feature enhancement operation on the basis of double-flow ViT; and fusing the features of the original face image and the high-frequency noise residual feature image after the secondary feature enhancement operation, performing true and false prediction on the fused features, and outputting a detection result. The objective of the invention is to solve the technical problems of feature splitting, local-global imbalance and insufficient cross-modal interaction in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and pattern recognition, and particularly relates to a deep fake face picture detection method based on a dual-flow CNN and ViT hybrid model. BACKGROUND

[0002] With the breakthrough progress of Generative Adversarial Networks (GANs) technology, deep fake technology has achieved a leap in the field of image and video generation. Based on GANs and its derivative architectures (such as StyleGAN series, CycleGAN, etc.), deep fake technology has formed a technology system with identity replacement, face replay, attribute editing, and full face generation as the core. Among them, the identity replacement technology realizes the cross-identity face feature replacement through three-dimensional face modeling, feature transfer and texture mapping, etc. For example, the classic algorithm FaceSwap completes face feature alignment through key point positioning and three-dimensional reconstruction technology, and FaceShifter further introduces a masking perception module, which significantly improves the face changing realism in complex scenes. The face replay technology realizes the cross-subject mapping of target person's expression and posture based on dynamic expression transfer and spatio-temporal consistency constraints. For example, the real-time replay framework based on three-dimensional dense feature point tracking and reflection transformation proposed by Thies et al. The attribute editing technology realizes the fine control of face attributes such as age, gender, expression, etc. through dual-space GAN architecture and Transformer feature decoupling mechanism. The full face generation technology relies on hidden space decoupling and progressive generation strategy to realize the unsupervised generation from random noise to high-resolution face images. In terms of technology evolution, the progressive training strategy proposed by ProGAN solves the mode collapse problem in high-resolution image generation through resolution layer optimization mechanism. StyleGAN further introduces a mapping network and a style mixing module to realize hierarchical control of the visual features of generated images. With the improvement of the complexity of the generation model, deep fake technology has made significant breakthroughs in visual fidelity, semantic consistency, and operation robustness, and its generated results have reached the level of fake-to-real, posing a serious challenge to personal privacy, social security, and public opinion ecology.

[0003] Early detection methods mainly rely on traditional image classification networks such as Xception and EfficientNet, which improve model performance through depth separable convolution and compound scaling strategy respectively, but their detection robustness is insufficient when facing compression distortion, view changes, and unseen fake methods.

[0004] Current deep fake detection research mainly focuses on the following technical routes:

[0005] Local feature extraction based on CNN (Convolutional Neural Network): Deep convolutional networks represented by Xception and EfficientNet achieve preliminary detection by optimizing feature abstraction, but they lack modeling of global semantic associations.

[0006] Fine-grained and attention enhancement: The positioning ability of fake artifacts is enhanced through multi-attention mechanisms such as spatial-texture aggregation modules and operation trace extractors, but the coupling relationship between high-frequency noise and RGB features has not been fully explored.

[0007] Dual-flow heterogeneous feature fusion: Parallel networks are used to process original RGB images and high-frequency noise residual feature images, but the local limitations of CNN branches in traditional dual-flow architecture and the global feature fragmentation problem have not been solved.

[0008] Frequency and time domain analysis: Frequency domain filtering based on fast Fourier transform (such as rPPG signal heat map) can capture physiological signal abnormalities, but is significantly affected by image compression and lighting interference; noise fingerprint analysis (such as camera model artifact enhancement) has limited discrimination in high-fidelity fake scenarios.

[0009] Global modeling of Vision Transformer (ViT): ViT achieves long-range dependency modeling through self-attention mechanisms, but its sensitivity to local subtle artifacts (such as edge seams and texture inconsistencies) is weaker than CNN, and it lacks a mechanism for aligning multi-modal features.

[0010] The shortcomings of existing deep fake detection techniques can be summarized as follows:

[0011] Feature fragmentation problem: Single reliance on RGB space or high-frequency noise results in ineffective utilization of macro-micro feature complementarity;

[0012] Local-global imbalance: There is a modeling gap between the local inductive bias of CNN and the global attention of ViT, making it difficult to capture cross-scale fake traces collaboratively;

[0013] Dynamic adaptation defect: Traditional multi-scale fusion methods (such as feature pyramids) lack learnability in responding to frequency domain fake patterns and cannot adapt to changes in noise distribution;

[0014] Insufficient cross-modal interaction: Dual-flow networks usually use late feature concatenation and do not establish an interactive mechanism for hierarchical feature transfer and attention guidance. SUMMARY

[0015] The present application aims to provide a deep fake face picture detection method based on a dual-flow CNN and ViT hybrid model, which aims to solve the technical problems of feature fragmentation, local-global imbalance, and insufficient cross-modal interaction in existing technologies.

[0016] To achieve the above object, the application provides a deep fake face picture detection method based on a double-flow CNN and ViT hybrid model, comprising the following steps:

[0017] S1: pre-processing real videos and fake videos to obtain original face images;

[0018] S2: improving a high-frequency noise residual feature extraction module of a traditional SRM (Spatial Rich Model) filter to obtain an improved high-frequency noise residual feature extraction module, and obtaining high-frequency noise residual features of the original face images based on the improved high-frequency noise residual feature extraction module to obtain high-frequency noise residual feature images;

[0019] The improvement includes:

[0020] establishing a cascade structure of an SRM filter set and a gradient operator;

[0021] adopting a per-channel gradient calculation strategy to independently process each feature channel with a Sobel operator;

[0022] introducing a gradient amplitude normalization operation to control the dynamic range of high-frequency features through a differentiable boundary constraint function;

[0023] S3: performing a first feature enhancement operation on the original face images and the high-frequency noise residual feature images based on a double-flow CNN; wherein the first feature enhancement operation comprises, in sequence, multi-scale feature extraction, frequency domain enhancement, feature dimension alignment, and multi-scale feature fusion;

[0024] S4: performing a second feature enhancement operation on the original face images and the high-frequency noise residual feature images subjected to the first feature enhancement operation based on a double-flow ViT; wherein the second feature enhancement operation comprises, in sequence, parallel encoding and bidirectional feature enhancement;

[0025] S5: fusing the features of the original face images and the high-frequency noise residual feature images subjected to the second feature enhancement operation, performing true-false prediction on the fused features based on a multi-layer perceptron classification head, and outputting a detection result.

[0026] Optionally, the pre-processing includes video frame extraction, training set and test set division, face region positioning, image size standardization, tensor format conversion, and normalization processing.

[0027] Optionally, the high-frequency noise residual features of the original face images are obtained to obtain high-frequency noise residual feature images, comprising:

[0028] The original face image is processed using a traditional SRM filter to obtain an initial high-frequency noise residual feature image;

[0029] In combination with a Sobel gradient operator, gradient calculation is performed on each channel of the initial high-frequency noise residual feature image, and a region with a gradient greater than a preset threshold is extracted as a final high-frequency noise residual feature image corresponding to the original face image.

[0030] Optionally, the first feature enhancement operation in sequence is multi-scale feature extraction, frequency domain enhancement, feature dimension alignment, and multi-scale feature fusion, specifically:

[0031] Multi-scale feature extraction: the original face image and the high-frequency noise residual feature image are respectively input into a double-flow CNN branch, each branch sequentially passes through four levels of convolution layer integration modules, and four feature maps with decreasing resolutions are obtained;

[0032] Frequency domain enhancement: a fast Fourier transform is used to construct a learnable spatial frequency filter and a learnable channel frequency filter, and the learnable high-frequency spatial filter and the learnable high-frequency channel filter are used to extract a feature map of high-frequency spatial information and a feature map of cross-channel high-frequency anomalies of the four feature maps of the original face image, and a high-frequency noise residual feature corresponding to the high-frequency noise residual feature image at the current scale level, at each scale level, the feature map of high-frequency spatial information and the feature map of cross-channel high-frequency anomalies of the original face image are added, and then processed by an SE attention mechanism to obtain an attention weight, and the result of the addition is multiplied by the attention weight, and then added to the high-frequency noise residual feature of the current scale level of the high-frequency noise residual feature image to realize fusion;

[0033] Feature dimension alignment: feature mapping is performed on the features of the last three resolutions, channel expansion and spatial compression are implemented, and the features are converted into 768-dimensional vectors; at the same time, the four-dimensional tensor format is reconstructed into a three-dimensional tensor to adapt to the feature format requirements of the ViT network;

[0034] Multi-scale feature fusion: the image features of the last three resolutions in three-dimensional tensor format obtained from the two branches are formed into multi-scale feature blocks by cross-layer splicing.

[0035] Optionally, the construction of the learnable high-frequency spatial filter includes:

[0036] First, a two-dimensional fast Fourier transform is used to convert the input image tensor from the spatial domain to the frequency domain, and the zero-frequency point in the frequency domain is shifted to the center of the frequency spectrum;

[0037] Second, a learnable parameter is defined as the standard deviation of a Gaussian function, which is used to construct a low-pass mask in the shape of a two-dimensional Gaussian function;

[0038] Then, the low-pass mask of the two-dimensional Gaussian function shape is reversed into a high-pass mask with a center value of 0 and a value of 1 around the center, and the frequency domain image obtained by the two-dimensional fast Fourier transform is multiplied element by element with the high-pass mask;

[0039] Finally, the frequency domain signal after element-by-element multiplication is converted back to the image space domain by inverse Fourier transform, the real part of the complex result obtained by the inverse Fourier transform is taken, and a ReLU activation function is applied, and finally a feature map containing only high-frequency spatial information is output.

[0040] Optionally, the construction of the learnable high-frequency channel filter comprises:

[0041] First, the channel dimension of the image tensor is processed by one-dimensional fast Fourier transform, and the zero frequency point of the channel dimension is shifted to the center;

[0042] Second, a learnable parameter is defined to construct a low-pass mask of one-dimensional Gaussian function shape;

[0043] Then, the low-pass mask of the one-dimensional Gaussian function shape is reversed into a high-pass mask, and the channel frequency domain signal obtained by the one-dimensional fast Fourier transform is multiplied element by element with the high-pass mask;

[0044] Finally, the channel frequency domain signal after element-by-element multiplication is converted back to the original channel representation by inverse Fourier transform, the real part of the complex result obtained by the inverse Fourier transform is taken, and a ReLU activation function is applied, and finally a feature map containing cross-channel high-frequency anomalies is output.

[0045] Optionally, the secondary feature enhancement operation is parallel coding and bidirectional feature enhancement in sequence, and the bidirectional feature enhancement comprises:

[0046] Parallel coding: first, the original face image and the high-frequency noise residual feature image obtained by the first feature enhancement operation are respectively divided into image block sequences, then class Token and relative position coding are added in the sequence, and the parallel coded image block sequence is sent into a 12-layer Transform Block for processing;

[0047] Bidirectional feature enhancement: divide the 12-layer Transform Block into [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12] three groups, before sending the parallel coded image block sequence into the Transform Block layer of each group, use the ViT feature as Query, the CNN multi-scale convolution feature as Key-Value, and use the multi-head attention mechanism to realize the migration and enhancement of the CNN local feature to the ViT global feature, when the Transform Block layer of this group is completely processed, use the CNN multi-scale convolution feature as Query and the ViT feature as Key-Value, and use the multi-head attention mechanism to realize the migration and enhancement of the ViT global feature to the CNN local feature.

[0048] Optionally, the S5 is specifically:

[0049] S51: The original face image and the high-frequency noise residual feature image after the first feature enhancement operation are respectively processed by 12-layer Transform Block, and the processing includes: embedding a cross-attention module after the 5th and 9th Transform Block layers, establishing a feature interaction channel between the original face image branch and the high-frequency noise residual feature image branch, and performing migration and enhancement of the respective CNN local feature to the ViT global feature before the 1st, 5th and 6th layers, migration and enhancement of the ViT global feature to the CNN local feature after the 4th, 8th and 12th layers, and feature interaction between the two branches through the cross-attention module after the 5th and 9th layers;

[0050] S52: After obtaining the features of the two branches after 12-layer Transform Block processing, first, the features of the two branches are spliced to form a joint feature, then a fully connected layer is used to preliminarily fuse the spliced features, and then the preliminarily fused features are sent into a channel attention module for weighted processing to obtain the final fused features;

[0051] S53: A multi-layer perceptron classification head is used to perform true or false prediction based on the final fused features, and output the final deep fake face picture detection result.

[0052] The beneficial effects of the present application are: the present application constructs a "filter-gradient" two-stage residual extraction module, realizes decoupling and enhancement of high-frequency noise and RGB features; an adaptive frequency domain filter bank is designed, which dynamically optimizes multi-scale feature fusion in Fourier space; a bidirectional feature migration mechanism of CNN and ViT is introduced, realizing collaborative enhancement of local details and global semantics; a hierarchical cross-attention module is embedded, establishing spatial-frequency feature correlation reasoning in the deep layer of the Transformer, thereby enhancing the detection performance of the model on deep fake face pictures. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The step flow chart of the present application is shown in the figure;

[0054] Figure 2 The first layer 56x56 resolution of the multi-scale convolution feature extraction module of the present application is shown in the figure;

[0055] Figure 3 The second layer 28x28 resolution, the third layer 14x14 resolution and the fourth layer 7x7 resolution of the multi-scale convolution feature extraction module of the present application are shown in the figure;

[0056] Figure 4 The spatial-frequency joint feature extraction module of the present application is shown in the figure;

[0057] Figure 5 The bimodal cross-attention mechanism module of the present application is shown in the figure;

[0058] Figure 6 The bimodal feature fusion module of the present application is shown in the figure;

[0059] Figure 7 The overall structure diagram of the deep fake face picture detection model network of the present application is shown in the figure;

[0060] Figure 8 The confusion matrix of the deep fake face picture detection model of the present application on various deep fake methods is shown in the figure. DETAILED DESCRIPTION

[0061] The present application will be further described below with reference to the accompanying drawings. The example embodiments can be implemented in many forms, but the protection scope of the present application is not limited to the present embodiments. The embodiments of the present application and the features in the embodiments can be combined with each other without conflict. Therefore, the present application describes many details to facilitate the technical personnel to have a good understanding of the implementation mode of the present method.

[0062] Example 1: as shown in the figure, the step flow chart of a deep fake face detection method based on a bimodal CNN and ViT hybrid model of the present application includes the following steps: Figure 1

[0063] ​S1: Preprocess the real video and the fake video to obtain the original face image.

[0064] Optionally, video frame extraction and dataset division are performed, 50 frames of video frames are extracted from each real video and its corresponding fake video, and all the extracted video frames are divided into a training set and a test set in proportion, and the storage location information of the pictures and the true or false labels are stored in a CSV file according to the division of the dataset, so as to load data during subsequent training and testing.

[0065] Optionally, face region positioning is performed. In the training stage, since the extracted video frames are the entire screenshots of the video, after loading the pictures according to the CSV file of the divided dataset, a face detection algorithm (Dlib library) is used to accurately identify and extract the face region in the image to obtain the face region picture of each video frame, providing key information for subsequent analysis.

[0066] Optionally, image standardization and normalization processing is performed. The extracted face region picture is standardized to unify the picture size to a fixed size of 224x224 pixels, converted to a tensor format, and the image is normalized to make the data meet the training and testing requirements of the model.

[0067] S2: Improve the high-frequency noise residual feature extraction module of the traditional SRM filter to obtain an improved high-frequency noise residual feature extraction module, and obtain high-frequency noise residual features of the original face image based on the improved high-frequency noise residual feature extraction module to obtain a high-frequency noise residual feature image.

[0068] Optionally, the improved high-frequency noise residual feature extraction module is improved based on the traditional SRM filter as follows:

[0069] Improvement I: A two-stage feature enhancement mechanism is proposed, which uses a cascade structure of an SRM filter set (first-stage feature extraction) and a gradient operator (second-stage feature enhancement) to construct a "filter-gradient" two-stage feature enhancement mechanism. Compared with the traditional single-stage SRM method, this design can capture high-frequency fake traces across scales more finely.

[0070] Improvement II: Channel-independent gradient analysis is proposed, which uses a channel-by-channel gradient calculation strategy to independently process each feature channel with a Sobel operator. This preserves the differentiated feature responses between different filter channels and avoids the high-frequency feature confusion problem caused by traditional multi-channel mixed processing.

[0071] Improvement III: A dynamic range constraint mechanism is proposed, which introduces a gradient amplitude normalization operation to realize dynamic range control of high-frequency features through a derivable boundary constraint function (clamp(0,1)). This can suppress extreme noise interference while preserving the interpretability of the features.

[0072] Further, in the SRM filtering stage, the method of extracting high-frequency noise residual feature images uses three traditional SRM filters with fixed weights to obtain high-frequency features and edge information from images, specifically:

[0073] The first filter weight filter1: used to detect the change of horizontal gradient, reflecting the intensity of horizontal texture and edge. The formula is as follows:

[0074] (1)

[0075] The second filter weight filter2: used to detect the change of vertical gradient, reflecting the intensity of vertical texture and edge. The formula is as follows:

[0076] (2)

[0077] The third filter weight filter3: used to detect the second-order horizontal direction texture feature, fine line noise. The formula is as follows:

[0078] (3)

[0079] Stack the filters, the formula is as follows:

[0080] (4)

[0081] Further, in the image processing process, the original image x is subjected to a two-dimensional convolution operation with the stacked filter set . Specifically, each channel of the filter will multiply the corresponding channel of the input image element by element (i.e., point by point), and then sum the resulting products to generate a single-channel output. This convolution process involves the filter sliding in the H (image height) x W (image width) spatial domain to traverse the entire input image and finally generate a complete output feature map . This convolution operation can be expressed by the following mathematical formula:

[0082] (5)

[0083] Where b represents the bth sample, c represents the output channel number, (h, w) represents the position coordinates (row and column) on the output feature map, i represents the input channel number, and (u, v) represents the position index inside the filter. represents the pixel value of the input image at the nth sample, the i th channel, and the position (h+u, w+v), represents the weight value of the convolution kernel at the output channel c, the input channel i, and the position (u, v) inside the filter.

[0084] Further, in the gradient enhancement stage, Sobel gradient analysis is independently performed on each feature channel , specifically:

[0085] In the horizontal gradient . The formula is as follows:

[0086] (6)

[0087] In the vertical gradient . The formula is as follows:

[0088] (7)

[0089] The gradient amplitude is obtained . The formula is as follows:

[0090] (8)

[0091] Further, the multi-channel features are synthesized and normalized, and the final output is obtained as the high-frequency noise residual feature, where B represents the size of the training data fed into the model in each batch during model training, and the high-frequency noise residual feature expression is:

[0092] (9)

[0093] where is a calculation function in PyTorch, which is used to limit the numerical range.

[0094] S3: Perform a first feature enhancement operation on the original face image and the high-frequency noise residual feature image based on the dual-flow CNN, respectively; wherein the first feature enhancement operation is in turn multi-scale feature extraction, frequency domain enhancement, feature dimension alignment, and multi-scale feature fusion.

[0095] It should be understood that deep fake images often have inconsistencies or noise abnormalities in high-frequency regions or specific channel frequencies. In order to enhance the modeling ability of the model for small tampering traces in deep fake images, this step constructs a space-frequency joint enhancement prior module. This module fuses a learnable frequency domain filtering mechanism and an SE attention mechanism to guide the network to capture more subtle frequency and texture residual features in the forged area. Two learnable parameter frequency domain filtering modules are used to suppress low frequencies in image spatial domain and channel dimension, respectively, to obtain more discriminative fake traces. This is used to assist the backbone network in mining more rich and sensitive fake clues from the original image. The structure of this module is mainly designed as follows:

[0096] As Figure 2 , 3As shown, it is a multi-scale convolution feature extraction module: multi-scale feature extraction is performed, the original face image and the high-frequency noise residual feature image are respectively input into the dual-stream CNN branch, and each branch passes through the four-level convolution layer integration module in turn to gradually extract feature multi-scale features of 56×56 pixels (64 dimensions), 28×28 pixels (128 dimensions), 14×14 pixels (256 dimensions) and 7×7 pixels (512 dimensions).

[0097] Spatial-frequency domain joint feature extraction module: performs frequency domain enhancement, specifically: uses fast Fourier transform to construct a learnable spatial-frequency domain filter and a learnable channel-frequency domain filter, uses the learnable high-frequency spatial filter and the learnable high-frequency channel filter to respectively extract the feature map of the high-frequency spatial information of the four feature maps of the original face image and the feature map of the cross-channel high-frequency anomaly, as well as the high-frequency noise residual features of the corresponding scale level of the high-frequency noise residual feature image. At each scale level, the feature map of the high-frequency spatial information of the original face image and the feature map of the cross-channel high-frequency anomaly are added together, and then processed using the SE attention mechanism to obtain an attention weight, and the result of the addition is multiplied by the attention weight, and then added to the high-frequency noise residual features of the current scale level of the high-frequency noise residual feature image to achieve fusion;

[0098] It is understandable that if Figure 4 As shown in the figure, in the joint spatial-frequency domain feature extraction module, low-frequency components are suppressed by Gaussian masks in the learnable spatial-frequency domain filters and the learnable channel-frequency domain filters, which can retain more sensitive high-frequency features for forgeries. This is used to adaptively enhance the frequency components in the image that are sensitive to deepfakes and capture subtle traces of tampering. Furthermore, the SE attention mechanism can adaptively control the influence of the frequency-domain enhancement features on the branch features of the high-frequency noise residual feature image, highlighting the tampered areas.

[0099] Feature dimension alignment: Feature mapping is performed on the features of the last three resolutions, channel expansion and spatial compression are implemented, and they are converted into 768-dimensional vectors. At the same time, the four-dimensional tensor format is reconstructed into a three-dimensional tensor to adapt to the feature format requirements of the ViT network.

[0100] Multi-scale feature fusion: The image features in the three-dimensional tensor format of the last three resolutions of the two branches are respectively formed into multi-scale feature blocks through cross-layer splicing, which are used for the subsequent Vision Transform network to further fuse the spatial-frequency joint multi-scale convolution feature information.

[0101] Furthermore, the construction of a learnable spatial frequency domain filter and a learnable channel frequency domain filter in this embodiment is described in detail:

[0102] It can be understood that the learnable high-frequency spatial filter (LearnableHFS) acts on the input feature map , a two-dimensional fast Fourier transform is performed, and a learnable Gaussian mask is calculated to suppress the central low-frequency region and enhance the high-frequency texture response.

[0103] Specifically, the two-dimensional fast Fourier transform (FFT2) is as follows:

[0104] (10)

[0105] The two-dimensional fast Fourier transform is performed on the two-dimensional feature map of each channel of the input image x, and the image is converted from the spatial domain to the frequency domain to obtain . The fake region is often accompanied by local frequency abnormalities, which are more obvious in the frequency domain.

[0106] Further, a two-dimensional Gaussian function shape mask in the frequency domain is constructed , a two-dimensional Gaussian kernel centered on the spectrum center is generated, and the formula is as follows:

[0107] (11)

[0108] In the formula, represents the coordinates of the spectrum graph, and , represents the lowest frequency at the center of the spectrum. The mask is largest at the center low frequency and gradually decays outward, suppressing the central low frequency and retaining the high frequency information. is a learnable parameter used to control the mask size. The network can learn a high-frequency filtering window suitable for the current task.

[0109] Further, the frequency suppression and enhancement formula is as follows:

[0110] (12)

[0111] The spectrum graph is multiplied by the inverse Gaussian mask to retain the high-frequency components of the image to obtain . After processing, the low-frequency information such as large-area outlines and background smooth regions is weakened, and the edge, texture and other local sensitive regions are enhanced.

[0112] Further, the two-dimensional inverse Fourier transform converts the frequency domain enhanced data back to the spatial domain image, and the formula is as follows:

[0113] (13)

[0114] ​Where Re (real part) IFFT output is a complex number, taking its real part as the image value. ReLU activation function prevents negative values, ensuring that image features are non-negative. Image feature maps containing only high-frequency information can be obtained , sensitive to the boundaries and residues of the forged region.

[0115] It can be understood that the learnable high-frequency channel filter (LearnableHFC) performs Fourier transform in the channel dimension, FFT along the channel axis, to simulate the frequency distribution on the channel, especially for the cross-channel disturbance of high-frequency forged texture. The mask is constructed as follows.

[0116] Specifically, the one-dimensional Fourier transform (FFT) formula is as follows:

[0117] (14)

[0118] Perform one-dimensional Fourier transform on the input features in the channel dimension (dim=1). When the image color or texture is tampered with, inconsistencies may occur between the RGB domain or feature channels. Abnormalities can be captured by obtaining the channel spectrum.

[0119] Further, a channel one-dimensional Gaussian function shape mask is constructed , generating a one-dimensional Gaussian kernel centered on the spectrum center, the formula is as follows:

[0120] (15)

[0121] where, is a learnable parameter that controls the mask size. represents the channel index, represents the middle position of the low-frequency corresponding channel. The low-frequency component in the channel is suppressed, and the high-frequency feature part between channels is retained.

[0122] Further, the frequency suppression and enhancement formula is as follows

[0123] (16)

[0124] Multiply the frequency spectrum by the inverse Gaussian mask to retain the high-frequency components of the image to get .

[0125] Further, one-dimensional inverse Fourier transform IFFT channel frequency enhancement and restoration, the formula is as follows:

[0126] (17)

[0127] It can be understood that, similarly to the suppression of the low-frequency part, one-dimensional inverse Fourier transform restores the frequency domain enhanced data to the channel domain image. The output image features The high-frequency information change across channels is reserved, which is beneficial for the model to capture color artifacts, channel discordance and other fake traces.

[0128] It can be understood that the SE attention fusion mechanism module can adaptively select key information in the frequency domain enhanced feature, highlight the forged significant area, and be used for re-weighting the fused high-frequency feature. The feature fusion formula is as follows:

[0129] (18)

[0130] wherein, represents a high-frequency noise residual feature image stream feature obtained by the current level convolution module, represents an original image stream feature obtained by the current level convolution module. represents a high-frequency enhanced feature obtained by performing frequency domain spatial filtering on the original image stream feature, represents a high-frequency enhanced feature obtained by performing frequency domain channel filtering on the original image stream. represents a fusion feature obtained by fusing the high-frequency noise residual feature image stream and the frequency domain enhanced feature of the original image stream. The purpose is to enhance the high-frequency information only in the forged area and keep stable in the non-forged area.

[0131] S4: performing a second feature enhancement operation on the original face image and the high-frequency noise residual feature image subjected to the first feature enhancement operation based on the dual-flow ViT; wherein the second feature enhancement operation is parallel encoding and bidirectional feature enhancement in sequence.

[0132] It should be understood that the ViT network can utilize the self-attention mechanism to establish a global dependency relationship between patches on the basis of dividing the image into multiple patches, so as to efficiently capture the global inconsistency existing in the deep fake image. The model has strong high-dimensional feature representation modeling capability and can detect subtle fake traces such as frequency domain artifacts and detail textures in the deep fake image.

[0133] Optionally, the second feature enhancement operation is parallel encoding and bidirectional feature enhancement in sequence, specifically:

[0134] Parallel encoding: first, the original face image and the high-frequency noise residual feature image subjected to the first feature enhancement operation are respectively divided into image block sequences, then, class Token and relative position encoding are added in the sequence, and the parallel encoded image block sequence is sent into 12-layer Transform Block processing;

[0135] Specifically, in the present embodiment, the image is first divided into a series of patch blocks, and then the patches are flattened and converted into a patch sequence. Each image block is 16x16 pixels in size, and the image is divided into 196 image blocks. At this time, the dimensions of the image are changed from [Batch_Size, 3, 224, 224] to the sequence [Batch_Size, 196, 768]. In order to introduce position information and pay attention to the positional relationship between each patch, position encoding and class token are added to the patch sequence, so that the dimensions become [Batch_Size, 197, 768]. The processed patch sequence is then sent to the Transformer Encoder composed of multiple stacked Blocks for fake feature extraction. The self-attention mechanism (Self-Attention) and feed forward neural network (Feed Forward Network, FFN) inside the Transformer Encoder can learn the complex relationships between different elements in the patch sequence, and then extract the global information of the image. It should be understood that the Block combines the multilayer perceptron MLP and the multi-head self-attention mechanism to model the global dependency.

[0136] Bidirectional feature enhancement: divide the 12-layer Transform Block into [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12] three groups, before sending the parallel encoded image block sequence into each group of Transform Block layer, use ViT features as Query, CNN multi-scale convolutional features as Key-Value, and use multi-head attention mechanism to realize the migration and enhancement of CNN local features to ViT global features, when all the Transform Block layers in this group are processed, then use CNN multi-scale convolutional features as Query, ViT features as Key-Value, and use multi-head attention mechanism to realize the migration and enhancement of ViT global features to CNN local features. In this process, it includes:

[0137] Injector module: realizes the attention injection from the local detail information of the convolutional features to the global structural information of the ViT features, which improves the fake clue detection ability of the information of the original image and the high-frequency noise residual feature image in the global modeling process.

[0138] Extractor module: realizes the reverse fusion from the ViT features to the convolutional features, and guides the enhancement of the low-level texture information of the convolutional network through the high-level semantic information of the ViT, so as to further improve the recognition accuracy of the fake pictures.

[0139] S5: Fuse the features of the original face image and the high-frequency noise residual feature image subjected to the secondary feature enhancement operation, perform true and false prediction on the fused features based on a multi-layer perception classification head, and output a detection result.

[0140] S51: The original face image and the high-frequency noise residual feature image subjected to the first feature enhancement operation are respectively processed by 12 layers of Transform Block, and the processing includes: embedding a cross-attention module after the 5th and 9th Transform Block processing to establish a feature interaction channel between the original face image branch and the high-frequency noise residual feature image branch, and migrating and enhancing the CNN local features of the two branches to the ViT global features before the 1st, 5th and 6th processing, migrating and enhancing the ViT global features to the CNN local features after the 4th, 8th and 12th processing, and realizing feature interaction between the two branches through the cross-attention module after the 5th and 9th processing of the two branches;

[0141] Specifically, in the present embodiment, as shown in Figure 5 , a double-modal cross-attention mechanism module is constructed, which aims to establish the correlation between the two input modalities (i.e., the original face image and the high-frequency noise residual feature image) through the attention mechanism. Specifically, a cross-modal self-attention mechanism is introduced to capture and fuse the interaction information between the two different modal data.

[0142] In the calculation process, the original face image x and the high-frequency noise residual feature image y are regarded as two keys. First, the two keys are projected into a shared low-dimensional space by using a fully connected layer, and then the similarity, i.e., the energy, between them is calculated by matrix operation. Based on this energy value, the attention weight matrix of x on y and the attention weight matrix of y on x are calculated respectively.

[0143] Next, x and y are mapped to a new feature space by a fully connected layer. Using the previously calculated attention weight matrix, the weighted output of y on x and the weighted output of x on y are calculated respectively. These weighted outputs are weighted and superimposed with the original inputs to obtain the final output and . Taking the calculation process of as an example, the calculation process is the same.

[0144] The energy calculation formula is as follows:

[0145] (19)

[0146] In the formula, With represents the projection of input features x, y into the same low-dimensional space. represents the attention energy matrix obtained by calculating the matching relationship between x and y of two modalities.

[0147] The attention weight matrix calculation formula is as follows:

[0148] (20)

[0149] In the formula, represents the attention weight matrix of y to x, that is, when x is enhanced, which positions of y should each position of x pay attention to. represents normalization, which normalizes the result into a probability distribution, represents a fully connected layer, which performs an additional learning transformation on the energy matrix.

[0150] The weighted output calculation formula is as follows:

[0151] (21)

[0152] In the formula, represents the representation of the Value feature of modality y after projection, which is used to pass the information of y to x, represents a learnable scalar parameter used to control the influence strength of y to x fusion features.

[0153] It can be understood that, represents the enhanced x obtained after x fuses the information of y, represents the enhanced y obtained after y fuses the information of x. The cross-modal cross-attention mechanism module only performs feature interaction after the 5th and 9th layers of the 12-layer Transform Block of the original face image (x) and the high-frequency noise residual feature image (y) two branches, and obtains With y is then taken as the new x and y, and sent to the next layer of Transform Block for further processing.

[0154] It can be understood that this model pays special attention to the differences in details, because the real areas and tampered areas of the forged images often have different noise patterns. By calculating the interaction information between the two modalities, the model can capture these differences and effectively identify the traces of forgery. By integrating the information of the two modalities, the model can obtain a more comprehensive feature representation, thereby enhancing the discriminative ability of the model for forged images. When processing images with minor traces of forgery, this attention mechanism can significantly improve the sensitivity of the model, enabling it to more accurately identify the forged content.

[0155] S52: After obtaining the features of the two branches after 12-layer Transform Block processing, first, the features of the two branches are spliced to form joint features, then a fully connected layer is used to preliminarily fuse the spliced features, and then the preliminarily fused features are sent to a channel attention module for weighted processing to obtain final fused features;

[0156] Specifically, in the embodiment, a dual-modal feature fusion module is constructed, aiming to realize the fusion of the original face image and the high-frequency noise residual feature image data, and to provide more rich and discriminative feature information for subsequent classification tasks by enhancing the feature representation. The feature fusion module process is as shown in Figure 6 .

[0157] In the calculation process, first, the original image x and the high-frequency noise residual feature image y are spliced along the feature dimension to form a joint feature map. Then, a fully connected layer is applied for feature fusion, and the spliced feature map is converted into a new and more compact feature representation through linear transformation. In order to accelerate the model training process and ensure the stability of learning, batch normalization processing is performed on the output of the fully connected layer. Then, the ReLU activation function is introduced to map the features in a nonlinear way, so that the network can learn more complex feature representations.

[0158] After the above processing, the fused feature map is sent to the channel attention module for further weighting. This module uses attention mechanism to dynamically adjust the features according to the importance of each channel, so as to highlight the key information and suppress the irrelevant features. Finally, a weighted feature representation is obtained, which retains important features while reducing redundant information, providing strong support for subsequent classification tasks. The specific calculation is as follows.

[0159] The feature splicing calculation formula is as follows:

[0160] (22)

[0161] In the formula, x and y are the final image features of the original image and the high-frequency noise residual feature image after 12-layer Transform Block processing. represents the splicing of x and y, represents the result of splicing x and y.

[0162] Weighted feature The calculation formula is as follows:

[0163] (23)

[0164] In the formula, represents a linear layer, x and y are spliced to double the channel dimension, and a linear layer is used to process the spliced result of x and y Projection is the same channel dimension as x and y.

[0165] The channel attention fusion calculation formula is as follows:

[0166] (24)

[0167] In the formula, is the attention weight obtained by the channel attention module, is the final output result.

[0168] S53: A multi-layer perceptron classification head is used to predict true and false based on the final fused features, and the final deep fake face image detection result is output.

[0169] In summary, the application adopts a dual-flow collaborative architecture of original face image and high-frequency noise residual feature image, a multi-modal feature fusion and attention guiding structure design, aiming to mine potential forgery traces from different frequency domains. The model constructs a framework for identifying forgery traces through multi-modal feature fusion and frequency domain enhancement technology. The core design is the improved high-frequency noise residual feature extraction module, which adopts a two-stage enhancement strategy: first, use the SRM filter set to capture cross-scale high-frequency residuals (such as edge abnormalities and texture discontinuities), then perform channel-independent dynamic gradient analysis through the Sobel gradient operator, and combine with the normalized constraint to suppress noise interference, generating a 3-channel high-frequency residual feature map. LearnableHFS and LearnableHFC are designed in the frequency domain to suppress image spatial low-frequency components and channel low-frequency disturbances, respectively, and SE attention mechanism is used to fuse the original image and frequency domain enhanced features to strengthen the response of the forged area. The model uses a dual-flow ViT architecture to process the original image and the high-frequency noise residual feature image in parallel. The ViT network uses self-attention mechanism to model global dependency, and through the bidirectional feature interaction modules Injector and Extractor, it realizes the dynamic complementarity of CNN local details and ViT global semantics: the former injects CNN features into ViT to enhance the modeling of forged details, and the latter uses ViT semantics to optimize CNN texture perception. In the cross-modal decision stage, the energy matrix of the original image and the high-frequency mode is calculated through cross-modal self-attention, the complementary information is weighted and fused, and the channel attention (CA module) is used to select key features, and finally the detection result is output through the MLP classification head. The overall structure of the deep fake face image detection model network is shown in Figure 7 .

[0170] The model has the advantages of high frequency sensitivity and multi-modal cooperation: two-stage residual extraction locates edge and texture abnormalities, spatial-frequency domain joint modeling effectively suppresses interference, dual-flow ViT and bidirectional interaction consider global context and local fake traces, and cross-attention mechanism enhances inter-modal collaborative discrimination.

[0171] The effectiveness of the technical solutions of the present application is illustrated by experiments based on the specific implementation described below. Comparative experiments were conducted on four different tampering methods (Deepfake, Face2Face, FaceSwap, and NeuralTextures) of the c23 version on the FF++ dataset, and the training set and test set used the same tampering method dataset. The experimental results are shown in Table 1.

[0172] Table 1 Test results of different tampering methods of FF++ (c23) dataset

[0173]

[0174] The experiment uses accuracy ACC to evaluate the detection performance of the model. The experimental results show that the accuracy of the present application can reach more than 90%, and is better than the existing mainstream model.

[0175] Further, as shown in Figure 8 To visualize and analyze the detection performance of the present application, the confusion matrix obtained by training and testing the model on different fake methods is plotted. The confusion matrix depicts the ability of the model to distinguish between positive and negative samples. According to the labels of the horizontal and vertical coordinates of the confusion matrix and the values in the four regions, the true positive examples (fake pictures identified as fake pictures), true negative examples (real pictures identified as real pictures), false positive examples (real pictures identified as fake pictures), and false negative examples (fake pictures identified as real pictures) can be obtained. From the confusion matrix, it can be seen that the true positive and true negative samples are the most numerous, and only a small number of samples are false positive and false negative.

[0176] The above is only a description of the preferred embodiments of the present application and the principles of the technology used. The protection scope of the present application is not limited to this. For those skilled in the art, within the technical scope disclosed by the present application, various corresponding changes and modifications can be made according to the above technical solutions, but all should be covered within the protection scope of the present application.

Claims

1. A deep fake face detection method based on a dual-flow CNN and ViT hybrid model, characterized in that, The method comprises the following steps: S1: preprocessing real videos and fake videos to obtain original face images; S2: improving a high-frequency noise residual feature extraction module of a traditional SRM filter to obtain an improved high-frequency noise residual feature extraction module, and obtaining high-frequency noise residual features of the original face images based on the improved high-frequency noise residual feature extraction module to obtain a high-frequency noise residual feature image; The improvement comprises: establishing a cascade structure of an SRM filter set and a gradient operator; adopting a channel-by-channel gradient calculation strategy to independently perform Sobel operator processing on each feature channel; introducing a gradient amplitude normalization operation to control the dynamic range of high-frequency features through a derivable boundary constraint function; S3: performing a first feature enhancement operation on the original face images and the high-frequency noise residual feature images based on a double-flow CNN; wherein the first feature enhancement operation comprises, in sequence, multi-scale feature extraction, frequency domain enhancement, feature dimension alignment, and multi-scale feature fusion; S4: performing a second feature enhancement operation on the original face images and the high-frequency noise residual feature images subjected to the first feature enhancement operation based on a double-flow ViT; wherein the second feature enhancement operation comprises, in sequence, parallel encoding and bidirectional feature enhancement; S5: fusing the features of the original face images and the high-frequency noise residual feature images subjected to the second feature enhancement operation, performing true-false prediction on the fused features based on a multilayer perceptron classification head, and outputting a detection result.

2. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 1, characterized in that, The preprocessing comprises video frame extraction, training set and test set division, face region positioning, image size standardization, tensor format conversion, and normalization processing.

3. The deep fake face detection method based on the dual-stream CNN and ViT hybrid model according to claim 1 is characterized in that The high-frequency noise residual of the original face images is obtained to obtain a high-frequency noise residual feature image, comprising: processing the input original face images using a traditional SRM filter to obtain an initial high-frequency noise residual feature image; combining a Sobel gradient operator to perform gradient calculation on each channel of the initial high-frequency noise residual feature image, and extracting a region with a gradient greater than a preset threshold as a final high-frequency noise residual feature image corresponding to the original face images.

4. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 1, characterized in that, The first feature enhancement operation comprises, in sequence, multi-scale feature extraction, frequency domain enhancement, feature dimension alignment, and multi-scale feature fusion, specifically comprising: multi-scale feature extraction: inputting the original face images and the high-frequency noise residual feature images into double-flow CNN branches, respectively, and passing each branch through four levels of convolution layer integration modules to obtain four feature maps with decreasing resolutions; Frequency domain enhancement: a learnable spatial frequency domain filter and a learnable channel frequency domain filter are constructed using fast Fourier transform, high-frequency spatial information feature maps and cross-channel high-frequency anomaly feature maps of four feature maps of the original face image are extracted using the learnable high-frequency spatial filter and the learnable high-frequency channel filter respectively, and high-frequency noise residual features corresponding to the scale level of the high-frequency noise residual feature image are extracted, at each scale level, the high-frequency spatial information feature maps and the cross-channel high-frequency anomaly feature maps of the original face image are added, then processed by SE attention mechanism to obtain an attention weight, and then multiplied by the added result and added to the high-frequency noise residual features of the current scale level of the high-frequency noise residual feature image to realize fusion; Feature dimension alignment: the features of the last three resolutions are mapped to implement channel expansion and spatial compression, and are converted into 768-dimensional vectors; at the same time, the four-dimensional tensor format is reconstructed into a three-dimensional tensor to adapt to the feature format requirements of the ViT network; Multi-scale feature fusion: the image features of the two branches obtained in the last three resolutions are formed into multi-scale feature blocks through cross-layer splicing.

5. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 4, characterized in that, The construction of the learnable high-frequency spatial filter includes: Firstly, the input image tensor is converted from the spatial domain to the frequency domain by using two-dimensional fast Fourier transform, and the zero frequency point in the frequency domain is moved to the center of the frequency spectrum; Secondly, a learnable parameter is defined as the standard deviation of the Gaussian function, which is used to construct a low-pass mask in the shape of a two-dimensional Gaussian function; Then, the low-pass mask in the shape of a two-dimensional Gaussian function is inverted into a high-pass mask, the center value of the high-pass mask is 0, and the surrounding is 1, the frequency domain image obtained by two-dimensional fast Fourier transform is multiplied by the high-pass mask element by element; Finally, the element-wise multiplied frequency domain signal is converted back to the image spatial domain by using inverse Fourier transform, the real part of the complex result obtained by inverse Fourier transform is taken and the ReLU activation function is applied, and finally the feature map containing only high-frequency spatial information is output.

6. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 4, characterized in that, The construction of the learnable high-frequency channel filter includes: Firstly, the channel dimension of the image tensor is processed by using one-dimensional fast Fourier transform, and the zero frequency point of the channel dimension is moved to the center; Secondly, a learnable parameter is defined to construct a low-pass mask in the shape of a one-dimensional Gaussian function; Then, the low-pass mask in the shape of a one-dimensional Gaussian function is inverted into a high-pass mask, and the channel frequency domain signal obtained by one-dimensional fast Fourier transform is multiplied by the high-pass mask element by element; Finally, the element-wise multiplied channel frequency domain signal is converted back to the original channel representation by using inverse Fourier transform, the real part of the complex result obtained by inverse Fourier transform is taken and the ReLU activation function is applied, and finally the feature map containing cross-channel high-frequency anomaly is output.

7. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 1, characterized in that, The secondary feature enhancement operation is parallel coding and bidirectional feature enhancement in sequence, which is specifically Parallel encoding: first, the original face image and the high-frequency noise residual feature image after the first feature enhancement operation are respectively divided into image block sequences, then, class Token and relative position encoding are added in the sequence, and the parallel encoded image block sequence is sent into the 12-layer Transform Block processing; Bidirectional feature enhancement: the 12-layer Transform Block is divided into [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12] three groups, before the parallel encoded image block sequence is sent into the Transform Block layer of each group, the ViT feature is taken as the Query, the CNN multi-scale convolution feature is taken as the Key-Value, and the multi-head attention mechanism is used to realize the migration and enhancement of the CNN local feature to the ViT global feature, when the Transform Block layers of the group are all processed, the CNN multi-scale convolution feature is taken as the Query, and the ViT feature is taken as the Key-Value, and the multi-head attention mechanism is used to realize the migration and enhancement of the ViT global feature to the CNN local feature.

8. The deep fake face detection method based on a dual-flow CNN and ViT hybrid model according to claim 1, characterized in that, The S5 is specifically: S51: the original face image and the high-frequency noise residual feature image after the first feature enhancement operation are respectively processed by 12-layer Transform Block, the processing includes: after the 5th and 9th Transform Block layers are processed, the cross-attention module is embedded to establish the feature interaction channel between the original face image branch and the high-frequency noise residual feature image branch, the two branches are migrated and enhanced from the CNN local feature to the ViT global feature before the 1st, 5th and 6th layers are processed, and the ViT global feature is migrated and enhanced to the CNN local feature after the 4th, 8th and 12th layers are processed, and the feature interaction between the two branches is realized through the cross-attention module after the 5th and 9th layers are processed; S52: after the features of the two branches after 12-layer Transform Block processing are obtained, the features of the two branches are first spliced to form a joint feature, then a fully connected layer is used to preliminarily fuse the spliced features, and then the preliminarily fused features are sent into the channel attention module for weighted processing to obtain the final fused features; S53: a multi-layer perceptron classification head is used to predict true and false based on the final fused features, and the final deep fake face picture detection result is output.

Images