A multi-step sequence deepfake detection method and system fusing frequency domain

Abstract

The application provides a multi-step sequence deep forgery detection method and system fusing frequency domains, which solves the technical problems of low recognition accuracy, poor interpretability and weak anti-interference ability when recognizing complex forged content. It includes obtaining a face image to be detected, performing feature extraction to obtain a dual-domain fusion feature; a dual-domain enhanced sequence decoder is constructed based on an autoregressive mechanism, and an attention prior graph is combined to perform autoregressive sequence decoding on the dual-domain fusion feature to obtain a predicted forgery operation sequence; the model is optimized by target loss collaborative optimization and reverse sequential modeling training on the predicted forgery operation sequence, the optimized model is used to calculate the forgery operation sequence, and the type and sequence of the multi-step forgery operation are obtained. The application can be widely applied to the field of forged image detection technology.

Description

Technical Field

[0001] This application belongs to the field of forgery detection technology, and more specifically, relates to a multi-step sequence deep forgery detection method and system that integrates frequency domain. Background Technology

[0002] With the rapid development of generative artificial intelligence and deep learning technologies, deepfake technology has been able to synthesize highly realistic facial images and videos. Its misuse poses a serious threat to personal privacy, social trust, and even national security. The deepfake process is essentially a multi-step, sequential image manipulation, typically involving a combination of operations such as face replacement, expression replay, and attribute editing. These operations not only leave subtle, unnatural traces of tampering in the spatial domain (i.e., at the pixel level) but also introduce regular, anomalous patterns in the frequency domain (i.e., at the spectrum level).

[0003] However, existing detection methods either treat forged content as the result of a single operation or only perform binary classification on the final mixed product. For example, Chinese invention patent CN118968269A provides a deep forgery detection method that integrates spatial texture differences and frequency domain information. Although it can achieve binary classification of genuine and fake images by combining spatial texture differences and frequency domain information, this detection method fails to deeply model the complex sequence relationships between multi-step forgery operations, nor does it effectively capture the trace superposition effect brought about by different operations. As a result, its recognition accuracy is insufficient when facing complex and progressive forgery attacks.

[0004] In addition, existing methods also have shortcomings in dual-domain feature fusion. Specifically, most current methods focus on extracting visual features from the spatial domain. Although some studies have introduced frequency domain analysis as an aid, it usually only stays at the level of simple feature stitching or post-processing, failing to achieve adaptive and dynamic fusion of spatial and frequency domain features in deep networks. This makes it difficult for the model to fully capture the more significant forgery artifacts in the frequency domain, resulting in weak anti-interference ability and significant degradation in detection performance when faced with image compression, resolution changes, or adversarial perturbations.

[0005] Furthermore, existing methods generally lack the ability to model multi-step forgery sequences. Most can only determine "whether it is forgery" but cannot further infer "what kind of forgery operation was performed" or "the order in which the operation was executed," which limits the interpretability of the technology and its application value in real-world tracing scenarios. Even though a few studies have attempted to introduce sequence models, their attention mechanisms are often not deeply coupled with the dual-domain (spatial and frequency domain) evidence of images, making them susceptible to inter-step interference when decoding multi-step sequences, resulting in insufficient accuracy and robustness of sequence reasoning. Summary of the Invention

[0006] The purpose of this application is to provide a multi-step sequence deep forgery detection method and system that integrates frequency domains, so as to solve the technical problems of low recognition accuracy, poor interpretability and weak anti-interference ability in the prior art when identifying complex forged content.

[0007] To achieve the above objectives, a first aspect of this application provides a multi-step sequence deep forgery detection method that integrates the frequency domain, comprising the following steps: Acquire the face image to be detected and perform feature extraction to obtain dual-domain fusion features; A dual-domain enhanced sequence decoder is constructed based on an autoregressive mechanism. The predicted forgery operation sequence is obtained by autoregressive sequence decoding of the dual-domain fused features in combination with an attention prior graph. The model is optimized by performing target loss co-optimization and reverse order modeling training on the predicted forgery operation sequence. The optimized model is then used to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

[0008] Preferably, the feature extraction process includes: inputting the face image to be detected into parallel global feature embedding branches and local feature embedding branches, extracting global structural semantics and local high-frequency details to obtain a first label sequence and a second label sequence, and performing scale adaptive fusion on the first label sequence and the second label sequence to obtain an initial visual label sequence; The initial visual label sequence is input into the parallel spatial domain modeling branch and frequency domain modeling branch respectively to obtain structural semantic features and frequency domain features. The structural semantic features and frequency domain features are dynamically weighted and fused to obtain dual-domain fused features.

[0009] Preferably, the formula for scale-adaptive fusion is: ; In the formula, For the initial visual label sequence, The first labeled sequence, For learnable weights, This is the second marker sequence.

[0010] Preferably, the formula for dynamic weighted fusion is: ; In the formula, As a dual-domain fusion feature, For frequency domain characteristics, For learnable domain weights, These are structural semantic features.

[0011] Preferably, the process of autoregressive sequence decoding includes: mapping the dual-domain fusion features and the forged operation sequence labels to be predicted to a core feature matrix, calculating the sequence relationship features by combining the multi-head attention mechanism enhanced by the attention prior graph, and performing autoregressive prediction to obtain the predicted forged operation sequence.

[0012] Preferably, the formula for mapping to the core feature matrix is: ; In the formula, For querying the matrix, It is a single fully connected layer. The sequence of forgery operations to be predicted is labeled. The key matrix, For value matrices, This is a dual-domain fusion feature.

[0013] Preferably, the formula for calculating the sequence relationship features is: ; In the formula, For the first The sequence relationship features captured by the attention head, It is the first Each attention head corresponds to the transpose of the key matrix. and These are the query submatrix and value submatrix obtained by splitting the attention head, respectively. Attention scaling factor This is an attention prior diagram.

[0014] Preferably, the formula for the attention prior map is: ; In the formula, For attention priors, For learnable parameters, This is a spatial attention weight map. This is the frequency domain attention weight map.

[0015] Preferably, reverse order modeling training refers to inputting the predicted forgery operation sequence into the model in reverse order, so that the model first predicts the last operation executed in the forgery operation sequence, and then gradually backtracks to reason about the preceding operations. The formula for the collaborative optimization of the target loss is: ; In the formula, For the overall optimization function of the model, For sequence decoding cross-entropy loss, and All of these are hyperparameters. This is the image sequence contrast loss.

[0016] A second aspect of this application provides a multi-step sequence deep forgery detection system that integrates the frequency domain, including: a dual-domain feature image encoding module, a dual-domain enhanced sequence decoding module, and an image sequence joint inference module; The dual-domain feature image encoding module is used to acquire the face image to be detected and extract features to obtain dual-domain fusion features; The dual-domain enhanced sequence decoding module is used to construct a dual-domain enhanced sequence decoder based on an autoregressive mechanism. It combines the attention prior map to perform autoregressive sequence decoding on the dual-domain fused features to obtain the predicted forgery operation sequence. The image sequence joint inference module is used to optimize the model by performing target loss co-optimization and reverse order modeling training on the predicted forgery operation sequence. The optimized model is then used to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

[0017] The beneficial effects of this application are as follows: This application provides a multi-step sequence deep forgery detection method and system that integrates the frequency domain. First, combining the characteristics that forgery traces are hidden in the spatial domain but more easily exposed in the frequency domain, dual-domain fusion features of the spatial and frequency domains are extracted, making full use of the complementary advantages of dual-domain information to lay the foundation for accurate extraction of forgery operation sequences in the later stage. Next, a dual-domain enhanced sequence encoder is constructed based on an autoregressive mechanism, and combined with an attention prior map, when decoding each step of the forgery operation, the model is explicitly guided to focus on the spatial tampering region and the frequency anomaly response simultaneously, accurately capturing the fine-grained traces of complex forgery operations and improving the model's recognition accuracy. At the same time, through the dynamic fusion of spatial structure semantics and frequency anomaly patterns by learnable domain weights in the target loss co-optimization, and by adopting the dual-domain enhanced sequence decoding and image sequence joint inference mechanism in the target loss co-optimization, the accurate identification and inference of the multi-step face forgery operation type and its strict execution order are completed, realizing the transformation from simple true or false judgment to operation steps, improving the model's interpretability and semantic understanding ability. Furthermore, by using domain weight regularization to force equalization and utilize dual-domain information, the robustness and generalization ability of the model under complex interferences such as image compression and noise are significantly enhanced, ultimately achieving detection results with high recognition accuracy, good interpretability and strong anti-interference ability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A schematic diagram of the overall process of a multi-step sequence deep forgery detection method with fused frequency domain provided in an embodiment of this application; Figure 2 This is a structural architecture diagram of a multi-step sequence deep forgery detection system with fused frequency domain provided in an embodiment of this application. Detailed Implementation

[0020] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0021] This application provides a multi-step sequence deep forgery detection method and system that integrates the frequency domain. By constructing a dual-domain feature image encoding module that deeply integrates the spatial and frequency domains, a dual-domain enhanced sequence decoding module, and an image sequence joint inference module, fine-grained recognition and inference of the types and execution order of multi-step face forgery operations are achieved. Furthermore, by utilizing adaptive domain weight fusion, attention prior guidance, and reverse order modeling strategies, the detection stability and generalization ability of the model under complex interference environments such as noise and compression are significantly improved.

[0022] Please see Figure 1 The first embodiment of this application provides a multi-step sequence deep forgery detection method in the fused frequency domain, comprising: S1: Obtain the face image to be detected and perform feature extraction to obtain dual-domain fusion features.

[0023] The image of the face to be detected is acquired and input into the dual-domain feature image encoding module for feature extraction, resulting in dual-domain fusion features containing spatial and frequency domains.

[0024] Specifically, firstly, a multi-scale feature embedding mechanism is used to input the face image to be detected. This mechanism includes parallel global feature embedding branches and local feature embedding branches. The global feature embedding branch is used to extract the global structural semantics of the face image to be detected, resulting in a first label sequence. The local feature embedding branch is used to extract local high-frequency details from the face image to be detected, resulting in a second label sequence. Utilizing learnable weights For the first labeled sequence Second marker sequence Scale-adaptive fusion is performed to obtain the initial visual label sequence. The formula is as follows: ; Subsequently, the initial visual marker sequence Input is used in a multi-layer dual-domain fusion Transformer encoder. In each coding layer... In this process, dual-path parallel processing is performed. Specifically, the dual paths include a spatial domain modeling branch and a high-frequency domain modeling branch. In the spatial domain modeling branch, a multi-head self-attention mechanism is used to model the long-range dependencies between all spatial locations to obtain structural semantic features. In the frequency domain modeling branch, the features are first transformed to the frequency domain using a two-dimensional fast Fourier transform (2-D FFT), and anomaly pattern extraction is performed in the frequency domain. Then, an inverse transform (2-D IFFT) is used to transform them back to the spatial domain to obtain the frequency domain features. Finally, through the independent learnable domain weights of this layer. The output of the dual-path (structural semantic features) and frequency domain features Dynamic weighted fusion is performed to generate the dual-domain fusion features of the current layer. The formula is: ; By setting independent learning domain weights, the model can adaptively adjust its dependence on spatial and frequency domain information according to the content of the input image. The final output deep features are sensitive to forgery traces and have strong robustness to common interferences such as compression and noise.

[0025] S2: Construct a dual-domain enhanced sequence decoder based on an autoregressive mechanism, and combine the attention prior map to perform autoregressive sequence decoding on the dual-domain fused features to obtain the predicted forgery operation sequence.

[0026] This application constructs a dual-domain enhanced sequence decoder based on an autoregressive mechanism to infer fine-grained information about forgery operation sequences. The core of this decoder is the fusion of image features in both domains. With the forged operation sequence tag to be predicted Establish a cross-attention mechanism between the two domains to fuse image features. Using the input as input, a sequence of predicted forgery operations is gradually generated through autoregressive sequence decoding.

[0027] While decoding each forged operation sequence marker, the model simultaneously generates a spatial attention weight map indicating key spatial regions. And a frequency domain attention weight map highlighting anomalous frequency responses. Both are achieved through learnable parameters. Merge into a unified attention prior map The formula is: ; To standardize the start and end markers of forgery operation sequences, a start marker [SOS] and an end marker [EOS] are introduced at the beginning and end of the sequence, respectively, to construct complete sequence relationship features. The formula is: ; ; ; In the formula, For a single fully connected layer, the forged operation sequence to be predicted is labeled. The query matrix is ​​obtained through mapping by the fully connected layer. Dual-domain fusion image features The key matrix obtained by transformation through independent fully connected layers Sum matrix , , , These three elements together constitute the core feature matrix of the cross-attention mechanism. For the first Each attention head corresponds to the transpose of the key matrix. and These are the query submatrix and value submatrix obtained by splitting the attention head, respectively. This is the attention scaling factor. For the first The sequence relationship features captured by each attention head, ultimately The sequence relationship features captured by all attention heads are integrated through a splicing operation. The cross-attention mechanism and attention prior map of the size The logarithms are added element by element to enhance the cross-attention effect of the image.

[0028] By injecting this prior image into the cross-attention computation in logarithmic form, the model can be explicitly guided to accurately focus on the spatial region and frequency response most relevant to the operation at each step of decoding. This effectively alleviates the problems of attention diffusion and inference ambiguity caused by the superposition of forgery traces in multiple steps, and generates a clear and accurate sequence of predicted forgery operations.

[0029] S3: The model is optimized by performing multi-objective loss co-optimization and reverse order modeling training on the predicted forgery operation sequence. The optimized model is then used to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

[0030] This application introduces multi-objective loss collaborative optimization (a joint optimization mechanism based on multi-objective loss) and a reverse sequential modeling training strategy to improve the performance and stability of the entire system. Its operation process is divided into two stages: training and inference.

[0031] During the training phase, a reverse-order modeling training strategy is adopted: the predicted fake operation sequence is input into the model in reverse order, so that it prioritizes predicting the last operation executed in the fake operation sequence, and then gradually backtracks to reason about the preceding operations. This reduces the forward dependency coupling within the sequence, allowing the model to focus more on the visual traces introduced by the current operation when decoding, and improves the robustness of sequence reasoning.

[0032] During the inference phase, to avoid performance degradation and overfitting caused by the model's over-reliance on single-domain features, a domain weight regularization constraint is introduced: for the first domain feature... Learnable domain weights of layer encoders Applying L2 regularization forces a balance in information utilization between the image domain and the sequence domain, maintaining the complementarity of features across both domains. The formula is as follows: ; In the formula, This is the loss for domain weight regularization.

[0033] Based on this, a contrast loss is constructed through image sequence contrast learning. In the latent feature space, this approach shortens the distance between genuine matching image features and forged operation sequence features, pushes apart mismatched feature pairs, and strengthens the ability to model the association between visual content and manipulation semantics. The formula is as follows: ; In the formula, Image sequence contrast loss, Let be the expectation operator, representing the expectation of the loss over all samples in the training batch. For cosine similarity calculation, For temperature coefficient, For image features, These represent sequence features that match the image features and sequence features that do not match, respectively.

[0034] Furthermore, sequence decoding cross-entropy loss is employed. This is used to measure the difference between the model-predicted forged operation sequences and the true labeled sequences. In the dual-domain enhanced sequence decoding module, the model uses an autoregressive mechanism to progressively generate predicted forged operation sequences. For each decoding time step... The dual-domain enhanced sequence decoder is based on the currently generated sequence tags. and dual-domain fusion image features The context features are obtained through a cross-attention mechanism, and the probability distribution of the next operation label is output through a linear layer and a softmax function. This distribution is consistent with the true labeled sequence. The cross-entropy loss between them is defined as: ; The overall decoding loss for the entire forgery sequence (including the start marker [SOS] and the end marker [EOS]) is the average of the losses at each time step: ; In the formula, The length of the real forgery operation sequence. This loss function drives the model to accurately predict the type of the current forgery operation based on the dual-domain features of the image at each decoding step, thus providing a basis for subsequent joint optimization.

[0035] Finally, the three types of losses are integrated through multi-objective loss collaborative optimization: sequence decoding cross-entropy loss. Image sequence contrast loss With domain weight regularization loss And through hyperparameters To balance the contributions of each loss, an overall optimization function is constructed to collaboratively update the model parameters. (Model overall optimization function) Defined as: ; Through the overall optimization function of the model The loss of the model is calculated to optimize the model. The optimized model is then used to calculate the sequence of forgery operations, thus obtaining the type and order of the multi-step forgery operations.

[0036] Please see Figure 2 The second embodiment of this application provides a multi-step sequence deep forgery detection system that integrates the frequency domain, including: a dual-domain feature image encoding module, a dual-domain enhanced sequence decoding module, and an image sequence joint inference module.

[0037] The dual-domain feature image encoding module is used to acquire the face image to be detected and extract features to obtain dual-domain fused features.

[0038] The dual-domain enhanced sequence decoding module is used to construct a dual-domain enhanced sequence decoder based on an autoregressive mechanism. It combines the attention prior map to perform autoregressive sequence decoding on the dual-domain fused features to obtain the predicted forgery operation sequence.

[0039] The image sequence joint inference module is used to optimize the model by performing multi-objective loss collaborative optimization and reverse order modeling training on the predicted forgery operation sequence. The optimized model is then used to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

[0040] This application provides a multi-step sequence deep forgery detection method and system that integrates the frequency domain. It can combine the characteristics of forgery traces being hidden in the spatial domain and more easily exposed in the frequency domain, and make full use of the complementary advantages of dual-domain information. By dynamically fusing spatial structure semantics and frequency anomaly patterns through learnable domain weights, and by adopting a dual-domain enhanced sequence decoding and image sequence joint reasoning mechanism, it can accurately identify and infer the types of multi-step face forgery operations and their strict execution order. It effectively utilizes the synergistic effect of dual-domain fusion and sequence modeling to significantly improve fine-grained detection and semantic understanding capabilities, thereby achieving a deep forgery detection effect that is both accurate and interpretable.

[0041] This application has strong generalization ability and robustness, and can be used in various complex interference scenarios for detecting fake content, overcoming the shortcomings of existing methods in terms of single feature representation and weak sequence reasoning ability.

[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0043] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-step sequence deep forgery detection method integrating frequency domain, characterized in that, Includes the following steps: Acquire the face image to be detected and perform feature extraction to obtain dual-domain fusion features; A dual-domain enhanced sequence decoder is constructed based on an autoregressive mechanism. The dual-domain fusion features are then autoregressively decoded using an attention prior map to obtain the predicted forgery operation sequence. The model is optimized by performing target loss co-optimization and reverse order modeling training on the predicted forgery operation sequence. The optimized model is then used to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

2. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 1, characterized in that, The feature extraction process includes: inputting the face image to be detected into parallel global feature embedding branches and local feature embedding branches, extracting global structural semantics and local high-frequency details to obtain a first label sequence and a second label sequence, and performing scale adaptive fusion on the first label sequence and the second label sequence to obtain an initial visual label sequence; The initial visual label sequence is input into the parallel spatial domain modeling branch and frequency domain modeling branch respectively to obtain structural semantic features and frequency domain features. The structural semantic features and the frequency domain features are dynamically weighted and fused to obtain the dual-domain fused features.

3. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 2, characterized in that, The formula for performing the scale-adaptive fusion is: ； In the formula, For the initial visual label sequence, The first labeled sequence, For learnable weights, This is the second marker sequence.

4. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 2, characterized in that, The formula for performing the dynamic weighted fusion is: ； In the formula, As a dual-domain fusion feature, For frequency domain characteristics, For learnable domain weights, These are structural semantic features.

5. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 1, characterized in that, The process of performing the autoregressive sequence decoding includes: mapping the dual-domain fusion features and the forgery operation sequence label to be predicted to a core feature matrix, calculating the sequence relationship features by combining the multi-head attention mechanism enhanced by the attention prior graph, and performing autoregressive prediction to obtain the predicted forgery operation sequence.

6. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 5, characterized in that, The formula for mapping to the core feature matrix is: ； In the formula, For querying the matrix, It is a single fully connected layer. The sequence of forgery operations to be predicted is labeled. The key matrix, For value matrices, This is a dual-domain fusion feature.

7. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 5, characterized in that, The formula for calculating the sequence relationship features is as follows: ； In the formula, For the first The sequence relationship features captured by the attention head, It is the first Each attention head corresponds to the transpose of the key matrix. and These are the query submatrix and value submatrix obtained by splitting the attention head, respectively. Attention scaling factor This is an attention prior diagram.

8. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 1, characterized in that, The formula for the attention prior map is: ； In the formula, For attention priors, For learnable parameters, This is a spatial attention weight map. This is the frequency domain attention weight map.

9. The multi-step sequence deep forgery detection method in the fused frequency domain as described in claim 1, characterized in that, The reverse order modeling training refers to inputting the predicted forgery operation sequence into the model in reverse order, so that the model first predicts the last operation executed in the forgery operation sequence, and then gradually backtracks to reason about the preceding operations. The formula for the collaborative optimization of the target loss is: ； In the formula, For the overall optimization function of the model, For sequence decoding cross-entropy loss, and All of these are hyperparameters. Image sequence contrast loss, This is the loss for domain weight regularization.

10. A multi-step sequence deep forgery detection system integrating frequency domains, applied to the multi-step sequence deep forgery detection method integrating frequency domains as described in any one of claims 1-9, characterized in that, include: Dual-domain feature image encoding module, dual-domain enhanced sequence decoding module, and image sequence joint inference module; The dual-domain feature image encoding module is used to acquire the face image to be detected and perform feature extraction to obtain dual-domain fusion features; The dual-domain enhanced sequence decoding module is used to construct a dual-domain enhanced sequence decoder based on an autoregressive mechanism, and combine the attention prior map to perform autoregressive sequence decoding on the dual-domain fusion features to obtain the predicted forgery operation sequence. The image sequence joint inference module is used to optimize the model by performing target loss co-optimization and reverse order modeling training on the predicted forgery operation sequence, and to use the optimized model to calculate the forgery operation sequence to obtain the type and order of multi-step forgery operations.

Images