An adaptive deepfake detection method based on modal interaction analysis

Abstract

The present application belongs to the technical field of forgery detection, and specifically relates to a self-adaptive deep forgery detection method based on modal interaction analysis, steps of which comprise: using a pre-trained backbone network as an image encoder to extract spatial feature vectors of different scales from an input image; using interpolation and convolution operations to generate enhanced spatial features; constructing a wavelet transform feature decoupler to obtain high-frequency frequency domain features and low-frequency frequency domain features; establishing a frequency domain guided trade-off mechanism to obtain post-trade-off frequency domain features; constructing a modal interaction quantitative evaluation module to generate an interaction type discrimination signal; constructing a gate routing and expert fusion module to obtain spatial frequency domain fusion features; constructing a main classification head for outputting a prediction result; and using the trained overall model to judge whether the input image is a real image or a fake image based on the spatial frequency domain fusion features. The present application can accurately cope with the diversity and complexity of fake samples, and improve the precision, robustness and interpretability of deep forgery detection.

Description

Technical Field

[0001] This invention belongs to the field of forgery detection technology, specifically relating to an adaptive deep forgery detection method based on modal interaction analysis. Background Technology

[0002] The rapid development of deepfake technology has posed a serious challenge to network security and information authenticity, prompting continuous iteration and updates in detection technologies. Most current mainstream deepfake detection methods combine spatial and frequency domain features for comprehensive judgment, identifying forgery traces by analyzing the differences in image characteristics across both spatial and frequency dimensions. This has improved detection accuracy to some extent.

[0003] However, existing deepfake detection methods have significant limitations in the feature fusion stage. Most existing methods employ fixed fusion strategies, such as simple feature concatenation or attention-based weighted fusion, essentially a "one-size-fits-all" approach. This fixed fusion method struggles to adapt to the diversity and complexity of forged samples, because the relationship between spatial and frequency domain features dynamically changes across different forgery techniques or samples. When faced with complex forged samples exhibiting inconsistent or ambiguous modal signals, spatial and frequency domain features may corroborate each other, conflict, or one may be deceptive. Fixed fusion strategies often fail to effectively handle these complex relationships, are easily misled, and thus limit the accuracy and robustness of detection. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide an adaptive deepfake detection method based on modal interaction analysis. Through adaptive modal interaction analysis and expert fusion mechanism, it can accurately deal with the diversity and complexity of fake samples, significantly improve the accuracy, robustness and interpretability of deepfake detection, and overcome the limitations of the fixed fusion strategy of the prior art.

[0005] To achieve the above objectives, this invention provides an adaptive deepfake detection method based on modal interaction analysis, comprising the following steps:

[0006] S1. For the input image, a pre-trained backbone network is used as an image encoder to extract spatial feature vectors of different scales from the input image, including a batch of shallow spatial features. and three batches of deep space features , , Among them, the deepest spatial features will be... This is denoted as the final depth space feature;

[0007] S2. The shapes of the three batches of deep spatial features are adjusted using interpolation and convolution operations, and then stitched together to generate enhanced spatial features. ;

[0008] S3. Construct a wavelet transform feature decoupler to... Decomposed into high-frequency domain features and low-frequency domain characteristics ;

[0009] S4. Establish a frequency domain guidance and trade-off mechanism. and The importance of each factor is weighed to obtain the frequency domain features after the trade-off. ;

[0010] S5. Construct a modal interaction quantification evaluation module to quantify the interaction relationship between the final depth space features and the trade-off frequency domain features, and generate interaction type discrimination signals. ;

[0011] S6. Construct a gated routing and expert fusion module to... To guide, and Adaptive fusion is performed to obtain spatial-frequency domain fusion features. ;

[0012] S7. Construct the main classification head to output the prediction results;

[0013] S8. Construct a unified loss function and train the overall model consisting of S1-S7;

[0014] S9. Using the trained overall model, based on the spatial frequency domain fusion features, determine whether the input image is a real image or a fake image.

[0015] As a preferred embodiment of the present invention, in S1, for the input image I, feature extraction is performed using a pre-trained Xception backbone network, as follows:

[0016] (1);

[0017] In the formula, This represents the Xception backbone network;

[0018] The scale of the input image I is represented as ,in express The real space of dimension , the rest Similarly, H and W are the height and width of the input image I, respectively; The scale is C represents the number of channels for the feature; the scales of the three batches of deep spatial features are respectively... , , .

[0019] As a preferred embodiment of the present invention, in S2, the process of generating enhanced spatial features involves adjusting the shape of three batches of deep spatial features through interpolation and convolution operations, and then concatenating the adjusted three batches of deep spatial features, as shown below:

[0020] (2);

[0021] In the formula, This indicates the enhanced spatial features after splicing; Indicates that the convolution kernel is Convolution operations; Indicates feature concatenation operation; This indicates an interpolation operation that uses bilinear interpolation to adjust the three batches of deep spatial features to the same size.

[0022] As a preferred embodiment of the present invention, in S3, the process of decomposing the enhanced spatial features into high-frequency domain features and low-frequency domain features is as follows:

[0023] S3.1, Enhancing spatial features of the input using a wavelet transform feature decoupler. The decoupler performs processing, including Haar wavelet transform and convolution operations, for... Each feature channel independently performs a first-order Haar wavelet transform, applying different types of filtering in the horizontal and vertical directions to generate a horizontal low-pass filter component. and a vertical high-pass filter component , represented as:

[0024] (3);

[0025] In the formula, Indicates enhanced spatial features The feature map of the c-th channel, express In position The value at that location, the rest Similarly; , These represent the positions of the c-th channel after transformation. The low-frequency and high-frequency approximation coefficient values ​​at each location Components of a horizontal low-pass filter Each position Composition of vertical high-pass filter components i and j are the row and column indices of the transformed coefficient matrix, respectively.

[0026] S3.2, To and A second round of filtering is performed, further decomposing it into four frequency components, as follows:

[0027] (4);

[0028] In the formula, Indicates to After low-pass filtering, at position Low-frequency information captured at the location, express In position The value at that location, the rest Similarly; Indicates to After performing vertical high-pass filtering, at position High-frequency information captured at the location express In position The value at that location, the rest Similarly; Indicates to After performing a horizontal high-pass filter, at the position High-frequency information captured at the location; Indicates to After performing diagonal high-pass filtering, at position High-frequency information captured at the location;

[0029] Each position Composition of low-frequency components Each position Composition of vertical high frequency components Each position Composition of horizontal high-frequency components Each position Composition of diagonal high-frequency components ;

[0030] S3.3. The three high-frequency components are combined by a concatenation operation, and then point-convolutional low-dimensional mapping is used to obtain the high-frequency and low-frequency features after wavelet decomposition, as follows:

[0031] (5);

[0032] In the formula, , These represent the high-frequency and low-frequency domain characteristics of the wavelet transform feature decoupler output, respectively. Representation layer normalization calculation; Indicates that the convolution kernel is The convolution operation.

[0033] As a preferred embodiment of the present invention, in S4, the frequency domain guided trade-off mechanism includes two independent but complementary attention modules, namely the position attention module and the channel attention module, which act on the spatial dimension and channel dimension of the feature map, respectively.

[0034] For high frequency domain characteristics The calculation steps for the position attention module are as follows:

[0035] S4.1 Input Feature Map Three new feature maps, B, C1, and D, are generated through convolution operations, denoted as follows: ;

[0036] Reassemble the three new feature maps ,in ;

[0037] S4.2 Calculate the positional attention map using the Softmax layer, as follows:

[0038] (6);

[0039] In the formula, This represents the attention weight of position a to position b; N is the number of positions. The feature vector representing position a; The feature vector representing position b;

[0040] S4.3, Utilization right We perform weighted summation to obtain the output:

[0041] (7);

[0042] In the formula, This represents the output feature at position b after weighting by the position attention map; α is a learnable parameter initialized to 0. This represents the feature vector of the input feature at position a; Indicates the bias term;

[0043] each Composition of the overall positional attention map ;

[0044] The calculation steps for the channel attention module are as follows:

[0045] S4.4 Input Feature Map Computing channel attention maps ;

[0046] feature map Reorganized into Then to The matrix is ​​multiplied by its transpose, and then a Softmax layer is applied to obtain the channel attention map. :

[0047] (8);

[0048] In the formula, This represents the attention weight of channel c on channel z; This represents the feature vector of channel c; The eigenvectors of channel z;

[0049] S4.5, Utilization Get the output:

[0050] (9);

[0051] In the formula, β is a learnable parameter, initialized to 0; This represents the feature vector of the input feature at channel c; Indicates the bias term;

[0052] each Composition of the overall positional attention map ;

[0053] S4.6, to and Recombination is performed to obtain a trade-off guided feature map of high-frequency domain features. :

[0054] (10);

[0055] S4.7, Regarding low-frequency domain characteristics Similarly, based on S4.1-S4.6, the trade-off guided feature map of low-frequency domain features is obtained. ;

[0056] S4.8, to and Reorganization is represented as:

[0057] (11);

[0058] In the formula, For the frequency domain features after the trade-off, Conv represents convolution.

[0059] As a preferred embodiment of the present invention, the process of generating the interaction type discrimination signal in S5 is as follows:

[0060] S5.1, The input to the modal interaction quantification evaluation module is: and First, independent unimodal prediction distributions are generated, represented as:

[0061] (12);

[0062] (13);

[0063] In the formula, Softmax represents the normalized exponential function, which converts the output into a probability distribution; GAP represents the global average pooling operation. , Fully connected layer classifiers representing spatial and frequency domain branches, respectively; Indicates correspondence The predicted distribution of the output, Indicates correspondence The predicted distribution of the output, , Both are probability vectors containing two categories: "real" and "fake".

[0064] S5.2, For , The quantification of modal prediction divergence is expressed as:

[0065] (14);

[0066] In the formula, Denotes KL divergence, Indicates JS divergence; M represents the modal prediction divergence of sample x; M represents the mean distribution.

[0067] (15);

[0068] S5.3, For and The intrinsic similarity between spatial and frequency domain features after projection is calculated to obtain the modal feature alignment, which is expressed as:

[0069] (16);

[0070] (17);

[0071] (18);

[0072] In the formula, This represents the modal feature alignment of sample x; , They represent , Feature vector after global average pooling; , Represents a learnable linear projection matrix; , Indicates correspondence , The projected eigenvectors; Represents the L2 norm;

[0073] S5.4. Convert the divergence and alignment degrees into discrete interaction relationship types: corresponding to , Define two binary pointers respectively. , , used to discretize continuous values, is represented as:

[0074] (19);

[0075] (20);

[0076] In the formula, , This indicates the preset threshold for divergence and alignment. This indicates an indicator function that returns 1 if the condition is met, otherwise returns 0.

[0077] Will , Combine into an integer label :

[0078] (twenty one);

[0079] This is the interaction type discrimination signal. This indicates consistent predictions and feature alignment. This indicates that the predictions are consistent, but the features are misaligned. This indicates inconsistent predictions but aligned features. This indicates inconsistent predictions and misaligned features.

[0080] As a preferred embodiment of the present invention, the process of obtaining the spatial frequency domain fusion features in S6 is as follows:

[0081] S6.1, will and As input, it is transformed into a feature vector through global average pooling, and then concatenated, as follows:

[0082] (twenty two);

[0083] In the formula, Concat represents the concatenation operation; This represents the fused vector obtained by concatenating spatial and frequency domain feature vectors;

[0084] Will Input Interaction Gating Network Obtain the raw output of the route. , represented as:

[0085] (twenty three);

[0086] The Softmax function will This is converted into a routing decision vector, which is a four-dimensional probability distribution:

[0087] (twenty four);

[0088] In the formula, The final routing decision vector is a probability distribution with four components. , , , These correspond to the weights that route the samples to the four fusion expert networks, and their sum is 1;

[0089] S6.2 Construct four fusion expert networks with identical structures but independent parameters, optimized for modal interaction relationships. , , , The four fusion expert networks are represented as a set. ;

[0090] Each fusion expert network is a function that takes features from two modalities and outputs a fused feature, represented as:

[0091] (25);

[0092] In the formula, k is the index of the fusion expert network, which can be AA, AM, DA or DM; This represents the fusion feature output by the k-th fusion expert network;

[0093] S6.3, will As a weight, the fusion output of all experts By performing a weighted summation, the final spatial frequency domain fusion feature is obtained. :

[0094] (26);

[0095] (27);

[0096] In the formula, for , , or .

[0097] As a preferred embodiment of the present invention, in S7, the main classification header is a fully connected layer. Its input is and real labels To obtain the final predicted probability. , represented as:

[0098] (28);

[0099] In the formula, Sigmoid represents the activation function.

[0100] As a preferred embodiment of the present invention, the process of constructing the unified loss function in S8 is as follows:

[0101] S8.1 Calculate the main task loss using binary classification cross-entropy. , represented as:

[0102] (29);

[0103] S8.2, For , Calculate the BCE loss for each of the two single-mode branches, and then average them to obtain the auxiliary single-mode loss. , represented as:

[0104] (30);

[0105] (31);

[0106] (32);

[0107] In the formula, Indicates correspondence BCE loss; Indicates correspondence BCE loss; BCE represents the BCE loss function;

[0108] S8.3, Configure gated routing loss :

[0109] (33);

[0110] In the formula, CE represents the cross-entropy loss function;

[0111] Setting Router-Z loss :

[0112] (34);

[0113] In the formula, express The original output of the k-th fusion expert network;

[0114] Set load balancing loss :

[0115] (35);

[0116] In the formula, LoadBalancingLoss represents the load balancing loss function;

[0117] Total interaction gating loss Represented as:

[0118] (36);

[0119] In the formula, , The weight hyperparameters representing the regularization loss;

[0120] S8.4, , , Together they constitute a unified loss function.

[0121] As a preferred embodiment of the present invention, in S9, the output of the main classification head is a continuous probability value between 0 and 1, indicating that the input image is a "fake image". This probability value is compared with a pre-set decision threshold. If the probability value is greater than or equal to the decision threshold, the image is finally determined to be a "fake image"; if it is less than the decision threshold, it is finally determined to be a "real image".

[0122] The beneficial effects of this invention are:

[0123] This invention uses a modal interaction quantification evaluation module to accurately quantify the spatial and frequency domain feature relationships of samples. It introduces a gated routing and expert fusion module to match the most suitable fusion expert network for different samples based on the analysis results, thereby achieving targeted processing and significantly improving the model's detection accuracy for complex and ambiguous forged samples. This effectively solves the problem that fixed fusion strategies in existing technologies are unable to cope with the diversity and complexity of forged samples.

[0124] The multi-head classification and unified loss function constructed in this invention provide comprehensive supervision for model training, ensuring that each module is trained fully and effectively while enabling the model to learn more discriminative feature representations. This further improves the robustness and interpretability of the model and ensures that the model makes stable and reliable decisions when faced with different modal information relationships. Attached Figure Description

[0125] Figure 1 This is a flowchart illustrating the principle of this invention. Detailed Implementation

[0126] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0127] Example 1: As Figure 1 As shown, an adaptive deepfake detection method based on modal interaction analysis includes the following steps:

[0128] S1. For the input image, a pre-trained backbone network is used as an image encoder to extract spatial feature vectors of different scales from the input image, including a batch of shallow spatial features. and three batches of deep space features , , Among them, the deepest spatial features will be... This is denoted as the final depth space feature;

[0129] S2. The shapes of the three batches of deep spatial features are adjusted using interpolation and convolution operations, and then stitched together to generate enhanced spatial features. ;

[0130] S3. Construct a wavelet transform feature decoupler to... Decomposed into high-frequency domain features and low-frequency domain characteristics ;

[0131] S4. Establish a frequency domain guidance and trade-off mechanism. and The importance of each factor is weighed to obtain the frequency domain features after the trade-off. ;

[0132] S5. Construct a modal interaction quantification evaluation module to quantify the interaction relationship between the final depth space features and the trade-off frequency domain features, and generate interaction type discrimination signals. ;

[0133] S6. Construct a gated routing and expert fusion module to... To guide, and Adaptive fusion is performed to obtain spatial-frequency domain fusion features. ;

[0134] S7. Construct the main classification head to output the prediction results;

[0135] S8. Construct a unified loss function and train the overall model consisting of S1-S7;

[0136] S9. Using the trained overall model, based on the spatial frequency domain fusion features, determine whether the input image is a real image or a fake image.

[0137] In S1, for the input image I, feature extraction is performed using the pre-trained Xception backbone network (an existing deep convolutional neural network architecture), as shown below:

[0138] (1);

[0139] In the formula, This represents the Xception backbone network;

[0140] The scale of the input image I is represented as ,in express The real space of dimension , the rest Similarly, H and W are the height and width of the input image I, respectively; The scale is C represents the number of channels for the feature; the scales of the three batches of deep spatial features are respectively... , , .

[0141] In S2, the process of generating enhanced spatial features involves adjusting the shape of three batches of deep spatial features through interpolation and convolution operations, and then concatenating the adjusted three batches of deep spatial features, as shown below:

[0142] (2);

[0143] In the formula, This indicates the enhanced spatial features after splicing; Indicates that the convolution kernel is Convolution operations; Indicates feature concatenation operation; This indicates an interpolation operation that uses bilinear interpolation to adjust the three batches of deep spatial features to the same size.

[0144] In S3, the process of decomposing the enhanced spatial features into high-frequency domain features and low-frequency domain features is as follows:

[0145] S3.1, Enhancing spatial features of the input using a wavelet transform feature decoupler. The decoupler performs processing, including Haar wavelet transform and convolution operations, for... Each feature channel independently performs a first-order Haar wavelet transform, applying different types of filtering in the horizontal and vertical directions to generate a horizontal low-pass filter component. and a vertical high-pass filter component , represented as:

[0146] (3);

[0147] In the formula, Indicates enhanced spatial features The feature map of the c-th channel, express In position The value at that location, the rest Similarly; , These represent the positions of the c-th channel after transformation. The low-frequency and high-frequency approximation coefficient values ​​at each location Components of a horizontal low-pass filter Each position Composition of vertical high-pass filter components i and j are the row and column indices of the transformed coefficient matrix, respectively.

[0148] S3.2, To and A second round of filtering is performed, further decomposing it into four frequency components, as follows:

[0149] (4);

[0150] In the formula, Indicates to After low-pass filtering, at position Low-frequency information captured at the location, express In position The value at that location, the rest Similarly; Indicates to After performing vertical high-pass filtering, at position High-frequency information captured at the location express In position The value at that location, the rest Similarly; Indicates to After performing a horizontal high-pass filter, at the position High-frequency information captured at the location; Indicates to After performing diagonal high-pass filtering, at position High-frequency information captured at the location;

[0151] Each position Composition of low-frequency components Each position Composition of vertical high frequency components Each position Composition of horizontal high-frequency components Each position Composition of diagonal high-frequency components ;

[0152] S3.3. The three high-frequency components are combined by a concatenation operation, and then point-convolutional low-dimensional mapping is used to obtain the high-frequency and low-frequency features after wavelet decomposition, as follows:

[0153] (5);

[0154] In the formula, , These represent the high-frequency and low-frequency domain characteristics of the wavelet transform feature decoupler output, respectively. Representation layer normalization calculation; Indicates that the convolution kernel is The convolution operation.

[0155] In S4, the frequency domain guided trade-off mechanism includes two independent but complementary attention modules: the position attention module and the channel attention module, which act on the spatial dimension and channel dimension of the feature map, respectively.

[0156] For high frequency domain characteristics The calculation steps for the position attention module are as follows:

[0157] S4.1 Input Feature Map Three new feature maps, B, C1, and D, are generated through convolution operations, denoted as follows: ;

[0158] Reassemble the three new feature maps ,in ;

[0159] S4.2 Calculate the positional attention map using the Softmax layer, as follows:

[0160] (6);

[0161] In the formula, This represents the attention weight of position a to position b; N is the number of positions. The feature vector representing position a; The feature vector representing position b;

[0162] S4.3, Utilization right We perform weighted summation to obtain the output:

[0163] (7);

[0164] In the formula, This represents the output feature at position b after weighting by the position attention map; α is a learnable parameter initialized to 0. This represents the feature vector of the input feature at position a; Indicates the bias term;

[0165] each Composition of the overall positional attention map ;

[0166] The calculation steps for the channel attention module are as follows:

[0167] S4.4 Input Feature Map Computing channel attention maps ;

[0168] feature map Reorganized into Then to The matrix is ​​multiplied by its transpose, and then a Softmax layer is applied to obtain the channel attention map. :

[0169] (8);

[0170] In the formula, This represents the attention weight of channel c on channel z; This represents the feature vector of channel c; The eigenvectors of channel z;

[0171] S4.5, Utilization Get the output:

[0172] (9);

[0173] In the formula, β is a learnable parameter, initialized to 0; This represents the feature vector of the input feature at channel c; Indicates the bias term;

[0174] each Composition of the overall positional attention map ;

[0175] S4.6, to and Recombination is performed to obtain a trade-off guided feature map of high-frequency domain features. :

[0176] (10);

[0177] S4.7, Regarding low-frequency domain characteristics Similarly, based on S4.1-S4.6, the trade-off guided feature map of low-frequency domain features is obtained. ;

[0178] S4.8, to and Reorganization is represented as:

[0179] (11);

[0180] In the formula, For the frequency domain features after the trade-off, Conv represents convolution.

[0181] In S5, the process of generating the interaction type discrimination signal is as follows:

[0182] S5.1 The Modal Interaction Quantization Evaluation Module first generates independent single-modal prediction distributions, constructing independent classification paths for spatial and frequency domain features respectively, to obtain their respective prediction probability distributions for the forgery detection task. The input to the Modal Interaction Quantization Evaluation Module is... and First, independent unimodal prediction distributions are generated, represented as:

[0183] (12);

[0184] (13);

[0185] In the formula, Softmax represents the normalized exponential function, which converts the output into a probability distribution; GAP represents the global average pooling operation. , Fully connected layer classifiers representing spatial and frequency domain branches, respectively; Indicates correspondence The predicted distribution of the output, Indicates correspondence The predicted distribution of the output, , Both are probability vectors containing two categories: "real" and "fake".

[0186] S5.2, For , To quantify the modal prediction divergence, the difference between two unimodal prediction distributions is calculated to quantify their consistency in prediction opinions, expressed as:

[0187] (14);

[0188] In the formula, KL divergence is used to measure the difference between two probability distributions. Indicates JS divergence; M represents the modal prediction divergence of sample x; M represents the mean distribution.

[0189] (15);

[0190] S5.3, For and The intrinsic similarity between spatial and frequency domain features after projection is calculated to obtain the modal feature alignment, which is expressed as:

[0191] (16);

[0192] (17);

[0193] (18);

[0194] In the formula, This represents the modal feature alignment of sample x, with a value range of [-1, 1]. The closer the value is to 1, the higher the alignment. , They represent , Feature vector after global average pooling; , Represents a learnable linear projection matrix used to map two feature vectors to a shared space of the same dimension; , Indicates correspondence , The projected eigenvectors; Represents the L2 norm;

[0195] S5.4. Based on a preset threshold, the continuous divergence and alignment indices are converted into discrete interaction relationship types, which serve as the final discrimination signal: corresponding to , Define two binary pointers respectively. , , used to discretize continuous values, is represented as:

[0196] (19);

[0197] (20);

[0198] In the formula, , These represent the preset thresholds for divergence and alignment, and are hyperparameters. This indicates an indicator function that returns 1 if the condition is met, otherwise returns 0. This indicates the consistency indicator of predictions (1 = consistent, 0 = inconsistent). Indicates the feature alignment indicator (1 = aligned, 0 = unaligned).

[0199] Will , Combine into an integer label :

[0200] (twenty one);

[0201] This is the interaction type discrimination signal. This indicates that predictions are consistent and features are aligned (AA). This indicates that the predictions are consistent but the features are misaligned (AM). This indicates inconsistent predictions but aligned features (DA). This indicates inconsistent predictions and misaligned features (DM).

[0202] In S6, the core function of the gated routing and adaptive expert fusion module is to dynamically select the most suitable expert fusion network to perform feature fusion based on the modal interaction type evaluated by S5, thereby achieving adaptive and targeted information processing. It typically consists of one or more fully connected layers. The process of obtaining spatial-frequency domain fusion features is as follows:

[0203] S6.1, will and As input, it is transformed into a feature vector through global average pooling, and then concatenated, as follows:

[0204] (twenty two);

[0205] In the formula, Concat represents the concatenation operation; This represents the fused vector obtained by concatenating spatial and frequency domain feature vectors;

[0206] Will Input Interaction Gating Network Obtain the raw output (logits) of the route. , represented as:

[0207] (twenty three);

[0208] An interactive gating network is a decision network that receives features from two modalities and outputs a probability distribution indicating the probability that the current sample should be assigned to each of the four fusion expert networks. This is achieved through the Softmax function. This is converted into a routing decision vector, which is a four-dimensional probability distribution:

[0209] (twenty four);

[0210] In the formula, The final routing decision vector is a probability distribution with four components. , , , These correspond to the weights that route the samples to the four fusion expert networks, and their sum is 1;

[0211] S6.2 Construct four fusion expert networks with identical structures but independent parameters, optimized for modal interaction relationships. , , , (The internal implementation can be any existing fusion structure, such as cross-attention mechanism, bidirectional gating fusion, etc.), representing the four fusion expert networks as a set. ;

[0212] Each fusion expert network is a function that takes features from two modalities and outputs a fused feature, represented as:

[0213] (25);

[0214] In the formula, k is the index of the fusion expert network, which can be AA, AM, DA or DM; This represents the fusion feature output by the k-th fusion expert network. During training, the outputs of all experts (i.e., fusion expert networks) are calculated.

[0215] S6.3, will As a weight, the fusion output of all experts By performing a weighted summation, the final spatial frequency domain fusion feature is obtained. :

[0216] (26);

[0217] (27);

[0218] In the formula, for , , or . It dynamically integrates the wisdom of all experts, with the expert who makes the greatest contribution usually having the highest weight assigned by the gating network.

[0219] Alternatively, you can select only the expert with the highest weight for calculation to obtain... .

[0220] In S7, the main classification head is a fully connected layer. Its input is and real labels To obtain the final predicted probability. , represented as:

[0221] (28);

[0222] In the formula, Sigmoid represents the activation function.

[0223] In S8, the process of constructing a unified loss function is as follows:

[0224] S8.1 Calculate the main task loss using binary classification cross-entropy. , represented as:

[0225] (29);

[0226] S8.2, Auxiliary Single-Mode Loss, is used to independently supervise the classification capabilities of both the spatial and frequency domain branches, ensuring that each mode learns discriminative features and thus playing a regularization role. For , Calculate the BCE loss for each of the two single-mode branches, and then average them to obtain the auxiliary single-mode loss. , represented as:

[0227] (30);

[0228] (31);

[0229] (32);

[0230] In the formula, Indicates correspondence BCE loss; Indicates correspondence BCE loss; BCE represents the BCE loss function (binary cross-entropy loss).

[0231] S8.3, Configure gated routing loss :

[0232] (33);

[0233] In the formula, CE represents the cross-entropy loss function;

[0234] Setting Router-Z loss :

[0235] (34);

[0236] In the formula, express The original output of the k-th fusion expert network;

[0237] Set load balancing loss This encourages the gating network to distribute data roughly evenly among the four experts:

[0238] (35);

[0239] In the formula, LoadBalancingLoss represents the load balancing loss function;

[0240] To ensure expert load balancing and training stability, two regularization terms are introduced. Total interaction gating loss. Represented as:

[0241] (36);

[0242] In the formula, , The weight hyperparameters representing the regularization loss;

[0243] S8.4, , , Together they constitute a unified loss function.

[0244] In S9, the output of the main classification head is a continuous probability value between 0 and 1, indicating that the input image is a "fake image". This probability value is compared with a pre-set decision threshold. If the probability value is greater than or equal to the decision threshold, the image is finally determined to be a "fake image"; if it is less than the decision threshold, it is finally determined to be a "real image".

[0245] Based on the above description, the judgment process for any new image to be detected is as follows:

[0246] First, the input image to be detected is fed into the pre-trained model. The image sequentially passes through feature extraction and processing modules S1 to S4, generating its spatial and frequency domain features. Next, the modal interaction quantization evaluation module S5 analyzes the intrinsic relationship between these two features and generates an interaction type signal. Then, the interaction gating network S6 uses this signal to perform a weighted summation of the fusion outputs of all experts, obtaining the final spatial-frequency domain fusion feature. This generates a final, comprehensive spatial-frequency domain fusion feature (or alternatively, the most suitable expert for processing the current image feature relationships can be dynamically selected and activated from four dedicated fusion expert networks, and this selected expert network performs deep fusion of spatial and frequency domain features).

[0247] Then, this final fused feature is fed into the main classification head. The main classification head analyzes and processes this feature and outputs a continuous probability value indicating that the image is a "fake image". This probability value is between 0 and 1.

[0248] Finally, this probability value is compared with a pre-set decision threshold (usually 0.5). If the probability value is greater than or equal to the decision threshold, the image is ultimately determined to be a "fake image"; if it is less than the threshold, it is ultimately determined to be a "real image".

[0249] During the inference phase, the model's network parameters remain fixed, and no loss function is calculated. All auxiliary modules used only for training, such as the unimodal classification in S5.1 and the interactive gating loss supervision part in S8.3, are inactive. The final decision depends entirely on the output of the main classification head.

[0250] Example 2: An adaptive deepfake detection device based on modal interaction analysis, comprising:

[0251] One or more processors;

[0252] Memory, used to store one or more computer programs;

[0253] When one or more programs are executed by one or more processors, the one or more processors execute the method in Example 1.

[0254] Example 3: A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method in Example 1.

Claims

1. An adaptive deepfake detection method based on modal interaction analysis, characterized in that... Includes the following steps: S1. For the input image, a pre-trained backbone network is used as an image encoder to extract spatial feature vectors of different scales from the input image, including a batch of shallow spatial features. and three batches of deep space features , , Among them, the deepest spatial features will be... This is denoted as the final depth space feature; S2. The shapes of the three batches of deep spatial features are adjusted using interpolation and convolution operations, and then stitched together to generate enhanced spatial features. ; S3. Construct a wavelet transform feature decoupler to... Decomposed into high-frequency domain features and low-frequency domain characteristics ; S4. Establish a frequency domain guidance and trade-off mechanism. and The importance of each factor is weighed to obtain the frequency domain features after the trade-off. ; S5. Construct a modal interaction quantification evaluation module to quantify the interaction relationship between the final depth space features and the trade-off frequency domain features, and generate interaction type discrimination signals. ; S6. Construct a gated routing and expert fusion module to... To guide, and Adaptive fusion is performed to obtain spatial-frequency domain fusion features. ; S7. Construct the main classification head to output the prediction results; S8. Construct a unified loss function and train the overall model consisting of S1-S7; S9. Using the trained overall model, based on the spatial frequency domain fusion features, determine whether the input image is a real image or a fake image.

2. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S1, for the input image I, feature extraction is performed using a pre-trained Xception backbone network, as shown below: （1）； In the formula, This represents the Xception backbone network; The scale of the input image I is represented as ,in express The real space of dimension , the rest Similarly, H and W are the height and width of the input image I, respectively; The scale is C represents the number of channels for the feature; the scales of the three batches of deep spatial features are respectively... , , .

3. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S2, the process of generating enhanced spatial features involves adjusting the shape of three batches of deep spatial features through interpolation and convolution operations, and then concatenating the adjusted three batches of deep spatial features, as shown below: （2）； In the formula, This indicates the enhanced spatial features after splicing; Indicates that the convolution kernel is Convolution operations; Indicates feature concatenation operation; This indicates an interpolation operation that uses bilinear interpolation to adjust the three batches of deep spatial features to the same size.

4. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S3, the process of decomposing the enhanced spatial features into high-frequency domain features and low-frequency domain features is as follows: S3.1, Enhancing spatial features of the input using a wavelet transform feature decoupler. The decoupler performs processing, including Haar wavelet transform and convolution operations, for... Each feature channel independently performs a first-order Haar wavelet transform, applying different types of filtering in the horizontal and vertical directions to generate a horizontal low-pass filter component. and a vertical high-pass filter component , represented as: （3）； In the formula, Indicates enhanced spatial features The feature map of the c-th channel, express In position The value at that location, the rest Similarly; , These represent the positions of the c-th channel after transformation. The low-frequency and high-frequency approximation coefficient values ​​at each location Components of a horizontal low-pass filter Each position Composition of vertical high-pass filter components i and j are the row and column indices of the transformed coefficient matrix, respectively. S3.2, To and A second round of filtering is performed, further decomposing it into four frequency components, as follows: （4）； In the formula, Indicates to After low-pass filtering, at position Low-frequency information captured at the location, express In position The value at that location, the rest Similarly; Indicates to After performing vertical high-pass filtering, at position High-frequency information captured at the location express In position The value at that location, the rest Similarly; Indicates to After performing a horizontal high-pass filter, at the position High-frequency information captured at the location; Indicates to After performing diagonal high-pass filtering, at position High-frequency information captured at the location; Each position Composition of low-frequency components Each position Composition of vertical high frequency components Each position Composition of horizontal high frequency components Each position Composition of diagonal high-frequency components ; S3.

3. The three high-frequency components are combined by a concatenation operation, and then point convolution low-dimensional mapping is used to obtain the high-frequency and low-frequency domain features after wavelet decomposition, as follows: （5）； In the formula, , These represent the high-frequency and low-frequency domain characteristics of the wavelet transform feature decoupler output, respectively. Representation layer normalization calculation; Indicates that the convolution kernel is The convolution operation.

5. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S4, the frequency domain guided trade-off mechanism includes two independent but complementary attention modules, namely the position attention module and the channel attention module, which act on the spatial dimension and channel dimension of the feature map, respectively. For high frequency domain characteristics The calculation steps for the position attention module are as follows: S4.1 Input Feature Map Three new feature maps, B, C1, and D, are generated through convolution operations, denoted as follows: ; Reassemble the three new feature maps ,in ; S4.2 Calculate the positional attention map using the Softmax layer, as follows: （6）； In the formula, This represents the attention weight of position a to position b; N is the number of positions. The feature vector representing position a; The feature vector representing position b; S4.3, Utilization right We perform weighted summation to obtain the output: （7）； In the formula, This represents the output feature at position b after weighting by the position attention map; α is a learnable parameter, initialized to 0; This represents the feature vector of the input feature at position a; Indicates the bias term; each Composition of the overall positional attention map ; The calculation steps for the channel attention module are as follows: S4.4 Input Feature Map Computing channel attention maps ; feature map Reorganized into Then to The matrix is ​​multiplied by its transpose, and then a Softmax layer is applied to obtain the channel attention map. : （8）； In the formula, This represents the attention weight of channel c on channel z; This represents the feature vector of channel c; The eigenvectors of channel z; S4.5, Utilization Get the output: （9）； In the formula, β is a learnable parameter, initialized to 0; This represents the feature vector of the input feature at channel c; Indicates the bias term; each Composition of the overall positional attention map ; S4.6, to and Recombination is performed to obtain a trade-off guided feature map of high-frequency domain features. : （10）； S4.7, Regarding low-frequency domain characteristics Similarly, based on S4.1-S4.6, the trade-off guided feature map of low-frequency domain features is obtained. ; S4.8, to and Reorganization is represented as: （11）； In the formula, For the frequency domain features after the trade-off, Conv represents convolution.

6. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S5, the process of generating the interaction type discrimination signal is as follows: S5.1, The input to the modal interaction quantification evaluation module is: and First, generate independent unimodal prediction distributions, represented as: （12）； （13）； In the formula, Softmax represents the normalized exponential function, which converts the output into a probability distribution; GAP represents the global average pooling operation. , Fully connected layer classifiers representing spatial and frequency domain branches, respectively; Indicates correspondence The predicted distribution of the output, Indicates correspondence The predicted distribution of the output, , Both are probability vectors containing two categories: "real" and "fake". S5.2, For , The quantification of modal prediction divergence is expressed as: （14）； In the formula, Denotes KL divergence, Indicates JS divergence; M represents the modal prediction divergence of sample x; M represents the mean distribution. （15）； S5.3, For and The intrinsic similarity between spatial and frequency domain features after projection is calculated to obtain the modal feature alignment, which is expressed as: （16）； （17）； （18）； In the formula, This represents the modal feature alignment of sample x; , They represent , Feature vector after global average pooling; , Represents a learnable linear projection matrix; , Indicates correspondence , The projected eigenvectors; Represents the L2 norm; S5.

4. Convert the divergence and alignment degrees into discrete interaction relationship types: corresponding to , Define two binary pointers respectively. , , used to discretize continuous values, is represented as: （19）； （20）； In the formula, , This indicates the preset threshold for divergence and alignment. This indicates an indicator function that returns 1 if the condition is met, otherwise returns 0. Will , Combine into an integer label : （21）； This is the interaction type discrimination signal. This indicates consistent predictions and feature alignment. This indicates that the predictions are consistent, but the features are misaligned. This indicates inconsistent predictions but aligned features. This indicates inconsistent predictions and misaligned features.

7. The adaptive deepfake detection method based on modal interaction analysis according to claim 6, characterized in that, In S6, the process of obtaining the spatial frequency domain fusion features is as follows: S6.1, will and As input, it is transformed into a feature vector through global average pooling, and then concatenated, as follows: （22）； In the formula, Concat represents the concatenation operation; This represents the fused vector obtained by concatenating spatial and frequency domain feature vectors; Will Input Interaction Gating Network Obtain the raw output of the route. , represented as: （23）； The Softmax function will This is converted into a routing decision vector, which is a four-dimensional probability distribution: （24）； In the formula, The final routing decision vector is a probability distribution with four components. , , , These correspond to the weights that route the samples to the four fusion expert networks, and their sum is 1; S6.2 Construct four fusion expert networks with identical structures but independent parameters, optimized for modal interaction relationships. , , , The four fusion expert networks are represented as a set. ; Each fusion expert network is a function that takes features from two modalities and outputs a fused feature, represented as: （25）； In the formula, k is the index of the fusion expert network, which can be AA, AM, DA or DM; This represents the fusion feature output by the k-th fusion expert network; S6.3, will As a weight, the fusion output of all experts By performing a weighted summation, the final spatial frequency domain fusion feature is obtained. : （26）； （27）； In the formula, for , , or .

8. The adaptive deepfake detection method based on modal interaction analysis according to claim 7, characterized in that, In S7, the main classification header is a fully connected layer. Its input is and real labels To obtain the final predicted probability. , represented as: （28）； In the formula, Sigmoid represents the activation function.

9. The adaptive deepfake detection method based on modal interaction analysis according to claim 8, characterized in that, In S8, the process of constructing the unified loss function is as follows: S8.1 Calculate the main task loss using binary classification cross-entropy. , represented as: （29）； S8.2, For , Calculate the BCE loss for each of the two single-mode branches, and then average them to obtain the auxiliary single-mode loss. , represented as: （30）； （31）； （32）； In the formula, Indicates correspondence BCE loss; Indicates correspondence BCE loss; BCE represents the BCE loss function; S8.3, Configure gated routing loss : （33）； In the formula, CE represents the cross-entropy loss function; Setting Router-Z loss : （34）； In the formula, express The original output of the k-th fusion expert network; Set load balancing loss : （35）； In the formula, LoadBalancingLoss represents the load balancing loss function; Total interaction gating loss Represented as: （36）； In the formula, , The weight hyperparameters representing the regularization loss; S8.4, , , Together they constitute a unified loss function.

10. The adaptive deepfake detection method based on modal interaction analysis according to claim 1, characterized in that, In S9, the output of the main classification head is a continuous probability value between 0 and 1, indicating that the input image is a "fake image". This probability value is compared with a pre-set decision threshold. If the probability value is greater than or equal to the decision threshold, the image is finally determined to be a "fake image". If the image is less than the decision threshold, it is ultimately determined to be a "real image".

Images