Multi-modal deep fake detection method based on frequency domain guided spatio-temporal feature fusion

Abstract

The application discloses a kind of multi-modal deep fake detection methods based on frequency domain guided space-frequency feature fusion, in reasoning process to the input multi-modal deep fake content is executed standardization pretreatment, multiscale space-frequency dual-domain feature extraction, frequency domain guided cross-modal feature fusion, end-to-end model optimization and explainable authenticity discrimination, while completing the visual positioning of fake area, under the premise of guaranteeing detection whole process data controllable, realize the high robust detection of deep fake content, the deep collaborative fusion of space-frequency heterogeneous feature, the accurate visual positioning of fake trace.This method can improve the security, stability and applicability of multi-modal deep fake detection system in resource-limited, heterogeneous multi-modal network scene, taking into account the overall detection accuracy and generalization adaptation demand in complex scene.

Description

Technical Field

[0001] This invention relates to the fields of cyberspace security, artificial intelligence security, and multimodal network intrinsic security, and particularly to a multimodal deep forgery detection method and system based on frequency domain-guided spatial-frequency feature fusion. Background Technology

[0002] With the rapid development of multimodal network technology and the fast iteration of generative artificial intelligence (AIGC), deepfake generation technology, centered on large multimodal models, has matured significantly, capable of generating highly realistic images, videos, and other multimodal digital content that are difficult for the human eye to distinguish. While this technology has achieved positive applications in film and television production, digital content creation, and industrial design, its malicious misuse has also posed serious threats to personal privacy, social trust systems, and cyberspace security. Deepfake detection technology has become a core research direction in the field of cyberspace security and a key underlying technology for ensuring the healthy development of the multimodal network environment.

[0003] However, deepfake detection technologies for multimodal heterogeneous network environments still face significant performance and reliability challenges. Because content in multimodal networks typically undergoes multiple degradation processes such as strong compression, low-resolution downsampling, format transcoding, and cross-modal transmission, microscopic tampering clues in deepfakes are highly susceptible to semantic collapse, leading to a precipitous drop in detection performance. Simultaneously, AIGC generation technologies are continuously iterating, with novel high-fidelity forged content based on diffusion models and large multimodal models generating more subtle artifacts, making traditional detection methods prone to false negatives and missed detections. Furthermore, the heterogeneous content sources and dynamically changing propagation environments in multimodal networks make it difficult for existing detection technologies to simultaneously achieve detection accuracy, scenario robustness, and cross-domain generalization capabilities.

[0004] Existing technologies suffer from two main shortcomings: First, mainstream detection methods rely heavily on single spatial domain features, making it easy to lose forgery clues in scenarios with severe degradation. Existing space-frequency fusion methods generally employ shallow fusion strategies with hard-joining at the end, lacking collaborative adaptation to heterogeneous space-frequency features. This results in core defects such as feature conflicts and semantic dilution, making it difficult to adapt to dynamically changing multimodal network propagation environments. Second, existing detection methods generally suffer from a black-box decision-making problem, only outputting true / false classification results and failing to provide visual evidence of forged regions. The interpretability of detection results is poor, making it difficult to meet the actual needs of auditing and evidence collection in multimodal network environments. Furthermore, the separate processing of robust defense and scenario adaptation capabilities makes it difficult to simultaneously consider both global detection accuracy and the generalization adaptation requirements of heterogeneous scenarios.

[0005] Therefore, there is an urgent need for a multimodal deepfake detection method and system based on frequency domain-guided spatial-frequency feature fusion, so as to achieve robust detection of multimodal deepfake content, deep collaborative fusion of spatial-frequency heterogeneous features, and accurate visualization and localization of forgery traces in multimodal network environments with strong compression degradation, heterogeneous propagation, and unknown AIGC generation algorithms, while taking into account global detection performance and generalization adaptability to complex scenarios. Summary of the Invention

[0006] This invention addresses the core problems of existing deepfake detection methods in multimodal network environments, including low detection accuracy in scenarios with heavy compression and degradation of deepfake content, conflicts in the fusion of heterogeneous spatial-frequency features, weak cross-domain generalization ability, and insufficient interpretability of the decision-making black box. It proposes a multimodal deepfake detection method and system based on frequency-domain guided spatial-frequency feature fusion. During inference, the method performs standardized preprocessing, multi-scale spatial-frequency dual-domain feature extraction, frequency-domain guided cross-modal feature fusion, end-to-end model optimization, and interpretability-based authenticity judgment on the input multimodal deepfake content. Simultaneously, it achieves visual localization of the forged region. While ensuring data controllability throughout the detection process, it achieves highly robust detection of deepfake content, deep collaborative fusion of heterogeneous spatial-frequency features, and accurate visual localization of forgery traces. This method improves the security, stability, and applicability of multimodal deepfake detection systems in resource-constrained and heterogeneous multimodal network scenarios, balancing global detection accuracy with the generalization adaptation requirements of complex scenarios. This system is applied to multimodal network content security governance scenarios consisting of multimodal content input terminals and central detection service nodes. It is suitable for detection scenarios involving strong compression and degradation, heterogeneous cross-domain propagation, coexistence of unknown AIGC generation algorithms, and the presence of hidden forgery traces.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion, comprising the following steps:

[0009] Its characteristic is that it includes the following steps:

[0010] S1. Perform standardized preprocessing on multimodal AIGC content to obtain standardized content data;

[0011] S2. Multi-scale spatial-frequency dual-domain feature extraction: Standardized content data is subjected to multi-scale spatial feature extraction and wavelet frequency domain feature extraction based on sub-band decoupling through spatial and frequency domain branches respectively, to obtain spatial semantic features and frequency prior representations.

[0012] S3. Frequency-domain guided cross-modal feature fusion: The frequency-domain prior representation output by the frequency-domain branch is used as a conditional guiding variable to perform pixel-level reweighting on the spatial semantic features output by the spatial branch, and output cross-modal fusion features.

[0013] S4. Input the cross-modal fusion features into the classification head composed of a multilayer perceptron, and output a scalar probability that the input content is AIGC deep forgery through the Sigmoid activation function.

[0014] Further, step S1 includes:

[0015] For video samples, perform equal-interval frame extraction, extracting a fixed number of keyframes from a single video; for image samples, perform subsequent processing directly.

[0016] An object detection algorithm is used to locate and crop the effective area of ​​the input content, and irrelevant background interference is removed. A bicubic interpolation algorithm is used to uniformly scale the effective content area to a standard resolution of 299×299 to match the input size specification of the backbone network.

[0017] Further, in step S2, the multi-scale spatial feature extraction includes:

[0018] Using the pre-trained Xception network as the backbone, three progressive feature interception points are set at the end of the Entry Flow, the middle of the Middle Flow, and the end of the Exit Flow of the Xception network. These points extract shallow texture features from high resolution with small receptive fields, mid-level structural features from medium resolution with moderate receptive fields, and deep semantic features from low resolution with global receptive fields. These three types of features correspond to different scales of forgery traces generated during the AIGC generation process, such as interpolation noise, local structural distortion, and inconsistency between global illumination and semantics.

[0019] By using 1×1 pointwise convolution to align the channel dimensions of the three types of heterogeneous features, they are uniformly mapped to the same number of channels, eliminating the heterogeneity of feature dimensions and obtaining multi-scale spatial features after channel alignment.

[0020] Furthermore, the channel-aligned multi-scale spatial features are enhanced to obtain enhanced spatial semantic features through bidirectional interactive enhancement, including:

[0021] The attention-guided bidirectional feature interaction algorithm achieves deep fusion of cross-level features through bottom-up and top-down dual-path interaction.

[0022] After bidirectional interaction, the multi-scale features are upsampled to a unified scale, cascaded and aggregated in the channel dimension, and then a convolutional block attention module is introduced. The fused features are recalibrated sequentially through channel attention and spatial attention, and finally the enhanced spatial semantic features are output.

[0023] Furthermore, the attention-guided bidirectional feature interaction algorithm includes:

[0024] The bottom-up path transmits shallow, fine-grained forgery clues to the deep semantic space, while the top-down path provides global content location constraints for shallow features, guiding them to focus on key generated anomalous regions.

[0025] Further, in step S2, the wavelet frequency domain feature extraction based on sub-band decoupling includes:

[0026] First, the preprocessed 299×299 standardized content data is converted into a single-channel luminance map using ITU-R BT.709 standard weights for RGB image / video frames;

[0027] Subsequently, the single-channel luminance map was subjected to Min-Max normalization, which linearly mapped the pixel / amplitude dynamic range to the [0,1] interval;

[0028] Using Haar wavelets as basis functions, a single-level two-dimensional discrete wavelet transform is performed on the preprocessed single-channel data. By alternating convolution and downsampling in the row and column directions using low-pass and high-pass filters, the data is decomposed into four orthogonal frequency sub-bands, including: low-frequency approximate sub-band LL, horizontal high-frequency sub-band LH, vertical high-frequency sub-band HL, and diagonal high-frequency sub-band HH.

[0029] A divide-and-conquer strategy is adopted to extract features independently from the low-frequency and high-frequency sub-bands, thereby obtaining frequency domain prior representations.

[0030] Furthermore, the divide-and-conquer strategy includes:

[0031] For the low-frequency approximate subband LL, a standard residual convolutional block containing nonlinear activation and batch normalization is used for processing. For the horizontal high-frequency subband LH, vertical high-frequency subband HL, and diagonal high-frequency subband HH, they are concatenated along the channel dimension, and feature extraction is performed using a grouped convolutional layer with 3 groups. This allows the high-frequency subbands in the three directions to be processed independently within completely isolated channel subsets, and assigns dedicated directional filters to frequency anomalies in different directions. Finally, the low-frequency and high-frequency features are concatenated along the channel dimension, and cross-band adaptive reweighting is completed through a channel attention mechanism, ultimately outputting a frequency domain prior representation with high-frequency artifact directionality.

[0032] Further, step S3 includes:

[0033] The frequency domain prior representation applies max pooling and average pooling operations in parallel along the channel dimension to capture local significant high-frequency abrupt changes and global smooth frequency distribution drift, respectively. After concatenating the two pooling results along the channel dimension, spatial topology modeling is performed through a two-dimensional convolutional layer with a large receptive field of 7×7 kernel size. A two-dimensional frequency domain spatial prior attention mask with a value range of [0,1] is generated in conjunction with the Sigmoid activation function.

[0034] The frequency domain spatial prior attention mask is extended to the same number of channels as the spatial semantic features, and element-wise Hadamard products are performed with the spatial semantic features to perform pixel-level reweighting of the spatial semantic features.

[0035] The original spatial semantic features are added to the weighted spatial semantic features to finally output cross-modal fusion features.

[0036] Secondly, this invention proposes a multimodal deep forgery detection system based on frequency domain-guided spatial-frequency feature fusion, comprising the following modules:

[0037] The user management module is used to receive user authentication information and detection requests, verify user permissions, and manage user operation logs and access records.

[0038] The image processing module is used to preprocess the input multimodal image data, including image decoding, size normalization, and pixel value standardization.

[0039] The forgery detection module is used to perform the method described in any of the above, guide the fusion of spatial and frequency domain features based on frequency domain features, perform deep forgery discrimination on the image, and output the detection result;

[0040] The record management module is used to store the input image information, detection time, detection results and confidence scores for each detection task, and supports historical query, export and deletion operations of detection records;

[0041] The model management module is used to load, update, and manage the version of deep learning models used for forgery detection, and provides model running status monitoring and anomaly recovery functions.

[0042] Furthermore, the forgery detection module includes the following sub-modules:

[0043] The spatial feature extraction module is used for multi-scale spatial feature extraction to obtain spatial semantic features.

[0044] The frequency domain feature extraction module is used for wavelet frequency domain feature extraction based on sub-band decoupling to obtain frequency domain prior characterization.

[0045] The frequency-domain guided cross-modal feature fusion module is used to take the frequency-domain prior representation output by the frequency-domain branch as a conditional guiding variable, and perform pixel-level reweighting on the spatial semantic features output by the spatial-domain branch to output cross-modal fused features.

[0046] The present invention provides a multimodal deep forgery detection method and system based on frequency domain-guided space-frequency feature fusion, which has the following advantages compared with the prior art:

[0047] (1) This invention constructs a multi-scale spatial-frequency dual-stream parallel feature extraction architecture to perform complementary representation of spatial semantic features and frequency artifact features on the input multimodal content, thereby improving the detection robustness and stability of the system in complex multimodal network environments.

[0048] (2) The present invention adopts a frequency domain-guided high-frequency attention relay fusion mechanism, which, compared with the traditional space-frequency fusion method that relies on hard splicing at the end, helps to achieve deep collaborative adaptation of space-frequency heterogeneous features and improves the effectiveness of multimodal feature fusion.

[0049] (3) By combining wavelet subband decoupling frequency domain feature extraction with bidirectional interactive spatial domain feature enhancement mechanism, this invention can uncover the underlying generation distortion rules of AIGC deepfake technology, thereby improving the cross-domain generalization ability of the model and adapting to the dynamic multimodal network propagation scenario.

[0050] (4) Based on the robust optimization of the global model, this invention further constructs an interpretable reasoning mechanism for defense awareness, which helps to balance the requirements of global detection accuracy and the interpretability of detection results, breaks the black box barrier of deep network decision-making, and is more suitable for the actual deployment of content security governance in multimodal network environments. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is an overall framework diagram of the multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion provided in the embodiments of the present invention.

[0053] Figure 2 This is a schematic diagram of a multimodal deep forgery detection system based on frequency domain-guided spatial-frequency feature fusion, provided in an embodiment of the present invention.

[0054] Figure 3This is a schematic diagram illustrating the key mechanism of the multimodal deep forgery detection method and system based on frequency domain-guided spatial-frequency feature fusion provided in the embodiments of the present invention. Detailed Implementation

[0055] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0056] This invention proposes a multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion. The overall algorithm framework is as follows: Figure 1 As shown, it includes the following steps:

[0057] Step S1: Initialize the detection system and preprocess the multimodal data.

[0058] Let the set of multimodal content to be detected be... It covers modalities such as images and videos, and the number of model training iterations is [number missing]. The initial weights of the frequency-domain guided high-frequency attention detection network FGHA-Net are: For the input multimodal content, modality separation and standardization preprocessing are performed: for video samples, equal-interval frame extraction is performed, extracting a fixed number of keyframes from each video; for image samples, subsequent processing is performed directly. An object detection algorithm is used to locate and crop the effective region of the input content, removing irrelevant background interference. A bicubic interpolation algorithm is used to uniformly scale the effective content region to a standard resolution of 299×299, matching the input size specification of the backbone network. During the training phase, data augmentation operations such as random Gaussian blur, Gaussian noise, image translation and rotation, JPEG compression, and resolution downsampling are introduced to simulate the real content degradation propagation process in multimodal networks, improving the model's generalization robustness. After preprocessing, the standardized content data is input into the spatial and frequency domain dual-branch feature extraction modules as input for subsequent processing.

[0059] Step S2: Multi-scale spatial feature extraction and bidirectional interactive enhancement.

[0060] To capture multi-modal AIGC-generated forgery features across all levels, from micro-texture to macro-semantics, and to address the semantic collapse issue caused by single spatial features in degraded scenarios, the system constructs a multi-scale spatial feature extraction branch. Using a pre-trained Xception network as its backbone, three progressive feature interception points are set at the ends of the Entry Flow, Middle Flow, and Exit Flow of the Xception network. These points extract shallow texture features from high-resolution, small receptive fields, mid-level structural features from medium-resolution, moderately receptive fields, and deep semantic features from low-resolution, global receptive fields, respectively. These three types of features correspond to different scales of forgery traces generated during AIGC generation, such as interpolation noise, local structural distortion, and inconsistencies between global illumination and semantics. A 1×1 pointwise convolution is used to align the channel dimensions of the three heterogeneous features, uniformly mapping them to the same number of channels to eliminate feature dimensional heterogeneity. The aligned feature representation is as follows:

[0061]

[0062] in, The basic features extracted for the l-th interception point This represents the multi-scale spatial features after channel alignment.

[0063] Subsequently, an attention-guided bidirectional feature interaction algorithm was designed. Through bottom-up and top-down dual-path interaction, deep fusion of cross-level features was achieved. The bottom-up path transmits shallow, fine-grained forgery clues to the deep semantic space, while the top-down path provides global content position constraints for shallow features, guiding them to focus on key generated anomaly regions. The multi-scale features after bidirectional interaction were upsampled to a unified scale and cascaded and aggregated along the channel dimension. A convolutional block attention module was then introduced to recalibrate the fused features sequentially through channel attention and spatial attention, automatically strengthening the feature channels and spatial regions carrying forgery clues, suppressing background redundancy interference, and finally outputting enhanced spatial semantic features.

[0064] Step S3: Wavelet frequency domain feature extraction and characterization based on subband decoupling.

[0065] To extract high-purity, high-frequency generated forgery features that are detached from compressed noise and address the issues of insufficient compression resistance and loss of spatial location information in traditional frequency domain methods, a wavelet frequency domain feature extraction branch parallel to the spatial domain branch is constructed. First, the preprocessed 299×299 standardized content data is converted into a single-channel luminance map using ITU-R BT.709 standard weights for RGB image / video frames. This process removes redundant channel information while preserving the amplitude distribution features highly correlated with AIGC-generated forgery traces to the greatest extent possible. Then, the single-channel data undergoes Min-Max normalization, linearly mapping the pixel / amplitude dynamic range to the [0,1] interval, enhancing the numerical stability of the frequency domain transformation. Using the orthogonal and tightly supported Haar wavelet as the basis function, a single-level two-dimensional discrete wavelet transform is performed on the preprocessed single-channel data. Through alternating convolution and downsampling in the row and column directions using low-pass and high-pass filters, the data is decomposed into four orthogonal frequency subbands: the low-frequency approximate subband LL, the horizontal high-frequency subband LH, the vertical high-frequency subband HL, and the diagonal high-frequency subband HH. The decomposition process is represented as follows:

[0066]

[0067] in, For normalized single-channel data, This is a two-dimensional discrete wavelet transform based on Haar wavelets.

[0068] A divide-and-conquer strategy is employed to extract features independently from the low-frequency and high-frequency subbands: For the low-frequency LL subband, a standard residual convolutional block containing nonlinear activation and batch normalization is used to maintain the model's structural robustness under compression degradation scenarios; for the three high-frequency subbands LH, HL, and HH, they are concatenated along the channel dimension, and then feature extraction is performed using a grouped convolutional layer with 3 groups, allowing the three high-frequency subbands to be computed independently within completely isolated channel subsets. Dedicated directional filters are assigned to frequency anomalies in different directions to prevent weak artifacts from canceling each other out and causing feature aliasing during cross-channel fusion. Finally, the low-frequency and high-frequency features are concatenated along the channel dimension, and cross-band adaptive reweighting is performed through a channel attention mechanism, ultimately outputting a frequency domain prior representation with high-frequency artifact directionality. .

[0069] Step S4: Construction of a frequency-domain guided high-frequency attention cross-modal fusion mechanism.

[0070] To abandon the traditional shallow fusion strategy of hard-joining at the end and address the core issues of spatial-frequency feature heterogeneity conflict and semantic dilution, a frequency-domain guided high-frequency attention-guided method is designed to achieve deep fusion of spatial-frequency features in relay mode. This method uses the frequency-domain prior representation output in step S3. As input, max pooling and average pooling operations are applied in parallel along the channel dimension to capture local significant high-frequency abrupt changes and global smooth frequency distribution drift, respectively. The results of the two pooling operations are concatenated along the channel dimension and then spatial topology modeled through a 2D convolutional layer with a large receptive field of 7×7 kernel size. A 2D frequency domain spatial prior attention mask with a value range of [0,1] is generated using the Sigmoid activation function. The calculation formula is as follows:

[0071]

[0072] in, For the Sigmoid activation function, This is the frequency domain prior mask for the generated frequency domain.

[0073] Subsequently, a cross-modal residual fusion mechanism was constructed, using the frequency domain prior mask. Extended to the spatial semantic features output in step S2 With the same number of channels, element-wise Hadamard products are performed with the spatial features to reweight the spatial features at the pixel level. To avoid the gradient vanishing problem in the early stages of deep network training, a residual connection structure is introduced to add the original spatial features to the weighted features, ultimately outputting cross-modal fusion features. The calculation formula is as follows:

[0074]

[0075] in, For Hadamard product operation, The output is the fused feature.

[0076] This mechanism enables early intervention and guidance of high-frequency priors in the frequency domain on spatial semantic features, allowing the spatial network to accurately focus on areas with high incidence of AIGC-generated forgery traces, thus resolving the feature heterogeneity conflict problem in traditional end-point fusion.

[0077] Step S5: Build the overall architecture of the end-to-end modal detection network FGHA-Net.

[0078] Based on the aforementioned dual-branch feature extraction structure and FGHA fusion module, the system constructs an end-to-end frequency-domain guided high-frequency attention deep forgery detection network, FGHA-Net. The network adopts a dual-stream parallel and relay fusion topology. The input consists of standardized preprocessed multimodal content data, which is input in parallel to the multi-scale spatial feature extraction branch and the wavelet frequency domain feature extraction branch. After attention recalibration is completed in the spatial branch, the FGHA module is introduced, using the frequency domain prior representation output from the frequency domain branch as a conditional guiding variable to perform cross-modal recalibration of the spatial features, outputting fused calibrated features. A feature concatenation mechanism is designed at the network end, performing global average pooling on the calibrated spatial fusion features and the original frequency domain prior representation, compressing them into one-dimensional modal feature vectors, and then explicitly concatenating them in the channel dimension to construct global features containing complete spatial-frequency feature information. Finally, the global fusion features are input into a classification head composed of multi-layer perceptrons, and the Sigmoid activation function outputs a probability scalar indicating that the input content is AIGC deep forgery, completing the end-to-end network architecture. The classification output is represented as...

[0079]

[0080] in, For global fusion features, For fully connected classification heads, The predicted probability for input content being a deepfake.

[0081] Step S6: Layered training strategy and robust optimization of the global model.

[0082] To improve the model's convergence stability and generalization performance, the system employs a two-stage hierarchical training strategy to optimize the FGHA-Net network, while designing a loss function adapted to the deepfake binary classification task. Mainstream AIGC deepfake benchmark datasets are selected as training and testing data sources, covering various types of deepfake content generated using mainstream technologies such as GANs, VAEs, diffusion models, and multimodal large models, including lossless, lightly compressed, and heavily compressed quality levels. Considering the characteristics of the deepfake detection binary classification task, the standard binary cross-entropy loss function is used as the objective function for network optimization, quantifying the distribution difference between the network's predicted probabilities and the true labels. The calculation formula is as follows:

[0083]

[0084] Where N is the total number of samples in each training batch. Let be the true label of the i-th sample (0 for true content and 1 for AIGC deepfake content). Let be the probability of the network predicting a forgery for the i-th sample. The two-stage training strategy is as follows: In the first stage, the pre-trained weights of the backbone network are frozen, and only the frequency domain feature extraction module, FGHA fusion module, and classification head are trained, iterating for a fixed number of rounds to ensure the convergence stability of the cross-modal fusion module; In the second stage, the weights of the entire network are unfrozen, and the parameters of the entire network are jointly optimized in an end-to-end manner, iterating for the remaining rounds. At the same time, a learning rate cosine annealing strategy is introduced to dynamically adjust the learning rate, avoid the model getting stuck in local optima, and improve the convergence accuracy and generalization ability of the model. By minimizing the loss function, the learnable parameters of the entire network are updated synchronously using the backpropagation algorithm, driving the network to converge to the optimal decision boundary in the multimodal feature space, and completing the deep optimization of the global model.

[0085] Step S7: Explainable reasoning and fake region localization for defense perception.

[0086] To break down the "black box" barrier of deep network decision-making and meet the practical needs of auditing and evidence collection in multimodal network environments, the system constructs an interpretable reasoning mechanism based on Grad-CAM to achieve accurate visual localization of forged regions. For the trained FGHA-Net model, gradient-weighted class activation mapping (JEM) is introduced to visualize and analyze the fused feature map of the last layer of the network. By calculating the gradient information of the classification result with respect to the fused feature map, the gradient of the target class score with respect to the last layer feature map is generated. The formula for generating the class activation heatmap is as follows:

[0087]

[0088] in, The weight coefficient for the k-th feature channel is... This is the fused feature map of the k-th channel. It is a linear rectification activation function.

[0089] The heatmap is overlaid and rendered on top of the original input content. The more red the color, the higher the model's attention to that area. This accurately locates the core areas where tampering traces are concentrated during AIGC generation, transforming the model's classification decisions into visual physical evidence. The final output includes complete inference results, such as content authenticity labels, forgery confidence scores, tampering area locations, and a visual heatmap.

[0090] Step S8: Full-process detection inference closed loop and multi-scenario adaptation deployment.

[0091] Based on a trained FGHA-Net model, the system constructs an end-to-end multimodal deep forgery detection inference loop, supporting various modal input formats such as single images, batch images, and video files. For each input sample, it automatically executes steps S1 to S7, ultimately outputting a standardized detection report. The report can be exported to PDF format with one click, fully adapting to the engineering application needs of content security auditing in multimodal network environments. The system can flexibly adjust the model size according to the computing power of the deployment environment, supporting lightweight edge deployment and large-scale cloud service deployment. It can be directly embedded into multimodal network deployment systems as a core functional module for intrinsic security protection in multimodal networks. The training and optimization process is repeated until the maximum number of training rounds is reached or the preset convergence accuracy is met. After training, the system outputs the final optimized FGHA-Net global model and standardized detection inference interface. Through the above process, the present invention sequentially performs standardized preprocessing, multi-scale spatial-frequency dual-domain feature extraction, frequency domain-guided cross-modal fusion, model robust optimization, and interpretable reasoning, enabling frequency domain prior information, fusion features, and reliability results to be continuously reused throughout the entire process, thereby forming a closed loop of detection and training that connects the front and back ends, taking into account both global detection accuracy and the generalization adaptation requirements of heterogeneous scenarios.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion, characterized in that, Includes the following steps: S1. Perform standardized preprocessing on multimodal AIGC content to obtain standardized content data; S2. Multi-scale spatial-frequency dual-domain feature extraction: Standardized content data is subjected to multi-scale spatial feature extraction and wavelet frequency domain feature extraction based on sub-band decoupling through spatial and frequency domain branches respectively, to obtain spatial semantic features and frequency prior representations. S3. Frequency-domain guided cross-modal feature fusion: The frequency-domain prior representation output by the frequency-domain branch is used as a conditional guiding variable to perform pixel-level reweighting on the spatial semantic features output by the spatial branch, and output cross-modal fusion features. S4. Input the cross-modal fusion features into the classification head composed of a multilayer perceptron, and output a scalar probability that the input content is AIGC deep forgery through the Sigmoid activation function.

2. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 1, characterized in that, Step S1 includes: For video samples, perform equal-interval frame extraction, extracting a fixed number of keyframes from a single video; for image samples, perform subsequent processing directly. An object detection algorithm is used to locate and crop the effective area of ​​the input content, and irrelevant background interference is removed. A bicubic interpolation algorithm is used to uniformly scale the effective content area to a standard resolution of 299×299 to match the input size specification of the backbone network.

3. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 1, characterized in that, In step S2, the multi-scale spatial feature extraction includes: Using the pre-trained Xception network as the backbone, three progressive feature interception points are set at the end of the Entry Flow, the middle of the Middle Flow, and the end of the Exit Flow of the Xception network. These points extract shallow texture features from high resolution with small receptive fields, mid-level structural features from medium resolution with moderate receptive fields, and deep semantic features from low resolution with global receptive fields. These three types of features correspond to different scales of forgery traces generated during the AIGC generation process, such as interpolation noise, local structural distortion, and inconsistency between global illumination and semantics. By using 1×1 pointwise convolution to align the channel dimensions of the three types of heterogeneous features, they are uniformly mapped to the same number of channels, eliminating the heterogeneity of feature dimensions and obtaining multi-scale spatial features after channel alignment.

4. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 3, characterized in that, The channel-aligned multi-scale spatial features are enhanced to obtain enhanced spatial semantic features through bidirectional interactive enhancement, including: The attention-guided bidirectional feature interaction algorithm achieves deep fusion of cross-level features through bottom-up and top-down dual-path interaction. After bidirectional interaction, the multi-scale features are upsampled to a unified scale, cascaded and aggregated in the channel dimension, and then a convolutional block attention module is introduced. The fused features are recalibrated sequentially through channel attention and spatial attention, and finally the enhanced spatial semantic features are output.

5. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 1, characterized in that, The attention-guided bidirectional feature interaction algorithm includes: The bottom-up path transmits shallow, fine-grained forgery clues to the deep semantic space, while the top-down path provides global content location constraints for shallow features, guiding them to focus on key generated anomalous regions.

6. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 1, characterized in that, In step S2, the wavelet frequency domain feature extraction based on sub-band decoupling includes: First, the preprocessed 299×299 standardized content data is converted into a single-channel luminance map using ITU-R BT.709 standard weights for RGB image / video frames; Subsequently, the single-channel luminance map was subjected to Min-Max normalization, which linearly mapped the pixel / amplitude dynamic range to the [0,1] interval; Using Haar wavelets as basis functions, a single-level two-dimensional discrete wavelet transform is performed on the preprocessed single-channel data. By alternating convolution and downsampling in the row and column directions using low-pass and high-pass filters, the data is decomposed into four orthogonal frequency sub-bands, including: low-frequency approximate sub-band LL, horizontal high-frequency sub-band LH, vertical high-frequency sub-band HL, and diagonal high-frequency sub-band HH. A divide-and-conquer strategy is adopted to extract features independently from the low-frequency and high-frequency sub-bands, thereby obtaining frequency domain prior representations.

7. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 6, characterized in that, The divide-and-conquer strategy includes: For the low-frequency approximate subband LL, a standard residual convolutional block containing nonlinear activation and batch normalization is used for processing. For the horizontal high-frequency subband LH, vertical high-frequency subband HL, and diagonal high-frequency subband HH, they are concatenated along the channel dimension, and feature extraction is performed using a grouped convolutional layer with 3 groups. This allows the high-frequency subbands in the three directions to be processed independently within completely isolated channel subsets, and assigns dedicated directional filters to frequency anomalies in different directions. Finally, the low-frequency and high-frequency features are concatenated along the channel dimension, and cross-band adaptive reweighting is completed through a channel attention mechanism, ultimately outputting a frequency domain prior representation with high-frequency artifact directionality.

8. The multimodal deep forgery detection method based on frequency domain-guided spatial-frequency feature fusion according to claim 1, characterized in that, Step S3 includes: The frequency domain prior representation applies max pooling and average pooling operations in parallel along the channel dimension to capture local significant high-frequency abrupt changes and global smooth frequency distribution drift, respectively. After concatenating the two pooling results along the channel dimension, spatial topology modeling is performed through a two-dimensional convolutional layer with a large receptive field of 7×7 kernel size. A two-dimensional frequency domain spatial prior attention mask with a value range of [0,1] is generated in conjunction with the Sigmoid activation function. The frequency domain spatial prior attention mask is extended to the same number of channels as the spatial semantic features, and element-wise Hadamard products are performed with the spatial semantic features to perform pixel-level reweighting of the spatial semantic features. The original spatial semantic features are added to the weighted spatial semantic features to finally output cross-modal fusion features.

9. A multimodal deep forgery detection system based on frequency domain-guided spatial-frequency feature fusion, characterized in that, include: The user management module is used to receive user authentication information and detection requests, verify user permissions, and manage user operation logs and access records. The image processing module is used to preprocess the input multimodal image data, including image decoding, size normalization, and pixel value standardization. The forgery detection module is used to execute the method as described in any one of claims 1 to 8, guide the fusion of spatial and frequency domain features based on frequency domain features, perform deep forgery discrimination on the image, and output the detection result; The record management module is used to store the input image information, detection time, detection results and confidence scores for each detection task, and supports historical query, export and deletion operations of detection records; The model management module is used to load, update, and manage the version of deep learning models used for forgery detection, and provides model running status monitoring and anomaly recovery functions.

10. The multimodal deep forgery detection system based on frequency domain-guided space-frequency feature fusion according to claim 9, characterized in that, The forgery detection module includes the following sub-modules: The spatial feature extraction module is used for multi-scale spatial feature extraction to obtain spatial semantic features. The frequency domain feature extraction module is used for wavelet frequency domain feature extraction based on sub-band decoupling to obtain frequency domain prior characterization. The frequency-domain guided cross-modal feature fusion module is used to take the frequency-domain prior representation output by the frequency-domain branch as a conditional guiding variable, and perform pixel-level reweighting on the spatial semantic features output by the spatial-domain branch to output cross-modal fused features.

Images