A facial deepfake detection method based on enhanced dual-branch fusion model

Abstract

This invention discloses a face deepfake detection method based on an enhanced dual-branch fusion model. The invention employs a novel enhanced dual-branch fusion model to improve the robustness and generalization ability of CNN-based face deepfake detectors. First, high-frequency RGB noise in the face image is enhanced to extract its anomalous features, followed by preservation fusion. Specifically, a depthwise separable convolution module is used to improve model performance when extracting image features. A selective kernel module is also used to adaptively extract more representative noise features by dynamically adjusting the convolution kernel. Furthermore, this invention designs a multi-scale channel spatial attention fusion module, effectively fusing feature information from various parts, thereby reducing model overfitting and enhancing the model's robustness and generalization ability. Finally, comprehensive evaluation on multiple benchmark datasets confirms that this method significantly improves robustness and generalization.

Description

Technical Field

[0001] This invention belongs to the field of deepfake face detection, specifically relating to a deepfake face detection method based on an enhanced dual-branch fusion model. Background Technology

[0002] With the development of deepfake technology, the generated fake images and videos are visually almost indistinguishable from real content, posing a serious threat to personal privacy and public safety. Therefore, deepfake detection technology has become an important direction in information security research. Current detection methods are mainly divided into two categories: feature engineering-based and deep learning-based. The former relies on manually designed feature extraction, such as facial texture anomalies and inconsistent lighting, to identify signs of forged images; while the latter automatically learns image features through convolutional neural networks (CNNs), demonstrating higher accuracy. However, despite the existence of various methods, deepfake detection still faces problems of insufficient robustness and poor generalization ability, especially when faced with datasets generated by different forgery methods, where detection performance is often affected. To address the challenge of insufficient generalization ability, researchers have gradually explored more refined detection methods, such as two-branch fusion strategies, transfer learning, and multi-task learning. These methods aim to improve the model's adaptability across different datasets. The two-branch fusion method, by combining information from different feature extraction branches, can more comprehensively capture multi-level features in forged images, thereby improving detection accuracy and robustness. This strategy typically combines spatial and frequency domain information to more effectively identify the possibility of forgery.

[0003] In recent years, researchers have also begun to focus on extracting frequency domain information, such as using Discrete Cosine Transform (DCT) and Fourier Transform to enhance face classification capabilities. However, many existing methods fail to fully exploit the interactions between features in normal images, which limits detection performance. Previous dual-branch structures used the same processing method for the RGB and high-frequency noise branches, causing feature extraction to focus on regions unrelated to forgery detection, further affecting generalization ability. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a face deepfake detection method based on an enhanced dual-branch fusion model, thereby improving the robustness and generalization ability of CNN-based face deepfake detectors.

[0005] In a first aspect, the present invention provides a method for detecting deepfake faces based on an enhanced dual-branch fusion model, the method comprising the following steps:

[0006] Step 1: Construct the training set

[0007] Align the tagged fake face videos with the original videos and preprocess them;

[0008] Step 2: Build an enhanced dual-branch fusion model and train it using the training set;

[0009] The enhanced dual-branch fusion model EDFM includes a parallel RGB branch, a high-frequency noise branch, and a multi-scale branch fusion module BFM.

[0010] The RGB branch includes a depthwise separable convolution module (DSCM) and a cross-modal consistency enhancement module (CMCE).

[0011] The high-frequency noise branch includes a spatially rich model (SRM), a selective kernel module (SKM), and a cross-modal consistency enhancement module (CMCE) shared with the RGB branch.

[0012] Step 3: Test the trained enhanced two-branch fusion model using multiple face spoofing datasets, and detect the focus points of the trained enhanced two-branch fusion model during testing, obtaining only the results of detecting true and false faces; finally, use the trained and tested enhanced two-branch fusion model to achieve face spoofing detection.

[0013] Preferably, the preprocessing in step one involves cropping the face regions of the fake face video and the original video, uniformly normalizing them, and enhancing the image.

[0014] Preferably, the input image for the RGB branch in step two is the original image that still needs to be processed by the shallow part of the backbone network of an extremely deep separable convolutional network;

[0015] The features processed by the Spatial Rich Model (SRM) still need to be processed using the shallow part of the backbone network of the Extremely Deep Separable Convolutional Network to extract low-level features from the input image.

[0016] The shallow portion of the backbone network of the extremely deep separable convolutional network includes two sets of convolutional layers, a normalization layer, and a ReLU activation function.

[0017] Preferably, the depth-separable convolutional module (DSCM) in step two includes a series of sequentially connected two-dimensional convolutional layers, a normalization layer, a LeakyReLU activation layer, and a pointwise convolutional layer.

[0018] Preferably, the specific steps of the Spatial Enrichment Model (SRM) in step two are as follows:

[0019] 1) Perform multiple filtering operations on the image to extract the image residual;

[0020] 4) Image residuals are convolved with filters of different directions and scales to obtain multiple residual sub-models;

[0021] 5) The residuals of the above residual sub-model are statistically analyzed to extract high-order statistical features and form a vector with high-frequency features.

[0022] Preferably, the specific implementation process of the selective core module SKM in step two is as follows:

[0023] The output feature X of the SRM is processed by four convolutional kernels of different sizes. s The process involves extracting feature maps from receptive fields of different sizes; then, the outputs of four convolutional kernels of different sizes are summed to obtain the fused feature map X. f ;

[0024] Subsequently, the fused feature map X f After processing with a global average pooling layer and a fully connected layer, the dimensionality-reduced feature map X is obtained. d Then for X d Each layer is processed through a fully connected layer, and the concatenation is followed by Softmax activation and normalization to obtain an attention vector. This attention vector is then applied to the corresponding feature map X. d The final output X is obtained by weighted summation of all feature maps. skm .

[0025] Preferably, the specific implementation process of the cross-modal consistency enhancement module CMCE in step two is as follows:

[0026] The output F of the depthwise separable convolutional module (DSCM) r and the output F of the selective core module SKM n Calculate the cross-modal consistency graph using element-wise inner product;

[0027] Further, the cross-modal consistency fusion feature map F is obtained. r 'and F n '.

[0028] Preferably, the output of the cross-modal consistency enhancement module CMCE in step two also needs to use two cascaded, extremely deep separable convolutional blocks to perform efficient feature extraction in the feature space, progressively extracting and enhancing image features;

[0029] The extremely deep separable convolutional block adopts a Block block in an extremely deep separable convolutional network, and the Block block includes a series of skip connection layers and a normalization layer.

[0030] Preferably, the specific implementation process of the multi-scale branch fusion module (BFM) in step two is as follows:

[0031] The final features T1 and T2 obtained from the RGB branch and the high-frequency noise branch are convolved with three kernels of different sizes. Then, the three convolution results are summed to obtain the multi-scale feature map T.i ';

[0032] For multi-scale feature maps T i Pooling operations are performed in both the channel and spatial dimensions, specifically:

[0033] Global average pooling (AvgPool) and global max pooling (MaxPool) operations are performed on the multi-scale feature maps T1' and T2' in the channel dimension and spatial dimension, respectively, to obtain... and

[0034] The calculated channel pooling and space pooling results and Connect them to obtain the channel features. and spatial features Then, a one-dimensional convolution is performed on the connected spatial and channel features to obtain two different spatial weights; these weights are then normalized using softmax.

[0035] Finally, the calculated channel weights W1 and spatial weights W2 are used to perform weighted summation and weighted multiplication on the final features T1 and T2 obtained from the RGB branch and high-frequency noise branch processing to obtain the fused feature map F.

[0036] In a second aspect, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described thereon.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. The present invention proposes a novel enhanced dual-branch fusion model for face spoofing images to improve the robustness and generalization ability of CNN-based face deepfake detectors.

[0039] 2. The method of this invention employs functional analysis of the RGB and high-frequency noise branches, and proposes to optimize the corresponding branches using depthwise separable convolutions and selective kernel modules. For the RGB branch, the performance overhead of the branch is reduced while maintaining feature extraction capabilities, while for the high-frequency branch, depthwise separable convolutions are used to more accurately capture tampered regions.

[0040] 3. In the dual-branch fusion process, a multi-scale dual-branch fusion module is proposed to extract features from the RGB and high-frequency noise branches at multiple scales, and then cross-fuse them in space and channels to make full use of the features extracted from the RGB and high-frequency noise branches. Attached Figure Description

[0041] Figure 1This is a diagram of the dual-branch overall process and depth-separable modules of the model in this invention;

[0042] Figure 2 This is a diagram of the selective network structure in this invention;

[0043] Figure 3 This is a structural diagram of the multi-scale branch fusion module in this invention;

[0044] Figure 4 These are experimental results from this invention; Detailed Implementation

[0045] The present invention will be further analyzed below with reference to specific implementations.

[0046] An Enhanced Dual-Branch Fusion Model (EDFM) is proposed. The model consists of two branches: an RGB branch and a high-frequency noise branch. These two branches learn interactively and are finally fused by a Branch Fusion Module (BFM) for classification. In the shallow RGB layer, the advantages of a Depthwise Separable Convolutional Module (DSCM)—few parameters and high performance—are fully utilized to reduce the performance overhead of the RGB stream while ensuring stable performance. In the high-frequency noise branch, since SRM operates through three fixed filters, its adaptive ability to features is limited. This method addresses this issue by introducing selective convolutional units, enabling each neuron in the convolutional neural network to adaptively adjust its receptive field size based on multi-scale input information. This dynamic selection mechanism within the neural network effectively adjusts the receptive field size, significantly improving the performance of SRM. Therefore, a Selective Kernel Module (SKM) is used in conjunction with the high-frequency noise of SRM to further enhance SRM's ability to extract effective representations. Furthermore, cross-modal consistency is utilized for interactive learning in the middle layers. Finally, we designed a multi-scale channel spatial attention fusion module, supporting multi-scale channel spatial attention fusion of RGB and high-frequency branch streams, fully integrating the representations learned from the two branches for classification. Each component will be discussed in detail below. The specific steps of this deepfake face detection method are as follows:

[0047] Step 1: Construct the training set

[0048] This embodiment uses five common face spoofing datasets: Faceforencis++ (FF++), Celeb-DF, DeepFake Detection (DFD), and Deepfake Detection Challenge (DFDC). Notably, a high-quality version of FF++ (c23) is used, which contains 4,000 spoofed videos generated by four algorithms: DeepFake (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT).

[0049] Align the tagged fake face video (FF++) with the original video and preprocess it;

[0050] The preprocessing involves cropping the face regions of both the fake face video and the original video to 299×299 pixels, then uniformly normalizing them to [0,1]. Finally, image enhancement is performed, such as flipping, contrasting, scaling, and blurring.

[0051] Furthermore, we used random cropping to increase dataset diversity while ensuring annotation alignment with images. We employed Xception for pre-trained model initialization, using betas 0.9 and 0.999, and the Adam optimizer with epsilon1e-8, with a batch size of 32. Adam was used for optimization, and the learning rate was 0.0002. All experiments were implemented using PyTorch on an NVIDIA RTX 4096 24GB platform.

[0052] Step 2: Build an enhanced dual-branch fusion model and train it using the training set;

[0053] The enhanced dual-branch fusion model (EDFM) includes parallel RGB branches, high-frequency noise branches, and a multi-scale branch fusion module (BFM).

[0054] (1) The RGB branch is an enhanced source surface feature extraction module used to capture high-frequency noise information introduced by image sensor and device characteristics. These noises exhibit unique patterns due to different devices and sources, and can be used for identification as inherent characteristics of the image.

[0055] The RGB branch includes Depthwise Separable Convolution (DSCM) and Cross-Modality Consistency Enhancement (CMCE).

[0056] The depthwise separable convolutional (DSCM) comprises sequentially concatenated two-dimensional convolutions, normalization layers, LeakyReLU activation layers, and pointwise convolutional layers; specifically, as shown... Figure 1 (b) First, feature map X R The input is fed into a deep convolutional layer, then passes through a 2D convolution, a normalization and LeakyReLU activation layer, and a pointwise convolutional layer to obtain the output X. D A pointwise convolutional layer is a convolutional layer with a kernel of 1, the only difference being the number of channels.

[0057] In the RBG branch, replacing regular convolution with a depthwise separable convolution module improves computational efficiency. This is because depthwise separable convolution breaks down regular convolution into two steps: depthwise convolution and pointwise convolution. This decomposition significantly reduces computation, making the model more lightweight and efficient during training and fusion. Secondly, it significantly reduces the number of parameters. Since each depthwise convolution operates on only a single channel, while pointwise convolution handles inter-channel information fusion, this design drastically reduces the number of required parameters, thus lowering the model's storage and memory footprint. Furthermore, model accuracy typically remains stable, and feature extraction in the RBG branch suffers almost no performance loss. Therefore, using a depthwise separable convolution module in the RBG branch maintains model performance while greatly improving computational efficiency.

[0058] (2) The high-frequency noise branch is a selective kernel module in high-frequency noise, used to focus on the inconsistency after image manipulation, analyze the damage of the tampering operation to the spatial features of noise, and dig out the residual manipulation traces.

[0059] The high-frequency noise branch includes the Spatial Rich Model (SRM), the Selectable Kernel Module (SKM), and the CMCE shared with the RGB branch.

[0060] The specific steps of the Spatial Enrichment Model (SRM) are as follows:

[0061] 1) Perform three filtering operations on the image sequentially to extract the image residual;

[0062] 6) Image residuals are convolved with filters of different directions and scales to obtain multiple residual sub-models.

[0063] The filters of different directions and scales are as follows:

[0064] 0 0 0 0 0 0 -1 2 -1 0 0 2 -4 2 0 0 -1 2 -1 0 0 0 0 0 0

[0065] -1 2 -2 2 -1 2 -6 8 -6 2 -2 8 -12 8 -2 2 -6 8 -6 2 -1 2 -2 2 -1

[0066] 0 0 0 0 0 0 0 0 0 0 0 1 -2 1 0 0 0 0 0 0 0 0 0 0 0

[0067] 7) The residuals of the above residual sub-model are statistically analyzed to extract high-order statistical features and form a vector with high-frequency features.

[0068] The three filtering operations include a high-pass filter, a first-order residual filter, and a higher-order residual filter;

[0069] The higher-order statistical features include mean, variance, and covariance.

[0070] The input image for the RGB branch is the original image processed by the shallow layer of the backbone network of the Xception deep separable convolutional network.

[0071] The features processed by the Spatial Rich Model (SRM) still need to be processed using the shallow part of the Xception backbone network to extract low-level features from the input image, providing a foundation for subsequent deep learning of more complex semantic information.

[0072] The shallow portion of the Xception backbone network consists of two sets of convolutional layers, normalization layers, and ReLU activation functions. The convolutional layers use a 3x3 Convolutional layer.

[0073] The specific implementation process of the selective core module SKM is as follows:

[0074] SKM utilizes a building block called a selective kernel unit, which contains multiple branches with different kernel sizes. These branches are fused through a softmax attention mechanism, guided by information from within them. This fusion process allows neurons to adaptively adjust their effective receptive field size based on input. The specific structure of the module is as follows: Figure 2 As shown. Specifically, the output feature X of the SRM is processed by four convolutional kernels of different sizes. s The process involves extracting feature maps from receptive fields of different sizes; then, the outputs of four convolutional kernels of different sizes are summed to obtain the fused feature map X. f The description is as follows:

[0075]

[0076] Where F i (·) represents a convolution operation with i as the kernel;

[0077] Subsequently, the fused feature map X f After processing with a global average pooling layer and a fully connected layer, the dimensionality-reduced feature map X is obtained. d Then for X d Each layer is processed through a fully connected layer, and the concatenation is followed by Softmax activation and normalization to obtain an attention vector. This attention vector is then applied to the corresponding feature map X. dThe final output X is obtained by weighted summation of all feature maps. skm .

[0078] Because the filter used in preprocessing, with its three fixed convolutional kernels, cannot adaptively update to fit the data, it has limitations in perceiving the size of the receptive field for noise features. Therefore, we propose using the SKM (Sensitive Selection Model) approach. This module implements a dynamic selection mechanism within the convolutional neural network, allowing each neuron to adaptively adjust its receptive field size based on the input. This effectively addresses the shortcomings of SRM in extracting noise features, enhancing the ability of the high-frequency noise branch to extract receptive fields of different noise sizes, thus providing more effective features for subsequent classification.

[0079] The specific implementation process of the CMCE module is as follows:

[0080] Output of the RGB branch DSCM module and the output of another branch of the SKM module Calculate the cross-modal consistency graph using element-wise inner product:

[0081]

[0082] in and Representing F respectively r and F n The i-th element, where i ∈ {1,...,hw}, is then used to obtain the cross-modal consistency fusion feature map F using the following formula. r 'and F n ':

[0083]

[0084] The obtained F r 'and F n 'Features are used for subsequent fusion.'

[0085] After CMCE processing, two cascaded, highly depthwise separable convolutional blocks are used to perform efficient feature extraction in the feature space, which is used to progressively extract and enhance image features.

[0086] The extremely deep separable convolutional block adopts the Block block in the extremely deep separable convolutional network (Xception), and the Block block includes a series of skip connection layers and a normalization layer.

[0087] The specific implementation process of the multi-scale branch fusion module (BFM) is as follows:

[0088] Since this invention employs a dual-branch learning model, effectively fusing the features from both branches is a crucial issue. Traditional simple addition or multiplication feature fusion methods have significant limitations when handling these tasks, being susceptible to noise interference and struggling to dynamically adjust feature importance. Our proposed multi-scale dual-time fusion module, combining multi-scale feature extraction and channel attention mechanisms, enhances the robustness of feature fusion. In this process, multi-scale convolution captures feature information at different scales, while the attention mechanism dynamically adjusts feature importance, giving key features higher weights during the fusion process.

[0089] The final features T1 and T2 obtained from the RGB branch and the high-frequency noise branch are convolved with three kernels of different sizes. Then, the three convolution results are summed to obtain the multi-scale feature map T. i ';

[0090]

[0091] For multi-scale feature maps T i Pooling operations are performed in both the channel and spatial dimensions, specifically:

[0092] Global average pooling (AvgPool) and global max pooling (MaxPool) operations are performed on the multi-scale feature maps T1' and T2' in the channel dimension and spatial dimension, respectively, as described in the following details:

[0093]

[0094] Among them, AdvAvgPool is adaptive average pooling, and AdvMaxPool is adaptive max pooling, which can adaptively change to the specified HxW size output.

[0095] Next, we perform pooling operations on the space, as described below:

[0096]

[0097]

[0098] Where Cat represents the connection operation, 2DConv1 and 1DConv1 represent the convolution operation at time 1, Soft represents SoftMax, and W1 and W2 represent the channel weight and spatial weight, respectively.

[0099] Ultimately, as Figure 3 As shown, the calculated results of channel pooling and spatial pooling are presented. and Connect them to obtain the channel features. and spatial features Then, a one-dimensional convolution is performed on the concatenated spatial and channel features to obtain two different spatial weights. These weights are then normalized using softmax to ensure that their sum is 1, which facilitates subsequent weighted fusion operations.

[0100] Finally, using the channel weights W1 and spatial weights W2 calculated by formulas (6) and (7), the final features T1 and T2 obtained from the RGB branch and high-frequency noise branch are weighted and multiplied to obtain the fused feature map F. The specific description is as follows:

[0101] F = T1W1 + T2W2 (8)

[0102] After obtaining the fused feature map through BFM fusion, the Flatten operation is used to obtain the vector used for final classification.

[0103] The multi-scale branch fusion module abandons traditional simple addition or multiplication feature fusion methods, enhances its resistance to noise interference, and achieves dynamic adjustment of feature regions during fusion. The multi-scale dual-time fusion module improves the robustness of feature fusion by combining multi-scale feature extraction and channel attention mechanisms. In this process, multi-scale convolution can capture feature information at different scales, while the attention mechanism can dynamically adjust the importance of features, giving key features higher weights during fusion.

[0104] Step 3: Test the trained enhanced two-branch fusion model using multiple face spoofing datasets, and detect the focus points of the trained enhanced two-branch fusion model during testing, obtaining only the results of detecting true and false faces; finally, use the trained and tested enhanced two-branch fusion model to achieve face spoofing detection.

[0105] During the training of the enhanced two-branch fusion model, cross-entropy loss is used to supervise network learning;

[0106]

[0107] Where y is the binary label. It is the binary representation of the model's prediction results.

[0108] For example, this embodiment also trains and evaluates the above model.

[0109] In-dataset evaluation. Table 1 lists the results of training with FF++ using four different methods and testing on four spoofing methods. As shown by the gray-marked data in the table, our method achieves similar high performance to other methods in in-dataset testing.

[0110] Cross-dataset evaluation. When applied to cross-dataset scenarios, some methods exhibited very low AUCs during testing. In Table 1, the DCL method achieved 52.1% AUC in the NT to F2F scenario, while Xception achieved 49% AUC in the DF to FS scenario; and Face x-ray only achieved 45.8% AUC in the FS to DF scenario. In contrast, our method showed a more balanced performance, with AUCs generally above 70%, demonstrating good stability. Furthermore, in DF to other datasets and NT to other datasets, our method consistently achieved the best AUC results. For example, in the NT to four other methods evaluation, it achieved an excellent average of 94.7%, and particularly in the DF to F2F cross-dataset scenario, our method showed a significant improvement of 9.4% AUC over the second-best DCL method.

[0111] Table 1 shows the AUC evaluation across datasets on FF++ (HQ).

[0112]

[0113] Note: Bold text represents the best value for each column, and underlined text represents the second-best value for each column.

[0114] Table 2 Comparison of accuracy of various ablation models in this invention

[0115]

[0116] To evaluate the performance of the three modules, we conducted relevant ablation experiments. As shown in Table 2, we first trained the FF++ dataset using three models: RGB, SRM, and Two-stream, to obtain benchmark data. Then, we sequentially trained and tested the DSCM, SKNT, and BFM modules using the same method. The specific process is as follows: first, we evaluated the data using two single-input streams, RGB and SRM, and then evaluated the data by fusing them using Two-stream. As can be seen from the data in the third row of the table, the fused result outperforms the single-branch method in terms of DF and NT evaluations. From the fourth row onwards, we present the DSCM, SKNT, and BFM methods of this paper, representing depthwise separable convolution, selective modules, and dual-temporal fusion modules, respectively.

[0117] The table analysis shows that the basic DSCM module provides a certain improvement over the conventional two-stream model, especially in the NT benchmark, where it improves by 4.7%. The combination of the DSCM and SKM modules further enhances the performance compared to the single DSCM module, improving by 1.4% in both the FS and NT benchmarks. This further demonstrates the SKNT module's ability to enhance high-frequency feature extraction in SRM. Finally, adding the BFM module further improves the overall results, particularly in the NT benchmark, increasing the accuracy from 95.5% to 96.7%. The analysis of the last three rows shows that the overall model performance continuously improves with the addition of each module, demonstrating the effectiveness of each module.

[0118] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting deepfake faces based on an enhanced dual-branch fusion model, characterized in that... The method includes the following steps: Step 1: Construct the training set Align the tagged fake face videos with the original videos and preprocess them; Step 2: Build an enhanced dual-branch fusion model and train it using the training set; The enhanced dual-branch fusion model includes a parallel RGB branch, a high-frequency noise branch, and a multi-scale branch fusion module (BFM). The RGB branch includes a depthwise separable convolution module (DSCM) and a cross-modal consistency enhancement module (CMCE). The high-frequency noise branch includes a spatially rich model (SRM), a selective kernel module (SKM), and a cross-modal consistency enhancement module (CMCE) shared with the RGB branch. The multi-scale branch fusion module BFM performs multi-scale feature extraction on the RGB branch and the high-frequency noise branch, and then cross-fused them in space and channels. Step 3: Test the trained enhanced two-branch fusion model using multiple face spoofing datasets, and detect the focus points of the trained enhanced two-branch fusion model during testing, obtaining only the results of detecting true and false faces; finally, use the trained and tested enhanced two-branch fusion model to achieve face spoofing detection.

2. The method according to claim 1, characterized in that, The preprocessing described in step one involves cropping the face regions of the fake face video and the original video, uniformly normalizing them, and enhancing the image.

3. The method according to claim 1, characterized in that, The input image for the RGB branch mentioned in step two is the original image that needs to be processed by the shallow part of the backbone network of an extremely deep separable convolutional network. The features processed by the Spatial Rich Model (SRM) still need to be processed using the shallow part of the backbone network of the Extremely Deep Separable Convolutional Network to extract low-level features from the input image. The shallow portion of the backbone network of the extremely deep separable convolutional network includes two sets of convolutional layers, a normalization layer, and a ReLU activation function.

4. The method according to claim 1, characterized in that, The depthwise separable convolutional module (DSCM) mentioned in step two includes a series of sequentially connected two-dimensional convolutional layers, a normalization layer, a LeakyReLU activation layer, and a pointwise convolutional layer.

5. The method according to claim 1, characterized in that, The specific steps of the Spatial Enrichment Model (SRM) described in Step Two are as follows: 1) Perform multiple filtering operations on the image to extract the image residual; 2) Image residuals are convolved with filters of different directions and scales to obtain multiple residual sub-models; 3) The residuals of the above residual sub-model are statistically analyzed to extract high-order statistical features and form a vector with high-frequency features.

6. The method according to claim 1, characterized in that, The specific implementation process of the selective core module SKM mentioned in step two is as follows: The output features of the SRM are analyzed using four convolutional kernels of different sizes. The process involves extracting feature maps from receptive fields of different sizes; then, the outputs of four convolutional kernels of different sizes are summed to obtain a fused feature map. The description is as follows: (1) in Indicates The convolution operation is performed on the convolution kernel; Subsequently, the fused feature map Global average pooling and fully connected layers are applied to obtain the dimensionality-reduced feature map. Then on Each layer is processed through a fully connected layer, and the concatenation is followed by Softmax activation and normalization to obtain an attention vector. This attention vector is then applied to the corresponding feature map. The final output is obtained by weighted summation of all feature maps. .

7. The method according to claim 1, characterized in that, The specific implementation process of the cross-modal consistency enhancement module CMCE mentioned in step two is as follows: Output of the Depthwise Separable Convolutional Module (DSCM) and the output of the selective core module SKM Calculate the cross-modal consistency graph using element-wise inner product: (2) in and Respectively represent and The i-th element; Further, cross-modal consistency fusion feature maps were obtained. and : (3)。 8. The method according to claim 1, characterized in that, The output of the cross-modal consistency enhancement module CMCE mentioned in step two also needs to use two cascaded, extremely deep separable convolutional blocks to perform efficient feature extraction in the feature space, progressively extracting and enhancing image features; The extremely deep separable convolutional block adopts a Block block in an extremely deep separable convolutional network, and the Block block includes a series of skip connection layers and a normalization layer.

9. The method according to claim 1, characterized in that, The specific implementation process of the multi-scale branch fusion module (BFM) mentioned in step two is as follows: The final features T1 and T2 obtained from processing the RGB branch and the high-frequency noise branch are convolved with three convolutional kernels of different sizes. Then, the three convolutional results are summed to obtain a multi-scale feature map. ; Feature maps at multiple scales Pooling operations are performed in both the channel and spatial dimensions, specifically: Feature maps at multiple scales Global average pooling (AvgPool) and global max pooling (MaxPool) operations are performed in the channel dimension and spatial dimension, respectively, as described in the following descriptions: (4) in It is adaptive average pooling. It is adaptive max pooling; Next, we perform pooling operations on the space, as described below: (5) (6) Where Cat represents the connection operation, 2DConv1 and 1DConv1 represent the convolution operation at time 1, and Soft represents SoftMax; , These represent channel weights and spatial weights, respectively. The calculated channel pooling and space pooling results and Connect them to obtain the channel features. and spatial features Then, a one-dimensional convolution is performed on the concatenated spatial and channel features to obtain two different spatial weights; these weights are then normalized using softmax. Finally, using the channel weights W1 and spatial weights W2 calculated by formulas (5) and (6), the final features T1 and T2 obtained from the RGB branch and high-frequency noise branch are weighted and multiplied to obtain the fused feature map F; the specific description is as follows: (7)。 10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images