Face depth forgery detection method based on enhanced double-branch fusion model

Abstract

The invention discloses a face depth forgery detection method based on an enhanced double-branch fusion model. According to the method, a novel enhanced double-branch fusion model is adopted, so that the robustness and generalization ability of a CNN-based face depth forgery detector are improved. Firstly, RGB high-frequency noise in a face image is enhanced to extract abnormal features of the face image, and then reservation fusion is carried out. Specifically, a depth separable convolution module is used to improve model performance when image features are extracted. A selective kernel module is also used to adaptively extract more representative noise features by dynamically adjusting a convolution kernel. Besides, a multi-scale channel space attention fusion module is designed, and feature information of each part is effectively fused, so that overfitting of the model is reduced, and robustness and generalization ability of the model are enhanced. And finally, comprehensive evaluation on a plurality of reference data sets proves that the method is remarkably improved in the aspects of robustness and generalization.

Description

Technical Field

[0001] The present invention belongs to the field of face deep fake detection, and specifically relates to a face deep fake detection method based on an enhanced dual-branch fusion model. Background Art

[0002] With the development of deep fake technology, the generated fake images and videos are almost visually indistinguishable from the real content, posing a serious threat to personal privacy and public safety. Therefore, deep fake detection technology has become an important direction of information security research. Current detection methods are mainly divided into two categories: feature engineering-based and deep learning-based. The former relies on manually designed feature extraction, such as facial texture anomalies and inconsistent lighting, to identify signs of forged images; while the latter automatically learns image features through convolutional neural networks (CNNs) and shows higher accuracy. However, despite the various methods proposed, deep fake detection still faces the problems of insufficient robustness and poor generalization ability, especially when facing data sets generated by different forgery methods, the detection performance is often affected. In order to address the challenge of insufficient generalization ability, researchers have gradually explored more sophisticated detection methods, such as dual-branch fusion strategies, transfer learning, and multi-task learning. These methods aim to improve the adaptability of the model between different data sets. The dual-branch fusion method combines the information of different feature extraction branches to more comprehensively capture the multi-level features in the forged image, thereby improving the accuracy and robustness of detection. This strategy usually combines information in the spatial domain and the frequency domain to more effectively identify the possibility of forgery.

[0003] In recent years, researchers have also begun to focus on the extraction of frequency domain information, such as using discrete cosine transform (DCT) and Fourier transform to enhance face classification capabilities. However, many existing methods fail to fully exploit the interaction between normal image features, which limits the detection effect. Previous dual-branch structures used the same processing method for RGB and high-frequency noise branches, resulting in feature extraction concentrated in areas unrelated to forgery detection, further affecting generalization capabilities. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and propose a face deep fake detection method based on an enhanced dual-branch fusion model to improve the robustness and generalization ability of the CNN-based face deep fake detector.

[0005] In a first aspect, the present invention provides a method for detecting deep fake faces based on an enhanced dual-branch fusion model, the method comprising the following steps:

[0006] Step 1: Build a training set

[0007] Align the labeled face forgery video with the original video and perform preprocessing;

[0008] Step 2: Build an enhanced dual-branch fusion model and train it using the training set;

[0009] The enhanced dual-branch fusion model EDFM includes an RGB branch, a high-frequency noise branch, and a multi-scale branch fusion module BFM arranged in parallel;

[0010] The RGB branch includes a depth separable convolution module DSCM and a cross-modal consistency enhancement module CMCE;

[0011] The high-frequency noise branch includes a spatial enrichment model SRM, a selective kernel module SKM, and a cross-modal consistency enhancement module CMCE shared with the RGB branch;

[0012] Step 3. Use a variety of face forgery data sets to test the trained enhanced two-branch fusion model, and detect the focus points of the trained enhanced two-branch fusion model during the test to only obtain the authenticity of the detection result; finally, use the trained and tested enhanced two-branch fusion model to realize face forgery detection.

[0013] Preferably, the preprocessing in step 1 is cropping, uniform normalization, and image enhancement of the face area of ​​the forged face video and the original video.

[0014] Preferably, the input image to the RGB branch in step 2 is an original image that needs to be processed by a shallow layer of the backbone network of an extremely deep separable convolutional network;

[0015] The features processed by the spatial enrichment model SRM need to be processed by the shallow part of the backbone network of the extremely deep separable convolutional network to extract low-level features in the input image;

[0016] The shallow part of the backbone network of the extremely deep separable convolutional network includes two groups of convolutional layers, a normalization layer and a ReLU activation function.

[0017] Preferably, the depthwise separable convolution module DSCM in step 2 comprises a two-dimensional convolution, a normalization layer, a LeakyReLU activation layer, and a point-by-point convolution layer connected in series in sequence.

[0018] Preferably, the spatial enrichment model SRM in step 2 comprises the following specific steps:

[0019] 1) Perform multiple filtering operations on the image to extract the image residual;

[0020] 4) The image residual is convolved with filters of different directions and scales to obtain multiple residual sub-models;

[0021] 5) The residual of the above residual sub-model is subjected to statistical analysis to extract high-order statistical features to form a vector with high-frequency features.

[0022] Preferably, the specific implementation process of the optional core module SKM in step 2 is:

[0023] The output feature X of the SRM is processed by four convolution kernels of different sizes. s Processing is performed to extract feature maps of receptive fields of different sizes; then the outputs of the four convolution kernels of different sizes are summed to obtain the fused feature map X f ;

[0024] Then, the fusion feature map X f Perform global average pooling layer and full connection layer processing to obtain the feature map X after dimensionality reduction d ; Then for X d Each layer is processed by a fully connected layer, and after concatenation, Softmax activation and normalization are used to obtain the attention vector, which is applied to the corresponding feature map X d And all feature maps are weighted summed to get the final output X skm .

[0025] Preferably, the specific implementation process of the cross-modal consistency enhancement module CMCE in step 2 is:

[0026] The output F of the depthwise separable convolution module DSCM r and the output F of the optional nuclear module SKM n The cross-modal consistency graph is calculated by element-wise inner product;

[0027] Further obtain the cross-modal consistency fusion feature map F r ' and F n '.

[0028] Preferably, the output of the cross-modal consistency enhancement module CMCE in step 2 also needs to use two extremely deep separable convolution blocks connected in series to perform efficient feature extraction in the feature space, and gradually extract and enhance image features;

[0029] The extremely deep separable convolution block adopts the Block block in the extremely deep separable convolution network, and the Block block includes a serially connected jump connection layer and a normalization layer.

[0030] Preferably, the specific implementation process of the multi-scale branch fusion module BFM in step 2 is:

[0031] The final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch are convolved with three convolution kernels of different sizes, and then the three convolution results are added to obtain a multi-scale feature map Ti ';

[0032] For multi-scale feature maps T i 'Pooling operations are performed in the channel dimension and spatial dimension, specifically:

[0033] Perform global average pooling (AvgPool) and global maximum pooling (MaxPool) operations on the multi-scale feature maps T1' and T2' in the channel dimension and spatial dimension respectively, and obtain and

[0034] The calculated channel pooling and spatial pooling results and Connect them to get the channel features respectively and spatial characteristics Then, one-dimensional convolution is performed on the connected spatial and channel features to obtain two different spatial weights; these weights are then normalized using softmax;

[0035] Finally, the calculated channel weight W1 and spatial weight W2 are used to perform weighted addition and weighted multiplication on the final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch to obtain the fused feature map F.

[0036] In a second aspect, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. The method of the present invention proposes a novel enhanced dual-branch fusion model for forged face images to improve the robustness and generalization ability of the CNN-based deep fake face detector.

[0039] 2. The method of the present invention adopts the functional analysis of RGB and high-frequency noise branches, and proposes to use deep separable convolution and selective kernel modules to optimize the functions of the corresponding branches. On the RGB branch, the performance overhead of the branch is reduced while ensuring the feature extraction capability, and deep separable convolution is used for the high-frequency branch to adaptively capture the tampered area more accurately.

[0040] 3. In the dual-branch fusion process, a multi-scale dual-branch fusion module is proposed to perform multi-scale feature extraction on the RGB and high-frequency noise branches, and then cross-fuse them in space and channels to make full use of the features extracted by the RGB and high-frequency noise branches. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1It is the dual-branch overall process and deeply separable module diagram of the model in the present invention;

[0042] Figure 2 is a selective network structure diagram in the present invention;

[0043] Figure 3 It is a structural diagram of the multi-scale branch fusion module in the present invention;

[0044] Figure 4 It is an experimental effect diagram of the present invention; DETAILED DESCRIPTION

[0045] The present invention is further analyzed below in conjunction with specific implementation.

[0046] Enhanced dual-branch fusion model (EDFM). The model mainly consists of two branches, one is composed of RGB branch and the other is high-frequency noise branch. These two branches are interactively learned and finally fused by Branch Fusion Module (BFM) for classification. In the shallow RGB layer, the advantages of Depthwise Separable Convolutional Module (DSCM) with small number of parameters and high performance are fully utilized to reduce the performance overhead of RGB stream while ensuring stable performance. In the high-frequency noise branch, since SRM works through three fixed filters, its adaptive ability to features is limited. This method introduces a selective convolution unit, so that each neuron in the convolutional neural network can adaptively adjust its receptive field size according to multi-scale input information. The dynamic selection mechanism of the neural network can adjust the receptive field size, which solves the problem of SRM. Therefore, we use the selective network module Selective Kernel Module (SKM) and work together with SRM high-frequency noise to further enhance the ability of SRM to extract effective representations. In addition, we also use cross-modal consistency to interactively learn in the middle layer. Finally, we designed a multi-scale channel spatial attention fusion module that supports multi-scale channel spatial attention fusion of RGB and high-frequency branch streams, and fully integrates the representations learned by the two branches for classification. In this regard, each component will be discussed in detail below. The specific steps of this face deep fake detection method are as follows:

[0047] Step 1: Build a training set

[0048] This example uses five common face forgery datasets: Faceforencis++ (FF++), Celeb-DF, DeepFake Detection (DFD), and Deepfake Detection Challenge (DFDC). In particular, the high-quality version (c23) of FF++ is used, which contains 4,000 forged videos generated by four algorithms: DeepFake (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT).

[0049] Align the labeled face forgery video FF++ with the original video and perform preprocessing;

[0050] The preprocessing is to crop the face regions of the forged face video and the original video to 299×299, then uniformly normalize them to [0,1], and finally perform image enhancement, such as flipping, contrasting, scaling, and blurring.

[0051] In addition, we use random cropping to increase the diversity of the dataset while ensuring that the annotations are aligned with the images. We use Xception for pre-trained model initialization and use betas 0.9 and 0.999, and Adam optimizer with epsilon 1e-8, batch size is set to 32, Adam is used for optimization, and the learning rate is 0.0002. All experiments are implemented using PyTorch on an NVIDIA RTX409624GB platform.

[0052] Step 2: Build an enhanced dual-branch fusion model and train it using the training set;

[0053] The enhanced dual-branch fusion model EDFM (Enhanced dual-branch fusion model) includes a parallel RGB branch, a high-frequency noise branch, and a multi-scale branch fusion module BFM (Branch Fusion Module);

[0054] (1) The RGB branch is an enhanced source feature extraction module, which is used to capture high-frequency noise information introduced by image sensors and device characteristics. These noises show unique patterns due to differences in devices and sources, and can be used for recognition as inherent characteristics of images;

[0055] The RGB branch includes deep separable convolution DSCM and cross-modal consistency enhancement CMCE;

[0056] The depthwise separable convolution DSCM includes a two-dimensional convolution, a normalization layer, a LeakyReLU activation layer, and a point-by-point convolution layer connected in series in sequence; specifically, Figure 1 (b), first feature map X R The input is sent to the deep convolution layer, and then goes through a two-dimensional convolution, a normalization and LeakyReLU activation layer, and a point-by-point convolution layer to get the output X. D The point-by-point convolution layer includes the convolution layer with a convolution kernel of 1, which only changes the number of channels.

[0057] In the RBG branch, the computational efficiency of the model can be improved by using a depthwise separable convolution module instead of a conventional convolution. Because the depthwise separable convolution decomposes the conventional convolution into two steps: depthwise convolution and pointwise convolution. This decomposition greatly reduces the amount of computation, making the model lighter and more efficient during training fusion. Secondly, the number of parameters can be significantly reduced. Since each depthwise convolution only acts on a single channel, and the pointwise convolution is responsible for information fusion between channels, this design can significantly reduce the number of required parameters, thereby reducing the storage and memory usage of the model. In addition, while the model accuracy is usually kept stable, there is almost no performance loss when the RBG branch extracts features from the image. Therefore, using a depthwise separable convolution module in the RBG branch can greatly improve computational efficiency while maintaining model performance.

[0058] (2) The high-frequency noise branch is a selective kernel module in the high-frequency noise, which is used to focus on the inconsistency after the image is manipulated, analyze the damage of the tampering operation to the noise space characteristics, and dig out the remaining manipulation traces;

[0059] The high-frequency noise branch includes a spatial rich model SRM (Spatial Rich Model), a selective kernel module SKM, and a CMCE shared with the RGB branch;

[0060] The specific steps of the spatial enrichment model SRM are as follows:

[0061] 1) Perform three filtering operations on the image in sequence to extract the image residual;

[0062] 6) The image residual is convolved with filters of different directions and scales to obtain multiple residual sub-models.

[0063] The filters of different directions and scales are specifically as follows:

[0064] 0 0 0 0 0 0 -1 2 -1 0 0 2 -4 2 0 0 -1 2 -1 0 0 0 0 0 0

[0065] -1 2 -2 2 -1 2 -6 8 -6 2 -2 8 -12 8 -2 2 -6 8 -6 2 -1 2 -2 2 -1

[0066] 0 0 0 0 0 0 0 0 0 0 0 1 -2 1 0 0 0 0 0 0 0 0 0 0 0

[0067] 7) The residual of the above residual sub-model is subjected to statistical analysis to extract high-order statistical features to form a vector with high-frequency features.

[0068] The three filtering operations include a high-pass filter, a first-order residual filter and a high-order residual filter;

[0069] The high-order statistical features include mean, variance, and covariance.

[0070] The input image to the RGB branch is an image of the original image processed by the shallow part of the backbone network of the extremely deep separable convolutional network (Xception);

[0071] The features processed by the spatial enrichment model SRM also need to be processed using the shallow part of the Xception backbone network to extract low-level features in the input image, providing a basis for subsequent deep learning of more complex semantic information.

[0072] The shallow part of the Xception backbone network includes two sets of convolutional layers, a normalization layer and a ReLU activation function. The convolutional layer uses 3x3 Conv;

[0073] The specific implementation process of the optional core module SKM is:

[0074] SKM utilizes a building block called the selective kernel unit, which contains multiple branches with different kernel sizes. These branches are fused through a softmax attention mechanism, guided by the information in these branches. This fusion process enables neurons to adaptively adjust their effective receptive field size according to the input. The specific structure of the module is as follows Figure 2 Specifically, the output feature X of the SRM is processed by four convolution kernels of different sizes. s Processing is performed to extract feature maps of receptive fields of different sizes; then the outputs of the four convolution kernels of different sizes are summed to obtain the fused feature map X f , described as follows:

[0075]

[0076] where F i (·) represents the convolution operation with i as the convolution kernel;

[0077] Then, the fusion feature map X f Perform global average pooling layer and full connection layer processing to obtain the feature map X after dimensionality reduction d ; Then for X d Each layer is processed by a fully connected layer, and after concatenation, Softmax activation and normalization are used to obtain the attention vector, which is applied to the corresponding feature map X dAnd all feature maps are weighted summed to get the final output X skm .

[0078] Because the filter cannot be adaptively updated to adapt to the data when used for preprocessing because it uses three fixed convolution kernels, there are certain limitations in the perception of the size of the receptive field of noise features. Therefore, we propose to use the SKM method, which can implement a dynamic selection mechanism in the convolutional neural network, allowing each neuron to adaptively adjust its receptive field size according to the input. Therefore, it can well solve the shortcomings of SRM in extracting noise features, enhance the ability of the high-frequency noise branch to extract receptive fields of different sizes of noise, and provide more effective features for subsequent classification.

[0079] The specific implementation process of the CMCE module is:

[0080] Output of DSCM module for RGB branch And the output of another branch SKM module Compute the cross-modal consistency graph by element-wise inner product:

[0081]

[0082] in and Represents F r and F n The i-th element and i∈{1,...,hw}, then the cross-modal consistency fusion feature map F is obtained by the following formula r ' and F n ':

[0083]

[0084] The obtained F r ' and F n 'Features are used for subsequent fusion.

[0085] After CMCE processing, two extremely deep separable convolutional blocks are connected in series to perform efficient feature extraction in the feature space, which is used to gradually extract and enhance image features.

[0086] The extremely deep separable convolution block adopts the Block block in the extremely deep separable convolution network (Xception), and the Block block includes a serially connected jump connection layer and a normalization layer.

[0087] The specific implementation process of the multi-scale branch fusion module BFM is:

[0088] Since the dual-branch learning mode is adopted in the present invention, how to effectively fuse the features of the dual branches is a very important issue. The traditional simple addition or multiplication feature fusion method has significant limitations when processing these tasks, is easily interfered by noise, and is difficult to dynamically adjust the importance of features. The multi-scale dual-time fusion module we proposed, combined with multi-scale feature extraction and channel attention mechanism, can improve the robustness of feature fusion. In this process, multi-scale convolution can capture feature information of different scales, and the attention mechanism can dynamically adjust the importance of features, so that key features get higher weights in the fusion process.

[0089] The final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch are convolved with three convolution kernels of different sizes, and then the three convolution results are added to obtain a multi-scale feature map T i ';

[0090]

[0091] For multi-scale feature maps T i 'Pooling operations are performed in the channel dimension and spatial dimension, specifically:

[0092] The multi-scale feature maps T1', T2' are subjected to global average pooling (AvgPool) and global maximum pooling (MaxPool) operations in the channel dimension and spatial dimension respectively, which are specifically described as follows:

[0093]

[0094] Among them, AdvAvgPool is adaptive average pooling, and AdvMaxPool is adaptive maximum pooling, which can adaptively change to the specified HxW size output.

[0095] Next, the pooling operation is performed on the space, which is described as follows:

[0096]

[0097]

[0098] Among them, Cat represents the connection operation, 2DConv1 and 1DConv1 represent the convolution operation at time 1, Soft represents SoftMax, and W1 and W2 represent channel weight and spatial weight respectively.

[0099] Finally, if Figure 3 As shown, the calculated channel pooling and spatial pooling results are and Connect them to get the channel features respectively and spatial characteristics Then, one-dimensional convolution is performed on the concatenated spatial and channel features to obtain two different spatial weights. These weights are then normalized using softmax to ensure that their sum is 1, which is convenient for subsequent weighted fusion operations.

[0100] Finally, the channel weight W1 and spatial weight W2 calculated by formulas (6) and (7) are used to perform weighted addition and weighted multiplication on the final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch to obtain the fused feature map F. The specific description is as follows:

[0101] F=T1W1+T2W2 (8)

[0102] After the fused feature map is obtained through BFM fusion, the Flatten operation is used to obtain the vector used for the final classification.

[0103] The multi-scale branch fusion module abandons the traditional simple addition or multiplication feature fusion method, enhances the ability to resist noise interference, and realizes the fusion of dynamically adjusted feature areas. The multi-scale dual-time fusion module can improve the robustness of feature fusion by combining multi-scale feature extraction and channel attention mechanism. In this process, multi-scale convolution can capture feature information of different scales, and the attention mechanism can dynamically adjust the importance of features, so that key features get higher weights in the fusion process.

[0104] Step 3. Use a variety of face forgery data sets to test the trained enhanced two-branch fusion model, and detect the focus points of the trained enhanced two-branch fusion model during the test to only obtain the authenticity of the detection result; finally, use the trained and tested enhanced two-branch fusion model to realize face forgery detection.

[0105] During the training of the enhanced two-branch fusion model, cross entropy loss is used to supervise network learning;

[0106]

[0107] where y is the binary label, is the binary prediction result of the model.

[0108] By way of example, this embodiment also performs training evaluation on the above-mentioned model.

[0109] Evaluation on the dataset. Table 1 lists the results of testing on four forgery methods using FF++ training in four different ways. As shown in the gray data in the table, when tested on the dataset, our method can achieve similar performance to other methods.

[0110] Cross-dataset evaluation. When applied to cross-dataset scenarios, some methods have very low AUC during testing. In Table 1, the DCL method has a 52.1% AUC in the NT to F2F scenario, while in the DF to FS scenario, Xception has a 49% AUC evaluation; in the FS to DF scenario, the Face x-ray result is only 45.8%. In comparison, our method shows a balance, and the AUC is generally above 70%, which shows that our method has good stability. In addition, in the methods from DF to other datasets and NT to other datasets, it can be seen that our method has basically the best results in the AUC evaluation results. For example, in the evaluation results from NT to the other four methods, the average excellent result reached 94.7%. Especially in the DF to F2F cross-dataset, our method has a significant improvement of 9.4% over the AUC of the second DCL.

[0111] Table 1 AUC evaluation across datasets on FF++(HQ)

[0112]

[0113] Note: The bold fonts are the optimal values ​​in each column, and the underlined fonts are the suboptimal values ​​in each column.

[0114] Table 2 Comparison of the accuracy of various ablation models of the present invention

[0115]

[0116] In order to evaluate the effects of the three modules, we conducted relevant ablation experiments. As shown in Table 2, we first used the RGB, SRM and Two-stream models to train the FF++ dataset to obtain benchmark data, and then used the DSCM, SKNT and BFM modules to perform the same training and testing. The specific process is as follows: first, we use two single input streams of RGB and SRM for evaluation, and then use the Two-stream fusion to evaluate the two. According to the data in the third row of the table, it can be seen that the evaluation results of the fusion results on DF and NT are better than those of the single branch. From the fourth row onwards, we have the DSCM, SKNT and BFM methods of this article, which represent the depth-separable convolution, the selective module and the dual-time fusion module, respectively.

[0117] From the table analysis, it can be seen that the basic DSCM has a certain improvement compared to the conventional Two-stream, especially in the NT evaluation, which has increased by 4.7%. The combination of the following two modules, DSCM and SKM, has a certain degree of improvement compared to the single DSCM module, and has increased by 1.4% in the FS and NT evaluations. It is further demonstrated that the SKNT module has an enhanced effect on the high-frequency feature extraction of SRM. Finally, with the addition of the BFM module, the overall results are further improved, especially in the NT evaluation, from 95.5% to 96.7%. Through the analysis of the last three rows of results, with the addition of each module, the overall performance of the model is constantly improving, which also illustrates the effectiveness of each module.

[0118] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting deep fake faces based on an enhanced dual-branch fusion model, characterized in that The method comprises the following steps: Step 1: Build a training set Align the labeled face forgery video with the original video and perform preprocessing; Step 2: Build an enhanced dual-branch fusion model and train it using the training set; The enhanced dual-branch fusion model EDFM includes an RGB branch, a high-frequency noise branch, and a multi-scale branch fusion module BFM arranged in parallel; The RGB branch includes a depth separable convolution module DSCM and a cross-modal consistency enhancement module CMCE; The high-frequency noise branch includes a spatial enrichment model SRM, a selective kernel module SKM, and a cross-modal consistency enhancement module CMCE shared with the RGB branch; Step 3. Use a variety of face forgery data sets to test the trained enhanced two-branch fusion model, and detect the focus points of the trained enhanced two-branch fusion model during the test to only obtain the authenticity of the detection result; finally, use the trained and tested enhanced two-branch fusion model to realize face forgery detection.

2. The method according to claim 1, characterized in that: The preprocessing described in step 1 is to crop the face area of ​​the forged face video and the original video, uniformly normalize, and enhance the image.

3. The method according to claim 1, characterized in that: The input image for the RGB branch in step 2 is the original image that needs to be processed by the shallow part of the backbone network of the extremely deep separable convolutional network; The features processed by the spatial enrichment model SRM need to be processed by the shallow part of the backbone network of the extremely deep separable convolutional network to extract low-level features in the input image; The shallow part of the backbone network of the extremely deep separable convolutional network includes two groups of convolutional layers, a normalization layer and a ReLU activation function.

4. The method according to claim 1, characterized in that: The depthwise separable convolution module DSCM described in step 2 includes a two-dimensional convolution, a normalization layer, a LeakyReLU activation layer, and a point-by-point convolution layer connected in series.

5. The method according to claim 1, characterized in that: The specific steps of the spatial enrichment model SRM described in step 2 are: 1) Perform multiple filtering operations on the image to extract the image residual; 2) The image residual is convolved with filters of different directions and scales to obtain multiple residual sub-models; 3) The residuals of the above residual sub-model are statistically analyzed to extract high-order statistical features to form a vector with high-frequency features.

6. The method according to claim 1, characterized in that: The specific implementation process of the optional core module SKM described in step 2 is: The output feature X of the SRM is processed by four convolution kernels of different sizes. s Processing is performed to extract feature maps of receptive fields of different sizes; then the outputs of the four convolution kernels of different sizes are summed to obtain the fused feature map X f , described as follows: where F i (·) represents the convolution operation with i as the convolution kernel; Then, the fusion feature map X f Perform global average pooling layer and full connection layer processing to obtain the feature map X after dimensionality reduction d ; Then for X d Each layer is processed by a fully connected layer, and after concatenation, Softmax activation and normalization are used to obtain the attention vector, which is applied to the corresponding feature map X d And all feature maps are weighted summed to get the final output X skm .

7. The method according to claim 1, characterized in that: The specific implementation process of the cross-modal consistency enhancement module CMCE in step 2 is: The output F of the depthwise separable convolution module DSCM r and the output F of the optional nuclear module SKM n Compute the cross-modal consistency graph by element-wise inner product: where f r i and f n i Represents F r and F n The i-th element; Further obtain the cross-modal consistency fusion feature map F r ' and F n ':

8. The method according to claim 1, characterized in that: The output of the cross-modal consistency enhancement module CMCE described in step 2 also needs to use two extremely deep separable convolution blocks connected in series to perform efficient feature extraction in the feature space, and gradually extract and enhance image features; The extremely deep separable convolution block adopts the Block block in the extremely deep separable convolution network, and the Block block includes a serially connected jump connection layer and a normalization layer.

9. The method according to claim 1, characterized in that: The specific implementation process of the multi-scale branch fusion module BFM in step 2 is: The final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch are convolved with three convolution kernels of different sizes, and then the three convolution results are added to obtain a multi-scale feature map T i '; For multi-scale feature maps T i 'Pooling operations are performed in the channel dimension and spatial dimension, specifically: The multi-scale feature maps T1', T2' are subjected to global average pooling (AvgPool) and global maximum pooling (MaxPool) operations in the channel dimension and spatial dimension respectively, which are specifically described as follows: Among them, AdvAvgPool is adaptive average pooling, and AdvMaxPool is adaptive maximum pooling; Next, the pooling operation is performed on the space, which is described as follows: Among them, Cat represents the connection operation, 2DConv1 and 1DConv1 represent the convolution operation at time 1, Soft represents SoftMax; W1 and W2 represent channel weight and spatial weight respectively; The calculated channel pooling and spatial pooling results and Connect them to get the channel features respectively and spatial characteristics Then, one-dimensional convolution is performed on the connected spatial and channel features to obtain two different spatial weights; these weights are then normalized using softmax; Finally, the channel weight W1 and spatial weight W2 calculated by formulas (5) and (6) are used to perform weighted addition and weighted multiplication on the final features T1 and T2 obtained by processing the RGB branch and the high-frequency noise branch to obtain the fused feature map F. The specific description is as follows: F=T1W1+T2W2 (7).

10. A computing device comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.

Images