Double-domain face counterfeiting positioning method and system based on deep learning, and medium

Abstract

The invention discloses a double-domain face forgery positioning method and system based on deep learning and a medium. Diversified information is collected from RGB and frequency domains for deep forgery detection. Specifically, firstly, a multi-frequency-band feature sensor is introduced to obtain frequency features of different frequency bands, and face counterfeit traces are deeply excavated. Then, a spatial frequency feature fusion module is designed, RGB-frequency domain information is comprehensively fused, more low-level counterfeit traces and structural textures are utilized, a face counterfeit region is positioned through a cross-domain feature fusion mode, and the safety and reliability of face recognition are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a dual-domain face forgery localization method, system and medium based on deep learning. Background Art

[0002] With the rapid development of digital image processing and artificial intelligence technology, technologies such as face recognition and face synthesis have been widely used. However, this has also raised the risks and challenges of face forgery. Malicious users can use face synthesis technology to create forged face images to deceive face recognition systems and conduct illegal activities. Therefore, developing a reliable face forgery location algorithm is crucial to improving the security and reliability of face recognition.

[0003] Traditional classification-based deep learning methods can usually only determine whether an image is forged, but cannot provide accurate forgery location information. Traditional face forgery detection networks regard face forgery detection as a binary classification task and cannot locate the main area of ​​face forgery. However, in some multimedia forensics scenarios, it is crucial to locate the area of ​​face forgery. With the continuous development and evolution of face forgery technology, new forgery methods continue to emerge, such as the stable diffusion model. Traditional face forgery networks have only been experimented on forged photos generated based on GAN technology, and may not be able to adapt to and identify new forgery technologies in a timely manner, resulting in missed detection or false detection problems. Some existing detection methods have poor robustness to image changes such as lighting changes, posture changes, and blur. This means that in complex real-world scenarios, the detection algorithm may produce a high false alarm rate or missed alarm rate, reducing the overall detection accuracy.

[0004] Therefore, the prior art has defects and is in urgent need of improvement. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a dual-domain face forgery positioning method, system and medium based on deep learning, which can locate the face forgery area more effectively and quickly, solve the main problem that the existing face forgery detection network cannot locate the forged area and the main problem that the traditional detection method may not be able to adapt to and identify new forgery technology in time.

[0006] A first aspect of the present invention provides a dual-domain face forgery positioning method based on deep learning, comprising:

[0007] Get face RGB image X image ;

[0008] Extract the face RGB image X image The semantic feature X semantic, by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out ,x 2_out ,x 3_out ,x 4_out};

[0009] The multi-scale feature combination {x 1_out ,x 2_out ,x 3_out ,x 4_out}Converted to frequency domain feature combination

[0010] Combining the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ;

[0011] A rough prediction result Y of the forged face positioning coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features

[0012] For the multi-scale feature combination {x 1_out ,x 2_out ,x 3_out ,x 4_out} Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ;

[0013] Calculate the feature X fusion The local channel vector V local and the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ;

[0014] According to the rough prediction result Y of the face forgery positioning coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

[0015] In this solution, the face RGB image X is extracted image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out ,x 2_out ,x 3_out ,x 4_out},include:

[0016] The face RGB image X image Feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic ;

[0017] The semantic feature X semantic Input the dilated layered decoupled convolution unit to perform dilated convolution to obtain a multi-scale feature combination {x 1_out ,x 2_out ,x 3_out ,x 4_out};

[0018] The dilated convolution process is expressed as:

[0019] x1, x2, x3, x4 = Conv 1×1 (X semantic )

[0020] x1_out=x1

[0021] x2_out = DWConv r＝1 (x2)+x2

[0022] x3_out = DWConv r＝2 (x2+x3)+x3

[0023] x4_out = DWConv r＝3 (x3+x4)+x4

[0024] Among them, x1, x2, x3, x4 are the features obtained by the 1×1 convolution layer, Conv 1×1 For 1×1 convolution, DWConv r＝i is a depthwise separable convolution with dilation rate i, x 1_out , x 2_out , x 3_out , x 4_out is a multi-scale feature, X semantic is a semantic feature.

[0025] In this solution, the multi-scale feature combination {x1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination include:

[0026] According to the channel dimension, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out} to split and obtain the multi-scale feature combination after splitting

[0027] The multi-scale features after splitting are combined by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination

[0028] In this solution, the frequency domain features are combined Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ,include:

[0029] The frequency domain feature combination Perform channel dimension splicing to obtain frequency domain features

[0030] The frequency domain features Send it to the prediction head to get the rough prediction result Y of face forgery positioning coarse .

[0031] In this solution, the rough prediction result Y of the forged face location is coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features include:

[0032] The frequency domain characteristics and the rough prediction result Y of the face forgery positioning coarse Perform Hadamard dot product operation to obtain frequency domain features

[0033] The frequency domain characteristics Perform graph convolution operation to obtain frequency domain features

[0034] In this solution, according to the feature X fusion The local channel vector V local and the global channel vector V no-localCalculate and obtain the refined prediction result Y of face forgery positioning fine include:

[0035] For the feature X fusion The local channel vector V local and the global channel vector V no-local Add them together, and pass the Sigmoid activation function to obtain the new vectors, which are then combined with the spatial features S C and frequency domain characteristics Multiply them together to get the spatial features of the fused frequency domain information and frequency domain features that fuse spatial information

[0036] The spatial domain features of the fused frequency domain information are and frequency domain features that fuse spatial information Add together to get fused cross-domain features

[0037] The fused cross-domain features Send it to the prediction head to get the refined prediction result Y of face forgery positioning fine .

[0038] In this solution, the rough prediction result Y based on the face forgery positioning coarse And the refined prediction result Y fine Calculate the final loss function include:

[0039] Calculate the rough prediction results Y of face forgery positioning respectively coarse And the refined prediction result Y fine The damage function of and refined prediction damage function

[0040] The refined prediction result Y fine Perform binary classification weighted loss calculation to obtain the binary classification weighted loss function

[0041] The rough prediction damage function Refined prediction of damage function and binary classification weighted loss function; Add them together to get the final loss function

[0042] A second aspect of the present invention provides a dual-domain face forgery positioning system based on deep learning, comprising:

[0043] Data acquisition module, used to obtain face RGB image X image ;

[0044] Feature extraction conversion module module, used to extract the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}; The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination

[0045] A rough prediction module for face forgery positioning is used to combine the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ;

[0046] A cross-domain feature fusion module is used to roughly predict the result Y of the forged face location. coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features For the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out} Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ;

[0047] The face forgery positioning fine prediction module is used to calculate the feature X fusion The local channel vector V local and the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ;

[0048] A loss function evaluation module is used to roughly predict the result Y of the face forgery positioning according to the coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

[0049] In this solution, the face RGB image X is extracted image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out},include:

[0050] The face RGB image X image Feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic ;

[0051] The semantic feature X semantic Input the dilated layered decoupled convolution unit to perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out};

[0052] The dilated convolution process is expressed as:

[0053] x1, x2, x3, x4 = Conv 1×1 (X semantic )

[0054] x1_out=x1

[0055] x2_out = DWConv r＝1 (x2)+x2

[0056] x3_out = DWConv r＝2 (x2+x3)+x3

[0057] x4_out = DWConv r＝3 (x3+x4)+x4

[0058] Among them, x1, x2, x3, x4 are the features obtained by the 1×1 convolution layer, Conv 1×1 For 1×1 convolution, DWConv r＝i is a depthwise separable convolution with dilation rate i, x 1_out , x 2_out , x 3_out , x 4_out is a multi-scale feature, Xsemantic is a semantic feature.

[0059] In this solution, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination include:

[0060] According to the channel dimension, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out} to split and obtain the multi-scale feature combination after splitting

[0061] The multi-scale features after splitting are combined by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination

[0062] In this solution, the frequency domain features are combined Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ,include:

[0063] The frequency domain feature combination Perform channel dimension splicing to obtain frequency domain features

[0064] The frequency domain features Send it to the prediction head to get the rough prediction result Y of face forgery positioning coarse .

[0065] In this solution, the rough prediction result Y of the forged face location is coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features include:

[0066] The frequency domain characteristics and the rough prediction result Y of the face forgery positioning coarse Perform Hadamard dot product operation to obtain frequency domain features

[0067] The frequency domain characteristics Perform graph convolution operation to obtain frequency domain features

[0068] In this solution, according to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine include:

[0069] For the feature X fusion The local channel vector V local and the global channel vector V no-local Add them together, and pass the Sigmoid activation function to obtain the new vectors, which are then combined with the spatial features S C and frequency domain characteristics Multiply them together to get the spatial features of the fused frequency domain information and frequency domain features that fuse spatial information

[0070] The spatial domain features of the fused frequency domain information are and frequency domain features that fuse spatial information Add together to get fused cross-domain features

[0071] The fused cross-domain features Send it to the prediction head to get the refined prediction result Y of face forgery positioning fine .

[0072] In this solution, the rough prediction result Y based on the face forgery positioning coarse And the refined prediction result Y fine Calculate the final loss function include:

[0073] Calculate the rough prediction results Y of face forgery positioning respectively coarse And the refined prediction result Y fine The damage function of and refined prediction damage function

[0074] The refined prediction result Y fine Perform binary classification weighted loss calculation to obtain the binary classification weighted loss function

[0075] The rough prediction damage function Refined prediction of damage function and binary classification weighted loss function; Add them together to get the final loss function

[0076] A third aspect of the present invention provides a computer-readable storage medium, which includes a dual-domain face forgery positioning method program based on deep learning. When the dual-domain face forgery positioning method program based on deep learning is executed by a processor, the steps of the dual-domain face forgery positioning method based on deep learning as described in any one of the above items are implemented.

[0077] The present invention discloses a dual-domain face forgery positioning method, system and medium based on deep learning. The present invention collects diversified information from RGB and frequency domains for deep forgery detection. Specifically, a multi-band feature sensor is first introduced to obtain frequency features of different frequency bands to deeply explore the traces of face forgery. Then, a spatial frequency feature fusion module is designed to fully integrate RGB-frequency domain information and utilize more low-level forgery traces and structural textures. The face forgery area is located by cross-domain feature fusion, thereby improving the security and reliability of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flow chart of a dual-domain face forgery location method based on deep learning of the present invention is shown;

[0079] Figure 2 The present invention shows a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Get the flowchart of the method;

[0080] Figure 3 The present invention shows a method of combining multi-scale features {x 1_out , x 2_out , x 3_out , x 4_out}Flowchart of the method for transforming from spatial domain to frequency domain;

[0081] Figure 4 A block diagram of a dual-domain face forgery positioning system based on deep learning of the present invention is shown. DETAILED DESCRIPTION

[0082] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0083] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0084] Figure 1 A flow chart of a dual-domain face forgery positioning method based on deep learning of the present invention is shown.

[0085] like Figure 1 As shown, the present invention discloses a dual-domain face forgery positioning method based on deep learning, comprising:

[0086] S102, obtain the face RGB image X image ;

[0087] S104, extracting the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out};

[0088] S106, by discrete cosine transform, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination

[0089] S108, combining the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ;

[0090] S110, roughly predicting the forged face location result Y coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features

[0091] S112, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out} Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ;

[0092] S114, calculating the feature X fusion The local channel vector V localand the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ;

[0093] S116, based on the rough prediction result Y of the face forgery positioning coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

[0094] According to an embodiment of the present invention, the face RGB image X is extracted by the ResNet18 encoder. image The semantic feature X semantic , and decouple the convolutional units by dilating the RGB image X image The semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}.

[0095] Then, multi-band frequency features are extracted based on the multi-scale features in the spatial domain, and the multi-scale features are combined {x 1_out , x 2_out , x 3_out , x 4_out}Transform from spatial domain to frequency domain. Combine the obtained frequency domain features The channel dimension is spliced ​​and sent to the prediction head, and the rough prediction result Y of face forgery positioning is obtained through the prediction head coarse .

[0096] Traditional semantic segmentation networks are not suitable for facial manipulation localization tasks because they interfere with the network's tracking of forgery clues. This paper designs a multi-band feature perceptron that uses frequency domain differences to extract frequency information to suppress the interference of semantic target features. The multi-band feature perceptron learns semantically irrelevant forgery features from the perspective of different frequency bands to improve the effect of facial manipulation localization.

[0097] The discrete cosine filter is defined as follows:

[0098]

[0099] stu∈{0, 1,…,U-1}, v∈{0, 1,…,V-1},

[0100] in is a two-dimensional feature matrix, is the spectral feature of 2D discrete cosine transform. U and V represent the number of frequency basis functions of DCT transform. u∈{0, 1, ..., U-1} and v∈{0, 1, ..., V-1} represent the index of frequency basis functions, and their values ​​range from 0 to U-1 and from 0 to V-1. By changing the values ​​of U and V, we can control the accuracy and frequency resolution of DCT transform.

[0101] Then, cross-domain feature fusion is performed on the frequency domain features and spatial domain features. In the face forgery positioning task, the frequency domain features and spatial domain features describe the characteristics of the signal or image from different angles. The frequency domain features can provide frequency components and spectrum information about the signal by converting the signal to the frequency domain. The spatial domain features directly describe the distribution and structure of the signal or image in space. By fusing these two different features, a more comprehensive and diverse feature representation can be obtained. Frequency domain features and spatial domain features can capture different information, such as frequency components, textures, shapes, etc. By fusing them, this information can be comprehensively utilized and a richer and more accurate feature representation can be provided. Therefore, the present invention designs a spatial frequency feature fusion module to fully fuse spatial-frequency domain information and utilize more low-level forgery traces and structural textures.

[0102] In the process of cross-domain feature fusion, feature X fusion The local channel vector V local and the global channel vector V no-local The calculation method is expressed as follows:

[0103] V local =BN(Conv point-wise (ReLU(BN(Conv point-wise (X fusion )))))

[0104] V no-local =BN(Conv point-wise (ReLU(BN(Conv point-wise (GAP(X fusion ))))))

[0105] Among them, V local represents the local channel vector, V no-local represents the global channel vector, BN is the normalization layer, ReLU is the activation function, Conv point-wise is a point-by-point convolution operation, GAP is a global average pooling operation, and X fusion Represents the spatial feature S C Frequency domain characteristics Add up to get the features.

[0106] The global channel vector V no-local With the local channel vector V local Add, and pass through the Sigmoid activation function, and finally add to S C and Multiply, add the obtained fusion features and pass them through the prediction head to finally obtain the refined prediction result Y of face forgery positioning fine The model outputs a rough prediction result Y of the forged face location. coarse And the refined prediction result Y fine , and calculate Y coarse and Y fine The loss function is used to roughly predict the damage function and refined prediction damage function Based on Y fine Calculate the binary classification weighted loss function right and The final damage function is obtained by adding them together, and the model performance is evaluated by the final damage function.

[0107] Figure 2 The present invention shows a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Get the flowchart of the method.

[0108] like Figure 2 As shown, according to an embodiment of the present invention, the extraction of the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out},include:

[0109] S202, the face RGB image X image Feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic ;

[0110] S204, the semantic feature X semantic Input the dilated layered decoupled convolution unit to perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out};

[0111] The process of dilated convolution by dilated layered decoupled convolution unit is expressed as:

[0112] x1, x2, x3, x4 = Conv 1×1 (X semantic )

[0113] x1_out=x1

[0114] x2_out = DWConv r＝1 (x2)+x2

[0115] x3_out = DWConv r＝2 (x2+x3)+x3

[0116] x4_out = DWConv r＝3 (x3+x4)+x4

[0117] Among them, x1, x2, x3, x4 are the features obtained by the 1×1 convolution layer, Conv 1×1 For 1×1 convolution, DWConv r＝i is a depthwise separable convolution with dilation rate i, x 1_out , x 2_out , x 3_out , x 4_out is a multi-scale feature, X semantic is a semantic feature.

[0118] It should be noted that for the input face RGB image X image , first feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic . However, ResNet18 has no spatial context and cannot obtain multi-scale semantic information of faces. To solve this problem, the present invention creates a dilated hierarchical decoupled convolution unit. Adding dilated convolution can expand the receptive field without changing the number of network parameters and improve the network's ability to extract multi-scale information. The convolution feature map input of each layer comes from the independent output of the previous layer. Each layer uses a different convolution rate for dilated convolution and uses residual connections to avoid the loss of local information.

[0119] Then the semantic feature X semantic Input expansion layer decoupled convolution unit, first semantic feature X semanticIt will pass through a 1×1 convolution layer to get the feature combination {x1, x2, x3, x4}. x1 is x1_out, then x2 passes through a depthwise separable convolution with a dilation rate of 1 and is added to x2 to get x2_out. Then x2+x3 passes through a depthwise separable convolution with a dilation rate of 2 and is added to x3 to get x3_out. Then x3+x4 passes through a depthwise separable convolution with a dilation rate of 3 and is added to x4 to get x4_out. The final output is the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}.

[0120] Figure 3 The present invention shows a method of combining multi-scale features {x 1_out , x 2_out , x 3_out , x 4_out}Flowchart of the method for transforming from spatial domain to frequency domain.

[0121] like Figure 3 As shown, according to an embodiment of the present invention, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination include:

[0122] S302, combining the multi-scale features {x 1_out , x 2_out , x 3_out , x 4_out} to split and obtain the multi-scale feature combination after splitting

[0123] S304, combining the split multi-scale features by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination

[0124] It should be noted that the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Transform from spatial domain to frequency domain, for the multi-scale feature combination of the input discrete cosine transform module {x 1_out , x 2_out , x 3_out , x 4_out}, first split it according to the channel dimension. For example, split into in Among them, C represents the number of channels of the input feature map, h represents the height of the input feature map, w represents the width of the input feature map, and i is the number of features. Apply the discrete cosine transform algorithm separately to obtain its corresponding spectral characteristics, and then splice them according to the channel dimension. The specific formula is as follows:

[0125]

[0126]

[0127]

[0128]

[0129] in, is the frequency domain feature, is the multi-scale feature after splitting, i∈{0, 1,…, n}. n is the number of frequency basis functions of DCT transform, It is the discrete cosine transform algorithm under the corresponding number of transformations.

[0130] By performing domain conversion on the multi-scale feature combination, the frequency domain feature combination is obtained It not only has multi-scale semantic information, but also has multi-band frequency domain features, which can help neural networks better mine forgery clues and improve the accuracy of face forgery positioning.

[0131] According to an embodiment of the present invention, the frequency domain features are combined Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ,include:

[0132] The frequency domain feature combination Perform channel dimension splicing to obtain frequency domain features

[0133] The frequency domain features Send it to the prediction head to get the rough prediction result Y of face forgery positioning coarse .

[0134] It should be noted that the frequency domain features are combined Perform channel dimension splicing to obtain frequency domain features To highlight The present invention adds an auxiliary supervision for the forgery feature in A rough prediction result Y of face forgery positioning will be obtained through a prediction head coarse The prediction head consists of a 3×3 convolution layer, an upsampling layer, and a sigmoid activation function. The specific formula is as follows:

[0135]

[0136] Among them, prediction head represents the prediction head, concat is the concatenation operation of the channel dimension, It is the frequency domain feature.

[0137] According to an embodiment of the present invention, the rough prediction result Y of the forged face location coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features include:

[0138] The frequency domain characteristics and the rough prediction result Y of the face forgery positioning coarse Perform Hadamard dot product operation to obtain frequency domain features

[0139] The frequency domain characteristics Perform graph convolution operation to obtain frequency domain features

[0140] It should be noted that the Hadamard dot product is then used to activate the forged features in the multi-band features. The Hadamard dot product can extract local correlations in the feature map. By multiplying the two feature maps element by element, the Hadamard dot product can emphasize the similar parts of the two feature maps and suppress the dissimilar parts. This helps to extract important information in the feature map and reduce noise or irrelevant features.

[0141]

[0142] in, is the frequency domain feature obtained after the Hadamard dot product operation, is the frequency domain feature obtained by splicing, Y coarse This is a rough prediction result for face forgery positioning. represents the Hadamard dot product operation, for The matrix transpose of .

[0143] Then, graph learning is used to fuse multi-band features. Frequency domain features of different frequency bands are often interrelated, and there may be certain correlations or dependencies between them. Through graph learning, the association relationship between different frequency bands can be established, thereby realizing cross-band feature association learning. This helps to extract a more comprehensive and consistent frequency domain feature representation.

[0144]

[0145] in, is the frequency domain feature obtained after the graph convolution operation, It is the frequency domain feature obtained after the Hadamard dot product operation, and GCN is a graph convolution operation.

[0146] According to an embodiment of the present invention, according to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine include:

[0147] For the feature X fusion The local channel vector V local and the global channel vector V no-local Add them together, and pass the Sigmoid activation function to obtain the new vectors, which are then combined with the spatial features S C and frequency domain characteristics Multiply them together to get the spatial features of the fused frequency domain information and frequency domain features that fuse spatial information

[0148] The spatial domain features of the fused frequency domain information are and frequency domain features that fuse spatial information Add together to get fused cross-domain features

[0149] The fused cross-domain features Send it to the prediction head to get the refined prediction result Y of face forgery positioning fine .

[0150] It should be noted that the feature X fusion The global channel vector V no-local and the local channel vector V local Add, then pass through the Sigmoid activation function, and finally add to S C and Multiplying them together, we can get the spatial features of the fused frequency domain information. and frequency domain features that fuse spatial information The specific formula is as follows:

[0151]

[0152]

[0153] in, represents the spatial domain features of the fused frequency domain information, represents the frequency domain features of the fused spatial domain information, S C Represents the spatial characteristics, represents the frequency domain characteristics, V no-local represents the global channel vector, V local Represents the local channel vector, and sigmoid is the Sigmoid activation function.

[0154] Then and Add together to get the fused cross-domain features Will Send it to the prediction head to get the final refined prediction result Y of face forgery positioning fine .

[0155] According to an embodiment of the present invention, the rough prediction result Y based on the forged face positioning coarse And the refined prediction result Y fine Calculate the final loss function include:

[0156] Calculate the rough prediction results Y of face forgery positioning respectively coarse And the refined prediction result Y fine The damage function of and refined prediction damage function

[0157] The refined prediction result Y fine Perform binary classification weighted loss calculation to obtain the binary classification weighted loss function

[0158] The rough prediction damage function Refined prediction of damage function and binary classification weighted loss function; Add them together to get the final loss function

[0159] It should be noted that the rough prediction result Y of the face forgery positioning output by the model is coarse And the refined prediction result Y fine , use the MSE loss function to calculate, and the loss function is calculated as and However, face forgery localization is a special task. Sometimes the forged area is very small. Small targets usually have a lower signal-to-noise ratio and more noise interference, which places more stringent requirements on the regression results. By using a weighted regression loss function, the contribution of small target samples to the model training can be increased by adjusting the weights, reducing the impact of noise, and improving the robustness and stability of the model. Therefore, the present invention designs a binary classification weighted loss function. Specifically, for Y fine A threshold of 0.1 is designed. We set the pixels greater than 0.1 to 1 and the rest to 0. Then we calculate the density function of the tampered pixels and calculate the weight w based on the density function. j .

[0160]

[0161] Among them, PDF(i) represents the density function of the i-th category, w i is the weight of the corresponding category.

[0162] The binary classification weighted loss function can be expressed as:

[0163]

[0164] in, is the binary weighted loss function, S is the final prediction result, G s It’s a real label.

[0165] The calculation method of the final loss function is expressed as follows:

[0166]

[0167] in, represents the final loss function, represents the rough prediction damage function, represents the refined prediction damage function, Represents the binary classification weighted loss function.

[0168] Figure 4 A block diagram of a dual-domain face forgery positioning system based on deep learning of the present invention is shown.

[0169] like Figure 4 As shown, the second aspect of the present invention provides a dual-domain face forgery positioning system based on deep learning, comprising:

[0170] Data acquisition module, used to obtain face RGB image X image ;

[0171] Feature extraction conversion module module, used to extract the face RGB image X image The semantic feature Xsemantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}; The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination

[0172] A rough prediction module for face forgery positioning is used to combine the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ;

[0173] A cross-domain feature fusion module is used to roughly predict the result Y of the forged face location. coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features For the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out} Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ;

[0174] The face forgery positioning fine prediction module is used to calculate the feature X fusion The local channel vector V local and the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ;

[0175] A loss function evaluation module is used to roughly predict the result Y of the face forgery positioning according to the coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

[0176] According to an embodiment of the present invention, the face RGB image X is extracted by the ResNet18 encoder. image The semantic feature X semantic , and decouple the convolutional units by dilating the RGB image X image The semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}.

[0177] Then, multi-band frequency features are extracted based on the multi-scale features in the spatial domain, and the multi-scale features are combined {x 1_out , x 2_out , x 3_out , x 4_out}Transform from spatial domain to frequency domain. Combine the obtained frequency domain features The channel dimension is spliced ​​and sent to the prediction head, and the rough prediction result Y of face forgery positioning is obtained through the prediction head coarse .

[0178] Traditional semantic segmentation networks are not suitable for facial manipulation localization tasks because they interfere with the network's tracking of forgery clues. This paper designs a multi-band feature perceptron that uses frequency domain differences to extract frequency information to suppress the interference of semantic target features. The multi-band feature perceptron learns semantically irrelevant forgery features from the perspective of different frequency bands to improve the effect of facial manipulation localization.

[0179] The discrete cosine filter is defined as follows:

[0180]

[0181] stu∈{0, 1,…,U-1}, v∈{0, 1,…,V-1},

[0182] in is a two-dimensional feature matrix, is the spectral feature of 2D discrete cosine transform. U and V represent the number of frequency basis functions of DCT transform. u∈{0, 1, ..., U-1} and v∈{0, 1, ..., V-1} represent the index of frequency basis functions, and their values ​​range from 0 to U-1 and from 0 to V-1. By changing the values ​​of U and V, we can control the accuracy and frequency resolution of DCT transform.

[0183] Then, cross-domain feature fusion is performed on the frequency domain features and spatial domain features. In the face forgery positioning task, the frequency domain features and spatial domain features describe the characteristics of the signal or image from different angles. The frequency domain features can provide frequency components and spectrum information about the signal by converting the signal to the frequency domain. The spatial domain features directly describe the distribution and structure of the signal or image in space. By fusing these two different features, a more comprehensive and diverse feature representation can be obtained. Frequency domain features and spatial domain features can capture different information, such as frequency components, textures, shapes, etc. By fusing them, this information can be comprehensively utilized and a richer and more accurate feature representation can be provided. Therefore, the present invention designs a spatial frequency feature fusion module to fully fuse spatial-frequency domain information and utilize more low-level forgery traces and structural textures.

[0184] In the process of cross-domain feature fusion, feature X fusion The local channel vector V local and the global channel vector V no-local The calculation method is expressed as follows:

[0185] V local =BN(Conv point-wise (ReLU(BN(Conv point-wise (X fusion )))))

[0186] V no-local =BN(Conv point-wise (ReLU(BN(Conv point-wise (GAP(X fusion ))))))

[0187] Among them, V local represents the local channel vector, V no-local represents the global channel vector, BN is the normalization layer, ReLU is the activation function, Conv point-wise is a point-by-point convolution operation, GAP is a global average pooling operation, and X fusion Represents the spatial feature S C Frequency domain characteristics Add up to get the features.

[0188] The global channel vector V no-local With the local channel vector V local Add, and pass through the Sigmoid activation function, and finally add to S C and Multiply, add the obtained fusion features and pass them through the prediction head to finally obtain the refined prediction result Y of face forgery positioning fine The model outputs a rough prediction result Y of the forged face location. coarseAnd the refined prediction result Y fine , and calculate Y coarse and Y fine The loss function is used to roughly predict the damage function and refined prediction damage function Based on Y fine Calculate the binary classification weighted loss function right and The final damage function is obtained by adding them together, and the model performance is evaluated by the final damage function.

[0189] According to an embodiment of the present invention, the extraction of the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out},include:

[0190] The face RGB image X image Feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic ;

[0191] The semantic feature X semantic Input the dilated layered decoupled convolution unit to perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out};

[0192] The process of dilated convolution by dilated layered decoupled convolution unit is expressed as:

[0193] x1, x2, x3, x4 = Conv 1×1 (X semantic )

[0194] x1_out=x1

[0195] x2_out = DWConv r＝1 (x2)+x2

[0196] x3_out = DWConv r＝2 (x2+x3)+x3

[0197] x4_out = DWConv r＝3 (x3+x4)+x4

[0198] Among them, x1, x2, x3, x4 are the features obtained by the 1×1 convolution layer, Conv 1×1 For 1×1 convolution, DWConv r＝i is a depthwise separable convolution with dilation rate i, x 1_out , x 2_out , x 3_out , x 4_out is a multi-scale feature, X semantic is a semantic feature.

[0199] It should be noted that for the input face RGB image X image , first feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic . However, ResNet18 has no spatial context and cannot obtain multi-scale semantic information of faces. To solve this problem, the present invention creates a dilated hierarchical decoupled convolution unit. Adding dilated convolution can expand the receptive field without changing the number of network parameters and improve the network's ability to extract multi-scale information. The convolution feature map input of each layer comes from the independent output of the previous layer. Each layer uses a different convolution rate for dilated convolution and uses residual connections to avoid the loss of local information.

[0200] Then the semantic feature X semantic Input expansion layer decoupled convolution unit, first semantic feature X semantic It will pass through a 1×1 convolution layer to get the feature combination {x1, x2, x3, x4}. x1 is x1_out, then x2 passes through a depthwise separable convolution with a dilation rate of 1 and is added to x2 to get x2_out. Then x2+x3 passes through a depthwise separable convolution with a dilation rate of 2 and is added to x3 to get x3_out. Then x3+x4 passes through a depthwise separable convolution with a dilation rate of 3 and is added to x4 to get x4_out. The final output is the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}.

[0201] According to an embodiment of the present invention, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Converted to frequency domain feature combination include:

[0202] According to the channel dimension, the multi-scale feature combination {x 1_out , x 2_out , x3_out , x 4_out} to split and obtain the multi-scale feature combination after splitting

[0203] The multi-scale features after splitting are combined by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination

[0204] It should be noted that the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out}Transform from spatial domain to frequency domain, for the multi-scale feature combination of the input discrete cosine transform module {x 1_out , x 2_out , x 3_out , x 4_out}, first split it according to the channel dimension. For example, split into in Among them, C represents the number of channels of the input feature map, h represents the height of the input feature map, w represents the width of the input feature map, and i is the number of features. Apply the discrete cosine transform algorithm separately to obtain its corresponding spectral characteristics, and then splice them according to the channel dimension. The specific formula is as follows:

[0205]

[0206]

[0207]

[0208]

[0209] in, is the frequency domain feature, is the multi-scale feature after splitting, i∈{0, 1,…, n}. n is the number of frequency basis functions of DCT transform, It is the discrete cosine transform algorithm under the corresponding number of transformations.

[0210] By performing domain conversion on the multi-scale feature combination, the frequency domain feature combination is obtained It not only has multi-scale semantic information, but also has multi-band frequency domain features, which can help neural networks better mine forgery clues and improve the accuracy of face forgery positioning.

[0211] According to an embodiment of the present invention, the frequency domain features are combined Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ,include:

[0212] The frequency domain feature combination Perform channel dimension splicing to obtain frequency domain features

[0213] The frequency domain features Send it to the prediction head to get the rough prediction result Y of face forgery positioning coarse .

[0214] It should be noted that the frequency domain features are combined Perform channel dimension splicing to obtain frequency domain features To highlight The present invention adds an auxiliary supervision for the forgery feature in A rough prediction result Y of face forgery positioning will be obtained through a prediction head coarse The prediction head consists of a 3×3 convolution layer, an upsampling layer, and a sigmoid activation function. The specific formula is as follows:

[0215]

[0216] Among them, prediction head represents the prediction head, concat is the concatenation operation of the channel dimension, It is the frequency domain feature.

[0217] According to an embodiment of the present invention, the rough prediction result Y of the forged face location coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features include:

[0218] The frequency domain characteristics and the rough prediction result Y of the face forgery positioning coarse Perform Hadamard dot product operation to obtain frequency domain features

[0219] The frequency domain characteristics Perform graph convolution operation to obtain frequency domain features

[0220] It should be noted that the Hadamard dot product is then used to activate the forged features in the multi-band features. The Hadamard dot product can extract local correlations in the feature map. By multiplying the two feature maps element by element, the Hadamard dot product can emphasize the similar parts of the two feature maps and suppress the dissimilar parts. This helps to extract important information in the feature map and reduce noise or irrelevant features.

[0221]

[0222] in, is the frequency domain feature obtained after the Hadamard dot product operation, is the frequency domain feature obtained by splicing, Y coarse This is a rough prediction result for face forgery positioning. represents the Hadamard dot product operation, for The matrix transpose of .

[0223] Then, graph learning is used to fuse multi-band features. Frequency domain features of different frequency bands are often interrelated, and there may be certain correlations or dependencies between them. Through graph learning, the association relationship between different frequency bands can be established, thereby realizing cross-band feature association learning. This helps to extract a more comprehensive and consistent frequency domain feature representation.

[0224]

[0225] in, is the frequency domain feature obtained after the graph convolution operation, It is the frequency domain feature obtained after the Hadamard dot product operation, and GCN is a graph convolution operation.

[0226] According to an embodiment of the present invention, according to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine include:

[0227] For the feature X fusion The local channel vector V local and the global channel vector V no-local Add them together, and pass the Sigmoid activation function to obtain the new vectors, which are then combined with the spatial features S C and frequency domain characteristics Multiply them together to get the spatial features of the fused frequency domain information and frequency domain features that fuse spatial information

[0228] The spatial domain features of the fused frequency domain information are and frequency domain features that fuse spatial information Add together to get fused cross-domain features

[0229] The fused cross-domain features Send it to the prediction head to get the refined prediction result Y of face forgery positioning fine .

[0230] It should be noted that the feature X fusion The global channel vector V no-local and the local channel vector V local Add, then pass through the Sigmoid activation function, and finally add to S C and Multiplying them together, we can get the spatial features of the fused frequency domain information. and frequency domain features that fuse spatial information The specific formula is as follows:

[0231]

[0232]

[0233] in, represents the spatial domain features of the fused frequency domain information, represents the frequency domain features of the fused spatial domain information, S C Represents the spatial characteristics, represents the frequency domain characteristics, V no-local represents the global channel vector, V local Represents the local channel vector, and sigmoid is the Sigmoid activation function.

[0234] Then and Add together to get the fused cross-domain features Will Send it to the prediction head to get the final refined prediction result Y of face forgery positioning fine .

[0235] According to an embodiment of the present invention, the rough prediction result Y based on the forged face positioning coarse And the refined prediction result Y fine Calculate the final loss function include:

[0236] Calculate the rough prediction results Y of face forgery positioning respectively coarse And the refined prediction result Y fine The damage function of and refined prediction damage function

[0237] The refined prediction result Y fine Perform binary classification weighted loss calculation to obtain the binary classification weighted loss function

[0238] The rough prediction damage function Refined prediction of damage function and binary classification weighted loss function; Add them together to get the final loss function

[0239] It should be noted that the rough prediction result Y of the face forgery positioning output by the model is coarse And the refined prediction result Y fine , use the MSE loss function to calculate, and the loss function is calculated as and However, face forgery localization is a special task. Sometimes the forged area is very small. Small targets usually have a lower signal-to-noise ratio and more noise interference, which places more stringent requirements on the regression results. By using a weighted regression loss function, the contribution of small target samples to the model training can be increased by adjusting the weights, reducing the impact of noise, and improving the robustness and stability of the model. Therefore, the present invention designs a binary classification weighted loss function. Specifically, for Y fine A threshold of 0.1 is designed. We set the pixels greater than 0.1 to 1 and the rest to 0. Then we calculate the density function of the tampered pixels and calculate the weight w based on the density function. i .

[0240]

[0241] Among them, PDF(i) represents the density function of the i-th category, w i is the weight of the corresponding category.

[0242] The binary classification weighted loss function can be expressed as:

[0243]

[0244] in, It is a binary weighted loss function, S is the final predicted result, and Gs is the actual label.

[0245] The calculation method of the final loss function is expressed as follows:

[0246]

[0247] in, represents the final loss function, represents the rough prediction damage function, represents the refined prediction damage function, Represents the binary classification weighted loss function.

[0248] A third aspect of the present invention provides a computer-readable storage medium, which includes a dual-domain face forgery positioning method program based on deep learning. When the dual-domain face forgery positioning method program based on deep learning is executed by a processor, the steps of the dual-domain face forgery positioning method based on deep learning as described in any one of the above items are implemented.

[0249] The present invention discloses a dual-domain face forgery positioning method, system and medium based on deep learning. The present invention collects diversified information from RGB and frequency domains for deep forgery detection. Specifically, a multi-band feature sensor is first introduced to obtain frequency features of different frequency bands to deeply explore the traces of face forgery. Then, a spatial frequency feature fusion module is designed to fully integrate RGB-frequency domain information and utilize more low-level forgery traces and structural textures. The face forgery area is located by cross-domain feature fusion, thereby improving the security and reliability of face recognition.

[0250] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0251] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0252] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0253] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0254] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A dual-domain face forgery location method based on deep learning, characterized in that: include: Get face RGB image X image ; Extract the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }; The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }Converted to frequency domain feature combination Combining the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ; A rough prediction result Y of the forged face positioning coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features For the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out } Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ; Calculate the feature X fusion The local channel vector V local and the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ; According to the rough prediction result Y of the face forgery positioning coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

2. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: Extracting the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out },include: The face RGB image X image Feed it into the ResNet18 encoder to extract X image Semantic information, get semantic feature X semantic ; The semantic feature X semantic Input the dilated layered decoupled convolution unit to perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }; The dilated convolution process is expressed as: x1,x2,x3,x4=Conv 1×1 (X semantic ) x 1_ out=x1 x 2_ out=DWConv r＝1 (x2)+x2 x 3- out=DWConv r＝2 (x2+x3)+x3 x 4- out=DWConv r＝3 (x3+x4)+x4 Among them, x1, x2, x3, x4 are the features obtained by the 1×1 convolution layer, Conv 1×1 For 1×1 convolution, DWConv r＝i is a depthwise separable convolution with dilation rate i, x 1_out , x 2_out , x 3_out , x 4_out is a multi-scale feature, X semantic is a semantic feature.

3. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }Converted to frequency domain feature combination include: According to the channel dimension, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out } to split and obtain the multi-scale feature combination after splitting The multi-scale features after splitting are combined by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination 4. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: The frequency domain features are combined Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ,include: The frequency domain feature combination Perform channel dimension splicing to obtain frequency domain features The frequency domain features Send it to the prediction head to get the rough prediction result Y of face forgery positioning coarse .

5. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: The rough prediction result Y of the forged face positioning coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features include: The frequency domain characteristics and the rough prediction result Y of the face forgery positioning coarse Perform Hadamard dot product operation to obtain frequency domain features The frequency domain characteristics Perform graph convolution operation to obtain frequency domain features 6. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine include: For the feature X fusion The local channel vector V local and the global channel vector V no-local Add them together, and pass the Sigmoid activation function to obtain the new vectors, which are then combined with the spatial features S C and frequency domain characteristics Multiply them together to get the spatial features of the fused frequency domain information and frequency domain features that fuse spatial information The spatial domain features of the fused frequency domain information are and frequency domain features that fuse spatial information Add together to get fused cross-domain features The fused cross-domain features Send it to the prediction head to get the refined prediction result Y of face forgery positioning fine .

7. The dual-domain face forgery location method based on deep learning according to claim 1, characterized in that: The rough prediction result Y based on the forged face positioning coarse And the refined prediction result Y fine Calculate the final loss function include: Calculate the rough prediction results Y of face forgery positioning respectively coarse And the refined prediction result Y fine The damage function of and refined prediction damage function The refined prediction result Y fine Perform binary classification weighted loss calculation to obtain the binary classification weighted loss function The rough prediction damage function Refined prediction of damage function and binary classification weighted loss function; Add them together to get the final loss function 8. A dual-domain face forgery positioning system based on deep learning, characterized in that: include: Data acquisition module, used to obtain face RGB image X image ; Feature extraction conversion module module, used to extract the face RGB image X image The semantic feature X semantic , by dilating the layered decoupling convolutional units for the semantic feature X semantic Perform dilated convolution to obtain a multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }; The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }Converted to frequency domain feature combination A rough prediction module for face forgery positioning is used to combine the frequency domain features Frequency domain features obtained by channel dimension splicing Send it to the prediction head and output the rough prediction result Y of face forgery positioning coarse ; A cross-domain feature fusion module is used to roughly predict the result Y of the forged face location. coarse Perform Hadamard dot product operations and graph convolution operations to obtain frequency domain features For the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out } Perform channel dimension splicing to obtain spatial domain features S C , the spatial feature S C With the frequency domain characteristics Add together to get feature X fusion ; The face forgery positioning fine prediction module is used to calculate the feature X fusion The local channel vector V local and the global channel vector V no-local According to the feature X fusion The local channel vector V local and the global channel vector V no-local Calculate and obtain the refined prediction result Y of face forgery positioning fine ; A loss function evaluation module is used to roughly predict the result Y of the face forgery positioning according to the coarse And the refined prediction result Y fine Calculate the final loss function Based on the final loss function Evaluate the prediction results and output face forgery location prediction data.

9. The deep learning-based dual-domain face forgery positioning system according to claim 8, characterized in that: The multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out }Converted to frequency domain feature combination include: According to the channel dimension, the multi-scale feature combination {x 1_out , x 2_out , x 3_out , x 4_out } to split and obtain the multi-scale feature combination after splitting The multi-scale features after splitting are combined by discrete cosine transform algorithm Calculate and obtain the corresponding spectrum features, and concatenate the spectrum features according to the channel dimension to obtain the frequency domain feature combination 10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a dual-domain face forgery positioning method program based on deep learning. When the dual-domain face forgery positioning method program based on deep learning is executed by a processor, the steps of the dual-domain face forgery positioning method based on deep learning as described in any one of claims 1 to 7 are implemented.

Images