Forged face detection method, medium and device based on frequency attention feature fusion

Abstract

The present invention relates to a method, medium, and device for detecting fake faces based on frequency-attention feature fusion, which can fully aggregate a variety of facial feature information, effectively enhance the expressiveness of facial features, and thus improve the accuracy of fake face detection. The detection method fully utilizes the spatial information of the image by adding a spatial-frequency feature fusion module. At the same time, it can also focus on the fake traces produced by upsampling operations during the fake face generation process through frequency-domain information, thereby achieving multi-dimensional information aggregation. The variant attention module can obtain the optimal feature representation through the enhanced attention operation, greatly improving the generalization of fake face detection and the robustness against various unknown tampering methods in real scenes, and improving the resolution ability of the model, so that it can maintain good results in real scenes.

Description

Technical Field

[0001] The present invention relates to the field of deep fake detection technology, and in particular to a method, medium and device for detecting fake faces based on frequency attention feature fusion. Background Art

[0002] Thanks to significant breakthroughs in computer vision technologies such as audio and video synthesis, face recognition, generative adversarial networks, and variational autoencoders, deepfake technology, led by DeepFake, has emerged. Deepfakes use intelligent methods such as variational autoencoders and generative adversarial networks to reproduce, synthesize, edit, and replace images, videos, or audio to achieve realistic facial replacement. These technologies allow users to arbitrarily tamper with faces in target videos, mimicking the expressions and gestures of real people, and create fictitious faces. This has led to a large number of fake videos online, which not only violates personal privacy but also poses a threat to social stability. Therefore, the development of deepfake detection technology is urgent.

[0003] Existing deepfake detection technologies can be divided into those based on traditional image forensics and those based on deep learning. Traditional image forensics methods use traditional digital image processing techniques to detect and identify fake faces in images. They compare different image features between fake and real faces, such as texture, lighting, facial shape, and shadows, and analyze these features to detect fake faces. Deep learning methods typically utilize convolutional neural network models to extract and analyze deep features such as facial attributes, structure, and details in an image to determine whether the image is real or fake. The primary challenge in fake face detection technology lies in effectively identifying different types of fake attacks, including image synthesis, deepfakes, and video synthesis. Therefore, current research is focusing on combining multiple techniques, including texture analysis, spectral analysis, motion analysis, and 3D models, to improve the accuracy and robustness of fake face detection.

[0004] Although current deep learning models are capable of extracting features effectively, the difference between real and fake faces is becoming increasingly difficult to discern as GAN-based images become increasingly realistic. The features extracted by convolutional neural network models lack sufficient differentiation between different datasets, resulting in reduced accuracy in fake face detection models. To effectively utilize the different levels of features extracted by convolutional neural networks, many researchers have resorted to simple addition and concatenation operations to fuse features. However, since features from different sources and levels can exhibit significant inconsistencies at different scales and semantic levels, this approach can compromise model performance. Summary of the Invention

[0005] In view of the above problems, the present invention provides a method, medium and device for detecting fake faces based on frequency attention feature fusion, which solves the problem of low accuracy of existing deep learning models in identifying fake face detection models.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting fake faces based on frequency-attention feature fusion, comprising the following steps:

[0007] Obtain a sample image, where the sample image is image information containing a face area;

[0008] The sample image is subjected to image enhancement processing by an image enhancement module to obtain a first face image and a second face image, wherein the first face image and the second face image are enhanced using different image enhancement methods;

[0009] The first and second facial images are converted into the frequency domain using discrete cosine transform, and the first-level spectrograms corresponding to the two are obtained respectively. The spectrum image information of different frequency domains in the first-level spectrogram is obtained, and the spectrum image information of different frequency domains is spliced along the channel dimension to obtain the second-level spectrograms corresponding to the two. The second-level spectrograms are converted back to the spatial domain using inverse discrete cosine transform to obtain a first frequency domain enhanced image corresponding to the first facial image and a second frequency domain enhanced image corresponding to the second facial image respectively.

[0010] performing feature extraction on the first face image, the second face image, the first frequency-domain enhanced image, and the second frequency-domain enhanced image, and fusing the extracted feature information to obtain a first fused feature and a second fused feature containing spatial domain information and frequency domain information, and processing the first fused feature and the second fused feature using an attention mechanism to obtain a feature representation vector;

[0011] The feature representation vector is input into the first branch and the second branch respectively for network training, and the training results of the first branch and the second branch are superimposed to obtain a trained deep fake face detection model; the first branch is configured to use a pooling layer and a fully connected layer to obtain a first prediction result for judging authenticity, and to optimize the first prediction result using a cross-entropy loss function; the second branch is configured to use a cosine similarity loss function to calculate the similarity of the feature representation vectors of the first face image and the second face image, and to shorten the distance between the same real samples.

[0012] In some embodiments, the discrete cosine transform is represented by formula (1), which is as follows:

[0013] ;

[0014] in, is the coefficient after discrete cosine transform, and the inverse discrete cosine transform is expressed by formula (2), which is as follows:

[0015] ,

[0016] ;

[0017] in, is the sample image after image enhancement processing, N is the number of pixels on the side of the sample image after image enhancement processing, is the compensation coefficient, which can make the discrete cosine transform matrix an orthogonal matrix.

[0018] In some embodiments, spectrum image information in different frequency domains is obtained by formula (3), which is as follows:

[0019] ;

[0020] in, Represents the operation of concatenating feature maps along the channel dimension; and represent discrete cosine transform and inverse discrete cosine transform respectively; Respectively represent low frequency, medium frequency and high frequency filtering, .

[0021] In some embodiments, a convolutional neural network is used to extract features from the first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image. The convolutional neural network uses Xception as a backbone network. The feature extraction step includes:

[0022] Perform low-level feature extraction on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively, where the low-level features are the Separable Conv layer and the ReLU layer;

[0023] The extracted low-level feature information is input into the Xception network, and the middle-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The middle layer consists of 12 Block modules of the Xception network.

[0024] The extracted middle-layer feature information is input into the Xception network, and high-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The high-level layers are Separable Conv layer, ReLU layer and Pooling layer.

[0025] In some embodiments, feature information extracted from the first facial image, the second facial image, the first frequency domain enhanced image, and the second frequency domain enhanced image is calculated by a spatial domain and frequency domain feature fusion module, which is calculated based on a channel attention module. The calculation process of the channel attention module includes the following steps:

[0026] For the input feature map X The first function analysis and the second function analysis are performed respectively. The first function analysis includes extracting local face image information in the first point-by-point convolution layer, the nonlinear activation function ReLU layer, and the second point-by-point convolution layer. The extraction result is obtained by formula (4), which is as follows:

[0027] ;

[0028] in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, and the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, and the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate;

[0029] The second function analysis includes extracting global face image information in the global average pooling layer, the first point-by-point convolution layer, the nonlinear activation function ReLU layer, and the second point-by-point convolution layer. The global average pooling is obtained by formula (5), which is as follows:

[0030] ;

[0031] in, H.W. is the size of the feature map, is the global pooling information, global information Obtained by formula (6), formula (6) is as follows:

[0032] ;

[0033] in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, and the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, and the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate;

[0034] Using nonlinear activation function Sigmoid The polarity features of local face image information extraction and global face image information extraction are fused to obtain a feature map. The feature fusion is obtained by formula (7), which is as follows:

[0035] ;

[0036] in, represents a nonlinear activation function, Represents an element-wise addition operation.

[0037] In some embodiments, the calculation process of the spatial and frequency domain feature fusion module includes the following steps:

[0038] A convolutional neural network is used to extract spatial domain feature maps and frequency domain feature maps, and the spatial domain feature maps and frequency domain feature maps are added together to obtain a first-level feature map;

[0039] The first-level feature map is input into the channel attention module to obtain the second-level feature map containing the image spatial domain, frequency domain, global and local information;

[0040] The secondary feature map is residually connected with the spatial domain feature map and the frequency domain feature map and outputted to obtain the final fusion feature. The final fusion feature is obtained by formula (8) and (9). Formula (8) is as follows:

[0041] ;

[0042] Formula (9) is as follows:

[0043] ;

[0044] in, 、 Represent the characteristics of the spatial domain feature map and the frequency domain feature map respectively, 、 They represent the features of the spatial domain feature map and the frequency domain feature map after passing through the spatial domain and frequency domain fusion modules, and M represents the calculation process of the channel attention module.

[0045] In some embodiments, the attention mechanism includes a variant attention module, and the computation process of the variant attention module includes the following steps:

[0046] The output results of the spatial domain and frequency domain feature fusion modules are summed and input into the channel attention module to obtain its output feature map;

[0047] Perform attention calculation on the output results of the spatial domain and frequency domain feature fusion modules to obtain the corresponding attention maps of global and local artifact connections;

[0048] The attention maps connecting global and local artifacts are summed and input into the channel attention module to obtain the final attention weight matrix;

[0049] Perform the first attention analysis calculation and the second attention analysis calculation on the final attention weight matrix. The first attention analysis calculation multiplies the final attention weight matrix with the spatial feature map to obtain the first attention analysis eigenvalue. The second attention analysis calculation performs an inverted attention operation on the final attention weight matrix and then multiplies it with the frequency domain feature map to obtain the second attention analysis eigenvalue.

[0050] The first attention analysis eigenvalue and the second attention analysis eigenvalue are added together to calculate a feature representation vector;

[0051] The feature representation vector is obtained by formula (10) and formula (11). Formula (10) is as follows:

[0052] ;

[0053] Formula (11) is as follows:

[0054] ;

[0055] Among them, A is the feature obtained by the first attention calculation, and Z is the final feature representation output by the encoding network. Represents an element-wise multiplication operation.

[0056] In some embodiments, the cross entropy loss function is expressed by formula (12), which is as follows:

[0057] ;

[0058] in, for The output probability of the function, For the During the training process, we define 1 as a fake face and 0 as a real face. is the cross entropy loss function value;

[0059] The cosine similarity loss function is expressed by formula (13), which is as follows:

[0060] ;

[0061] in, Indicates calculating the cosine distance between two features. is the cosine similarity loss function value;

[0062] Face detection model Obtained by formula (14), formula (14) is as follows:

[0063] ;

[0064] Where m represents the weight balancing parameter.

[0065] In a second aspect, the present invention further provides a computer-readable storage medium storing computer program instructions, which implement the method described in the first aspect when executed by a processor.

[0066] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0067] Different from the existing technology, the above technical solution can fully aggregate a variety of facial feature information, effectively enhance the expressive ability of facial features, and thus improve the accuracy of fake face detection; by adding a spatial domain and frequency domain feature fusion module, it fully utilizes the spatial domain information of the image. At the same time, it can also use the frequency domain information to pay attention to the fake traces caused by upsampling operations in the fake face generation process, thereby realizing multi-dimensional information aggregation; the variant attention module can obtain the optimal feature representation through enhanced attention operations, greatly improving the generalization of fake face detection and its robustness against various unknown tampering methods in real scenes, and improving the resolution ability of the model, so that it can maintain good results in real scenes.

[0068] The above-mentioned description of the invention content is only an overview of the technical solution of the present invention. In order to enable ordinary technicians in this field to more clearly understand the technical solution of the present invention, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned objects and other objects, features and advantages of the present invention easier to understand, the following is an explanation in combination with the specific implementation methods and drawings of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are only used to illustrate the principles, implementations, applications, features, and effects of the specific embodiments of the present invention and other related contents, and are not to be considered as limiting the present invention.

[0070] In the drawings of the specification:

[0071] Figure 1 This is the first flow chart of the forged face detection method of the present invention;

[0072] Figure 2 This is a second flow chart of the forged face detection method of the present invention;

[0073] Figure 3 This is a convolutional neural network flow chart of the fake face detection method of the present invention;

[0074] Figure 4 This is a functional block diagram of the channel attention module of the present invention;

[0075] Figure 5 This is a principle block diagram of the spatial domain and frequency domain feature fusion module according to the present invention;

[0076] Figure 6 This is a principle block diagram of the variant attention module according to the present invention;

[0077] Figure 7 The figure is a schematic diagram of an electronic device storing the forged face detection method according to a specific embodiment.

[0078] The reference numerals in the above drawings are described as follows:

[0079] 1. Electronic equipment;

[0080] 11. Memory;

[0081] 12. Processor. DETAILED DESCRIPTION

[0082] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of the present invention, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0083] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the term "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in the present invention, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0084] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which the present invention belongs. The use of relevant terms herein is only for describing specific embodiments and is not intended to limit the present invention.

[0085] In the description of the present invention, the term "and / or" is used to describe a logical relationship between objects, indicating that three possible relationships exist. For example, A and / or B means: A exists, B exists, and both A and B exist. Furthermore, the character " / " generally indicates that the objects are in a logical "or" relationship.

[0086] In the present invention, terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0087] Without further limitations, in the present invention, the words "include", "comprise", "have" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product that includes the elements, so that the process, method or product that includes a series of elements may include not only those limited elements, but also other elements that are not explicitly listed, or also include elements inherent to such process, method or product.

[0088] Consistent with the understanding in the Examination Guidelines, in this disclosure, expressions such as "greater than," "less than," and "exceed" are understood to exclude the number itself; expressions such as "above," "below," and "within" are understood to include the number itself. Furthermore, in the description of the embodiments of this disclosure, "multiple" means two or more (including two). Similar expressions related to "multiple," such as "multiple groups" and "multiple times," are also understood in this manner, unless otherwise specifically defined.

[0089] See also Figures 1 to 6 This embodiment provides a method for detecting fake faces based on frequency-attention feature fusion, comprising the following steps:

[0090] S1. Obtain a sample image, where the sample image is image information containing a face area;

[0091] S2. Performing image enhancement processing on the sample image through an image enhancement module to obtain a first face image and a second face image, wherein the first face image and the second face image are enhanced using different image enhancement methods;

[0092] S3. Using discrete cosine transform to convert the first and second facial images into the frequency domain, respectively obtaining primary spectrograms corresponding to the two, obtaining spectrum image information of different frequency domains in the primary spectrograms, and splicing the spectrum image information of different frequency domains along the channel dimension to obtain secondary spectrograms corresponding to the two, and using inverse discrete cosine transform to convert the secondary spectrograms back to the spatial domain to obtain a first frequency domain enhanced image corresponding to the first facial image and a second frequency domain enhanced image corresponding to the second facial image;

[0093] S4. Extract features from the first face image, the second face image, the first frequency-domain enhanced image, and the second frequency-domain enhanced image, fuse the extracted feature information to obtain a first fused feature and a second fused feature containing spatial domain information and frequency domain information, and process the first fused feature and the second fused feature using an attention mechanism to obtain a feature representation vector;

[0094] S5. Input the above two feature representation vectors into the first branch and the second branch for network training, and superimpose the training results of the first branch and the second branch to obtain a trained deep fake face detection model; the first branch is configured to use a pooling layer and a fully connected layer to obtain a first prediction result for judging authenticity, and use a cross-entropy loss function to optimize the first prediction result; the second branch is configured to use a cosine similarity loss function to calculate the similarity of the feature representation vectors of the first face image and the second face image, and shorten the distance between the same real samples.

[0095] In this embodiment, the sample image is represented by the letter X. The sample image needs to be preprocessed to become an image of a preset size. For example, the preset size can be 299x299x3. The source of the sample image is image information obtained from real face videos and forged face-related videos from the same source. The specific method of obtaining the facial image information can be: detecting the face through the MTCNN face detector; then cropping the face area to a preset size. After preprocessing, the sample image is image information within a preset size range containing only the face area. Optionally, the collected sample images are divided into training and test sets. The forged face datasets used include: Face Forensics++ (FF++), Celeb-DF, and DFDC.

[0096] The sample image is enhanced by the image enhancement module. The image enhancement methods mainly include random flipping, random cropping, Gaussian blur, random occlusion, color change, etc. Each input sample image will randomly select two of the above image enhancement methods for image transformation. The two randomly selected image enhancement methods are respectively recorded as T 1 and T 2 ， Get the first face image and the second face image .

[0097] Using discrete cosine transform , Convert to the frequency domain to obtain the corresponding first-level spectrum, and use the learnable filter to adaptively obtain , The first-level spectrum diagram contains spectrum image information of different views of low-frequency, medium-frequency and high-frequency components; then the spectrum image information of different frequency domains obtained are spliced along the channel dimension to obtain a second-level spectrum diagram; and then the inverse discrete cosine transform is used to convert the second-level spectrum diagram back to the spatial domain to obtain the first frequency domain enhanced image processed in the frequency domain. , the second frequency domain enhanced image .

[0098] See also Figure 3 In this embodiment, feature extraction is performed on the first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image. The feature extraction is performed through a convolutional neural network, and the convolutional neural network is divided into three levels: low level, middle level, and high level.

[0099] Will , , , Input to the lower layer of the convolutional neural network, for the original image , , the encoding network extracts the basic visual features of the image, namely , ; For the frequency domain processed , , the encoding network extracts the artifact information generated during the generation of fake face images. The obtained frequency information can enhance the visual feature representation of the image, that is, , ;

[0100] and , and They respectively contain the spatial domain information and frequency domain information of the same image, so and Input it into the spatial domain and frequency domain feature fusion module, fuse it, and obtain low-level fusion features that contain both spatial domain information and frequency domain information; and Do the same operation.

[0101] The obtained low-level fusion features are input into the middle layer of the convolutional neural network, and then into the spatial and frequency domain feature fusion module, and fused to obtain the middle-level fusion features containing both spatial and frequency domain information; the obtained middle-level fusion features are then input into the high-level convolutional neural network, and then into the spatial and frequency domain feature fusion module, and fused to obtain the high-level fusion features containing both spatial and frequency domain information; the high-level fusion features are then input into the variant attention module, and the obtained features are respectively obtained. , Feature representation vector that aggregates diverse information (i.e., the first eigenvector) and (i.e. the second eigenvalue vector).

[0102] The above two feature representation vectors are input into the first branch and the second branch for network training, and the training results of the first branch and the second branch are superimposed to obtain a trained deep fake face detection model; the first branch is configured to use a pooling layer and a fully connected layer to obtain a first prediction result for judging authenticity, and to optimize the first prediction result using a cross-entropy loss function; the second branch is configured to use a cosine similarity loss function to calculate the similarity of the feature representation vectors of the first face image and the second face image, and to shorten the distance between the same real samples.

[0103] The above-mentioned embodiment can fully aggregate a variety of facial feature information, effectively enhance the expressiveness of facial features, and thus improve the accuracy of forged face detection; by adding a spatial domain and frequency domain feature fusion module, the spatial domain information of the image is fully utilized. At the same time, the forged traces caused by upsampling operations in the process of forged face generation can be paid attention to through frequency domain information, thereby realizing multi-dimensional information aggregation; the variant attention module can obtain the optimal feature representation through enhanced attention operations, greatly improving the generalization of forged face detection and its robustness against various unknown tampering methods in real scenes, and improving the resolution ability of the model, so that it can maintain good results in real scenes.

[0104] In some embodiments, the discrete cosine transform is represented by formula (1), which is as follows:

[0105] ;

[0106] in, is the coefficient after discrete cosine transform, and the inverse discrete cosine transform is expressed by formula (2), which is as follows:

[0107] ,

[0108] ;

[0109] in, is the sample image after image enhancement processing, is the pixel value, N is the number of pixels on the side of the sample image after image enhancement processing, is the compensation coefficient, It is the coefficient after discrete cosine transform, which can make the discrete cosine transform matrix an orthogonal matrix.

[0110] In some embodiments, spectrum image information in different frequency domains is obtained by formula (3), which is as follows:

[0111] ;

[0112] in, Represents the operation of concatenating feature maps along the channel dimension; and represent discrete cosine transform and inverse discrete cosine transform respectively; represents the input sample image, Respectively represent low frequency, medium frequency and high frequency filtering, .

[0113] See also Figure 3 In some embodiments, a convolutional neural network is used to extract features from the first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image. The convolutional neural network uses Xception as the backbone network. The feature extraction step includes:

[0114] Perform low-level feature extraction on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively, where the low-level features are the Separable Conv layer and the ReLU layer;

[0115] The extracted low-level feature information is input into the Xception network, and the middle-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The middle layer consists of 12 Block modules of the Xception network.

[0116] The extracted middle-layer feature information is input into the Xception network, and high-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The high-level layers are Separable Conv layer, ReLU layer and Pooling layer.

[0117] The first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image are input into the Xception convolutional network. The image information corresponding to the first face image and the second face image is recorded as RGB stream images, and the image information corresponding to the first frequency domain enhanced image and the second frequency domain enhanced image is recorded as frequency stream images. The features extracted from the RGB stream image and the frequency stream image respectively through the convolutional network are fused through the spatial domain and frequency domain feature fusion module to obtain a fused feature that contains both spatial domain information and frequency domain information. Taking the low-level, mid-level, and high-level feature extraction and fusion as an example, this process specifically includes the following steps:

[0118] Input RGB stream image and frequency stream image, RGB stream image and frequency stream image are in one-to-one correspondence;

[0119] The RGB stream image and the frequency stream image are input to the lower layer (Separable Conv + ReLU)*3 of the Xception network. The feature maps output by the RGB stream image and the frequency stream image are fused through the spatial domain and frequency domain feature fusion module. The fused low-level features are residually connected with the input of the previous step.

[0120] The RGB stream image and frequency stream image obtained by low-level fusion are input to the middle layer (Block)*3 of the Xception network. The output feature map is fused through the spatial domain and frequency domain feature fusion module. The fused features are residually connected with the input of the previous step respectively.

[0121] The RGB stream image and frequency stream image obtained by the middle layer fusion are input to the high layer [(Separable Conv+ReLU)*2+Pooling] of the Xception network. The output feature map is fused through the spatial domain and frequency domain feature fusion module. The fused features are residually connected with the input of the previous step.

[0122] The fused high-level input is fed into the variant attention module to obtain an enhanced feature representation that aggregates diverse information and has strong robustness.

[0123] See also Figure 4 In some embodiments, feature information extracted from the first facial image, the second facial image, the first frequency domain enhanced image, and the second frequency domain enhanced image is calculated by a spatial domain and frequency domain feature fusion module, which is calculated based on a channel attention module. The calculation process of the channel attention module includes the following steps:

[0124] For the input feature map XThe first function analysis and the second function analysis are performed respectively. The first function analysis includes extracting local face image information in the first point-by-point convolution layer, the nonlinear activation function ReLU layer, and the second point-by-point convolution layer. The extraction result is obtained by formula (4), which is as follows:

[0125] ;

[0126] in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, and the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, and the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate;

[0127] The second function analysis includes extracting global face image information in the global average pooling layer, the first point-by-point convolution layer, the nonlinear activation function ReLU layer, and the second point-by-point convolution layer. The global average pooling is obtained by formula (5), which is as follows:

[0128] ;

[0129] in, H.W. is the size of the feature map, is the global pooling information, global information Obtained by formula (6), formula (6) is as follows:

[0130] ;

[0131] in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, and the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, and the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate;

[0132] Using nonlinear activation function Sigmoid The polarity features of local face image information extraction and global face image information extraction are fused to obtain a feature map. The feature fusion is obtained by formula (7), which is as follows:

[0133] ;

[0134] in, represents a nonlinear activation function, Represents an element-wise addition operation.

[0135] In this embodiment, the first function analysis is mainly used to extract local face image information. Point-wise convolution can reduce the number of channels of the model, thereby reducing the amount of calculation and parameters, and thus reducing the complexity of the model. The second function analysis is mainly used to extract global face image information. Global average pooling is used to obtain global pooling information on each channel. GAP(X) , and then combined with the global pooling information GAP(X) Adopt and L(X) A consistent calculation formula to calculate global information G (X) .

[0136] The channel attention module can help the network dig out the forgery traces of deep fake faces by learning the intrinsic connection between local and global faces.

[0137] See also Figure 5 In some embodiments, the calculation process of the spatial and frequency domain feature fusion module includes the following steps:

[0138] A convolutional neural network is used to extract spatial domain feature maps and frequency domain feature maps, and the spatial domain feature maps and frequency domain feature maps are added together to obtain a first-level feature map;

[0139] The first-level feature map is input into the channel attention module to obtain the second-level feature map containing the image spatial domain, frequency domain, global and local information;

[0140] The secondary feature map is residually connected with the spatial domain feature map and the frequency domain feature map and outputted to obtain the final fusion feature. The final fusion feature is obtained by formula (8) and (9). Formula (8) is as follows:

[0141] ;

[0142] Formula (9) is as follows:

[0143] ;

[0144] in, 、 Respectively represent the features of the spatial domain feature map and the frequency domain feature map, that is, the features of the RGB stream and the frequency stream, 、 They represent the features of the spatial domain feature map and the frequency domain feature map (i.e., RGB stream and frequency stream) after the spatial domain and frequency domain fusion modules respectively. M represents the calculation process of the channel attention module. Figure 5 、 Figure 6 CAM stands for Channel Attention Module. The output of this algorithm will be used as input for the next layer of the convolutional neural network. For example, if the output of this algorithm is a low-level fusion feature value, it will be used as the input feature value for the middle-level fusion feature step, and so on.

[0145] See also Figure 6 In some embodiments, the attention mechanism includes a variant attention module, and the computation process of the variant attention module includes the following steps:

[0146] The output results of the spatial domain and frequency domain feature fusion modules are summed and input into the channel attention module to obtain its output feature map;

[0147] Perform attention calculation on the output results of the spatial domain and frequency domain feature fusion modules to obtain the corresponding attention maps of global and local artifact connections;

[0148] The attention maps connecting global and local artifacts are summed and input into the channel attention module to obtain the final attention weight matrix;

[0149] Perform the first attention analysis calculation and the second attention analysis calculation on the final attention weight matrix. The first attention analysis calculation multiplies the final attention weight matrix with the spatial feature map to obtain the first attention analysis eigenvalue. The second attention analysis calculation performs an inverted attention operation on the final attention weight matrix and then multiplies it with the frequency domain feature map to obtain the second attention analysis eigenvalue.

[0150] The first attention analysis eigenvalue and the second attention analysis eigenvalue are added together to calculate a feature representation vector;

[0151] The feature representation vector is obtained by formula (10) and formula (11). Formula (10) is as follows:

[0152] ;

[0153] Formula (11) is as follows:

[0154] ;

[0155] Among them, A is the feature obtained by the first attention calculation, and Z is the final feature representation output by the encoding network. Represents an element-wise multiplication operation.

[0156] In some embodiments, the cross entropy loss function is expressed by formula (12), which is as follows:

[0157]

[0158] in, for The output probability of the function, For the During the training process, we define 1 as a fake face and 0 as a real face. is the cross entropy loss function value;

[0159] The cosine similarity loss function is expressed by formula (13), which is as follows:

[0160] ;

[0161] in, Indicates calculating the cosine distance between two features. is the cosine similarity loss function value;

[0162] Face detection model Obtained by formula (14), formula (14) is as follows:

[0163] ;

[0164] Where m represents the weight balancing parameter.

[0165] The above-mentioned embodiment can fully aggregate a variety of facial feature information, effectively enhance the expressiveness of facial features, and thus improve the accuracy of forged face detection; by adding a spatial domain and frequency domain feature fusion module, the spatial domain information of the image is fully utilized. At the same time, the forged traces caused by upsampling operations in the process of forged face generation can be paid attention to through frequency domain information, thereby realizing multi-dimensional information aggregation; the variant attention module can obtain the optimal feature representation through enhanced attention operations, greatly improving the generalization of forged face detection and its robustness against various unknown tampering methods in real scenes, and improving the resolution ability of the model, so that it can maintain good results in real scenes.

[0166] In a second aspect, this embodiment further provides a computer-readable storage medium storing computer program instructions, which implement the method described in the first aspect when executed by a processor.

[0167] See also Figure 7In a third aspect, this embodiment further provides an electronic device 1, comprising a memory 11 and a processor 12, wherein the memory 11 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor 12 to implement the method described in the first aspect.

[0168] The electronic device 1 may be a tablet, mobile phone, laptop computer, desktop computer, etc. The storage medium / memory 11 includes, but is not limited to, RAM, ROM, magnetic disk, tape, optical disk, flash memory, USB flash drive, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc. The processor 12 includes, but is not limited to, a CPU (central processing unit 12), a GPU (graphics processing unit 12), an MCU (microprocessor 12), etc.

[0169] The above technical solution can fully aggregate a variety of facial feature information, effectively enhance the expressiveness of facial features, and thus improve the accuracy of forged face detection; by adding a spatial-frequency domain feature fusion module, the spatial information of the image is fully utilized. At the same time, the frequency domain information can also be used to focus on the forged traces caused by upsampling operations during the forged face generation process, thereby realizing multi-dimensional information aggregation; the variant attention module can obtain the optimal feature representation through enhanced attention operations, greatly improving the generalization of forged face detection and its robustness against various unknown tampering methods in real scenes, and improving the model's resolution ability, so that it can maintain good results in real scenes.

[0170] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present invention, this does not limit the scope of patent protection of the present invention. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concepts of the present invention using the contents recorded in the specification and drawings of the present invention, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of the present invention.

Claims

1. A fake face detection method based on frequency attention feature fusion, characterized in that: The following steps are involved: Acquire a sample image, wherein the sample image is image information containing a face area; Performing image enhancement processing on the sample image through an image enhancement module to obtain a first face image and a second face image, wherein the first face image and the second face image are enhanced using different image enhancement methods; Using a discrete cosine transform to convert the first facial image and the second facial image into the frequency domain, respectively obtaining primary spectrograms corresponding to the first and second facial images, obtaining spectrum image information of different frequency domains in the primary spectrograms, and splicing the spectrum image information of different frequency domains along the channel dimension to obtain secondary spectrograms corresponding to the first and second facial images, and using an inverse discrete cosine transform to convert the secondary spectrograms back into the spatial domain to obtain a first frequency domain enhanced image corresponding to the first facial image and a second frequency domain enhanced image corresponding to the second facial image, respectively; performing feature extraction on the first face image, the second face image, the first frequency-domain enhanced image, and the second frequency-domain enhanced image, and fusing the extracted feature information to obtain a first fused feature and a second fused feature containing spatial domain information and frequency domain information, and processing the first fused feature and the second fused feature using an attention mechanism to obtain a feature representation vector; The feature representation vector is respectively input into the first branch and the second branch for network training, and the training results of the first branch and the second branch are superimposed to obtain a trained deep fake face detection model; the first branch is configured to use a pooling layer and a fully connected layer to obtain a first prediction result for judging authenticity, and to optimize the first prediction result using a cross-entropy loss function; the second branch is configured to use a cosine similarity loss function to calculate the similarity of the feature representation vectors of the first face image and the second face image, and to shorten the distance between the same real samples.

2. The forged face detection method based on frequency attention feature fusion according to claim 1 is characterized in that: The discrete cosine transform is expressed by formula (1), which is as follows: ； in, is the coefficient after discrete cosine transformation, and the inverse discrete cosine transformation is expressed by formula (2), which is as follows: ， ； in, is the sample image after image enhancement processing, N is the number of pixels on the side of the sample image after image enhancement processing, is the compensation coefficient, which can make the discrete cosine transform matrix an orthogonal matrix.

3. The forged face detection method based on frequency attention feature fusion according to claim 1 or 2, characterized in that: The spectrum image information of different frequency domains is obtained by formula (3), which is as follows: ； in, Represents the operation of concatenating feature maps along the channel dimension; and represent discrete cosine transform and inverse discrete cosine transform respectively; Respectively represent low frequency, medium frequency and high frequency filtering, .

4. The method for detecting fake faces based on frequency-attention feature fusion according to claim 1 or 2, characterized in that: A convolutional neural network is used to extract features from the first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image. The convolutional neural network uses Xception as a backbone network. The feature extraction step includes: Performing low-level feature extraction on the first face image, the second face image, and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image, respectively, wherein the low-level features are a Separable Conv layer and a ReLU layer; The extracted low-level feature information is input into the Xception network, and the middle-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The middle layer is 12 Block modules of the Xception network; The extracted middle-layer feature information is input into the Xception network, and high-level feature extraction is performed on the first face image, the second face image and the corresponding first frequency domain enhanced image and the second frequency domain enhanced image respectively. The high-level layers are the Separable Conv layer, the ReLU layer and the Pooling layer.

5. The method for detecting fake faces based on frequency-attention feature fusion according to claim 4, characterized in that: The feature information extracted from the first face image, the second face image, the first frequency domain enhanced image, and the second frequency domain enhanced image is calculated by a spatial domain and frequency domain feature fusion module, and the spatial domain and frequency domain feature fusion module is calculated based on a channel attention module. The calculation process of the channel attention module includes the following steps: For the input feature map X The first function analysis and the second function analysis are performed respectively. The first function analysis includes extracting local face image information in the first point-by-point convolution layer, the nonlinear activation function ReLU layer, and the second point-by-point convolution layer. The extraction result is obtained by formula (4), which is as follows: ； in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate; The second function analysis includes extracting global face image information in a global average pooling layer, a first point-by-point convolution layer, a nonlinear activation function ReLU layer, and a second point-by-point convolution layer. The global average pooling is obtained by formula (5), which is as follows: ； in, H.W. is the size of the feature map, is the global pooling information, global information Obtained by formula (6), formula (6) is as follows: ； in, represents normalization, represents a nonlinear activation function, represents point-wise convolution, Represents the first point-by-point convolution layer, and the convolution kernel size of the first point-by-point convolution layer is , Represents the second point-by-point convolution layer, and the convolution kernel size of the second point-by-point convolution layer is , represents the channel reduction rate; Using nonlinear activation function Sigmoid The polarity features of local face image information extraction and global face image information extraction are fused to obtain a feature map. The feature fusion is obtained by formula (7), which is as follows: ； in, represents a nonlinear activation function, Represents an element-wise addition operation.

6. The method for detecting fake faces based on frequency-attention feature fusion according to claim 5, characterized in that: The calculation process of the spatial domain and frequency domain feature fusion module includes the following steps: A convolutional neural network is used to extract a spatial domain feature map and a frequency domain feature map, and the spatial domain feature map and the frequency domain feature map are added together to obtain a primary feature map; The first-level feature map is input into the channel attention module to obtain the second-level feature map containing the image spatial domain, frequency domain, global and local information; The secondary feature map is residually connected with the spatial domain feature map and the frequency domain feature map and outputted to obtain the final fusion feature. The final fusion feature is obtained by formula (8) and (9). Formula (8) is as follows: ； The formula (9) is as follows: ； in, 、 Represent the characteristics of the spatial domain feature map and the frequency domain feature map respectively, 、 They represent the features of the spatial domain feature map and the frequency domain feature map after the spatial domain and frequency domain fusion modules respectively, and M represents the calculation process of the channel attention module.

7. The method for detecting fake faces based on frequency-attention feature fusion according to claim 6, characterized in that: The attention mechanism includes a variant attention module, and the calculation process of the variant attention module includes the following steps: The output results of the spatial domain and frequency domain feature fusion modules are summed and input into the channel attention module to obtain its output feature map; Perform attention calculation on the output results of the spatial domain and frequency domain feature fusion modules to obtain the corresponding attention maps of global and local artifact connections; The attention maps of the global and local artifacts are summed and input into the channel attention module to obtain the final attention weight matrix; Performing a first attention analysis calculation and a second attention analysis calculation on the final attention weight matrix, wherein the first attention analysis calculation multiplies the final attention weight matrix with the spatial feature map to obtain a first attention analysis eigenvalue, and the second attention analysis calculation performs an inverted attention operation on the final attention weight matrix, and then multiplies it with the frequency domain feature map to obtain a second attention analysis eigenvalue; The first attention analysis eigenvalue and the second attention analysis eigenvalue are added together to calculate a feature representation vector; The feature representation vector is obtained by formula (10) and formula (11), and the formula (10) is as follows: ； The formula (11) is as follows: ； Among them, A is the feature obtained by the first attention calculation, and Z is the final feature representation output by the encoding network. Represents an element-wise multiplication operation.

8. The method for detecting fake faces based on frequency-attention feature fusion according to claim 1, characterized in that: The cross entropy loss function is expressed by formula (12), which is as follows: ； in, for The output probability of the function, For the During the training process, we define 1 as a fake face and 0 as a real face. is the cross entropy loss function value; The cosine similarity loss function is expressed by formula (13), which is as follows: ； in, Indicates calculating the cosine distance between two features. is the cosine similarity loss function value; The face detection model It is obtained by formula (14), which is as follows: ； Where m represents the weight balancing parameter.

9. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 8 when executed by a processor.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

Images