A deep fake face detection method based on self-information enhanced contrastive learning
Through the self-information enhanced contrast learning method, the overfitting problem of deep forged face detection model in new scene processing is solved, the generalization and robustness of the model are improved, and the better forged feature recognition ability is achieved.
Patent Information
- Application Number
- CN202411346926.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-26
Smart Images

Figure CN118887722B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of deep fake detection, and mainly to a deep fake face detection method based on self-information enhanced contrast learning. Background Art
[0002] The great progress of artificial intelligence technology has introduced open source algorithms such as DeepFaceLab, DFaker, AvatarMe, and application software such as ZAO and FakeApp that can easily manipulate multimedia content and spread false information through social media platforms. The emergence of these fake contents has exacerbated the public's concerns about information security.
[0003] DeepFake is a technology that creates realistic artificial synthetic content based on deep learning models.
[0004] With the continuous advancement of deep fake technology, it is difficult for the naked eye to tell whether the faces in images and videos have been tampered with. Therefore, there is an urgent need for a technology that can effectively detect fake faces in multimedia content. Deep fake face detection technology has come into being. It helps protect society from the influence of false information, enhance the security of face recognition and identity authentication systems, and prevent fake face images from being used to deceive access control systems. In addition, the research on deep fake face detection technology will also help promote the establishment of regulatory and ethical frameworks, and clarify the boundaries of the legal use and abuse of technology.
[0005] Deep fake detection technology can be divided into two categories according to the different model processing objects: one is deep fake detection at the image level; the other is deep fake detection at the video level. Among them, image-level detection mainly detects whether the image has been deeply faked by identifying the face part in a single frame image and comparing the face information in the original image; video-level detection methods mainly target continuous video frames and analyze and process each frame in the video. This type of method not only considers the detection accuracy of a single frame image, but also needs to take into account the temporal continuity and inter-frame correlation between frames in the video.
[0006] However, although many deep fake face detection methods can achieve quite impressive results in in-domain evaluation, as deep fake technology is constantly developing and evolving, new generation methods and tools are constantly emerging. Each method has its own specific traces and artifacts. During the model training process, the model will focus too much on specific local face forgery patterns in the training data generated by specific methods, and has not really learned the general forgery features that are more general and discriminative for generating faces, resulting in overfitting problems in the model and poor performance when dealing with new scenes. All signs indicate that it is necessary to further explore and mine the common features of deep fake content and develop a more universal and efficient detection framework based on these features. Summary of the invention
[0007] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a deep fake face detection method based on self-information enhanced contrastive learning. The method uses self-information measurement to evaluate whether any local area of the face image is significantly different or abnormal from its adjacent areas, and further counts the amount of local self-information carried by these areas; at the same time, a self-information enhanced learning module and a contrastive learning paradigm are designed to improve the model's ability to recognize common forgery features and common forgery patterns generated by different forgery methods, so as to improve the generalization and robustness of the model.
[0008] According to one aspect of the present invention, a deep fake face detection method based on self-information enhanced contrast learning is proposed, comprising:
[0009] S1. Preprocessing the real video and the forged video to obtain an image containing only the face area, generating a sample image, performing data enhancement on the sample image using an image enhancement module, and randomly generating two images with different data enhancement methods, namely a first view and a second view;
[0010] S2, using Xception as the backbone network to extract features from the first view and the second view, and learning shallow features through a multi-scale feature enhancement module to obtain multi-scale enhanced features, and calculating the multi-scale enhanced features through self-information metric to obtain a self-information feature map;
[0011] S3, inserting self-information enhancement learning modules into the shallow and middle layers of the backbone network respectively, the self-information feature graph outputs self-information enhancement features through the self-information enhancement learning module, and encoding the self-information enhancement features using the deep network of the backbone network, and outputting the encoded features and , the coding feature and Splice to obtain the final encoding features ;
[0012] S4, using cascaded Projector and Predictor to encode the features and Mapped to a high-dimensional space, the high-dimensional feature vectors of the first view are obtained respectively and , and the high-dimensional feature vector of the second view and , respectively, close and , and distance, forming a mutual prediction mechanism;
[0013] S5. Use two different branches to perform model optimization training. The first branch obtains a prediction result through a pooling layer and a fully connected layer, uses the prediction result to judge the authenticity, and optimizes it with a cross entropy loss function; the second branch introduces a consistency loss function to maximize the similarity of feature representations between the first view and the second view; the cross entropy loss function and the consistency loss function are used for joint training to obtain a trained deep fake face detection model.
[0014] Furthermore, the preprocessing obtains an image containing only a face area, and generating a sample image specifically includes: detecting faces using an MTCNN face detector for real videos and forged videos, cropping the face area and preprocessing it to a size of 299×299×3.
[0015] Obtaining images containing only the face area can reduce the interference of model prediction and improve the accuracy and efficiency of subsequent processing.
[0016] Preprocessing the image to a size of 299×299 can meet the format requirements for input data during model training; and ensure that the image has three channels (RGB) to be consistent with the input requirements of the model.
[0017] Furthermore, the image enhancement module performs data enhancement on the sample image, and the data enhancement methods include: random patch, random flip, random cropping, Gaussian blur, random occlusion, color change, and frequency domain enhancement. Each input image will randomly select two of the image transformation methods.
[0018] Furthermore, the step of performing image processing using the frequency domain enhancement includes:
[0019] S1.1. Convert the first view and the second view into the frequency domain using discrete cosine transform to obtain the corresponding spectrum diagram;
[0020] S1.2, using a learnable filter to adaptively acquire a frequency domain image containing low-frequency, medium-frequency, and high-frequency components;
[0021] S1.3, splicing the obtained spectrograms of different spectral components along the channel dimension;
[0022] S1.4. Use inverse discrete cosine transform to convert the spliced spectrum back to the spatial domain, and obtain the first view processed in the frequency domain and the second view with enhanced frequency information.
[0023] Spectral enhancement can capture the global information of the image, enhance the robustness and generalization ability of the model, reduce overfitting, and improve the model's resistance to noise by converting the image from the spatial domain to the frequency domain. Through frequency domain transformation, the model can learn more types of image features and perform better in various tasks.
[0024] Furthermore, the discrete cosine change calculation formula is as follows:
[0025]
[0026]
[0027] Among them, F(u,v) represents the coefficient of the inverse discrete cosine transform, that is, represents the original image, N represents the number of points in the original image, c(v) represents the compensation coefficient, c(u) represents the normalization coefficient, i represents the horizontal coordinate of the image pixel point, j represents the vertical coordinate of the image pixel point, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image; the inverse discrete cosine transform calculation formula is as follows:
[0028]
[0029] Among them, f(i,j) represents the image after being converted back to the spatial domain, that is, the original image.
[0030] Discrete cosine transform (DCT) and inverse discrete cosine transform (IDCT) are of great significance in image processing and compression. DCT transforms images from the spatial domain to the frequency domain, which helps to remove redundancy, concentrate energy, and filter in the frequency domain. IDCT transforms the image represented in the frequency domain back to the spatial domain for image decompression and reconstruction after frequency domain processing. Through these transformations, the efficiency and quality of image processing can be effectively improved.
[0031] Furthermore, the specific formula of the frequency domain enhanced data enhancement method is expressed as follows:
[0032]
[0033] Among them, Concat means concatenating feature maps along the channel dimension, D means discrete cosine transform, represents the inverse discrete cosine transform, represents filtering, Indicates low frequency, medium frequency, and high frequency filtering. Indicates the first view or the second view.
[0034] Furthermore, the feature extraction performed by Xception as a backbone network specifically includes: shallow feature extraction including three Separable Conv layers + ReLU layers, middle feature extraction including twelve Block modules of the Xception network, and deep feature extraction including two Separable Conv layers + ReLU layers and one Pooling layer.
[0035] As an efficient deep convolutional neural network, the Xception network significantly reduces the amount of computation and parameters while maintaining powerful feature extraction capabilities through deep separable convolution technology.
[0036] Furthermore, the multi-scale feature enhancement module is composed of cascaded 1×1conv, 3×3conv and 5×5conv, and the formula of the output multi-scale enhanced feature is expressed as follows:
[0037]
[0038] in, represents multi-scale enhanced features, m represents the specific scale or level of the enhanced features, and Concat represents concatenating feature maps along the channel dimension. , , They represent 1×1conv, 3×3conv, and 5×5conv respectively, where conv represents convolution operations at different levels. Feature map representing the output of a shallow network of Xception.
[0039] Furthermore, the self-information metric calculation step is: generating local anomalies generated by approximating the human face through the joint distribution of the current pixel and its neighborhood of the Gaussian kernel function, and searching the local receptive field area from the spatial dimension and the channel dimension for self-information metric, wherein the formulas of the self-information metric of the spatial dimension and the self-information metric of the channel dimension are respectively expressed as:
[0040]
[0041] in, Represents multi-scale enhanced features The pixel coordinates (i, j) of the nth channel, and They represent the receptive field area of the pixel (i, j) spatial dimension and channel dimension, which includes all channel pixels of the current pixel and its (3×3) neighborhood; p, q and a, b represent and The pixel index in ; is the Gaussian kernel function, h represents the bandwidth;
[0042] The overall formula of the self-information metric outputting the self-information feature map is expressed as follows: in, represents the equilibrium parameter.
[0043] Furthermore, the self-information enhanced learning module includes a self-information measurement process and a self-information enhanced attention calculation. The self-information enhanced learning module receives a dual-stream input, one branch is an output feature map of a layer on the network, and the other branch is a feature map after self-information measurement. The specific steps of the self-information measurement process include:
[0044] The self-information metric is output as the self-information feature map I( ) and the multi-scale enhanced feature map are added element by element to obtain the feature map ;
[0045] For the feature map The self-information metric calculation is performed again, and the second self-information metric feature map I ( );
[0046] Starting from two orthogonal dimensions, the second self-information metric feature map I( ) calculating the self-information enhancement feature of the forged face, wherein the self-information enhancement feature includes a spatial dimension and a channel dimension, wherein the attention of the channel dimension amplifies the channel dependency across dimensions through a multi-layer perceptron;
[0047] Learn the global context information of each channel, use the Sigmoid function to perform normalization, and further integrate it with the feature map Perform element-wise multiplication to obtain the channel attention weight based on the self-information metric , the specific formula is as follows:
[0048]
[0049] Among them, Sigmoid represents the normalization function, MLP represents the multi-layer perceptron, I( ) represents the second self-information metric feature map, represents element-wise multiplication, Represents the feature map I( ) performs element-by-element addition operation with the multi-scale enhanced feature map to obtain a feature map;
[0050] Two cascaded convolutional layers are used to fuse spatial information, and the Sigmoid function is used for normalization, and the feature map is combined with the Perform element-wise multiplication to obtain the spatial attention weight based on the self-information metric , the specific formula can be expressed as:
[0051]
[0052] Among them, Conv represents 7X7 convolution operation;
[0053] The comprehensive formula of self-information enhancement features is expressed as follows:
[0054]
[0055] in, Represents the self-information enhancement feature.
[0056] The self-information enhanced learning module will be inserted into the middle and deep structures of the backbone network to guide the network to perform feature learning, and the self-information enhanced learning module simultaneously outputs two types of feature maps: one is the self-information feature map output after self-information measurement, and the self-information feature map will be used as the input of the next self-information enhanced learning module; the other is the self-information enhanced feature map, and the self-information enhanced feature map will be input into the next layer of the network for feature calculation.
[0057] Furthermore, the coding features of the first view and the second view, and the formula of the final coding features are specifically expressed as follows:
[0058]
[0059]
[0060]
[0061] in, represents the encoding network, represents convolution concatenation, represents the self-information encoding feature of the first view, Represents the self-information encoding feature of the second view.
[0062] Furthermore, the high-dimensional feature vector of the first view and and the high-dimensional feature vector of the second view and Respectively expressed as:
[0063]
[0064]
[0065] Furthermore, the specific formula of the cross entropy loss function is as follows:
[0066]
[0067] in, is the output probability of the sigmoid function, is the label value of the i-th input, j represents 0 or 1. During the training process, 1 is defined as a fake face, 0 is defined as a real face, and N represents the number of points in the original image;
[0068] Introduce mutual prediction consistency loss to close and , and The distance in the high-dimensional representation space is used to form a mutual prediction mechanism. The mutual prediction mechanism makes predictions from different data perspectives to help the model discover the diversity of potential forgery patterns and facial shape structure types generated in the process of forged face generation. The specific formula of the consistency loss function is as follows:
[0069] in, Representation calculation and The cosine distance between two features, Representation calculation and The cosine distance between two features;
[0070] The specific formula of the loss function of the deep fake face detection model is as follows:
[0071]
[0072] in, represents the weight balancing parameter, represents the cross entropy loss function, represents the consistency loss function.
[0073] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.
[0074] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0075] 1) The present invention provides a deep fake face detection method based on self-information enhanced contrastive learning, and proposes a deep fake face detection method based on the combination of self-information attention and contrastive learning. This method can effectively alleviate the overfitting problem of the model by improving the model's ability to recognize common features and common fake patterns generated by different fake methods;
[0076] 2) The present invention designs a multi-scale feature enhancement module and a self-information enhancement learning module, wherein the multi-scale feature enhancement module can improve the local and global feature modeling capabilities of the network; the self-information enhancement learning module uses self-information metrics to quantify the self-information differences in local areas of the generated face, highlighting high-information textures and shape features, and uses an attention mechanism to reduce the network's attention to specific forgery traces and strengthen attention to general shape features and common forgery patterns, thereby alleviating the problem of cross-domain performance degradation when the model deals with different data sets;
[0077] 3) The present invention designs a new contrastive learning paradigm, which uses two cascaded Projectors and Predictors to map features to a high-dimensional space. By using mutual prediction, it can make predictions from different data perspectives, helping the model discover the potential forgery patterns and diversity of facial shape and structure types generated in the process of forged face generation, thereby gaining a more comprehensive understanding of different forgery types. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.
[0079] Figure 1 A schematic flow chart of a method for detecting deep fake face videos based on self-information enhanced contrast learning according to an embodiment of the present invention is shown;
[0080] Figure 2 A structural diagram of a deep fake face video detection model based on self-information enhanced contrast learning according to a specific embodiment of the present invention is shown;
[0081] Figure 3 A schematic diagram of a self-information measurement process flow according to a specific embodiment of the present invention is shown;
[0082] Figure 4 A principle block diagram of a self-information enhanced learning module according to a specific embodiment of the present invention is shown;
[0083] Figure 5 A detailed flowchart of a deep fake face video detection method based on self-information enhanced contrast learning according to a specific embodiment of the present invention is shown. DETAILED DESCRIPTION
[0084] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0085] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0086] Figure 1 FIG. 1 is a flow chart of a method for detecting deep fake face videos based on self-information enhanced contrast learning according to an embodiment of the present invention. Figure 1 As shown:
[0087] S1. Preprocessing the real video and the forged video to obtain an image containing only the face area, generating a sample image, performing data enhancement on the sample image using an image enhancement module, and randomly generating two images with different data enhancement methods, namely a first view and a second view;
[0088] The preprocessing obtains an image containing only a face area, and generating a sample image specifically includes: using an MTCNN face detector to detect faces in real videos and forged videos, cropping and preprocessing the face area to a size of 299×299×3, and dividing the face data into a training set and a test set;
[0089] In this embodiment, the forged face datasets used include: Face Forensics++ (FF++), Celeb-DF, and DFDC.
[0090] The image enhancement module performs data enhancement on the sample image. The data enhancement methods include: random patch, random flip, random cropping, Gaussian blur, random occlusion, color change, and frequency domain enhancement. Two of the image transformation methods are randomly selected for each input image.
[0091] The step of acquiring the frequency domain enhanced image comprises:
[0092] S1.1. Convert the first view and the second view into the frequency domain using discrete cosine transform to obtain the corresponding spectrum diagram;
[0093] S1.2, using a learnable filter to adaptively acquire a frequency domain image containing low-frequency, medium-frequency, and high-frequency components;
[0094] S1.3, splicing the obtained spectrograms of different spectral components along the channel dimension;
[0095] S1.4. Use inverse discrete cosine transform to convert the spliced spectrum back to the spatial domain, and obtain the first view processed in the frequency domain and the second view with enhanced frequency information.
[0096] The discrete cosine variation calculation formula is as follows:
[0097]
[0098]
[0099] Among them, F(u,v) represents the coefficient after inverse discrete cosine transform, N represents the number of points in the original image, c(v) represents the compensation coefficient, c(u) represents the normalization coefficient, i represents the horizontal coordinate of the image pixel, j represents the vertical coordinate of the image pixel, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image;
[0100] The inverse discrete cosine transform calculation formula is as follows:
[0101]
[0102] Among them, f(i,j) represents the image after being converted back to the spatial domain, that is, the original image.
[0103] The specific formula of the frequency domain enhanced image is as follows:
[0104]
[0105] Among them, Concat means concatenating feature maps along the channel dimension, D means discrete cosine transform, represents the inverse discrete cosine transform, represents filtering, Indicates low frequency, medium frequency, and high frequency filtering. Indicates the first view or the second view.
[0106] Further, such as Figure 2A structural diagram of a deep fake face video detection model based on self-information enhanced contrast learning according to a specific embodiment of the present invention is shown, and the specific steps include: inputting an initialized face image (input x), the face image is outputted through a data enhancement module to output a first view x1 and a second view x2, and then through network coding to output fout1 and fout2, and finally using cascaded projectors and predictors for mutual prediction.
[0107] S2, using Xception as the backbone network to extract features from the first view and the second view, and learning shallow features through a multi-scale feature enhancement module to obtain multi-scale enhanced features, and calculating the multi-scale enhanced features through self-information metric to obtain a self-information feature map;
[0108] The feature extraction performed by Xception as a backbone network specifically includes: shallow feature extraction including three Separable Conv layers + ReLU layers, middle feature extraction including twelve Block modules of the Xception network, and deep feature extraction including two Separable Conv layers + ReLU layers and one Pooling layer.
[0109] The multi-scale feature enhancement module consists of cascaded 1×1conv, 3×3conv and 5×5conv. The formula of the output multi-scale enhanced feature is expressed as follows:
[0110]
[0111] in, represents multi-scale enhanced features, m represents the specific scale or level of the enhanced features, and Concat represents concatenating feature maps along the channel dimension. , , They represent 1×1conv, 3×3conv, and 5×5conv respectively, where conv represents convolution operations at different levels. Feature map representing the output of a shallow network of Xception.
[0112] The self-information metric is calculated as Figure 3 As shown, the specific steps include: generating local anomalies generated by approximating human faces through the joint distribution of the current pixel and its neighborhood of the Gaussian kernel function, and searching the local receptive field area from the spatial dimension and the channel dimension for self-information measurement, wherein the formulas for the self-information measurement of the spatial dimension and the self-information measurement of the channel dimension are respectively expressed as:
[0113]
[0114]
[0115] in, Represents multi-scale enhanced features The pixel coordinates (i, j) of the nth channel, and They represent the receptive field area of the pixel (i, j) spatial dimension and channel dimension, which includes all channel pixels of the current pixel and its (3×3) neighborhood; p, q and a, b represent and The pixel index in ; is the Gaussian kernel function, h represents the bandwidth;
[0116] The overall formula of the self-information metric outputting the self-information feature map is expressed as follows:
[0117]
[0118] in, represents the equilibrium parameter.
[0119] S3, inserting self-information enhancement learning modules into the shallow and middle layers of the Xception backbone network respectively, the self-information feature map outputs self-information enhancement features through the self-information enhancement learning module, and encoding the self-information enhancement features using the deep network of the Xception backbone network, and outputting the encoded features and , the coding feature and Splice to obtain the final encoding features ;
[0120] The self-information enhanced learning module is as follows Figure 4 As shown, it includes a self-information measurement process and a self-information enhanced attention calculation. The self-information enhanced learning module receives a dual-stream input, one branch is an output feature map of a layer on the network, and the other branch is a feature map after self-information measurement. The specific steps of the self-information measurement process include:
[0121] The self-information metric is output as the self-information feature map I( ) and the multi-scale enhanced feature map are added element by element to obtain the feature map ;
[0122] For the feature map The self-information metric calculation is performed again, and the second self-information metric feature map I ( );
[0123] Starting from two orthogonal dimensions, the second self-information metric feature map I( ) calculating the self-information enhancement feature of the forged face, wherein the self-information enhancement feature includes a spatial dimension and a channel dimension, wherein the attention of the channel dimension amplifies the channel dependency across dimensions through a multi-layer perceptron;
[0124] Learn the global context information of each channel, use the Sigmoid function to perform normalization, and further integrate it with the feature map Perform element-wise multiplication to obtain the channel attention weight based on the self-information metric , the specific formula is as follows:
[0125]
[0126] Among them, Sigmoid represents the normalization function, MLP represents the multi-layer perceptron, I( ) represents the second self-information metric feature map, represents element-wise multiplication, Represents the feature map I( ) performs element-by-element addition operation with the multi-scale enhanced feature map to obtain a feature map;
[0127] Two cascaded convolutional layers are used to fuse spatial information, and the Sigmoid function is used for normalization, and the feature map is combined with the Perform element-wise multiplication to obtain the spatial attention weight based on the self-information metric , the specific formula can be expressed as:
[0128]
[0129] Among them, Conv represents 7X7 convolution operation;
[0130] The comprehensive formula of self-information enhancement features is expressed as follows:
[0131]
[0132] in, Represents the self-information enhancement feature.
[0133] The self-information enhanced learning module will be inserted into the middle and deep structures of the backbone network to guide the network to perform feature learning, and the self-information enhanced learning module simultaneously outputs two types of feature maps: one is the self-information feature map output after self-information measurement, which will serve as the input of the next self-information enhanced learning module; the other is the self-information enhanced feature map, which will be input into the next layer of the network for feature calculation.
[0134] The self-information enhanced learning module can calculate the attention weights based on the self-information measurement results, thereby guiding the network to reduce the correlation between the output of each layer and the low-information area in the input to reduce the attention to the specific face forgery patterns and susceptible local forgery features generated by different face forgery methods, while strengthening the correlation between the output of each layer of the network and the high-information area in the input to guide the network to pay attention to more general shape features and common face forgery patterns generated by different deep forgery methods.
[0135] S4, using cascaded Projector and Predictor to encode the features and Mapped to a high-dimensional space, the high-dimensional feature vectors of the first view are obtained respectively and , and the high-dimensional feature vector of the second view and , respectively, close and , and distance, forming a mutual prediction mechanism;
[0136] The encoding features of the first view and the second view, as well as the formula of the final encoding features are specifically expressed as follows:
[0137]
[0138]
[0139]
[0140] in, represents the encoding network, represents convolution concatenation, represents the self-information encoding feature of the first view, Represents the self-information encoding feature of the second view.
[0141] The high-dimensional feature vector of the first view and and the high-dimensional feature vector of the second view and Respectively expressed as:
[0142]
[0143]
[0144] Both Projector and Predictor are small neural networks. Projector is used to map input data to a lower-dimensional representation space. Predictor is used to predict the representation vector generated by Projector.
[0145] Through Projector, the original input data can be converted into a more compact representation, which usually has better generalization ability. The goal of Predictor is to predict the Projector output of another view to enhance the learning effect of the representation.
[0146] Projector can capture complex patterns and nonlinear relationships in the input data. The learning effect of representation can be further enhanced by letting Predictor learn to predict the output of Projector of another view. The output dimension of Predictor is the same as that of Projecto for contrastive loss calculation.
[0147] S5. Use two different branches to perform network optimization training. The first branch obtains a prediction result through a pooling layer and a fully connected layer, uses the prediction result to judge the authenticity, and optimizes it with a cross entropy loss function; the second branch introduces a consistency loss function to maximize the similarity of feature representations between the first view and the second view; the cross entropy loss function and the consistency loss function are used for joint training to obtain a trained deep fake face detection model.
[0148] The specific formula of the cross entropy loss function is as follows:
[0149]
[0150] in, is the output probability of the sigmoid function, is the label value of the i-th input, j represents 0 or 1. During the training process, 1 is defined as a fake face, 0 is defined as a real face, and N represents the number of points in the original image;
[0151] Introduce mutual prediction consistency loss to close and , and The distance in the high-dimensional representation space is used to form a mutual prediction mechanism. The mutual prediction mechanism makes predictions from different data perspectives to help the model discover the diversity of potential forgery patterns and facial shape structure types generated in the process of forged face generation. The specific formula of the consistency loss function is as follows:
[0152]
[0153] in, Representation calculation and The cosine distance between two features, Representation calculation and The cosine distance between two features;
[0154] The specific formula of the loss function of the deep fake face detection model is as follows:
[0155]
[0156] in, represents the weight balancing parameter, represents the cross entropy loss function, represents the consistency loss function.
[0157] Further, in some optional embodiments, such as Figure 5 The detailed process steps of the deep fake face video detection method based on self-information enhanced contrast learning are as follows:
[0158] The input image x is preprocessed into a preset size of 299×299×3, and then x is passed through the data enhancement module to obtain two different views of the image. ;
[0159] The data enhancement methods include random patch, random flip, random crop, Gaussian blur, random occlusion, color change, and frequency domain enhancement. Two of these image transformation methods are randomly selected for each input image. The frequency domain enhanced image is obtained by using discrete cosine transform to transform the visual image. Figure 1 With Vision Figure 2 The image is converted to the frequency domain to obtain its spectrum; three learnable filters are designed to adaptively obtain the frequency domain images containing low-frequency, medium-frequency and high-frequency components respectively; the spectrum of different spectral components is then spliced along the channel dimension; the spliced spectrum is converted back to the spatial domain using the inverse discrete cosine transform to obtain the frequency domain processed visual images. Figure 1 With Vision Figure 2 Image after frequency information enhancement.
[0160] The backbone network used is divided into three levels: low layer, middle layer and deep layer. Figure 1 With Vision Figure 2 First, input it into the lower layer of the backbone network to extract the visual Figure 1 With Vision Figure 2 The shallow characteristics of
[0161] The backbone network is the Xception network. Xception is divided into shallow feature extraction: including 3 (Separable Conv layers + ReLU layers); middle feature extraction: including 12 Block modules of the Xception network; deep feature extraction: including two (Separable Conv layers + ReLU layers) and one Pooling layer.
[0162] The obtained shallow features are input into the multi-scale feature enhancement module to obtain multi-scale enhanced features , the multi-scale feature enhancement module consists of cascaded 1×1conv, 3×3conv and 5×5conv.
[0163] Using self-information metric to enhance multi-scale features Perform the first self-information calculation to obtain the first self-information feature , self-information can approximate the local anomaly generated by the face through the joint distribution of the current pixel and its neighborhood of the Gaussian kernel function, and search the local 3×3 receptive field area from the spatial dimension and channel dimension to measure the self-information.
[0164] The first self-information feature With multi-scale features Input into the self-information enhancement learning module to obtain the self-information enhancement feature The second self-information feature , the self-information enhanced learning module receives dual-stream input, one branch is the output feature map of the upper layer of the network, and the other branch is the feature map after self-information measurement. And the self-information enhanced learning module will be inserted into the middle and deep structures of the backbone network to guide the network to learn features. The module outputs two types of feature maps at the same time: one is the self-information feature map output by self-information measurement, which will serve as the input of the next self-information enhanced learning module; the other is the self-information enhanced feature map, which will be input to the next layer of the network for feature calculation.
[0165] The obtained self-information enhancement feature Is input into the middle layer of the backbone network to obtain the visual Figure 1 With Vision Figure 2 The middle-level characteristics of
[0166] The second self-information feature The middle-level features are input into the second self-information enhancement learning module for further feature learning, and the self-information enhancement features after the second self-information enhancement learning module are output. ;
[0167] The feature calculation process of the second self-information enhanced learning module is consistent with that of the first self-information enhanced learning module.
[0168] The obtained self-information enhancement feature Input to the deep layer of the backbone network to obtain visual Figure 1 With Vision Figure 2 The final feature representation and , and and Concatenate along the channels to obtain the final feature representation of the input image x ;
[0169] For output A prediction result will be obtained through the pooling layer and the fully connected layer in turn to determine the authenticity, and optimized with the cross entropy loss function;
[0170] For output and , the cascaded Projector and Predictor are mapped to the high-dimensional space, and the visual Figure 1 The high-dimensional feature vector and And visual Figure 2 The high-dimensional feature vector and , the network will be closer and , and The distance between the two images f1 and f2 is used to form a mutual prediction mechanism, and the mutual prediction consistency loss function is introduced to maximize the similarity of feature representation between the two enhanced images f1 and f2 obtained from the same input x. The two loss functions are used for joint training to obtain a trained deep fake face detection model.
[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0172] The modules involved in the embodiments of the present application may be implemented by software or by hardware.
[0173] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A deep fake face detection method based on self-information enhanced contrast learning, characterized in that: The following steps are involved: S1. Preprocessing the real video and the forged video to obtain an image containing only the face area, generating a sample image, performing data enhancement on the sample image using an image enhancement module, and randomly generating two images with different data enhancement methods, namely a first view and a second view; S2, using Xception as the backbone network to extract features from the first view and the second view, and learning shallow features through a multi-scale feature enhancement module to obtain multi-scale enhanced features, and calculating the multi-scale enhanced features through self-information metric to obtain a self-information feature map; S3, inserting self-information enhancement learning modules into the shallow and middle layers of the backbone network respectively, the self-information feature graph outputs self-information enhancement features through the self-information enhancement learning module, and encoding the self-information enhancement features using the deep network of the backbone network, and outputting the encoded features f out1 and f out2 , the encoding feature f out1 and f out2 Concatenate to obtain the final encoding feature f out , wherein the self-information enhanced learning module includes a self-information measurement process and a self-information enhanced attention calculation, wherein the self-information enhanced learning module receives a dual-stream input, wherein one input branch is an output feature map of a layer on the network, and the other input branch is a feature map after self-information measurement, wherein the specific steps of the self-information measurement process include: The self-information metric outputs the self-information feature map I(f m ) and the multi-scale enhanced feature map are added element by element to obtain the first self-information metric feature map f′ m ; For the feature map f′ m The self-information metric calculation is performed again, and the second self-information metric feature map I(f l ); Starting from two orthogonal dimensions, the second self-information measurement feature map I(f l ) calculating the self-information enhancement feature of the forged face, wherein the self-information enhancement feature includes a spatial dimension and a channel dimension, wherein the attention of the channel dimension amplifies the channel dependency across dimensions through a multi-layer perceptron; Learn the global context information of each channel, use the Sigmoid function to perform normalization, and the normalized data is consistent with the feature map f′ m Perform element-by-element multiplication to obtain the channel attention weight C based on the self-information metric n , the specific formula is as follows: C n =Sigmoid(MLP(I(f l )))⊙f′ m Among them, Sigmoid represents the normalization function, MLP represents the multi-layer perceptron, I(f l ) represents the second self-information metric feature map, ⊙ represents element-by-element multiplication, f′ m Represents the first self-information metric feature map; Two cascaded convolutional layers are used to fuse spatial information, and the Sigmoid function is used for normalization and combined with the feature map f′ m Perform element-by-element multiplication to obtain the spatial attention weight S based on the self-information metric n , the specific formula can be expressed as: S n =Sigmoid(Conv(I(f l )))⊙f′ m Among them, Conv represents a 7×7 convolution operation; The comprehensive formula of self-information enhancement features is expressed as follows: f l =C n ⊙S n ⊙f′ m +f′ m Among them, f l represents the self-information enhancement feature; S4, using the cascaded Projector and Predictor to transform the encoded feature f out1 and f out2 Mapping to a high-dimensional space, respectively obtaining high-dimensional feature vectors z1 and p1 of the first view, and high-dimensional feature vectors z2 and p2 of the second view, respectively shortening the distances between z1 and z2, and between p1 and p2, to form a mutual prediction mechanism; S5. Use two different branches to perform model optimization training. The first branch obtains a prediction result through a pooling layer and a fully connected layer, uses the prediction result to judge the authenticity, and optimizes it with a cross entropy loss function. The second branch introduces a consistency loss function to maximize the similarity of feature representations between the first view and the second view. The cross entropy loss function and the consistency loss function are used for joint training to obtain a trained deep fake face detection model, wherein the cross entropy loss function is specifically expressed as follows: in, is the output probability of the sigmoid function, b i is the label value of the i-th input, j represents 0 or 1. During the training process, 1 is defined as a fake face, 0 is defined as a real face, and N represents the number of points in the original image; The mutual prediction consistency loss is introduced to narrow the distance between z1 and z2, p1 and p2 in the high-dimensional representation space, so as to form a mutual prediction mechanism. The mutual prediction mechanism predicts from different data perspectives, helping the model to discover the potential forgery patterns and diversity of facial shape structure types generated in the process of forged face generation. The specific formula of the consistency loss function is as follows: L cos =l cos (p1,z2) / 2+l cos (z1, p2) / 2 in, It means calculating the cosine distance between the two features p1 and z2. Indicates calculating the cosine distance between the two features z1 and p2; The specific formula of the loss function of the deep fake face detection model is as follows: L=L ce +βL cos Among them, β represents the weight balance parameter, L ce represents the cross entropy loss function, L cos represents the consistency loss function.
2. The deep fake face detection method according to claim 1, characterized in that: The preprocessing obtains an image containing only a face area, and generating a sample image specifically includes: using an MTCNN face detector to detect faces in real videos and forged videos, cropping the face area and preprocessing it to a size of 299×299×3, where 299×299 represents the image pixel size, and 3 represents the 3 channels of the image RGB.
3. The deep fake face detection method according to claim 1, characterized in that: The image enhancement module performs data enhancement on the sample image. The data enhancement methods include: random patch, random flip, random cropping, Gaussian blur, random occlusion, color change, and frequency domain enhancement. Two of the data enhancement methods are randomly selected for each input image.
4. The deep fake face detection method according to claim 3, characterized in that: The steps of performing image processing using the frequency domain enhancement include: S1.
1. Convert the first view and the second view into the frequency domain using discrete cosine transform to obtain the corresponding spectrum diagram; S1.2, using a learnable filter to adaptively acquire a frequency domain image containing low-frequency, medium-frequency, and high-frequency components; S1.3, splicing the obtained spectrograms of different spectral components along the channel dimension; S1.
4. Use inverse discrete cosine transform to convert the spliced spectrum back to the spatial domain, and obtain the first view processed in the frequency domain and the second view with enhanced frequency information.
5. The deep fake face detection method according to claim 4, characterized in that: The calculation formula of the discrete cosine change is as follows: The inverse discrete cosine transform calculation formula is as follows: Among them, f(i,j) represents the image after being converted back to the spatial domain, F(u,v) represents the coefficient of the inverse discrete cosine transform, N represents the number of points in the original image, c(v) represents the compensation coefficient, c(u) represents the normalization coefficient, i represents the horizontal coordinate of the image pixel, j represents the vertical coordinate of the image pixel, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image.
6. The deep fake face detection method according to claim 4, characterized in that: The specific formula of the data enhancement method of frequency domain enhancement is as follows: Among them, Concat represents concatenating feature maps along the channel dimension, D represents discrete cosine transform, and D -1 represents the inverse discrete cosine transform, f w represents filtering, Indicates low frequency, medium frequency, and high frequency filtering, x k , k∈{1, 2} represents the first view or the second view, and y represents the image data after frequency domain enhancement.
7. The deep fake face detection method according to claim 1, characterized in that: The feature extraction performed by Xception as a backbone network specifically includes: shallow feature extraction including three Separable Conv layers + ReLU layers, middle feature extraction including twelve Block modules of the Xception network, and deep feature extraction including two Separable Conv layers + ReLU layers and one Pooling layer.
8. The deep fake face detection method according to claim 1, characterized in that: The multi-scale feature enhancement module consists of cascaded 1×1conv, 3×3conv and 5×5conv. The formula of the output multi-scale enhanced feature is expressed as follows: f m =Concat(connv1(X layer1 ),conv2(X layer1 ),conv3(X layer1 )) Among them, f m represents multi-scale enhanced features, m represents the specific scale or level of the enhanced features, Concat represents concatenating feature maps along the channel dimension, conv1, conv2, and conv3 represent 1×1conv, 3×3conv, and 5×5conv, respectively, and conv represents convolution operations at different levels. laver1 Feature map representing the output of a shallow network of Xception.
9. The deep fake face detection method according to claim 1, characterized in that: The specific steps of calculating the self-information metric are: generating local anomalies generated by approximating a human face through the joint distribution of the current pixel and its neighborhood of the Gaussian kernel function, and searching the local receptive field area from the spatial dimension and the channel dimension for self-information metric, wherein the formulas of the self-information metric of the spatial dimension and the self-information metric of the channel dimension are respectively expressed as: in, Represents the multi-scale enhanced feature f m The pixel coordinates (i, j) of the nth channel, R S and R C represents the receptive field area of the pixel (i, j) spatial dimension and channel dimension, which includes all channel pixels of the current pixel and its (3×3) neighborhood; P, q and a, b represent R S and R C The pixel index in ; is the Gaussian kernel function, h represents the bandwidth, represents the self-information measure of the spatial dimension, Represents the self-information measure of the channel dimension; The overall formula of the self-information metric outputting the self-information feature map is expressed as follows: Here, γ represents the equilibrium parameter.
10. The deep fake face detection method according to claim 1, characterized in that: The self-information enhanced learning module will be inserted into the middle and deep structures of the backbone network to guide the network to perform feature learning, and the self-information enhanced learning module simultaneously outputs two types of feature maps: one is the self-information feature map output after self-information measurement, and the self-information feature map will be used as the input of the next self-information enhanced learning module; the other is the self-information enhanced feature map, and the self-information enhanced feature map will be input into the next layer of the network for feature calculation.
11. The deep fake face detection method according to claim 1, characterized in that: The encoding feature f of the first view out1 and the encoded feature f of the second view out2 , and the final encoding feature f out The formula is specifically expressed as follows: f out1 =Encoder(f1) f out2 =Encoder(f2) f out =Concat(f out1 ,f out2 ) Among them, Encoder represents the encoding network, Concat represents convolution concatenation, f1 represents the self-information encoding feature of the first view, and f2 represents the self-information encoding feature of the second view.
12. The deep fake face detection method according to claim 1, characterized in that: The high-dimensional feature vectors z1 and p1 of the first view and the high-dimensional feature vectors z2 and p2 of the second view are respectively expressed as: z1=Projector(f out1 ),z2=Projector(f out2 ) p1=predictor(f out1 ),p2=predictor(f out2 )。 13. A computer program product, characterized in that A computer program is stored thereon, which implements the method according to any one of claims 1 to 12 when executed by a processor.
14. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 12.
