Generalizable deep fake image detection method and system based on noise perception

Abstract

The present invention belongs to the field of computer vision technology, and provides a generalizable deep fake image detection method and system based on noise perception. The method comprises the following steps: obtaining a video containing a face identity to be detected; preprocessing the video data to be detected to obtain a facial image; inputting the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; the deep fake image detection model mines the noise information in the image as fake evidence through an adaptive noise perception attention model; and uses the features as fake evidence to determine whether the facial image has been deeply faked. The present invention has lower testing costs, higher efficiency, better generalization, and can cope with a wide range of deep fake image detection scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a generalizable deep fake image detection method and system based on noise perception. Background Art

[0002] Deepfake is content generated by artificial intelligence, but it is difficult for humans to distinguish whether it is real or fake. The term Deepfake is a combination of "deep learning" and "fake", and mainly refers to content generated by artificial neural networks. The most common form of Deepfake is the generation and manipulation of human imagery. This technology has creative and productive applications. Deepfake brings entertainment to people while also posing a huge security threat. Since the technical threshold of deep learning is relatively low and many open source technologies can be directly used, some criminals can easily forge fake videos of specific people without permission and use them maliciously.

[0003] Therefore, how to reduce this risk has attracted widespread attention from researchers, and the topic of Deepfake detection has gained traction in academia, and the technology has developed rapidly in the past few years. In recent years, many Deepfake detection methods have been proposed, which detect forgeries from multiple angles such as spatial artifacts, frequency domain, temporal inconsistency or multimodal conflict, and have achieved certain success on specific datasets. Although many scholars have made considerable efforts in the field of deep fake face detection, the practicality of related detection technologies is greatly restricted due to the lack of high-quality datasets, inconsistent evaluation mechanisms, and serious mismatches between experimental environments and real scene data. Most models use deep convolutional networks (e.g., ResNet, Xception, EfficientNet) as the backbone to extract high-level semantic features of suspicious images.

[0004] These features are then input into a binary classification network to achieve the identification of real or fake features. Although these detection methods perform well when the forgery method is known (e.g., evaluation in a dataset), these detection algorithms have limited generalization when encountering unseen forgeries (e.g., evaluation across datasets). Therefore, how to improve the generalization performance of detection algorithms is the focus and difficulty of current research. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention develops a generalizable deep fake image detection method and system based on noise perception.

[0006] The technical solution of the present invention to solve the technical problem is as follows: On the one hand, the present invention provides a generalizable deep fake image detection method based on noise perception, which comprises the following steps:

[0007] S110. Obtaining a face image to be detected;

[0008] S120. Input the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; the deep fake image detection model mines the noise information in the image as fake identification evidence through an adaptive noise-aware attention model;

[0009] The specific process is as follows:

[0010] The noise enhancement module is used to enable the model to capture information related to the intrinsic noise of the image. The non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions. The multi-scale fusion module is used to fuse feature information of different scales.

[0011] The deep forgery detection model uses a local noise data enhancement flow to generate more diverse training data through a model generalization mechanism based on local noise data enhancement, enabling the model to mine more robust, forgery method-independent, and generalizable image forgery evidence;

[0012] S130. Use the features as forgery evidence to determine whether the facial image has been deeply forged, and finally return the identification result.

[0013] Based on the above-mentioned method of training a deep fake image detection model, the following steps are specifically included:

[0014] S1201. Obtaining a face image to be identified from a video data set containing a portrait; annotating the original video image with information on facial key points through a pre-trained face detection model to obtain the corresponding face image and facial key point position information;

[0015] S1202. Mining noise information in the image as forgery identification evidence through an adaptive noise-aware attention model;

[0016] The noise enhancement module is used to enhance the model's ability to perceive noise information in the input image; the noise enhancement module is embedded in the backbone network so that the entire backbone network has noise perception ability and obtains image features after noise enhancement at multiple scales; the non-local consistency module is used to obtain inconsistency information between local regions based on the image features after noise enhancement; the multi-scale fusion module is used to obtain fused image features based on the fusion of deep inconsistency information and shallow noise enhancement features; the above non-local inconsistency information and image features at multiple scales are used to obtain the final identification result through global average pooling, splicing, and fully connected classifiers;

[0017] S1203. Through a model generalization mechanism based on local noise data enhancement, a local noise data enhancement flow is used to generate more diverse training data according to the input image, so that the model can mine more robust, forgery method-independent, and generalizable image forgery evidence;

[0018] S1204. Using a loss function to perform joint training constraints on the noise-aware image features and non-local inconsistency information;

[0019] S1205. Obtain a trained deep fake image detection model.

[0020] Based on the above-mentioned method for training a deep fake detection image detection model, a method for pre-acquiring facial images and facial key point position information includes the following steps:

[0021] Obtain an image containing a portrait to be processed from a video dataset containing a portrait;

[0022] The pre-trained face detection model is used to annotate the facial key point position information of the original image containing the portrait, and the corresponding facial key point information is obtained;

[0023] Obtaining a face image: Using the face key point information to locate the local position of the face area in the original image; Correcting the faces of different images to the same perspective and transforming them to the same size by performing an affine transformation; Using the mask video of the forged data also adopts the above scheme to obtain a forged area mask image, with the original area part set to 0 and the forged area part set to 1;

[0024] Obtaining facial key point location information: The original image containing the portrait is annotated with facial key point location information through the pre-trained face detection model. The location information contains 68 key points in total, and the location of the key points in the image is represented by a 68*2 matrix; the facial key point location information is multiplied by the affine transformation matrix to obtain the key point location information after the face image is aligned, which is also represented by a 68*2 matrix.

[0025] Based on the above-mentioned method for training a deep fake image detection model, S1203 specifically includes the following steps: the noise distribution in the forged area is highly inconsistent with the noise distribution in the original image area, but the noise distribution between the original image areas is highly consistent; the detection method includes an adaptive noise-aware attention model and a model generalization mechanism based on local noise data enhancement; the adaptive noise-aware attention model includes a noise enhancement module, a non-local consistency module, a multi-scale fusion module, and a loss function for supervising model training optimization.

[0026] Based on the above-mentioned deep fake image detection model training method, the noise enhancement of the image noise-level fake detection evidence mining method using the adaptive noise-aware attention model includes the following process:

[0027] The input image is represented as The current work mainly focuses on Input deep convolutional neural networks such as Resnet, Xception, and Efficientnet to extract feature maps for the final classifier. The features extracted from the backbone network are defined as ,in is the feature extractor. Since the traditional convolutional neural network contains multiple stages of different scales, the formula of the feature extraction process is rewritten as ,in It is The backbone layers with different depths can extract feature information of different scales. It is divided into four stages, namely high, medium, low and final. , , and As these noise enhancement modules have the same structure, the superscript To represent noise enhancement modules of different scales;

[0028] The noise of a digital image is defined as: in It is an ideal image of the real world without any noise. It is the only noise generated during image acquisition, transmission and other processes. This unique noise distribution can be used as key evidence for image tracing and has been widely used in image forensics, image tracing and other fields. Inspired by this, we attempt to use the noise knowledge of digital images for deep fake image detection.

[0029] The noise enhancement module is embedded into the backbone layer to mine and enhance the noise information in the image and features. The specific method is as follows:

[0030] a. Represents input features The noise information of It is a denoising algorithm that spatially subtracts the input features from the denoising features;

[0031] b. Use the sigmoid function to reduce the feature noise Map to ;

[0032] c. Pass the characteristic noise through In the convolution layer, the noise information on different channels is integrated and the intensity of the noise information is adjusted;

[0033] d. Input feature map through residual connection Add to the enhanced noise map to obtain the noise enhanced feature map, and use Representation, and used as input for subsequent modules;

[0034] e. The whole process is defined as:

[0035] ,

[0036] ,

[0037] in, represents the sigmoid function, The kernel size is The convolutional layer.

[0038] Based on the above-mentioned deep fake image detection model training method, the non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions. The specific process is as follows:

[0039] The feature map is firstly The size is divided into Blocks, with To express, express The scale Feature information of each block;

[0040] Apply block embedding to map feature blocks into the same space ,in It is the output dimension of Embedding, and is expressed as follows:

[0041] It is an Embedding operator. The kernel size is and the bar is The convolutional layer, is the Embedding vector, It is used to represent the positional embedding of different blocks. It is a learnable parameter that adaptively adjusts the ratio of block embedding and position embedding;

[0042] Predicted nonlocal consistency matrix It can be calculated by the following formula:

[0043]

[0044] in, It is Blocks and The prediction consistency between blocks, the sigmoid function is used to limit the output to ,like The closer it is to 0, the Blocks and The lower the consistency of the blocks, the lower the The closer it is to 1, the Blocks and The higher the consistency of the blocks.

[0045] Based on the above-mentioned method of training the deep fake image detection model, the process of multi-scale information interaction fusion through the adaptive noise-aware attention model is as follows:

[0046] Will Adjust to A three-dimensional tensor of size , Represented as nonlocal consistency, using The lightweight attention block composed of convolutional layer, batch normalization layer and nonlinear activation layer ReLU In The information of the channel is compressed into Channel, using consistency mapping express;

[0047] Use bilinear interpolation to compress Upsample to feature map Same size;

[0048] The information is integrated using element multiplication operations, and finally the fusion features are obtained.

[0049] Final Features and the consistency matrix from the three scales Connected as the final feature Used for the final deep fake detector to discriminate.

[0050] Based on the above method of training the deep fake image detection model, the total loss function is defined as:

[0051] , is the total loss function of the above model, , Respectively represent the above classification loss and consistency loss, the two are , Carry out weight control;

[0052] Obtaining the consistency map of ground truth , which is expressed as:

[0053] ,in Indicates Blocks and The ground truth consistency of blocks, for each If it is close to 1, it means that the consistency of the two blocks is high, otherwise it is close to 0. The consistency loss is calculated by the following formula:

[0054] ,in Represents different scales, Represents different blocks, Represents the binary cross entropy loss, which is used to supervise the non-local consistency of the model Consistency diagram with ground truth The distance between.

[0055] Based on the above-mentioned deep fake image detection model training method, the process of constructing a deep fake image detection model generalization mechanism based on local noise data enhancement is as follows:

[0056] Given a real face image , take its 68 facial key points, and based on facial geometry, use the facial key points to divide the facial image into 9 key areas: 'left edge', 'right edge', 'upper edge', 'left eyebrow', 'right eyebrow', 'nose', 'left eye', 'right eye' and 'mouth'; generate a mask , by calculating a number of randomly selected different regions and constructing the convex hull ; Based on the mask, the real image is mixed with the noise-enhanced image to obtain ,in represents a randomly generated convex hull mask, represents the original image, Represents the mixed image, that is, the image enhanced with local noise data.

[0057] On the other hand, an embodiment of the present invention provides a generalizable deep fake image detection system based on noise perception, comprising:

[0058] A data acquisition unit, which acquires the video image data of the portrait to be detected, wherein the detection image is obtained from different data sources to be detected through a detection interface provided by the system;

[0059] A feature extraction unit, inputting the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image;

[0060] The deep fake image detection model uses an adaptive noise-aware attention model to mine noise information in the image as evidence for forgery detection. Specifically, it uses a noise enhancement module to capture information related to the intrinsic noise of the image, uses a non-local consistency module to analyze the similarity of homologous regions and the inconsistency of heterologous regions, and uses a multi-scale fusion module to fuse feature information of different scales. The deep fake image detection model uses a model generalization mechanism based on local noise data enhancement and uses a local noise data enhancement flow to generate more diverse training data, so that the model can mine more robust, forgery method-independent, and generalizable image forgery evidence.

[0061] The result recognition unit uses the features as forgery evidence to determine whether the facial image has been deeply forged, and returns the detection result through the above-mentioned detection interface.

[0062] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0063] 1) The noise-aware generalizable deep fake image detection method of the present invention is jointly trained through an adaptive noise-aware attention model and a model generalization mechanism based on local noise data enhancement, fully utilizing image noise information as key evidence for image authentication, mining the potential consistency correlation characteristics between local areas of the image and other areas, fully integrating and utilizing information of features of different scales, and building a highly robust, highly discriminative, highly generalizable, end-to-end deep learning network through a local noise data enhancement mechanism, thereby constructing an intelligent and generalizable deep fake image detection model.

[0064] 2) The noise-aware generalizable deepfake image detection method of the present invention is systematically and comprehensively evaluated on seven existing deepfake image detection datasets, including FaceForensics++, CelebDF-v1, CelebDF-v2, DeepFakeDetection, deepfake detection Challenge, UADFV, and FaceShifter. The experimental results show that the features extracted by the above model have strong discrimination and generalization capabilities, and the above model outperforms the SOTA method in all datasets, cross-datasets, and cross-operation evaluations. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0066] Figure 1 Schematic diagram of a process of a generalizable deep fake image detection method based on noise perception according to Embodiment 1 of the present invention;

[0067] Figure 2 The model architecture of the deep fake image detection model based on adaptive noise-aware attention in the noise-aware generalizable deep fake image detection method according to Embodiment 1 of the present invention;

[0068] Figure 3 This is a diagram of the non-local consistency module architecture of a noise-aware generalizable deep fake image detection method according to Embodiment 1 of the present invention;

[0069] Figure 4 This is a multi-scale fusion module architecture diagram of a noise-aware generalizable deep fake image detection method according to Embodiment 1 of the present invention;

[0070] Figure 5 Schematic diagram of a model generalization mechanism based on local noise data enhancement of a noise-aware generalizable deep fake image detection method according to Embodiment 1 of the present invention;

[0071] Figure 6 4 is a logical structure diagram of a noise-aware generalizable deep fake image detection system according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Without departing from the principles of the present invention, changes, modifications, replacements and deformations made to the embodiments of the present invention are all within the scope of protection of the present invention.

[0073] Example 1

[0074] The present invention provides a generalizable deep fake image detection method based on noise perception to overcome the technical problems existing in the prior art.

[0075] Specifically, as an example, Figure 1 The following is a flow chart of a generalizable deep fake image detection method based on noise perception provided by an embodiment of the present invention. Figure 1As shown, the present invention provides a generalizable deep fake image detection method based on noise perception, which can be performed by a device, and the device can be implemented by software and / or hardware. The generalizable deep fake image detection method based on noise perception includes steps S110 to S130, which are as follows:

[0076] S110. Obtaining a face image to be detected;

[0077] S120. Input the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; the deep fake image detection model mines the noise information in the image as fake identification evidence through an adaptive noise-aware attention model;

[0078] The specific process is as follows:

[0079] The noise enhancement module is used to enable the model to capture information related to the intrinsic noise of the image. The non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions. The multi-scale fusion module is used to fuse feature information of different scales.

[0080] The deep forgery detection model uses a local noise data enhancement flow to generate more diverse training data through a model generalization mechanism based on local noise data enhancement, enabling the model to mine more robust, forgery method-independent, and generalizable image forgery evidence;

[0081] S130. Use the features as forgery evidence to determine whether the facial image has been deeply forged, and finally return the identification result.

[0082] Figure 2 : is a principle framework diagram of an adaptive noise-aware attention model in a noise-aware generalizable deep fake image detection method according to an embodiment of the present invention; Figure 2 As shown, the present invention provides a deep fake image detection model, including a backbone network, a noise enhancement module, a non-local consistency module and a multi-scale fusion module.

[0083] The following are the specific operation process and principles of each step.

[0084] The input image is represented as The current work mainly focuses on Input deep convolutional neural networks such as Resnet, Xception, and Efficientnet to extract feature maps for the final classifier;

[0085] The features extracted from the backbone network are defined as ,in is the feature extractor. Since the traditional convolutional neural network contains multiple stages of different scales, the formula of the feature extraction process is rewritten as ,in It is The backbone layers with different depths can extract feature information of different scales. It is divided into four stages, namely high, medium, low and final. , , and As these noise enhancement modules have the same structure, the superscript To represent noise enhancement modules of different scales;

[0086] The noise of a digital image is defined as: in It is an ideal image of the real world without any noise. It is the only noise generated during image acquisition, transmission and other processes. This unique noise distribution can be used as key evidence for image tracing and has been widely used in image forensics, image tracing and other fields. Inspired by this, we attempt to use the noise knowledge of digital images for deep fake image detection.

[0087] The noise enhancement module is embedded into the backbone layer to mine and enhance the noise information in the image and features. The specific method is as follows:

[0088] a. Represents input features The noise information of It is a denoising algorithm that spatially subtracts the input features from the denoising features;

[0089] b. Use the sigmoid function to reduce the feature noise Map to ;

[0090] c. Pass the characteristic noise through In the convolution layer, the noise information on different channels is integrated and the intensity of the noise information is adjusted;

[0091] d. Input feature map through residual connection Add to the enhanced noise map to obtain the noise enhanced feature map, and use Representation, and used as input for subsequent modules;

[0092] e. The whole process is defined as:

[0093] ,

[0094] ,

[0095] in, represents the sigmoid function, The kernel size is The design of the noise enhancement module starts from the definition of noise. Compared with the general convolutional neural network, it can improve the noise perception ability and has better model interpretability.

[0096] like Figure 3 As shown, the adaptive noise-aware attention model performs a non-local consistency module for local region homology or heterology as follows. The non-local consistency module consists of an attention module and a block pairwise consistency calculator. Through the attention module, more attention is paid to the noise information in different channels and spaces, and the non-local consistency module is gradually guided to reduce the attention to the information that is not conducive to identifying authenticity (such as semantics, GAN fingerprints). The consistency calculator is mainly implemented by calculating the dot product similarity of each piece; in this module, an attention module (CBAM) is first adopted. It is a lightweight but effective attention module for feedforward convolutional neural networks, and it is a plug-and-play lightweight module that can be easily integrated into various networks. The attention map is sequentially inferred along two independent dimensions (channel and space), and then the attention map is multiplied by the input feature map for adaptive feature refinement. Together with the non-local consistency module training objective, the use of CBAM can make the model pay more attention to more generalized features that help distinguish between true and false. Next, the feature map obtained by the attention module is fed to the non-local consistency operator to compute the non-local consistency that can represent the difference in noise distribution between the forged and original regions.

[0097] Specifically, they are as follows:

[0098] The feature map is firstly The size is divided into Blocks, with To express, express The scale Feature information of each block;

[0099] Apply block embedding to map feature blocks into the same space ,in It is the output dimension of Embedding, and is expressed as follows:

[0100] It is an Embedding operator. The kernel size is and the bar is The convolutional layer, is the Embedding vector, It is used to represent the positional embedding of different blocks. It is a learnable parameter that adaptively adjusts the ratio of block embedding and position embedding;

[0101] Predicted nonlocal consistency matrix It can be calculated by the following formula:

[0102] ,

[0103] in It is Blocks and The prediction consistency between blocks, the sigmoid function is used to limit the output to ,like The closer it is to 0, the Blocks and The lower the consistency of the blocks, the lower the The closer it is to 1, the Blocks and The higher the consistency of the blocks.

[0104] like Figure 4 As shown, the multi-scale information fusion module of the adaptive noise-aware attention model for multi-scale information interactive fusion is specifically as follows:

[0105] The feature map output by NE is defined as The non-local consistency of the output of the non-local inconsistency module is ,in and Represent the small scale and large scale respectively, Adjust to A three-dimensional tensor of size , Represented as nonlocal consistency, using The lightweight attention block composed of convolutional layer, batch normalization layer and nonlinear activation layer ReLU In The information of the channel is compressed into Channel, using consistency mapping express;

[0106] Use bilinear interpolation to compress Upsample to feature map Same size;

[0107] The information is integrated using element multiplication operations, and finally the fusion features are obtained.

[0108] In order to effectively utilize noise information, non-local consistency information and better distinguish whether the image is real or fake, the final feature and the consistency matrix from the three scales Connected as the final feature Used for the final deep fake detector to discriminate.

[0109] The deep fake image detection task is usually regarded as a binary classification problem, so we use the cross entropy loss to calculate the classification loss that represents the image classification result. To further improve the generalization ability, the consistency loss is used as an auxiliary task to supervise the training in a fine-grained mode. The consistency loss makes the model pay more attention to features with generalization ability such as noise distribution, learn the consistency between original regions and the inconsistency between original and forged regions. Therefore, the total loss function is defined as:

[0110] ,

[0111] in, is the total loss function of the above model, , Respectively represent the above classification loss and consistency loss, the two are , Perform weight control.

[0112] For the consistency loss, we can calculate the predicted non-local consistency (extracted by the non-local consistency module) and the forgery mask (obtained by comparing the real image and the forged image), and define the forgery mask as , where the mask is a number that contains only 0 and 1. We divide the mask into Blocks, Indicates The pixels of the block are averaged. To indicate the The forgery score of each block is used to obtain the consistency map of the ground truth. , which is expressed as:

[0113] ,

[0114] in Indicates Blocks and The ground truth consistency of blocks, for each If it is close to 1, it means that the consistency of the two blocks is high, otherwise it is close to 0. The consistency loss is calculated by the following formula:

[0115] ,

[0116] in Represents different scales, Represents different blocks, Represents the binary cross entropy loss, which is used to supervise the non-local consistency of the model Consistency diagram with ground truth The distance between.

[0117] Noise-aware generalizable deepfake image detection methods, such as Figure 5 The generalization mechanism of the deep fake image detection model based on local noise data enhancement is shown as follows:

[0118] Given a real face image , we take its 68 facial key points and use them to divide the facial image into 9 key areas based on facial geometry: 'left edge', 'right edge', 'upper edge', 'left eyebrow', 'right eyebrow', 'nose', 'left eye', 'right eye' and 'mouth'. Next, we generate a mask , by calculating a number of randomly selected different regions and constructing the convex hull Finally, based on the mask, we mix the real image with the noise-enhanced image (after deformation, noise, blur, compression and other data enhancement methods) to obtain The generated images are put into the above model for joint training, and finally a deep fake image detection model with better generalization can be obtained.

[0119] Using the trained deep fake image detection model, the features are used as forgery evidence to determine whether the facial image has been deeply faked, and finally the identification result is returned.

[0120] The noise-aware generalizable deepfake image detection method of the present invention is systematically and comprehensively evaluated on seven existing deepfake image detection datasets, including FaceForensics++, CelebDF-v1, CelebDF-v2, DeepFakeDetection, deepfake detection Challenge, UADFV, and FaceShifter. The experimental results show that the features extracted by the above model have strong discrimination and generalization capabilities, and the above model outperforms the SOTA method in all datasets, cross-datasets, and cross-operation evaluations.

[0121] Example 2

[0122] Corresponding to the above-mentioned generalizable deep fake image detection method based on noise perception, the present invention also provides a generalizable deep fake image detection system based on noise perception. Figure 4 The functional modules of a noise-aware generalizable deep fake image detection system according to an embodiment of the present invention are shown.

[0123] according to Figure 6 As shown, the present invention provides a generalizable deep fake image detection system 400 based on noise perception that can be installed in an electronic device. The system may include a data acquisition unit 410, a feature extraction unit 420, and a result recognition unit 430 according to its functional implementation. These units may also be called modules, which refer to a series of computer program segments that can be executed by a processor of an electronic device and perform specific functions, and are stored in the memory of the electronic device.

[0124] In this embodiment, the functions of each module / unit are as follows:

[0125] The data acquisition unit 410 acquires the video image data of the portrait to be detected, wherein the detection image is obtained from different data sources to be detected through the detection interface provided by the system;

[0126] The feature extraction unit 420 inputs the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; the deep fake image detection model mines the noise information in the image as fake identification evidence through an adaptive noise-aware attention model. Specifically, the noise enhancement module is used to enable the model to capture information related to the intrinsic noise of the image, the non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions, and the multi-scale fusion module is used to fuse feature information of different scales; the deep fake image detection model uses a local noise data enhancement flow to generate more diverse training data through a model generalization mechanism based on local noise data enhancement, so that the model can mine more robust, fake method-independent, and generalizable image fake evidence;

[0127] The result recognition unit 430 uses the feature as forgery evidence to determine whether the facial image has been deeply forged, and returns the detection result through the above-mentioned detection interface.

[0128] More specific implementation methods of the above-mentioned generalizable deep fake image detection system 400 based on noise perception provided by the present invention can all be described with reference to the above-mentioned embodiment of the generalizable deep fake image detection method based on noise perception, and will not be listed one by one here.

[0129] In the above-mentioned generalizable deep fake image detection system based on noise perception provided by the present invention, the deep fake image detection model mines the noise information in the image as the evidence for forgery through the adaptive noise perception attention model. Specifically, the noise enhancement module is used to enable the model to capture information related to the intrinsic noise of the image, the non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions, and the multi-scale fusion module is used to fuse feature information of different scales; the deep fake image detection model uses a model generalization mechanism based on local noise data enhancement and uses a local noise data enhancement flow to generate more diverse training data, so that the model can mine more robust, forgery method-independent, and generalizable image forgery evidence; the features are used as forgery evidence to determine whether the facial image has been deeply forged. The generalizable deep fake image detection method based on noise perception of the present invention systematically and comprehensively evaluates the proposed method on 7 existing deep fake image detection datasets, including FaceForensics++, CelebDF-v1, CelebDF-v2, DeepFakeDetection, deepfake detection Challenge, UADFV and FaceShifter. Experimental results show that the features extracted by the above model have strong discrimination and generalization capabilities, and the above model outperforms the SOTA method in all datasets, cross-datasets and cross-operation evaluations.

[0130] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A generalizable deep fake image detection method based on noise perception, characterized in that: The following steps are involved: S110. Obtaining a face image to be detected; S120. Input the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; the deep fake image detection model mines the noise information in the image as authentication evidence through an adaptive noise-aware attention model; The specific process is as follows: The noise enhancement module is used to enable the model to capture information related to the intrinsic noise of the image. The non-local consistency module is used to analyze the similarity of homologous regions and the inconsistency of heterologous regions. The multi-scale fusion module is used to fuse feature information of different scales. The deep fake image detection model uses a local noise data enhancement flow to generate more diverse training data through a model generalization mechanism based on local noise data enhancement, enabling the model to mine more robust, forgery method-independent, and generalizable image forgery evidence; The noise distribution in the forged area is highly inconsistent with the noise distribution in the original image area, but the noise distribution between the original image areas is highly consistent. The detection method includes an adaptive noise-aware attention model and a model generalization mechanism based on local noise data enhancement. The adaptive noise-aware attention model includes a noise enhancement module, a non-local consistency module, a multi-scale fusion module, and a loss function for supervising model training optimization; The noise enhancement of the method for mining forgery evidence at the image noise level using the adaptive noise-aware attention model includes the following steps: The input image is represented as , to extract feature maps for the final classifier; The features extracted from the backbone network are defined as ,in is the feature extractor. Since the traditional convolutional neural network contains multiple stages of different scales, the formula of the feature extraction process is rewritten as ,in It is The backbone layers with different depths can extract feature information of different scales. It is divided into four stages, namely high, medium, low and final. , , and As these noise enhancement modules have the same structure, the superscript To represent noise enhancement modules of different scales; The noise of a digital image is defined as: in represents the position of the pixel, It is an ideal image of the real world without any noise. It is the only noise generated during image acquisition, transmission and other processes; The noise enhancement module is embedded into the backbone layer to mine and enhance the noise information in the image and features. The specific method is as follows: a. Represents input features The noise information of It is a denoising algorithm that spatially subtracts the input features from the denoising features; b. Use the sigmoid function to reduce the feature noise Map to ; c. Pass the characteristic noise through In the convolution layer, the noise information on different channels is integrated and the intensity of the noise information is adjusted; d. Input feature map through residual connection Add to the enhanced noise map to obtain the noise enhanced feature map, and use Representation, and used as input for subsequent modules; e. The whole process is defined as: ， ， in, represents the sigmoid function, The kernel size is The convolutional layer; The specific process of using the non-local consistency module to analyze the similarity of homologous regions and the inconsistency of heterologous regions is as follows: The feature map is firstly The size is divided into Blocks, with To express, express The scale Feature information of each block; Apply block embedding to map feature blocks into the same space ,in It is the output dimension of Embedding, and is expressed as follows: ， in, It is an Embedding operator. The kernel size is and the bar is The convolutional layer, is the Embedding vector, It is used to represent the positional embedding of different blocks. It is a learnable parameter that adaptively adjusts the ratio of block embedding and position embedding; Predicted nonlocal consistency matrix It can be calculated by the following formula: ， in, It is Block and The prediction consistency between blocks, the sigmoid function is used to limit the output to ,like The closer it is to 0, the Block and The lower the consistency of the blocks, the lower the The closer it is to 1, the Block and The higher the consistency of the blocks; The process of multi-scale information interaction fusion through the adaptive noise-aware attention model is as follows: Will Adjust to A three-dimensional tensor of size , Represented as nonlocal consistency, using The lightweight attention block composed of convolutional layer, batch normalization layer and nonlinear activation layer ReLU In The information of the channel is compressed into Channel, using consistency mapping express; Use bilinear interpolation to compress Upsample to feature map Same size; The information is integrated using element multiplication operations, and finally the fusion features are obtained. Final Features and the consistency matrix from the three scales Connected as the final feature Used for the final deep fake detector’s discrimination; The total loss function is defined as: , is the total loss function of the above model, , Respectively represent the above classification loss and consistency loss, the two are expressed by the loss coefficient , Perform weight control; Obtaining the consistency map of ground truth , which is expressed as: ,in Indicates Block and The ground truth consistency of blocks, for each If it is close to 1, it means that the consistency of the two blocks is high, otherwise it is close to 0. The consistency loss is calculated by the following formula: ， in Represents different scales, Indicates different blocks, Represents the binary cross entropy loss, which is used to supervise the non-local consistency of the model Consistency diagram with ground truth The distance between S130. Use the features as forgery evidence to determine whether the facial image has been deeply forged, and finally return the identification result.

2. The noise-aware generalizable deep fake image detection method according to claim 1, characterized in that: The method for training a deep fake image detection model specifically includes the following steps: S1201. Obtaining a face image to be identified from a video data set containing a portrait; annotating the original video image with information on facial key points through a pre-trained face detection model to obtain the corresponding face image and facial key point position information; S1202. Mining noise information in the image as forgery identification evidence through an adaptive noise-aware attention model; The noise enhancement module is used to enhance the model's ability to perceive noise information in the input image. The noise enhancement module is embedded in the backbone network, so that the entire backbone network has noise perception ability and obtains image features after noise enhancement at multiple scales. The non-local consistency module is used to obtain inconsistency information between local regions based on the image features of noise enhancement. The multi-scale fusion module is used to obtain fused image features based on the fusion of deep inconsistency information and shallow noise enhancement features. The above non-local inconsistency information and image features at multiple scales are combined through global average pooling, splicing, and fully connected classifiers to obtain the final identification results. S1203. Through a model generalization mechanism based on local noise data enhancement, a local noise data enhancement flow is used to generate more diverse training data according to the input image, so that the model can mine more robust, forgery method-independent, and generalizable image forgery evidence; S1204. Using the loss function to perform joint training constraints on noise-aware image features and non-local inconsistency information; S1205. Obtain a trained deep fake image detection model.

3. The noise-aware generalizable deep fake image detection method according to claim 2, characterized in that: A method for pre-acquiring facial images and facial key point position information comprises the following steps: Obtain an image containing a portrait to be processed from a video dataset containing a portrait; The pre-trained face detection model is used to annotate the facial key point position information of the original image containing the portrait, and the corresponding facial key point information is obtained; Obtaining a face image: Using the face key point information to locate the local position of the face area in the original image; Correcting the faces of different images to the same perspective and transforming them to the same size by performing an affine transformation; Obtaining a forged area mask image based on the mask video of the forged data, with the original area part set to 0 and the forged area part set to 1; Obtaining facial key point location information: The original image containing the portrait is annotated with facial key point location information through the pre-trained face detection model. The location information contains 68 key points in total, and the location of the key points in the image is represented by a 68*2 matrix; the facial key point location information is multiplied by the affine transformation matrix to obtain the key point location information after the face image is aligned, which is also represented by a 68*2 matrix.

4. The noise-aware generalizable deep fake image detection method according to claim 1, characterized in that: The process of constructing the generalization mechanism of the deep fake image detection model based on local noise data enhancement is as follows: Given a real face image , take its 68 facial key points, and based on facial geometry, use the facial key points to divide the facial image into 9 key areas: 'left edge', 'right edge', 'upper edge', 'left eyebrow', 'right eyebrow', 'nose', 'left eye', 'right eye' and 'mouth'; generate a mask , by calculating a number of randomly selected different regions and constructing the convex hull ; Based on the mask, the real image is mixed with the noise-enhanced image. ,in represents a randomly generated convex hull mask, represents the original image, Represents the mixed image, that is, the image enhanced with local noise data.

5. A noise-aware generalizable deep fake image detection system, which executes the noise-aware generalizable deep fake image detection method according to any one of claims 1 to 4, characterized in that: include: A data acquisition unit, which acquires the video image data of the portrait to be detected, wherein the detection image is obtained from different data sources to be detected through a detection interface provided by the system; A feature extraction unit, inputting the facial image into a pre-trained deep fake image detection model to extract deep fake evidence features of the image; The deep fake image detection model uses an adaptive noise-aware attention model to mine the noise information in the image as evidence for authentication. Specifically, it uses the noise enhancement module to capture information related to the intrinsic noise of the image, uses the non-local consistency module to analyze the similarity of homologous regions and the inconsistency of heterologous regions, and uses the multi-scale fusion module to fuse feature information of different scales. The deep fake image detection model uses a local noise data enhancement flow to generate more diverse training data through a model generalization mechanism based on local noise data enhancement, enabling the model to mine more robust, forgery method-independent, and generalizable image forgery evidence; The result recognition unit uses the features as forgery evidence to determine whether the facial image has been deeply forged, and returns the detection result through the above-mentioned detection interface.

Images