Generable deepfake image detection method and system based on double data enhancement

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a generalizable deep fake image detection method and system based on double data enhancement. Specifically, the extracted facial image is preprocessed and labeled; a landmark diffusion generation model is constructed to generate new fake samples; real samples, fake samples and new fake samples are respectively input into three teacher models to obtain corresponding identity semantic features; the identity semantic features are enhanced; the enhanced features are transferred to a student model based on a KL divergence distillation architecture; the facial image to be detected is input into the student model to extract fake features, which are then input into a discriminant decision module to output a judgment result. The present application has lower testing cost, higher efficiency, better generalizability and stronger robustness, and is suitable for efficient and low-cost detection of a wide range of deep fake images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a generable deepfake image detection method and system based on double data enhancement. BACKGROUND

[0002] The rapid development of deepfake technology makes face identity tampering present an unprecedented visual authenticity, and its application in the fields of film and television production, digital entertainment, etc. is attracting much attention. Building a reliable and universal deepfake detection system has become a core demand to protect digital content security.

[0003] Although the current mainstream detection methods can achieve very high detection accuracy within a specific dataset, they generally face significant performance degradation in cross-dataset testing. The reason is that the current mainstream methods (such as artifact region positioning, noise pattern analysis, and frequency domain feature extraction) are mostly optimized for specific fake traces (such as GAN-generated fingerprints). Although these methods have been successful in specific scenarios, they are essentially still over-fitted to known fake patterns and have not yet achieved the ability to generalize to unseen situations in reality, making it difficult to cope with cross-domain changes brought about by emerging fake technologies (such as diffusion-based Deepfake) and complex real-world scenarios.

[0004] In order to break through the generalization bottleneck, many methods use data augmentation to expand the decision boundary of fake feature representation. In order to overcome the overfitting problem of specific datasets, some studies show that data augmentation is an effective tool that can significantly improve the generalization ability. Typical methods include GAN artifact simulation based on regional blur, multi-source image mixing to generate composite fake samples, and adversarial training to enhance model robustness. By generating new mixed images from two original images, and by mixing fake images in a single image with target images to generate fake images, these methods provide the model with a more general and more difficult face fake detection task. Their common point is to generate fake samples containing face fake traces that are more difficult to identify, thereby encouraging the model to learn a more general and more robust face fake detection representation. However, the existing enhancement strategies have fundamental flaws: the visual quality and realism of the synthesized samples are insufficient, and they rely heavily on traditional generation methods, which not only lack diversity in synthetic data, but also are difficult to effectively generalize to unknown fake types, which is contrary to their original intention.

[0005] In order to solve the above problems, it is urgent to develop an enhanced strategy that can systematically expand the distribution of forged space, generate high-visual-fidelity forged samples, and has good migration ability. Moreover, the rise of generative models, especially diffusion models, provides a new way to construct high-quality and controllable forged samples. Diffusion models have high image generation ability while preserving image details and semantic consistency, and are particularly suitable for injecting local forged features while maintaining the integrity of the identity. Combining the diffusion generation mechanism with the spatial constraint strategy guided by the forged mask, it is expected to generate more challenging forged samples that are closer to the real distribution, thereby promoting the deep forged detection model to learn more robust, generalizable and method-independent feature representations and breaking through the limitations of existing methods in cross-domain generalization performance.

[0006] Therefore, the present application proposes a generable deep forged image detection method and system based on double data enhancement to solve the above problems. SUMMARY

[0007] The present application aims to improve the deep forged detection ability, make the test cost lower, the efficiency higher, the generalization better and the robustness stronger.

[0008] On the one hand, the technical scheme for solving the technical problem of the present application is a generable deep forged image detection method based on double data enhancement, and the specific steps are as follows:

[0009] S1, extracting facial images from videos containing human face identities and pre-processing, then marking the pre-processed facial images as real or fake, as real samples and forged samples respectively;

[0010] S2, constructing a landmark diffusion generation model, inputting the real samples into the landmark diffusion generation module to synthesize new forged samples with high fidelity and difficult to identify, and performing data enhancement on the new forged samples;

[0011] S3, constructing three teacher models, inputting the real samples, forged samples and new forged samples into the three teacher models respectively to obtain real identity semantic features, forged identity semantic features and new forged identity semantic features, and guiding the teacher models to optimize and train through class loss and inter-class contrast loss;

[0012] S4, using the constructed deep weight enhancement module to perform data enhancement on the real identity semantic features to obtain real enhanced features, and using the constructed class enhancement module to perform systematic data enhancement on the forged identity semantic features and the new forged identity semantic features to obtain forged enhanced features;

[0013] S5. Based on the KL divergence distillation architecture, the real enhancement features and fake enhancement features are transferred to the student model to obtain a student fake image detector learned from the teacher model.

[0014] S6. Input the face image to be detected into the student fake image detector to extract fake features, and then input the extracted fake features into the discrimination decision module to output the judgment result.

[0015] S1 is as follows:

[0016] S1.1, Obtain facial image:

[0017] Retrieve videos from an existing database containing real and fake faces, and extract facial images from videos containing facial identities based on custom frame count, height, width, and frame rate.

[0018] A pre-trained facial landmark model from the machine learning library dlib is loaded as a face detection model. The face detection model is used to detect facial landmarks in the extracted facial images, locate the facial regions, crop them, and then determine the position information of the facial landmarks. Each facial region includes 68 landmarks, and the position information of the facial landmarks in each facial image is represented by a 68×2 two-dimensional matrix. The facial images are aligned according to the position information of the facial landmarks to eliminate pose and angle differences and obtain standardized facial images.

[0019] S1.2 Facial image preprocessing:

[0020] The standardized facial image is subjected to controllable image perturbation operations, including random horizontal flipping, random angle rotation and blur perturbation operations. The perturbation image is then adjusted to a uniform size and normalized according to the set mean and standard deviation to obtain the preprocessed facial image.

[0021] S1.3, Authenticity Marking:

[0022] The preprocessed facial images are labeled to distinguish between real and fake facial images, and the two types of facial images are used as real samples and fake samples, respectively.

[0023] S2 is as follows:

[0024] S2.1 Based on the facial key point location information of real samples, the face is divided into multiple semantically specific regions, and all semantic regions are represented as follows: The semantic regions include the left eye, right eye, nose, mouth, and combined regions. Indicates the first A semantic region, Indicates the number of semantic regions;

[0025] At the same time, a corresponding text description is constructed based on the face region mask, which serves as a text description prompt;

[0026] S2.2 Construct a landmark diffusion generation module. Input the face region mask and text description prompts mentioned above into the landmark diffusion generation module to generate a new type of fake sample. The landmark diffusion generation module adopts a conditional input image generation model as the image generation backbone and a multi-scale U-Net network as the generator backbone in the conditional diffusion model. In the image generation stage, a cross-attention mechanism is introduced to fuse semantic regions and text description prompts.

[0027] In the process of generating novel fake samples, a denoising optimizer is used to obtain samples from a standard normal distribution. latent variables Sampling is performed, represented as The denoised latent variables were obtained through multiple iterations using a multi-scale U-Net network. The latent variables after denoising Input to image decoder Image decoder It consists of a series of convolutional layers, activation functions, and normalization layers, and is processed by an image decoder. Denoising the latent variables Gradually restore the image to the same resolution as the input facial image to generate a new type of forged sample that is high-resolution and difficult to detect;

[0028] S2.3. Expand the new forged samples to achieve data augmentation.

[0029] S3 is as follows:

[0030] S3.1 Constructing a teacher model:

[0031] The teacher models include the first teacher model, the second teacher model, and the third teacher model;

[0032] The ArcFace face recognition model, pre-trained on the MS1MV2 dataset, was used as the first teacher model.

[0033] The EfficientNet-B4 deep convolutional neural network model pre-trained on the ImageNet dataset was used as the second and third teacher models.

[0034] S3.2 Input the image samples into the teacher model:

[0035] Real samples are input into the first teacher model to extract real identity semantic features. ;

[0036] The forged samples and novel forged samples were input into the second teacher model and the third teacher model, respectively, to extract semantic features of the forged identities. and new semantic features of forged identities ;

[0037] S3.3 Constructing a loss function to optimize and train the teacher model:

[0038] (1) Constructing the inter-class comparison loss function :

[0039] A cosine similarity-based comparison mechanism is used to integrate the semantic features of real identities. Semantic features of forged identities and new semantic features of forged identities The similarity metric is calculated using the following formula:

[0040] ,

[0041] in, Indicates the number of fake samples. Indicates the number of real samples. A function that measures the similarity between features;

[0042] (2) Constructing the category loss function :

[0043] Based on semantic features of real identity Semantic feature set of forged identity Any semantic feature of a forged identity and new semantic features of forged identities Construct the category loss function The calculation formula is as follows:

[0044] ,

[0045] in, Indicates the first Semantic features of a forged identity This represents the number of forged identity semantic features, which is equal to the number of forged samples. This represents a binary classification label, where 0 indicates that the input is a real sample, and 1 indicates that the input is a fake sample. This represents the classification loss function used for binary classification tasks.

[0046] The specific operations in the deep weight enhancement module are as follows:

[0047] A deep weight enhancement module is constructed, which consists of three parts: channel partitioning, convolution calculation, and channel concatenation.

[0048] The channelizing part is composed of a set of feature vector slicing operations in channel dimensions;

[0049] The convolution calculation part includes three convolution layers, a 3x3 convolution layer , an 11x1 convolution layer , and a 1x11 convolution layer ;

[0050] The channel splicing part is composed of a channel splicing operation;

[0051] The real identity semantic feature is input into the deep weight enhancement module, and the input real identity semantic feature is divided along the channel dimension into four sub-channel regions, namely the original channel feature , the spatial direction information enhancement feature , the vertical direction information enhancement feature , and the horizontal direction information enhancement feature ;

[0052] The spatial direction information enhancement feature is extracted using a depth separable convolution with a 3x3 convolution kernel ;

[0053] The spatial direction information enhancement feature is extracted using a longitudinal convolution with an 11x1 convolution kernel ;

[0054] The horizontal direction information enhancement feature is extracted using a transverse convolution with a 1x11 convolution kernel ;

[0055] The output results of the three convolution branches are spliced in the channel dimension with the original channel feature , and the dimension of the output feature is adjusted to the same feature map size as , to obtain the real enhanced feature with fused multi-dimensional spatial structure information .

[0056] The operation of the category enhancement module is as follows:

[0057] A category enhancement module is constructed, which includes three parts, namely a five-image transformation strategy pool, a random selector and feature splicing. The random selector randomly selects one image transformation strategy for each of the two input features, which is realized by the random function in the Python standard library, and then the enhanced feature vectors of the two input features are spliced along the spatial dimension by a channel splicing function.

[0058] The fake identity semantic feature and the new fake identity semantic feature are input into the category enhancement module, the input features are expanded and enhanced by image transformation strategies, a continuous fake feature variant space is constructed, and fake enhanced features are obtained.

[0059] Among them, the image transformation strategies are clustering transformation CT, extrapolation transformation ET, mixed transformation MT, scaling transformation ZT and noise transformation NT, and one image transformation strategy is randomly selected for expansion and enhancement.

[0060] (1) Clustering transformation CT: simulate different fake sources or subclasses by feature clustering disturbance;

[0061] First, the feature average of the fake identity semantic feature and the new fake identity semantic feature is calculated, and the calculation formula is

[0062] ,

[0063] Then, adjust and according to the feature average , and the adjustment formula is as follows:

[0064] ,

[0065] wherein is a randomly sampled proportion factor between 0 and 1, represents the hyperbolic tangent function;

[0066] (2) Extrapolation transformation ET: interpolate boundary variants samples at the edge of the feature space to strengthen decision boundary learning, the specific process is as follows:

[0067] ,

[0068] wherein represents the feature vector guiding the extrapolation process, , represents the fake identity semantic feature vector guiding the extrapolation process, ​This represents a novel semantic feature vector for forged identities that guides the extrapolation process. This represents the scaling factor for random sampling between 0 and 1;

[0069] (3) Hybrid Transform (MT): This method fuses multiple features of forged images to synthesize complex hybrid forged images. The specific process is as follows:

[0070] ,

[0071] in, , for hour, for , for hour, for ; Indicates from distributed The random number drawn from the sample is used to control the mixing ratio of the two input samples. The value is between 0 and 1;

[0072] (4) Scaling transformation ZT: by applying the affine transformation matrix Affine scaling of the image simulates the effect of resolution or size changes on forgery; affine transformation matrix. Affine transformation matrix combines rotation and scaling operations. The definition is as follows:

[0073] ,

[0074] in, Indicates the rotation angle. Indicates the scaling factor;

[0075] affine transformation matrix Application to features spatial coordinates This yields the enhanced features, and the specific process is as follows:

[0076] ,

[0077] in, Indicates transpose. Represents the x-axis, Represents the ordinate;

[0078] (5) Noise Transformation NT: Different types of random noise are injected into the forged image. The calculation process is as follows:

[0079] ,

[0080] wherein, is according to generated Gaussian noise, denotes the standard deviation of the noise, denotes a coefficient that controls the intensity of the noise addition, randomly sampled between 0 and 1;

[0081] wherein, denotes the enhanced feature, , denotes the adjusted fake identity semantic feature, denotes the new fake identity semantic feature, and refers to the feature or , .

[0082] S5 is specifically as follows:

[0083] S5.1, construct a student model, use the ImageNet pre-trained model EfficientB4 as the student model;

[0084] S5.2, construct a distillation architecture from three teacher models and one student model, align the output features of the student model with the output features of the teacher models by using the normalized exponential function , and , the alignment method is KL divergence alignment, the specific process is as follows:

[0085] ,

[0086] wherein, denotes the distillation loss, refers to the feature or , , refers to the real identity semantic feature , the fake identity semantic feature or the new fake identity semantic feature , when is the real enhanced feature output by the teacher model , corresponds to is the real identity semantic feature extracted by the student model , when is the fake enhanced feature output by the teacher model , corresponds to is the fake identity semantic feature extracted by the student model and the new fake identity semantic feature ; denotes the total amount of input samples, denotes the first feature or , denotes the first real identity semantic feature , fake identity semantic feature or new type of fake identity semantic feature ; denotes the divergence alignment operation; denotes the smoothness temperature parameter of the adjusted normalization exponential function ;

[0087] During the training process of the student model, the distillation loss is used as a loss function to back-propagate gradients and update model parameters, and the parameters of the student model are optimized by minimizing the distillation loss to obtain a student fake image detector learned from the teacher model.

[0088] S6 is as follows:

[0089] The discriminant decision module includes two convolutional activation units, a global average pooling layer, a flattening layer, a fully connected layer, and a Dropout inactivation layer. The fake features output by the student fake image detector are input into the discriminant decision module, which performs dimension compression and semantic mapping on the input features, and finally outputs a binary classification result of fake or real and the corresponding probability of fake or real.

[0090] On the other hand, the application also provides a generable deep fake image detection system based on double data enhancement, which includes modules for executing processing instructions of each step in the generable deep fake image detection method based on double data enhancement, including an image input module, a artifact feature extraction module, and a discriminant decision module.

[0091] The face image to be detected is input into the system through the image input module, then passes through the artifact feature extraction module to extract fake features, and then the fake features are input into the discriminant decision module to output a classification prediction result.

[0092] Among them, the artifact feature extraction module uses a pre-trained student model to extract features, and the discriminant decision module classifies and predicts according to the fake features.

[0093] The effects provided in the summary of the invention are only the effects of the embodiments, not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects:

[0094] (1) High-precision, strong generalization of deep forgery detection: Through the deep weight enhancement module and the category enhancement module for feature-level enhancement, the detection performance and robustness of the detection method are improved; at the same time, the advanced knowledge transfer from the teacher model to the student model is realized by using the KL divergence distillation loss to enhance the generalization ability of the student model, which further explores and expands the broader space of forged features, so that the model can learn the decision boundary covering various forgery patterns, and finally improve the detection performance of various unknown forgery types;

[0095] (2) High-fidelity, high-resolution deep forgery synthesis: By combining the diffusion model with the forged image data, the face organs (eyes, nose, mouth) are used as spatial constraints to ensure that the generated image retains the original identity features in the ungenerated area; the newly generated forged image has high visual fidelity and high resolution, which is almost indistinguishable from the real picture; the method uses facial region key points (such as eyes, nose, mouth, etc.) as structural conditions, and uses a conditional diffusion model to generate images with specific forged regions; the method fuses real images, mask images and text prompts through cross-attention, uses a decoder to restore high-resolution forged samples, and retains the edge features of the real identity, thereby generating deep forged images that are difficult to distinguish from real images in vision.

[0096] (3) Innovative data enhancement strategy: The synergistic enhancement strategy of data enhancement and data expansion effectively solves the limitations of existing methods in capturing sample-level diversity and feature-level complexity, and obtains better generalization performance compared with methods that only rely on enhancement or expansion.

[0097] Therefore, the method proposed in the present application can generate more realistic distribution and challenging forgery samples, which can promote the student model to learn more robust, generalizable and method-independent feature representations. BRIEF DESCRIPTION OF DRAWINGS

[0098] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, which together with the embodiments of the present application, serve to explain the present application, and do not constitute a limitation of the present application.

[0099] Figure 1 The flowchart of the method of the present application.

[0100] Figure 2 The noise robustness comparison chart of the method of the present application and the existing method. DETAILED DESCRIPTION

[0101] In order to clearly illustrate the technical features of the present scheme, the present application will be described in detail below with reference to the specific embodiments and the accompanying drawings.

[0102] Example 1

[0103] As Figure 1 shown, a generable deep fake image detection method based on double data enhancement, the specific steps are as follows:

[0104] S1, extract the face image from the video containing the face identity and pre-process it, then mark the pre-processed face image as real or fake, as real sample and fake sample respectively;

[0105] S2, construct a landmark diffusion generation model, input the real sample into the landmark diffusion generation module, synthesize new fake samples with high fidelity and difficult to identify, and perform data enhancement on the new fake samples;

[0106] S3, construct three teacher models, input the real sample, fake sample and new fake sample into the three teacher models respectively, get the real identity semantic feature, fake identity semantic feature and new fake identity semantic feature respectively, and guide the teacher model to optimize training through class loss and inter-class contrast loss;

[0107] S4, use the constructed deep weight enhancement module to perform data enhancement on the real identity semantic feature to obtain real enhanced features, and use the constructed class enhancement module to perform systematic data enhancement on the fake identity semantic feature and the new fake identity semantic feature to obtain fake enhanced features;

[0108] S5, based on the KL divergence distillation architecture, migrate the real enhanced features and fake enhanced features to the student model to obtain a student fake image detector learned from the teacher model;

[0109] S6, input the face image to be detected into the student fake image detector to extract the fake features, and then input the extracted fake features into the discriminant decision module to output the judgment result.

[0110] In the specific implementation, S1 is as follows:

[0111] S1.1, obtain the face image:

[0112] Obtain the video from the existing database containing real face and fake face, extract the face image from the video containing the face identity according to the self-defined frame number, height, width and frame rate;

[0113] Load the pre-trained face key point model in the machine learning tool library dlib ) As a face detection model, the face key point detection is performed on the extracted face image through the face detection model, the face region is located after cutting, and then the face key point position information is determined. The face region includes 68 key points. The face key point position information of each face image is represented by a 68x2 two-dimensional matrix. The face image is aligned according to the face key point position information, and the posture and angle difference is eliminated to obtain a standardized face image.

[0114] S1.2, face image preprocessing:

[0115] The standardized face image is subjected to controllable image disturbance operation, the image disturbance operation includes random horizontal flip, random angle rotation and blur disturbance operation, then the image after the disturbance operation is adjusted to a uniform size, and is subjected to normalization processing according to the set mean and standard deviation to obtain a preprocessed face image.

[0116] S1.3, true and false marking:

[0117] The preprocessed face image is marked to mark the real face image and the fake face image, and the two kinds of face images are respectively taken as real samples and fake samples.

[0118] In the specific implementation, S2 is specifically as follows:

[0119] S2.1, according to the face key point position information of the real sample, a multi-region face organ semantic coordinate graph is constructed, the face region is divided into a plurality of semantic regions according to a face region mask, and the semantic region is represented as The semantic region includes left eye, right eye, nose, mouth and combined region, represents the th semantic region, represents the number of semantic regions;

[0120] According to the face region mask, a corresponding text description is constructed as a text description prompt;

[0121] S2.2, a landmark diffusion generation module is constructed, the face region mask and the text description prompt are input into the landmark diffusion generation module, a new type of fake sample is generated, the landmark diffusion generation module uses a conditional input image generation model as an image generation backbone, uses a multi-scale U-Net network as a generator backbone in the conditional diffusion model, and introduces a cross attention mechanism in the image generation stage to fuse the semantic region and the text description prompt;

[0122] In the process of generating the new type of fake sample, a denoising optimizer is used to sample latent variables from a standard normal distribution , represented as and denoised latent variables are obtained by denoising through the multi-scale U-Net network in multiple iterations the denoised latent variables are input into the image decoder the image decoder is composed of a series of convolutional layers, activation functions and normalization layers, and the denoised latent variables are gradually restored to the same resolution as the input face image through the image decoder the denoised latent variables are gradually restored to the same resolution as the input face image, generating a new type of fake sample with high resolution and difficult to identify;

[0123] S2.3, the new type of fake sample is expanded to realize data enhancement.

[0124] In the specific implementation, S3 is as follows:

[0125] S3.1, a teacher model is constructed:

[0126] The teacher model includes a first teacher model, a second teacher model and a third teacher model;

[0127] The face recognition model ArcFace pre-trained using the MS1MV2 dataset is used as the first teacher model, and the face recognition model ArcFace does not process any fake class data and is dedicated to modeling the structural expression of real data;

[0128] The EfficientNet-B4 deep convolutional neural network model pre-trained using the ImageNet dataset is used as the second teacher model and the third teacher model;

[0129] S3.2, the image sample is input into the teacher model:

[0130] The real sample is input into the first teacher model to extract the real identity semantic feature ;

[0131] The fake sample and the new type of fake sample are input into the second teacher model and the third teacher model respectively to extract the fake identity semantic feature and the new type of fake identity semantic feature ;

[0132] S3.3, a loss function is constructed to optimize and train the teacher model:

[0133] (1) an inter-class contrast loss function is constructed :

[0134] A contrast mechanism based on cosine similarity is adopted, and the real identity semantic feature is respectively compared with the fake identity semantic feature and the new type of fake identity semantic feature The similarity measurement calculation is performed, and the calculation formula is as follows:

[0135] ,

[0136] wherein, represents the number of fake samples, represents the number of real samples, represents a function of similarity between the measurement features, and specifically, a cosine similarity is used as a distance measurement index to reflect the closeness of the angle between vectors;

[0137] (2) Constructing a category loss function :

[0138] Based on any one of the real identity semantic feature , the fake identity semantic feature set , the fake identity semantic feature , and the new fake identity semantic feature , a category loss function is constructed, and the calculation formula is as follows:

[0139] ,

[0140] wherein, represents the fake identity semantic feature, represents the number of fake identity semantic features, and the number of fake identity semantic features is equal to the number of fake samples, represents a binary classification label, 0 represents that the input is a real sample, and 1 represents that the input is a fake sample, represents a classification loss function for a binary classification task.

[0141] In the specific implementation, the operations in the deep weight enhancement module are as follows:

[0142] The deep weight enhancement module is constructed, and the module includes three parts, namely a channel division part, a convolution calculation part, and a channel splicing part;

[0143] The channel division part is composed of a group of channel dimension feature vector cutting operations;

[0144] The convolution calculation part includes three convolution layers, a 3x3 convolution layer , a 11x1 convolution layer , and a 1x11 convolution layer ;

[0145] The channel splicing part is composed of a channel splicing operation;

[0146] ​real identity semantic features to the deep weight enhancement module, the input real identity semantic features are divided into four sub-channel regions along the channel dimension, which are original channel features , spatial direction information enhanced features , vertical direction information enhanced features and horizontal direction information enhanced features ;

[0147] The spatial direction information enhanced features are extracted by using a depth separable convolution with a convolution kernel of 3x3 ;

[0148] The spatial direction information enhanced features are extracted by using a longitudinal convolution with a convolution kernel of 11x1 ;

[0149] The horizontal direction information enhanced features are extracted by using a transverse convolution with a convolution kernel of 1x11 ;

[0150] The output results of the three convolution branches are spliced along the channel dimension with the original channel features , and the dimension of the output features is adjusted to the same feature map size as , to obtain real enhanced features with fused multi-dimensional spatial structure information .

[0151] In the specific implementation, the operation of the category enhancement module is as follows:

[0152] The category enhancement module is constructed, which includes three parts, namely a five-image transformation strategy pool, a random selector and feature splicing. The random selector randomly selects one image transformation strategy for each of the two input features, which is realized by the random function in the Python standard library. Then the enhanced feature vectors of the two input features are spliced along the spatial dimension using a channel splicing function, which is a tensor operation function torch.cat provided by the top-level module torch of the PyTorch framework.

[0153] The fake identity semantic features and the new fake identity semantic features are input into the category enhancement module, and the input features are expanded and enhanced through image transformation strategies to construct a continuous fake feature variant space, thereby obtaining fake enhanced features ;

[0154] Wherein, the image transformation strategies are clustering transformation CT, extrapolation transformation ET, mixed transformation MT, scaling transformation ZT and noise transformation NT, and one image transformation strategy is randomly selected for expansion and enhancement;

[0155] (1) Clustering transformation CT: simulate different fake sources or sub-classes by feature clustering disturbance;

[0156] First, calculate the feature mean values of the fake identity semantic features and the new fake identity semantic features , and the calculation formula is as follows:

[0157]

[0158] Then, adjust and according to the feature mean values , and the adjustment formula is as follows:

[0159]

[0160] wherein, is a randomly sampled proportion factor between 0 and 1, represents the hyperbolic tangent function;

[0161] (2) Extrapolation transformation ET: interpolate boundary variants samples at the edge of the feature space to strengthen the decision boundary learning, and the specific process is as follows:

[0162]

[0163] wherein, represents the feature vector guiding the extrapolation process, , represents the fake identity semantic feature vector guiding the extrapolation process, represents the new fake identity semantic feature vector guiding the extrapolation process, represents a randomly sampled proportion factor between 0 and 1;

[0164] (3) Mixed transformation MT: fuse multiple fake image features to synthesize complex mixed fake images, and the specific process is as follows:

[0165]

[0166] wherein, , when , is , when , is​​​​​ ; denotes the random number extracted from distribution for controlling the mixing ratio of two input samples, whose numerical value is between 0 and 1;

[0167] (4) Scaling transformation ZT: affine scaling is applied to the image by applying an affine transformation matrix to simulate the impact of resolution or size changes on forgery, the affine transformation matrix combines rotation and scaling operations, and the affine transformation matrix is defined as follows:

[0168] ,

[0169] wherein, denotes the rotation angle, denotes the scaling coefficient;

[0170] The affine transformation matrix is applied to the spatial coordinates of the feature , thereby obtaining the enhanced feature, and the specific process is as follows:

[0171] ,

[0172] wherein, denotes the transpose, denotes the horizontal coordinate, denotes the vertical coordinate;

[0173] (5) Noise transformation NT: different types of random noise are injected into the forged image, and the calculation process is as follows:

[0174] ,

[0175] wherein, is the Gaussian noise generated according to , denotes the standard deviation of the noise, denotes the coefficient for controlling the noise addition intensity, which is randomly sampled between 0 and 1; wherein,

[0176] denotes the enhanced feature, , denotes the adjusted forged identity semantic feature, denotes the new forged identity semantic feature, and refer to the feature or , . .

[0177] In the detailed description, S5 is specifically as follows:

[0178] S5.1, construct a student model, use the ImageNet pre-trained model EfficientB4 as the student model;

[0179] S5.2, construct a distillation architecture from three teacher models and one student model, and align the output of the student model with the features output by the teacher model through a normalized exponential function The output of the student model is aligned with the features output by the teacher model And The alignment method is KL divergence alignment, which can realize multi-level feature transfer between the teacher model and the student model, and the specific process is as follows:

[0180] ,

[0181] Wherein, The distillation loss is represented by L, Refers to the features Or , , Refers to the real identity semantic features , fake identity semantic features Or new fake identity semantic features , When The real enhanced features output by the teacher model are Corresponding The real identity semantic features extracted by the student model are When The fake enhanced features output by the teacher model are Corresponding The fake identity semantic features extracted by the student model are And new fake identity semantic features ; Indicates the total amount of input samples, Indicates the Features Or , Indicates the Real identity semantic features , fake identity semantic features Or new fake identity semantic features ; Indicates the divergence alignment operation, which is used to measure the difference between two probability distributions Kullback-Leibler divergence; Indicates the adjustment of the normalized exponential function a smoothness temperature parameter of the teacher model;

[0182] During the training process of the student model, the distillation loss The gradient is back-propagated as the loss function to update the model parameters by minimizing the distillation loss The parameters of the student model are optimized to obtain a pre-trained student fake image detector with strong generalization ability after learning from the teacher model.

[0183] In the specific implementation, S6 is specifically as follows:

[0184] The discriminant decision module includes two convolution activation units, a global average pooling layer, a flattening layer, a fully connected layer, and a Dropout inactivation layer. The fake features output by the student fake image detector are input into the discriminant decision module. The module performs dimension compression and semantic mapping on the input features, and finally outputs a binary classification result of fake or real and the corresponding probability of fake or real.

[0185] Embodiment 2

[0186] A generable deep fake image detection system based on double data enhancement, which executes various step processing instructions in a generable deep fake image detection method based on double data enhancement, includes the following:

[0187] An image input module, a artifact feature extraction module, and a discriminant decision module.

[0188] The face image to be detected is input into the system through the image input module, and then passes through the artifact feature extraction module to extract the fake features, and then the fake features are input into the discriminant decision module to output the classification prediction result.

[0189] Among them, the artifact feature extraction module uses a pre-trained student model to extract features, and the discriminant decision module classifies and predicts according to the fake features.

[0190] Embodiment 3

[0191] In order to better prove the technical effect of the present application, the present application is applied to the actual application to train the deep fake detection model, and then improve the prediction accuracy.

[0192] As Figure 2As shown, the method of the present application is compared with the noise robustness performance of the existing methods LSDA (Latent Space Data Augmentation, a deep fake image detection algorithm based on latent space data augmentation), UCF (Uncovering Common Features, a deep fake image detection algorithm based on discovering common features), and FWA (Face Warp Artifacts, a deep fake image detection algorithm based on face deformation artifacts), and the evaluation index is AUC (Area Under the Curve). Under the same noise level, the AUC of the method of the present application is higher, and as the noise intensity (6 levels: 0-5) increases, the performance gap becomes more obvious, thereby proving the high robustness of the method of the present application.

[0193] The generation quality and similarity of the new fake images generated by the landmark diffusion generation module are evaluated. The source image is processed by the method of the present application and the existing methods SBI (Self-Blended Images, a deep fake image detection model based on self-mixed images) and CADDM (Cross-scale Artifact Deepfake Detection Module, a deep fake image detection model based on multi-scale artifact detection) to obtain generated images. As shown in Table 1, the source image is evaluated by SSIM, PSNR, and LPIPS indexes with the images generated by the three methods. SSIM is the structural similarity index, and the larger the value, the more similar the generated image. PSNR is the peak signal-to-noise ratio, and the larger the value, the better the generation effect. LPIPS is an index for measuring the perceptual similarity between images, and the smaller the value, the more similar the image. The superiority of the method of the present application in the image generation scenario can be verified.

[0194] Table 1 Performance comparison table of the present application and existing methods

[0195]

[0196] From the generation quality and similarity evaluation, the method of the present application is superior to the traditional methods (SBI, CADDM) in image generation, and the method of the present application shows significant advantages. Through the experimental results, it can be proved that the method of the present application is highly efficient in image generation, and provides a reliable scheme for image generation.

[0197] Although the specific embodiments of the application have been described in conjunction with the accompanying drawings, they are not a limitation on the scope of protection of the present application. Various modifications or variations made by those skilled in the art without creative labor on the basis of the technical solutions of the present application are still within the scope of protection of the present application.

Claims

1. A generalizable deepfake image detection method based on dual data augmentation, characterized in that, Includes the following steps: S1. Extract facial images from videos containing facial identities and preprocess them. Then, mark the preprocessed facial images as real and fake samples, respectively. S2. Construct a landmark diffusion generation model, input real samples into the landmark diffusion generation module, synthesize a new type of fake sample that is high-fidelity and difficult to identify, and perform data augmentation on the new fake sample; S3. Construct three teacher models, input real samples, fake samples and novel fake samples into the three teacher models respectively, and obtain real identity semantic features, fake identity semantic features and novel fake identity semantic features respectively, and guide the teacher models to optimize training through category loss and inter-class comparison loss; S4. Use the constructed deep weight enhancement module to perform data augmentation on the real identity semantic features to obtain real enhanced features, and use the constructed category enhancement module to perform systematic data augmentation on the fake identity semantic features and novel fake identity semantic features to obtain fake enhanced features. The specific operations in the deep weight enhancement module are as follows: A deep weight enhancement module is constructed, which consists of three parts: channel partitioning, convolution calculation, and channel concatenation. The channelized component consists of a set of feature vector segmentation operations along the channel dimension; The convolution computation part consists of three convolutional layers, including one 3×3 convolutional layer. An 11×1 convolutional layer A 1×11 convolutional layer ; The channel splicing section consists of channel splicing operations; semantic features of real identity The input is fed into the deep weighting augmentation module to process the semantic features of the real identity of the input. Dividing it along the channel dimension, it is divided into four sub-channel regions, which are the original channel features. Spatial orientation information enhancement features Vertical information enhancement features and horizontal information enhancement features ; Features that enhance spatial orientation information Depth-separable convolution with a kernel of 3×3 Extract multi-dimensional spatial features; Features that enhance spatial orientation information Vertical convolution with an 11×1 kernel is used. Extract vertical structural information; Features that enhance horizontal information Use lateral convolution with a kernel of 1×11 Extract horizontal structural information; The outputs of the three convolutional branches are compared with the original channel features. Perform concatenation along the channel dimension and adjust the dimension of the output features to match. Consistent feature map size yields true augmented features that fuse multidimensional spatial structural information. ; The specific operation of the category enhancement module is as follows: A category enhancement module is constructed, which consists of three parts: a five-element image transformation policy pool, a random selector, and feature concatenation. The random selector randomly selects an image transformation policy for each of the two input features, which is implemented by the random function in the Python standard library. Then, the enhanced feature vectors of the two input features are concatenated along the spatial dimension using the channel concatenation function. Semantic features of forged identities and new semantic features of forged identities The input is fed into the category enhancement module, which expands and enhances the input features through image transformation strategies, constructing a continuous variant space of fake features, thereby obtaining the fake enhanced features. ; The image transformation strategies are clustering transformation (CT), extrapolation transformation (ET), hybrid transformation (MT), scaling transformation (ZT), and noise transformation (NT). One image transformation strategy is randomly selected for extended enhancement. S5. Based on the KL divergence distillation architecture, the real enhancement features and fake enhancement features are transferred to the student model to obtain a student fake image detector learned from the teacher model. S6. Input the face image to be detected into the student fake image detector to extract fake features, and then input the extracted fake features into the discrimination decision module to output the judgment result.

2. The generalizable deepfake image detection method based on dual data augmentation according to claim 1, characterized in that, S1 is as follows: S1.1, Obtain facial image: Retrieve videos from an existing database containing real and fake faces, and extract facial images from videos containing facial identities based on custom frame count, height, width, and frame rate. A pre-trained facial landmark model from the machine learning library dlib is loaded as a face detection model. The face detection model is used to detect facial landmarks in the extracted facial images, locate the facial regions, crop them, and then determine the position information of the facial landmarks. Each facial region includes 68 landmarks, and the position information of the facial landmarks in each facial image is represented by a 68×2 two-dimensional matrix. The facial images are aligned according to the position information of the facial landmarks to eliminate pose and angle differences and obtain standardized facial images. S1.2 Facial image preprocessing: The standardized facial image is subjected to controllable image perturbation operations, including random horizontal flipping, random angle rotation and blur perturbation operations. The perturbation image is then adjusted to a uniform size and normalized according to the set mean and standard deviation to obtain the preprocessed facial image. S1.3, Authenticity Marking: The preprocessed facial images are labeled to distinguish between real and fake facial images, and the two types of facial images are used as real samples and fake samples, respectively.

3. The generalizable deepfake image detection method based on dual data augmentation according to claim 2, characterized in that, S2 is as follows: S2.1 Based on the facial key point location information of real samples, the face is divided into multiple semantically specific regions, and all semantic regions are represented as follows: The semantic regions include the left eye, right eye, nose, mouth, and combined regions. Indicates the first A semantic region, Indicates the number of semantic regions; At the same time, a corresponding text description is constructed based on the face region mask, which serves as a text description prompt; S2.2 Construct a landmark diffusion generation module. Input the face region mask and text description prompts mentioned above into the landmark diffusion generation module to generate a new type of fake sample. The landmark diffusion generation module adopts a conditional input image generation model as the image generation backbone and a multi-scale U-Net network as the generator backbone in the conditional diffusion model. In the image generation stage, a cross-attention mechanism is introduced to fuse semantic regions and text description prompts. In the process of generating novel fake samples, a denoising optimizer is used to obtain samples from a standard normal distribution. latent variables Sampling is performed, represented as The denoised latent variables were obtained through multiple iterations using a multi-scale U-Net network. The latent variables after denoising Input to image decoder Image decoder It consists of a series of convolutional layers, activation functions, and normalization layers, and is processed by an image decoder. Denoising the latent variables Gradually restore the image to the same resolution as the input facial image to generate a new type of forged sample that is high-resolution and difficult to detect; S2.

3. Expand the new forged samples to achieve data augmentation.

4. The generalizable deepfake image detection method based on dual data augmentation according to claim 3, characterized in that, S3 Specifically as follows: S3.1 Constructing a teacher model: The teacher models include the first teacher model, the second teacher model, and the third teacher model; The ArcFace face recognition model, pre-trained on the MS1MV2 dataset, was used as the first teacher model. The EfficientNet-B4 deep convolutional neural network model pre-trained on the ImageNet dataset was used as the second and third teacher models. S3.2 Input the image samples into the teacher model: Real samples are input into the first teacher model to extract real identity semantic features. ; The forged samples and novel forged samples were input into the second teacher model and the third teacher model, respectively, to extract semantic features of the forged identities. and new semantic features of forged identities ; S3.3 Constructing a loss function to optimize and train the teacher model: (1) Constructing the inter-class comparison loss function : A cosine similarity-based comparison mechanism is used to integrate the semantic features of real identities. Semantic features of forged identities and new semantic features of forged identities The similarity metric is calculated using the following formula: ， in, Indicates the number of fake samples. Indicates the number of real samples. A function that measures the similarity between features; (2) Constructing the category loss function : Based on semantic features of real identity Semantic feature set of forged identity Any semantic feature of a forged identity and new semantic features of forged identities Construct the category loss function The calculation formula is as follows: ， in, Indicates the first Semantic features of a forged identity This represents the number of forged identity semantic features, which is equal to the number of forged samples. This represents a binary classification label, where 0 indicates that the input is a real sample, and 1 indicates that the input is a fake sample. This represents the classification loss function used for binary classification tasks.

5. The generalizable deepfake image detection method based on dual data augmentation according to claim 4, characterized in that, The image transformation strategy operates as follows: (1) Cluster Transformation CT: Simulates different forgery sources or subclasses through feature cluster perturbation; First, calculate the semantic features of the forged identity. and new semantic features of forged identities The characteristic average The calculation formula is: ， Then, based on the characteristic average Adjustment and The formula is adjusted as follows: ， in, It is a scaling factor for random sampling between 0 and 1. Represents the hyperbolic tangent function; (2) Extrapolation Transformation (ET): Interpolation is performed at the edge of the feature space to generate boundary variant samples, which strengthens the learning of decision boundaries. The specific process is as follows: ， in, This represents the feature vector that guides the extrapolation process. , This represents the semantic feature vector of the forged identity that guides the extrapolation process. This represents a novel semantic feature vector for forged identities that guides the extrapolation process. This represents the scaling factor for random sampling between 0 and 1; (3) Hybrid Transform (MT): This method fuses multiple features of forged images to synthesize complex hybrid forged images. The specific process is as follows: ， in, , for hour, for , for hour, for ; Indicates from distributed The random number drawn from the sample is used to control the mixing ratio of the two input samples. The value is between 0 and 1; (4) Scaling transformation ZT: by applying the affine transformation matrix Affine scaling of the image simulates the effect of resolution or size changes on forgery; affine transformation matrix. Affine transformation matrix combines rotation and scaling operations. The definition is as follows: ， in, Indicates the rotation angle. Indicates the scaling factor; affine transformation matrix Application to features spatial coordinates This yields the enhanced features, and the specific process is as follows: ， in, Indicates transpose. Represents the x-axis, Represents the ordinate; (5) Noise Transformation NT: Different types of random noise are injected into the forged image. The calculation process is as follows: ， in, It is based on The generated Gaussian noise, The standard deviation of noise The coefficient representing the control of the added noise intensity is randomly sampled between 0 and 1; in, Indicates the enhanced features, , This represents the adjusted semantic features of the forged identity. This indicates the semantic features of a new type of forged identity. and Referential features or , .

6. The generalizable deepfake image detection method based on dual data augmentation according to claim 5, characterized in that, S5 is detailed below: S5.1 Build a student model, using the ImageNet pre-trained model EfficientB4 as the student model; S5.

2. A distillation architecture is constructed using three teacher models and one student model, through a normalized exponential function. Features of the student model output and the teacher model output and The alignment is performed using KL divergence alignment, and the specific process is as follows: ， in, Indicates distillation loss, Referential features or , , The real identity semantic features extracted by the student model Semantic features of forged identities Or new semantic features of forged identities , ,when Realistic augmented features output for the teacher model When, corresponding Extracting real identity semantic features for student models ,when Fake augmented features output for the teacher model When, corresponding Semantic features of forged identities extracted for student models and new semantic features of forged identities ; This represents the total number of input samples. Indicates the first Features or , Indicates the first Semantic features of a real identity Semantic features of forged identities Or new semantic features of forged identities ; This indicates a divergence alignment operation; Represents the adjustment of the normalized exponential function Smoothness temperature parameters; During the training of the student model, distillation loss will be applied. The gradient is backpropagated as the loss function to update the model parameters, thereby minimizing the distillation loss. The parameters of the student model are optimized to obtain a student fake image detector that has learned from the teacher model.

7. The generalizable deepfake image detection method based on dual data augmentation according to claim 6, characterized in that, S6 is detailed below: The discrimination decision module includes two convolutional activation units, a global average pooling layer, a flattening layer, a fully connected layer, and a Dropout deactivation layer. The forgery features output by the student forgery image detector are input into the discrimination decision module. This module performs dimensionality compression and semantic mapping on the input features, and finally outputs the binary classification result of forgery or real and the probability of forgery or real.

8. A generalizable deepfake image detection system based on dual data augmentation, comprising a module that executes the processing instructions for each step of the generalizable deepfake image detection method based on dual data augmentation as described in any one of claims 1-7, characterized in that, include: Image input module, artifact characterization extraction module, and discrimination decision module; The face image to be detected is input into the system through the image input module, then passes through the artifact representation extraction module to extract forgery features, and then the forgery features are input into the decision discrimination module to output the classification prediction result; The forgery characterization extraction module uses a pre-trained student model to extract features, while the decision discrimination module performs classification and prediction based on the forgery features.

Images