Generalized deeply-forged image detection method and system based on dual data enhancement

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a generalization deeply-forged image detection method and system based on dual data enhancement. The method specifically comprises the following steps: preprocessing and marking an extracted face image; the constructed landmark diffusion generation model generates a novel forged sample; respectively inputting the real sample, the forged sample and the novel forged sample into the three teacher models to respectively obtain corresponding identity semantic features; the identity semantic features are enhanced; on the basis of a KL divergence distillation framework, the enhanced feature is migrated to a student model; and inputting a to-be-detected face image into the student model to extract counterfeit features, then inputting the counterfeit features into the judgment decision module, and outputting a judgment result. The method is lower in test cost, higher in efficiency, better in generalization, higher in robustness, and suitable for efficient and low-cost detection scenes of wide deep forged images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a generalizable deep fake image detection method and system based on double data enhancement. Background Art

[0002] The rapid development of deepfake technology has enabled facial identity manipulation to achieve unprecedented visual authenticity, attracting significant attention in areas such as film and television production and digital entertainment. Building a reliable and universal deepfake detection system has become a core requirement for ensuring the security of digital content.

[0003] While current mainstream detection methods can achieve very high accuracy within specific datasets, they generally experience significant performance degradation when tested across datasets. This is because most current methods (such as artifact region localization, noise pattern analysis, and frequency domain feature extraction) are optimized for specific forgery traces (such as fingerprints generated by GANs). While successful in specific scenarios, these methods inherently overfit to known forgery patterns and lack the ability to generalize to unseen real-world scenarios. This makes them difficult to handle the cross-domain variations brought about by emerging forgery techniques (such as diffusion-based deepfakes) and complex real-world scenarios.

[0004] To overcome the generalization bottleneck, many methods utilize data augmentation to expand the decision boundary of forged feature representations. To overcome the overfitting problem on specific datasets, several studies have demonstrated that data augmentation is an effective tool that can significantly improve generalization. Typical approaches include region-blurring-based GAN artifact simulation, generating composite forged samples by mixing multiple source images, and adversarial training to enhance model robustness. These methods reproduce the mixing artifact by generating new mixed images from two original image pairs, and generate forged images by mixing forged images from a single image with the target image. These methods offer models a more general and challenging task for face forgery detection. Their commonality is that they generate forged samples that are more difficult to identify and contain traces of face forgery, thereby encouraging models to learn more general and robust representations for face forgery detection. However, existing augmentation strategies suffer from fundamental flaws: the visual quality and realism of the synthesized samples are insufficient, and they often rely on traditional generation methods. This lacks diversity in the synthesized data and hinders effective generalization to unknown forgery types, defeating their original purpose.

[0005] To address the above issues, an enhancement strategy is urgently needed that can systematically expand the spatial distribution of forgeries, generate high-visual-fidelity forged samples, and possess good transferability. Furthermore, the rise of generative models, especially diffusion models, has provided a new approach for constructing high-quality, controllable forged samples. Diffusion models possess extremely high image generation capabilities while preserving image details and semantic consistency, making them particularly suitable for injecting local forged features while maintaining identity integrity. Combining the diffusion generation mechanism with a spatial constraint strategy guided by forgery masks is expected to generate challenging forged samples that are closer to the true distribution, thereby promoting deep forgery detection models to learn more robust, generalizable, and method-independent feature representations, thus breaking through the limitations of existing methods in cross-domain generalization performance.

[0006] Therefore, the present invention proposes a generalizable deep fake image detection method and system based on double data enhancement to solve the above problems. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention develops a generalizable deep fake image detection method and system based on double data enhancement. The main purpose is to improve the deep fake detection capability, make the testing cost lower, the efficiency higher, the generalization better, and the robustness stronger.

[0008] On the one hand, the technical solution to the technical problem of the present invention is a generalizable deep fake image detection method based on double data enhancement, which specifically comprises the following steps: S1. Extract facial images from videos containing facial identities and preprocess them. Then, mark the preprocessed facial images as true and false, respectively, as real samples and forged samples. S2. Build a landmark diffusion generation model, input real samples into the landmark diffusion generation module, synthesize high-fidelity and difficult-to-identify new forged samples, and perform data enhancement on the new forged samples; S3. Construct three teacher models, input the real samples, forged samples and new forged samples into the three teacher models respectively, obtain the real identity semantic features, forged identity semantic features and new forged identity semantic features respectively, and guide the teacher models to perform optimization training through category loss and inter-class contrast loss; S4. Use the constructed deep weight enhancement module to perform data enhancement on the real identity semantic features to obtain real enhanced features, and use the constructed category enhancement module to perform systematic data enhancement on the forged identity semantic features and the new forged identity semantic features to obtain forged enhanced features; S5. Based on the KL divergence distillation architecture, the real enhancement features and the forged enhancement features are transferred to the student model to obtain a student forged image detector learned from the teacher model; S6. Input the facial image to be detected into the student forged image detector to extract forged features, and then input the extracted forged features into the discrimination decision module to output the judgment result.

[0009] S1 is as follows: S1.1. Obtain facial image: Obtain videos from an existing database containing real and fake faces, and extract facial images from videos containing face identities based on customized frame count, height, width, and frame rate; A pre-trained facial landmark model from the machine learning tool library dlib is loaded as a face detection model. The model then performs facial landmark detection on the extracted facial image. The facial region is located and cropped, and the locations of the facial landmarks are determined. The facial region contains 68 landmarks, and the locations of the facial landmarks for each facial image are represented as a 68×2 two-dimensional matrix. The facial images are aligned based on the facial landmark location information to eliminate differences in posture and angle, resulting in a standardized facial image. S1.2, facial image preprocessing: Perform controllable image perturbations on the standardized facial images, including random horizontal flipping, random angle rotation, and blurring. The perturbed images are then resized to a uniform size and normalized according to the set mean and standard deviation to obtain the preprocessed facial images. S1.3、Authenticity mark: The preprocessed facial images are marked to identify real facial images and forged facial images, and the two facial images are used as real samples and forged samples respectively.

[0010] S2 is as follows: S2.1. Based on the facial key point position information of the real sample, the face is divided into multiple regions with specific semantics. All semantic regions are represented as ,Semantic regions include left eye, right eye, nose, mouth and combined regions, Indicates the semantic regions, Indicates the number of semantic regions; At the same time, the corresponding text description is constructed according to the face area mask as a text description prompt; S2.2. Construct a landmark diffusion generation module. Input the above-mentioned face region mask and text description prompt into the landmark diffusion generation module to generate new forged samples. The landmark diffusion generation module uses a conditional input image generation model as the image generation backbone, adopts a multi-scale U-Net network as the generator backbone in the conditional diffusion model, and introduces a cross-attention mechanism in the image generation stage to fuse the semantic region and text description prompt. In the process of generating new fake samples, the denoising optimizer is used to extract the data from the standard normal distribution. latent variables Sampling is performed, which is expressed as , and iterates multiple times through the multi-scale U-Net network to obtain the denoised latent variables , the denoised latent variables Input to image decoder , image decoder It is composed of a series of convolutional layers, activation functions and normalization layers, and is decoded by the image decoder. The denoised latent variables Gradually restore the image to the same resolution as the input face image, generating new forged samples that are high-resolution and difficult to identify; S2.3. Expand new forged samples to achieve data enhancement.

[0011] S3 is as follows: S3.1. Constructing the teacher model: The teacher models include the first teacher model, the second teacher model and the third teacher model; Use the face recognition model ArcFace pre-trained on the MS1MV2 dataset as the first teacher model; Use the EfficientNet-B4 deep convolutional neural network model pre-trained on the ImageNet dataset as the second and third teacher models; S3.2. Input image samples into the teacher model: Input the real sample into the first teacher model to extract the real identity semantic features ; The forged samples and new forged samples are input into the second teacher model and the third teacher model respectively to extract the forged identity semantic features. and new forged identity semantic features ; S3.3. Construct a loss function to optimize the teacher model training: (1) Constructing inter-class contrast loss function : Using a cosine similarity-based comparison mechanism, the real identity semantic features Semantic features of forged identities and new forged identity semantic features The similarity measurement calculation is performed as follows: , in, represents the number of forged samples, represents the number of real samples, Represents a function that measures the similarity between features; (2) Constructing category loss function : Based on real identity semantic features , forged identity semantic feature set Any of the forged identity semantic features and new forged identity semantic features , construct the category loss function , the calculation formula is as follows: , in, Indicates the Item forged identity semantic features, represents the number of forged identity semantic features, which is equal to the number of forged samples. Represents a binary classification label, 0 indicates that the input is a real sample, 1 indicates that the input is a fake sample, Represents the classification loss function for the binary classification task.

[0012] The operations in the depth weight enhancement module are as follows: Construct a deep weight enhancement module, which consists of three parts: channel division, convolution calculation, and channel splicing. The channelization part consists of a set of feature vector segmentation operations on the channel dimension; The convolution calculation part includes three convolution layers, a 3×3 convolution layer , an 11×1 convolutional layer , a 1×11 convolutional layer ; The channel splicing part consists of channel splicing operations; Real identity semantic features Input to the deep weight enhancement module to input the true identity semantic features Divide it along the channel dimension into four sub-channel areas, which are the original channel features , spatial direction information enhancement features , vertical information enhancement features and horizontal information enhancement features ; Enhance features of spatial direction information Use depth-separable convolution with a convolution kernel of 3×3 , extract multidimensional spatial features; Enhance features of spatial direction information Use vertical convolution with a convolution kernel of 11×1 , extract vertical structural information; Enhance features for horizontal information Use horizontal convolution with a convolution kernel of 1×11 , extract horizontal structural information; Combine the output results of the three convolution branches with the original channel features Perform concatenation on the channel dimension and adjust the dimension of the output feature to be the same as Consistent feature map size, to obtain real enhanced features that integrate multi-dimensional spatial structure information .

[0013] The operations of the category enhancement module are as follows: Build a category enhancement module, which consists of three parts: a five-element image transformation strategy pool, a random selector, and feature splicing. The random selector randomly selects an image transformation strategy for each of the two input features, implemented by the random function in the Python standard library. It then splices the feature vectors of the two enhanced input features along the spatial dimension using the channel splicing function. Forged identity semantic features and new forged identity semantic features The input is sent to the category enhancement module, and the input features are expanded and enhanced through the image transformation strategy to construct a continuous forged feature variant space, thereby obtaining the forged enhanced features. ; Among them, the image transformation strategies are clustering transformation CT, extrapolation transformation ET, hybrid transformation MT, scaling transformation ZT and noise transformation NT. One image transformation strategy is randomly selected for expansion enhancement. (1) Clustering transformation (CT): simulating different forgery sources or subclasses through feature clustering perturbations; First, calculate the forged identity semantic features and new forged identity semantic features The characteristic mean , the calculation formula is , Then, according to the feature mean Adjustment and , the adjustment formula is as follows: , in, is the scaling factor for random sampling between 0 and 1, represents the hyperbolic tangent function; (2) Extrapolation Transformation ET: Generate boundary variant samples by interpolation at the edge of the feature space to strengthen decision boundary learning. The specific process is as follows: , in, represents the eigenvector guiding the extrapolation process, , represents the forged identity semantic feature vector that guides the extrapolation process, represents a new forged identity semantic feature vector that guides the extrapolation process, Represents the scaling factor for random sampling between 0 and 1; (3) Hybrid Transformation MT: Fusion of multiple forged image features to synthesize complex hybrid forged images. The specific process is as follows: , in, , for hour, for , for hour, for ; Indicates from distributed A random number drawn from the input to control the mixing ratio of the two input samples, The value of is between 0 and 1; (4) Scaling transformation ZT: by applying the affine transformation matrix Perform affine scaling on an image to simulate the artifacts caused by resolution or size changes, using the affine transformation matrix Combined with rotation and scaling operations, the affine transformation matrix The definition is as follows: , in, Indicates the rotation angle, represents the scaling factor; The affine transformation matrix Apply to Features The spatial coordinates of , thus obtaining the enhanced features, the specific process is as follows: , in, represents transpose, represents the horizontal axis, Indicates the vertical coordinate; (5) Noise Transformation NT: Different types of random noise are injected into the forged image. The calculation process is as follows: , in, is based on The generated Gaussian noise, represents the standard deviation of the noise, Represents the coefficient that controls the intensity of noise addition, which is randomly sampled between 0 and 1; in, represents the enhanced features, , represents the adjusted forged identity semantic features, Represents new forged identity semantic features, and Referential features or , .

[0014] S5 is as follows: S5.1. Build a student model and use the ImageNet pre-trained model EfficientB4 as the student model; S5.2. Construct a distillation architecture by three teacher models and one student model, using a normalized exponential function The output of the student model is compared with the features of the teacher model output and Alignment is performed using KL divergence alignment. The specific process is as follows: , in, represents the distillation loss, Referential features or , , Refers to the true identity semantic features extracted by the student model , forged identity semantic features or new semantic features of forged identities , ,when True enhanced features output by the teacher model When Ground truth identity semantic features extracted for the student model ,when Forged enhanced features output by the teacher model When Forged identity semantic features extracted for the student model and new forged identity semantic features ; Indicates the total number of input samples, Indicates the Features or , Indicates the True identity semantic features , forged identity semantic features or new semantic features of forged identities ; represents the divergence alignment operation; Represents the adjusted normalized exponential function Smoothness temperature parameter; During the training of the student model, the distillation loss Backpropagating gradients as a loss function, updating model parameters by minimizing the distillation loss The parameters of the student model are optimized to obtain a student forged image detector learned from the teacher model.

[0015] S6 is as follows: The discriminant decision module includes two convolutional activation units, a global average pooling layer, a flattening layer, a fully connected layer and a Dropout inactivation layer. The forged features output by the student forged image detector are input into the discriminant decision module, which performs dimensionality compression and semantic mapping on the input features, and finally outputs a forged or real binary classification result and the corresponding probability of forged or real.

[0016] On the other hand, the present invention also provides a generalizable deep fake image detection system based on dual data enhancement, comprising a module for executing processing instructions for each step in the generalizable deep fake image detection method based on dual data enhancement, including: an image input module, an artifact representation extraction module, and a discrimination decision module; The face image to be detected is input into the system through the image input module, and then passes through the artifact representation extraction module to extract the forged features. The forged features are then input into the decision discrimination module to output the classification prediction results. Among them, the artifact representation extraction module uses the pre-trained student model to extract features, and the decision discrimination module performs classification prediction based on the forged features.

[0017] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects: (1) High-precision and strong generalization deepfake detection: Feature-level enhancement is performed through the deep weight enhancement module and the category enhancement module to improve the detection performance and robustness of the detection method. At the same time, the KL divergence distillation loss is used to achieve high-level knowledge transfer from the teacher model to the student model to enhance the generalization ability of the student model. This method deeply explores and expands the broader space of forged features, enabling the model to learn decision boundaries covering various forgery patterns, and ultimately improves its detection performance on various unknown forgery types. (2) High-fidelity and high-resolution deep fake synthesis: By combining the diffusion model with the forged image data, the facial organs (eyes, nose, mouth) are used as spatial constraints to ensure that the generated image retains the original identity features in the ungenerated areas; the newly generated forged images have high visual fidelity and high resolution, and are almost indistinguishable from the real pictures; the method uses the key points of the facial region (such as eyes, nose, mouth, etc.) as structural conditions and adopts the conditional diffusion model to generate images with specific forged areas; the method uses the decoder to restore the high-resolution forged samples by cross-attention fusion of the real image, mask map and text prompts, and retains the edge features of the real identity, thereby generating deep fake images that are visually difficult to distinguish from the real image.

[0018] (3) Innovative data augmentation strategy: The collaborative augmentation strategy of data augmentation and data expansion effectively addresses the limitations of existing methods in capturing sample-level diversity and feature-level complexity, and achieves better generalization performance than methods that rely solely on augmentation or expansion.

[0019] Therefore, the method proposed in this invention can generate challenging forged samples that are closer to the real distribution, and can promote the student model to learn more robust, generalizable and method-independent feature representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0021] Figure 1 Schematic diagram of the process of the present invention.

[0022] Figure 2 A comparison chart of the noise robustness of the method of the present invention and the existing method. DETAILED DESCRIPTION

[0023] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0024] Example 1 like Figure 1As shown in Figure 1, a generalizable deep fake image detection method based on double data augmentation is proposed. The specific steps are as follows: S1. Extract facial images from videos containing facial identities and preprocess them. Then, mark the preprocessed facial images as true and false, respectively, as real samples and forged samples. S2. Build a landmark diffusion generation model, input real samples into the landmark diffusion generation module, synthesize high-fidelity and difficult-to-identify new forged samples, and perform data enhancement on the new forged samples; S3. Construct three teacher models, input the real samples, forged samples and new forged samples into the three teacher models respectively, obtain the real identity semantic features, forged identity semantic features and new forged identity semantic features respectively, and guide the teacher models to perform optimization training through category loss and inter-class contrast loss; S4. Use the constructed deep weight enhancement module to perform data enhancement on the real identity semantic features to obtain real enhanced features, and use the constructed category enhancement module to perform systematic data enhancement on the forged identity semantic features and the new forged identity semantic features to obtain forged enhanced features; S5. Based on the KL divergence distillation architecture, the real enhancement features and the forged enhancement features are transferred to the student model to obtain a student forged image detector learned from the teacher model; S6. Input the facial image to be detected into the student forged image detector to extract forged features, and then input the extracted forged features into the discrimination decision module to output the judgment result.

[0025] In a specific implementation manner, S1 is specifically as follows: S1.1. Obtain facial image: Obtain videos from an existing database containing real and fake faces, and extract facial images from videos containing face identities based on customized frame count, height, width, and frame rate; Load the pre-trained facial key point model in the machine learning tool library dlib ( ) as a face detection model, which detects facial key points on the extracted facial images, locates the facial region and then crops it, and then determines the location information of the facial key points. The facial region includes 68 key points, and the location information of the facial key points of each facial image is represented by a 68×2 two-dimensional matrix. The facial images are aligned based on the facial key point location information to eliminate posture and angle differences and obtain a standardized facial image; S1.2, facial image preprocessing: Perform controllable image perturbations on the standardized facial images, including random horizontal flipping, random angle rotation, and blurring. The perturbed images are then resized to a uniform size and normalized according to the set mean and standard deviation to obtain the preprocessed facial images. S1.3、Authenticity mark: The preprocessed facial images are marked to identify real facial images and forged facial images, and the two facial images are used as real samples and forged samples respectively.

[0026] In a specific implementation manner, S2 is specifically as follows: S2.1. Based on the facial key point position information of the real sample, a multi-region facial organ semantic coordinate map is constructed. The face region is divided into multiple semantic regions according to the face region mask. The semantic region generation is represented as ,Semantic regions include left eye, right eye, nose, mouth and combined regions, Indicates the semantic regions, Indicates the number of semantic regions; Construct a corresponding text description based on the face area mask as a text description prompt; S2.2. Construct a landmark diffusion generation module. Input the above-mentioned face region mask and text description prompt into the landmark diffusion generation module to generate new forged samples. The landmark diffusion generation module uses a conditional input image generation model as the image generation backbone, adopts a multi-scale U-Net network as the generator backbone in the conditional diffusion model, and introduces a cross-attention mechanism in the image generation stage to fuse the semantic region and text description prompt. In the process of generating new fake samples, the denoising optimizer is used to extract the data from the standard normal distribution. latent variables Sampling is performed, which is expressed as , and iterates multiple times through the multi-scale U-Net network to obtain the denoised latent variables , the denoised latent variables Input to image decoder , image decoder It is composed of a series of convolutional layers, activation functions and normalization layers, and is decoded by the image decoder. The denoised latent variables Gradually restore the image to the same resolution as the input face image, generating new forged samples that are high-resolution and difficult to identify; S2.3. Expand new forged samples to achieve data enhancement.

[0027] In a specific implementation, S3 is as follows: S3.1. Constructing the teacher model: The teacher models include the first teacher model, the second teacher model and the third teacher model; The face recognition model ArcFace pre-trained on the MS1MV2 dataset is used as the first teacher model. The face recognition model ArcFace does not process any fake data and is dedicated to modeling the structural expression of real data; Use the EfficientNet-B4 deep convolutional neural network model pre-trained on the ImageNet dataset as the second and third teacher models; S3.2. Input image samples into the teacher model: Input the real sample into the first teacher model to extract the real identity semantic features ; The forged samples and new forged samples are input into the second teacher model and the third teacher model respectively to extract the forged identity semantic features. and new forged identity semantic features ; S3.3. Construct a loss function to optimize the teacher model training: (1) Constructing inter-class contrast loss function : Using a cosine similarity-based comparison mechanism, the real identity semantic features Semantic features of forged identities and new forged identity semantic features The similarity measurement calculation is performed as follows: , in, represents the number of forged samples, represents the number of real samples, A function that measures the similarity between features. Specifically, cosine similarity is used as a distance metric to reflect the closeness of the angle between vectors. (2) Constructing category loss function : Based on real identity semantic features , forged identity semantic feature set Any of the forged identity semantic features and new forged identity semantic features , construct the category loss function , the calculation formula is as follows: , in, Indicates the Item forged identity semantic features, represents the number of forged identity semantic features, which is equal to the number of forged samples. Represents a binary classification label, 0 indicates that the input is a real sample, 1 indicates that the input is a fake sample, Represents the classification loss function for the binary classification task.

[0028] In a specific implementation, the operations in the depth weight enhancement module are as follows: Construct a deep weight enhancement module, which consists of three parts: channel division, convolution calculation, and channel splicing. The channelization part consists of a set of feature vector segmentation operations on the channel dimension; The convolution calculation part includes three convolution layers, a 3×3 convolution layer , an 11×1 convolutional layer , a 1×11 convolutional layer ; The channel splicing part consists of channel splicing operations; Real identity semantic features Input to the deep weight enhancement module to input the true identity semantic features Divide it along the channel dimension into four sub-channel areas, which are the original channel features , spatial direction information enhancement features , vertical information enhancement features and horizontal information enhancement features ; Enhance features of spatial direction information Use depth-separable convolution with a convolution kernel of 3×3 , extract multidimensional spatial features; Enhance features of spatial direction information Use vertical convolution with a convolution kernel of 11×1 , extract vertical structural information; Enhance features for horizontal information Use horizontal convolution with a convolution kernel of 1×11 , extract horizontal structural information; Combine the output results of the three convolution branches with the original channel features Perform concatenation on the channel dimension and adjust the dimension of the output feature to be the same as Consistent feature map size, to obtain real enhanced features that integrate multi-dimensional spatial structure information .

[0029] In a specific implementation, the operations of the category enhancement module are as follows: Build a category enhancement module, which consists of three parts: a five-element image transformation strategy pool, a random selector, and feature splicing. The random selector randomly selects an image transformation strategy for each of the two input features, implemented by the random function in the Python standard library. It then concatenates the enhanced feature vectors of the two input features along the spatial dimension using a channel splicing function. The channel splicing function is specifically the tensor operation function torch.cat provided by the torch top-level module in the PyTorch framework. Forged identity semantic features and new forged identity semantic features The input is sent to the category enhancement module, and the input features are expanded and enhanced through the image transformation strategy to construct a continuous forged feature variant space, thereby obtaining the forged enhanced features. ; Among them, the image transformation strategies are clustering transformation CT, extrapolation transformation ET, hybrid transformation MT, scaling transformation ZT and noise transformation NT. One image transformation strategy is randomly selected for expansion enhancement. (1) Clustering transformation (CT): simulating different forgery sources or subclasses through feature clustering perturbations; First, calculate the forged identity semantic features and new forged identity semantic features The characteristic mean , the calculation formula is as follows: , Then, according to the feature mean Adjustment and , the adjustment formula is as follows: , in, is the scaling factor for random sampling between 0 and 1, represents the hyperbolic tangent function; (2) Extrapolation Transformation ET: Generate boundary variant samples by interpolation at the edge of the feature space to strengthen decision boundary learning. The specific process is as follows: , in, represents the eigenvector guiding the extrapolation process, , represents the forged identity semantic feature vector that guides the extrapolation process, represents a new forged identity semantic feature vector that guides the extrapolation process, Represents the scaling factor for random sampling between 0 and 1; (3) Hybrid Transformation MT: Fusion of multiple forged image features to synthesize complex hybrid forged images. The specific process is as follows: , in, , for hour, for , for hour, for ; Indicates from distributed A random number drawn from the input to control the mixing ratio of the two input samples, The value of is between 0 and 1; (4) Scaling transformation ZT: by applying the affine transformation matrix Perform affine scaling on an image to simulate the artifacts caused by resolution or size changes, using the affine transformation matrix Combined with rotation and scaling operations, the affine transformation matrix The definition is as follows: , in, Indicates the rotation angle, represents the scaling factor; The affine transformation matrix Apply to Features The spatial coordinates of , thus obtaining the enhanced features, the specific process is as follows: , in, represents transpose, represents the horizontal axis, Indicates the vertical coordinate; (5) Noise Transformation NT: Different types of random noise are injected into the forged image. The calculation process is as follows: , in, is based on The generated Gaussian noise, represents the standard deviation of the noise, Represents the coefficient that controls the intensity of noise addition, which is randomly sampled between 0 and 1; in, represents the enhanced features, , represents the adjusted forged identity semantic features, Represents new forged identity semantic features, and Referential features or , .

[0030] In a specific implementation, S5 is as follows: S5.1. Build a student model and use the ImageNet pre-trained model EfficientB4 as the student model; S5.2. Construct a distillation architecture by three teacher models and one student model, using a normalized exponential function The output of the student model is compared with the features of the teacher model output and Alignment is performed using KL divergence alignment, which can achieve multi-level feature migration between the teacher model and the student model. The specific process is as follows: , in, represents the distillation loss, Referential features or , , Refers to the true identity semantic features extracted by the student model , forged identity semantic features or new semantic features of forged identities , ,when True enhanced features output by the teacher model When Ground truth identity semantic features extracted for the student model ,when Forged enhanced features output by the teacher model When Forged identity semantic features extracted for the student model and new forged identity semantic features ; Indicates the total number of input samples, Indicates the Features or , Indicates the True identity semantic features , forged identity semantic features or new semantic features of forged identities ; represents the divergence alignment operation, which is used to measure the difference between two probability distributions, Kullback-Leibler scattering; Represents the adjusted normalized exponential function Smoothness temperature parameter; During the training of the student model, the distillation loss Backpropagating gradients as a loss function, updating model parameters by minimizing the distillation loss The parameters of the student model are optimized to obtain a student forged image detector that is lightweight and has strong generalization ability after being pre-trained from the teacher model.

[0031] In a specific implementation manner, S6 is specifically as follows: The discriminant decision module includes two convolutional activation units, a global average pooling layer, a flattening layer, a fully connected layer and a Dropout inactivation layer. The forged features output by the student forged image detector are input into the discriminant decision module, which performs dimensionality compression and semantic mapping on the input features, and finally outputs a forged or real binary classification result and the corresponding probability of forged or real.

[0032] Example 2 A generalizable deepfake image detection system based on dual data augmentation, including a module that executes processing instructions for each step in a generalizable deepfake image detection method based on dual data augmentation, including the following: Image input module, artifact representation extraction module and discrimination decision module; The face image to be detected is input into the system through the image input module, and then passes through the artifact representation extraction module to extract the forged features. The forged features are then input into the decision discrimination module to output the classification prediction results. Among them, the artifact representation extraction module uses the pre-trained student model to extract features, and the decision discrimination module performs classification prediction based on the forged features.

[0033] Example 3 To better demonstrate the technical effects of the present invention, the present invention was applied to practical applications to train a deepfake detection model, thereby improving the accuracy of predictions. like Figure 2As shown, the noise robustness performance of the method of the present invention is compared with that of the existing methods LSDA (Latent Space Data Augmentation - a deep fake image detection algorithm based on latent space data enhancement), UCF (Uncovering Common Features - a deep fake image detection algorithm based on discovering common features), and FWA (Face Warp Artifacts - a deep fake image detection algorithm based on face deformation artifacts). The evaluation indicator is AUC (Area Under the Curve). Under the same noise level, the method of the present invention has a higher AUC. As the noise intensity (a total of 6 levels: 0-5) increases, the performance gap becomes more obvious, which proves the high robustness of the method of the present invention. The generation quality and similarity of the new forged images generated by the landmark diffusion generation module were evaluated. The source images were processed using the method of the present invention and the existing methods SBI (Self-Blended Images - a deep forged image detection model based on self-blended images) and CADDM (Cross-scale Artifact Deepfake Detection Module - a deep forged image detection model based on multi-scale artifact detection) to obtain generated images. As shown in Table 1, the source images were evaluated using the SSIM, PSNR, and LPIPS indicators compared with the images generated by the three methods. SSIM is the structural similarity index, and a larger value indicates a more similar generated image. PSNR is the peak signal-to-noise ratio, and a larger value indicates a better generation effect. LPIPS is an indicator that measures the perceptual similarity between images, and a smaller value indicates a more similar image. This verifies the superiority of the method of the present invention in the image generation scenario. Table 1 Performance comparison between the present invention and existing methods From the perspective of generation quality and similarity evaluation, the method of the present invention is superior to traditional methods (SBI, CADDM) in generating images. The method of the present invention shows significant advantages. Through the experimental results, it can be proved that the method of the present invention is highly efficient in image generation and provides a reliable solution for image generation.

[0034] Although the above describes the specific implementation methods of the invention in conjunction with the accompanying drawings, it does not limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A generalizable deep fake image detection method based on double data augmentation, characterized by: The following steps are involved: S1. Extract facial images from videos containing facial identities and preprocess them. Then, mark the preprocessed facial images as true and false, respectively, as real samples and forged samples. S2. Build a landmark diffusion generation model, input real samples into the landmark diffusion generation module, synthesize high-fidelity and difficult-to-identify new forged samples, and perform data enhancement on the new forged samples; S3. Construct three teacher models, input the real samples, forged samples and new forged samples into the three teacher models respectively, obtain the real identity semantic features, forged identity semantic features and new forged identity semantic features respectively, and guide the teacher models to perform optimization training through category loss and inter-class contrast loss; S4. Use the constructed deep weight enhancement module to perform data enhancement on the real identity semantic features to obtain real enhanced features, and use the constructed category enhancement module to perform systematic data enhancement on the forged identity semantic features and the new forged identity semantic features to obtain forged enhanced features; S5. Based on the KL divergence distillation architecture, the real enhancement features and the forged enhancement features are transferred to the student model to obtain a student forged image detector learned from the teacher model; S6. Input the facial image to be detected into the student forged image detector to extract forged features, and then input the extracted forged features into the discrimination decision module to output the judgment result.

2. The generalizable deep fake image detection method based on dual data enhancement according to claim 1, characterized in that: S1 is as follows: S1.

1. Obtain facial image: Obtain videos from an existing database containing real and fake faces, and extract facial images from videos containing face identities based on customized frame count, height, width, and frame rate; A pre-trained facial landmark model from the machine learning tool library dlib is loaded as a face detection model. The model then performs facial landmark detection on the extracted facial image. The facial region is located and cropped, and the locations of the facial landmarks are determined. The facial region contains 68 landmarks, and the locations of the facial landmarks for each facial image are represented as a 68×2 two-dimensional matrix. The facial images are aligned based on the facial landmark location information to eliminate differences in posture and angle, resulting in a standardized facial image. S1.2, facial image preprocessing: Perform controllable image perturbations on the standardized facial images, including random horizontal flipping, random angle rotation, and blurring. The perturbed images are then resized to a uniform size and normalized according to the set mean and standard deviation to obtain the preprocessed facial images. S1.3、Authenticity mark: The preprocessed facial images are marked to identify real facial images and forged facial images, and the two facial images are used as real samples and forged samples respectively.

3. The generalizable deep fake image detection method based on dual data enhancement according to claim 2, characterized in that: S2 is as follows: S2.

1. Based on the facial key point position information of the real sample, the face is divided into multiple regions with specific semantics. All semantic regions are represented as ,Semantic regions include left eye, right eye, nose, mouth and combined regions, Indicates the semantic regions, Indicates the number of semantic regions; At the same time, the corresponding text description is constructed according to the face area mask as a text description prompt; S2.

2. Construct a landmark diffusion generation module. Input the above-mentioned face region mask and text description prompt into the landmark diffusion generation module to generate new forged samples. The landmark diffusion generation module uses a conditional input image generation model as the image generation backbone, adopts a multi-scale U-Net network as the generator backbone in the conditional diffusion model, and introduces a cross-attention mechanism in the image generation stage to fuse the semantic region and text description prompt. In the process of generating new fake samples, the denoising optimizer is used to extract the data from the standard normal distribution. latent variables Sampling is performed, which is expressed as , and iterates multiple times through the multi-scale U-Net network to obtain the denoised latent variables , the denoised latent variables Input to image decoder , image decoder It is composed of a series of convolutional layers, activation functions and normalization layers, and is decoded by the image decoder. The denoised latent variables Gradually restore the image to the same resolution as the input face image, generating new forged samples that are high-resolution and difficult to identify; S2.

3. Expand new forged samples to achieve data enhancement.

4. The generalizable deep fake image detection method based on dual data enhancement according to claim 3, characterized in that S3 The details are as follows: S3.

1. Constructing the teacher model: The teacher models include the first teacher model, the second teacher model and the third teacher model; Use the face recognition model ArcFace pre-trained on the MS1MV2 dataset as the first teacher model; Use the EfficientNet-B4 deep convolutional neural network model pre-trained on the ImageNet dataset as the second and third teacher models; S3.

2. Input image samples into the teacher model: Input the real sample into the first teacher model to extract the real identity semantic features ; The forged samples and new forged samples are input into the second teacher model and the third teacher model respectively to extract the forged identity semantic features. and new forged identity semantic features ; S3.

3. Construct a loss function to optimize the teacher model training: (1) Constructing inter-class contrast loss function : Using a cosine similarity-based comparison mechanism, the real identity semantic features Semantic features of forged identities and new forged identity semantic features The similarity measurement calculation is performed as follows: ， in, represents the number of forged samples, represents the number of real samples, Represents a function that measures the similarity between features; (2) Constructing category loss function : Based on real identity semantic features , forged identity semantic feature set Any of the forged identity semantic features and new forged identity semantic features , construct the category loss function , the calculation formula is as follows: ， in, Indicates the Item forged identity semantic features, represents the number of forged identity semantic features, which is equal to the number of forged samples. Represents a binary classification label, 0 indicates that the input is a real sample, 1 indicates that the input is a fake sample, Represents the classification loss function for the binary classification task.

5. The generalizable deep fake image detection method based on dual data enhancement according to claim 4 is characterized in that: The operations in the depth weight enhancement module are as follows: Construct a deep weight enhancement module, which consists of three parts: channel division, convolution calculation, and channel splicing. The channelization part consists of a set of feature vector segmentation operations on the channel dimension; The convolution calculation part includes three convolution layers, a 3×3 convolution layer , an 11×1 convolutional layer , a 1×11 convolutional layer ; The channel splicing part consists of channel splicing operations; Real identity semantic features Input to the deep weight enhancement module to input the true identity semantic features Divide it along the channel dimension into four sub-channel areas, which are the original channel features , spatial direction information enhancement features , vertical information enhancement features and horizontal information enhancement features ; Enhance features of spatial direction information Use depth-separable convolution with a convolution kernel of 3×3 , extract multidimensional spatial features; Enhance features of spatial direction information Use vertical convolution with a convolution kernel of 11×1 , extract vertical structural information; Enhance features for horizontal information Use horizontal convolution with a convolution kernel of 1×11 , extract horizontal structural information; Combine the output results of the three convolution branches with the original channel features Perform concatenation on the channel dimension and adjust the dimension of the output feature to be the same as Consistent feature map size, to obtain real enhanced features that integrate multi-dimensional spatial structure information .

6. The generalizable deep fake image detection method based on dual data enhancement according to claim 4 is characterized in that: The operations of the category enhancement module are as follows: Build a category enhancement module, which consists of three parts: a five-element image transformation strategy pool, a random selector, and feature splicing. The random selector randomly selects an image transformation strategy for each of the two input features, implemented by the random function in the Python standard library. It then splices the feature vectors of the two enhanced input features along the spatial dimension using the channel splicing function. Forged identity semantic features and new forged identity semantic features The input is sent to the category enhancement module, and the input features are expanded and enhanced through the image transformation strategy to construct a continuous forged feature variant space, thereby obtaining the forged enhanced features. ; Among them, the image transformation strategies are clustering transformation CT, extrapolation transformation ET, hybrid transformation MT, scaling transformation ZT and noise transformation NT. One image transformation strategy is randomly selected for expansion enhancement. (1) Clustering transformation (CT): simulating different forgery sources or subclasses through feature clustering perturbations; First, calculate the forged identity semantic features and new forged identity semantic features The characteristic mean , the calculation formula is ， Then, according to the feature mean Adjustment and , the adjustment formula is as follows: ， in, is the scaling factor for random sampling between 0 and 1, represents the hyperbolic tangent function; (2) Extrapolation Transformation ET: Generate boundary variant samples by interpolation at the edge of the feature space to strengthen decision boundary learning. The specific process is as follows: ， in, represents the eigenvector guiding the extrapolation process, , represents the forged identity semantic feature vector that guides the extrapolation process, represents a new forged identity semantic feature vector that guides the extrapolation process, Represents the scaling factor for random sampling between 0 and 1; (3) Hybrid Transformation MT: Fusion of multiple forged image features to synthesize complex hybrid forged images. The specific process is as follows: ， in, , for hour, for , for hour, for ; Indicates from distributed A random number drawn from the input to control the mixing ratio of the two input samples, The value of is between 0 and 1; (4) Scaling transformation ZT: by applying the affine transformation matrix Perform affine scaling on an image to simulate the artifacts caused by resolution or size changes, using the affine transformation matrix Combined with rotation and scaling operations, the affine transformation matrix The definition is as follows: ， in, Indicates the rotation angle, represents the scaling factor; The affine transformation matrix Apply to Features The spatial coordinates of , thus obtaining the enhanced features, the specific process is as follows: ， in, represents transpose, represents the horizontal axis, Indicates the vertical coordinate; (5) Noise Transformation NT: Different types of random noise are injected into the forged image. The calculation process is as follows: ， in, is based on The generated Gaussian noise, represents the standard deviation of the noise, Represents the coefficient that controls the intensity of noise addition, which is randomly sampled between 0 and 1; in, represents the enhanced features, , represents the adjusted forged identity semantic features, Represents new forged identity semantic features, and Referential features or , .

7. The generalizable deep fake image detection method based on dual data enhancement according to claim 6, characterized in that: S5 is as follows: S5.

1. Build a student model and use the ImageNet pre-trained model EfficientB4 as the student model; S5.

2. Construct a distillation architecture by three teacher models and one student model, using a normalized exponential function The output of the student model is compared with the features of the teacher model output and Alignment is performed using KL divergence alignment. The specific process is as follows: ， in, represents the distillation loss, Referential features or , , Refers to the true identity semantic features extracted by the student model , forged identity semantic features or new semantic features of forged identities , ,when True enhanced features output by the teacher model When Ground truth identity semantic features extracted for the student model ,when Forged enhanced features output by the teacher model When Forged identity semantic features extracted for the student model and new forged identity semantic features ; Indicates the total number of input samples, Indicates the Features or , Indicates the True identity semantic features , forged identity semantic features or new semantic features of forged identities ; represents the divergence alignment operation; Represents the adjusted normalized exponential function Smoothness temperature parameter; During the training of the student model, the distillation loss Backpropagating gradients as a loss function, updating model parameters by minimizing the distillation loss The parameters of the student model are optimized to obtain a student forged image detector learned from the teacher model.

8. The generalizable deep fake image detection method based on dual data augmentation according to claim 7, characterized in that: S6 is as follows: The discriminant decision module includes two convolutional activation units, a global average pooling layer, a flattening layer, a fully connected layer and a Dropout inactivation layer. The forged features output by the student forged image detector are input into the discriminant decision module, which performs dimensionality compression and semantic mapping on the input features, and finally outputs a forged or real binary classification result and the corresponding probability of forged or real.

9. A generalizable deepfake image detection system based on dual data augmentation, comprising a module for executing the processing instructions for each step of the generalizable deepfake image detection method based on dual data augmentation as claimed in any one of claims 1 to 8, characterized in that: include: Image input module, artifact representation extraction module and discrimination decision module; The face image to be detected is input into the system through the image input module, and then passes through the artifact representation extraction module to extract the forged features. The forged features are then input into the decision discrimination module to output the classification prediction results. Among them, the artifact representation extraction module uses the pre-trained student model to extract features, and the decision discrimination module performs classification prediction based on the forged features.

Images