Deep forgery attribution and detection method based on staring guide CLIP model

Abstract

The invention relates to the technical field of deep faking attribution and detection methods, in particular to a deep faking attribution and detection method based on a gaze guide CLIP model, and the method specifically comprises the following steps: collecting face images, constructing a deep faking attribution and detection data set, enabling each face image in the data set to have a corresponding faking attribution label and a faking detection label, preprocessing images in the data set, and then dividing into a training set and a test set; constructing a zero-sample deep counterfeit attribution and detection model, inputting the preprocessed data set into the model, and performing model processing to obtain predicted image counterfeit detection category features and predicted image counterfeit attribution category features; and performing optimization training on the deep counterfeit attribution and detection model through the training set to obtain an optimized and trained model, and testing the optimized and trained model by using data in the test set. According to the invention, the source of the deep pseudo image is traced through the deep learning method, and the faked attribution and detection can be carried out more accurately.

Description

Technical Field

[0001] This invention relates to the field of deepfake attribution and detection methods, and in particular to a deepfake attribution and detection method based on the gaze-guided CLIP model. Background Technology

[0002] With the continuous development of AI-generated content, advanced generative models like deepfakes have enabled more sophisticated facial image editing, gradually blurring the lines between real and fake faces. Deepfake technology utilizes advanced generators such as Generative Adversarial Networks (GANs) and diffusion models to manipulate facial images with virtually no perceptible traces. While deepfake technology has brought benefits to filmmaking and virtual reality, it has also contributed to the increase in malicious face manipulation and unauthorized use. Therefore, the need for attribution and detection of deepfake images has become both urgent and important.

[0003] The goal of deepfake image attribution and detection is to trace the origin of deepfake images and detect their authenticity using deep learning methods. Current deepfake image attribution and detection methods generally suffer from poor generalization ability when faced with novel generative models. Furthermore, current deepfake image attribution and detection methods fail to perform fine-grained tracing or detection evaluation of unseen advanced generators.

[0004] Therefore, this invention proposes a deepfake attribution and detection method based on the gaze-guided CLIP model to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention develops a deepfake attribution and detection method based on the gaze-guided CLIP model. This invention traces the source of deepfake images through deep learning methods, which can better detect the authenticity of images.

[0006] The technical solution of this invention to solve the technical problem is a method for deepfake attribution and detection based on the gaze-guided CLIP model, comprising the following steps: S1. Collect face images and construct a deep forgery attribution and detection dataset. Each face image in the dataset has a corresponding forgery attribution label and forgery detection label. Preprocess the images in the dataset and then divide them into training set and test set. S2. Construct a zero-shot deep forgery attribution and detection model. The model includes a fine-grained text generator, a visual perception encoder, a gaze-conscious image encoder, a language refinement encoder, a detection module, and an attribution module. Input the preprocessed dataset into the model, and after processing by the model, obtain the predicted image forgery detection category features and the predicted image forgery attribution category features. S3. Optimize the deep forgery attribution and detection model using the training set, calculate the total loss function, and use the Adam optimizer to optimize the trainable parameters in the model to obtain the optimized model. Test the optimized model using data from the test set.

[0007] S1 is as follows: The width x height of each face image in the deepfake attribution and detection face dataset was uniformly adjusted to... The face images are normalized by dividing the pixel values ​​of each image by 255, and the normalized face images are then encapsulated to obtain the face image tensor. , , Representing a vector space, This indicates the number of images in each batch. This indicates that the number of channels in a face image tensor is 3; The forgery attribution label and forgery detection label corresponding to each face image are processed by the torch.tensor() function in PyTorch, an open-source deep learning framework for machine learning and deep learning, to obtain the preprocessed forgery attribution label tensor y and forgery detection label tensor z. The attribution labels for forgery are as follows: 0 represents a real image, 1 represents a DiffFace generator image, 2 represents a Diffae generator image, 3 represents a LatentDiffusion generator image, 4 represents a CollaborativeDiffusion generator image, 5 represents a DDPM generator image, 6 represents an FSLSD generator image, 7 represents a FaceSwapper generator image, 8 represents a LatentTransformer generator image, 9 represents a StyleGAN2 generator image, and 10 represents a StyleGAN3 generator image. The forgery detection labels are: 0 for a real image and 1 for a fake image.

[0008] The specific operations in the zero-shot deepfake attribution and detection model are as follows: The preprocessed forged attribution label tensor is input into the fine-grained text generator to obtain the fine-grained attribution text. ; The preprocessed face image tensor The face image is obtained by dividing it into multiple image blocks. ; The fine-grained attribution text Face image blocks and the preprocessed face image tensor Composition of face image text pairs ; The preprocessed face image tensor The visual features of the gaze are obtained by inputting them into the visual perception encoder. ; Fine-grained attribution text The input is fed into the language refining encoder to obtain refined global language features. ; Face image blocks The input is fed into a gaze-conscious image encoder to obtain general facial image forgery features. ; Gaze at visual features Common facial image forgery features After fusion, gaze perception visual manipulation features are obtained. , The features are input into the detection module and the attribution module respectively to obtain the predicted image forgery detection category features. and predict image forgery attribution category features .

[0009] Construct a visual perception encoder, which includes a gaze encoder and an appearance gaze perception module; A gaze encoder includes a gaze evaluator and a gaze adaptor; The appearance gaze perception module includes an appearance encoder and an appearance transformer block. The appearance encoder module includes several stacked convolutional layers, and the appearance transformer block is the same as the transformer block of the convolutional visual transformer CviT. Face image text image block pair Preprocessed face image tensor The input is fed into the visual perception encoder, and after passing through the frozen pre-trained gaze estimator in the gaze encoder, the output dimension is... The facial gaze features are then input into a trainable gaze adaptor to obtain global gaze manipulation features. , , Representing an image Feature dimensions, This represents the number of images in each batch, relative to the face image tensor. of The quantities are consistent; Preprocessed face image tensor The input is fed into the appearance gaze perception module, passes through the appearance encoder, and outputs a dimension of [dimensional value missing]. Local feature map of facial image appearance , The number of channels in the facial image feature map. This represents the height of the facial image's appearance feature map. Represents the width of the facial image's appearance feature map; Then, the local feature map of the face image is... Flatten along the channel and map the dimension as eigenvectors Then, with the gaze manipulation feature and learnable classes The fused feature embeddings are obtained after embedding in parallel. Feature embedding Add location information The input is an appearance transformer block, and the output dimension is... Facial appearance gaze global manipulation features , .

[0010] Fine-grained text The word segmenter obtains a word tag sequence, and the word tags in the word tag sequence are mapped to word embedding tensors through a word embedding layer. According to the word embedding tensor and the position of the automatically generated word embedding tensor Obtain fine-grained attribution text sequence vectors with location information ; Constructing a language refining encoder, which includes... A series of consecutive transformer blocks, each including an adaptive word selector, a frozen CLIP text multi-head attention module, and a frozen CLIP text feedforward neural network module. The layer preceding both the CLIP text multi-head attention module and the CLIP text feedforward neural network module is... Normalized layer, the next layer is a residual layer; Fine-grained text sequence vectors with location information The input is fed into the language refining encoder and then processed. After normalization, the normalized layer is fed into the adaptive word selector of the first consecutive transformer block to calculate refined language features. ; Refined language features The input is fed into the CLIP text multi-head attention module for global multi-head attention calculation, and then passed through a residual layer to obtain the text global semantic features of the first transformer block. ; Global semantic features of the first transformer block go through After normalization by the normalization layer, the data is input into the feedforward neural network module, and then passes through the residual layer to obtain the refined global language features of the first transformer block. ; The output of the first transformer block of the language refining encoder is used as the input of the second transformer block, and the output of the second transformer block is used as the input of the third transformer block. This process is repeated multiple times until the first transformer block is completed. The operations of each transformer block yield fine-grained language features. From fine-grained language features Take the last word Obtain global language features .

[0011] Constructing a gaze-conscious image encoder, which includes... The system consists of several consecutive transformer blocks, each including a frozen CLIP image multi-head attention module, a fine-tuned gaze injection module, and a frozen CLIP image feedforward neural network module. The layer preceding the CLIP image multi-head attention module, the fine-tuned gaze injection module, and the CLIP image feedforward neural network module is... Normalized layer, the next layer is a residual layer; Face image blocks Input is fed into the gaze-conscious image encoder and processed along the channel using Curry The reshaping function flattens the image into a sequence of two-dimensional image patches. , , express Number of patches Indicates the first A two-dimensional face image block; Calculate a sequence of two-dimensional image patches with location information The formula is expressed as follows: , in, This represents an automatically generated, learnable class tensor. Represents the mapping of latent vectors. , Indicates the first A two-dimensional face image block The mapping of latent vectors, This indicates the position of the automatically generated two-dimensional face image block sequence. This indicates element-wise addition; Two-dimensional face image block sequence The input is fed into M transformer blocks. After passing through the first transformer block, the input passes through a multi-head self-attention module in sequence, and the first two-dimensional global face feature map is output. The formula is expressed as follows: , in, This indicates the operation of the normalization layer. This indicates the operation of the multi-head self-attention module. Then With gaze manipulation characteristics The input is fed into the fine-tuned gaze injector module to perform image gaze interaction computation, and outputs the global semantic features of the image gaze in the first transformer block. ; Image gaze global semantic features go through After normalization, the input is fed into the CLIP text feedforward neural network module, and then through the residual layer to obtain the globally universal gaze manipulation features output by the first transformer block. ; The output of the first transformer block Use it as input to the second transformer block, and then use the output of the second transformer block. As input to the third transformer block, the process iterates multiple times until the operation of the Mth transformer block is completed, resulting in the Mth 2D face gaze manipulation global feature map. ; Manipulate the global feature map of the Mth two-dimensional face gaze. Learnable class tensors are extracted to obtain general gaze spoofing features of faces. .

[0012] Gaze at visual features Common facial image forgery features After fusion, gaze perception visual manipulation features are obtained. ,Will The inputs are respectively fed into an attribution module consisting of fully connected layers and a detection module consisting of fully connected layers to obtain the predicted image forgery detection category features. and predicting forged attribution category features .

[0013] Cross-modal contrast loss Functions for language global features Visual manipulation features related to gaze perception The optimization is performed, and the formula is expressed as follows: , , , , , in, This represents the loss of contrast between visual and linguistic perception. This represents the contrast loss from language to vision. Indicates the first Sample pairs of input face images Label, Indicates the first A similarity matrix from language to vision, Represents the cosine similarity function. This represents the trainable temperature parameter. Indicates the first A visual-to-language similarity matrix, Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global feature of language Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global linguistic feature; By deepfake attribution loss function Attribution category features for predicted image forgery With images Generator-level tags The optimization is performed, and the formula is expressed as follows: , in, Indicates the first One image Generator-level tags Indicates the first Each predicted image is a forged attribution category feature; Loss function for deepfake detection Predicting image forgery detection category features With images Generator-level tags The optimization is performed, and the formula is expressed as follows: , in, Indicates the first One image Generator-level tags Indicates the first Predictive image forgery detection category features.

[0014] The face images to be detected in the test set are input into the optimized and trained deep forgery attribution and detection model to obtain the predicted image forgery detection category features and the predicted forgery attribution category features. The predicted forgery attribution category features are processed by a softmax activation function for binary classification to obtain the final true / false judgment result; The predicted image forgery detection category features are transformed into a probability distribution of categories using a softmax activation function, and the class with the highest probability is selected as the predicted attribution label using an argmax operation.

[0015] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects: This invention designs a visual perception encoder that leverages inherent gaze differences to mine global forgery cue embeddings across appearance and gaze domains. By analyzing and utilizing different gaze patterns, the visual perception encoder effectively captures subtle differences in human eye annotations, thereby improving the detection accuracy of deepfake images, especially when subtle differences between forged and genuine images are difficult for deep learning models to detect. This invention also designs a gaze-aware image encoder that fuses gaze cues extracted by the gaze encoder with regular image cues, transforming features into a more stable and universal feature space for deepfake image attribution and detection. In this process, the introduction of gaze information effectively enhances the model's ability to recognize subtle and complex features in forged images. In particular, guided by gaze patterns, the model can more accurately extract key clues related to forgery from images. Furthermore, this invention constructs a language refinement encoder that automatically selects the most relevant words based on context, improving the accuracy and contextual adaptability of language representation. Through this dynamic adjustment, the language refinement encoder can better understand image-related textual information, further enhancing the fusion effect of cross-modal information. This method demonstrates stronger language processing capabilities in image forgery detection tasks.

[0016] This invention also constructs a plug-in adaptive word selector module, which can be easily integrated into any transformer-based language model. The plug-in adaptive word selector module not only greatly improves the language model's ability to process dynamic language information, but also requires very few additional parameters and computational overhead. The flexibility of this module makes it widely applicable to various existing language model systems, and it has good scalability and adaptability. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0019] Figure 2 This is the feature capture result of the method of the present invention in fake face images generated by different generators. Detailed Implementation

[0020] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0021] Example 1 like Figure 1 As shown, a deepfake attribution and detection method based on the gaze-guided CLIP model includes the following steps: S1. Collect face images and construct a deep forgery attribution and detection dataset. Each face image in the dataset has a corresponding forgery attribution label and forgery detection label. Preprocess the images in the dataset and then divide them into training set and test set. S2. Construct a zero-shot deep forgery attribution and detection model. The model includes a fine-grained text generator, a visual perception encoder, a gaze-conscious image encoder, a language refinement encoder, a detection module, and an attribution module. Input the preprocessed dataset into the model, and after processing by the model, obtain the predicted image forgery detection category features and the predicted image forgery attribution category features. S3. Optimize the deep forgery attribution and detection model using the training set, calculate the total loss function, and use the Adam optimizer to optimize the trainable parameters in the model to obtain the optimized model. Test the optimized model using data from the test set.

[0022] In a specific implementation, S1 is as follows: The width x height of each face image in the deepfake attribution and detection face dataset was uniformly adjusted to... The face images are normalized by dividing the pixel values ​​of each image by 255, and the normalized face images are then encapsulated to obtain the face image tensor. , , Representing a vector space, This indicates the number of images in each batch. This indicates that the number of channels in a face image tensor is 3; The forgery attribution label and forgery detection label corresponding to each face image are processed by the torch.tensor() function in PyTorch, an open-source deep learning framework for machine learning and deep learning, to obtain the preprocessed forgery attribution label tensor y and forgery detection label tensor z. The attribution labels for forgery are as follows: 0 represents a real image, 1 represents a DiffFace generator image, 2 represents a Diffae generator image, 3 represents a LatentDiffusion generator image, 4 represents a CollaborativeDiffusion generator image, 5 represents a DDPM generator image, 6 represents an FSLSD generator image, 7 represents a FaceSwapper generator image, 8 represents a LatentTransformer generator image, 9 represents a StyleGAN2 generator image, and 10 represents a StyleGAN3 generator image. DiffFace generator is an image generation tool based on the Stable Diffusion model. It is mainly used to generate or modify face images. Its core functions include facial feature restoration, local redrawing and style transfer. It supports adjusting the generation effect through prompts and parameters. The Diffae generator is a generative architecture that combines a diffusion model and an autoencoder, and is often used for image reconstruction and semantic editing. The LatentDiffusion generator is a model that performs the diffusion process in the latent space (rather than the pixel space), which can significantly reduce computational costs and is widely used for efficient image generation. CollaborativeDiffusion generators involve multiple diffusion models collaborating to generate content, improving generation quality by sharing information, and are often used in complex scenarios. The DDPM generator is the basic framework of the diffusion model. It generates images by progressively adding and removing noise and is widely used in high-fidelity image synthesis. The FSLSD generator focuses on image generation with a small number of samples, combining latent spatial diffusion and semantic control, and is often used for personalized image editing; FaceSwapper generator is a model specifically designed for swapping faces in images. It is often based on GAN or diffusion techniques and is suitable for entertainment applications. The LatentTransformer generator uses the Transformer architecture to manipulate sequential data in the latent space for image generation or editing; StyleGAN2 generator is a generative adversarial network (GAN) model developed by NVIDIA, which focuses on high-resolution image synthesis and fine-grained style control, and is widely used in face generation and art design; The StyleGAN3 generator, an improved version of StyleGAN2, optimizes motion blur and texture details, enhancing the generation of dynamic scenes (such as video sequences), making it suitable for film special effects and game development. The forgery detection labels are: 0 for a real image and 1 for a fake image.

[0023] In a specific implementation, the operations in the zero-sample deep forgery attribution and detection model are as follows: The preprocessed forged attribution label tensor is input into the fine-grained text generator to obtain the fine-grained attribution text. ; The preprocessed face image tensor The face image is obtained by dividing it into multiple image blocks. ; The fine-grained attribution text Face image blocks and the preprocessed face image tensor Composition of face image text pairs ; The preprocessed face image tensor The visual features of the gaze are obtained by inputting them into the visual perception encoder. ; Fine-grained attribution text The input is fed into the language refining encoder to obtain refined global language features. ; Face image blocks The input is fed into a gaze-conscious image encoder to obtain general facial image forgery features. ; Gaze at visual features Common facial image forgery features After fusion, gaze perception visual manipulation features are obtained. , The features are input into the detection module and the attribution module respectively to obtain the predicted image forgery detection category features. and predict image forgery attribution category features .

[0024] In a specific implementation, a visual perception encoder is constructed, which includes a gaze encoder and an appearance gaze perception module. A gaze encoder includes a gaze evaluator and a gaze adaptor; The appearance gaze perception module includes an appearance encoder and an appearance transformer block. The appearance encoder module includes several stacked convolutional layers, and the appearance transformer block is the same as the transformer block of the convolutional visual transformer CviT. Face image text image block pair Preprocessed face image tensor The input is fed into the visual perception encoder, and after passing through the frozen pre-trained gaze estimator in the gaze encoder, the output dimension is... The facial gaze features are then input into a trainable gaze adaptor to obtain global gaze manipulation features. , The formula is as follows: , in, Representing an image Feature dimensions, This represents the number of images in each batch, relative to the face image tensor. of The quantities are consistent. This indicates the operation of the gaze encoder GE; Preprocessed face image tensor The input is fed into the appearance gaze perception module, passes through the appearance encoder, and outputs a dimension of [dimensional value missing]. Local feature map of facial image appearance The formula is expressed as follows: , in, This represents the backbone operations of a convolutional neural network. The parameters representing the backbone of the convolutional neural network, The number of channels in the facial image feature map. This represents the height of the facial image's appearance feature map. Represents the width of the facial image's appearance feature map; Then, the local feature map of the face image is... Flatten along the channel and map the dimension as eigenvectors Then, with the gaze manipulation feature and learnable classes The fused feature embeddings are obtained after embedding in parallel. Feature embedding Add location information The input is an appearance transformer block, and the output dimension is... Facial appearance gaze global manipulation features , ; , , , in, This indicates tensor parallel operations. This indicates an element-wise addition operation. Appearance transformer block operations.

[0025] In a specific implementation, fine-grained text The word segmenter obtains a word tag sequence, and the word tags in the word tag sequence are mapped to word embedding tensors through a word embedding layer. According to the word embedding tensor and the position of the automatically generated word embedding tensor Obtain fine-grained attribution text sequence vectors with location information The formula is expressed as follows: ; Constructing a language refining encoder, which includes... A series of consecutive transformer blocks, each including an adaptive word selector, a frozen CLIP text multi-head attention module, and a frozen CLIP text feedforward neural network module. The layer preceding both the CLIP text multi-head attention module and the CLIP text feedforward neural network module is... Normalized layer, the next layer is a residual layer; Fine-grained text sequence vectors with location information The input is fed into the language refining encoder and then processed. After normalization, the normalized layer is fed into the adaptive word selector of the first consecutive transformer block to calculate refined language features. The calculation formula is as follows: , in, This represents a diagonal matrix in the adaptive word selector, where the elements of the matrix follow a standard normal distribution. Refined language features The input is fed into the CLIP text multi-head attention module for global multi-head attention calculation, and then passed through a residual layer to obtain the text global semantic features of the first transformer block. The formula is expressed as follows: , in, This indicates a normalization operation. This indicates the operation of the multi-head attention module; Global semantic features of the first transformer block go through After normalization by the normalization layer, the data is input into the feedforward neural network module, and then passes through the residual layer to obtain the refined global language features of the first transformer block. The formula is expressed as follows: , in, This indicates the operation of the feedforward neural network module; The output of the first transformer block of the language refining encoder is used as the input of the second transformer block, and the output of the second transformer block is used as the input of the third transformer block. This process is repeated multiple times until the first transformer block is completed. The operations of each transformer block yield fine-grained language features. From fine-grained language features Take the last word Obtain global language features .

[0026] In a specific implementation, a gaze-conscious image encoder is constructed, which includes... The system consists of several consecutive transformer blocks, each including a frozen CLIP image multi-head attention module, a fine-tuned gaze injection module, and a frozen CLIP image feedforward neural network module. The layer preceding the CLIP image multi-head attention module, the fine-tuned gaze injection module, and the CLIP image feedforward neural network module is... Normalized layer, the next layer is a residual layer; Face image blocks Input is fed into the gaze-conscious image encoder and processed along the channel using Curry The reshaping function flattens the image into a sequence of two-dimensional image patches. , , express Number of patches Indicates the first A two-dimensional face image block; Calculate a sequence of two-dimensional image patches with location information The formula is expressed as follows: , in, This represents an automatically generated, learnable class tensor. Represents the mapping of latent vectors. , Indicates the first A two-dimensional face image block The mapping of latent vectors, Indicates the position of the automatically generated two-dimensional face image block sequence; Two-dimensional face image block sequence The input is fed into M transformer blocks. After passing through the first transformer block, the input passes through a multi-head self-attention module in sequence, and the first two-dimensional global face feature map is output. The formula is expressed as follows: , in, This indicates the operation of the normalization layer. This indicates the operation of the multi-head self-attention module. Then With gaze manipulation characteristics The input is fed into the fine-tuned gaze injector module to perform image gaze interaction computation, and outputs the global semantic features of the image gaze in the first transformer block. The formula is expressed as follows: , , , , , in, , and These represent the query vector, key vector, and value vector, respectively. , and These represent the query matrix, key matrix, and value matrix, respectively. Representing an image Feature dimensions, This represents the weight projection matrix in the fine-tuned gaze injector module. This represents the Softmax activation function. Indicates the scaling factor. This indicates the output characteristics of gaze interaction; Image gaze global semantic features go through After normalization, the input is fed into the CLIP text feedforward neural network module, and then through the residual layer to obtain the globally universal gaze manipulation features output by the first transformer block. The formula is expressed as follows: ; The output of the first transformer block Use it as input to the second transformer block, and then use the output of the second transformer block. As input to the third transformer block, the process iterates multiple times until the operation of the Mth transformer block is completed, resulting in the Mth 2D face gaze manipulation global feature map. ; Manipulate the global feature map of the Mth two-dimensional face gaze. Learnable class tensors are extracted to obtain general gaze spoofing features of faces. .

[0027] In a specific implementation, the visual characteristics of the gaze appearance will be... Common facial image forgery features After fusion, gaze perception visual manipulation features are obtained. ,Will The inputs are respectively fed into an attribution module consisting of fully connected layers and a detection module consisting of fully connected layers to obtain the predicted image forgery detection category features. and predicting forged attribution category features The formula is expressed as follows: , , , in, The matrix representing the attribution model, The matrix represents the detection model.

[0028] In a specific implementation, cross-modal contrast loss is used. Functions for language global features Visual manipulation features related to gaze perception The optimization is performed, and the formula is expressed as follows: , , , , , in, This represents the loss of contrast between visual and linguistic perception. This represents the contrast loss from language to vision. Indicates the first Sample pairs of input face images Label, Indicates the first A similarity matrix from language to vision, Represents the cosine similarity function. This represents the trainable temperature parameter. Indicates the first A visual-to-language similarity matrix, Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global feature of language Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global linguistic feature; By deepfake attribution loss function Attribution category features for predicted image forgery With images Generator-level tags The optimization is performed, and the formula is expressed as follows: , in, Indicates the first One image Generator-level tags Indicates the first Each predicted image is a forged attribution category feature; Loss function for deepfake detection Predicting image forgery detection category features With images Generator-level tags The optimization is performed, and the formula is expressed as follows: , in, Indicates the first One image Generator-level tags Indicates the first Predictive image forgery detection category features.

[0029] In a specific implementation, the face images to be detected in the test set are input into the optimized and trained deep forgery attribution and detection model to obtain the predicted image forgery detection category features and the predicted forgery attribution category features. The predicted forgery attribution category features are processed by a softmax activation function for binary classification to obtain the final true / false judgment result; The predicted image forgery detection category features are transformed into a probability distribution of categories using a softmax activation function, and the class with the highest probability is selected as the predicted attribution label using an argmax operation.

[0030] Example 2 To demonstrate the attribution and detection performance of the method of this invention, the attribution and detection performance of the method of this invention are compared with those of different existing models. For the attribution part, the high-quality face benchmark dataset CelebA-HQ is selected as the experimental dataset. After training with the seen generator and CelebA-HQ, the accuracy (ACC) score on the unseen generator and FFHQ is obtained. For the detection part, the average of the accuracy (ACC) and the area under the curve (AUC) score on the unseen generator is used. The seen generator refers to the generator that has been encountered by the model and whose features have been learned during the model training stage. The unseen generator refers to the generator that appears during the model testing stage and has never been encountered during the training stage. In Table 1, bold and underline represent the best and second-best results, respectively. The attributed objects include: FFHQ represents a high-quality face dataset, SimSwap represents a face swapping framework based on feature decoupling, InSwapper can quickly achieve real-time face swapping in videos using a pre-trained face feature encoder, UniFace represents a unified face representation framework, e4s represents an explicit expression embedding face swapping technique based on a single image, VQGAN represents an image generation model combining vector quantization and GAN, SD-2.1 represents a text-based image diffusion model, DiT represents a diffusion model architecture that replaces UNet with Transformer, SiT represents a video frame interpolation and generation model, and REFace represents a face editing technique based on reference images. The attribution part, Average, represents the average accuracy of each model on the attribution object; The Average component represents the average accuracy and the area under the curve. Existing models include: ResNet (Residual Network), Xception (Extreme Convolution), ViT (Visual Transformer), CviT (Compact Visual Transformer), CAEL (Contrast-Supervised Reinforcement Learning), CLIP (Multimodal Model), MFCLIP (Multimodal Fusion CLIP), DNA-Net (Dynamic Kernel Adaptation Network), DE-FAKE (Deep Forgery Exposure Network), ForensicAda (Forensic Person Adaptation Network), Lin et al. (Research on Referential Face Anti-Spoofing), OmniDFA (Holographic Deep Forgery Analysis), and CDAL (Cross-Domain Adversarial Learning). Table 1. Comparison of attribution and detection performance of different depth-of-forgery attribution and detection methods After training with a seen generator and CelebA-HQ, this invention tested the model's performance on various unseen generators and FFHQ. Table 1 shows that the accuracy (ACC) of the proposed method surpasses most existing techniques, demonstrating its superior capabilities in attribution and detection. Specifically, the proposed model achieves an attribution average accuracy (ACC) of approximately 6% higher than CLIP on the unseen generator. Unlike CLIP, which only mines global image and language forgery embeddings, the proposed network focuses on general gaze-aware image forgery features and performs fine-grained visual-language alignment. Furthermore, the detection accuracy of the proposed model is generally higher than its attribution accuracy. For example, CAEL and DNA-Det have average attribution accuracies of 25.74% and 33.14%, respectively. In contrast, their detection accuracies are 81.71% and 86.99%, respectively. Therefore, attribution is generally more challenging than detection.

[0031] Example 3 like Figure 2 As shown, Figure 2 The first row of images shows forged face images generated using Grad-CAM visualization technology and various generation methods. The generated images are then analyzed in depth using the method described in this invention. The second row shows the detection results of this invention. Based on the predicted image forgery detection category features and predicted forgery attribution category features output by the model of this invention, the output features are presented in the form of a heatmap. This clearly reveals the capture of forgery features in forged face images generated by different methods using the method of this invention. The hotter (redder) the position in the image, the more forgery features the method of this invention captures. Figure 2 The experimental results shown demonstrate that the method of this invention can deeply explore and effectively capture forgery details at the far-distance gaze perception level, providing a new perspective and strong evidence for the field of forged face detection.

[0032] Figure 2 In this context, StyleGAN2 represents a style-based generative adversarial network 2, LatTrans represents a latent transformer, DDPM represents a denoising diffusion probabilistic model, Diffac represents a hybrid approach combining diffusion models and reinforcement learning, VQGAN represents an image generation model combining vector quantization and GAN, UniFace represents a unified face representation framework, InSwapper enables real-time video face swapping using a pre-trained face feature encoder, SimSwap represents a face swapping framework based on feature decoupling, e4s represents an explicit expression embedding face swapping technique based on a single image, REFace represents a face editing technique based on a reference image, SiT represents a video frame interpolation and generation model, SD-2.1 represents a text-based image diffusion model, and DiT represents a diffusion model architecture that replaces UNet with Transformer.

[0033] Although the specific embodiments of the invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. A gaze-guided CLIP model based deepfake attribution and detection method, characterized in that, Includes the following steps: S1. Collect face images and construct a deep forgery attribution and detection dataset. Each face image in the dataset has a corresponding forgery attribution label and forgery detection label. Preprocess the images in the dataset and then divide them into training set and test set. S2. Construct a zero-shot deep forgery attribution and detection model. The model includes a fine-grained text generator, a visual perception encoder, a gaze-conscious image encoder, a language refinement encoder, a detection module, and an attribution module. Input the preprocessed dataset into the model, and after processing by the model, obtain the predicted image forgery detection category features and the predicted image forgery attribution category features. S3. Optimize the deep forgery attribution and detection model using the training set, calculate the total loss function, and use the Adam optimizer to optimize the trainable parameters in the model to obtain the optimized model. Test the optimized model using data from the test set.

2. The gaze-guided CLIP model-based deepfake attribution and detection method of claim 1, wherein, S1 is as follows: The width x height of each face image in the face dataset is uniformly adjusted to , and the pixel value of each face image is divided by 255 to normalize the face image, and the normalized face image is packaged to obtain a face image tensor , , represents a vector space, represents the number of images in each batch, represents that the channel number of a face image tensor is 3. The forgery attribution label and forgery detection label corresponding to each face image are processed by the torch.tensor() function in PyTorch, an open-source deep learning framework for machine learning and deep learning, to obtain the preprocessed forgery attribution label tensor y and forgery detection label tensor z. The attribution labels for forgery are as follows: 0 represents a real image, 1 represents a DiffFace generator image, 2 represents a Diffae generator image, 3 represents a LatentDiffusion generator image, 4 represents a CollaborativeDiffusion generator image, 5 represents a DDPM generator image, 6 represents an FSLSD generator image, 7 represents a FaceSwapper generator image, 8 represents a LatentTransformer generator image, 9 represents a StyleGAN2 generator image, and 10 represents a StyleGAN3 generator image. The forgery detection labels are: 0 for a real image and 1 for a fake image.

3. The gaze-guided CLIP model-based deepfake attribution and detection method of claim 2, wherein, The specific operations in the zero-shot deepfake attribution and detection model are as follows: input the preprocessed fake attribution label tensor into the fine-grained text generator to obtain fine-grained attribution text ; the pre-processed face image tensor divided into a plurality of image blocks to obtain face image blocks ; the fine-grained attribution text , the face image block , and the pre-processed face image tensor composing the face image text pair ; a pre-processed face image tensor input into a visual perception encoder to obtain gaze appearance visual features ; Coarse-grained attribution text Input into a language distillation encoder to obtain distilled language global features ; blocks of a face image of a person input into a gaze-aware image encoder to obtain general face image forgery features ; Visual features of gaze appearance With general face image forgery features Fusion to obtain gaze perception visual manipulation features , Respectively input into the detection module and the attribution module to obtain predicted image forgery detection class features And predicted image forgery attribution class features .

4. The method for deepfake attribution and detection based on a gaze-guided CLIP model according to claim 3, characterized in that: Construct a visual perception encoder, which includes a gaze encoder and an appearance gaze perception module; A gaze encoder includes a gaze evaluator and a gaze adaptor; The appearance gaze perception module includes an appearance encoder and an appearance transformer block. The appearance encoder module includes several stacked convolutional layers, and the appearance transformer block is the same as the transformer block of the convolutional visual transformer CviT. Face image text image patch pair Face image tensor after preprocessing in the middle Input into the visual perception encoder, through the frozen pre-training gaze evaluator in the gaze encoder, output the face gaze feature with the dimension of , and then input into the trainable gaze adapter to obtain the global gaze manipulation feature , , The feature dimension of the image , The number of images in each batch is consistent with the number of face image tensors​ pre-processed face image tensor input to the appearance gaze perception module, passed through an appearance encoder, outputting a face image appearance local feature map of dimension , , denotes a number of channels of the face image appearance feature map, denotes a height of the face image appearance feature map, denotes a width of the face image appearance feature map. The face image appearance local feature map is flattened along the channel and mapped to a feature vector of dimension Then the gaze manipulation feature and the learnable class embeddings are concatenated to obtain the fused feature embedding The feature embedding is augmented with position information and input into an appearance transformer block to output a face appearance gaze global manipulation feature of dimension , .​​ 5. A method for attribution and detection of deepfakes based on a gaze-guided CLIP model according to claim 4, characterized in that: Fine-grained text After the word tokenizer, the word tokens in the word token sequence are mapped to word embedding tensors through the word embedding layer , the fine-grained attribution text sequence vector with position information is obtained according to the word embedding tensor and the position of the automatically generated word embedding tensor ;​ A language refinement encoder is constructed, the language refinement encoder comprising transformer blocks, each transformer block comprising an adaptive word selector, a frozen CLIP text multi-head attention module, and a frozen CLIP text feed-forward neural network module, each of the CLIP text multi-head attention module and the CLIP text feed-forward neural network module having a previous layer that is a normalization layer, and a next layer that is a residual layer; Fine-grained text sequence vectors with location information The input is fed into the language refining encoder and then processed. After normalization, the normalized layer is fed into the adaptive word selector of the first consecutive transformer block to calculate refined language features. ; Refined language features The input is fed into the CLIP text multi-head attention module for global multi-head attention calculation, and then passed through a residual layer to obtain the text global semantic features of the first transformer block. ; Global semantic features of the first transformer block go through After normalization by the normalization layer, the data is input into the feedforward neural network module, and then passes through the residual layer to obtain the refined global language features of the first transformer block. ; The output of the first transformer block of the language refining encoder is used as the input of the second transformer block, and the output of the second transformer block is used as the input of the third transformer block. This process is repeated multiple times until the first transformer block is completed. The operations of each transformer block yield fine-grained language features. From fine-grained language features Take the last word Obtain global language features .

6. The method for deepfake attribution and detection based on a gaze-guided CLIP model according to claim 5, characterized in that: Constructing a gaze-conscious image encoder, which includes... The system consists of several consecutive transformer blocks, each including a frozen CLIP image multi-head attention module, a fine-tuned gaze injection module, and a frozen CLIP image feedforward neural network module. The layer preceding the CLIP image multi-head attention module, the fine-tuned gaze injection module, and the CLIP image feedforward neural network module is... Normalized layer, the next layer is a residual layer; Face image blocks Input is fed into the gaze-conscious image encoder and processed along the channel using Curry The reshaping function flattens the image into a sequence of two-dimensional image patches. , , express Number of patches Indicates the first A two-dimensional face image block; Calculate a sequence of two-dimensional image patches with location information The formula is expressed as follows: ， in, This represents an automatically generated, learnable class tensor. Represents the mapping of latent vectors. , Indicates the first A two-dimensional face image block The mapping of latent vectors, This indicates the position of the automatically generated two-dimensional face image block sequence. This indicates element-wise addition; Two-dimensional face image block sequence The input is fed into M transformer blocks. After passing through the first transformer block, the input passes through a multi-head self-attention module in sequence, and the first two-dimensional global face feature map is output. ; Then With gaze manipulation characteristics The input is fed into the fine-tuned gaze injector module to perform image gaze interaction computation, and outputs the global semantic features of the image gaze in the first transformer block. ; Image gaze global semantic features go through After normalization, the input is fed into the CLIP text feedforward neural network module, and then through the residual layer to obtain the globally universal gaze manipulation features output by the first transformer block. ; The output of the first transformer block Use it as input to the second transformer block, and then use the output of the second transformer block. As input to the third transformer block, the process iterates multiple times until the operation of the Mth transformer block is completed, resulting in the Mth 2D face gaze manipulation global feature map. ; Manipulate the global feature map of the Mth two-dimensional face gaze. Learnable class tensors are extracted to obtain general gaze spoofing features of faces. .

7. A method for attribution and detection of deepfakes based on a gaze-guided CLIP model according to claim 6, characterized in that: Gaze at visual features Common facial image forgery features After fusion, gaze perception visual manipulation features are obtained. ,Will The inputs are respectively fed into an attribution module consisting of fully connected layers and a detection module consisting of fully connected layers to obtain the predicted image forgery detection category features. and predicting forged attribution category features .

8. A method for attribution and detection of deepfakes based on a gaze-guided CLIP model according to claim 7, characterized in that: Cross-modal contrast loss Functions for language global features Visual manipulation features related to gaze perception The optimization is performed, and the formula is expressed as follows: ， ， ， ， ， in, This represents the loss of contrast between visual and linguistic perception. This represents the contrast loss from language to vision. Indicates the first Sample pairs of input face images Label, Indicates the first A similarity matrix from language to vision, Represents the cosine similarity function. This represents the trainable temperature parameter. Indicates the first A visual-to-language similarity matrix, Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global feature of language Indicates the first A gaze-perception visual manipulation feature Indicates the first A concise global linguistic feature; By deepfake attribution loss function Attribution category features for predicted image forgery With images Generator-level tags The optimization is performed, and the formula is expressed as follows: ， in, Indicates the first One image Generator-level tags Indicates the first Each predicted image is a forged attribution category feature; Loss function for deepfake detection Predicting image forgery detection category features With images Generator-level tags The optimization is performed, and the formula is expressed as follows: ， in, Indicates the first One image Generator-level tags Indicates the first Predictive image forgery detection category features.

9. A method for attribution and detection of deepfakes based on a gaze-guided CLIP model according to claim 8, characterized in that: The face images to be detected in the test set are input into the optimized and trained deep forgery attribution and detection model to obtain the predicted image forgery detection category features and the predicted forgery attribution category features. The predicted forgery attribution category features are processed by a softmax activation function for binary classification to obtain the final true / false judgment result; The predicted image forgery detection category features are transformed into a probability distribution of categories using a softmax activation function, and the class with the highest probability is selected as the predicted attribution label using an argmax operation.

Images