Diffusion forged face detection and localization method based on multi-domain visual language reconstruction

Abstract

The present invention relates to the technical field of forged face detection methods, and in particular to a method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction. First, spatial information learning is performed on a diffuse forged face image to extract local and global spatial image features respectively, and the fine-grained text prompt corresponding to the image is learned to extract the global language representation, and then the image is reconstructed using the local spatial features to enhance the spatial representation and obtain the residual image, and the residual image is learned to extract the residual features, and then the image features are combined with the residual features to obtain the visual manipulation features, and the global language representation is reconstructed to strengthen the language representation, and finally the language features are combined with the visual features to learn the multimodal representation of the diffuse forged face, and in the inference stage, the diffuse forged face is used to learn the visual manipulation features to realize the detection and positioning of the diffuse forged face. This method can improve the efficiency of forged detection and positioning and the cross-modal detection capability.

Description

Technical Field

[0001] The present invention relates to the technical field of forged face detection methods, and in particular to a diffusion forged face detection and positioning method based on multi-domain visual language reconstruction. Background Art

[0002] Diffusion fake face is to use the latest deep learning technology based on diffusion model to generate extremely realistic fake face images or videos. So far, a large number of diffusion fake face videos that cannot be distinguished by human eyes have been circulated on the Internet, which greatly damages personal reputation and threatens social stability.

[0003] Current face forgery detection and localization models have made great progress in capturing face forgery patterns in spatial and frequency domains using image modalities, but other modalities such as fine-grained category-aware text have not been comprehensively studied, which limits the generalization ability of the model. In addition, current face forgery detection and localization methods tend to analyze facial images created by GAN, but have difficulty identifying and localizing facial images synthesized based on diffusion models.

[0004] Therefore, the present invention provides a method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention develops a diffusion forged face detection and positioning method based on multi-domain visual language reconstruction, the main purpose of which is to improve the forged detection and positioning efficiency and cross-modal detection capability.

[0006] The technical solution of the present invention to solve the technical problem is a method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction, and the specific steps are as follows:

[0007] S1. Constructing a hierarchical and fine-grained face dataset , The dataset contains several face images, each of which has a corresponding fine-grained class-aware text and a true mask image. The face image, the corresponding fine-grained class-aware text, and the true mask image are preprocessed to obtain the preprocessed face image, the corresponding fine-grained class-aware text tensor, and the true mask image tensor.

[0008] S2. Use the multi-domain visual encoder module to obtain local features of face images Visually manipulated features fused with global image residual ;

[0009] S3: Local image features of the face extracted by the multi-domain visual encoder module Input into the visual decoder module and the mask decoder module respectively to obtain the predicted face appearance images and the predicted mask image ;

[0010] S4, after adding the position information to the text tensor, input it into the language encoder, obtain a new text tensor based on the text tensor and add the position information, and then input it into the language decoder to obtain the fine-grained class language features, fine-grained source language features and predicted target language features of the text;

[0011] S5, the residual visual manipulation feature of the global face image obtained in step S2 Input into the adapter and multi-layer perceptron respectively to obtain the predicted language features and predicted image category features ;

[0012] S6: The predicted face appearance image obtained in step S3 is Tensor with face images Using appearance reconstruction loss function Optimize and transform the predicted mask image obtained in step S3 Tensor with the true mask image Using manipulation of the localization loss function Optimize and convert the fine-grained language features obtained in step S4 into Visually manipulated features fused with the global image residual obtained in step S2 and the predicted language features obtained in step S5 Use cross-modal contrast loss respectively Functions and Divergence loss function Optimize and convert the predicted image category features obtained by S5 Using manipulation detection loss function with real image labels Optimize and transform the predicted label vector of the text The true labels of the words in the text By optimizing the language reconstruction loss function, the adapter and multi-layer perceptron are trained to obtain a multi-domain fine-grained visual language reconstruction model.

[0013] S7. Use the multi-domain visual encoder, visual decoder and multi-layer perceptron in the trained model to perform binary classification detection on the face image, and generate a predicted mask image to locate the manipulation.

[0014] S1 is as follows:

[0015] S1.1. Adjust the width × height of each face image in the hierarchical fine-grained face dataset , the face image is normalized using the dimension with a mean of [0.4718, 0.3467, 0.3154] and a variance of [0.1656, 0.1432, 0.1364]. The three values ​​in the two sets of data correspond to the mean and variance of the red, green, and blue channels in the RGB channels of the face image, respectively. The normalized face image is encapsulated as Tensor , represents a vector space, Represents a tensor of any face image in the hierarchical fine-grained face dataset. 3 indicates that the number of channels of the face image tensor is 3. The label corresponding to the face image is recorded as or , Indicates the number of images in each batch, 0 represents true images and 1 represents false images;

[0016] S1.2. The fine-grained class-aware text corresponding to the face image is processed by the word segmenter to obtain a word tag sequence, and the word embedding layer is used to map the word tags into a word embedding tensor. , Indicates the number of texts in each batch, Indicates the number of word tags, Represents the feature dimension of word embedding;

[0017] S1.3. Encapsulate the real mask image as Tensor , Indicates the number of images in each batch.

[0018] S2 is as follows:

[0019] S2.1. Construct a multi-domain visual encoder module. The multi-domain visual encoder module includes a The encoder and an image encoder, where the image The encoder is A continuous Module composition, each The module consists of a multi-head self-attention module and Module composition, multi-head self-attention module and The upper layer of the module is layer, the next layer is the residual layer;

[0020] S2.2, the tensor of the face image Input to the multi-domain visual encoder module, first through Encoder, the output dimension is Local feature map of face image , the specific process is as follows:

[0021] ,

[0022] in, , express The parameters of the encoder, Represents the number of channels of the local feature map of the face image, Represents the height of the local feature map of the face image, Indicates the width of the local feature map of the face image;

[0023] S2.3, local feature map of face image Utilize along the channel Curry's The reshape function flattens the image into a sequence of two-dimensional blocks. , , express The number of patches, , Indicates A two-dimensional image block, Indicates the size of a two-dimensional image block;

[0024] Calculate a sequence of 2D image blocks with position information , the specific calculation is as follows:

[0025] ,

[0026] in, represents the automatically generated learnable class tensor, represents the mapping latent vector, , Represents the dimension of the mapping hidden vector, dimension Equal to dimension , , Indicates 2D image patch The mapping hidden vector of represents the position of the automatically generated two-dimensional image block sequence, ;

[0027] S2.4. Input to Image In the encoder, First go through the first layer Module, on the first level The module goes through the multi-head self-attention module and Module, last layer Module Output , Represents the two-dimensional spatial feature map of the first layer. The specific process is as follows:

[0028] First, after normalization, it is input into the first-layer multi-head self-attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the two-dimensional global spatial features. ,Will After normalization, it is input to the first layer module, and then after the residual layer, we finally get the first layer Module Output , the calculation process is as follows:

[0029] ,

[0030] ,

[0031] in, represents the operation of the normalization layer, represents the operation of the multi-head self-attention module, express Operation of the module;

[0032] Then the first layer The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , Indicates 2D spatial feature map of the layer;

[0033] S2.5. Input the appearance decoder module to get the predicted face appearance image. The calculation process is as follows:

[0034] ,

[0035] in, represents the predicted face appearance image, , Represents the parameters of the appearance decoder AD;

[0036] After the residual image is obtained through the residual generation module, the calculation process is as follows:

[0037] ,

[0038] in, represents the residual image, represents the input face image, ;

[0039] S2.6, establish a residual encoder with the same structure and shared parameters as the image encoder, and convert the residual image Input the residual encoder to obtain the global residual image features. The calculation process is as follows:

[0040] ,

[0041] in, represents the global residual image feature, , represents the parameters of the residual encoder;

[0042] Will and To merge and then pass The visual manipulation features of the global image residual fusion are obtained by operation , the specific process is as follows:

[0043] ,

[0044] in, , Represents a reshape operation.

[0045] S3 is as follows:

[0046] S3.1. Construct a visual decoder module, which includes a UNet decoder and an appearance reconstruction layer, and the appearance reconstruction layer includes a convolutional upsampling layer; construct a mask decoder module, which includes a UNet decoder and a manipulation positioning layer, and the manipulation positioning layer is composed of a convolutional upsampling layer. The UNet decoders in the visual decoder and the mask decoder share weights;

[0047] S3.2. Input the visual decoder module to get the predicted face appearance image , ,Will Input the mask decoder module to get the predicted mask image , the calculation process is as follows:

[0048] ,

[0049] in, , is the number of detection categories, Represents the operation of the mask decoder module.

[0050] S4 is as follows:

[0051] S4.1. Based on word embedding tensor and the position of the automatically generated word embedding tensor Get fine-grained class-aware text sequence vector with position information , the calculation process is as follows:

[0052] ;

[0053] S4.2. Construct a language encoder, the language encoder includes A continuous Module, each The modules include multi-head attention module, visual injection module and Modules, including multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer;

[0054] S4.3. Fine-grained class-aware text sequence vector with position information Input to the speech encoder, First, after normalization, it is input into the first-layer multi-head attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the global semantic features of the text. , the calculation process is as follows:

[0055] ,

[0056] in, represents the normalization operation, Represents the operation of the multi-head attention module;

[0057] Then Visually manipulated features fused with global image residual after normalization Input to the visual attention module, and then pass through the residual layer to obtain the global visual language features through cross attention calculation. , the specific process is as follows:

[0058] Will Input to the query linear layer to get the query tensor , the calculation process is as follows:

[0059] ,

[0060] in, , Represents the query linear layer parameters in the first layer multi-head attention module;

[0061] Visually manipulated features fused with global image residuals Input to the key mapping layer to get the key tensor , the calculation process is as follows:

[0062] ,

[0063] in, , Represents the key mapping layer parameters in the first layer of multi-head attention module;

[0064] Then Input to the value mapping layer to get the value tensor , the calculation process is as follows:

[0065] ,

[0066] in, , Represents the parameters of the median mapping layer of the first-layer multi-head attention module;

[0067] Then the global visual language features are obtained through cross attention calculation , the calculation process is as follows:

[0068] ,

[0069] in, express The operation of the function, ;

[0070] Then After layer normalization, input to In the module, the refined global visual language fusion features are obtained through the residual module , the calculation process is as follows:

[0071] ;

[0072] Then the first layer of the language encoder The output of the module is used as the first layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated several times until the layer Module Output , the final output of the language encoder That is, the fine-grained source language features. Take the last one Text fragments get fine-grained language features ;

[0073] S4.4. In the word embedding tensor Attach a and remove the last Get the new word embedding tensor ,pass and Location Get fine-grained class-aware target text sequence vector with position information , the calculation process is as follows:

[0074] ;

[0075] S4.5. Construct a language decoder. The language decoder consists of A continuous Module composition, each The modules include mask multi-head attention module, multi-head attention module, visual injection module and Modules, mask multi-head attention module, multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer;

[0076] S4.6. Fine-grained class-aware target text sequence vector with position information Input to the language decoder, First, it is normalized and input into the first-layer mask multi-head attention module, and then passes through the residual layer to obtain the semantic features of the target text. , the calculation process is as follows:

[0077] ,

[0078] in, represents the normalization operation, Represents the operation of the masked multi-head attention module;

[0079] Then After normalization, They are input into the multi-head attention module for cross attention calculation, and then pass through the residual layer to obtain the source and target fusion semantic features. , the calculation process is as follows:

[0080] ,

[0081] in, Represents the operation of the multi-head attention module;

[0082] Will Visually manipulated features fused with global image residual after normalization Input to the visual injection module, and then pass through the residual layer to obtain the global visual language fusion feature , the calculation process is as follows:

[0083] ,

[0084] in, Represents the operation of the visual injection module;

[0085] Will After normalization, input to module, and then pass through the residual layer to obtain the refined global visual language fusion features , the calculation process is as follows:

[0086] ,

[0087] in, express Operation of the module;

[0088] Then the first layer of the language decoder The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , the final output of the language decoder That is the predicted target language feature.

[0089] S5 is as follows:

[0090] Will Input to an adapter consisting of a fully connected layer to obtain the predicted language features , the calculation process is as follows:

[0091] ,

[0092] in, , represents the adapter parameters, ;

[0093] Then Input into a multi-layer perceptron composed of fully connected layers to obtain the predicted image category features , the calculation process is as follows:

[0094] ,

[0095] in, , represents the multi-layer perceptron parameters, .

[0096] S6 is as follows:

[0097] S6.1. Total loss function:

[0098] ,

[0099] ;

[0100] ,

[0101] ;

[0102] ,

[0103] ,

[0104] ,

[0105] ,

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] in, represents transpose, represents the appearance reconstruction loss function, represents the predicted face appearance image, Indicates the number of images The index of represents the manipulation localization loss function, represents the predicted mask image, A tensor representing the true mask image, represents the cross-modal contrast loss function, Represents fine-grained language features, Indicates The image visual features and The similarity score of the language features of the image text, Indicates The image text language features and The similarity score of the visual features of the images, represents the contrastive loss function from vision to language, represents the contrast loss function from language to vision, Indicates the number of images The index of Represents the one-hot label of the sample. represents the cosine similarity function, represents a trainable temperature parameter, express Divergence loss function, represents the language reconstruction loss function, Indicates the number of word tags The index of Represents the predicted label vector of the text, Indicates The image text The predicted label vector of each word, Indicates The image text The true one-hot label of each word, Indicates The true one-hot labels of the images, Indicates images to predict image category features, represents the vocabulary, , Indicates the number of words, represents the manipulation detection loss function, Represents a real image t tag;

[0111] S6.2. Optimize the total loss function using the Adam optimizer , complete the training of the adapter and the multi-layer perceptron, and obtain a multi-domain fine-grained visual language reconstruction model.

[0112] S7 is as follows:

[0113] Image Visual encoders as input to multi-domain fine-grained visual-language reconstruction models Get local image features respectively , and then the residual image is calculated by the visual decoder , the calculation process is as follows:

[0114] ,

[0115] Then and Input to the vision encoder Get visual manipulation features , the final predicted image true or false category is calculated by multi-layer perceptron , the calculation process is as follows:

[0116] ,

[0117] Finally, the final predicted mask image is obtained by calculation , the calculation process is as follows:

[0118] ,

[0119] in, represents the visual encoder, represents the visual decoder, Represents a multi-layer perceptron.

[0120] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0121] (1) Efficient forgery detection and localization: The present invention can improve the detection accuracy of forged face images generated by the diffusion model, especially in terms of fine-grained recognition and localization. By integrating multimodal information, the present invention can effectively improve the generalization ability of the model, so that it can not only handle forged faces generated by traditional GAN, but also cope with images generated by the diffusion model.

[0122] (2) Cross-modal detection capability: By combining images, texts and other modalities, the present invention can utilize multi-dimensional contextual information to improve the accuracy of detection and positioning. This cross-modal detection method helps to overcome the limitations of relying on a single modality.

[0123] (3) Improved Diffusion Model Analysis: Different from existing methods, the present invention can analyze and identify the unique generation process and features of forged images generated by the diffusion model, thereby improving the detection capability of such images. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0125] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0126] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.

[0127] Example 1

[0128] like Figure 1As shown in FIG. 1 , a method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction is described. The specific steps are as follows:

[0129] S1. Constructing a hierarchical and fine-grained face dataset , The dataset contains several face images, each of which has a corresponding fine-grained class-aware text and a true mask image. Each face image has a corresponding fine-grained image label. Based on the label, the corresponding fine-grained class-aware text is generated. The face image, the corresponding fine-grained class-aware text and the true mask image are preprocessed respectively to obtain the preprocessed face image, the corresponding fine-grained class-aware text tensor and the true mask image tensor.

[0130] S2. Use the multi-domain visual encoder module to obtain local features of face images Visually manipulated features fused with global image residual ;

[0131] S3: Local image features of the face extracted by the multi-domain visual encoder module Input into the visual decoder module and the mask decoder module respectively to obtain the predicted face appearance images and the predicted mask image ;

[0132] S4, after adding the position information to the text tensor, input it into the language encoder, obtain a new text tensor based on the text tensor and add the position information, and then input it into the language decoder to obtain the fine-grained class language features, fine-grained source language features and predicted target language features of the text;

[0133] S5, the residual visual manipulation feature of the global face image obtained in step S2 Input into the adapter and multi-layer perceptron respectively to obtain the predicted language features and predicted image category features ;

[0134] S6: The predicted face appearance image obtained in step S3 is Tensor with face images Using appearance reconstruction loss function Optimize and transform the predicted mask image obtained in step S3 Tensor with the true mask image Using manipulation of the localization loss function Optimize and convert the fine-grained language features obtained in step S4 into Visually manipulated features fused with the global image residual obtained in step S2 and the predicted language features obtained in step S5 Use cross-modal contrast loss respectively Functions and Divergence loss function Optimize and convert the predicted image category features obtained by S5 Using manipulation detection loss function with real image labels Optimize and transform the predicted label vector of the text The true labels of the words in the text By optimizing the language reconstruction loss function, the adapter and multi-layer perceptron are trained to obtain a multi-domain fine-grained visual language reconstruction model.

[0135] S7. Use the multi-domain visual encoder, visual decoder and multi-layer perceptron in the trained model to perform binary classification detection on the face image, and generate a predicted mask image to locate the manipulation.

[0136] S1 is as follows:

[0137] S1.1. Adjust the width × height of each face image in the hierarchical fine-grained face dataset , the face image is normalized using the dimension with a mean of [0.4718, 0.3467, 0.3154] and a variance of [0.1656, 0.1432, 0.1364]. The three values ​​in the two sets of data correspond to the mean and variance of the red, green, and blue channels in the RGB channels of the face image, respectively. The normalized face image is encapsulated as Tensor , represents a vector space, Represents a tensor of any face image in the hierarchical fine-grained face dataset. 3 indicates that the number of channels of the face image tensor is 3. The label corresponding to the face image is recorded as or , Indicates the number of images in each batch, 0 represents true images and 1 represents false images;

[0138] S1.2. The fine-grained class-aware text corresponding to the face image is processed by the word segmenter to obtain a word tag sequence, and the word embedding layer is used to map the word tags into a word embedding tensor. , Indicates the number of texts in each batch, Indicates the number of word tags, Represents the feature dimension of word embedding;

[0139] S1.3. Encapsulate the real mask image as Tensor , Indicates the number of images in each batch.

[0140] S2 is as follows:

[0141] S2.1. Construct a multi-domain visual encoder module. The multi-domain visual encoder module includes a The encoder and an image encoder, where the image The encoder is A continuous Module composition, each The module consists of a multi-head self-attention module and Module composition, multi-head self-attention module and The upper layer of the module is layer, the next layer is the residual layer;

[0142] S2.2, the tensor of the face image Input to the multi-domain visual encoder module, first through Encoder, the output dimension is Local feature map of face image , the specific process is as follows:

[0143] ,

[0144] in, , express The parameters of the encoder, Represents the number of channels of the local feature map of the face image, Represents the height of the local feature map of the face image, Indicates the width of the local feature map of the face image;

[0145] S2.3, local feature map of face image Utilize along the channel Curry's The reshape function flattens the image into a sequence of two-dimensional blocks. , , express The number of patches, , Indicates A two-dimensional image block, Indicates the size of a two-dimensional image block;

[0146] Calculate a sequence of 2D image blocks with position information , the specific calculation is as follows:

[0147] ,

[0148] in, represents the automatically generated learnable class tensor, represents the mapping latent vector, , Represents the dimension of the mapping hidden vector, dimension Equal to dimension , , Indicates 2D image patch The mapping hidden vector of represents the position of the automatically generated two-dimensional image block sequence, ;

[0149] S2.4. Input to Image In the encoder, First go through the first layer Module, on the first level The module goes through the multi-head self-attention module and Module, last layer Module Output , Represents the two-dimensional spatial feature map of the first layer. The specific process is as follows:

[0150] First, after normalization, it is input into the first-layer multi-head self-attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the two-dimensional global spatial features. ,Will After normalization, it is input to the first layer module, and then after the residual layer, we finally get the first layer Module Output , the calculation process is as follows:

[0151] ,

[0152] ,

[0153] in, represents the operation of the normalization layer, represents the operation of the multi-head self-attention module, express Operation of the module;

[0154] Then the first layer The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , Indicates 2D spatial feature map of the layer;

[0155] S2.5. Input the appearance decoder module to get the predicted face appearance image. The calculation process is as follows:

[0156] ,

[0157] in, represents the predicted face appearance image, , Represents the parameters of the appearance decoder AD;

[0158] After the residual image is obtained through the residual generation module, the calculation process is as follows:

[0159] ,

[0160] in, represents the residual image, represents the input face image, ;

[0161] S2.6, establish a residual encoder with the same structure and shared parameters as the image encoder, and convert the residual image Input the residual encoder to obtain the global residual image features. The calculation process is as follows:

[0162] ,

[0163] in, represents the global residual image feature, , represents the parameters of the residual encoder;

[0164] Will and To merge and then pass The visual manipulation features of the global image residual fusion are obtained by operation , the specific process is as follows:

[0165] ,

[0166] in, , Represents a reshape operation.

[0167] S3 is as follows:

[0168] S3.1. Construct a visual decoder module, which includes a UNet decoder and an appearance reconstruction layer, and the appearance reconstruction layer includes a convolutional upsampling layer; construct a mask decoder module, which includes a UNet decoder and a manipulation positioning layer, and the manipulation positioning layer is composed of a convolutional upsampling layer. The UNet decoders in the visual decoder and the mask decoder share weights;

[0169] S3.2. Input the visual decoder module to get the predicted face appearance image , ,Will Input the mask decoder module to get the predicted mask image , the calculation process is as follows:

[0170] ,

[0171] in, , is the number of detection categories, Represents the operation of the mask decoder module.

[0172] S4 is as follows:

[0173] S4.1. Based on word embedding tensor and the position of the automatically generated word embedding tensor Get fine-grained class-aware text sequence vector with position information , the calculation process is as follows:

[0174] ;

[0175] S4.2. Construct a language encoder, the language encoder includes A continuous Module, each The modules include multi-head attention module, visual injection module and Modules, including multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer;

[0176] S4.3. Fine-grained class-aware text sequence vector with position information Input to the speech encoder, First, after normalization, it is input into the first-layer multi-head attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the global semantic features of the text. , the calculation process is as follows:

[0177] ,

[0178] in, represents the normalization operation, Represents the operation of the multi-head attention module;

[0179] Then Visually manipulated features fused with global image residual after normalization Input to the visual attention module, and then pass through the residual layer to obtain the global visual language features through cross attention calculation. , the specific process is as follows:

[0180] Will Input to the query linear layer to get the query tensor , the calculation process is as follows:

[0181] ,

[0182] in, , Represents the query linear layer parameters in the first layer multi-head attention module;

[0183] Visually manipulated features fused with global image residuals Input to the key mapping layer to get the key tensor , the calculation process is as follows:

[0184] ,

[0185] in, , Represents the key mapping layer parameters in the first layer of multi-head attention module;

[0186] Then Input to the value mapping layer to get the value tensor , the calculation process is as follows:

[0187] ,

[0188] in, , Represents the parameters of the median mapping layer of the first-layer multi-head attention module;

[0189] Then the global visual language features are obtained through cross attention calculation , the calculation process is as follows:

[0190] ,

[0191] in, express The operation of the function, ;

[0192] Then After layer normalization, input to In the module, the refined global visual language fusion features are obtained through the residual module , the calculation process is as follows:

[0193] ;

[0194] Then the first layer of the language encoder The output of the module is used as the first layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated several times until the layer Module Output , the final output of the language encoder That is, the fine-grained source language features. Take the last one Text fragments get fine-grained language features ;

[0195] S4.4. In the word embedding tensor Attach a and remove the last Get the new word embedding tensor ,pass and Location Get fine-grained class-aware target text sequence vector with position information , the calculation process is as follows:

[0196] ;

[0197] S4.5. Construct a language decoder. The language decoder consists of A continuous Module composition, each The modules include mask multi-head attention module, multi-head attention module, visual injection module and Modules, mask multi-head attention module, multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer;

[0198] S4.6. Fine-grained class-aware target text sequence vector with position information Input to the language decoder, First, it is normalized and input into the first-layer mask multi-head attention module, and then passes through the residual layer to obtain the semantic features of the target text. , the calculation process is as follows:

[0199] ,

[0200] in, represents the normalization operation, Represents the operation of the masked multi-head attention module;

[0201] Then After normalization, They are input into the multi-head attention module for cross attention calculation, and then pass through the residual layer to obtain the source and target fusion semantic features. , the calculation process is as follows:

[0202] ,

[0203] in, Represents the operation of the multi-head attention module;

[0204] Will Visually manipulated features fused with global image residual after normalization Input to the visual injection module, and then pass through the residual layer to obtain the global visual language fusion feature , the calculation process is as follows:

[0205] ,

[0206] in, Represents the operation of the visual injection module;

[0207] Will After normalization, input to module, and then pass through the residual layer to obtain the refined global visual language fusion features , the calculation process is as follows:

[0208] ,

[0209] in, express Operation of the module;

[0210] Then the first layer of the language decoder The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , the final output of the language decoder That is the predicted target language feature.

[0211] S5 is as follows:

[0212] Will Input to an adapter consisting of a fully connected layer to obtain the predicted language features , the calculation process is as follows:

[0213] ,

[0214] in, , represents the adapter parameters, ;

[0215] Then Input into a multi-layer perceptron composed of fully connected layers to obtain the predicted image category features , the calculation process is as follows:

[0216] ,

[0217] in, , represents the multi-layer perceptron parameters, .

[0218] S6 is as follows:

[0219] S6.1. Total loss function:

[0220] ,

[0221] ;

[0222] ,

[0223] ;

[0224] ,

[0225] ,

[0226] ,

[0227] ,

[0228] ;

[0229] ;

[0230] ;

[0231] ;

[0232] in, represents transpose, represents the appearance reconstruction loss function, represents the predicted face appearance image, Indicates the number of images The index of represents the manipulation localization loss function, represents the predicted mask image, A tensor representing the true mask image, represents the cross-modal contrast loss function, Represents fine-grained language features, Indicates The image visual features and The similarity score of the language features of the image text, Indicates The image text language features and The similarity score of the visual features of the images, represents the contrastive loss function from vision to language, represents the contrast loss function from language to vision, Indicates the number of images The index of Represents the one-hot label of the sample. represents the cosine similarity function, represents a trainable temperature parameter, express Divergence loss function, represents the language reconstruction loss function, Indicates the number of word tags The index of Represents the predicted label vector of the text, Indicates The image text The predicted label vector of each word, Indicates The image text The true one-hot label of each word, Indicates The true one-hot labels of the images, Indicates images to predict image category features, represents the vocabulary, , Indicates the number of words, represents the manipulation detection loss function, Represents a real image t tag;

[0233] S6.2. Optimize the total loss function using the Adam optimizer , complete the training of the adapter and the multi-layer perceptron, and obtain a multi-domain fine-grained visual language reconstruction model.

[0234] S7 is as follows:

[0235] Image Visual encoders as input to multi-domain fine-grained visual-language reconstruction models Get local image features respectively , and then the residual image is calculated by the visual decoder , the calculation process is as follows:

[0236] ,

[0237] Then and Input to the vision encoder Get visual manipulation features , the final predicted image true or false category is calculated by multi-layer perceptron , the calculation process is as follows:

[0238] ,

[0239] Finally, the final predicted mask image is obtained by calculation , the calculation process is as follows:

[0240] ,

[0241] in, represents the visual encoder, represents the visual decoder, Represents a multi-layer perceptron.

[0242] Example 2

[0243] Taking this case background as an example, the method of the present invention is used as follows:

[0244] Case background: On a well-known social media platform, the account of a public figure A was suddenly flooded with a series of highly realistic fake face videos. The content of these videos showed A's false remarks and behaviors, which quickly caused an uproar on the Internet and caused serious damage to A's personal reputation. At the same time, it also triggered widespread public concern about the authenticity of the information. After technical identification, these videos were produced using the latest diffusion fake face technology, and it was difficult to distinguish the authenticity with the naked eye.

[0245] Application process and situation:

[0246] (1) Initial detection stage: Social media platforms first used existing image-based face forgery detection models to conduct preliminary screening of these videos. However, since these models are mainly optimized for GAN-generated images, their recognition accuracy is greatly reduced when faced with content generated by diffusion forgery technology, and they can only identify some relatively obvious forgery traces.

[0247] (2) Introducing multimodal detection technology: In view of the shortcomings of the initial detection, the platform decided to introduce an advanced face forgery detection and localization model that combines the image modality and the fine-grained category-aware text modality. This model can not only capture the forgery patterns in the image in the spatial and residual domains, but also enhance the detection capability through the fine-grained text modality;

[0248] (3) Fine-grained analysis and localization: With the help of multimodal detection technology, the system successfully identified and located multiple diffuse forged face regions in the video;

[0249] (4) Verification and feedback: The platform will submit the test results to a professional third-party verification agency for review to ensure accuracy. Once the video is confirmed to be fake, it will be immediately removed from the shelves, and a warning will be issued to users. At the same time, an investigation procedure will be initiated against the rumor maker;

[0250] (5) Results: Through this process, not only were all the spread fake face videos involving public figure A successfully identified and removed, but the platform also established a more complete and multi-modal face fake detection mechanism, effectively improving the authenticity and credibility of the platform content.

[0251] After this incident, social media platforms strengthened their cooperation with scientific research institutions and continued to optimize detection technology, especially in the identification capabilities of new counterfeiting technologies (such as diffusion models). More importantly, this incident aroused widespread public concern about the authenticity of information and promoted consensus among all sectors of society on combating online rumors and protecting personal privacy and data security.

[0252] This embodiment demonstrates that in practical applications, combining multimodal detection technology is of great significance in dealing with complex and changeable face forgery problems, and is an effective means to maintain the health of cyberspace and promote the authentic dissemination of information.

[0253] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction, characterized in that: The following steps are involved: S1. Constructing a hierarchical and fine-grained face dataset , The dataset contains several face images, each of which has a corresponding fine-grained class-aware text and a true mask image. The face image, the corresponding fine-grained class-aware text, and the true mask image are preprocessed to obtain the preprocessed face image, the corresponding fine-grained class-aware text tensor, and the true mask image tensor. S2, use the multi-domain visual encoder module to obtain the local features of the face image Visual manipulation features ; S3: Local image features of the face extracted by the multi-domain visual encoder module Input into the visual decoder module and the mask decoder module respectively to obtain the predicted face appearance images and the predicted mask image ; S4. After adding the position information to the fine-grained class-aware text tensor, it is input into the language encoder. Based on the fine-grained class-aware text tensor, a new fine-grained class-aware text tensor is obtained and the position information is taken into account, and then input into the language decoder to obtain the fine-grained class language features, fine-grained source language features and predicted target language features of the text. , from fine-grained source language features Take the last one Text fragments get fine-grained language features ; S5, the visual manipulation features obtained in step S2 Input into the adapter and multi-layer perceptron respectively to obtain the predicted language features and predicted image category features ; S6: The predicted face appearance image obtained in step S3 is Tensor with face images Using appearance reconstruction loss function Optimize and transform the predicted mask image obtained in step S3 Tensor with the true mask image Using manipulation of the localization loss function Optimize and convert the fine-grained language features obtained in step S4 into The visual manipulation features obtained in step S2 and the predicted language features obtained in step S5 Use cross-modal contrast loss respectively Functions and Divergence loss function Optimize and convert the predicted image category features obtained by S5 Using manipulation detection loss function with real image labels Optimize and transform the predicted label vector of the text The true labels of the words in the text By optimizing the language reconstruction loss function, the adapter and multi-layer perceptron are trained to obtain a multi-domain fine-grained visual language reconstruction model. Among them, the predicted label vector of the text The calculation formula is as follows: ， in, represents the vocabulary, , Indicates the number of words, Represents the dimension; S7. Use the multi-domain visual encoder, visual decoder and multi-layer perceptron in the trained model to perform binary classification detection on the face image, and generate a predicted mask image to locate the manipulation.

2. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 1 is characterized in that: S1 is as follows: S1.

1. Adjust the width × height of each face image in the hierarchical fine-grained face dataset , the face image is normalized using the dimension with a mean of [0.4718, 0.3467, 0.3154] and a variance of [0.1656, 0.1432, 0.1364]. The three values ​​in the two sets of data correspond to the mean and variance of the red, green, and blue channels in the RGB channels of the face image, respectively. The normalized face image is encapsulated as Tensor , represents a vector space, Represents a tensor of any face image in the hierarchical fine-grained face dataset. 3 indicates that the number of channels of the face image tensor is 3. The label corresponding to the face image is recorded as or , Indicates the number of images in each batch, 0 represents true images and 1 represents false images; S1.

2. The fine-grained class-aware text corresponding to the face image is processed by the word segmenter to obtain a word tag sequence, and the word embedding layer is used to map the word tags into a fine-grained class-aware text tensor. , Indicates the number of texts in each batch, Indicates the number of word tags, Represents the feature dimension of word embedding; S1.

3. Encapsulate the real mask image as Tensor , Indicates the number of images in each batch.

3. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 2 is characterized in that: S2 is as follows: S2.

1. Construct a multi-domain visual encoder module. The multi-domain visual encoder module includes a The encoder and an image encoder, where the image The encoder is A continuous Module composition, each The module consists of a multi-head self-attention module and Module composition, multi-head self-attention module and The upper layer of the module is layer, the next layer is the residual layer; S2.2, the tensor of the face image Input to the multi-domain visual encoder module, first through Encoder, the output dimension is Local feature map of face image , the specific process is as follows: ， in, , express The parameters of the encoder, Represents the number of channels of the local feature map of the face image, Represents the height of the local feature map of the face image, Indicates the width of the local feature map of the face image; S2.3, local feature map of face image Utilize along the channel Curry's The reshape function flattens the image into a sequence of two-dimensional blocks. , , express The number of patches, , Indicates A two-dimensional image block, Indicates the size of a two-dimensional image block; Calculate a sequence of 2D image blocks with position information , the specific calculation is as follows: ， in, represents the automatically generated learnable class tensor, represents the mapping latent vector, , Represents the dimension of the mapping hidden vector, dimension Equal to dimension , , Indicates 2D image patch The mapping hidden vector of represents the position of the automatically generated two-dimensional image block sequence, ; S2.

4. Input to Image In the encoder, First go through the first layer Module, on the first level The module goes through the multi-head self-attention module and Module, last layer Module Output , Represents the two-dimensional spatial feature map of the first layer. The specific process is as follows: First, after normalization, it is input into the first-layer multi-head self-attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the two-dimensional global spatial features. ,Will After normalization, it is input to the first layer module, and then after the residual layer, we finally get the first layer Module Output , the calculation process is as follows: ， ， in, represents the operation of the normalization layer, represents the operation of the multi-head self-attention module, express Operation of the module; Then the first layer The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , Indicates 2D spatial feature map of the layer; S2.

5. Input the visual decoder module to get the predicted face appearance image. The calculation process is as follows: ， in, represents the predicted face appearance image, , represents the operation of the visual decoder module, Represents the parameters of the visual decoder module; Then the residual image is obtained through the residual generation module, and the calculation process is as follows: ， in, represents the residual image, represents a tensor, ; S2.

6. Establishment and Image The residual encoder with the same encoder structure and shared parameters converts the residual image Input the residual encoder to obtain the global residual image features. The calculation process is as follows: ， in, represents the global residual image feature, , Residual encoder Parameters; Will and To merge and then pass Operation to obtain visual manipulation features , the specific process is as follows: , in, , Represents a reshape operation.

4. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 3, wherein S3 The details are as follows: S3.

1. Build a visual decoder module. The visual decoder module includes Decoder and appearance reconstruction layer, the appearance reconstruction layer includes a convolutional upsampling layer; build a mask decoder module, the mask decoder module includes The decoder and manipulation positioning layer, the manipulation positioning layer consists of a convolutional upsampling layer, the visual decoder module and the mask decoder module Decoder shared weights; S3.

2. Input the visual decoder module to get the predicted face appearance image , ,Will Input the mask decoder module to get the predicted mask image , the calculation process is as follows: ， in, , is the number of detection categories, Represents the operation of the mask decoder module.

5. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 4 is characterized in that: S4 is as follows: S4.

1. Based on fine-grained class-aware text tensor and the location of the automatically generated fine-grained class-aware text tensor Get fine-grained class-aware text sequence vector with position information , the calculation process is as follows: ； S4.

2. Construct a language encoder, the language encoder includes A continuous Module, each The modules include multi-head attention module, visual injection module and Feedforward neural network module, including multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer; S4.

3. Fine-grained class-aware text sequence vector with position information Input to the speech encoder, First, after normalization, it is input into the first-layer multi-head attention module for global multi-head attention calculation, and then passes through the residual layer to obtain the global semantic features of the text. , the calculation process is as follows: ， in, represents the normalization operation, Represents the operation of the multi-head attention module; Then After normalization and visual manipulation features Input to the visual attention module, and then pass through the residual layer to obtain the global visual language features through cross attention calculation. , the specific process is as follows: Will Input to the query linear layer to get the query tensor , the calculation process is as follows: ， in, , Represents the query linear layer parameters in the first layer multi-head attention module; Visual manipulation features Input to the key mapping layer to get the key tensor , the calculation process is as follows: ， in, , Represents the key mapping layer parameters in the first layer of multi-head attention module; Then Input to the value mapping layer to get the value tensor , the calculation process is as follows: ， in, , Represents the parameters of the median mapping layer of the first-layer multi-head attention module; Then the global visual language features are obtained through cross attention calculation , the calculation process is as follows: ， in, express The operation of the function, ; Then After layer normalization, input to In the module, the refined global visual language fusion features are obtained through the residual module , the calculation process is as follows: ； Then the first layer of the language encoder The output of the module is used as the first layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated several times until the layer Module Output , the final output of the speech encoder That is, the fine-grained source language features. Take the last one Text fragments get fine-grained language features ; S4.

4. Fine-grained Class-aware Text Tensor Attach a and remove the last Get a new fine-grained class-aware text tensor ,pass and Location Get fine-grained class-aware target text sequence vector with position information , the calculation process is as follows: ； S4.

5. Construct a language decoder. The language decoder consists of A continuous Module composition, each The modules include mask multi-head attention module, multi-head attention module, visual injection module and Modules, mask multi-head attention module, multi-head attention module, visual injection module and The upper layer of the module is layer, the next layer is the residual layer; S4.

6. Fine-grained class-aware target text sequence vector with position information Input to the language decoder, First, it is normalized and input into the first-layer mask multi-head attention module, and then passes through the residual layer to obtain the semantic features of the target text. , the calculation process is as follows: ， in, represents the normalization operation, Represents the operation of the masked multi-head attention module; Then After normalization, They are input into the multi-head attention module for cross attention calculation, and then pass through the residual layer to obtain the semantic features fused with the source target. , the calculation process is as follows: ， in, Represents the operation of the multi-head attention module; Will After normalization and visual manipulation features Input to the visual injection module, and then pass through the residual layer to obtain the global visual language fusion feature , the calculation process is as follows: ， in, Represents the operation of the visual injection module; Will After normalization, input to module, and then pass through the residual layer to obtain the refined global visual language fusion features , the calculation process is as follows: ， in, express Operation of the module; Then the first layer of the language decoder The output of the module is used as the second layer The input of the module is the second layer The output of the module is used as the third layer The input of the module is repeated many times until the layer Module Output , the final output of the language decoder That is the predicted target language feature.

6. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 5 is characterized in that: S5 is as follows: Will Input to an adapter consisting of a fully connected layer to obtain the predicted language features , the calculation process is as follows: ， in, , represents the adapter parameters, ; Then Input into a multi-layer perceptron composed of fully connected layers to obtain the predicted image category features , the calculation process is as follows: ， in, , represents the multi-layer perceptron parameters, .

7. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 6, wherein S6 The details are as follows: S6.

1. Total loss function: ， ； ； ， ， ， ， ； ； ； ； in, represents transpose, represents the appearance reconstruction loss function, represents the predicted face appearance image, Indicates the number of images The index of represents the manipulation localization loss function, represents the predicted mask image, A tensor representing the true mask image, represents the cross-modal contrast loss function, Represents fine-grained language features, Indicates The image visual features and The similarity score of the language features of the image text, Indicates The image text language features and The similarity score of the visual features of the images, represents the contrastive loss function from vision to language, represents the contrast loss function from language to vision, Indicates the number of images The index of Represents the one-hot label of the sample. represents the cosine similarity function, represents a trainable temperature parameter, express Divergence loss function, represents the language reconstruction loss function, Indicates the number of word tags The index of Represents the predicted label vector of the text, Indicates The image text The predicted label vector of each word, Indicates The image text The true one-hot label of each word, Indicates The true one-hot labels of the images, Indicates images to predict image category features, represents the manipulation detection loss function, Represents a real image t tag; S6.

2. Optimize the total loss function using the Adam optimizer , complete the training of the adapter and the multi-layer perceptron, and obtain a multi-domain fine-grained visual language reconstruction model.

8. The method for detecting and locating diffuse forged faces based on multi-domain visual language reconstruction according to claim 7 is characterized in that: S7 is as follows: The tensor Visual encoders as input to multi-domain fine-grained visual-language reconstruction models Get local image features respectively , and then the residual image is calculated through the visual decoder module , the calculation process is as follows: ， Then and Input to multi-domain visual encoder , the final predicted image true or false category is calculated by multi-layer perceptron , the calculation process is as follows: ， Finally, the final predicted mask image is obtained by calculation , the calculation process is as follows: ， in, represents the visual encoder, represents the operation of the visual decoder module, Represents a multilayer perceptron.

Images