Face forgery detection method based on identity decoupling and adaptive cosine embedding loss

Abstract

This invention discloses a face forgery detection method based on identity decoupling and adaptive cosine embedding loss. Addressing the problems of existing deepfake detection methods being susceptible to interference from identity information and lacking flexibility due to fixed boundary constraints, this invention constructs an identity-independent deepfake detection network. It decouples the input face image into forensic features containing forgery clues and content features containing identity information, eliminating identity interference by ignoring content features. Simultaneously, it designs an adaptive cosine embedding loss based on visual differences, using absolute error and the visual differences between real and fake image pairs as adaptive boundary values ​​to dynamically adjust feature spacing constraints. This invention significantly improves the model's generalization ability when facing unknown forgery types through the synergistic optimization of decoupled representation learning and adaptive metric learning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image content security technology, and in particular to a face forgery detection method based on identity decoupling and adaptive cosine embedding loss. Background Technology

[0002] In recent years, with the rapid development of generative artificial intelligence technology, especially the widespread application of technologies such as Generative Adversarial Networks (GANs), Diffusion Models (DMs), and Autoencoders, the realism of deepfake content has reached unprecedented levels. The malicious spread of this fake content in areas such as social media, financial payments, and judicial evidence collection poses a serious threat to the security of facial recognition systems, the authenticity of information, and the protection of personal privacy. Therefore, there is an urgent need to develop efficient and reliable detection technologies to address these challenges.

[0003] Existing deepfake detection methods primarily explore two dimensions. First, from the perspective of forgery cues, current techniques typically rely on single forgery traces for detection, such as detecting fusion boundaries, inconsistencies between image patches, and frequency domain differences between real and forged images. However, such methods over-rely on specific forgery cues; when the test image does not contain the specific forgery traces from the training set, detection performance drops sharply, resulting in insufficient generalization ability. Second, from the perspective of training data, given the constant emergence of new forgery methods, attempting to train a general model by collecting data generated by all forgery methods is impractical. Some studies have attempted to simulate forgery samples by mixing real face images, but such synthetic data differs significantly in distribution from real deepfake data, making it difficult to effectively improve the model's generalization performance.

[0004] Current mainstream methods mostly employ Convolutional Neural Networks (CNNs) frameworks for end-to-end binary classification training for real and fake faces. These methods often capture global content features containing facial identity information during feature learning. This identity-irrelevant information is considered interference for forgery detection tasks, significantly impacting the model's ability to capture subtle forgery clues. Specifically, facial images of different identities naturally differ in visual features. The model easily misinterprets these identity-related features as forgery traces or obscures genuine forgery clues within the identity information, leading to overfitting to specific identities or forgery patterns and severely limiting generalization performance across datasets and forgery types.

[0005] Furthermore, traditional metric learning methods, such as cosine embedding loss, typically use a fixed minimum distance margin to constrain the feature distance between real and fake samples when optimizing the feature space distribution. This fixed margin ignores the diversity of visual differences between different image pairs: for real and fake image pairs with small visual differences, a fixed large margin forces the network to artificially widen the feature distance between similar images, increasing learning difficulty and easily leading to overfitting; for image pairs with large visual differences, a fixed small margin is insufficient to provide enough feature distance, weakening the model's discriminative ability. This fixed boundary constraint cannot adapt to the dynamic changes in visual content, limiting the flexibility of feature space optimization.

[0006] Therefore, there is an urgent need for a deepfake detection method that can effectively eliminate interference from identity information and can adaptively adjust and optimize the target based on the characteristics of the image, so as to improve the generalization ability and robustness of the model when facing unknown forgery techniques. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a face forgery detection method based on identity decoupling and adaptive cosine embedding loss.

[0008] The objective of this invention is achieved through the following technical solution: a face forgery detection method based on identity decoupling and adaptive cosine embedding loss, the method comprising:

[0009] Training image pairs are constructed using real face images and fake face images;

[0010] An identity-independent deep forgery detection network is constructed. The network uses a feature encoder to encode each input image pair to obtain corresponding feature maps. These feature maps are then fed into parallel evidence-gathering and content branches to extract evidence-gathering features with forgery traces and consistent content features containing identity information, respectively. The evidence-gathering features and consistent content features corresponding to the same feature map are concatenated by a feature decoder to obtain the reconstructed results of real and forged face images. A classifier is used to identify the authenticity of face images by performing average pooling on the vectors obtained from the evidence-gathering features, and outputs probability scores.

[0011] The network is trained by constructing a total loss function that includes content consistency loss, reconstruction loss, classification loss, and adaptive cosine embedding loss using the outputs of each branch and classifier. The feature encoder, forensic branch, and classifier of the trained identity-independent deep forgery detection network are then deployed to perform face forgery detection on the input face images.

[0012] Furthermore, the feature encoder uses the EfficientNet B3 model pre-trained on the ImageNet dataset to encode the input real face image and the corresponding fake face image into feature maps respectively.

[0013] Furthermore, the evidence collection branch is composed of It consists of convolutional layers and separable convolutional layers, which extract forensic features with traces of forgery from feature maps.

[0014] Furthermore, the content branches include Convolutional layers and Convolutional layers are used to extract consistent content features with identity information from feature maps.

[0015] Furthermore, the feature decoder includes multiple convolutional layers, residual blocks, and a sigmoid activation function, which concatenate the evidence features extracted from the evidence branch and the content features extracted from the content branch to obtain the reconstruction results of the corresponding real face image and the forged face image.

[0016] Furthermore, the total loss function is a weighted sum of content consistency loss, reconstruction loss, classification loss, and adaptive cosine embedding loss based on visual differences; when only training image pairs with consistent identities are used, the weights are equal and equal to 1; when training image pairs with consistent and inconsistent identities are used in combination, the number of image pairs of the two classes is equal in each batch, the content consistency loss weight of the image pairs with consistent identities is 1, the content consistency loss weight of the image pairs with inconsistent identities is 0, and the remaining weight coefficients are all 1.

[0017] Furthermore, the content consistency loss is the L2 norm of the difference between the content features of the real image and the content features of the forged image;

[0018] The reconstruction loss includes: the L2 norm of the difference between the input real face image and the reconstruction result of the real image, plus the L2 norm of the difference between the input fake face image and the reconstruction result of the fake image.

[0019] The classification loss is the binary cross-entropy loss calculated from the fake probability score output by the classifier and the true label of the input sample.

[0020] The adaptive cosine embedding loss includes: calculating the cosine similarity between the input image and the feature vectors obtained by feature encoder and forensic branch extraction and average pooling; when the two input images are both real samples or both are fake samples, the cosine similarity is selected as 1 minus the cosine similarity as the loss; when the two input images are samples of different types, and the cosine similarity of the forensic features of different types of samples is greater than the adaptive boundary value, the difference between the cosine similarity and the adaptive boundary value is selected as the loss.

[0021] Furthermore, the calculation of the adaptive boundary includes:

[0022] Calculate the sum of absolute errors between each pair of identical real and fake images within a batch to measure the visual difference between the image pairs; perform min-max normalization on the sum of absolute errors and scale it to... The interval will be used to transform the normalized absolute error and linearly to... The interval is used as the adaptive boundary value for the image pair. The adaptive boundary value is truncated to restrict it to a certain range. Within the range.

[0023] According to another aspect of the specification, a face forgery detection device based on identity decoupling and adaptive cosine embedding loss is disclosed, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the face forgery detection method based on identity decoupling and adaptive cosine embedding loss.

[0024] According to another aspect of the specification, a computer-readable storage medium is disclosed having a program stored thereon, which, when executed by a processor, implements the aforementioned face forgery detection method based on identity decoupling and adaptive cosine embedding loss.

[0025] The beneficial effects of this invention are:

[0026] This invention, through a carefully designed network architecture and loss function, forces the model to decompose the input image into two independent parts: "content features" related to identity and "evidence features" related to forgery traces. During training and inference, the model relies solely on pure evidence features for decision-making, thereby fundamentally eliminating the interference of identity information.

[0027] This invention abandons the traditional approach of fixing boundary values ​​and designs a loss function that can dynamically adjust the boundary values ​​based on the actual visual differences between each pair of real and fake images. For visually very similar pairs of real and fake images, their evidence feature distances are allowed to be relatively short to reduce the learning difficulty; for pairs of images with significant visual differences, their evidence feature distances are forced to be sufficiently large to enhance the model's discriminative power.

[0028] This invention effectively eliminates interference from identity information and significantly improves the model's generalization ability and detection accuracy across datasets and under unknown forgery techniques. It can be applied to fields such as digital forensics, financial payment security, social media content moderation, and forensic identification. Attached Figure Description

[0029] Figure 1 An architecture diagram of the feature decoder provided in an embodiment of the present invention;

[0030] Figure 2 This is a training network architecture diagram provided in an embodiment of the present invention;

[0031] Figure 3 The inference network architecture diagram provided for embodiments of the present invention;

[0032] Figure 4 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation

[0033] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] A face forgery detection method based on identity decoupling and adaptive cosine embedding loss includes the following three steps:

[0035] Step 1: Decouple the real and fake face images with consistent identities from the input identity-independent deepfake detection network for feature decoupling;

[0036] Step 2: Perform multi-objective joint optimization and adaptive feature constraints based on decoupling features, including content consistency loss, reconstruction loss, classification loss and adaptive cosine embedding loss based on visual differences;

[0037] Step 3: Model training and inference deployment.

[0038] The specific steps for Step One are as follows:

[0039] A training image pair is formed by selecting real face images with consistent identities and fake face images, and the training image pair is input into an identity-independent deep forgery detection network; the identity-independent deep forgery detection network includes a feature encoder. ), Forensic Branch Content Branch ), Feature Decoder The input image pairs are encoded by the feature encoder to obtain corresponding feature maps; the evidence-gathering branch extracts evidence-gathering features containing forgery clues from the feature maps, and the content branch extracts content features containing identity information from the feature maps.

[0040] (1) such as Figure 1 As shown, the feature encoder uses the EfficientNetB3 model pre-trained on the ImageNet dataset to process the input real face image. and corresponding fake face images Encode them into feature maps respectively and Its mathematical expression is:

[0041]

[0042] in, Indicates the space size as The input image, The space dimensions are The number of channels is Feature map, It is the feature encoder described above.

[0043] (2) The evidence collection branch consists of one It consists of a convolutional layer and a separable convolutional layer, first using... Convolutional processing performs channel dimensionality reduction and integrates cross-channel features. After BatchNorm and ReLU activation, it is then used... Separable convolutions are used to extract features from feature maps. and Extracting forensic features with traces of forgery. and .

[0044] (3) The content branch contains a Convolutional layer and one Convolutional layers, first use Convolution is used for channel reduction, followed by BatchNorm and ReLU activation, and then... Convolutions are used to extract features from the feature map. and Extract consistent content features containing identity information. and .

[0045] (4) The feature decoder consists of multiple convolutional layers, residual blocks, and a sigmoid activation function, and is used to extract the forensic features from the aforementioned forensic branches. , Content features extracted from the aforementioned content branches , The corresponding real face images obtained by stitching them together and fake facial images Reconstruction results and .

[0046] The specific steps for step two are as follows:

[0047] The evidence-gathering features and the content features are concatenated and then input into the feature decoder to reconstruct the original input image. A content consistency loss is calculated to constrain the content features of real and fake image pairs with the same identity to remain consistent in the feature space. A reconstruction loss is calculated to constrain the difference between the reconstructed image and the original input image. Only the evidence-gathering features are used for binary classification of real and fake images, and the classification loss is calculated. An adaptive cosine embedding loss based on visual differences is calculated, dynamically setting boundary values ​​according to the visual differences between each pair of real and fake images with the same identity to constrain the distribution of the evidence-gathering features in the feature space.

[0048] (1) The loss of content consistency Content Consistency Loss is used to constrain the extracted content features of real and fake images with the same identity to maintain a high degree of consistency in the feature space, ensuring that identity information is completely isolated from content features. Its mathematical expression is:

[0049]

[0050] in, Represents the content features of a real image. Indicates the content features of a forged image. This represents the L2 norm.

[0051] (2) The reconstruction loss (Reconstruction Loss) is used to constrain the difference between the image reconstructed by the feature decoder and the original input image, ensuring that content features and forensic features can completely reconstruct the original image. Its mathematical expression is:

[0052]

[0053] in, This represents the input of a real human face image. This indicates the corresponding fake face image. Represents a real image The reconstruction results Indicates a forged image The reconstruction results This represents the L2 norm.

[0054] (3) The classification loss (Classification Loss) employs binary cross-entropy loss to perform binary classification of images as true or false based solely on forensic features. Its mathematical expression is:

[0055]

[0056] in, This represents the forgery probability score output by the classifier. This represents the true label of the input sample; when the sample is a fake image... When the sample is a real image , () is the natural logarithm function.

[0057] (4) The adaptive cosine embedding loss based on visual differences The specific calculation steps for (Adaptive Cosine EmbeddingLoss) are as follows:

[0058] Calculate the sum of absolute errors (SAE) between each pair of identical real and fake images within a batch (the number of samples input to the model at one time during model training) to measure the visual difference between image pairs.

[0059]

[0060] in, and They are real images and fake images, respectively. This represents the L1 norm.

[0061] The absolute errors mentioned above are then subjected to minimum-maximum normalization and scaled to [value missing]. Interval:

[0062]

[0063] in, and These are the minimum and maximum values ​​of SAE in the current batch, respectively.

[0064] The normalized absolute error and linear transformation are then applied to... The interval serves as the adaptive boundary value for the image pair. ,Right now

[0065]

[0066] The adaptive boundary value is truncated to restrict it to... Within a certain range, to preserve subtle differences between highly similar image pairs and weak similarities between highly dissimilar image pairs.

[0067] The cosine embedding loss is calculated using the adaptive boundary value to increase the evidence feature distance between different types of samples and maintain the evidence feature distance between samples of the same type.

[0068] The cosine embedding loss is used to constrain the distance of forensic features between samples of different classes (i.e., pairs of real and fake images), while maintaining the distance of forensic features between samples of the same class. When the similarity of forensic features between samples of different classes is greater than the adaptive boundary value, i.e., the feature distance is too close, the loss function will impose constraints, prompting the network to learn more discriminative feature representations. Its mathematical expression is:

[0069]

[0070] in, and For the input image and The feature vector obtained after extraction by the feature encoder and the evidence-gathering branch, followed by average pooling. express and Are they both real samples or both fake samples? express and They are different types of samples. Represents the cosine similarity function. This represents the function that takes the maximum value. Indicates a pair based on the input image Adaptive boundary values ​​for pixel-level visual difference calculation.

[0071] Step 3 is detailed below:

[0072] like Figure 2 As shown, during the training phase, a real human face image is input. and corresponding fake face images Feature encoder They are encoded into feature maps respectively. and Content branches From feature maps and Extract consistent content features containing identity information. and Evidence-gathering branch Then from the feature map and Extracting forensic features with traces of forgery. and Then, they were assembled separately. and , and Input feature decoder To obtain a real human face image and fake facial images Corresponding reconstruction results and Furthermore, the classifier analyzes forensic features. and Vector obtained by average pooling and It is used to identify the authenticity of facial images and output probability scores.

[0073] The content consistency loss, reconstruction loss, classification loss, and visual difference-based adaptive cosine embedding loss are weighted and summed to obtain the total training loss (Total Loss). ), and update network parameters through backpropagation; such as Figure 3 As shown, during the inference phase, only the aforementioned feature encoder, evidence branch, and classifier are used to determine whether the input face image is real or fake.

[0074] (1) The overall training loss is a weighted sum of content consistency loss, reconstruction loss, classification loss, and visual difference-based adaptive cosine embedding loss, and its mathematical expression is:

[0075]

[0076] in, These are the weighting coefficients for each loss term; when only training image pairs with consistent identities are used, When training image pairs with consistent and inconsistent identities are used in combination, the number of image pairs of each class is equal in each batch, while the number of consistent image pairs is... Image pairs with inconsistent identities All other weighting coefficients are 1.

[0077] (2) Training parameter settings: The AdamW optimizer was used for network training, and the initial learning rate was set to The batch size was set to 32; after 75 training epochs, a cosine annealing strategy was used to reduce the learning rate to... Continue training for 25 cycles.

[0078] Corresponding to the aforementioned embodiment of a face forgery detection method based on identity decoupling and adaptive cosine embedding loss, the present invention also provides an embodiment of a face forgery detection device based on identity decoupling and adaptive cosine embedding loss.

[0079] See Figure 4The present invention provides a face forgery detection device based on identity decoupling and adaptive cosine embedding loss, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a face forgery detection method based on identity decoupling and adaptive cosine embedding loss in the above embodiment.

[0080] The embodiment of the face forgery detection device based on identity decoupling and adaptive cosine embedding loss provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing-capable device, including the face forgery detection device based on identity decoupling and adaptive cosine embedding loss provided by this invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0081] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0082] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0083] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a face forgery detection method based on identity decoupling and adaptive cosine embedding loss as described in the above embodiments.

[0084] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0085] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned face forgery detection method based on identity decoupling and adaptive cosine embedding loss.

[0086] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0087] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A face forgery detection method based on identity decoupling and adaptive cosine embedding loss, characterized in that, The method includes: Training image pairs are constructed using real face images and fake face images; An identity-independent deep forgery detection network is constructed. The network uses a feature encoder to encode each input image pair to obtain corresponding feature maps. These feature maps are then fed into parallel evidence-gathering and content branches to extract evidence-gathering features with forgery traces and consistent content features containing identity information, respectively. The evidence-gathering features and consistent content features corresponding to the same feature map are concatenated by a feature decoder to obtain the reconstructed results of real and forged face images. A classifier is used to identify the authenticity of face images by performing average pooling on the vectors obtained from the evidence-gathering features, and outputs probability scores. The network is trained by constructing a total loss function containing content consistency loss, reconstruction loss, classification loss, and adaptive cosine embedding loss using the outputs of each branch and classifier. The feature encoder, forensic branch, and classifier of the trained identity-agnostic deep forgery detection network are then deployed to perform face forgery detection on the input face images to distinguish between real and fake images. The adaptive cosine embedding loss includes: calculating the cosine similarity between the input image and the feature vectors obtained by the feature encoder and the evidence-gathering branch and average pooling. When the two input images are both real samples or both are fake samples, the loss is 1 minus the cosine similarity. When the two input images are samples of different types, and the cosine similarity of the evidence-gathering features of the different types of samples is greater than the adaptive boundary value, the loss is the difference between the cosine similarity and the adaptive boundary value. The calculation of the adaptive boundary includes: calculating the sum of absolute errors between each pair of identical real and fake images within a batch, used to measure the visual difference between the image pairs; performing min-max normalization on the sum of absolute errors, scaling it to... The interval is used to linearly transform the normalized absolute error to... The interval is used as the adaptive boundary value for the image pair. The adaptive boundary value is truncated to restrict it to a certain range. Within the range.

2. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The feature encoder uses the EfficientNet B3 model pre-trained on the ImageNet dataset to encode the input real face image and the corresponding fake face image into feature maps respectively.

3. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The evidence collection branch consists of It consists of convolutional layers and separable convolutional layers, which extract forensic features with traces of forgery from feature maps.

4. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The content branch includes Convolutional layers and Convolutional layers are used to extract consistent content features with identity information from feature maps.

5. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The feature decoder includes multiple convolutional layers, residual blocks, and a sigmoid activation function. It concatenates the evidence features extracted from the evidence branch and the content features extracted from the content branch to obtain the reconstruction results of the corresponding real face image and the fake face image.

6. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The total loss function is a weighted sum of content consistency loss, reconstruction loss, classification loss, and visual difference-based adaptive cosine embedding loss. When only training image pairs with consistent identities are used, the weights are equal and equal to 1. When training image pairs with consistent and inconsistent identities are used in combination, the number of image pairs of the two classes is equal in each batch, the content consistency loss weight of the consistent image pairs is 1, the content consistency loss weight of the inconsistent image pairs is 0, and the weights of the remaining weights are all 1.

7. The face forgery detection method based on identity decoupling and adaptive cosine embedding loss according to claim 1, characterized in that, The content consistency loss is the L2 norm of the difference between the content features of the real image and the content features of the fake image; The reconstruction loss includes: the L2 norm of the difference between the input real face image and the reconstruction result of the real image, plus the L2 norm of the difference between the input fake face image and the reconstruction result of the fake image. The classification loss is the binary cross-entropy loss calculated from the fake probability score output by the classifier and the true label of the input sample.

8. A face forgery detection device based on identity decoupling and adaptive cosine embedding loss, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a face forgery detection method based on identity decoupling and adaptive cosine embedding loss as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a face forgery detection method based on identity decoupling and adaptive cosine embedding loss as described in any one of claims 1-7.

Images