False face image identification method based on image block comparison and residual image estimation

Abstract

The invention relates to the technical field of digital image evidence obtaining and visual information security, in particular to a fake face image identification method based on image block comparison and residual image estimation. In addition, the invention also provides a fake face image identification model and an application thereof. The fake face image identification method based on pure image block comparison and reliable residual image estimation is innovatively provided, the cost of artificial identification and auditing of the fake face image is reduced, the accuracy and reliability of fake face image identification are improved, the accuracy, generalization and robustness are higher, and the method is suitable for popularization and application. The method has great practical application value and social practical significance.

Description

Technical Field

[0001] This invention relates to the fields of digital image forensics and visual information security, and in particular to a method for identifying forged face images based on image patch comparison and residual map estimation. Background Technology

[0002] Image synthesis technology has matured significantly, not only drastically reducing the cost of image forgery but also making it difficult to distinguish forged images from genuine ones. By modifying faces in images, it is possible to forge someone else's identity, posing a major threat to facial payment security and visual communication security. This necessitates the support of forged facial image identification technology to safeguard facial information security.

[0003] The challenges in identifying forged facial images lie in two main aspects. Firstly, the sheer number of methods used to forge facial images makes it impossible to obtain data on all types of forgeries. Secondly, there's the issue of neural network model design. Existing neural network models are prone to overfitting, resulting in current forged facial image identification technologies falling far short of user requirements in terms of accuracy.

[0004] Based on the above considerations, this invention provides a method for identifying fake face images based on clean image block comparison and reliable residual map estimation, which greatly improves the accuracy, generalization and robustness of fake face image identification.

[0005] The aforementioned background knowledge is intended to aid in understanding the concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content was disclosed before the filing date of this patent application, the aforementioned background knowledge should not be used to evaluate the technical innovation of this invention. Summary of the Invention

[0006] This invention provides a method for identifying forged face images based on clean image block comparison and reliable residual map estimation, which greatly reduces the cost of manually reviewing forged face images and improves the accuracy of identifying forged face images.

[0007] The technical solution adopted in this invention is as follows: The method for identifying fake face images based on clean image patch comparison and reliable residual map estimation includes the following two steps: Step 1: Design of the neural network model; Step 2: Training the neural network model; Step 3: Input the image to be detected into the trained neural network model for identifying fake faces, and output the fake face detection result.

[0008] The neural network model for identifying fake face images includes a ResNet-based backbone network, a clean image patch comparison module, and a reliable residual map prediction module.

[0009] The clean image patch comparison module identifies forged face images by comparing the feature differences between clean face image patches containing only face information and clean background image patches containing only background information; the reliable residual map prediction module guides the backbone network to focus on reliable information near the forged edges in the residual map during training, prompting the backbone network to extract effective features; the two network modules are trained simultaneously with the backbone network, and finally a trained neural network model for identifying forged face images is obtained.

[0010] The neural network model framework designed in step one is as follows: Figure 1 As shown, the specific steps are as follows: (1) Backbone network: The face image to be detected has 3 channels, a height of H, and a width of W. Feature extraction is performed using a ResNet-34-based backbone network, resulting in a channel count of [number missing]. Height is Width is Feature map In the feature map In the image, the feature vector at each pixel For a given local image patch, there are a total of Image blocks.

[0011] (2) Clean image patch comparison module: The open-source face detection library Dlib is used to extract the face images to be detected. Corresponding face area mask , feature map and its corresponding face region mask Input a clean image patch comparison module. First, use a module based on... Convolutional layers The function will feature map Mapped to a new feature space. Then, the face region mask is applied using bilinear interpolation. Adjust to feature map For the same dimensions, purity score diagrams were obtained. , where pixels The value represents the feature map. Feature vector of the corresponding pixel The purity score of the corresponding local image patch [0,1], where 0 represents a clean background image block, 1 represents a clean face image block, and values ​​between 0 and 1 represent mixed image blocks.

[0012] This invention defines a function for calculating the selection priority between clean face image patches and clean background image patches. as follows: (1) (2) (3) In formula (1), when When using the selection priority calculation function Select a clean face image patch; when When using the selection priority calculation function Select a clean background image block. In formulas (2) and (3), , and These represent the three weighting coefficients. and These represent clean face image patches and clean background image patches at the anchor points, respectively. A cleanliness score image is selected. The midpoint of the minimum bounding quadrilateral of the face region is in the feature map. The corresponding image block is an anchor point clean face image block. Select feature map The image block in the upper left corner is an anchor point with a clean background image block. . It is a function used to calculate the distance between two image patches, and the specific formula is as follows: (4) in This indicates that the clean image patch at the anchor point is in the feature map. Corresponding pixel in the middle eigenvectors, Indicates the current image patch in the feature map Corresponding pixel in the middle eigenvectors, and These represent the clean image patches at the anchor points in the feature map. Corresponding pixel in the middle x-axis and y-axis coordinates, and These represent the current image patch in the feature map. Corresponding pixel in the middle The x-axis and y-axis coordinates are used to calculate the selection priority value of all image patches according to formula (1). Select the first 16 priority values ​​in descending order. The largest clean face image patch and the largest clean background image patch have the following corresponding feature vectors: and .

[0013] Next, global average pooling is used. The pure face feature vectors are respectively and clean background feature vector Compress to and Then splice and get Finally, the feature vector Input fully connected layer and Activation function, output classification result The above process can be expressed using the following formula: (5) (6) (7) (8) (3) Reliable residual map estimation module: A reliable residual map estimation module is built using convolutional layers, residual blocks, and transposed convolutional layers. The feature maps... The input reliable residual map estimation module predicts the residual map between the input image and the ground truth image. .

[0014] The specific neural network model training steps designed in step two are as follows: (1) Loss function design The loss function used in this invention consists of two parts: the overall loss function. as follows: (9) Among them, classification loss Using the cross-entropy loss function, residual loss The mean square error function is weighted by the distance field. and Residual loss With classification loss The weighting coefficients.

[0015] Cross-entropy loss function is used as the classification loss. The following formula is used to constrain the clean image patch comparison module: (10) in, It is the total number of categories. This represents the result of the clean image patch comparison module predicting the true class of the sample. Indicates the true class label of the sample. This indicates that the sample is a real human face image. This indicates that the sample is a fake face image. This is the summation operator for all training samples.

[0016] The mean square error function weighted by the distance field is used as the residual loss. The reliable residual map estimation module is constrained, and the specific formula is as follows: (11) in To provide the residual map prediction results for the sample by the reliable residual map estimation module, The ground truth value of the sample residual map is obtained by calculating the absolute difference between a real face image and its corresponding fake face image. The sample distance field is created based on a face region mask. This is the summation operator for all training samples.

[0017] (2) Training parameter settings The AdamW optimizer is used to update the network model parameters, with the learning rate set to... The training image size was set to 256*256, the batch size was set to 128, and the training was conducted for a total of 200 rounds.

[0018] (3) Model training process After normalizing the facial image, it is input into the neural network model for identifying fake facial images designed in this invention, and the result is obtained through the overall loss function. Sample True Labels True value of residual plot To address the constraints, the open-source deep learning framework PyTorch is used to update the parameters of the network model until the overall loss function is reached. The training process ends upon convergence.

[0019] (4) Model testing methods The face image to be detected is normalized and then input into the backbone network for feature extraction to obtain a feature map. The feature map is then input into the clean image block comparison module to obtain the face image authenticity detection result.

[0020] In addition, this application also provides a model for identifying forged face images and its application.

[0021] This model can serve the task of identifying forged facial images in application fields such as digital forensics, forensic identification, visual communication, facial image fraud detection, and facial recognition security.

[0022] This invention primarily utilizes the deep learning framework PyTorch, the image processing library OpenCV, and the scientific computing library NumPy. The parameters of the neural network model are updated by establishing a loss function between the model's predictions and the true labels, until the overall loss function converges, thus completing the model training process.

[0023] This invention innovatively proposes a method for identifying forged face images based on clean image block comparison and reliable residual map estimation. This method reduces the cost of manual identification and verification of forged face images, improves the accuracy and reliability of forged face image identification, and has higher accuracy, generalization and robustness. It has significant practical application value and social significance. Attached Figure Description

[0024] Some of the accompanying drawings are provided to enable those skilled in the art to more quickly and clearly understand the above and / or other objects, features, advantages and examples of this application. It should be noted that the drawings, suitable embodiments and their descriptions constituting this application are used to provide a further understanding of this application and do not constitute an improper limitation of this application.

[0025] Figure 1 This is a diagram of the neural network model framework for identifying fake face images in step one. Detailed Implementation

[0026] 1. A method for identifying fake face images based on clean image patch comparison and reliable residual map estimation, comprising the following two steps: Step 1: Design of the neural network model in the method for identifying fake face images based on clean image patch comparison and reliable residual map estimation; Step 2: Training the neural network model in the method for identifying fake face images based on clean image patch comparison and reliable residual map estimation.

[0027] 2. The method for identifying forged face images based on clean image patch comparison and reliable residual map estimation according to claim 1, wherein the neural network model framework designed in step one is as follows: Figure 1 As shown, the specific steps are as follows: (1) Backbone network: The face image to be detected has 3 channels, a height of H, and a width of W. Feature extraction is performed using a ResNet-34-based backbone network, resulting in a channel count of [number missing]. Height is Width is Feature map In the feature map In the image, the feature vector at each pixel For a given local image patch, there are a total of Image blocks.

[0028] (2) Clean image patch comparison module: The open-source face detection library Dlib is used to extract the face images to be detected. Corresponding face area mask , feature map and its corresponding face region mask Input a clean image patch comparison module. First, use a module based on... Convolutional layers The function will feature map Mapped to a new feature space. Then, the face region mask is applied using bilinear interpolation. Adjust to feature map For the same dimensions, purity score diagrams were obtained. , where pixels The value represents the feature map. Feature vector of the corresponding pixel The purity score of the corresponding local image patch [0,1], where 0 represents a clean background image block, 1 represents a clean face image block, and values ​​between 0 and 1 represent mixed image blocks.

[0029] This invention defines a function for calculating the selection priority between clean face image patches and clean background image patches. as follows: (1) (2) (3) In formula (1), when When using the selection priority calculation function Select a clean face image patch; when When using the selection priority calculation function Select a clean background image block. In formulas (2) and (3), , and These represent the three weighting coefficients. and These represent clean face image patches and clean background image patches at the anchor points, respectively. A cleanliness score image is selected. The midpoint of the minimum bounding quadrilateral of the face region is in the feature map. The corresponding image block is an anchor point clean face image block. Select feature map The image block in the upper left corner is an anchor point with a clean background image block. . It is a function used to calculate the distance between two image patches, and the specific formula is as follows: (4) in This indicates that the clean image patch at the anchor point is in the feature map. Corresponding pixel in the middle eigenvectors, Indicates the current image patch in the feature map Corresponding pixel in the middle eigenvectors, and These represent the clean image patches at the anchor points in the feature map. Corresponding pixel in the middle x-axis and y-axis coordinates, and These represent the current image patch in the feature map. Corresponding pixel in the middle The x-axis and y-axis coordinates are used to calculate the selection priority value of all image patches according to formula (1). Select the first 16 priority values ​​in descending order. The largest clean face image patch and the largest clean background image patch have the following corresponding feature vectors: and .

[0030] Next, global average pooling is used. The pure face feature vectors are respectively and clean background feature vector Compress to and Then splice and get Finally, the feature vector Input fully connected layer and Activation function, output classification result The above process can be expressed using the following formula: (5) (6) (7) (8) (3) Reliable residual map estimation module: A reliable residual map estimation module is built using convolutional layers, residual blocks, and transposed convolutional layers. The feature maps... The input reliable residual map estimation module predicts the residual map between the input image and the ground truth image. .

[0031] 3. The method for identifying forged face images based on clean image patch comparison and reliable residual map estimation according to claim 1, the specific neural network model training steps in step two are as follows: (1) Loss function design The loss function used in this invention consists of two parts: the overall loss function. as follows: (9) Among them, classification loss Using the cross-entropy loss function, residual loss The mean square error function is weighted by the distance field. and Residual loss With classification loss The weighting coefficients.

[0032] Cross-entropy loss function is used as the classification loss. The following formula is used to constrain the clean image patch comparison module: (10) in, It is the total number of categories. This represents the result of the clean image patch comparison module predicting the true class of the sample. Indicates the true class label of the sample. This indicates that the sample is a real human face image. This indicates that the sample is a fake face image. This is the summation operator for all training samples.

[0033] The mean square error function weighted by the distance field is used as the residual loss. The reliable residual map estimation module is constrained, and the specific formula is as follows: (11) in To provide the residual map prediction results for the sample by the reliable residual map estimation module, The ground truth value of the sample residual map is obtained by calculating the absolute difference between a real face image and its corresponding fake face image. The sample distance field is created based on a face region mask. This is the summation operator for all training samples.

[0034] (2) Training parameter settings The AdamW optimizer is used to update the network model parameters, with the learning rate set to... The training image size was set to 256*256, the batch size was set to 128, and the training was conducted for a total of 200 rounds.

[0035] (3) Model training process After normalizing the facial image, it is input into the neural network model for identifying fake facial images designed in this invention, and the result is obtained through the overall loss function. Sample True Labels True value of residual plot To address the constraints, the open-source deep learning framework PyTorch is used to update the parameters of the network model until the overall loss function is reached. The training process ends upon convergence.

[0036] (4) Model testing methods The face image to be detected is normalized and then input into the backbone network for feature extraction to obtain a feature map. The feature map is then input into the clean image block comparison module to obtain the face image authenticity detection result.

[0037] On the commonly used fake face image identification dataset Celeb-DF, the present invention achieved an accuracy of 91.88%. The algorithm has strong generalization and high accuracy, and has great application value and market prospects.

[0038] The conventional techniques described in the above embodiments are existing technologies known to those skilled in the art, and therefore will not be described in detail here.

[0039] The specific embodiments described herein are merely illustrative examples of the invention. Those skilled in the art can make various modifications, additions, or similar substitutions to the specific embodiments described herein, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0040] Although the present invention has been described in detail and specific embodiments have been cited, it will be apparent to those skilled in the art that various changes or modifications can be made without departing from the spirit and scope of the invention.

[0041] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the protection scope of this application.

[0042] All matters not covered in this invention are common knowledge.

Claims

1. A method for identifying fake face images based on clean image patch comparison and reliable residual map estimation, comprising the following two steps: Step 1: Design of the neural network model; The neural network model for identifying fake face images includes a ResNet-based backbone network, a clean image patch comparison module, and a reliable residual map prediction module. Step 2: Training the neural network model; The clean image patch comparison module identifies fake face images by comparing the feature differences between clean face image patches containing only face information and clean background image patches containing only background information; the reliable residual map prediction module guides the backbone network to focus on reliable information near the fake edges in the residual map during training, prompting the backbone network to extract effective features; the two network modules are trained simultaneously with the backbone network, and finally a trained neural network model for identifying fake face images is obtained. Step 3: Input the image to be detected into the trained neural network model for identifying fake faces, and output the fake face detection result.

2. The method of claim 1, wherein the method is characterized by: Step 1: (1) Backbone network: a to-be-detected face image with a channel number of 3, a height of H, and a width of W is input into the backbone network The input is subjected to feature extraction based on a ResNet-34 backbone network, and a feature map with a channel number of , a height of , and a width of is obtained ; in the feature map , a feature vector at each pixel corresponds to a local image block, and there are image blocks in total; (2) Clean image patch comparison module: The open-source face detection library Dlib is used to extract the face images to be detected. Corresponding face region mask , feature map and its corresponding face region mask Input clean image patch comparison module; first use based on Convolutional layers The function will feature map Mapped to a new feature space; then the face region mask is applied using bilinear interpolation. Adjust to feature map For the same size, a purity score map was obtained. , where pixels The value represents the feature map. Feature vector of the corresponding pixel The purity score of the corresponding local image patch [0,1], where 0 represents a clean background image block, 1 represents a clean face image block, and values ​​between 0 and 1 represent mixed image blocks; Define a function to calculate the selection priority between a clean face image patch and a clean background image patch. as follows: (1) (2) (3) In formula (1), when When using the selection priority calculation function Select a clean face image patch; when When using the selection priority calculation function Select a clean background image block; in formulas (2) and (3), , and These represent the three weighting coefficients. and These represent the anchor point clean face image block and the anchor point clean background image block, respectively. Select purity fraction chart The midpoint of the minimum bounding quadrilateral of the face region is in the feature map. The corresponding image block is an anchor point clean face image block. Select feature map The image block in the upper left corner is an anchor point with a clean background image block. It is a function used to calculate the distance between two image patches, and the specific formula is as follows: (4) in This indicates that the clean image patch at the anchor point is in the feature map. Corresponding pixel in the middle eigenvectors, Indicates the current image patch in the feature map Corresponding pixel in the middle eigenvectors, and These represent the clean image patches at the anchor points in the feature map. Corresponding pixel in the middle x-axis and y-axis coordinates, and These represent the current image patch in the feature map. Corresponding pixel in the middle The x-axis and y-axis coordinates; the selection priority values ​​of all image patches are calculated according to formula (1). Select the first 16 priority values ​​in descending order. The largest clean face image patch and the largest clean background image patch have the following corresponding feature vectors: and ; Next, global average pooling is used. The pure face feature vectors are respectively and clean background feature vector Compress to and Then splice and get Finally, the feature vector Input fully connected layer and Activation function, output classification result The above process can be expressed using the following formula: (5) (6) (7) (8) (3) Reliable residual map estimation module: A reliable residual map estimation module is built using convolutional layers, residual blocks, and transposed convolutional layers; the feature map is... The input reliable residual map estimation module predicts the residual map between the input image and the ground truth image. .

3. The method for identifying forged face images based on clean image patch comparison and reliable residual map estimation according to claim 1, characterized in that: Step 2: The neural network model training steps are as follows: (1) Loss function design The loss function used in this invention consists of two parts: the overall loss function. as follows: (9) Among them, classification loss Using the cross-entropy loss function, residual loss The mean square error function is weighted by the distance field. and Residual loss With classification loss Weighting coefficients; Cross-entropy loss function is used as the classification loss. The following formula is used to constrain the clean image patch comparison module: (10) in, It is the total number of categories. This represents the result of the clean image patch comparison module predicting the true class of the sample. Indicates the true class label of the sample. This indicates that the sample is a real human face image. This indicates that the sample is a fake face image. This is the summation operator for all training samples; The mean square error function weighted by the distance field is used as the residual loss. The reliable residual map estimation module is constrained, and the specific formula is as follows: (11) in To provide the residual map prediction results for the sample by the reliable residual map estimation module, The ground truth value of the sample residual map is obtained by calculating the absolute difference between a real face image and its corresponding fake face image. The sample distance field is created based on a face region mask. This is the summation operator for all training samples; (2) Training parameter settings The AdamW optimizer is used to update the network model parameters, with the learning rate set to... The training image size was set to 256*256, the batch size was set to 128, and the training was conducted for a total of 200 rounds. (3) Model training process After normalizing the facial image, it is input into the forged facial image identification neural network model designed in this invention, and the result is obtained through the overall loss function. Authentic labels of samples True value of residual plot To address the constraints, the open-source deep learning framework PyTorch is used to update the parameters of the network model until the overall loss function is reached. The training process ends upon convergence. (4) Model testing methods The face image to be detected is normalized and then input into the backbone network for feature extraction to obtain a feature map. The feature map is then input into the clean image block comparison module to obtain the face image authenticity detection result.

4. A model for identifying fake face images based on clean image patch comparison and reliable residual map estimation, characterized in that: The model includes a ResNet-based backbone network, a clean image patch comparison module, and a reliable residual map prediction module. The clean image patch comparison module identifies fake face images by comparing the feature differences between clean face image patches containing only face information and clean background image patches containing only background information. The reliable residual map prediction module guides the backbone network to focus on reliable information near fake edges in the residual map during training, prompting the backbone network to extract effective features. The two network modules are trained simultaneously with the backbone network to obtain a trained neural network model for identifying fake face images.

5. The use of the forged face image identification model as described in claim 4 in digital forensics, forensic identification, visual communication, anti-forgery of face images, and face identification security for identifying forged face images.

Images