A fine-grained fake image detection method and a model training method thereof

Abstract

The application discloses a fine-grained fake image detection method and a model training method thereof. The detection method adopts a hierarchical fake attribute classification technology, outputs a classification prediction value of each layer of fake attributes, outputs a fake image prediction mask, meets the detection and fake region positioning requirements of different fine-grained fake images, and can be quickly and simply deployed and applied in the field related to fake image detection. The training method of the fine-grained fake image detection model adopts a hierarchical fake attribute classification prediction value calculation formula, adopts a plurality of loss functions to constrain model module parameter updating, designs a positioning loss function based on a deep metric learning method and a SoftTripleLoss classification loss function with an adaptive class center number, improves the fake attribute classification prediction performance and fake region positioning performance of the model, and thus improves the fake region positioning accuracy and fake attribute classification accuracy. The method can be popularized to more general fake image detection tasks.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, artificial intelligence, and forged image detection, and particularly to a fine-grained method for detecting forged images and a model training method thereof. Background Technology

[0002] Forged images, if misused, can have serious negative impacts. Different forged image generation methods lead to variations in forged attributes, making the development of a unified method for image forgery detection and localization highly challenging. Currently, forged image detection and forged region localization often employ deep learning-based methods targeting specific forged image generation techniques. However, these methods are often only effective under specific forged attributes and have poor generalization capabilities. This invention proposes a fine-grained forged image detection method. This method employs a hierarchical forged attribute classification technique, outputting classification prediction values ​​for each layer of forged attributes. The method also outputs a forged image prediction mask, meeting the requirements for fine-grained forged image detection and forged region localization. Furthermore, this invention proposes a training method for a fine-grained forged image detection model. This method uses a hierarchical forged attribute classification prediction value calculation formula and employs multiple loss functions to constrain the parameter updates of each module of the model, improving the model's performance in forged attribute classification prediction and forged region localization. This invention also has significant application prospects.

[0003] A fine-grained method for detecting forged images and its model training method are disclosed, mainly including a fine-grained method for detecting forged images and a training method for a fine-grained method for detecting forged images. The proposed method, "Deepfake Image Detection Method and Apparatus" (Authorization Announcement No.: CN112183501B), proposes using three convolutional neural networks to perform feature extraction, feature fusion, and feature re-extraction on the input face image, and outputting the probability confidence score for classifying the face image as real or fake through a linear classifier and normalization processing. The proposed method, apparatus, device, and medium for deepfake image detection and model training (Application Publication No.:

[0004] CN112598643A proposes using a Siamese network to extract feature vectors from a pair of genuine and fake images, calculating the distance between the feature vectors of the pair of genuine and fake images, the loss function of the APN, and training a detection model based on the loss function of the detection model determined by the loss function of the APN. CN115690501A, "A Deep Forged Image Detection Method and System Based on Fine-Grained Features," proposes using an image detection backbone network to extract features, dividing the global features horizontally into several blocks to construct fine-grained features, and using classification loss for constrained training.

[0005] The specific patent prior art documents mentioned above are as follows:

[0006] (1) Method and apparatus for detecting deepfake images (Authorization Announcement No.: CN112183501B)

[0007] This invention discloses a method and apparatus for deepfake image detection. The method includes: inputting a face image, which includes both a real face image and a fake face image; performing preliminary face feature extraction on the face image using a first convolutional neural network; extracting traditional image features, including texture features, from the face image using a second convolutional neural network, and dynamically adjusting and processing the traditional image features; superimposing the preliminary face features and traditional image features along the channel dimension to obtain fused features of the face image; further extracting features from the fused features using a third convolutional neural network, whereby the preliminary face features and the processed traditional image features interact; and outputting a probability confidence score for classifying the face image as real or fake based on the further extracted features. This invention integrates traditional image features and deep learning, significantly improving the accuracy and generalization performance of deepfake image detection models.

[0008] The forged image detection method described in this invention is designed for multiple categories of forged images. The feature recognition and detection module of the forged image detection network model has a hierarchical structure and adopts a hierarchical forgery attribute classification calculation formula. The detection network model outputs the probability confidence of forgery attributes at each level of the image to be detected, which is different from the comparison file.

[0009] (2) Methods, apparatus, equipment and media for deepfake image detection and model training (Application Publication No.: CN112598643A)

[0010] This patent discloses a method, apparatus, device, medium, and product for deepfake image detection and model training, relating to the field of image processing technology, specifically to artificial intelligence technologies such as deep learning and computer vision. The training method includes: obtaining a first ROI image based on a first image and ROI information output by an APN; obtaining a second ROI image based on the ROI information and a second image; using a Siamese network to extract feature vectors from the first image and the second image respectively; calculating the distance between the two image feature vectors, calculating the loss function of the APN based on the distance, such that the training objective of the APN is to maximize the distance; determining the loss function of the detection model based on the loss function of the APN, and training the detection model based on the loss function of the detection model. The method described in this patent can improve the accuracy of the APN, thereby improving the accuracy of deepfake image detection.

[0011] The forgery detection model training method described in this invention constructs multiple loss functions to constrain the parameter updates of the forgery detection model, including a binary cross-entropy loss function, a localization loss function based on depth metric learning, and a SoftTripleLoss classification loss function that adapts to the number of class centers. This effectively improves the model's performance in forgery region localization and forgery attribute classification, thereby increasing the accuracy of forgery region localization and forgery attribute classification. The detection model training method described in this invention differs from that in the comparative documents.

[0012] (3) A method and system for detecting deepfake images based on fine-grained features (Application Publication No.: CN115690501A)

[0013] This invention discloses a method and system for deepfake image detection based on fine-grained features. The method includes extracting features using a backbone network for image detection, the backbone network comprising four residual convolutional blocks; dividing the extracted global features horizontally into several blocks to construct fine-grained features, and using these fine-grained features to guide the output features of the residual convolutional blocks, constrained by a loss function; pooling the fine-grained features and global features, and training the pooled output features using a classification loss function. This invention, without requiring complex attention mechanisms or introducing additional training parameters, enables the model to acquire more robust features, significantly improving the accuracy of cross-domain deepfake detection.

[0014] The forged image detection method described in this invention uses the feature extraction module of the detection model to extract features, and performs hierarchical forged image classification by the hierarchical forged attribute classification calculation formula of the feature recognition and detection module. Finally, it outputs the forged attribute confidence prediction value of each layer, which is different from the comparison file. Summary of the Invention

[0015] To address the aforementioned technical problems, the purpose of this invention is to provide a method for detecting fine-grained forged images, achieving unified image forgery detection and localization, and meeting the needs for detecting and locating forged areas of different fine-grained forged images.

[0016] The objective of this invention is achieved through the following technical solution:

[0017] A method for detecting fine-grained forged images, comprising:

[0018] Step A1 inputs the image to be detected into the feature extraction module of the fake image detection model and outputs the feature information of the image to be detected;

[0019] Step A2 inputs the feature information and mask ground truth of the image to be detected into the feature recognition and detection module of the forgery image detection model, and outputs fine-grained forgery attribute classification prediction value, prediction mask and forgery region localization accuracy;

[0020] Step A3 evaluates the fine-grained forgery attribute classification prediction value, prediction mask and forgery region localization accuracy, and outputs the detection results.

[0021] A training method for a fine-grained forged image detection model, including...

[0022] Step B1: Create a fine-grained dataset for detecting forged images;

[0023] Step B2 inputs the forged images to be trained in the dataset into the feature extraction module of the forged image detection model, and outputs a feature set.

[0024] Step B3 will set the features Input the ground truth mask of the forged image to be trained into the feature recognition and detection module of the forged image detection model, and output the fine-grained forged attribute classification prediction value prob and the prediction mask pred_mask;

[0025] Step B4: Construct the loss function Constrain the updating of parameters for each module of the forged image detection model;

[0026] Step B5 sets the number of training iterations and training parameters. Each iteration repeats steps B2 to B4 until the iteration ends.

[0027] Step B6 sets up a verification step during the training iteration. If the prediction parameters of the forged image detection model meet the given conditions, the current model training parameters are saved.

[0028] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0029] The fine-grained forgery image detection method implemented in this invention can identify different categories of forgery attributes and output corresponding forgery attribute classification prediction values ​​in a hierarchical manner. The training method for the fine-grained forgery image detection model involves designing multiple loss functions to constrain the parameter updates of each module of the detection model. Specifically, it improves the model's forgery region localization performance by designing a localization loss function based on deep metric learning, and improves the model's forgery attribute classification performance by designing a classification loss function based on SoftTripleLoss. The method implemented in this invention can be rapidly applied to related fields of forgery image detection. Attached Figure Description

[0030] Figure 1 This is a flowchart of a fine-grained forged image detection method;

[0031] Figure 2 This is a flowchart of the training method for a fine-grained forged image detection model;

[0032] Figures 3a-3d The classification loss of each level of the HiFi-Net model picture;

[0033] Figure 4 It is the segmentation and localization loss of the HiFi-Net model. picture;

[0034] Figure 5 It is the binary cross-entropy loss of the HiFi-Net model. picture. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the embodiments and accompanying drawings.

[0036] like Figure 1 The image shown is a fine-grained method for detecting forged images, which includes the following steps:

[0037] Step 101: Input the image to be detected into the feature extraction module of the forgery image detection model, and output the feature information of the image to be detected;

[0038] Step 102: Input the feature information and the ground truth value of the mask of the image to be detected into the feature recognition and detection module of the forgery image detection model, and output the fine-grained forgery attribute classification prediction value, prediction mask, and forgery region localization accuracy.

[0039] Step 103: The forged image detection model comprehensively evaluates various prediction indicators and outputs the detection results.

[0040] In step 101 above: the forged image detection model is specifically the deep learning model HiFi-Net, which includes a feature extraction module UHRNet and a feature recognition and detection module NLCDetection. The feature extraction module takes the image to be detected as input and outputs feature information, including four multi-scale fused feature maps F1 to F4 of different sizes. The feature recognition and detection module has a multi-level classifier and a prediction mask generator; its input is the feature maps F1 to F4, and its output is the forged attribute classification prediction value p(y). b |X), Prediction Mask pred_mask.

[0041] In step 102 above: the ground truth of the mask of the image to be detected is specifically the binary image mask of the fake region mask of the image to be detected, where pixel value 0 represents the real region and 1 represents the fake region. The fine-grained fake attribute classification is specifically as follows: level zero - real / fake image; level one: partial region / full image region fake; level two: image editing method / CNN-based / GAN / Diffusion; level three: conditional / unconditional fake method; level four: various fake image generation methods (splicing, copymove, inpainting, face2face, faceswap, StarGANv2, HiSD, DDPM, DDIM). The specific method of outputting the fine-grained fake attribute classification is shown in formula (1), where for the image to be detected X, the logits function of the current level classifier outputs θ. b (X), the current level classifier outputs the predicted value p(y) for each fake attribute under the corresponding category after passing through the softmax function. b |X). The accuracy of fake region localization is obtained by calculating pixel-level accuracy using the predicted mask pred_mask and the ground truth mask mask of the fake region in the image to be detected.

[0042] p(y b |X)=softmax(θ b (X)⊙(1+p(y b-1 |X))) (1)

[0043] In step 103 above: the prediction metrics are specifically pixel-level accuracy and F1 score, image-level accuracy and F1 score, and forgery attribute classification prediction values ​​at each level. The output detection result is whether the image to be detected is a forgery image; if it is a forgery image, the forgery classification is identified, and a prediction mask is output.

[0044] [Example] Fine-grained fake image detection using real images

[0045] The result of the feature extraction module of the detection model after processing the real image is as follows:

[0046]

[0047] The result after feature recognition and detection module of the detection model is as follows:

[0048]

[0049]

[0050] The final prediction result of the detection model is:

[0051]

[0052]

[0053] like Figure 2 The diagram shows the training process for a fine-grained forged image detection model, which includes the following steps:

[0054] Step 201: Create a fine-grained fake image detection dataset;

[0055] Step 202: Input the forged images to be trained in the dataset into the feature extraction module of the forged image detection model, and output a feature set.

[0056] Step 203: Feature set Input the ground truth mask of the forged image to be trained into the feature recognition and detection module of the forged image detection model, and output the fine-grained forged attribute classification prediction value prob and the prediction mask pred_mask;

[0057] Step 204: Construct the loss function Constrain the updating of parameters for each module of the forged image detection model;

[0058] Step 205 sets the number of training iterations and training parameters. Steps 202 to 204 above are repeated for each iteration until the iteration ends.

[0059] Step 206 sets up a verification step in the training iteration. If the prediction parameters of the forged image detection model meet the given conditions, the current model training parameters are saved.

[0060] In step 201 above: the fine-grained fake image detection dataset consists of a training set, a validation set, and a test set. The original images are sourced from open-source datasets, including the MSCOCO dataset, FFHQ dataset, AFHQ dataset, CelebaHQ dataset, LSUN dataset, and YouTube Face dataset. Different fake image generation techniques are applied to the original images to generate fake images, including FaceSwap, Face2Face, StarGANv2, DDPM, DDIM, HiSD, Copymove, Inpainting, and Splicing. The training set contains 10,000 original images and 10,000 fake images; the validation set contains 200 original images and 200 fake images; and the test set contains 1,000 original images and 1,000 fake images.

[0061] In step 202 above: the forged images to be trained in the dataset are specifically the training set images, the forged image detection model is specifically the deep learning model HiFi-Net, and the output feature set is specifically the multi-scale fused feature maps F1 to F4.

[0062] In step 203 above: the ground truth mask of the forged image to be trained is specifically a binary image of the forged region mask of the image to be detected, where pixel value 0 represents the real region and 1 represents the forged region. The fine-grained forged attribute classification prediction value is specifically calculated by formula (1).

[0063] In step 204 above: loss function The sum of the following loss functions: binary cross-entropy (BCE) function Location loss function and classification loss function Formulas (2)-(5) represent the calculation methods of the above loss functions, where the binary cross-entropy function is the binary_cross_entropy function in the PyTorch function toolkit; the localization loss function adopts the deep metric learning method, and a reference center is constructed by the average feature pixel value of all real images in the training set mentioned in step 201. When a pixel in mask M is a non-forged pixel, then the pixel F corresponding to the multi-scale fused feature map described in step 202 is calculated. ij The Euclidean distance between the given boundary value τ and the reference center c is calculated; otherwise, the difference between the given boundary value τ and the above Euclidean distance is calculated and compared with 0. The maximum value of the two is then taken. The sum of the losses of all pixels is the localization loss; the classification loss function is calculated using SoftTripleLoss with an adaptive number of class centers.

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] Wherein, the embedding x of the i-th sample i The corresponding label is y i The final fully connected layer of the network C represents the number of classes, K represents the number of centers per class, d is the embedding dimension, λ is the scaling factor, δ is the inter-class margin, and S i ′ ,c For example x i The relaxed similarity between class c and class c, where γ and τ are scaling factors and N is the number of examples.

[0070] Step 205 above specifically includes: setting the number of training iterations to 10, using a Mini-batch strategy for loading the training set data, and N batch These can be set to positive integers 1, 2, 3, ... . The learning rate for the feature extraction module, feature recognition and detection module of the detection model is 0.0003, and the learning rate for the SoftTripleLoss module of the detection model is 0.01. The Adam optimizer is used, employing a learning strategy that reduces the learning rate when the loss no longer decreases or the accuracy no longer increases. During training, the model selects N from the training set for each training iteration. batch The images are analyzed. The classification loss at each level of the HiFi-Net model can be observed. (like Figures 3a-3d (As shown), segmentation and positioning loss (like Figure 4 (As shown), binary cross-entropy loss (like Figure 5 As shown in the figure, if all the above losses show a decreasing trend, then the HiFi-Net model converges.

[0071] Step 206 above specifically includes: In each training iteration, a validation step is performed every 500 steps. The validation step uses validation set images, which are processed by the HiFi-Net model to obtain fine-grained forgery attribute classification predictions and prediction masks. If the current fine-grained forgery attribute classification prediction accuracy and the pixel-level accuracy of the prediction mask are both improved compared to the previous prediction, then the weights of the model's various parameters are saved.

[0072] While the embodiments disclosed in this invention are as described above, the content is merely for the purpose of facilitating understanding of the invention and is not intended to limit the invention. Any person skilled in the art to which this invention pertains may make any modifications and variations in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the scope of patent protection for this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A method of detecting a fine-grained fake image, characterized by, The method comprises the following steps: Step A1: input the image to be detected into the feature extraction module of the fake image detection model, and output the feature information of the image to be detected; Step A2: input the feature information of the image to be detected and the mask true value into the feature recognition and detection module of the fake image detection model, and output the fine-grained fake attribute classification prediction value, the prediction mask and the fake region positioning accuracy rate; Step A3: evaluate the fine-grained fake attribute classification prediction value, the prediction mask and the fake region positioning accuracy rate, and output the detection result; The fake image detection model is a hierarchical fine-grained deep learning network model, which comprises the feature extraction module UHRNet and the feature recognition and detection module NLCDetection; The input of the feature extraction module UHRNet is the image to be detected, and the output is the feature information; the feature information comprises at least four different size multi-scale fusion feature maps; The feature recognition and detection module comprises a multi-level classification module, a positioning module, and a prediction mask generator, inputs at least four different sizes of multi-scale fusion feature maps, and outputs fine-grained fake attribute classification prediction value p(y b |X) and a prediction mask pred_mask, where X is an image to be detected.

2. The method of claim 1, wherein, The feature information in step A1 is a multi-scale fusion feature matrix set.

3. The method of claim 1, wherein, In step A2, the fake region mask true value of the image to be detected is a binary matrix; the prediction mask is generated by the positioning module, and the prediction mask is a binary matrix with the same shape as the image to be detected; The fake region positioning accuracy rate is obtained by calculating the pixel-level accuracy rate from the prediction mask and the fake region mask true value of the image to be detected.

4. The method of claim 1, wherein the step of detecting a fine-grained forged image is characterized by, The fine-grained fake attribute classification prediction value is generated by the multi-level classification module; each layer of the multi-level classification module is a classifier, and each classifier identifies multiple fake attributes in the corresponding category and outputs the prediction value of each fake attribute in the corresponding category; wherein the generation process of each fake attribute prediction value comprises: The prediction value of the current classifier is taken as the prior value to participate in the prediction value calculation of the next classifier, and the process is repeated until the last classifier is calculated. For the image X to be detected, the logits function output θ b (X) of the current classifier, the prediction value p(y b |X) of each fine-grained fake attribute classification under the corresponding category is output by the softmax function. p(y b |X) = softmax(θ b (X) ⊙(1 + p(y b-1 |X))) (1).

5. The method of fine-grained fake image detection of claim 1, wherein, Step A3 is a comprehensive evaluation of the fine-grained fake attribute classification prediction value, the prediction mask and the fake region positioning accuracy rate by the fake image detection model, and the detection result of the image is obtained; The detection result includes whether the image to be detected is a fake image; if it is a fake image, the fake classification is identified, and the prediction mask is output.

6. A method of training a fake image detection model according to any one of claims 1-5, characterized in that, The method comprises the following steps: Step B1: prepare a fine-grained fake image detection data set; Step B2 input the fake image to be trained in the data set into the feature extraction module of the fake image detection model, and output a feature set Step B3 collect features and the feature recognition and detection module of the fake image detection model input the fake image mask true value mask, output the fine-grained fake attribute classification prediction value p(y b |X), and the predicted mask pred_mask; Step B4 constructing loss function constraining the update of the parameters of the respective modules of the fake image detection model Step B5: set the training iteration number and the training parameters, repeat the above steps B2 to B4 in each iteration until the iteration is completed; Step B6: set a verification link in the training iteration, and save the current model training parameters when the prediction parameters of the fake image detection model meet the given conditions. 7.The method of claim 6, wherein, The fine-grained fake image detection data set in step B1 is composed of a training set, a verification set and a test set. 8.The method of claim 6, wherein, The fake image detection model in step B2 is a deep learning network model, which comprises a feature extraction module and a feature recognition and detection module; The input of the feature extraction module is an image, and the output is a feature set; The input of the feature recognition and detection module is a feature set, and the output is a fine-grained fake attribute classification prediction value and a prediction mask; The feature set is a multi-scale fusion feature map.

Images