A method and system for detecting and locating forged images based on region perception
Through the forged image detection and positioning method based on region perception, the technology of combining encoder-decoder network and classification prediction network is used to solve the problem of forged region positioning in the prior art, and the problem of forged region positioning is realized. The accurate positioning of the forged area and the authenticity of the image improve the accuracy of detection.
Patent Information
- Application Number
- CN202210320007.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-29
AI Technical Summary
The existing deep forgery detection methods mainly focus on the authenticity of images and fail to effectively locate the forgery area. Especially in the case of local forgery of the face area, the hard label selection is poor, making it difficult to accurately identify the forgery area.
Using a region-aware fake image detection and positioning method, the encoder-decoder network and the classification prediction network are combined, and soft labels and mixed loss functions (binary cross entropy loss and IoU loss) are trained to generate a fake area mask. It is combined with data preprocessing such as MTCNN network detecting face areas to realize the positioning and authenticity judgment of the forged area.
The forged area can be accurately positioned and the conversion between input images and labels can be effectively learned at the image level and pixel level, improving the accuracy of forged image detection and authenticity recognition performance.
Smart Images

Figure CN114821706B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for detecting and locating forged images based on region perception. Background Art
[0002] The deepfake detection task is usually regarded as a common binary classification problem, that is, judging the authenticity of the input image. During the model training stage, the current mainstream deepfake detection methods usually use hard labels, that is, 0 / 1 labels, where 0 means that the image is fake (forged) and 1 means that the image is real. However, only part of the forged image is fake. In other words, the forged area only appears in part of the image, usually on the face area, while the rest of the background area is real. Therefore, using hard labels is not the best choice.
[0003] Currently, deep convolutional neural networks have been widely used in the field of deep fake detection and have achieved good performance. However, few studies have focused on the problem of localizing the forged area.
[0004] For different forgery methods, the forged area is actually an important known condition. Taking face-changing forgery as an example, a typical face-changing forgery method consists of the following three steps: 1) Detecting the face area from the input original image; 2) Synthesizing the desired target face; 3) Fusioning the target face into the original image. Therefore, the forged area in the face-changing forgery image is actually the face area detected in the first step. For other forgery methods, as long as the specific type of forgery method is known, the location of the forged area can be clearly determined according to its generation principle. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes a forged image detection and positioning method and system based on region perception, which can not only determine whether a face image is forged or real, but also accurately locate the forged area. The method includes:
[0006] Perform data preprocessing on the input video as input image, and generate soft labels for each input image;
[0007] Inputting the input image into an encoder-decoder network for training, wherein the encoder-decoder network is trained in a supervised manner, wherein both the encoder and the decoder are composed of K convolutional modules connected in series, and the last layer of each convolutional module of the decoder is supervised by soft labels;
[0008] The output image of the encoder is trained through a classification prediction network, wherein the classification prediction network is composed of M convolution modules connected in series, and different classifiers are respectively set after each convolution module, and each classifier is supervised by hard labels.
[0009] Use a hybrid loss function that combines binary cross entropy loss and IoU loss to train the encoder-decoder network and the classification prediction network;
[0010] The image to be detected is input into the trained encoder-decoder network and classification prediction network, and the forged area and authenticity identification result of the image to be detected are detected and output.
[0011] The hybrid loss function L hybrid for:
[0012] L hybrid =δ1L BCE +δ2L IoU
[0013] L BCE =L BCE_map +L BCE_pixel
[0014]
[0015]
[0016]
[0017] Among them, L BCE is the binary cross entropy loss, L IoU is the IoU loss, δ1 and δ2 represent weights, and L BCE_map is the expression of binary cross entropy loss at the image level, L BCE_pixel is the expression of binary cross entropy loss at the pixel level, l (c) represents the binary cross entropy loss of the c-th classifier output in the classification prediction network, C represents the number of classifiers, α c represents the weight of the binary cross entropy loss of each classifier output, s (k) represents the binary cross entropy loss of the output of the kth convolutional module in the decoder, K represents the number of output masks, and β k represents the weight of the binary cross entropy loss output by each convolutional module in the decoder, t (k) represents the IoU loss of the output of the kth convolutional module in the decoder, γ k Represents the weight of the IoU loss output by each convolutional module in the decoder.
[0018] The binary cross entropy loss of a single classifier in the classification prediction network is:
[0019] l=-[y·log(p)+(1-y)·log(1-p)]
[0020] The binary cross entropy loss of a single convolutional module output in the decoder is:
[0021]
[0022] The IoU loss of a single convolutional module output in the decoder is
[0023]
[0024] Among them, y and p represent the true hard label of the input image and the probability value of predicting it to be a real image, respectively. G(m,n)∈{0,1} represents the soft label value of the input image at the pixel point (m,n). S(m,n) represents the probability value predicted by the model that the pixel point (m,n) is a forged area.
[0025] Furthermore, the encoder and decoder are each composed of 4 convolution modules, and the classification prediction network is composed of 3 convolution modules.
[0026] Furthermore, for the M prediction results generated by the classification prediction network during the training process, the principle of minority obeys majority is adopted to judge the authenticity of the input image.
[0027] Furthermore, the data preprocessing of the input video includes:
[0028] Convert the input video into a single frame image;
[0029] Use the MTCNN network to detect the face area and crop it;
[0030] The cropped face images are resized to a uniform size and used as input images.
[0031] Furthermore, the method of using the MTCNN network to detect the face area and perform cropping is specifically as follows: using the MTCNN network to detect the face area, and then expanding the detected face area outward by 1.2 times in width and height directions and then cropping it.
[0032] The present invention also provides a forged image detection and positioning system based on region perception, wherein the system detects forged images and forged regions by using any one of the above methods.
[0033] The present invention makes full use of the known condition of the forged area, and proposes a method and system for forged image detection and positioning based on region perception with a prediction optimization structure, as well as a new hybrid loss for region-aware deep forgery detection, which is specifically composed of a data preprocessing module, a supervised encoder-decoder network, and a classification prediction network. The encoder-decoder network converts the original input image into a grayscale image, and combined with the classification prediction network, it can not only determine whether a face image is forged or real, but also accurately locate the forged area. The hybrid loss function guides the network to learn the conversion between input images and labels at two different levels: image level (map-level) and pixel level (pixel-level) by fusing binary cross entropy (BCE) loss and IoU loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0035] Figure 1 Flowchart of the forged image detection and localization method based on region perception.
[0036] Figure 2 Schematic diagram of the framework of the forged image detection and localization method based on region perception. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0038] In one embodiment, a flowchart of a method for detecting and locating a forged image based on region perception is shown in FIG. Figure 1 As shown in the figure, a framework process of a forged image detection and positioning method based on region perception is as follows Figure 2The network structure is shown in Table 1. The method consists of a data preprocessing module, a supervised encoder-decoder network, and a classification prediction network. The encoder-decoder network transforms the original input image into a grayscale image. Combined with the classification prediction network, it can not only determine whether a face image is forged or real, but also accurately locate the forged area. The hybrid loss function guides the network to learn the conversion between the input image and the label at two different levels, the image level (map-level) and the pixel level (pixel-level) by fusing the binary cross entropy (BCE) loss and the IoU loss. The encoder-decoder network and the classification prediction network are improved from the Resnet-50 network. Each convolutional layer (conv) is followed by a batch normalization layer (BN) and a ReLU activation function.
[0039] Table 1 Network layer structure
[0040]
[0041] The method specifically comprises:
[0042] In the first step, the input video is preprocessed as input images and soft labels are generated for each input image.
[0043] Data preprocessing includes: converting the input video into a single frame image; using the MTCNN network to detect the face area and crop it to reduce the model processing range. In order to take into account potential forgery traces as much as possible, the detected face area can be expanded outward in width and height, such as 1.2 times; the cropped face image is uniformly adjusted to 224×224 size as the input of the network; and soft labels GTL are generated for all input images, namely ground truth label (GTL).
[0044] For a real image, GTL is a grayscale image in which the pixel value of each point is 0, indicating that there is no forged area in the image; for a forged image, a respective GTL is generated according to the difference in the forging method. GTL∈{0,1} is a grayscale image, and the white area in GTL (pixel value is 1) indicates the forged area.
[0045] In the second step, the input image is input into the encoder-decoder network for training. The encoder-decoder network is trained in a supervised manner. Both the encoder and the decoder are composed of K convolutional modules in series, and the last layer of each convolutional module of the decoder is supervised by soft labels.
[0046] The encoder-decoder network is mainly used to predict the mask representation of the forged area from the input image, that is, to locate the forged area. The encoder-decoder architecture can capture both low-level detail information and high-level global context information. Figure 2 As shown in the figure, the encoder consists of convolution module 1-convolution module 4, and the decoder consists of convolution module 8-convolution module 11. The encoder guides the network to learn advanced feature representation by stacking convolution modules and maximum pooling layers; the upsampling operation in the decoder gradually restores the size of the input image, and the role of the convolution module is to enrich the information of the upsampled image, so that the information lost in the pooling process can be learned by the decoder network.
[0047] The encoder-decoder network is trained in a supervised manner. To avoid overfitting, the last layer of each convolutional module of the decoder is supervised by GTL. Therefore, for an input image, the constructed encoder-decoder network generates 4 forged region mask images during training.
[0048] In the third step, the output image of the encoder is trained through a classification prediction network, where the classification prediction network is composed of M convolution modules connected in series. Different classifiers are set after each convolution module, and each classifier is supervised by hard labels.
[0049] The classification prediction network is used to predict the authenticity of the input image. Figure 2 As shown in FIG. 1 , after the image passes through the encoder, a high-level feature expression is obtained. Next, after passing through convolution modules 5 to 7, the receptive field is further increased and the feature space dimension is reduced to extract global semantic features.
[0050] In order to avoid overfitting, different classifiers are set after each convolution module to map the previously learned distributed feature representation to the sample label space, and then predict the authenticity of the image. Each classifier consists of 3 fully connected layers, with 1024, 500 and 2 neuron nodes respectively. The last layer is followed by a softmax layer, whose output corresponds to the probability value of the input image predicted as a true / false sample. Each classifier performs supervised learning through hard labels. Therefore, for an input image, the constructed classification prediction network generates 3 prediction results during the training process, namely S1, S2, and S3. Finally, the principle of minority obeys majority is adopted to judge the authenticity of the input image.
[0051] In the fourth step, a hybrid loss function that combines binary cross entropy loss and IoU loss is used to train the encoder-decoder network and the classification prediction network.
[0052] The present invention defines a new hybrid loss function, which combines the binary cross entropy (BCE) loss and the IoU loss, and guides the network to learn the conversion between input images and labels at two different levels: map level and pixel level. The hybrid loss function is expressed as:
[0053] L hybrid =δ1L BCE +δ2L IoU
[0054] In the above formula, L BCE and L IoU They represent BCE loss and IoU loss functions respectively, δ1 and δ2 represent the weights of each loss function, and δ1=δ2=1.
[0055] (1) BCE loss function
[0056] BCE is the most widely used loss in binary classification and segmentation tasks. In the classification prediction network, BCE loss is used to guide the network to learn the conversion between the input image and the hard label at the image level. The three prediction results S1, S2, and S3 are obtained by classifiers C1, C2, and C3 respectively. The expression of BCE loss at the image level is:
[0057]
[0058] l=-[y·log(p)+(1-y)·log(1-p)]
[0059] In the above formula, l (c) represents the loss of the c-th classifier output, C (C = 3) represents the number of classifiers, α c represents the weight of each loss, y and p represent the true hard label of the input image and the probability value of the model predicting that it is a real image, respectively.
[0060] In the encoder-decoder network, the BCE loss guides the network to learn the conversion between the input image and the soft label at the pixel level. Specifically, the encoder-decoder network generates 4 forged region mask maps by 4 convolution modules during training. The expression of BCE loss at the pixel level is:
[0061]
[0062]
[0063] In the above formula, s (k) represents the loss of the kth convolutional module output, K (K = 4) represents the number of output masks, and β krepresents the weight of each loss, G(m,n)∈{0,1} represents the GTL value of the input image at the pixel point (m,n), and S(m,n) represents the probability value predicted by the model that the pixel point (m,n) is a forged area.
[0064] Ultimately, the total BCE loss consists of two parts:
[0065] L BCE =L BCE_map +L BCE_pixel
[0066] (2) IoU loss function
[0067] IoU loss was originally proposed to measure the similarity between two sets, and later became a standard evaluation metric for object detection and segmentation. In the encoder-decoder network, the present invention samples IoU loss to guide the network to learn the conversion between input images and soft labels at the image level. Specifically, the encoder-decoder network generates 4 forged region mask maps by 4 convolutional modules during training. The expression of IoU loss is:
[0068]
[0069]
[0070] In the above formula, t (k) represents the loss of the output of the kth convolutional module, K (K = 4) represents the number of output masks, and γ k represents the weight of each loss, G(m,n)∈{0,1} represents the GTL value of the input image at the pixel point (m,n), and S(m,n) represents the probability value predicted by the model that the pixel point (m,n) is a forged area.
[0071] The fifth step is to input the image to be detected into the trained encoder-decoder network and classification prediction network, and detect and output the forged area and authenticity identification result of the image to be detected.
[0072] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. According to the above description, many modifications and changes can be made. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is only limited by the claims and their full scope and equivalents.
Claims
1. A forged image detection and localization method based on region perception, It is characterized in that The method comprises: After data preprocessing of the input video, the input image is obtained, and a soft label is generated for each input image; Inputting the input image into an encoder-decoder network for training, wherein the encoder-decoder network is trained in a supervised manner, wherein both the encoder and the decoder are composed of K convolutional modules connected in series, and the last layer of each convolutional module of the decoder is supervised by soft labels; The output image of the encoder is trained through a classification prediction network, wherein the classification prediction network is composed of M convolution modules connected in series, and different classifiers are respectively set after each convolution module, and each classifier is supervised by hard labels. Use a hybrid loss function that combines binary cross entropy loss and IoU loss to train the encoder-decoder network and the classification prediction network; Inputting the image to be detected into the trained encoder-decoder network and classification prediction network, detecting and outputting the forged area and the authenticity identification result of the image to be detected; The hybrid loss function L hybrid for: L hybrid =d 1 L BCE +d 2 L IoU L BCE =L BCE_map +L BCE_pixel Among them, L BCE is the binary cross entropy loss, L IoU is the IoU loss, δ 1 and δ 2 represents the weight, L BCE_map is the expression of binary cross entropy loss at the image level, L BCE_pixel is the expression of binary cross entropy loss at the pixel level, l (c) represents the binary cross entropy loss of the c-th classifier output in the classification prediction network, C represents the number of classifiers, α c represents the weight of the binary cross entropy loss of each classifier output, s (k) represents the binary cross entropy loss of the output of the kth convolutional module in the decoder, K represents the number of output masks, and β k represents the weight of the binary cross entropy loss output by each convolutional module in the decoder, t (k) represents the IoU loss of the output of the kth convolutional module in the decoder, γ k Represents the weight of the IoU loss output by each convolutional module in the decoder; The binary cross entropy loss of a single classifier in the classification prediction network is: l=-[y·log(p)+(1-y)·log(1-p)] The binary cross entropy loss of a single convolutional module output in the decoder is: The IoU loss of a single convolutional module output in the decoder is Among them, y and p represent the true hard label of the input image and the probability value of predicting it to be a real image, respectively. G(m,n)∈{0,1} represents the soft label value of the input image at the pixel point (m,n). S(m,n) represents the probability value predicted by the model that the pixel point (m,n) is a forged area.
2. The method according to claim 1, It is characterized in that The encoder and decoder are both composed of 4 convolution modules, and the classification prediction network is composed of 3 convolution modules.
3. The method according to claim 1, It is characterized in that The classification prediction network obtains M prediction results during the training process, and then uses the principle of minority obeys majority to judge the authenticity of the input image.
4. The method according to any one of claims 1 to 3, It is characterized in that The data preprocessing of the input video includes: Convert the input video into a single frame image; Use the MTCNN network to detect the face area and crop it; The cropped face images are resized to a uniform size and used as input images.
5. The method according to claim 4, It is characterized in that The method of detecting the face area using the MTCNN network and performing cropping is specifically as follows: detecting the face area using the MTCNN network, and then expanding the detected face area outward by 1.2 times in width and height directions and then cropping it.
6. A forged image detection and positioning system based on region perception, It is characterized in that The system adopts the method described in any one of claims 1 to 5 to detect the authenticity and forged area of the forged image.
