Face image forgery detection method and related device

Abstract

The application provides a face image forgery detection method and related equipment. The method comprises: obtaining a preprocessed face image to be tested, and performing conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested; inputting the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model respectively, and outputting a detection result label; in response to determining that the detection result label is a preset first value, the face image to be tested is a fake face image, and a fake mask is generated in the face image to be tested to label a fake area; in response to determining that the detection result label is a preset second value, the face image to be tested is a real face image. The characteristics of high-frequency information change details of a face forgery image, more forgery clues are mined through the frequency domain image of the face image to be tested, and the generalization of the face image forgery detection model improves the detection performance, thereby ensuring the reliability of the detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for detecting forged facial images and related equipment. Background Art

[0002] With the remarkable success of facial forgery technologies, such as deepfakes, face forgery detection has attracted widespread attention. However, most existing deep learning-based detection methods rely on stacked convolutions and engineered networks, which are incapable of learning generalizable feature representations. Consequently, they are unable to effectively detect and identify datasets not previously seen in the training set. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a facial image forgery detection method and related equipment to solve or partially solve the above technical problems.

[0004] Based on the above objectives, the first aspect of the present application provides a method for detecting facial image forgery, comprising:

[0005] Acquire a preprocessed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested;

[0006] Inputting the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model respectively, and outputting a detection result label;

[0007] In response to determining that the detection result label is a preset first value, the face image to be tested is a forged face image, and generating a forged mask in the face image to be tested to mark the forged area;

[0008] In response to determining that the detection result label is a preset second value, the face image to be tested is a real face image.

[0009] Optionally, obtaining a pre-processed face image to be tested includes:

[0010] Obtain the original face image to be tested;

[0011] The original face image to be tested is detected and located by a multi-task convolutional neural network to obtain a face area image;

[0012] The face region is adjusted according to a preset rule to obtain the pre-processed face image to be tested.

[0013] Optionally, the converting the face image to be tested to obtain a frequency domain image of the face image to be tested includes:

[0014] The frequency domain features of the face image to be tested are extracted to obtain a frequency domain map of the face image to be tested.

[0015] Optionally, the face image forgery detection model is pre-trained through the following process:

[0016] Obtaining a pre-processed pre-trained face image set, and performing conversion processing on the pre-trained face image set to obtain a frequency domain atlas of the pre-trained face image set;

[0017] Inputting the pre-trained face image set and the frequency domain atlas of the pre-trained face image set into a pre-built pre-training model respectively, wherein the pre-training model includes a feature extraction model, a pixel-level segmentation supervision model, a local patch relationship supervision model and an image-level classification supervision model, and the local patch relationship supervision model includes an inter-patch consistency supervision model and an inter-patch similarity supervision model;

[0018] Performing feature extraction on the pre-trained face image set and the frequency domain atlas of the pre-trained face image set using the feature extraction model to obtain feature maps;

[0019] Merging all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map, and performing pixel-level segmentation on the merged feature map to obtain a pixel-level segmentation loss function;

[0020] Based on the feature map, an inter-patch consistency loss function is obtained through the inter-patch consistency supervision model. At the same time, based on the feature map, an inter-patch similarity loss function is obtained through the inter-patch similarity supervision model. The inter-patch consistency loss function and the inter-patch similarity loss function are summed to obtain a patch-level relationship supervision loss function.

[0021] Obtaining an image-level classification loss function based on the feature map through the image-level classification supervision model;

[0022] Summing the pixel-level segmentation loss function, the patch-level relationship supervision loss function, and the image-level classification loss function to obtain a hybrid loss function;

[0023] Based on the hybrid loss function, the parameters of the pre-trained model are continuously adjusted until the hybrid loss function is minimized to obtain a trained pre-trained model, and the trained pre-trained model is used as the face image forgery detection model.

[0024] Optionally, merging all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map includes:

[0025] All the feature maps are upsampled using the pixel-level segmentation supervision model to obtain feature maps of the same specification, and all the feature maps of the same specification are merged to obtain a merged feature map.

[0026] Optionally, performing pixel-level segmentation on the merged feature map to obtain a pixel-level segmentation loss function includes:

[0027] Using the pixel-level segmentation supervision model, each pixel of the real face image in the merged feature map is labeled according to a preset first pixel value to obtain a first segmented region, and each pixel of the forged face image in the merged feature map is labeled according to a preset second pixel value to obtain a second segmented region;

[0028] generating a pre-trained forged mask based on the first segmented area and the second segmented area, and marking the forged face image in the merged feature map using the pre-trained forged mask;

[0029] The pixel set segmentation loss function is obtained according to the pre-trained forged mask and the merged feature map.

[0030] Based on the same inventive concept, the second aspect of the present application provides a facial image forgery detection device, comprising:

[0031] A data acquisition module is configured to acquire a pre-processed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested;

[0032] a forgery detection module configured to input the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model, and output a detection result label;

[0033] a first discrimination module configured to, in response to determining that the detection result label is a preset first value, determine that the face image to be tested is a forged face image, and generate a forged mask in the face image to mark the forged area;

[0034] The second discrimination module is configured to determine that, in response to determining that the detection result label is a preset second value, the face image to be tested is a real face image.

[0035] Based on the same inventive concept, the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0036] Based on the same inventive concept, the fourth aspect of this application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0037] As can be seen from the above, the facial image forgery detection method and related equipment provided by the present application convert the facial image to be tested to obtain a frequency domain map of the facial image to be tested, use the high-frequency information of the forged facial image to change the characteristic of detail loss, and mine more forgery trace clues through the frequency domain map of the facial image to be tested. Then, the facial image to be tested and the frequency domain map of the facial image to be tested are respectively input into the trained facial image forgery detection model, and the detection result label is output. The facial image forgery detection model is used to improve the generalization of the detection performance and ensure the reliability of the detection. When the detection result label is a preset first value, the facial image to be tested is a forged facial image, and a forged mask is generated in the facial image to be tested to mark the forged area. When the detection result label is a preset second value, the facial image to be tested is a real facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flowchart of a method for detecting forged facial images according to an embodiment of the present application;

[0040] Figure 2 This is a schematic diagram of a facial image forgery detection framework according to an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of a pre-trained forged mask according to an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a spatial attention model according to an embodiment of the present application;

[0043] Figure 5 A schematic diagram of an interactive attention model according to an embodiment of the present application;

[0044] Figure 6 This is a schematic diagram of the structure of a facial image forgery detection device according to an embodiment of the present application;

[0045] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0047] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0048] Related technologies often use deep learning detection methods that rely on stacked convolutions and designed networks. However, these networks are not capable of learning generalized feature representations and cannot effectively handle the detection of data that has not appeared in the training set.

[0049] The embodiments of the present application provide a facial image forgery detection method and related equipment, which detect an input facial image to be tested and a frequency domain map of the facial image to be tested through a facial image forgery detection model, and output a detection result label to avoid the loss of details due to changes in high-frequency information of the facial forgery image, thereby preventing the inability to mine more forgery trace clues, thereby improving the generalization of detection performance and ensuring the reliability of detection.

[0050] like Figure 1 As shown, the method includes:

[0051] Step 101: Acquire a pre-processed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested.

[0052] In this step, the face image to be tested is an image in RGB color mode, wherein the RGB color mode represents various colors obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other.

[0053] Since forged face images exhibit frequency statistical characteristics that are not obvious from real images, the high-frequency information of the forged image is changed and details are lost. Therefore, the face image to be tested is converted into a frequency domain map of the face image to explore more clues of forgery traces.

[0054] In some embodiments, in step 101, obtaining a pre-processed face image to be tested includes:

[0055] Step A1: Obtain the original face image to be tested.

[0056] Step A2: Detect and locate the face area of ​​the original face image to be tested through a multi-task convolutional neural network to obtain a face area image.

[0057] Step A3: Adjust the face region according to a preset rule to obtain the pre-processed face image to be tested.

[0058] In the above scheme, the original face image to be tested is a single-frame image after video decomposition. The multi-task convolutional neural network (MTCNN) is used to detect and locate the face area of ​​the original face image to be tested to obtain a face area image, wherein the multi-task convolutional neural network consists of a three-layer network architecture. The three-layer network architecture includes: P-Net layer, R-Net layer and O-Net layer. The P-Net layer is a fully convolutional neural network, which includes three convolutional layers; the R-Net layer is a convolutional neural network, which includes three convolutional layers and a fully connected layer; the O-Net layer is a convolutional neural network, which includes four convolutional layers.

[0059] The P-Net layer extracts the coordinates of possible faces for processing in the next layer. Then, the R-Net layer extracts the facial data in the original face image to be tested based on the facial coordinates extracted by the P-Net layer. The erroneous facial data is filtered out by the fully connected layer contained in the R-Net layer. Then, the non-maximum suppression (NMS) algorithm is used to filter and remove the erroneous facial data to obtain filtered facial data. Finally, the O-Net layer extracts the annotation point information in the original face image to be tested. The annotation point information in the original face image to be tested and the filtered facial data are used as the face area image obtained by detection and positioning.

[0060] The face region image is adjusted according to preset rules, for example, the face region is expanded outward by a set multiple along the width and height, and then it is cropped, and the cropped image size is adjusted to a uniform specification to obtain a preprocessed face image to be tested.

[0061] In some embodiments, in step 101, converting the face image to be tested to obtain a frequency domain image of the face image to be tested includes:

[0062] The frequency domain features of the face image to be tested are extracted to obtain a frequency domain map of the face image to be tested.

[0063] In the above scheme, a high-pass filter can be used to process the face image to be tested. The frequency threshold can be set according to the specific situation to suppress the low-frequency part of the face image to be tested that is below the frequency threshold, thereby extracting the frequency domain features and obtaining the frequency domain map of the face image to be tested. Since the high-frequency information of the forged image is changed and the details are lost, more clues of forgery traces can be mined through the frequency domain map of the face image to be tested.

[0064] Step 102: Input the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model, and output a detection result label.

[0065] In this step, detection is performed using a face image forgery detection model, and a detection result label is output, where the detection result label represents a binary classification label. For example, if the detection result label is 1, it indicates that the face image to be tested is a real face image. If the detection result label is 0, it indicates that the face image to be tested is a forged face image. There is no specific limitation on the numerical value of the detection result label representing the binary classification.

[0066] In some embodiments, in step 102, the facial image forgery detection model is pre-trained through the following process:

[0067] Step 1021 , obtaining a pre-processed pre-training face image set, and performing conversion processing on the pre-training face image set to obtain a frequency domain atlas of the pre-training face image set.

[0068] Step 1022: Input the pre-trained face image set and the frequency domain atlas of the pre-trained face image set into a pre-built pre-training model respectively, wherein the pre-training model includes a feature extraction model, a pixel-level segmentation supervision model, a local patch relationship supervision model and an image-level classification supervision model, and the local patch relationship supervision model includes an inter-patch consistency supervision model and an inter-patch similarity supervision model.

[0069] Step 1023 : performing feature extraction on the pre-trained face image set and the frequency domain atlas of the pre-trained face image set respectively through the feature extraction model to obtain feature maps.

[0070] In step 1024, all feature maps are merged using the pixel-level segmentation supervision model to obtain a merged feature map, and the merged feature map is segmented at the pixel level to obtain a pixel-level segmentation loss function.

[0071] Step 1025: obtain an inter-patch consistency loss function based on the feature map through the inter-patch consistency supervision model. At the same time, obtain an inter-patch similarity loss function based on the feature map through the inter-patch similarity supervision model, and sum the inter-patch consistency loss function and the inter-patch similarity loss function to obtain a patch-level relationship supervision loss function.

[0072] Step 1026: Obtain an image-level classification loss function based on the feature map through the image-level classification supervision model.

[0073] Step 1027 , summing the pixel-level segmentation loss function, the patch-level relationship supervision loss function, and the image-level classification loss function to obtain a hybrid loss function.

[0074] Step 1028: Continuously adjust the parameters of the pre-trained model based on the hybrid loss function until the hybrid loss function is minimized, thereby obtaining a trained pre-trained model, and using the trained pre-trained model as the face image forgery detection model.

[0075] In the above scheme, for example Figure 2 As shown, the RGB face image I (i.e., the pre-trained face image set) and the frequency domain map F (i.e., the frequency domain map set of the pre-trained face image set) are input into the pre-training model and processed in two branches. The feature extraction model extracts features from the RGB face image I and the frequency domain map F respectively to obtain feature maps. Among them, the RGB face image I and the frequency domain map F are processed by the spatial attention module respectively, so that the pre-training model focuses on important local areas related to forged face detection. After the RGB face image I and the frequency domain map F are processed by the spatial attention module respectively, they are respectively passed through four convolution modules: conv1_x, conv2_x, conv3_x and conv4_x. Each convolution layer is followed by a BN (Batch Normalization) layer and a ReLU (Linear rectification function) activation function. The network structure of the four convolution modules is shown in the following table:

[0076]

[0077]

[0078] The feature maps obtained by the four convolution modules of the RGB face image I are respectively recorded as and The feature maps obtained by the four convolution modules of image F are respectively recorded as and Among them, the two branches have the same semantic layer (e.g. and and ) are processed by the interactive attention module, and the two features are synergistically fused through the interactive attention module to obtain complementary feature representations.

[0079] The feature map obtained by the pixel-level segmentation supervision model and Upsampling is The same size, and then the three upsampled feature maps are combined with Merge by channel and finally input into the convolution module conv5_x. The output feature map is recorded as (i.e., merged feature maps), with a size of 120×120×3. The size of the feature map is not specifically limited here. Next, a separate forgery mask is generated for each input image, and pixel-level segmentation supervision training is performed. For a real face image, all pixels are considered to be real, and the forgery mask M is defined as a binary image with all pixel values ​​​​zero. For a forged face image, the forged area only appears on the face, so the pixels in the face area (the face area is detected and located using a multi-task convolutional neural network, and then the face area is framed with a rectangular frame) are considered to be false, and the pixel value is set to 1, while the other pixels are real, and the pixel value is set to 0. The forgery mask can be described by the following formula:

[0080]

[0081] Among them, x ij Represents the pixel of the input image at point (i, j). Then, the generated forged mask M is resized to 120×120×3 and denoted as f b , use it as a pixel-level segmentation supervision signal supervision feature map The learning of is used to guide the model to learn more accurate and discriminative forged region-aware features. Here, the pixel-level segmentation loss function is expressed as:

[0082]

[0083] Among them, Figure 3 Shown are examples of forged masks, where the first and third columns are real face images, and the second and fourth columns are forged face images.

[0084] The local patch relationship supervision model considers the correlation between local regions and learns a generalized feature representation, wherein the local patch relationship supervision model includes an inter-patch consistency supervision model and an inter-patch similarity supervision model. Since the RGB face image I and the frequency domain map F are different types of data and usually share the same deception pattern, the inter-patch consistency supervision model is used to perform inter-patch consistency measurement supervision to normalize the output consistency between the RGB face image I and the frequency domain map F, and improve the consistency of the output features in a self-supervised manner. The RGB face image I and the frequency domain map F are respectively obtained through the feature extraction model to obtain the corresponding feature maps, that is, and Will and The space is divided into s×s patches, denoted as c i and v i , where i∈{1,2,...,s 2}, then, c i and v i Flattened to a one-dimensional vector and Then calculate the cosine similarity between the two:

[0085]

[0086] in, The value range is 0 to 1. Higher values ​​indicate patch c i and v i Ideally, It should be close to 1, so we need to construct a full 1 matrix to guide Finally, the inter-patch consistency loss function is expressed as:

[0087]

[0088] The use of inter-similarity supervision model makes the pre-training model more stable, for feature maps patch and The cosine similarity between is calculated as:

[0089]

[0090] in, The value range is 0 to 1. The supervisory signal of H '×W'×C' First, divide M into s×s patches, denoted as m i ∈R (H' / s)×(W' / s)×C', where i∈{1,2,...,s 2 Then, the probability score m' of each patch being forged is obtained by calculating the ratio of the number of false pixels (pixel value = 1) in the patch to the total number of all pixels. i ∈[0,1]. After that, the patch and The relationship supervision signal between is expressed as:

[0091] r i,j =1-(m' i -m' j ) 2

[0092] Among them, r i,j The value range is 0 to 1. It is used to guide the learning of similarity relations between patches. Finally, the inter-patch similarity loss function is expressed as:

[0093]

[0094] The patch-level consistency loss function and the patch-level similarity loss function are summed to obtain the patch-level relationship supervision loss function, that is, the patch-level relationship supervision loss function is expressed as:

[0095] L patch =L patch_1 +L patch_2

[0096] Since the forgery traces are concentrated in the facial area (located in the center of the input image), the feature map is classified into and The patch at the center of the partition is flattened into a one-dimensional vector and passed through two fully connected layers (with 300 and 2 neuron nodes respectively) to form a classifier, obtaining two image-level classification prediction values, which are recorded as and The average of the two is then calculated as the final prediction score. The cross entropy loss function is used for two image-level classification supervisions as follows:

[0097]

[0098]

[0099] Where y represents the real image binary classification label. If the face image is a fake face image, y = 1, otherwise y = 0. The image-level classification loss function is expressed as a combination of two components, namely:

[0100] L classifier =L classifier_1 +L classifier_2

[0101] The pixel-level segmentation loss function, patch-level relationship supervision loss function, and image-level classification loss function are summed to obtain a hybrid loss function. The pre-trained model is trained end-to-end under three types of auxiliary supervision. The hybrid loss function represents the hybrid loss function of the overall training, namely:

[0102] L=λ1*L classifier +λ2*L map +λ3*L patch

[0103] Among them, L classifier represents the image-level classification loss function, L map represents the pixel-level segmentation loss function, L patch represents the patch-level relationship loss function, and λ1~λ3 are the weight factors of these terms, for example, set to λ1=1, λ2=λ3=0.1.

[0104] Iterative training is performed based on the hybrid loss function, so that the parameters of the pre-trained model are continuously adjusted until the hybrid loss function is minimized, thereby obtaining a trained pre-trained model, which is used as a face image forgery detection model.

[0105] In some embodiments, the feature extraction model includes a first branch model, a second branch model and an interactive attention model, wherein the first branch model includes a first spatial attention model and a first convolution model, and the two-branch model includes a second spatial attention model and a second convolution model;

[0106] Step 1023 includes:

[0107] Step 10231: extract important area features from the pre-trained face image set through the first spatial attention model and the first convolution model to obtain a first important area feature map. At the same time, extract important area features from the frequency domain atlas of the pre-trained face image set through the second spatial attention model and the second convolution model to obtain a second important area feature map.

[0108] Step 10232: Use the interactive attention model to complementarily fuse the first important region feature map and the second important region feature map to obtain a fused region feature, and use the fused region feature as the feature map.

[0109] In the above scheme, the first spatial attention model and the second spatial attention model are used to make the pre-trained model focus on the important local areas related to forged face detection (i.e., the first important area feature map and the second important area feature map), wherein the schematic diagram of the spatial attention model is as follows Figure 4As shown. Assume that the input feature map f∈R h×w×c First, f undergoes channel-dimensional average pooling (avg) and maximum pooling (max) operations respectively to obtain two feature maps with a channel number of 1 and a size of h×w×c. After merging the two by channel, a 3×3 convolution layer is used to change the channel number from 2 to 1, followed by a sigmoid function. The obtained feature map S is subjected to a spatial dimension dot multiplication operation with the original input feature map f, and then a 3×3 convolution layer is used.

[0110] The above process can be expressed as follows:

[0111] f avg =F avg (f)

[0112] f max =F max (f)

[0113] S=σ(Conv 3×3 (Concat(f avg ,f max )))

[0114]

[0115] The interactive attention model diagram is as follows Figure 5 As shown. Assume f1∈R h×w×c and f2∈R h×w×c are two feature maps from the same semantic layer in the first and second branch models. The goal of the interactive attention model is to synergistically fuse these two features to obtain complementary feature representations (i.e., feature maps), which are composed of spatial and channel-dimensional attention. First, the features are recalibrated in the spatial domain through a 1×1 convolution and a Sigmoid (S-shaped growth curve) function to create a reweighted matrix S, which is then dot-multiplied with the input feature map in the spatial dimension as shown below:

[0116]

[0117]

[0118] Where M1∈R h×w×c and M2∈R h×w×c Represent the spatial attention maps of f1 and f2 respectively. Represents the spatial dimension dot multiplication operation. Next, M1 and M2 are merged by channel to obtain the feature map V, and then a 1×1 convolution and a 3×3 convolution are used to fuse the two data streams. The process is expressed as:

[0119]

[0120] K=σ(Conv 3×3 (T))

[0121] Where T∈R h×w×2c , K∈R h×w×2c , β represents the batch normalization layer, Represents the ReLU function. Next, K is split into K1∈R by channel. h×w×1 and K2∈R h×w×1 , corresponding to the attention maps of f1 and f2 respectively. Finally, f1 and f2 are point-multiplied with K1 and K2 respectively to highlight the most important local areas in the image.

[0122] In some embodiments, in step 1024, merging all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map includes:

[0123] All the feature maps are upsampled using the pixel-level segmentation supervision model to obtain feature maps of the same specification, and all the feature maps of the same specification are merged to obtain a merged feature map.

[0124] In the above scheme, for example, the feature map obtained by the pixel-level segmentation supervision model and Upsampling is The same size, and then the three upsampled feature maps are combined with Merge by channel and finally input into the convolution module conv5_x. The output feature map is recorded as (i.e., merged feature maps), the size is 120×120×3, and the size of the feature map is not specifically limited here.

[0125] In some embodiments, in step 1024, performing pixel-level segmentation on the merged feature map to obtain a pixel-level segmentation loss function includes:

[0126] Step 10241: Use the pixel-level segmentation supervision model to label each pixel of the real face image in the merged feature map according to a preset first pixel value to obtain a first segmented area, and label each pixel of the forged face image in the merged feature map according to a preset second pixel value to obtain a second segmented area.

[0127] Step 10242: Generate a pre-trained forged mask based on the first segmented area and the second segmented area, and use the pre-trained forged mask to label the forged face image in the merged feature map.

[0128] Step 10243: Obtain the pixel set segmentation loss function according to the pre-trained forged mask and the merged feature map.

[0129] In the above scheme, for example, for a real face image, all pixels are considered real, and the forgery mask M is defined as a binary image with all pixel values ​​​​zero. However, for a forged face image, the forged area only appears on the face, so the pixels in the face area (using a multi-task convolutional neural network to detect and locate the face area, and then frame the face area with a rectangular frame) are considered fake and the pixel value is set to 1, while the other pixels are real and the pixel value is set to 0. The forgery mask can be described by the following formula:

[0130]

[0131] Among them, x ij Represents the pixel of the input image at point (i, j). Then, the generated forged mask M is resized to 120×120×3 and denoted as f b , use it as a pixel-level segmentation supervision signal supervision feature map The learning of is used to guide the model to learn more accurate and discriminative forged region-aware features. Here, the pixel-level segmentation loss function is expressed as:

[0132]

[0133] Step 103 : In response to determining that the detection result label is a preset first value, the face image to be tested is a forged face image, and generating a forged mask in the face image to be tested to mark the forged area.

[0134] In this step, for example, when the detection result label is 0, the face image to be tested is a forged face image, and a forged mask is generated in the face image to be tested to mark the forged area. The preset first value is not specifically limited here.

[0135] Step 104 : In response to determining that the detection result label is a preset second value, the face image to be tested is a real face image.

[0136] In this step, for example, when the detection result label is 1, the face image to be tested is a real face image, and the preset second value is not specifically limited here.

[0137] Through the above scheme, the face image to be tested is converted and processed to obtain the frequency domain map of the face image to be tested, and the high-frequency information of the forged face image is used to change the characteristic of detail loss. More forgery trace clues are mined through the frequency domain map of the face image to be tested, and then the face image to be tested and the frequency domain map of the face image to be tested are respectively input into the trained face image forgery detection model to output the detection result label. Among them, the face image forgery detection model introduces image-level classification supervision, local patch-level relationship supervision and pixel-level segmentation supervision during pre-training, avoiding the problem of poor generalization of extracted features caused by single global binary supervision. The reliability and generalization of the detection performance are further improved through the face image forgery detection model. When the detection result label is a preset first value, the face image to be tested is a forged face image, and a forged mask is generated in the face image to mark the forged area. When the detection result label is a preset second value, the face image to be tested is a real face image.

[0138] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0139] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a facial image forgery detection device.

[0141] refer to Figure 6 , the facial image forgery detection device comprises:

[0142] The data acquisition module 601 is configured to acquire a pre-processed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested;

[0143] a forgery detection module 602 configured to input the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model, and output a detection result label;

[0144] A first identification module 603 is configured to, in response to determining that the detection result label is a preset first value, determine that the face image to be tested is a forged face image, and generate a forged mask in the face image to mark the forged area;

[0145] The second discrimination module 604 is configured to determine that, in response to determining that the detection result label is a preset second value, the face image to be tested is a real face image.

[0146] In some embodiments, the data acquisition module 601 is specifically configured to:

[0147] Obtain the original face image to be tested;

[0148] The original face image to be tested is detected and located by a multi-task convolutional neural network to obtain a face area image;

[0149] The face region is adjusted according to a preset rule to obtain the pre-processed face image to be tested.

[0150] In some embodiments, the data acquisition module 601 is further specifically configured to:

[0151] The frequency domain features of the face image to be tested are extracted to obtain a frequency domain map of the face image to be tested.

[0152] In some embodiments, the facial image forgery detection apparatus further includes a model training module, including:

[0153] a data acquisition unit configured to acquire a pre-processed pre-trained face image set and perform conversion processing on the pre-trained face image set to obtain a frequency domain atlas of the pre-trained face image set;

[0154] a data input unit configured to input the pre-trained face image set and the frequency domain atlas of the pre-trained face image set into a pre-built pre-training model, respectively, wherein the pre-training model includes a feature extraction model, a pixel-level segmentation supervision model, a local patch relationship supervision model, and an image-level classification supervision model, and the local patch relationship supervision model includes an inter-patch consistency supervision model and an inter-patch similarity supervision model;

[0155] a feature extraction unit configured to perform feature extraction on the pre-trained face image set and the frequency domain atlas of the pre-trained face image set respectively through the feature extraction model to obtain a feature map;

[0156] a merging processing unit configured to merge all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map, and perform pixel-level segmentation on the merged feature map to obtain a pixel-level segmentation loss function;

[0157] A first loss function acquisition unit is configured to obtain an inter-patch consistency loss function based on the feature map through the inter-patch consistency supervision model, and at the same time, obtain an inter-patch similarity loss function based on the feature map through the inter-patch similarity supervision model, and sum the inter-patch consistency loss function and the inter-patch similarity loss function to obtain a patch-level relationship supervision loss function;

[0158] A second loss function acquisition unit is configured to obtain an image-level classification loss function based on the feature map through the image-level classification supervision model;

[0159] a hybrid loss function acquisition unit, configured to sum the pixel-level segmentation loss function, the patch-level relationship supervision loss function, and the image-level classification loss function to obtain a hybrid loss function;

[0160] The iterative training unit is configured to continuously adjust the parameters of the pre-trained model based on the hybrid loss function until the hybrid loss function is minimized, thereby obtaining a trained pre-trained model, and using the trained pre-trained model as the face image forgery detection model.

[0161] In some embodiments, the feature extraction model includes a first branch model, a second branch model and an interactive attention model, wherein the first branch model includes a first spatial attention model and a first convolution model, and the two-branch model includes a second spatial attention model and a second convolution model;

[0162] The feature extraction unit is specifically configured as follows:

[0163] Extracting important region features from the pre-trained face image set using the first spatial attention model and the first convolutional model to obtain a first important region feature map, and extracting important region features from the frequency domain atlas of the pre-trained face image set using the second spatial attention model and the second convolutional model to obtain a second important region feature map;

[0164] The interactive attention model is used to complementarily fuse the first important region feature map and the second important region feature map to obtain a fused region feature, and the fused region feature is used as the feature map.

[0165] In some embodiments, the merging processing unit is specifically configured to:

[0166] All the feature maps are upsampled using the pixel-level segmentation supervision model to obtain feature maps of the same specification, and all the feature maps of the same specification are merged to obtain a merged feature map.

[0167] In some embodiments, the merging processing unit is further specifically configured to:

[0168] Using the pixel-level segmentation supervision model, each pixel of the real face image in the merged feature map is labeled according to a preset first pixel value to obtain a first segmented region, and each pixel of the forged face image in the merged feature map is labeled according to a preset second pixel value to obtain a second segmented region;

[0169] generating a pre-trained forged mask based on the first segmented area and the second segmented area, and marking the forged face image in the merged feature map using the pre-trained forged mask;

[0170] The pixel set segmentation loss function is obtained according to the pre-trained forged mask and the merged feature map.

[0171] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0172] The apparatus of the above embodiment is used to implement the corresponding facial image forgery detection method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0173] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the facial image forgery detection method described in any of the above embodiments is implemented.

[0174] Figure 7 : A more specific hardware structure diagram of an electronic device provided in this embodiment is shown. The device may include: a processor 701, a memory 702, an input / output interface 703, a communication interface 704, and a bus 705. The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are communicatively connected to each other within the device via the bus 705.

[0175] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0176] The memory 702 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and called and executed by the processor 701.

[0177] The input / output interface 703 is used to connect to the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0178] The communication interface 704 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0179] The bus 705 comprises a pathway for transmitting information between various components of the device (eg, the processor 701 , the memory 702 , the input / output interface 703 , and the communication interface 704 ).

[0180] It should be noted that although the above device only shows the processor 701, the memory 702, the input / output interface 703, the communication interface 704, and the bus 705, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0181] The electronic device of the above embodiment is used to implement the corresponding facial image forgery detection method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0182] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the facial image forgery detection method described in any of the above embodiments.

[0183] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0184] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the facial image forgery detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0185] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0186] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0187] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0188] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A method for detecting forged facial images, characterized in that: include: Acquire a preprocessed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested; Inputting the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model respectively, and outputting a detection result label; In response to determining that the detection result label is a preset first value, the face image to be tested is a forged face image, and generating a forged mask in the face image to be tested to mark the forged area; In response to determining that the detection result label is a preset second value, the face image to be tested is a real face image; The face image forgery detection model is pre-trained through the following process: Obtaining a pre-processed pre-trained face image set, and performing conversion processing on the pre-trained face image set to obtain a frequency domain atlas of the pre-trained face image set; Inputting the pre-trained face image set and the frequency domain atlas of the pre-trained face image set into a pre-built pre-training model respectively, wherein the pre-training model includes a feature extraction model, a pixel-level segmentation supervision model, a local patch relationship supervision model and an image-level classification supervision model, and the local patch relationship supervision model includes an inter-patch consistency supervision model and an inter-patch similarity supervision model; Performing feature extraction on the pre-trained face image set and the frequency domain atlas of the pre-trained face image set using the feature extraction model to obtain feature maps; Merging all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map, and performing pixel-level segmentation on the merged feature map to obtain a pixel-level segmentation loss function; Based on the feature map, an inter-patch consistency loss function is obtained through the inter-patch consistency supervision model. At the same time, based on the feature map, an inter-patch similarity loss function is obtained through the inter-patch similarity supervision model. The inter-patch consistency loss function and the inter-patch similarity loss function are summed to obtain a patch-level relationship supervision loss function. Obtaining an image-level classification loss function based on the feature map through the image-level classification supervision model; Summing the pixel-level segmentation loss function, the patch-level relationship supervision loss function, and the image-level classification loss function to obtain a hybrid loss function; Based on the hybrid loss function, the parameters of the pre-trained model are continuously adjusted until the hybrid loss function is minimized to obtain a trained pre-trained model, and the trained pre-trained model is used as the face image forgery detection model.

2. The method according to claim 1, characterized in that The step of obtaining the pre-processed face image to be tested comprises: Obtain the original face image to be tested; The original face image to be tested is detected and located by a multi-task convolutional neural network to obtain a face area image; The face region is adjusted according to a preset rule to obtain the pre-processed face image to be tested.

3. The method according to claim 1, characterized in that The converting process of the face image to be tested to obtain a frequency domain image of the face image to be tested includes: The frequency domain features of the face image to be tested are extracted to obtain a frequency domain map of the face image to be tested.

4. The method according to claim 1, wherein The feature extraction model includes a first branch model, a second branch model and an interactive attention model, wherein the first branch model includes a first spatial attention model and a first convolution model, and the two-branch model includes a second spatial attention model and a second convolution model; The step of extracting features from the pre-trained face image set and the frequency domain atlas of the pre-trained face image set using the feature extraction model to obtain feature maps comprises: Extracting important region features from the pre-trained face image set using the first spatial attention model and the first convolutional model to obtain a first important region feature map, and extracting important region features from the frequency domain atlas of the pre-trained face image set using the second spatial attention model and the second convolutional model to obtain a second important region feature map; The interactive attention model is used to complementarily fuse the first important region feature map and the second important region feature map to obtain a fused region feature, and the fused region feature is used as the feature map.

5. The method according to claim 1, wherein The method of merging all feature maps using the pixel-level segmentation supervision model to obtain a merged feature map includes: All the feature maps are upsampled using the pixel-level segmentation supervision model to obtain feature maps of the same specification, and all the feature maps of the same specification are merged to obtain a merged feature map.

6. The method according to claim 1, wherein: The pixel-level segmentation of the merged feature map to obtain a pixel-level segmentation loss function includes: Using the pixel-level segmentation supervision model, each pixel of the real face image in the merged feature map is labeled according to a preset first pixel value to obtain a first segmented region, and each pixel of the forged face image in the merged feature map is labeled according to a preset second pixel value to obtain a second segmented region; generating a pre-trained forged mask based on the first segmented area and the second segmented area, and marking the forged face image in the merged feature map using the pre-trained forged mask; The pixel-level segmentation loss function is obtained according to the pre-trained forged mask and the merged feature map.

7. A facial image forgery detection device using the facial image forgery detection method according to any one of claims 1 to 6, characterized in that: include: A data acquisition module is configured to acquire a pre-processed face image to be tested, and perform conversion processing on the face image to be tested to obtain a frequency domain image of the face image to be tested; a forgery detection module configured to input the face image to be tested and the frequency domain image of the face image to be tested into a pre-trained face image forgery detection model, and output a detection result label; a first discrimination module configured to, in response to determining that the detection result label is a preset first value, determine that the face image to be tested is a forged face image, and generate a forged mask in the face image to mark the forged area; The second discrimination module is configured to determine that, in response to determining that the detection result label is a preset second value, the face image to be tested is a real face image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images