Detection model training method, detection method, device, equipment and medium

By integrating RGB and frequency domain information into the face synthesis graph detection model, the detection accuracy is improved by using instruction fine-tuning technology, solving the problem of insufficient detection accuracy of face synthesis graph in the existing technology, and achieving more effective defense against face synthesis attack.

CN117746515BActive Publication Date: 2024-11-29BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311766228.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-20
Publication Date
2024-11-29
Estimated Expiration
2043-12-20

AI Technical Summary

Technical Problem

The existing technology cannot meet the current accuracy requirements for face synthesis map detection, resulting in the inability to effectively defend face synthesis attacks.

Method used

By using the training data set of RGB face images to train the pre-trained model, and inserting learnable instruction fine-tuning parameters into the backbone network, the training data set of the frequency domain face images is used for fine-tuning instructions, and fuse the frequency domain information to improve detection accuracy .

Benefits of technology

It improves the accuracy and generalization of face synthesis map detection, effectively defends face synthesis attacks, and improves the defense effect of the detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117746515B_ABST
    Figure CN117746515B_ABST
Patent Text Reader

Abstract

The present disclosure provides a detection model training method, detection method, device, equipment and medium, which relate to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, face recognition, etc., and can be applied to scenarios such as smart cities and smart finance. The specific implementation scheme is: using the training data set of RGB face images, training to obtain a pre-trained model for detecting face synthesis images, wherein the pre-trained model includes an embedding layer, a backbone network and a classification layer; obtaining the frequency domain face image corresponding to the RGB face image, and obtaining the training data set of the frequency domain face image; inserting a learnable instruction fine-tuning parameter into the backbone network, and using the training data set of the frequency domain face image to perform instruction fine-tuning on the pre-trained model into which the instruction fine-tuning parameter is inserted, to obtain a face synthesis image detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, and face recognition, and can be applied to scenarios such as smart cities and smart finance. It specifically relates to a training method, detection method, device, equipment, and medium for a detection model. Background Art

[0002] Face composite image detection is to distinguish whether an image containing a face has been tampered with. It is a basic component module of the face recognition system and ensures the security of the face recognition system. For example, face composite image detection is applied in many scenarios such as attendance, access control, security and financial payment.

[0003] The accuracy of face synthesis image detection directly affects the effectiveness of defending against face synthesis attacks in various fields. However, the current existing technology still cannot meet the current accuracy requirements for face synthesis image detection. Summary of the invention

[0004] The present disclosure provides a detection model training method, detection method, device, equipment and medium.

[0005] According to one aspect of the present disclosure, a method for training a face composition image detection model is provided, comprising:

[0006] Using a training data set of RGB face images, a pre-trained model for detecting a composite face image is trained, wherein the pre-trained model includes an embedding layer, a backbone network, and a classification layer;

[0007] Obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image;

[0008] A learnable instruction fine-tuning parameter is inserted into the backbone network, and the pre-trained model into which the instruction fine-tuning parameter is inserted is fine-tuned using the training data set of the frequency domain face image to obtain a face synthesis image detection model.

[0009] According to another aspect of the present disclosure, a method for detecting a composite face image is provided, comprising:

[0010] Inputting the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained according to the training method for the face composite image detection model described in any embodiment of the present disclosure;

[0011] Obtaining a first prediction value output by the face composition image detection model;

[0012] If the first prediction value is higher than a first threshold, it is determined that the RGB image to be detected is a real face image.

[0013] According to another aspect of the present disclosure, a training device for a face composition image detection model is provided, comprising:

[0014] A pre-training module, used to train a pre-training model for detecting a composite face image using a training data set of RGB face images, wherein the pre-training model includes an embedding layer, a backbone network, and a classification layer;

[0015] A frequency domain image conversion module is used to obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image;

[0016] The instruction fine-tuning module is used to insert learnable instruction fine-tuning parameters into the backbone network, and use the training data set of the frequency domain face image to perform instruction fine-tuning on the pre-trained model into which the instruction fine-tuning parameters are inserted, so as to obtain a face synthesis image detection model.

[0017] According to another aspect of the present disclosure, a face composition image detection device is provided, comprising:

[0018] An input module, used for inputting the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained by the training device for the face composite image detection model according to any embodiment of the present disclosure;

[0019] An acquisition module, used to acquire a first prediction value output by the face synthesis image detection model;

[0020] A detection module is used to determine that the RGB image to be detected is a real face image if the first prediction value is higher than a first threshold.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the face synthesis image detection model and the face synthesis image detection method described in any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the training method of a face synthesis image detection model and the face synthesis image detection method described in any embodiment of the present disclosure.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0028] Figure 1 is a schematic diagram of a training method for a face synthesis image detection model according to an embodiment of the present disclosure;

[0029] Figure 2 is a schematic diagram of another method for training a face composite image detection model according to an embodiment of the present disclosure;

[0030] Figure 3 is a schematic diagram of another method for training a face composite image detection model according to an embodiment of the present disclosure;

[0031] Figure 4 is a schematic diagram of a method for detecting a synthetic face image according to an embodiment of the present disclosure;

[0032] Figure 5 is a schematic diagram of another method for detecting a synthetic face image according to an embodiment of the present disclosure;

[0033] Figure 6 is a structural schematic diagram of a training device for a face composition image detection model according to an embodiment of the present disclosure;

[0034] Figure 7 is a structural schematic diagram of a face composition image detection device according to an embodiment of the present disclosure;

[0035] Figure 8 It is a block diagram of an electronic device used to implement the training method of the face synthesis image detection model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] Figure 1This is a schematic diagram of a training method for a face composite image detection model according to an embodiment of the present disclosure. This embodiment can be applied to the case of detecting whether a face in an image is a composite image. It relates to the field of artificial intelligence technology, especially to computer vision, deep learning, face recognition and other technical fields, and can be applied to scenarios such as smart cities and smart finance. The method can be executed by a training device for a face composite image detection model, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device, a server or an intelligent terminal. Figure 1 As shown, the method specifically includes the following:

[0038] S101. Using a training data set of RGB face images, a pre-trained model for detecting a composite face image is trained, wherein the pre-trained model includes an embedding layer, a backbone network, and a classification layer.

[0039] S102: Obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image.

[0040] S103, inserting learnable instruction fine-tuning parameters into the backbone network, using a training data set of frequency domain face images to perform instruction fine-tuning on a pre-trained model with the inserted instruction fine-tuning parameters, and obtaining a face synthesis image detection model.

[0041] Specifically, the training method of the embodiment of the present disclosure includes a two-stage training process. The first stage is pre-training, and the second stage is instruction fine-tuning. The training data used in the pre-training stage is RGB face image data, and the training data used in the instruction fine-tuning stage is the frequency domain face image corresponding to the RGB face image.

[0042] The training dataset of RGB face images includes real RGB face images and synthetic RGB face images. Real RGB face images are used as positive samples, and synthetic RGB face images are used as negative samples. During pre-training, the RGB face images in the training dataset are input into the pre-training model, and the embedding vector of the RGB face image is obtained through the embedding layer. The backbone network extracts features of the RGB face image based on the embedding vector to obtain the image features of the RGB face image. Then the classification layer classifies the image features and outputs the prediction result of whether the RGB face image is a real face image. The binary cross entropy loss is calculated based on the prediction result and the pre-labeled true value, and the network parameters in the model are optimized through this loss.

[0043] Since the frequency domain information of the image can also assist in the judgment of the detected face composite image, therefore, in the embodiment of the present disclosure, after obtaining the pre-trained model, a training data set of the frequency domain face image corresponding to the RGB face image is further obtained, and the pre-trained model is fine-tuned using the training data set of the frequency domain face image. Specifically, firstly insert the learnable instruction fine-tuning parameters into the backbone network, and then input the training data set of the frequency domain face image into the pre-trained model into which the instruction fine-tuning parameters are inserted, and the instruction fine-tuning parameters are optimized by supervised training. Compared with the backbone network in the pre-trained model, the backbone network of the face composite image detection model obtained after instruction fine-tuning has an additional instruction fine-tuning parameter in its network parameters, so that when the face composite image detection model after instruction fine-tuning performs face composite image detection on the input RGB face image, the frequency domain information of the RGB face image can be considered at the same time, thereby improving the detection accuracy.

[0044] It should be noted that some existing technologies do not take into account the information brought by the frequency domain of the image, resulting in poor detection accuracy. Some existing technologies use the method of fusing RGB image features and frequency domain features or voting the results for detection, resulting in excessive reliance on frequency domain information. However, since the frequency domain information in the image is not as rich as the RGB information, the detection accuracy of over-reliance on frequency domain information is also not ideal. In the technical solution of the embodiment of the present disclosure, the training data set of RGB face images is first used for pre-training, and then the training data set of frequency domain face images corresponding to the RGB face images is used to fine-tune the pre-trained model. The frequency domain information is fused by fine-tuning the instructions, so that the frequency domain information of the image can also assist in the judgment of the face synthesis image detection, but it does not rely too much on the frequency domain information, thereby improving the accuracy and generalization of the detection, effectively preventing face synthesis attacks, and improving the defense effect of face synthesis image detection.

[0045] In one embodiment, ViT (Vision Transformer) can be used as the backbone network of the model for image feature extraction. The backbone network may include, for example, multiple encoder layers, and therefore, specifically, learnable instruction fine-tuning parameters may be inserted into at least one encoder layer of the backbone network. For example, if an instruction fine-tuning parameter is inserted into the first encoder layer of the backbone network, the inserted instruction fine-tuning parameter and the embedding vector output by the embedding layer are used as the basis for feature encoding of the first encoder layer; if an instruction fine-tuning parameter is inserted into any target encoder layer of the backbone network, the inserted instruction fine-tuning parameter and the output of the previous layer of the target encoder layer are used as the basis for feature encoding of the target encoder layer. In addition, the instruction fine-tuning parameter may be inserted into the backbone network in the form of a vector. When inserting the instruction fine-tuning parameter, the embodiment of the present disclosure does not limit the vector length of the instruction fine-tuning parameter, and may be specifically configured according to the structure of the backbone network and the needs of the actual scenario.

[0046] In one embodiment, during the instruction fine-tuning process, the parameters of the embedding layer and other parameters in the backbone network except the instruction fine-tuning parameters can be solidified. In this way, the training results of the pre-trained model in the first stage can be retained, and then the instruction fine-tuning process can be used to achieve the combination of frequency domain information as a supplement without losing RGB image information as much as possible, so as to improve the detection accuracy of the face synthesis image detection model.

[0047] In the process of fine-tuning the instructions, a binary classification supervision method can be used for training. Specifically, it includes: obtaining the first detection result of the pre-trained model for the frequency domain face image after the instruction fine-tuning parameters are inserted; according to the first detection result and the pre-labeled true value, the pre-trained model after the instruction fine-tuning parameters are inserted is subjected to binary classification supervision. That is, the frequency domain face image is input into the pre-trained model after the instruction fine-tuning parameters are inserted, the model is used for prediction and the first detection result of its output is obtained, and then the loss is calculated according to the first detection result and the pre-labeled true value, such as the binary cross entropy loss, and the model is supervised by the loss to optimize the model parameters so that the model has the ability to distinguish and synthesize face images, and the frequency domain information can also be considered as an auxiliary to improve the training effect of the model.

[0048] In addition, while the binary classification is supervised, the process of instruction fine-tuning can also be supervised simultaneously by feature distillation. Specifically, it includes: obtaining a pre-trained model without inserting instruction fine-tuning parameters, and a second detection result of the RGB face image corresponding to the frequency domain face image; according to the first detection result and the second detection result, the pre-trained model after inserting the instruction fine-tuning parameters is feature distilled. That is, each RGB face image and the corresponding frequency domain face image can form a training data pair. The second detection result is the detection result of the RGB face image by the pre-trained model without inserting the instruction fine-tuning parameters, and the first detection result is the detection result of the frequency domain face image in the training data pair by the pre-trained model after inserting the instruction fine-tuning parameters. The loss is calculated according to the first detection result and the second detection result, such as the L1 loss, and the model is supervised by the loss to optimize the model parameters. In this way, the original features of the RGB image are not forgotten by feature distillation.

[0049] As an embodiment of the present disclosure, the process of instruction fine-tuning adopts binary classification supervision and feature distillation for simultaneous supervision, so that the original features of the RGB image are not forgotten. At the same time, the face synthesis image detection model after instruction fine-tuning can not only obtain frequency domain features as auxiliary information on the basis of identifying RGB face image features, but also does not rely on frequency domain information, thereby improving the accuracy of the model in predicting whether the face image is a real image.

[0050] Figure 2FIG. 1 is a schematic diagram of another method for training a face composite image detection model according to an embodiment of the present disclosure. This embodiment is further optimized on the basis of the above embodiment. Figure 2 As shown, the method specifically includes the following:

[0051] S201, performing facial key point detection on an RGB original image data set containing human faces, and obtaining facial key point coordinate values ​​of each RGB original image.

[0052] S202: performing face alignment on the target face in the corresponding RGB original image according to the coordinate values ​​of the facial key points, and determining the center position and rotation angle of the target face.

[0053] S203, according to the center position and the rotation angle, an image of the target face is captured from the RGB original image according to a set size through affine transformation.

[0054] S204: Obtain a training data set of RGB face images according to the image of the target face of each RGB original image.

[0055] S205. Using a training data set of RGB face images, a pre-trained model for detecting a composite face image is trained, wherein the pre-trained model includes an embedding layer, a backbone network, and a classification layer.

[0056] S206: Obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image.

[0057] S207, inserting learnable instruction fine-tuning parameters into the backbone network, and using the training data set of frequency domain face images to perform instruction fine-tuning on the pre-trained model with the inserted instruction fine-tuning parameters, to obtain a face synthesis image detection model.

[0058] Among them, the RGB original image contains a face image, but the face image may have an incorrect posture. In order to improve the effect of model training, it is necessary to preprocess the RGB original image dataset and obtain RGB face images with correct posture as training data for the model.

[0059] First, obtain the coordinate values ​​of the facial key points of each RGB original image. In one embodiment, a pre-trained facial key point recognition model can be used to obtain the coordinate values ​​of the facial key points of each input RGB original image; in another embodiment, a pre-trained face recognition model can be used to perform face detection on the RGB original image containing the face to obtain the face area image, and then the pre-trained key point recognition model can be used to perform face key point recognition on the face area image to obtain the coordinate values ​​of the facial key points. The above-mentioned model for identifying facial areas and facial key points can be implemented using existing technologies and will not be described in detail here.

[0060] Then, the target face in the corresponding RGB original image is aligned according to the coordinate values ​​of the facial key points to determine the center position and rotation angle of the target face. That is, the center position and rotation angle of the area where the target face is located are determined in the RGB original image using the coordinate values ​​of the facial key points. Then, according to the center position and rotation angle, the image of the target face is intercepted from the RGB original image according to the set size through affine transformation. Finally, according to the image of the target face in each RGB original image, a training data set of RGB face images is obtained. The face image with incorrect posture in the RGB original image can be transformed to the correct position through affine transformation. At the same time, when intercepting the image, it is intercepted according to the set size, so that the RGB face images in the training data set have a uniform size, which is convenient for the model to process and improve the training effect.

[0061] In addition, before obtaining the training data set of RGB face images, each pixel value of the target face image can be normalized. For example, the pixel value of each pixel is first subtracted by 128 and then divided by 256, so that the pixel value of each pixel is between [-0.5, 0.5]. The deviation of model processing can be reduced through normalized training data. Then, data enhancement is performed on the normalized image to expand the training data, and finally a training data set of RGB face images with correct face posture, uniform size and normalization is obtained, which prepares data for improving the training effect of the model.

[0062] Figure 3 It is a schematic diagram of another training method of a face synthesis image detection model according to an embodiment of the present disclosure. As shown in the figure, in the technical solution of the embodiment of the present disclosure, first, an RGB face image containing a real face is used as a positive sample, and a synthetic face image after face replacement is used as a negative sample, and a training data set of RGB face images is formed. At the same time, all samples are Fourier transformed to obtain corresponding frequency domain images. An RGB face image and its corresponding frequency domain image can form an RGB face image-frequency domain image pair.

[0063] Then, image processing is performed on each image in the training set. First, the face is detected by the face detection model to obtain the face area image, and then the key point coordinates of the face in the face area image are detected by the RGB face key point detection model. For example, 72 face key point coordinates can be obtained, which are (x1, y1)...(x72, y72). Then, the target face of the RGB image is aligned according to the key point coordinates of the face, and the image containing only the face area is intercepted through affine transformation and adjusted to the same size, such as 224x224.

[0064] After the face alignment operation, further preprocessing can be performed, including normalization and data enhancement. That is, firstly, the RGB face image regions are normalized, and then the normalized images are subjected to random data enhancement, such as random flipping, random scaling, color perturbation and other data enhancement methods, to enrich the training data set.

[0065] The feature extraction network uses ViT-base as the backbone network for face synthesis image detection. The training process has two stages. The first stage is the pre-training stage. The RGB face images in the training data set are used for model training. The token output by the network at the end (such as cls_token_i) is used for prediction. The binary cross entropy loss is calculated based on the prediction result and the true value of the annotation, thereby optimizing the network. Through the pre-training in the first stage, the model has the ability to identify RGB face synthesis images, and the pre-trained model M is obtained. The second stage is the instruction fine-tuning stage. On the basis of M, prompt (instruction fine-tuning parameter) is added to obtain M', and the embedded layer of the trained M and the trained parameters in the backbone network are fixed. The frequency domain image is added to train the model after the prompt is inserted, and the token at the end of the network (such as cls_token_f) is used for prediction. Binary supervision and feature distillation are used for supervision at the same time. Among them, the binary supervision is to take the cls_token_f output by the network at the end for prediction, and calculate the binary cross entropy loss based on the prediction result and the true value of the annotation. The specific method of feature distillation is: input the frequency domain image into M' (insert prompt) to get cls_token_f, input the corresponding RGB face image into M (no prompt inserted) to get cls_token_i, and calculate the L1 loss based on the prediction results of cls_token_i and cls_token_f to achieve feature distillation.

[0066] It can be seen that the technical solution of the embodiment of the present disclosure, after completing the RGB image training, uses prompt tuning to fuse the frequency domain information, and combines the frequency domain supplementary information to supervise the model while minimizing the loss of RGB image information, so that the detection accuracy and generalization of the final face synthesis image detection model can be improved, effectively preventing face synthesis attacks and improving the defense effect of the face synthesis image detection algorithm.

[0067] Figure 4is a schematic diagram of a method for detecting a composite face image according to an embodiment of the present disclosure. This embodiment can be applied to detecting whether a face in an image is a composite image, and relates to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, face recognition, etc. The method can be executed by a composite face image detection device, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device, a server, or an intelligent terminal. Figure 4 As shown, the method specifically includes the following:

[0068] S401. Inputting the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained according to the training method for the face composite image detection model described in any embodiment of the present disclosure.

[0069] S402: Obtain a first prediction value output by a face synthesis image detection model.

[0070] S403: If the first prediction value is higher than the first threshold, determine that the RGB image to be detected is a real face image.

[0071] The RGB image to be detected may be an image captured after face area detection, face key point detection, face alignment and affine transformation, so as to improve the subsequent detection accuracy.

[0072] The RGB image to be detected is input into the face composite image detection model. The face composite image detection model extracts features from the RGB image to be detected and predicts the probability that it is a real face image to obtain a first prediction value. The first threshold is a pre-configured threshold. If the first prediction value is higher than the first threshold, it is determined that the RGB image to be detected is a real face image. Since the face composite image detection model is trained according to the training method of the face composite image detection model described in any embodiment of the present disclosure, it can not only use the information of the RGB image for prediction, but also use the frequency domain in the image as auxiliary information for prediction. Therefore, the accuracy of the final detection result can be improved.

[0073] In one implementation, a second threshold is also pre-configured, and the second threshold is lower than the first threshold. Then, if the first predicted value is lower than the second threshold, it is determined that the RGB image to be detected is a synthetic face image. Additionally, if the first predicted value is between the second threshold and the first threshold, the frequency-domain image to be detected of the RGB image to be detected is further obtained, the frequency-domain image to be detected is input into the face composite image detection model, and the second predicted value output by the face composite image detection model is obtained. According to the second predicted value and the third threshold, it is determined whether the RGB image to be detected is a real face image. That is, when an accurate prediction result cannot be obtained based on the RGB image, prediction is performed based on the corresponding frequency-domain image and compared with the third threshold. If it is higher than the third threshold, it is determined that the RGB image to be detected is a real face image; otherwise, it is a synthetic face image.

[0074] Figure 5 is a schematic diagram of another face composite image detection method according to an embodiment of the present disclosure. As Figure 5 shown, when performing face composite image detection, a collaborative inference method is adopted. The pre-processed RGB image is input into the trained face composite image detection model M', and the real-person prediction score R is obtained. High and low thresholds are set as t1 and t2 respectively. If R > t1, it is determined as a real person; if R < t2, it is determined as a synthetic image attack; if t1 > R > t2, the corresponding frequency-domain image is inferred using the face composite image detection model M' to obtain the score Q. If Q is greater than the third threshold (such as 0.5), it is determined as a real person; if Q is less than the third threshold, it is determined as a synthetic image attack.

[0075] Therefore, the technical solution of the embodiment of the present disclosure uses the prompt tuning method to fuse frequency-domain information after completing the training of the RGB image, combines the frequency-domain supplementary information while minimizing the loss of RGB image information, and finally uses the post-processing method of collaborative inference. When the confidence of the RGB image result is higher than the threshold, the prediction of the frequency-domain image is not performed, which improves the algorithm efficiency and further reduces the interference of the frequency-domain information on the RGB information, greatly improving the accuracy and generalization of the algorithm model, effectively preventing face synthesis attacks, and enhancing the defense effect of the face composite image detection algorithm. It can be applied to many applications such as attendance, access control, security, and financial payment in the field of face recognition.

[0076] Figure 6 is a schematic structural diagram of a training device for a face composite image detection model according to an embodiment of the present disclosure. This embodiment is applicable to the situation of detecting whether the face in the image is a synthetic image, relates to the field of artificial intelligence technology, especially relates to technical fields such as computer vision, deep learning, and face recognition, and can be applied to scenarios such as smart cities and smart finance. As Figure 6 shown, the device 600 specifically includes:

[0077] A pre-training module 601 is used to train a pre-training model for detecting a composite face image using a training data set of RGB face images, wherein the pre-training model includes an embedding layer, a backbone network and a classification layer;

[0078] A frequency domain image conversion module 602 is used to obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image;

[0079] The instruction fine-tuning module 603 is used to insert learnable instruction fine-tuning parameters into the backbone network, and use the training data set of the frequency domain face image to perform instruction fine-tuning on the pre-trained model into which the instruction fine-tuning parameters are inserted, so as to obtain a face synthesis image detection model.

[0080] Optionally, the backbone network includes multiple encoder layers;

[0081] The instruction fine-tuning module is specifically used to insert learnable instruction fine-tuning parameters in at least one encoder layer of the backbone network.

[0082] Optionally, during the instruction fine-tuning process, the parameters of the embedding layer and other parameters in the backbone network except the instruction fine-tuning parameters are solidified.

[0083] Optionally, the instruction fine-tuning module includes a first fine-tuning submodule, and the first fine-tuning submodule includes:

[0084] A first detection result acquisition unit, used to acquire a first detection result of the frequency domain face image by the pre-trained model after inserting the instruction fine-tuning parameters;

[0085] The binary classification supervision unit is used to perform binary classification supervision on the pre-trained model after the instruction fine-tuning parameters are inserted according to the first detection result and the pre-labeled true value.

[0086] Optionally, the instruction fine-tuning module includes a second fine-tuning submodule, and the second fine-tuning submodule includes:

[0087] A second detection result acquisition unit, used for acquiring a pre-trained model without inserting the instruction fine-tuning parameters, and a second detection result of the RGB face image corresponding to the frequency domain face image;

[0088] A feature distillation unit is used to perform feature distillation on the pre-trained model after inserting the instruction fine-tuning parameters according to the first detection result and the second detection result.

[0089] Optionally, the training data set of RGB face images includes real RGB face images and synthesized RGB face images.

[0090] Optionally, the device further includes a preprocessing module, and the preprocessing module includes:

[0091] A facial key point detection unit is used to perform facial key point detection on a dataset of RGB original images containing faces, and obtain the coordinate values ​​of facial key points of each RGB original image, before the pre-training module is trained to obtain a pre-training model for detecting a synthetic face image;

[0092] A face alignment unit, used to align the target face in the corresponding RGB original image according to the coordinate values ​​of the face key points, and determine the center position and rotation angle of the target face;

[0093] An affine transformation unit, configured to intercept an image of the target face from the RGB original image according to a set size through affine transformation based on the center position and the rotation angle;

[0094] The training data set determining unit is used to obtain the training data set of the RGB face image according to the image of the target face in each RGB original image.

[0095] Optionally, the training data set determining unit includes:

[0096] A normalization subunit, used for normalizing each pixel value of the target face image;

[0097] The data enhancement subunit is used to perform data enhancement on the normalized image to obtain a training data set of the RGB face image.

[0098] Figure 7 The present invention is a schematic diagram of the structure of a face composite image detection device according to an embodiment of the present invention. The present embodiment can be applied to the case of detecting whether a face in an image is a composite image, and relates to the field of artificial intelligence technology, in particular to the fields of computer vision, deep learning, face recognition, etc. Figure 7 As shown, the device 700 specifically includes:

[0099] An input module 701 is used to input the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained by the training device for the face composite image detection model according to any embodiment of the present disclosure;

[0100] An acquisition module 702 is used to acquire a first prediction value output by the face composition image detection model;

[0101] The detection module 703 is configured to determine that the RGB image to be detected is a real face image if the first prediction value is higher than a first threshold.

[0102] Optionally, the detection module is further used for:

[0103] If the first prediction value is lower than a second threshold, it is determined that the RGB image to be detected is a synthetic face image.

[0104] Optionally, the device further includes a collaborative reasoning module, and the collaborative reasoning module includes:

[0105] a frequency domain image acquisition unit to be detected, configured to acquire the frequency domain image to be detected of the RGB image to be detected if the first prediction value is between the second threshold and the first threshold;

[0106] A second prediction value acquisition unit, used for inputting the frequency domain image to be detected into the face synthesis image detection model, and acquiring a second prediction value output by the face synthesis image detection model;

[0107] An inference unit is used to determine whether the RGB image to be detected is a real face image according to the second prediction value and a third threshold.

[0108] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0109] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0110] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0111] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0112] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the training method of the face synthesis image detection model. For example, in some embodiments, the training method of the face synthesis image detection model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method of the face synthesis image detection model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the training method of the face composition image detection model in any other appropriate manner (for example, by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0121] Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0122] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.

[0123] In addition, according to an embodiment of the present disclosure, the present disclosure also provides another electronic device, another readable storage medium and another computer program product, which are used to execute one or more steps of the face composite image detection method described in any embodiment of the present disclosure. The specific structure and program code thereof can be found in Figure 8 The description of the contents of the illustrated embodiment will not be repeated here.

[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document is not limited here.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a face synthesis image detection model, comprising: Using a training data set of RGB face images, a pre-trained model for detecting a composite face image is trained, wherein the pre-trained model includes an embedding layer, a backbone network, and a classification layer; Obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image; Inserting learnable instruction fine-tuning parameters into the backbone network, using the training data set of the frequency domain face image to perform instruction fine-tuning on the pre-trained model into which the instruction fine-tuning parameters are inserted, to obtain a face synthesis image detection model, and during the instruction fine-tuning process, solidifying the parameters of the embedding layer and other parameters in the backbone network except the instruction fine-tuning parameters; The process of fine-tuning the instruction includes: Obtaining a first detection result of the frequency domain face image by the pre-trained model after inserting the instruction fine-tuning parameters; According to the first detection result and the pre-labeled true value, binary classification supervision is performed on the pre-trained model after the instruction fine-tuning parameters are inserted.

2. The method according to claim 1, wherein: The backbone network includes a plurality of encoder layers; Inserting learnable instructions to fine-tune parameters in the backbone network includes: A learnable instruction fine-tuning parameter is inserted into at least one encoder layer of the backbone network.

3. The method according to claim 1, wherein: The process of fine-tuning the instructions also includes: Obtaining a pre-trained model without inserting the instruction fine-tuning parameters, and a second detection result of the RGB face image corresponding to the frequency domain face image; According to the first detection result and the second detection result, feature distillation is performed on the pre-trained model after the instruction fine-tuning parameters are inserted.

4. The method according to claim 1, wherein: The training data set of the RGB face images includes real RGB face images and synthesized RGB face images.

5. The method according to claim 1, wherein: Before the training to obtain a pre-trained model for detecting a synthetic face image, the method further includes: Perform facial key point detection on the RGB original image data set containing human faces, and obtain the facial key point coordinate values ​​of each RGB original image; Perform face alignment on the target face in the corresponding RGB original image according to the coordinate values ​​of the facial key points to determine the center position and rotation angle of the target face; According to the center position and the rotation angle, an image of the target face is intercepted from the RGB original image according to a set size through affine transformation; A training data set of the RGB face images is obtained according to the image of the target face in each RGB original image.

6. The method according to claim 5, wherein: The step of obtaining a training data set of the RGB face images according to the image of the target face of each RGB original image comprises: Normalizing each pixel value of the target face image; Data enhancement is performed on the normalized image to obtain a training data set of the RGB face image.

7. A method for detecting a synthetic face image, comprising: Inputting the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained according to the training method for the face composite image detection model according to any one of claims 1 to 6; Obtaining a first prediction value output by the face composition image detection model; If the first prediction value is higher than a first threshold, it is determined that the RGB image to be detected is a real face image.

8. The method according to claim 7, further comprising: If the first prediction value is lower than a second threshold, it is determined that the RGB image to be detected is a synthetic face image.

9. The method according to claim 8, further comprising: If the first prediction value is between the second threshold and the first threshold, obtaining a frequency domain image to be detected of the RGB image to be detected; Inputting the frequency domain image to be detected into the face synthesis image detection model, and obtaining a second prediction value output by the face synthesis image detection model; According to the second prediction value and the third threshold, it is determined whether the RGB image to be detected is a real face image.

10. A training device for a face synthesis image detection model, comprising: A pre-training module, used to train a pre-training model for detecting a composite face image using a training data set of RGB face images, wherein the pre-training model includes an embedding layer, a backbone network, and a classification layer; A frequency domain image conversion module is used to obtain a frequency domain face image corresponding to the RGB face image to obtain a training data set of the frequency domain face image; An instruction fine-tuning module is used to insert learnable instruction fine-tuning parameters into the backbone network, and use the training data set of the frequency domain face image to perform instruction fine-tuning on the pre-trained model into which the instruction fine-tuning parameters are inserted, so as to obtain a face synthesis image detection model. During the instruction fine-tuning process, the parameters of the embedding layer and other parameters in the backbone network except the instruction fine-tuning parameters are solidified; The instruction fine-tuning module includes a first fine-tuning submodule, and the first fine-tuning submodule includes: A first detection result acquisition unit, used to acquire a first detection result of the frequency domain face image by the pre-trained model after inserting the instruction fine-tuning parameters; The binary classification supervision unit is used to perform binary classification supervision on the pre-trained model after the instruction fine-tuning parameters are inserted according to the first detection result and the pre-labeled true value.

11. The device according to claim 10, wherein: The backbone network includes a plurality of encoder layers; The instruction fine-tuning module is specifically used to insert learnable instruction fine-tuning parameters in at least one encoder layer of the backbone network.

12. The device according to claim 10, wherein: The instruction fine-tuning module includes a second fine-tuning submodule, and the second fine-tuning submodule includes: A second detection result acquisition unit, used for acquiring a pre-trained model without inserting the instruction fine-tuning parameters, and a second detection result of the RGB face image corresponding to the frequency domain face image; A feature distillation unit is used to perform feature distillation on the pre-trained model after inserting the instruction fine-tuning parameters according to the first detection result and the second detection result.

13. The device according to claim 10, wherein: The training data set of the RGB face images includes real RGB face images and synthesized RGB face images.

14. The apparatus according to claim 10, further comprising a preprocessing module, wherein the preprocessing module comprises: A facial key point detection unit is used to perform facial key point detection on a dataset of RGB original images containing faces, and obtain the coordinate values ​​of facial key points of each RGB original image, before the pre-training module is trained to obtain a pre-training model for detecting a synthetic face image; A face alignment unit, used to align the target face in the corresponding RGB original image according to the coordinate values ​​of the face key points, and determine the center position and rotation angle of the target face; An affine transformation unit, configured to intercept an image of the target face from the RGB original image according to a set size through affine transformation based on the center position and the rotation angle; The training data set determining unit is used to obtain the training data set of the RGB face image according to the image of the target face in each RGB original image.

15. The device according to claim 14, wherein: The training data set determination unit comprises: A normalization subunit, used for normalizing each pixel value of the target face image; The data enhancement subunit is used to perform data enhancement on the normalized image to obtain a training data set of the RGB face image.

16. A face composite image detection device, comprising: An input module, used for inputting the RGB image to be detected into a face composite image detection model, wherein the face composite image detection model is trained according to the training device of the face composite image detection model according to any one of claims 10 to 15; An acquisition module, used to acquire a first prediction value output by the face synthesis image detection model; A detection module is used to determine that the RGB image to be detected is a real face image if the first prediction value is higher than a first threshold.

17. The device according to claim 16, wherein: The detection module is also used for: If the first prediction value is lower than a second threshold, it is determined that the RGB image to be detected is a synthetic face image.

18. The apparatus according to claim 17, further comprising a collaborative reasoning module, wherein the collaborative reasoning module comprises: a frequency domain image acquisition unit to be detected, configured to acquire the frequency domain image to be detected of the RGB image to be detected if the first prediction value is between the second threshold and the first threshold; A second prediction value acquisition unit, used for inputting the frequency domain image to be detected into the face synthesis image detection model, and acquiring a second prediction value output by the face synthesis image detection model; An inference unit is used to determine whether the RGB image to be detected is a real face image according to the second prediction value and a third threshold.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the face synthesis image detection model described in any one of claims 1-6, and the face synthesis image detection method described in any one of claims 7-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the training method for a face composite image detection model according to any one of claims 1 to 6, and the face composite image detection method according to any one of claims 7 to 9.

21. A computer program product, comprising a computer program, which, when executed by a processor, implements the training method for a face composite image detection model according to any one of claims 1 to 6, and the face composite image detection method according to any one of claims 7 to 9.

Citation Information

Patent Citations

  • Detection model training method and device, target detection method and device, equipment and medium

    CN115880502A

  • Face living body detection method and device based on local attention mechanism

    CN117133059A