Face image detection method, device, electronic device and storage medium
Through multi-stage semantic representation model and feature fusion processing, combined with global and local feature extraction, the problem of poor detection accuracy of face image forgery is solved, and the accuracy of detection is significantly improved.
Patent Information
- Application Number
- CN202111272607.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-10-29
AI Technical Summary
The prior art has poor accuracy in facial image forgery detection, making it difficult to effectively judge whether the face image tampered by deep learning technology is true or forgery.
Multi-stage semantic representation model and feature fusion processing are adopted. By inputting image vectors into multiple semantic representation models and performing feature fusion, combining global and local feature extraction, the accuracy of face image detection is improved.
The accuracy of face image forgery detection is significantly improved. Through multi-level feature fusion and weight adjustment, the utilization of face image features is enhanced and the accuracy of real forgery judgment is improved.
Smart Images

Figure CN114078274B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing, computer vision, and deep learning technology, and in particular to a face image detection method, device, electronic device, and storage medium. Background Art
[0002] At present, the face forgery detection technology in the industry refers to the use of certain methods to determine whether a face image has been tampered with by deep learning technology (i.e., changing the original content and meaning). In related technologies, it is difficult to determine whether a face image tampered with by deep learning technology is real or forged. Summary of the invention
[0003] The present disclosure provides a facial image detection method, device, electronic device and storage medium.
[0004] According to one aspect of the present disclosure, a face image detection method is provided, comprising: inputting an image vector of a face image to be detected into a first semantic representation model and a second semantic representation model respectively, so as to obtain a first feature vector output by the i-th stage network in the first semantic representation model and a second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model are both N, and i is a positive integer from 1 to N-1; performing feature fusion processing according to the first feature vector and the second feature vector to obtain a fused first feature vector and a fused second feature vector; inputting the fused first feature vector into the i+1-th stage network in the first semantic representation model, and inputting the fused second feature vector into the i+1-th stage network in the second semantic representation model; and determining the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model.
[0005] According to another aspect of the present disclosure, a training method for a joint model is provided, comprising: constructing an initial joint model, wherein the joint model comprises: N stage networks of a first semantic representation model, a head network of a second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; obtaining training data, wherein the training data comprises sample face images and corresponding labels, wherein the labels represent whether the sample face images are real or forged; using the image vectors of the sample face images as the input of the joint model, and using the labels corresponding to the sample face images as the output of the joint model, to train the joint model.
[0006] According to another aspect of the present disclosure, a face image detection device is provided, including: a first input module, used to input the image vector of the face image to be detected into a first semantic representation model and a second semantic representation model respectively, so as to obtain a first feature vector output by the i-th stage network in the first semantic representation model and a second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, and i is a positive integer from 1 to N-1; a feature fusion module, used to perform feature fusion processing according to the first feature vector and the second feature vector to obtain a fused first feature vector and a fused second feature vector; a second input module, used to input the fused first feature vector into the i+1-th stage network in the first semantic representation model, and input the fused second feature vector into the i+1-th stage network in the second semantic representation model; a first determination module, used to determine the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model.
[0007] According to another aspect of the present disclosure, a training device for a joint model is provided, comprising: a construction module, used to construct an initial joint model, wherein the joint model comprises: N stage networks of a first semantic representation model, a head network of a second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; an acquisition module, used to acquire training data, wherein the training data comprises sample face images and corresponding labels, and the labels represent whether the sample face images are real or forged; a training module, used to train the joint model using the image vectors of the sample face images as inputs of the joint model and the labels corresponding to the sample face images as outputs of the joint model.
[0008] According to another aspect of the present disclosure, there is provided an electronic device, including:
[0009] at least one processor; and
[0010] a memory communicatively connected to the at least one processor; wherein,
[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the face image detection method proposed in the above-mentioned first aspect of the present disclosure; or, the training method of the joint model proposed in the other aspect.
[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the face image detection method proposed in the above-mentioned first aspect of the present disclosure; or, the joint model training method proposed in the other aspect.
[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the face image detection method proposed in the above-mentioned first aspect of the present disclosure; or, the steps of the joint model training method proposed in the other aspect.
[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0016] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;
[0017] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;
[0018] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;
[0019] Figure 4 is a schematic diagram of the joint model;
[0020] Figure 5 It is a schematic diagram of the feature fusion network in the joint model;
[0021] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;
[0022] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;
[0023] Figure 8 is a block diagram of an electronic device implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0025] At present, the face forgery detection technology in the industry refers to the use of certain methods to determine whether a face image has been tampered with by deep learning technology (i.e., changing the original content and meaning). In related technologies, simple neural networks are mainly used to classify true and false, or deep learning methods are used to automatically extract forged features to determine whether a face image is real or forged. The accuracy of face forgery detection is poor.
[0026] In view of the above problems, the present disclosure proposes a facial image detection method, device, electronic device and storage medium.
[0027] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the face image detection method of the embodiment of the present disclosure can be applied to a face image detection device, and the device can be configured in an electronic device so that the electronic device can perform a face image detection function.
[0028] The electronic device may be any device with computing capability, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal may be a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, or other hardware device with various operating systems, touch screens, and / or display screens.
[0029] like Figure 1 As shown, the face image detection method may include the following steps:
[0030] Step 101, input the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model are both N, and i is a positive integer from 1 to N-1.
[0031] In the embodiment of the present disclosure, before step 101, the method may further include a process of determining an image vector of a face image to be detected. The process may specifically include the following steps: determining a face image to be detected; performing block processing on the face image to obtain a plurality of image blocks; performing vector conversion processing on the plurality of image blocks respectively to determine a vector of each image block; and determining a vector sequence composed of the vectors of the plurality of image blocks as an image vector of the face image.
[0032] In the disclosed embodiment, the facial image is segmented into blocks and vectorized by the blocks, so that vectors of multiple image blocks can be obtained, and more features in the facial image can be retained to facilitate subsequent processing, thereby further improving the accuracy of facial forgery detection.
[0033] In the disclosed embodiment, the first semantic representation model can be used to extract global features of a face image, and the second semantic representation model can be used to extract local features of a face image. The first semantic representation model can be, for example, a transformer model, which has global feature extraction capabilities and can establish long-distance relationships between multiple image blocks in a face image from a global perspective. The second semantic representation model can be, for example, a CNN model, which can extract local artifact features in a face image. Thus, by extracting global features and local features, the comprehensiveness of the extracted features is ensured, thereby improving the accuracy of face forgery detection.
[0034] In the embodiment of the present disclosure, the first semantic representation model is a transformer model, which may include 4 stage networks, that is, N is 4, each stage network includes 3 transformer network layers, and the model includes a total of 12 transformer network layers.
[0035] In the embodiment of the present disclosure, taking the second semantic representation model as a CNN model as an example, the model may include a head network and four stage networks, and each stage network includes three CNN network layers.
[0036] In the embodiment of the present disclosure, when the first semantic representation model is not provided with a head network and the second semantic representation model is provided with a head network, the process of the face image detection device executing step 101 may, for example, be: inputting the image vector into the head network of the second semantic representation model to obtain the head feature vector output by the head network; performing feature fusion processing according to the image vector and the head feature vector to obtain a fused image vector and a fused head feature vector; inputting the fused image vector into the first semantic representation model, and inputting the fused head feature vector into the first stage network of the second semantic representation model, to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model.
[0037] The head feature vector is obtained by processing the second semantic representation model and can reflect the local features of the face image. The image vector can reflect global features and local features. Feature fusion processing is performed based on the image vector and the head feature vector, so that the fused feature vector can reflect global features and local features, and can utilize more features in the face image, thereby further improving the accuracy of face forgery detection.
[0038] Step 102: Perform feature fusion processing on the first feature vector and the second feature vector to obtain a fused first feature vector and a fused second feature vector.
[0039] In the disclosed embodiment, the first feature vector is obtained by processing the first semantic representation model, and can reflect the global features of the face image; the second feature vector is obtained by processing the second semantic representation model, and can reflect the local features of the face image. Feature fusion processing is performed based on the global features and local features of the face image, so that the fused feature vector can reflect the global features and local features, and can utilize more features in the face image, thereby further improving the accuracy of face forgery detection.
[0040] Step 103: input the fused first feature vector into the i+1th stage network in the first semantic representation model, and input the fused second feature vector into the i+1th stage network in the second semantic representation model.
[0041] In the implementation of the present disclosure, i is a positive integer from 1 to N-1. The initial value of i is 1, and i is incremented by 1 each time step 102 and step 103 are executed. Taking N as 4 as an example, that is, after the image vector of the face image to be detected is input into the first semantic representation model and the second semantic representation model respectively, the first feature vector output by the first stage network in the first semantic representation model and the second feature vector output by the first stage network in the second semantic representation model are first obtained; feature fusion processing is performed to obtain the fused first feature vector and the fused second feature vector; the fused first feature vector is input into the second stage network in the first semantic representation model; the fused second feature vector is input into the second stage network in the second semantic representation model.
[0042] Then, a first feature vector output by the second stage network in the first semantic representation model and a second feature vector output by the second stage network in the second semantic representation model are obtained; feature fusion processing is performed to obtain a fused first feature vector and a fused second feature vector; the fused first feature vector is input into the third stage network in the first semantic representation model; and the fused second feature vector is input into the third stage network in the second semantic representation model.
[0043] Then, the first feature vector output by the third segment network in the first semantic representation model and the second feature vector output by the third stage network in the second semantic representation model are obtained; feature fusion processing is performed to obtain the fused first feature vector and the fused second feature vector; the fused first feature vector is input into the fourth stage network in the first semantic representation model; and the fused second feature vector is input into the fourth stage network in the second semantic representation model.
[0044] Step 104: Determine a detection result of the face image according to a first detection result of the Nth stage network in the first semantic representation model and a second detection result of the Nth stage network in the second semantic representation model.
[0045] In the embodiment of the present disclosure, the first detection result may include a forged probability and a true probability; the second detection result may include a forged probability and a true probability. Correspondingly, in order to improve the detection accuracy of the detection result, the process of the face image detection device performing step 104 may be, for example, determining the forged probability of the face image according to the forged probability in the first detection result and the forged probability in the second detection result; determining the true probability of the face image according to the true probability in the first detection result and the true probability in the second detection result; determining the detection result of the face image according to the true probability and the forged probability of the face image.
[0046] The process of determining the forgery probability of the face image based on the forgery probability in the first detection result and the forgery probability in the second detection result may, for example, be to sum and average the forgery probability in the first detection result and the forgery probability in the second detection result, and use the average as the forgery probability of the face image. The process of determining the true probability of the face image based on the true probability in the first detection result and the true probability in the second detection result may, for example, be to sum and average the true probability in the first detection result and the true probability in the second detection result, and use the average as the true probability of the face image.
[0047] Among them, the process of determining the detection result of the face image based on the true probability and the forgery probability of the face image can be, for example, if the true probability is greater than the forgery probability, then the face image is determined to be true; if the true probability is less than the forgery probability, then the face image is determined to be forged; if the true probability is equal to the forgery probability, then the face image is determined to be true or forged, or the face image is re-detected, etc.
[0048] The face image detection method of the disclosed embodiment is to input the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, and i is a positive integer from 1 to N-1; perform feature fusion processing according to the first feature vector and the second feature vector to obtain the fused first feature vector and the fused second feature vector; input the fused first feature vector into the i+1-th stage network in the first semantic representation model, and input the fused second feature vector into the i+1-th stage network in the second semantic representation model; determine the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model. Thus, it is possible to combine the features extracted by multiple semantic representation models, fuse the extracted features at multiple levels, make a true forgery judgment based on the fused features, and improve the accuracy of face forgery detection.
[0049] In order to further improve the accuracy of face forgery detection, Figure 2 As shown, Figure 2 It is a schematic diagram according to the second embodiment of the present disclosure. In the embodiment of the present disclosure, the weight adjustment value of the two feature vectors can be determined based on the first feature vector and the second feature vector; and then the weight adjustment value and the initial weight value are combined to determine the two fused feature vectors, and then the detection result is determined. Figure 2 The illustrated embodiment may include the following steps:
[0050] Step 201, input the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model are both N, and i is a positive integer from 1 to N-1.
[0051] Step 202: Determine a weight adjustment value of the first eigenvector and a weight adjustment value of the second eigenvector according to the first eigenvector and the second eigenvector.
[0052] In the embodiment of the present disclosure, the process of the face image detection device executing step 202 may, for example, be to perform frequency domain conversion on the first eigenvector and the second eigenvector respectively to obtain the first frequency domain eigenvector and the second frequency domain eigenvector; determine the weight adjustment value of the second eigenvector based on the first frequency domain eigenvector; and determine the weight adjustment value of the first eigenvector based on the second frequency domain eigenvector.
[0053] Among them, the first feature vector is subjected to frequency domain conversion to obtain the first frequency domain feature vector. The first frequency domain feature vector contains more information than the first feature vector, especially the information of the forged feature. Then, the weight adjustment value of the second feature vector is determined based on the first frequency domain feature vector. That is to say, if the amount of information of the forged feature in the first frequency domain feature vector is large, the weight of the first feature vector needs to be increased, which can be achieved by reducing the weight of the second feature vector. Therefore, the weight adjustment value of the second feature vector is determined based on the first frequency domain feature vector, and the weight adjustment value of the first feature vector is determined based on the second frequency domain feature vector. The weights of the first feature vector and the second feature vector can be accurately determined, thereby increasing the amount of information of the forged feature and reducing the amount of information of the non-forged feature, thereby further improving the accuracy of face forgery detection.
[0054] Step 203: Determine the fused first eigenvector according to the first eigenvector and the corresponding initial weight value and weight adjustment value.
[0055] In the embodiment of the present disclosure, the initial weight value is 1. In one example, the weight adjustment value can be multiplied by the first eigenvector, and the first eigenvector can be added to the multiplication result to obtain the fused first eigenvector. In another example, the weight adjustment value can be added to the initial weight value, and the first eigenvector can be multiplied to the addition result to obtain the fused first eigenvector.
[0056] Step 204: Determine a fused second eigenvector according to the second eigenvector and the corresponding initial weight value and weight adjustment value.
[0057] In the embodiment of the present disclosure, the initial weight value is 1. In one example, the weight adjustment value can be multiplied by the second eigenvector, and the second eigenvector can be added to the multiplication result to obtain the fused second eigenvector. In another example, the weight adjustment value can be added to the initial weight value, and the second eigenvector can be multiplied to the addition result to obtain the fused second eigenvector.
[0058] Step 205: input the fused first feature vector into the i+1th stage network in the first semantic representation model, and input the fused second feature vector into the i+1th stage network in the second semantic representation model.
[0059] Step 206, determining a detection result of the face image according to a first detection result of the Nth stage network in the first semantic representation model and a second detection result of the Nth stage network in the second semantic representation model.
[0060] In the embodiment of the present disclosure, the detailed description of step 201, step 205 to step 206 can be referred to. Figure 1 The detailed description of step 101, step 103 to step 104 in the illustrated implementation will not be repeated here.
[0061] The face image detection method of the disclosed embodiment comprises the following steps: inputting the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model are both N, and i is a positive integer from 1 to N-1; determining the weight adjustment value of the first feature vector and the weight adjustment value of the second feature vector according to the first feature vector and the corresponding initial weight value and weight adjustment value; determining the fused first feature vector according to the first feature vector and the corresponding initial weight value and weight adjustment value; determining the fused second feature vector according to the second feature vector and the corresponding initial weight value and weight adjustment value; inputting the fused first feature vector into the i+1-th stage network in the first semantic representation model, and inputting the fused second feature vector into the i+1-th stage network in the second semantic representation model; determining the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model. In this way, the features extracted by multiple semantic representation models can be combined, and the extracted features can be weighted and fused at multiple levels, and authenticity judgment can be made based on the fused features, further improving the accuracy of face forgery detection.
[0062] Figure 3It is a schematic diagram according to the third embodiment of the present disclosure. It should be noted that the training method of the joint model of the embodiment of the present disclosure can be applied to the training device of the joint model. The device can be configured in an electronic device so that the electronic device can perform the training function of the joint model.
[0063] like Figure 3 As shown, the training method of the joint model may include the following steps:
[0064] Step 301, constructing an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, the head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and the two first stage networks; i is a positive integer from 1 to N-1.
[0065] In the disclosed embodiment, the first semantic representation model can be used to extract global features of a face image, and the second semantic representation model can be used to extract local features of a face image. The first semantic representation model can be, for example, a transformer model, which has global feature extraction capabilities and can establish long-distance relationships between multiple image blocks in a face image from a global perspective. The second semantic representation model can be, for example, a CNN model, which can extract local artifact features in a face image.
[0066] In the embodiment of the present disclosure, the schematic diagram of the joint model can be as follows: Figure 4 As shown, in Figure 4 In , the first semantic representation model can be a transformer model, which may include 4 stage networks, that is, N is 4, each stage network includes 3 transformer network layers, and the model includes a total of 12 transformer network layers. Figure 4 In the example, the second semantic representation model may be a CNN model, which may include a head network and four stage networks, each of which includes three CNN network layers.
[0067] exist Figure 4 In , the block division and vector conversion represent the block division and vector conversion processing of the face image, and obtain the vector sequence, which is used as the image vector of the face image. Figure 4In the figure, the input of the head feature fusion network includes: an image vector and a head feature vector output by the head network (bottleneck) of the CNN model; the fused image vector output by the head feature fusion network is provided to the first stage network of the transformer model, and the fused head feature vector output by the head feature fusion network is provided to the first stage network of the CNN model.
[0068] In the embodiment of the present disclosure, the schematic diagram of the feature fusion network in the joint model can be as follows: Figure 5 As shown. Figure 5 In the embodiment, a feature fusion network connecting two first-stage networks and two second-stage networks is taken as an example. The input of the feature fusion network includes: the first eigenvector (Xg) output by the first-stage network of the transformer model and the second eigenvector (Xl) output by the first-stage network of the CNN model. The first FFR module in the feature fusion network is used to perform frequency domain conversion on the first eigenvector, and determine the weight adjustment value of the second eigenvector based on the conversion result, so as to obtain the fused second eigenvector; the second FFR module in the feature fusion network is used to perform frequency domain conversion on the second eigenvector, and determine the weight adjustment value of the first eigenvector based on the conversion result, so as to obtain the fused first eigenvector.
[0069] Step 302: Acquire training data, wherein the training data includes sample facial images and corresponding labels, and the labels represent whether the sample facial images are real or forged.
[0070] Step 303: Use the image vector of the sample face image as the input of the joint model, use the label corresponding to the sample face image as the output of the joint model, and train the joint model.
[0071] In the disclosed embodiment, specifically, the image vector of the sample face image is input into the first stage network of the first semantic representation model and into the head network of the second semantic representation model to obtain the first prediction result of the last stage network output of the first semantic representation model and the second prediction result of the last stage network output of the second semantic representation model; the value of the loss function is determined according to the first prediction result, the second prediction result, the label of the sample face image and the preset loss function; the coefficients in the joint model are adjusted according to the value of the loss function to realize the training of the joint model.
[0072] The training method of the joint model of the disclosed embodiment is to construct an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, the head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; obtain training data, wherein the training data includes sample face images and corresponding labels, and the labels represent whether the sample face images are real or forged; the image vectors of the sample face images are used as the input of the joint model, and the labels corresponding to the sample face images are used as the output of the joint model, and the joint model is trained. Thus, the trained joint model can combine the features extracted from multiple semantic representation models, and the extracted features are fused at multiple levels, and the real and forged judgment is made based on the fused features, thereby improving the accuracy of face forgery detection.
[0073] In order to implement the above embodiment, the present disclosure also proposes a face image detection device.
[0074] like Figure 6 As shown, Figure 6 Schematic diagram according to the fourth embodiment of the present disclosure. The face image detection device 600 comprises: a first input module 610 , a feature fusion module 620 , a second input module 630 and a first determination module 640 .
[0075] The first input module 610 is used to input the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, and i is a positive integer from 1 to N-1;
[0076] A feature fusion module 620, configured to perform feature fusion processing according to the first feature vector and the second feature vector to obtain a fused first feature vector and a fused second feature vector;
[0077] A second input module 630, configured to input the fused first feature vector into the i+1th stage network in the first semantic representation model, and input the fused second feature vector into the i+1th stage network in the second semantic representation model;
[0078] The first determination module 640 is used to determine the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model.
[0079] As a possible implementation method of the embodiment of the present disclosure, the device also includes: a second determination module, a blocking module, a vector conversion module and a third determination module; the second determination module is used to determine the face image to be detected; the blocking module is used to block the face image to obtain multiple image blocks; the vector conversion module is used to perform vector conversion processing on the multiple image blocks respectively to determine the vector of each image block; the third determination module is used to determine the vector sequence composed of the vectors of the multiple image blocks as the image vector of the face image.
[0080] As a possible implementation manner of the embodiment of the present disclosure, the first semantic representation model is used to extract global features of the face image; and the second semantic representation model is used to extract local features of the face image.
[0081] As a possible implementation of an embodiment of the present disclosure, the first semantic representation model is not provided with a head network, and the second semantic representation model is provided with a head network; the first input module 610 is specifically used to input the image vector into the head network of the second semantic representation model to obtain a head feature vector output by the head network; perform feature fusion processing according to the image vector and the head feature vector to obtain a fused image vector and a fused head feature vector; input the fused image vector into the first semantic representation model, and input the fused head feature vector into the first stage network of the second semantic representation model to obtain a first feature vector output by the i-th stage network in the first semantic representation model and a second feature vector output by the i-th stage network in the second semantic representation model.
[0082] As a possible implementation method of the embodiment of the present disclosure, the feature fusion module 620 is specifically used to determine the weight adjustment value of the first feature vector and the weight adjustment value of the second feature vector based on the first feature vector and the second feature vector; determine the fused first feature vector based on the first feature vector and the corresponding initial weight value and weight adjustment value; determine the fused second feature vector based on the second feature vector and the corresponding initial weight value and weight adjustment value.
[0083] As a possible implementation method of the embodiment of the present disclosure, the feature fusion module 620 is specifically used to perform frequency domain conversion on the first feature vector and the second feature vector respectively to obtain a first frequency domain feature vector and a second frequency domain feature vector; determine a weight adjustment value of the second feature vector based on the first frequency domain feature vector; determine a weight adjustment value of the first feature vector based on the second frequency domain feature vector.
[0084] As a possible implementation method of an embodiment of the present disclosure, the first determination module 640 is specifically used to determine the forgery probability of the facial image based on the forgery probability in the first detection result and the forgery probability in the second detection result; determine the true probability of the facial image based on the true probability in the first detection result and the true probability in the second detection result; determine the detection result of the facial image based on the true probability and the forgery probability of the facial image.
[0085] The face image detection device of the disclosed embodiment inputs the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, and i is a positive integer from 1 to N-1; performs feature fusion processing according to the first feature vector and the second feature vector to obtain the fused first feature vector and the fused second feature vector; inputs the fused first feature vector into the i+1-th stage network in the first semantic representation model, and inputs the fused second feature vector into the i+1-th stage network in the second semantic representation model; determines the detection result of the face image according to the first detection result of the N-th stage network in the first semantic representation model and the second detection result of the N-th stage network in the second semantic representation model. Thus, it is possible to combine the features extracted by multiple semantic representation models, fuse the extracted features at multiple levels, make a true forgery judgment based on the fused features, and improve the accuracy of face forgery detection.
[0086] In order to implement the above embodiments, the present disclosure also proposes a training device for a joint model.
[0087] like Figure 7 As shown, Figure 7 7 is a schematic diagram of the fifth embodiment of the present disclosure. The training device 700 of the joint model includes: a construction module 710 , an acquisition module 720 and a training module 730 .
[0088] Among them, the construction module 710 is used to construct an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, the head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1;
[0089] An acquisition module 720 is used to acquire training data, wherein the training data includes a sample face image and a corresponding label, wherein the label indicates whether the sample face image is real or forged;
[0090] The training module 730 is used to train the joint model by taking the image vector of the sample face image as the input of the joint model and taking the label corresponding to the sample face image as the output of the joint model.
[0091] The training device of the joint model of the disclosed embodiment constructs an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, the head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; training data is obtained, wherein the training data includes sample face images and corresponding labels, and the labels represent whether the sample face images are real or forged; the image vectors of the sample face images are used as the input of the joint model, and the labels corresponding to the sample face images are used as the output of the joint model to train the joint model. Thus, the trained joint model can combine the features extracted from multiple semantic representation models, and the extracted features are fused at multiple levels, and the real and forged judgment is made based on the fused features, thereby improving the accuracy of face forgery detection.
[0092] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.
[0093] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0094] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0095] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0096] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0097] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a face image detection method or a joint model training method. For example, in some embodiments, the face image detection method or the joint model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the face image detection method described above may be executed; or one or more steps of the joint model training method. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the face image detection method or the joint model training method in any other appropriate manner (for example, by means of firmware).
[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0102] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0103] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A face image detection method, comprising: Inputting the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the first stage network in the first semantic representation model and the second feature vector output by the first stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, the first semantic representation model is used to extract the global features of the face image, and the second semantic representation model is used to extract the local features of the face image; For the i-th stage network in the first semantic representation model and the i-th stage network in the second semantic representation model, determine the weight adjustment value of the first feature vector and the weight adjustment value of the second feature vector according to the first feature vector output by the i-th stage network in the first semantic representation model and the second feature vector output by the i-th stage network in the second semantic representation model; determine the fused first feature vector according to the first feature vector and the corresponding initial weight value and weight adjustment value; determine the fused second feature vector according to the second feature vector and the corresponding initial weight value and weight adjustment value; i is a positive integer from 1 to N-1; the initial value of i is 1; Input the fused first feature vector into the i+1th stage network in the first semantic representation model, and input the fused second feature vector into the i+1th stage network in the second semantic representation model, and add 1 to i; repeat the above two steps until the first detection result of the Nth stage network in the first semantic representation model and the second detection result of the Nth stage network in the second semantic representation model are obtained; The detection result of the facial image is determined according to a first detection result of the Nth stage network in the first semantic representation model and a second detection result of the Nth stage network in the second semantic representation model, wherein the detection result includes whether the facial image to be detected is real or forged.
2. The method according to claim 1, wherein: Before inputting the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, the method further comprises: Determining the face image to be detected; Performing block processing on the face image to obtain multiple image blocks; Performing vector conversion processing on the multiple image blocks respectively to determine a vector of each image block; A vector sequence composed of the vectors of the multiple image blocks is determined as an image vector of the face image.
3. The method according to claim 1, wherein: The first semantic representation model is not provided with a head network, and the second semantic representation model is provided with a head network; The step of inputting the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively comprises: Inputting the image vector into a head network of the second semantic representation model to obtain a head feature vector output by the head network; Performing feature fusion processing according to the image vector and the head feature vector to obtain a fused image vector and a fused head feature vector; The fused image vector is input into the first semantic representation model, and the fused head feature vector is input into the first stage network of the second semantic representation model to obtain a first feature vector output by the first stage network in the first semantic representation model and a second feature vector output by the first stage network in the second semantic representation model.
4. The method according to claim 1, wherein: The determining, according to the first feature vector and the second feature vector, a weight adjustment value of the first feature vector and a weight adjustment value of the second feature vector comprises: Performing frequency domain conversion on the first eigenvector and the second eigenvector respectively to obtain a first frequency domain eigenvector and a second frequency domain eigenvector; Determining a weight adjustment value of the second eigenvector according to the first frequency domain eigenvector; A weight adjustment value of the first eigenvector is determined according to the second frequency domain eigenvector.
5. The method according to claim 1, wherein: Determining the detection result of the face image according to the first detection result of the Nth stage network in the first semantic representation model and the second detection result of the Nth stage network in the second semantic representation model includes: Determining a probability of forgery of the facial image according to the probability of forgery in the first detection result and the probability of forgery in the second detection result; Determining a true probability of the face image according to a true probability in the first detection result and a true probability in the second detection result; The detection result of the face image is determined according to the authenticity probability and the forgery probability of the face image.
6. A joint model training method, comprising: Construct an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, the head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; Acquire training data, wherein the training data includes sample face images and corresponding labels, wherein the labels represent whether the sample face images are real or forged; The image vector of the sample face image is used as the input of the joint model, and the label corresponding to the sample face image is used as the output of the joint model. The joint model is trained, and the trained joint model is used to implement the face image detection method in claim 1.
7. A facial image detection device, comprising: A first input module, used to input the image vector of the face image to be detected into the first semantic representation model and the second semantic representation model respectively, so as to obtain the first feature vector output by the first stage network in the first semantic representation model and the second feature vector output by the first stage network in the second semantic representation model; wherein the number of stage networks in the first semantic representation model and the second semantic representation model is N, the first semantic representation model is used to extract the global features of the face image, and the second semantic representation model is used to extract the local features of the face image; A feature fusion module, for performing feature fusion processing on the i-th stage network in the first semantic representation model and the i-th stage network in the second semantic representation model according to a first feature vector output by the i-th stage network in the first semantic representation model and a second feature vector output by the i-th stage network in the second semantic representation model, so as to obtain a fused first feature vector and a fused second feature vector; i is a positive integer from 1 to N-1; the initial value of i is 1; A second input module is used to input the fused first feature vector into the i+1th stage network in the first semantic representation model, and input the fused second feature vector into the i+1th stage network in the second semantic representation model, and add 1 to i; repeatedly perform feature fusion processing and input processing until a first detection result of the Nth stage network in the first semantic representation model and a second detection result of the Nth stage network in the second semantic representation model are obtained; A first determination module is used to determine the detection result of the face image according to the first detection result of the Nth stage network in the first semantic representation model and the second detection result of the Nth stage network in the second semantic representation model, wherein the detection result includes whether the face image to be detected is real or forged; The feature fusion module is specifically used for: Determining a weight adjustment value of the first feature vector and a weight adjustment value of the second feature vector according to the first feature vector and the second feature vector; Determine the fused first eigenvector according to the first eigenvector and the corresponding initial weight value and weight adjustment value; The fused second eigenvector is determined according to the second eigenvector and the corresponding initial weight value and weight adjustment value.
8. The device according to claim 7, wherein: The device further comprises: a second determination module, a block division module, a vector conversion module and a third determination module; The second determination module is used to determine the face image to be detected; The block division module is used to perform block processing on the face image to obtain multiple image blocks; The vector conversion module is used to perform vector conversion processing on the multiple image blocks respectively to determine the vector of each image block; The third determination module is used to determine the vector sequence composed of the vectors of the multiple image blocks as the image vector of the face image.
9. The device according to claim 7, wherein: The first semantic representation model is not provided with a head network, and the second semantic representation model is provided with a head network; the first input module is specifically used to: Inputting the image vector into a head network of the second semantic representation model to obtain a head feature vector output by the head network; Performing feature fusion processing according to the image vector and the head feature vector to obtain a fused image vector and a fused head feature vector; The fused image vector is input into the first semantic representation model, and the fused head feature vector is input into the first stage network of the second semantic representation model to obtain a first feature vector output by the first stage network in the first semantic representation model and a second feature vector output by the first stage network in the second semantic representation model.
10. The device according to claim 7, wherein: The feature fusion module is specifically used for: Performing frequency domain conversion on the first eigenvector and the second eigenvector respectively to obtain a first frequency domain eigenvector and a second frequency domain eigenvector; Determining a weight adjustment value of the second eigenvector according to the first frequency domain eigenvector; A weight adjustment value of the first eigenvector is determined according to the second frequency domain eigenvector.
11. The device according to claim 7, wherein: The first determination module is specifically configured to: Determining a probability of forgery of the facial image according to the probability of forgery in the first detection result and the probability of forgery in the second detection result; Determining a true probability of the face image according to a true probability in the first detection result and a true probability in the second detection result; The detection result of the face image is determined according to the authenticity probability and the forgery probability of the face image.
12. A training device for a joint model, comprising: A construction module, used to construct an initial joint model, wherein the joint model includes: N stage networks of the first semantic representation model, a head network of the second semantic representation model and N stage networks, N-1 feature fusion networks, and a head feature fusion network; wherein the i-th feature fusion network is respectively connected to two i-th stage networks and two i+1-th stage networks; the head feature fusion network is respectively connected to the head network and two first stage networks; i is a positive integer from 1 to N-1; An acquisition module, configured to acquire training data, wherein the training data includes sample face images and corresponding labels, wherein the labels represent whether the sample face images are real or forged; A training module is used to train the joint model using the image vector of the sample face image as the input of the joint model and the label corresponding to the sample face image as the output of the joint model. The trained joint model is used to implement the face image detection method in claim 1.
13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 5; or, execute the method according to claim 6.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 5; or, to execute the method according to claim 6.
15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5; or implements the steps of the method according to claim 6.
