A face deep fake identification method based on deep learning and related equipment

Abstract

This invention discloses a deep learning-based method and related equipment for identifying deep face forgeries. The method includes: Step 1: Training an Encoder-Decoder deep network to learn the distribution features of reconstructed face images and real face images; Step 2: Extracting real and forged face images from the target image; Step 3: Aligning the faces in the real and forged face images; Step 4: Reconstructing the real and forged face images using the Encoder-Decoder deep network, extracting the distribution features of the reconstructed real and forged face images; Step 5: Calculating the cosine similarity of the distribution features of the reconstructed real and forged face images, and classifying the real and forged face images accordingly to obtain the identification result. This method utilizes the distribution differences in the reconstructed features of real and forged faces to more effectively identify deep face forgeries, greatly improving the accuracy of the identification results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to the identification technology of fake faces in images. Background Technology

[0002] In recent years, computer vision technology has advanced rapidly, but this has also brought about the potential for its misuse. The rapid development of deepfake technology has created potential risks and hidden dangers. Besides the leakage of personal information, negative news, fake news, and statements by key figures created using deepfake tools are difficult to verify, spread rapidly, and seriously mislead public opinion, impacting national security. They may also become a new tool for cybercrime, infringing on citizens' legitimate rights. Current AI image synthesis technology can create realistic virtual characters, raising widespread concerns that "seeing is not always believing," which poses a serious challenge to information security.

[0003] The rapid and accurate identification of deepfake videos is a cutting-edge international challenge and a global problem. This method will greatly improve the efficiency of the public security system in investigating and collecting evidence from deepfake videos, safeguard national security and social stability, save police resources, and has significant social benefits.

[0004] Most existing methods for detecting fake faces identify whether a face is fake by analyzing the specific synthesis pattern used in the input image. For example, the Face X-Ray algorithm proposed by Microsoft Research Asia uses the boundaries of image fusion as a synthesis pattern to determine whether it is fake. This algorithm assumes that each synthesized face image is composed of at least two superimposed images, that is, the center of the face comes from one image, while the surrounding area comes from another. However, with the development of forgery technology, over-reliance on specific known synthesis patterns can easily lead to the inability to identify fake samples generated by entirely new synthesis methods. At the same time, noise such as compression, blurring, and saturation imbalance during image transmission can also destroy known synthesis patterns, thus affecting the accuracy of fake face detection algorithms. Summary of the Invention

[0005] To address the issues with the accuracy of existing video image face anti-spoofing identification technologies, the present invention aims to provide a deep learning-based face deep forgery identification scheme. This scheme utilizes the distribution differences in the reconstructed features of real and forged faces to more effectively identify deep forgeries, significantly improving the accuracy of the identification results.

[0006] To achieve the above objectives, a first aspect of the present invention provides a deep learning-based method for identifying deep face forgeries, comprising:

[0007] Step 1: Train an Encoder-Decoder deep network to learn the distribution features of reconstructed face images and real face images;

[0008] Step 2: Extract the real face and the fake face image of the target from the image;

[0009] Step 3: Align the faces in the real face image and the fake face image;

[0010] Step 4: Use an Encoder-Decoder deep network to reconstruct real and fake face images, and extract the distribution features of the reconstructed real and fake face images;

[0011] Step 5: Calculate the cosine similarity of the distribution features of the real face reconstruction image and the fake face reconstruction image, and classify the real face image and the fake face image accordingly to obtain the identification result.

[0012] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0013] First, the distribution features of the reconstructed face image and the real face image are extracted using an Encoder-Decoder deep network model;

[0014] Next, the model weight files for the reconstructed face image and the real face image are trained and saved.

[0015] Furthermore, step 2, when extracting the real face and fake face images of the target from the image, includes:

[0016] First, a face detection algorithm was trained using Retinaface;

[0017] Next, the trained face detection algorithm is used to extract the real face and fake face images of the target in the image.

[0018] Furthermore, step 3, when aligning the face images, includes:

[0019] First, use facial key points to adjust the angle and position of the face;

[0020] Next, the faces in the real face image and the fake face image are aligned based on the adjusted angle and position.

[0021] Furthermore, step 4, in the process of reconstruction and extraction of distribution features, includes:

[0022] First, the aligned real and fake face images are adjusted to a preset size, and then the real and fake face images are reconstructed using an Encoder-Decoder deep network.

[0023] Next, the input image is encoded by the Encoder-Decoder deep network and then decoded by the Decoder.

[0024] Finally, the distribution features of the real face reconstruction image and the fake face reconstruction image are extracted and saved.

[0025] Furthermore, step 5 calculates the cosine similarity of the distribution features of the real face reconstruction image and the fake face reconstruction image using the following formula:

[0026] S(Sa,Sb)=cos(Sa,Sb)

[0027] Where cos(Sa,Sb) represents the cosine similarity of the distribution features of the reconstructed image; Sa represents the distribution features of the real face reconstruction image; and Sb represents the distribution features of the fake face reconstruction image.

[0028] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the deep learning-based deep face forgery identification method described above.

[0029] To achieve the above objectives, a third aspect of the present invention provides a processor for running a program that executes the steps of the deep learning-based deep face forgery identification method described above.

[0030] To achieve the above objectives, a fourth aspect of the present invention provides a terminal device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program code is loaded and executed by the processor to implement the steps of the above-described deep learning-based deep face forgery identification method.

[0031] To achieve the above objectives, the fifth aspect of the present invention provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of the deep learning-based deep face forgery identification method described above.

[0032] The deep learning-based deepfake face identification scheme provided by this invention has the following advantages and positive effects compared with existing deepfake face identification methods:

[0033] (1) The proposed solution of this invention specifically utilizes the distribution differences of the reconstruction features of real faces and fake faces, enriches and develops the physical feature indicators of deepfake face identification methods, thereby effectively improving the accuracy of identification results.

[0034] (2) The proposed solution utilizes the distribution differences of the reconstructed features of real faces and fake faces without the need for comparison and recognition with other reference images. Therefore, it has strong robustness and generalization ability for identifying unknown images, and can more effectively identify deep face forgeries, thereby effectively improving the accuracy of the identification results. Attached Figure Description

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0036] Figure 1 This is a flowchart illustrating the process of deep learning-based deep face forgery identification in an embodiment of the present invention.

[0037] Figure 2 This is a structural diagram of the Encoder_Decoder reconstruction network in the example of this invention. Detailed Implementation

[0038] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0039] In view of the problems existing in the prior art, in order to improve the accuracy of deepfake face identification results, this invention abandons the technical approach of existing solutions and innovatively uses real face images to train a reconstruction network. The distribution features of the reconstructed images of real and fake faces are then input into an SVM classification network to classify the real and fake face images and output the identification results. Because fake faces and real faces have inconsistent data distributions, the reconstruction error of fake faces is more obvious, thus exhibiting strong robustness and generalization ability for identifying unknown images. This allows for more effective identification of deepfake faces, improving the accuracy of deepfake face identification results.

[0040] Accordingly, this invention provides a deep learning-based scheme for identifying deep face forgeries. See also... Figure 1 The deep learning-based face deepfake identification scheme proposed in this invention is mainly implemented through the following steps:

[0041] Step 1: Train an Encoder-Decoder deep network to learn the distribution features of reconstructed face images and real face images;

[0042] Step 2: Use a face detection algorithm to extract the real face and fake face images of the target in the image;

[0043] Step 3: Align the faces in the real face image and the fake face image;

[0044] Step 4: Use an Encoder-Decoder deep network to reconstruct real and fake face images, and extract the distribution features of the reconstructed real and fake face images;

[0045] Step 5: Calculate the cosine similarity of the distribution features of the real face reconstruction image and the fake face reconstruction image, input it into the SVM classification network, classify the real face image and the fake face image, and output the identification result.

[0046] The deep learning-based deepfake face identification scheme provided in this invention can be configured into a corresponding software program to form a deep learning-based deepfake face identification system. When running, this software program executes the aforementioned deep learning-based deepfake face identification method and stores it in a corresponding storage medium for the processor to retrieve and execute.

[0047] In actual operation, the resulting deepfake face identification system is able to identify deepfake faces based on deep learning.

[0048] As an example, the specific process of using deep learning to identify deepfake faces is as follows:

[0049] Step 1: Train an Encoder-Decoder deep network to learn the distribution features of reconstructed face images and real face images.

[0050] In its implementation, this step utilizes an Encoder-Decoder deep network model to extract the distribution features of the reconstructed face image and the real face image; and trains and saves the model weight files of the reconstructed face image and the real face image for easy subsequent use.

[0051] Step 2: Use a face detection algorithm to extract the real face and fake face images of the target in the image.

[0052] This step first uses Retinaface to train a face detection algorithm; then, the trained face detection algorithm is used to extract the real face and fake face images of the target in the image.

[0053] Step 3: Align the faces in the real face image and the fake face image.

[0054] In practice, this step first involves using facial key points to adjust the angle and position of the face.

[0055] Next, based on the adjusted angle and position data of the faces, the faces of the real face image and the fake face image are aligned.

[0056] Step 4: Use an Encoder-Decoder deep network to reconstruct real and fake face images, and extract the distribution features of the reconstructed real and fake face images.

[0057] In its specific implementation, this step includes the following sub-steps:

[0058] Step 4.1: Adjust the real and fake face images aligned in Step 3 to a preset size, and input them into the Encoder-Decoder deep network to reconstruct the real and fake face images;

[0059] Step 4.2: The Encoder-Decoder deep network first encodes the input image using an Encoder, and then decodes it using a Decoder (e.g., ...). Figure 2 (As shown).

[0060] As an example, the Encoder in this instance contains four convolutional blocks, each consisting of two 3x3 convolutions followed by a pooling layer. The last convolutional block does not use pooling. The two convolutional operations first double the number of channels, then convolve again with the number of channels unchanged.

[0061] As an example, the Decoder in this instance also consists of 4 convolutional blocks. In each convolutional block, the image size is first doubled by upsampling. At the same time, the feature vector after transpose convolution is concatenated with the feature vector generated by the corresponding Encoder layer, and then subjected to two 2*2 convolutions. The channels of the two convolutions are halved first, and then the channels remain unchanged.

[0062] Thus, if the input image size is 224x224, it will become four different feature maps of different sizes: 112x112, 56x56, 28x28, and 14x14. Then, the 14x14 feature map is upsampled or deconvolved to obtain a 28x28 feature map. The 28x28 feature map is then concatenated with the previous 28x28 feature map through channels. The concatenated feature map is then convolved and upsampled to obtain a 56x56 feature map. This is then concatenated with the previous 56x56 feature map, convolved, and upsampled again. After four upsampling operations, a processing result of 224x224 with the same size as the input image can be obtained.

[0063] Step 4.3: Then, save the distribution features of the extracted real face reconstruction image and the fake face reconstruction image.

[0064] Step 5: Calculate the cosine similarity of the distribution features of the real face reconstruction image and the fake face reconstruction image, input it into the SVM classification network, classify the real face image and the fake face image, and output the identification result.

[0065] This step specifically calculates the cosine similarity of the distribution features between the real face reconstruction image and the fake face reconstruction image using the following formula:

[0066] S(Sa,Sb)=cos(Sa,Sb)

[0067] Where cos(Sa,Sb) represents the cosine similarity of the distribution features of the reconstructed image; Sa represents the distribution features of the real face reconstruction image; and Sb represents the distribution features of the fake face reconstruction image.

[0068] In this step, during the classification calculation, the cosine similarity of the distribution features of the reconstructed real face image and the reconstructed fake face image is input into the support vector machine. The category of the face image belonging to real face or fake face is used as the label to train the support vector machine, resulting in a classification hyperplane. After training, the support vector machine classification model classifies the similarity of the processed face image to be detected according to the classification hyperplane of real face and fake face, and obtains the identification result.

[0069] As described above, the deep learning-based scheme for identifying deep face forgeries presented in this invention utilizes an Encoder-Decoder deep network to train and learn the distribution features of reconstructed face images and real face images. Then, the Retinaface face detection algorithm is used to extract the real and forged face images from the image. Facial key points are used to adjust the angle and position of the faces, aligning the real and forged face images. Next, the Encoder-Decoder deep network is used to simultaneously reconstruct the real and forged face images, extracting the distribution features of the reconstructed real and forged face images. Finally, the distribution features of the reconstructed real and forged face images are input into an SVM classification network to classify the real and forged face images and output the identification result.

[0070] The present invention utilizes the distribution differences of reconstructed features between real and fake faces, thus exhibiting strong robustness and generalization ability for identifying unknown images, and can more effectively identify deepfake faces.

[0071] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the steps of the above-described method for identifying deep face forgeries based on deep learning.

[0072] This invention also provides a processor for running a program, wherein the program executes the steps described above for the deep learning-based method for identifying deep face forgeries.

[0073] This invention also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The program code is loaded and executed by the processor to implement the steps of the above-described method for identifying deep face forgeries based on deep learning.

[0074] The present invention also provides a computer program product, which, when executed on a data processing device, is adapted to perform the steps of the above-described method for identifying deep face forgeries based on deep learning.

[0075] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0083] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] At the same time, it should be understood that any parts not described in detail in this specification belong to the prior art.

[0087] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based method for face deepfake identification, characterized in that, The face deep fake identification method utilizes the distribution difference of the reconstructed features of the real face and the fake face to identify the face deep fake, comprising: Step 1: learning the distribution features of the reconstructed face and the real face image through the training of the Encoder-Decoder deep network; Step 2: intercepting the real face and fake face images of the target in the image; Step 3: aligning the intercepted real face and fake face images; Step 4: using the Encoder-Decoder deep network to reconstruct the aligned real face and fake face images in step 3 respectively, and extracting the distribution features of the real face reconstructed image and the fake face reconstructed image; when reconstructing and extracting the distribution features, it includes: First, adjust the aligned real face and fake face images to the preset size, and reconstruct the real face and fake face images through the Encoder-Decoder deep network; Next, the input image is encoded and decoded by the Encoder-Decoder deep network; Finally, save the distribution features of the real face reconstructed image and the fake face reconstructed image; The Encoder in the Encoder encoding process includes 4 convolution blocks, each block is composed of two 3*3 convolution followed by a pooling layer, and the last convolution block does not have pooling. When performing Encoder encoding, two convolution operations first double the channel, and then the channel is unchanged and convolved again. The Decoder in the Decoder decoding process is composed of 4 convolution blocks. When performing Decoder decoding, each convolution block in the Decoder first upsamples to double the image size, and then connects the feature vector after transposed convolution to the feature vector generated by the corresponding Encoder layer. Then, after two 2*2 convolutions, the channel is first reduced by half, and then the channel is unchanged; Step 5: calculate the cosine similarity of the distribution features of the real face reconstructed image and the fake face reconstructed image extracted in step 4, input into the trained SVM classification network, and classify the real face image and the fake face image, and output the identification result; When the SVM classification network is trained, the cosine similarity of the distribution features of the real face reconstructed image and the fake face reconstructed image is input into the support vector machine to train the support vector machine with the class of the face image belonging to the real face or the fake face as the label, and obtain the classification hyperplane. The SVM classification network after training classifies the similarity of the face image to be detected according to the classification hyperplane of the real face and the fake face, and obtains the identification result.

2. The deep learning-based face deepfake authentication method of claim 1, wherein, In the specific implementation of step 1, first, the distribution features of the reconstructed face image and the real face image are extracted using the Encoder-Decoder deep network model; then, the model weight file of the reconstructed face image and the real face image is trained and saved. 3.The deep learning-based face deepfake identification method of claim 1, wherein, The step 2 comprises the following steps when intercepting the real face and the fake face image of the target in the image: First, a face detection algorithm is trained by using Retinaface; Then, the real face and the fake face image of the target in the image are intercepted by using the trained face detection algorithm. 4.The deep learning-based face deepfake identification method of claim 1, wherein, The step 3 comprises the following steps when performing face alignment: First, the angle and position of the face are adjusted by using the face key point; Then, the faces of the real face image and the fake face image are aligned based on the adjusted angle and position.

5. The deep learning-based face deepfake authentication method of claim 1, wherein, The step 5 calculates the cosine similarity of the distribution features of the real face reconstruction image and the fake face reconstruction image by the following formula: S(Sa, Sb) = cos(Sa, Sb) Wherein, cos(Sa, Sb) represents the cosine similarity of the distribution features of the reconstruction image; Sa represents the distribution features of the real face reconstruction image; Sb represents the distribution features of the fake face reconstruction image.

6. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps of the deep learning-based face deep fake identification method in any one of claims 1-5.

7. A processor for running a program, the processor comprising: The program is executed to perform the steps of the deep learning-based face deep fake identification method in any one of claims 1-5.

8. A terminal device, the device comprising a processor, a memory, and a program stored on the memory and executable on the processor, wherein the program comprises instructions for causing the processor to perform the method of any one of claims 1 to 6. The program is loaded and executed by the processor to implement the steps of the deep learning-based face deep fake identification method in any one of claims 1-5.

9. A computer program product, characterised in that, When executed on a data processing device, it is adapted to perform the steps of the deep learning-based face deep fake identification method in any one of claims 1-5.

Images