Forgery detection of facial images

Abstract

This disclosure proposes a method for detecting forged facial images. After inputting a facial image, the method detects whether there are fusion boundaries in the facial image caused by the fusion of different images. Then, based on the detection results, a corresponding grayscale image is generated. The generated grayscale image reveals whether the input facial image is formed by the fusion of different images. If a visible boundary corresponding to the fusion boundary exists in the generated grayscale image, it indicates that the facial image is forged; conversely, if no visible boundary exists in the grayscale image, it indicates that the facial image is genuine. Therefore, this disclosure, by detecting fusion boundaries in the input facial image, can more accurately detect forged facial images. Furthermore, the detection model according to this disclosure can be trained using real images in a self-supervised manner, thereby making the facial image forgery detection method more general.

Description

Technical Field

[0001] Various exemplary embodiments of this application relate to the field of image processing technology, and more particularly to methods, apparatuses, devices, and computer-readable storage media for detecting forgery of facial images. Background Technology

[0002] Artificial intelligence (AI) refers to the intelligence exhibited by machines created by humans. AI aims to produce intelligent machines and / or programs that can react in a manner similar to human intelligence. The implementation of AI models typically requires massive amounts of training data and powerful computing capabilities. With the continuous development of information technology and neural network research, AI is increasingly being applied to various fields and scenarios, including chatbots, image recognition, speech recognition, natural language processing, autonomous driving, and smart healthcare, among others.

[0003] AI face-swapping is an emerging AI technology application based on deep learning algorithms. It can replace one face in an image or video with another. The operation is simple, and the resulting image or video can be so realistic that even humans cannot detect whether the image has been tampered with. The rapid development of AI face-swapping has enabled attackers to tamper with facial areas of images and generate new facial images; for example, they can change identities or modify facial attributes. Summary of the Invention

[0004] This disclosure proposes a method for detecting forged facial images. After inputting a facial image, the method detects whether there are fusion boundaries in the facial image caused by the fusion of different images. Then, based on the detection results, a corresponding grayscale image is generated. The generated grayscale image reveals whether the input facial image is formed by the fusion of different images. If a visible boundary corresponding to the fusion boundary exists in the generated grayscale image, it indicates that the facial image is a forged image; conversely, if no visible boundary exists in the generated grayscale image, it indicates that the facial image is a real image. Therefore, this disclosure, by detecting fusion boundaries in the input facial image, can more accurately detect forged facial images. Furthermore, the detection model according to this disclosure can be trained using real images in a self-supervised manner, thereby making the facial image forgery detection method more general.

[0005] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0006] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0007] Figure 1 A block diagram of a computing device / server in which one or more embodiments of the present disclosure may be implemented is shown;

[0008] Figure 2 A schematic diagram illustrating noise analysis and error level analysis of real and fake images according to embodiments of the present disclosure is shown.

[0009] Figure 3 A flowchart of a method for detecting forgery of facial images according to an embodiment of the present disclosure is shown;

[0010] Figure 4 A schematic diagram of a framework for detecting forgery of facial images according to an embodiment of the present disclosure is shown;

[0011] Figure 5 A schematic diagram of the image fusion process based on face detection is shown;

[0012] Figure 6 A schematic diagram of a process for generating training data according to an embodiment of the present disclosure is shown;

[0013] Figure 7 A schematic diagram illustrating the determination of a grayscale image based on a mask image according to an embodiment of the present disclosure is shown; and

[0014] Figure 8 The illustration shows a result of detecting fake regions in a face image according to an embodiment of the present disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0017] AI face-swapping has become very popular, especially on social networks, raising significant concerns about its impact on social networks and society. The forged images created through AI face-swapping could be misused for malicious purposes, potentially causing serious trust and security issues. For example, some people use AI face-swapping to create fake images for deception or pranks.

[0018] Currently, there are many AI face-swapping methods or algorithms, such as DeepFakes (DF), Face2Face (F2F), FaceSwap (FS), NeuralTextures (NT), etc. These methods typically fuse the altered face into an existing background image. The resulting face image is called a forged face image. Currently, forged face images are so realistic that even humans have difficulty distinguishing them. Generally speaking, detecting the authenticity of AI face-swapping is a very difficult task because it usually requires knowing the face-swapping method used.

[0019] Traditionally, to distinguish between fake and real face images, a binary classifier is typically trained using real face images and fake face images generated through some fusion method to achieve high accuracy in fake image detection. Traditional methods are usually trained in a supervised manner and optimized against known face forgery methods, thus achieving high detection accuracy against fake face images generated by known methods. However, this may simply be overfitting, and detection is limited to the known forgery methods trained specifically. For fake face images generated by unknown forgery methods, traditional methods usually fail to identify them, resulting in a significant drop in accuracy. Therefore, traditional face image forgery detection methods are not universal, and their accuracy is unstable and not high.

[0020] To address this, embodiments of this disclosure propose a method for detecting forged facial images, innovatively proposing the detection of fusion boundaries in forged images, achieving high accuracy for general facial forgery detection. The inventors of this application have discovered that if an image is formed by the fusion of two images, there are essential image differences inside and outside the fusion boundary. Therefore, according to embodiments of this disclosure, after inputting a facial image, the method detects whether there are fusion boundaries in the facial image caused by the fusion of different images, and then generates a corresponding grayscale image based on the detection results. The generated grayscale image can reveal whether the input facial image is formed by the fusion of different images. In other words, for an input facial image, its corresponding grayscale image can be calculated based on the input facial image. This grayscale image can not only determine whether the input facial image is forged or real, but also identify the location of the fusion boundary (in the case of a fusion boundary) through visible boundaries.

[0021] According to embodiments of this disclosure, if a visible boundary (such as a bright white ring) exists in the generated grayscale image, the face image is considered a forged image; conversely, if no visible boundary exists in the generated grayscale image, the face image is considered a genuine image. Therefore, embodiments of this disclosure can more accurately detect forged face images by detecting fusion boundaries in the input face image. Since it only assumes the existence of the fusion step and does not rely on artifact knowledge associated with a specific face forgery method, the detection method according to embodiments of this disclosure is universal.

[0022] Furthermore, the detection models according to some embodiments of this disclosure can be trained using real images in a self-supervised manner. Since no forged images generated by traditional forgery methods are used for training, the face forgery detection methods according to embodiments of this disclosure are more versatile and can achieve high detection accuracy even for forged images generated by unknown forgery methods. In fact, some embodiments of this disclosure can even be trained without using forged images generated by traditional forgery methods, thereby achieving high detection accuracy for any forged image formed through image fusion.

[0023] The facial image forgery detection method according to embodiments of this disclosure can be applied to various forgery detection scenarios. For example, it can be used by social networking sites to automatically verify whether user-submitted facial images or videos are genuine; it can be used by search engines to filter out some forged search result images; it can be applied to the needs of credit and privacy management; and it can also be used as an application programming interface (API) of a cloud platform for third parties, such as a cloud platform providing a general facial image forgery detection interface, a facial video forgery detection interface, etc.

[0024] The following is for reference Figures 1 to 8 This will illustrate the basic principles of this disclosure and provide several example implementations. Figure 1 A block diagram of a computing device / server 100 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 1 The computing device / server 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0025] like Figure 1 As shown, the computing device / server 100 is in the form of a general-purpose computing device. Components of the computing device / server 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage devices 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160. The processing unit 110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device / server 100.

[0026] The computing device / server 100 typically includes multiple computer storage media. Such media can be any available media accessible to the computing device / server 100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device / server 100.

[0027] The computing device / server 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 1 As shown, disk drives for reading from or writing to removable, non-volatile disks and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 120 may include computer program product 125 having one or more program modules, such as a detection model according to embodiments of this disclosure, configured to perform various methods or processes of various embodiments of this disclosure.

[0028] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device / server 100 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device / server 100 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0029] Input device 150 can be one or more input devices, such as a mouse, keyboard, trackball, camera, etc. Output device 160 can be one or more output devices, such as a monitor, speaker, printer, etc. The computing device / server 100 can also communicate as needed with one or more external devices (not shown) via communication unit 140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with the computing device / server 100, or with any device (e.g., network card, modem, etc.) that enables the computing device / server 100 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0030] like Figure 1 As shown, the computing device / server 100 can receive an input image 171 including a face via the input device 150, then process the input image 171 using the program product 125, and output a grayscale image 172 as the processing result via the output device 160. Figure 1 In the example, input image 171 is a forged image formed by fusing two images. Therefore, the output grayscale image 172 contains a visible boundary with a bright white ring, indicating the fusion point of the two images. Since the white boundary in grayscale image 172 resembles an X-ray, the grayscale image in this embodiment is also referred to as a "face X-ray." Therefore, embodiments of this disclosure can quickly and accurately detect the fusion boundary (if present) in input image 171, thus not only indicating whether the input face image is a forged image, but also further pointing to the location of the forged region within the forged image.

[0031] Those skilled in the art should understand that, although Figure 1The diagram shows the input image 171 being received via input unit 150 and the grayscale image 172 being output via output unit 160. However, the input image 171 can also be received via communication unit 140 and / or the output image 172 can be sent, or the input image 171 can be obtained via internal or external storage devices or other means. The embodiments of this disclosure do not limit the method of obtaining the input image. The following references... Figure 2-8 An example embodiment of how program product 125 generates output image 172 based on input image 171 is described in detail.

[0032] Figure 2 Illustration 200 illustrates noise analysis and error level analysis of genuine and fake images according to embodiments of the present disclosure. Embodiments of the present disclosure propose an image representation—a grayscale image—for detecting forged face images. The inventors of this application have found that most conventional face forgery methods share a common step of fusing the altered face into an existing background image, and inherent image differences exist at the fusion boundary, making the fusion boundary inherently detectable. In fact, due to different acquisition methods, each image possesses its own unique markers introduced from hardware (e.g., sensors, lenses) or software components (e.g., compression, synthesis algorithms), and these markers tend to exhibit similarity across the entire image. Conversely, images obtained through different methods exhibit varying degrees of similarity.

[0033] For example, each image possesses its own unique signature or underlying statistics, primarily stemming from two sources: 1) hardware, such as color filter array (CFA) interpolation introducing periodic patterns, camera response functions that should be similar for each color channel, and sensor noise including a series of on-chip processing steps (e.g., quantization and white balance), which introduce a unique signature; and 2) software, such as lossy compression schemes introducing consistent clogging artifacts and synthesis algorithms based on generative adversarial networks (GANs), which may leave unique imprints. Both these hardware and software factors influence image formation, leaving specific signatures that tend towards periodicity or uniformity, which can be perturbed in modified images. Therefore, inconsistencies in underlying image statistics across fusion boundaries can be used to detect fusion boundaries, thereby identifying forged face images.

[0034] Noise analysis and Error Level Analysis (ELA) are two representative types of unique labeling. Regarding noise analysis, natural images are typically filled with noise. If an image is modified, visible traces are usually left within the noise. Therefore, noise analysis can be achieved by using a very simple noise reduction filter (a separable median filter) and inverting its result. Error Level Analysis works by analyzing the compression ratio of different regions of an image and calculating their error levels to determine if the image has undergone post-processing such as stitching, modification, or smoothing. In principle, the ELA values ​​of different regions in an original image obtained from a single sampling should be similar. If a region's ELA value differs significantly from the rest of the image, this region is likely to have been modified or replaced. The general principle of ELA is to divide the image into many 8×8 squares of pixels and perform a separate color space conversion on each small block. Each modification to the image triggers a second conversion. The two conversions will naturally produce differences, which ELA can use to determine which part of the image has been modified.

[0035] like Figure 2 As shown, for the genuine image 210, its noise analysis result 211 and error level analysis result 212 exhibit similarity throughout the entire image. However, for the forged image 220, its noise analysis result 221 and error level analysis result 222 exhibit non-uniformity throughout the entire image, where the brightness of the modified image region is higher than other regions. For example, the color of the middle region between images 221 and 222 differs from the outer regions, thus indicating that the middle region of image 220 has been replaced by another image. Based on this discovery, the inventors of this application propose using the fusion boundary of two images to detect whether an image has been forged.

[0036] Figure 3 A flowchart of a method 300 for detecting forgery of facial images according to an embodiment of the present disclosure is shown. It should be understood that method 300 can be derived from references... Figure 1 The described computing device / server 100 performs the operation. It should be understood that the method 300 of the embodiments of this disclosure may be executed entirely locally on a user device (such as a smartphone, tablet, desktop computer, laptop computer, smart wearable device, etc.), or entirely on the server side (such as a single server, distributed server, cloud, etc.), or partly deployed locally on the user device and partly deployed on the server side or in the cloud. The embodiments of this disclosure do not limit this.

[0037] At 302, an input image including a face is obtained. For example, the detection model obtains an input image 171 to be detected, which includes a human face. For example, the detection model according to an embodiment of the present disclosure can be a fully convolutional neural network (FCNN) model, and the input image 171 can be input into the FCNN model. In an embodiment of the present disclosure, the input image 171 can be a forged image formed by face replacement. Figure 5 An example forgery process for forming a forged image is further described. It should be understood that since embodiments of this disclosure detect the fusion boundary of two merged images, forged images generated through image fusion can be detected. However, for completely forged or generated images, since they lack fusion boundaries, they cannot be identified as forged images.

[0038] At step 304, a fusion boundary is detected in the input image. For example, the FCNN model can detect whether a fusion boundary exists in the input image 171 due to the fusion of different images. Generally, in a face spoofing scenario, the area inside the fusion boundary is a face or at least a portion of a face (e.g., a nose and / or mouth). In some embodiments, if a fusion boundary is detected, it indicates that the facial region within the fusion boundary in the input image is spoofed; if no fusion boundary is detected, it indicates that the input image is a real image.

[0039] At step 306, based on the detection, a grayscale image associated with the input image is generated, wherein the grayscale image indicates whether the face in the input image has been forged. For example, a grayscale image 172 can be generated based on the detection results of the fusion boundary. By observing or analyzing the grayscale image 172, it can be determined whether the face in the input image has been forged. In some embodiments, it can be determined whether there is a visible boundary in the grayscale image corresponding to the position of the fusion boundary; that is, the position of the visible boundary in the grayscale image corresponds to the position of the fusion boundary in the input image. For example, the visible boundary can be represented as a bright white ring in the grayscale image, which can indicate inconsistencies in the original image inside and outside the boundary and / or unevenness at the boundary. If it exists, it indicates that the face region within the fusion boundary in the input image has been forged; otherwise, if it does not exist, it indicates that the input image is a genuine image. Therefore, the method 300 of the embodiments of this disclosure can more accurately detect forged face images by detecting the fusion boundary in the input face image.

[0040] The grayscale image (i.e., "face X-ray") proposed in the embodiments of this disclosure enables the development of a general face forgery detection model because it only assumes the existence of a fusion step and does not rely on any artifact knowledge associated with a specific face forgery algorithm. This generality is applicable to most existing face forgery algorithms.

[0041] In some embodiments, the detection model can be trained using self-supervised learning, which utilizes a large number of fused images synthesized from real images, eliminating the need for fake images generated by any existing face forgery methods. Therefore, the methods of the embodiments of this disclosure remain effective when applied to forged images generated by unknown face forgery methods; conversely, the performance of most conventional face forgery detection methods degrades significantly. Self-supervised learning is a machine learning method where the model learns directly from unlabeled data without the need for manual annotation. Therefore, self-supervised learning can utilize data to derive various labels without additional annotation costs, saving the cost of obtaining training data and enabling the acquisition of a large amount of training data, thereby improving the model's detection accuracy.

[0042] Figure 4 A schematic diagram of a frame 400 for forgery detection of facial images according to an embodiment of the present disclosure is shown. Figure 4 As shown, the framework 400 mainly includes a fully convolutional neural network (FCNN) model 410 and a classifier 420. An FCNN model is a neural network composed entirely of convolutional layers. Its difference from a classic CNN model is that FCNN replaces all fully connected layers in a classic CNN with convolutional layers. Examples of FCNN include HRNet and CovNet. Therefore, an FCNN model can take an image as input and output an image as the processing result. In some embodiments, the FCNN model 410 may include a backbone network and an upsampling network. For example, representations from all four different resolutions can be connected to the same size 64×64, followed by a 1×1 convolutional layer with one output channel, a bilinear upsampling layer with a 256×256 output size, and a sigmoid function.

[0043] Classifier 420 can be a neural network model including average pooling layers, fully connected layers, and softmax activation layers, capable of determining the probability that a face in an input image is forged based on a grayscale image. Classifier 420 can essentially be a binary classifier that outputs a classification based on the input image, and can be a small neural network model.

[0044] like Figure 4 As shown, the FCNN model 410 generates a corresponding grayscale image 172 based on the input image 171, and the classifier 420 generates a detection result 173 for the input image 171 based on the grayscale image 172 (which includes a white visible boundary), indicating that the input image 171 is a fake image, and the fake region is the face region within the white visible boundary in the grayscale image 172.

[0045] Similarly, the FCNN model 410 generates a corresponding grayscale image 472 based on the input image 471, and the classifier 420 generates a detection result 473 for the input image 471 based on the grayscale image 472 (which is a completely black image), indicating that the input image 471 is a real image in which there is no fusion boundary and therefore no image fusion process.

[0046] In some embodiments, the framework 400 can perform not only image forgery detection but also video forgery detection. Specifically, multiple frames can be extracted from the video, for example, at a predetermined frame extraction frequency of one minute. Then, each frame can be analyzed. If a frame contains a face, the frame can be input into the FCNN model 410 for face forgery detection.

[0047] Figure 5 A schematic diagram of an image fusion process 500 based on face detection is shown, which can be an exemplary method for generating a forged image. It mainly includes three processes. In the first process (520), a face region 530 is detected in image 510, then a similar face is found, and in the second process (540), face 530 is replaced with a similar face 550. Next, in the third process (560), the similar face 550 is fused into the original image 510, thereby forming a forged image 570. Compared to the original image 510, the face in the forged image 570 has been replaced.

[0048] Generally, traditional face forgery detection methods typically focus on the second process described above, which is based on a dataset including synthetic videos generated by the forgery method and real videos, and trains a supervised binary classifier. While achieving high detection accuracy on test datasets, performance significantly degrades when the trained model is applied to unknown forged images. In contrast, the face forgery detection method according to embodiments of this disclosure focuses on the third process described above. Unlike the synthetic artifacts obtained in the second process, embodiments of this disclosure attempt to locate the fusion boundary typically introduced in the third process of face forgery. Embodiments of this disclosure are based on a key observation: when two images are fused to form a single image, there is a fundamental image difference inside and outside the fusion boundary.

[0049] Figure 6 A schematic diagram of a process 600 for generating training data according to an embodiment of the present disclosure is shown. According to an embodiment of the present disclosure, the process 600 for generating training data may mainly include three stages.

[0050] In the first stage of the process 600 of generating training data, real images 610 (i.e., image I) are obtained from the training dataset. BMultiple facial landmarks are extracted from image 610 to obtain image 620 with facial landmarks. Then, the extracted facial landmarks are used to search for the best matching image 640 of another face (i.e., image I). F The target image 640 is then used to replace the face in the real image 610. In some embodiments, a target image including another face that matches the facial keypoints in the real image 610 can be searched. For example, similar face images can be searched from a random subset of training videos or images based on the Euclidean distance between the facial keypoints. In some embodiments, to improve the randomness of the training data, a target image set 630 including other faces that match the face in the real image (e.g., it may include 100 images with the same or similar facial frames) can be searched, and then an image is randomly selected from the target image set 630 as the target image 640. In this way, the randomness and diversity of the training data can be further improved, thereby improving the generality of the detection model.

[0051] In the second stage of the process 600 of generating training data, a mask is generated to define the forgery region. In some embodiments, a mask image 650 for replacing at least a portion of the face can be determined based on multiple facial key points in image 620. For example, the outermost points of multiple facial key points can be connected to form a mask, for example, by means of a convex hull. That is, the initial mask can be defined as the convex hull of the real image 610. Since various face forgery methods do not always focus on the same part of the face, there may be various different shapes of forgery regions when forging images. For example, it is possible to forge the entire face or only a part of the face (such as the mouth). In order to cover as many mask shapes as possible, random shape deformation can be used (e.g., using a thin plate spline estimated from a source of 16 points selected from a 4×4 grid to a target of 16 points using random offset deformation), and then a Gaussian blur with a random kernel size can be applied to generate the final mask. That is, the mask region in the mask image 650 can be randomly deformed and the edges of the randomly deformed mask region can be Gaussian blurred to generate the final mask image 660. By using random deformation and / or Gaussian blur, the mask region can be made more random and the mask edges more blurred, which helps in the subsequent generation of visible boundaries.

[0052] In the third stage of the training data generation process 600, a fused image 680 is generated. At 675, the fused image 680 can be generated by image fusion based on the real image 610' (i.e., the real image 610 or a copy thereof), the target image 640, and the mask image 660. In some embodiments, before image fusion, the target image 640 can be color-corrected to match the colors in the real image 610. Furthermore, at 665, based on the mask image 660, the mask boundary (i.e., the white bright ring in the grayscale image 670) is obtained, thereby generating the corresponding grayscale image 670.

[0053] In some embodiments of this disclosure, given an input face image I, it is desirable to determine whether the image is obtained by combining two images I. F and Image I B The obtained image I M As shown in equation (1):

[0054] I M =M⊙I F +(1-M)⊙I B (1)

[0055] Where ⊙ represents element-wise multiplication, I F It is a foreground fake face image with desired facial attributes, I B The image provides the background. M is a mask that defines the forgery region, where each pixel of M has a grayscale value between 0 and 1. When all values ​​are capped to 0 or 1, it is a binary mask. Furthermore, some color correction techniques can be applied to the foreground image I before fusion. F This is done so that its color matches the color of the background image.

[0056] In some embodiments of this disclosure, a grayscale image can be defined as image B such that if the input image is a forged image, B will reveal the fusion boundary; while if the input image is a real image, the value of all pixels in B will be zero. For an input face image I, its grayscale image B can be defined by the following equation (2):

[0057] B i，j =4·M i，j ·(1-M i，j (2)

[0058] Here, the subscripts (i,j) represent the indexes of the pixel positions, and M is the mask determined by the input image I. If the input image is real, the mask M is a simple blank image with all pixel values ​​either 0 or all 1. Otherwise, the mask M will be a non-simple image defining the foreground image region. i,j ·(1-M i,jThe maximum value of ) is no greater than 0.25, and in fact only when M i,j The maximum value of 0.25 is reached only when the value is 0.5. For this reason, the pixel B in a grayscale image... i,j It always takes a value between 0 and 1.

[0059] refer to Figure 7 This illustrates a schematic diagram 700 showing the determination of a grayscale image B based on a mask image M according to an embodiment of the present disclosure. Figure 7 As shown, the grayscale image 710 is obtained by element-wise multiplication of the mask image 720 and its inverted inverse mask image 730. In this way, the grayscale image can be determined based on the mask image, and the visible boundaries in the determined grayscale image have a certain width.

[0060] Return to reference Figure 6 If mask image 660 is a binary mask, then a 3×3 Gaussian kernel is used before step 665 to convert the binary mask into a soft mask. In embodiments of this disclosure, it is generally assumed that the mask image 660 is not a binary mask. A binary mask would hinder the generation of grayscale images in embodiments of this disclosure, as a binary mask cannot generate the visible boundaries to be detected.

[0061] Therefore, the purpose of embodiments of this disclosure is to discover non-simple soft masks, thereby enabling the acquisition of fusion boundaries and obtaining a fused image 680 formed by fusing two images. As discussed above, images from different sources have inherent differences due to variations in the image acquisition process, even though these differences may be subtle or invisible to the human eye. Therefore, the grayscale image according to embodiments of this disclosure is a computational representation used to discover such differences in face images from unknown input.

[0062] according to Figure 6 The process 600 for generating training data dynamically and online as the training process progresses, and then trains the FCNN model 410 in a self-supervised manner. For example, the fused image 680 and the corresponding grayscale image 670 are used in pairs to train the FCNN model 410. Since the process 600 for generating the fused image does not use one or more traditional face image forgery methods, and adds many random factors, the generated training data is more random, enabling the FCNN model 410 to achieve high detection accuracy for forged images generated by various methods.

[0063] Through the above process 600, a large amount of training data can be generated using only real images 610. Let the generated training dataset be... Where I represents the image, B represents the corresponding grayscale image, and c is a binary scalar indicating whether image I is a fused image. The FCNN-based framework 400 disclosed herein can generate a grayscale image B from the input image I, and then output the probability that the input image I is true based on the grayscale image B.

[0064] exist Figure 4 The framework 400 shown can be trained using a joint training method, allowing For the predicted grayscale image, where N1 = N2 b for Figure 4 The FCNN model 410 shown is... Let be the predicted true or false probability, where NN c for Figure 4 The classifier 420 shown is an example. During the training phase, existing or future-developed loss functions can be used for the two predictions. For example, for the FCNN model 410, the cross-entropy loss function L can be used. b The accuracy of the prediction is estimated as shown in equation (3) below:

[0065]

[0066] Where N is the total number of pixels in the feature map, and i and j are the indices representing the pixel positions.

[0067] For classifier 420, the loss function L c It can be shown in the following formula (4):

[0068]

[0069] Therefore, the total loss function of frame 400 can be defined as L = λL b +L c , where λ is the equilibrium L b and L c The loss weights. In some embodiments, λ can be set to a large value (such as 100) to make the network more focused on learning grayscale image prediction. In some embodiments, the framework 400 in the embodiments of this disclosure can be trained using an end-to-end backpropagation approach.

[0070] Figure 8 An illustration 800 showing the effect of detecting fake regions in a face image according to an embodiment of the present disclosure is provided. Figure 8As shown, the first column represents the forged input image, the second column represents the true result of the fusion boundary, which can be obtained by calculating the absolute element-wise difference between the forged image and the real image, and the third column represents the predicted result of the fusion boundary according to the embodiments of this disclosure, which presents an indication of the forged region on the original input image (e.g., indicated by a white boundary). From the true result in the second column, it can be seen that the portion of the face below the eyelashes in input image 810 is forged, the complete face in input image 820 is forged, and a portion of the face (nose and mouth) in input image 830 is forged. By comparing the second and third columns, it can be found that regardless of whether a complete face or a portion of the face is replaced, the method of the embodiments of this disclosure can accurately determine whether a face image has been forged and the specific forged region through the fusion boundary, thereby achieving more accurate and more general detection of forged face images.

[0071] Furthermore, the inventors discovered through experiments that the face image forgery detection method of the embodiments of this disclosure can significantly improve generalization ability. The embodiments of this disclosure achieve high detection accuracy on unknown face forgery methods and can also achieve very high detection accuracy on various known face forgery methods.

[0072] Therefore, embodiments of this disclosure use a more general approach to achieve forgery detection: grayscale images (i.e., "face X-rays"). The generalization ability of some embodiments of this disclosure stems primarily from two factors: first, detecting the fusion boundaries of images rather than focusing on specific forgery artifacts; and second, automatically constructing a large number of training samples from real images, allowing the model to be adapted to focus more on the fusion boundaries of images. Finally, the methods of embodiments of this disclosure can achieve high detection accuracy even using only self-supervised training data.

[0073] Furthermore, the training data in the embodiments of this disclosure is not generated by one or more existing face image forgery algorithms; instead, it is generated more randomly from real images. Simultaneously, the regions within the boundaries of the fused image of the training data generated according to the embodiments of this disclosure are real images, not synthetic ones, which helps the model avoid overfitting and thus achieves good generalization performance.

[0074] The methods and functions described herein can be performed, at least in part, by one or more hardware logic components. Examples, but not limited to, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0075] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0077] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of a single implementation may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0078] The following are some example implementations of this disclosure.

[0079] In one aspect, a computer-implemented method is provided. The method includes: obtaining an input image including a face; detecting fusion boundaries in the input image; and based on the detection, generating a grayscale image associated with the input image, wherein the grayscale image indicates whether the face in the input image has been forged.

[0080] In some embodiments, the method further includes determining whether a visible boundary exists in the grayscale image, wherein the visible boundary corresponds to a fusion boundary; determining that a facial region within the fusion boundary in the input image is faked based on the determination that a visible boundary exists in the grayscale image; and determining that the input image is a real image based on the determination that no visible boundary exists in the grayscale image.

[0081] In some embodiments, generating a grayscale image associated with an input image includes generating the grayscale image using a fully convolutional neural network (FCNN) model, and further includes determining the probability that a face in the input image is forged based on the grayscale image by a classifier, the classifier being a neural network model including pooling layers, fully connected layers, and activation layers.

[0082] In some embodiments, the method further includes: generating a fused image and a corresponding grayscale image based on real images in the training dataset; and using the fused image and the corresponding grayscale image to train an FCNN model.

[0083] In some embodiments, generating the corresponding grayscale image includes: detecting multiple facial key points in a real image; determining a mask image for replacing at least a portion of the face based on the multiple facial key points; and generating the corresponding grayscale image based on the mask image.

[0084] In some embodiments, determining the mask image includes: randomly deforming a mask region in the mask image; and Gaussian blurring the edges of the randomly deformed mask region.

[0085] In some embodiments, generating the fused image includes: searching for a target image that includes another face matching a face in a real image based on facial key points; and generating the fused image based on the real image, the target image, and the mask image.

[0086] In some embodiments, the search for a target image that includes another face matching a face in a real image includes: searching a set of target images that include other faces matching a face in a real image; and randomly selecting an image from the set of target images as the target image.

[0087] In some embodiments, the method further includes: determining a fake region in an input image based on a visible boundary in a grayscale image, the fake region including at least a portion of a face; and presenting an indication of the fake region on the input image.

[0088] In some embodiments, obtaining an input image that includes a face includes: extracting an image frame from a video; and determining the image frame as an input image based on determining that the image frame includes a face.

[0089] In another aspect, an electronic device is provided. The electronic device includes: a processing unit; and a memory coupled to the processing unit and storing instructions that, when executed by the processing unit, perform actions including: acquiring an input image including a face; detecting a fusion boundary in the input image; and, based on the detection, generating a grayscale image associated with the input image, wherein the grayscale image indicates whether the face in the input image has been forged.

[0090] In some embodiments, the action further includes determining whether a visible boundary exists in the grayscale image, wherein the visible boundary corresponds to a fusion boundary; determining that a facial region within the fusion boundary in the input image is faked based on the determination that a visible boundary exists in the grayscale image; and determining that the input image is a real image based on the determination that no visible boundary exists in the grayscale image.

[0091] In some embodiments, generating a grayscale image associated with an input image includes generating the grayscale image by a fully convolutional neural network (FCNN) model, and further includes determining the probability that a face in the input image is forged based on the grayscale image by a classifier, the classifier being a neural network model including pooling layers, fully connected layers, and activation layers.

[0092] In some embodiments, the action further includes: generating a fused image and a corresponding grayscale image based on real images in the training dataset; and using the fused image and the corresponding grayscale image to train an FCNN model.

[0093] In some embodiments, generating the corresponding grayscale image includes: detecting multiple facial key points in a real image; determining a mask image for replacing at least a portion of the face based on the multiple facial key points; and generating the corresponding grayscale image based on the mask image.

[0094] In some embodiments, determining the mask image includes: randomly deforming a mask region in the mask image; and Gaussian blurring the edges of the randomly deformed mask region.

[0095] In some embodiments, generating the fused image includes: searching for a target image that includes another face matching a face in a real image based on facial key points; and generating the fused image based on the real image, the target image, and the mask image.

[0096] In some embodiments, the search for a target image that includes another face matching a face in a real image includes: searching a set of target images that include other faces matching a face in a real image; and randomly selecting an image from the set of target images as the target image.

[0097] In some embodiments, the action further includes: determining a fake region in the input image based on a visible boundary in the grayscale image, the fake region including at least a portion of a face; and indicating the presentation of the fake region on the input image.

[0098] In some embodiments, obtaining an input image that includes a face includes: extracting an image frame from a video; and determining the image frame as an input image based on determining that the image frame includes a face.

[0099] In another aspect, a computer program product is provided. This computer program product is stored in a non-transitory computer storage medium and includes machine-executable instructions that, when executed in a device, cause the device to: acquire an input image including a face; detect a fusion boundary in the input image; and, based on the detection, generate a grayscale image associated with the input image, the grayscale image indicating whether the face in the input image has been forged.

[0100] In some embodiments, the machine-executable instructions, when executed in the device, further cause the device to: determine whether a visible boundary exists in the grayscale image, wherein the visible boundary corresponds to a blending boundary; determine, based on the determination that a visible boundary exists in the grayscale image, that a facial region within the blending boundary in the input image is forged; and determine, based on the determination that a visible boundary does not exist in the grayscale image, that the input image is a real image.

[0101] In some embodiments, generating a grayscale image associated with an input image includes: generating the grayscale image by a fully convolutional neural network (FCNN) model, and the machine-executable instructions, when executed in the device, also cause the device to: determine, based on the grayscale image, the probability that a face in the input image is forged by a classifier, the classifier being a neural network model including pooling layers, fully connected layers, and activation layers.

[0102] In some embodiments, the machine-executable instructions, when executed in the device, also cause the device to: generate a fused image and a corresponding grayscale image based on real images in the training dataset; and train an FCNN model using the fused image and the corresponding grayscale image.

[0103] In some embodiments, generating the corresponding grayscale image includes: detecting multiple facial key points in a real image; determining a mask image for replacing at least a portion of the face based on the multiple facial key points; and generating the corresponding grayscale image based on the mask image.

[0104] In some embodiments, determining the mask image includes: randomly deforming a mask region in the mask image; and Gaussian blurring the edges of the randomly deformed mask region.

[0105] In some embodiments, generating the fused image includes: searching for a target image that includes another face matching a face in a real image based on facial key points; and generating the fused image based on the real image, the target image, and the mask image.

[0106] In some embodiments, the search for a target image that includes another face matching a face in a real image includes: searching a set of target images that include other faces matching a face in a real image; and randomly selecting an image from the set of target images as the target image.

[0107] In some embodiments, the machine-executable instructions, when executed in the device, further cause the device to: determine a fake region in an input image based on a visible boundary in a grayscale image, the fake region including at least a portion of a face; and provide an instruction to present the fake region on the input image.

[0108] In some embodiments, obtaining an input image that includes a face includes: extracting an image frame from a video; and determining the image frame as an input image based on determining that the image frame includes a face.

[0109] Although this disclosure has been described using language specific to structural and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A computer-implemented method, comprising: Obtain the input image, including the face; Detect the fusion boundary in the input image; as well as Based on the detection, a grayscale image associated with the input image is generated by a detection model. The grayscale image indicates whether the face in the input image has been forged. The detection model is trained as follows: Based on real images in the training dataset, generate fused images and corresponding grayscale images; as well as The detection model is trained using the fused image and the corresponding grayscale image. Generating the corresponding grayscale image includes: Detect multiple facial key points in real images; Based on the aforementioned multiple facial key points, a mask image for replacing at least a portion of the face is determined by randomly deforming the mask region in the mask image; and Based on the mask image, the corresponding grayscale image is generated.

2. The method according to claim 1, further comprising: Determine whether there is a visible boundary in the grayscale image, the visible boundary corresponding to the fusion boundary; Based on the determination that the visible boundary exists in the grayscale image, it is determined that the facial region within the fusion boundary in the input image is faked; as well as Based on the determination that the visible boundary does not exist in the grayscale image, the input image is determined to be a real image.

3. The method of claim 1, wherein generating a grayscale image associated with the input image through the detection model comprises: The grayscale image is generated using a fully convolutional neural network (FCNN) model, and the method further includes: A classifier determines the probability that the face in the input image is faked based on the grayscale image. The classifier is a neural network model that includes pooling layers, fully connected layers, and activation layers.

4. The method of claim 1, wherein determining the mask image further comprises: Gaussian blur is applied to the edges of the randomly deformed mask region.

5. The method according to claim 1, The generation of the fused image includes: Based on the facial key points, search for a target image that includes another face that matches the face in the real image; as well as The fused image is generated based on the real image, the target image, and the mask image.

6. The method of claim 5, wherein searching for a target image comprising another face matching the face in the real image comprises: The search includes a set of target images of other faces that match the face in the real image; as well as Randomly select one image from the set of target images as the target image.

7. The method according to claim 2, further comprising: Based on the visible boundaries in the grayscale image, a fake region in the input image is determined, the fake region including at least a portion of the face; as well as An indication of the fake region is presented on the input image.

8. The method of claim 1, wherein obtaining the input image including a face comprises: Extracting image frames from video; as well as The image frame is determined as the input image based on the determination that the image frame includes a face.

9. An electronic device, comprising: Processing unit; A memory, coupled to the processing unit and storing instructions that, when executed by the processing unit, perform actions, including: Obtain the input image, including the face; Detect the fusion boundary in the input image; and Based on the detection, a grayscale image associated with the input image is generated by a detection model. The grayscale image indicates whether the face in the input image has been forged by determining whether a bright ring exists in the grayscale image. The detection model is trained as follows: Based on real images from the training dataset, generate a fused image and its corresponding grayscale image; and The detection model is trained using the fused image and the corresponding grayscale image. Generating the corresponding grayscale image includes: Detect multiple facial key points in real images; Based on the aforementioned multiple facial key points, a mask image for replacing at least a portion of the face is determined by randomly deforming the mask region in the mask image; and Based on the mask image, the corresponding grayscale image is generated.

10. The device according to claim 9, wherein the action further includes: Determine whether there is a visible boundary in the grayscale image, and the visible boundary corresponds to the fusion boundary; Based on the determination that the visible boundary exists in the grayscale image, it is determined that the facial region within the fusion boundary in the input image is faked; as well as Based on the determination that the visible boundary does not exist in the grayscale image, the input image is determined to be a real image.

11. The apparatus of claim 10, wherein generating a grayscale image associated with the input image via the detection model comprises: The grayscale image is generated by a fully convolutional neural network (FCNN) model, and the action further includes: A classifier determines the probability that the face in the input image is faked based on the grayscale image. The classifier is a neural network model that includes pooling layers, fully connected layers, and activation layers.

12. The apparatus of claim 9, wherein determining the mask image comprises: Gaussian blur is applied to the edges of the randomly deformed mask region.

13. The apparatus of claim 9, wherein generating the fused image comprises: Based on the facial key points, search for a target image that includes another face that matches the face in the real image; as well as The fused image is generated based on the real image, the target image, and the mask image.

14. The device of claim 13, wherein searching for a target image comprising another face matching the face in the real image comprises: The search includes a set of target images of other faces that match the face in the real image; as well as Randomly select one image from the set of target images as the target image.

15. The device according to claim 10, wherein the action further includes: Based on the visible boundaries in the grayscale image, a fake region in the input image is determined, the fake region including at least a portion of the face; as well as An indication of the fake region is presented on the input image.

16. A computer program product stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed in a device, cause the device to: Obtain the input image, including the face; Detect the fusion boundary in the input image; as well as Based on the detection, a grayscale image associated with the input image is generated by a detection model. The grayscale image indicates whether the face in the input image has been forged by determining whether a bright ring exists in the grayscale image. The detection model is trained as follows: Based on real images in the training dataset, generate fused images and corresponding grayscale images; as well as The detection model is trained using the fused image and the corresponding grayscale image. Generating the corresponding grayscale image includes: Detect multiple facial key points in real images; Based on the aforementioned multiple facial key points, a mask image for replacing at least a portion of the face is determined by randomly deforming the mask region in the mask image; and Based on the mask image, the corresponding grayscale image is generated.

17. The computer program product of claim 16, wherein the device is further configured to: Determine whether there is a visible boundary in the grayscale image, the visible boundary corresponding to the fusion boundary; Based on the determination that the visible boundary exists in the grayscale image, it is determined that the facial region within the fusion boundary in the input image is forged; and Based on the determination that the visible boundary does not exist in the grayscale image, the input image is determined to be a real image.

18. The computer program product of claim 16, wherein generating a grayscale image associated with the input image through the detection model comprises: The grayscale image is generated by a fully convolutional neural network (FCNN) model, and the device is further configured to: A classifier determines the probability that the face in the input image is faked based on the grayscale image. The classifier is a neural network model that includes pooling layers, fully connected layers, and activation layers.

19. The computer program product of claim 16, wherein determining the mask image further comprises: Gaussian blur is applied to the edges of the randomly deformed mask region.

20. The computer program product of claim 16, wherein generating the fused image comprises: Based on the facial key points, search for a target image that includes another face that matches the face in the real image; as well as The fused image is generated based on the real image, the target image, and the mask image.

Images