Deepfake detection model learning system and method

Abstract

This invention provides a deepfake detection model learning system and method. [Solution] The deepfake detection model training method combines training datasets using Real face images, Fake face images, and Enhanced face images, and trains a deepfake detection model using the combined training datasets. The deepfake detection model is a deep learning network model that classifies input images into Real, Fake, and Enhancement classes. As a result, by classifying even Enhancement face images, which are frequently used in deepfake detection models these days, it becomes possible to prevent errors that would cause Enhancement face images to be mistakenly identified as Fake face images.

Description

[Technical Field]

[0001] This invention relates to deepfake detection, and more particularly to a deepfake detection model with subdivided discrimination functions, training data augmentation and a training system and method therefor. [Background technology]

[0002] Deepfake detection technology typically involves developing deep learning models and augmenting training data to successfully classify images into two classes: Real and Fake. However, this approach creates a weakness in deepfake detection models regarding face enhancement. Specifically, deepfake detection models may mistakenly identify enhanced face images, which should be classified as Real face images, as Fake images.

[0003] While typical deepfake detection models utilize the training dataset as is, the varying quality of the training data can make it difficult for the model to process, potentially leading to excessive focus on abnormal data. This can result in the model overfitting or being unable to accept certain data during the initial stages of training. [Overview of the project] [Problems that the invention aims to solve]

[0004] Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to enhance the Real Face Image and Fake Face Image, in addition to the Real Face Image and Fake Face Image. d The objective is to provide a deepfake detection model capable of classifying even facial images, as well as a learning system and method for it.

[0005] Another object of the present invention is to provide a system and method for expanding the training data for the deepfake detection model and for training the deepfake detection model more effectively using the expanded training data. [Means for solving the problem]

[0006] A deepfake detection model training method according to one embodiment of the present invention for achieving the above objective includes the steps of combining a training dataset using Real face images, Fake face images, and Enhanced face images, and training a deepfake detection model using the combined training dataset, wherein the deepfake detection model is a deep learning network model that classifies input images into Real class, Fake class, and Enhancement class.

[0007] The deepfake detection model training method according to an embodiment of the present invention may further include the step of correcting a Real face image and a Fake face image to generate an Enhanced face image. The generation step involves generating an Enhanced Face Image using the following formula: I E = αI enhance + (1-α)I

[0008] I E I is an Enhanced Face Image, and I is the original Face Image, which is either a Real Face Image or a Fake Face Image. enhance is the corrected face image generated by correcting I, and α is I enhance The combination parameter of and I may be 0 ≤ α ≤ 1.

[0009] The combining step may control the composition of the Fake face images in the training dataset to be different as the learning progresses. The combining step may increase the proportion of high-quality Fake face images with Fake quality above the standard in the training dataset as the learning progresses. The combining step may adjust the increase rate of the proportion of high-quality Fake face images. The combining step may increase the proportion of high-quality Fake face images using the following formula: r t = r min + (r max - r min )×(t / T) p

[0010] Here, r t is the proportion of high-quality Fake face images at epoch t, r min is the proportion of initial high-quality Fake face images, r max is the proportion of final high-quality Fake face images, and p may be a parameter that adjusts the increase rate of the proportion of high-quality Fake face images from the first epoch 1 to the final epoch T.

[0011] The high-quality Fake face image may be an image that has been fine-tuned and edited after face swapping or face blending.

[0012] The deepfake detection model learning method according to an embodiment of the present invention further includes a step of prediction correcting the Enhanced face image classified according to the Enhancement class, a step of restoring the original face image of the Enhanced face image with an inverse correction technique corresponding to the predicted correction technique, and a step of further training the deepfake detection model so as not to classify the restored original face image into the Enhancement class. [[ID=3……]]

[0013] A deepfake detection model learning system according to another embodiment of the present invention includes a combining unit that combines learning datasets using real face images, fake face images, and enhanced face images, and a learning unit that learns a deepfake detection model using the combined learning datasets. The deepfake detection model is a deep learning network model that classifies an input image into a real class, a fake class, and an enhancement class.

[0014] A deepfake detection method according to yet another embodiment of the present invention includes a step of acquiring an image, a step of inputting the acquired image into a deepfake detection model and classifying the image into a real class, a fake class, and an enhancement class, and a step of expressing the classification result. The deepfake detection model is a deep learning network model that classifies an input image into a real class, a fake class, and an enhancement class, and is learned using a learning dataset generated by combining real face images, fake face images, and enhanced face images.

Advantages of the Invention

[0015] As described above, according to the present invention, by classifying into a real face image, a fake face image, and an enhanced face image which is recently often used as a deepfake detection model, it becomes possible to prevent an error of misdetecting an enhanced face image as a fake face image and improve the accuracy of deepfake detection. d face image d As described above, according to the present invention, by classifying into a real face image, a fake face image, and an enhanced face image which is recently often used as a deepfake detection model, it becomes possible to prevent an error of misdetecting an enhanced face image as a fake face image and improve the accuracy of deepfake detection.

[0016] Note that according to an embodiment of the present invention, the enhanced dBy extending face images from both real and fake face images, it becomes possible to secure sufficient face images, which can then be used to improve the quality of the fake images and perform effective training. [Brief explanation of the drawing]

[0017] [Figure 1] This figure shows a deepfake detection model learning system according to one embodiment of the present invention. [Figure 2] This diagram shows a method for generating an enhanced face image. [Figure 3] This figure shows a method for generating a training dataset by combining training images. [Figure 4] This figure shows a deepfake detection model training method according to another embodiment of the present invention. [Figure 5] This figure shows a deepfake detection model learning system according to yet another embodiment of the present invention. [Figure 6] This figure shows a deepfake detection model training method according to yet another embodiment of the present invention. [Figure 7] This figure shows a deepfake detection system according to yet another embodiment of the present invention. [Modes for carrying out the invention]

[0018] The present invention will be described in more detail below with reference to the drawings.

[0019] In the embodiments of the present invention, a deepfake detection model learning system and method are presented. The deepfake detection model is a technique for classifying from Real Face Images and Fake Face Images to Enhanced Face Images (corrected face images), which are frequently used these days.

[0020] Furthermore, in the embodiments of the present invention, the Enhanced Face Images that the new type of deepfake detection model described above should distinguish are learned by extending them from Real Face Images and Fake Face Images, and for Fake Face Images, the Fake quality of the training data is improved and effective learning is performed as training progresses.

[0021] Figure 1 shows the configuration of a deepfake detection model learning system according to one embodiment of the present invention. The deepfake detection model learning system according to one embodiment of the present invention is configured to include a learning DB 110, an image correction unit 120, a learning image combination unit 130, and a learning unit 140, as shown in the figure.

[0022] The training DB110 stores training images and other data used to train the deepfake detection model (D). The stored training images include Real face images, Fake face images, and Enhanced face images. Real face images are labeled with the Real class, Fake face images are labeled with the Fake class, and Enhanced face images are labeled with the Enhancement class.

[0023] The image correction unit 120 corrects the Real face image and Fake face image stored in the learning DB 110 to generate an Enhanced face image. As a result, the Enhanced face image is distinguished into an image generated from the Real face image and an image generated from the Fake face image.

[0024] The learning image combination unit 130 generates a training dataset by combining Real face images, Fake face images, and Enhanced face images stored in the training DB 110, and adjusts the Fake quality of the Fake face images depending on the training stage.

[0025] The learning unit 140 trains the deepfake detection model (D) using the training dataset generated by the training image combination unit 130. The deepfake detection model (D) is a deep learning network model that classifies input images into Real, Fake, and Enhancement classes.

[0026] The learning unit 140 inputs images that make up the training dataset into the deepfake detection model (D) to classify them, calculates the difference between the classified class and the correct class labeled on the training data using a loss function, and updates the parameters of the deepfake detection model (D).

[0027] In the deepfake detection model (D) presented in the embodiment of the present invention, in addition to the Real class and Fake class, the Enhancement class is also classified, thereby preventing errors from occurring where the Enhancement class is mistakenly identified as the Fake class.

[0028] Below, the generation of the Enhanced Face Image by the Image Correction Unit 120 will be explained in detail with reference to Figure 2.

[0029] As shown in the upper part of Figure 2, the image correction unit 120 generates enhanced face images from real face images stored in the learning DB 110, while also generating enhanced face images from fake face images stored in the learning DB 110, as shown in the lower part of Figure 2.

[0030] The generation of the Enhanced Face Image is performed using the following formula 1. (Equation 1) I E = αI enhance + (1-α)I

[0031] Here, I EI is the Enhanced Face Image, and I is the Original Face Image (as mentioned above, either a Real Face Image or a Fake Face Image is possible), and I enhance is the corrected face image generated by inputting I into the face correction model (F), and α is I enhance The coupling parameter between and I is 0 ≤ α ≤ 1.

[0032] The face correction model (F) can utilize generative AI models, but it can also utilize other AI models, and can even be replaced by non-AI-based correction algorithms.

[0033] α is a parameter used to adjust the degree to which the original face image and the corrected face image are reflected when generating the Enhanced face image. The larger α is, the more of the corrected face image is reflected in the Enhanced face image, and the smaller α is, the more of the original face image is reflected in the Enhanced face image.

[0034] The following describes the generation of the training dataset by the training image combination unit 130. hand This will be explained in detail with reference to Figure 3.

[0035] The learning image combination unit 130 generates a training dataset by combining Real face images, Fake face images, and Enhanced face images stored in the learning DB 110. However, for the Fake face images that are combined, the Fake quality is improved during the learning process.

[0036] In other words, as the number of epochs increases, the training image combination unit 130 controls the training dataset so that the proportion of high-quality (Fake quality above a certain standard) Fake face images increases. This can be shown by the following equation 2. (Equation 2) rt = r min + (r max - r min )×(t / T) p

[0037] Here, r t This is the percentage of high-quality fake face images in epoch t, ​​and r min This is the percentage of initial high-quality fake face images, and r max is the percentage of final high-quality fake face images. p is a parameter that adjusts the rate of increase of the percentage of high-quality fake face images from the first epoch 1 to the final epoch T. When p=1, the rate of increase is linear; when p<1, the rate of increase is sharp in the first half; and when p>1, the rate of increase is sharp in the second half. to It will get more expensive.

[0038] High-quality fake face images can be used for face swapping or face swapping. stomach After blending 、 This refers to an image that has been fine-tuned and edited. On the other hand, a low-quality fake face image is a face swap or face stomach Only blending is performed, and then fine-tuning and editing are done afterward. stomach It refers to something that doesn't have an image.

[0039] In this way, the learning image combination unit 130 gradually increases the proportion of high-quality fake face images to fake face images as learning progresses. This ensures that the deepfake detection model (D) is trained with low-quality fake face images in the first half of the learning process, while the deepfake detection model (D) is trained with high-quality fake face images in the second half. As the amount of learning increases, the learning effect of the deepfake detection model (D) improves with learning using more difficult training data.

[0040] On the one hand, in order to train the deepfake detection model (D), the learning unit 140 can be applied so that the loss function dynamically varies according to the learning stage. Specifically, in the first half of the learning stage corresponding to easy learning (for example, when t < T / 2 in the above formula), RMSE (Root Mean Squared Error) is utilized as the loss function, and in the second half of the learning stage corresponding to difficult learning (for example, when t ≥ T / 2 in the above formula), M SE( M ean Squared Error) can be utilized as the loss function.

[0041] Since RMSE is the square root of MSE, the error is calculated to approximate the actual error, while MSE is not square-rooted, so the error is calculated more exaggeratedly than the actual error. When the error is calculated to be large, the deepfake detection model (D) is updated frequently during the learning process, while when the error is calculated to be small, the deepfake detection model (D) is updated less frequently during the learning process.

[0042] Focusing on this, in the embodiment of the present invention, in the simple first half of the learning stage, the error is calculated relatively small with RMSE so that the update of the deepfake detection model (D) is less frequent, and in the difficult second half of the learning stage, the error is calculated relatively large with MSE so that the update of the deepfake detection model (D) is more frequent.

[0043] Figure 4 is a diagram showing the flow of a deepfake detection model learning method according to another embodiment of the present invention.

[0044] As shown in the figure, first, in the learning images used to train the deepfake detection model (D), real face images and fake face images are acquired and stored in the learning DB 110 (S210).

[0045] Next, the image correction unit 120 corrects the Real face image and Fake face image acquired from step S210 to generate an Enhanced face image, and saves the generated Enhanced face image to the learning DB 110 (S220).

[0046] Subsequently, the learning image combination unit 130 generates a learning dataset by combining the Real face images, Fake face images, and Enhanced face images stored in the learning DB 110, while adjusting the Fake quality of the Fake face images according to the learning stage (S230).

[0047] Then, the learning unit 140 trains the deepfake detection model (D) using the training dataset generated in step S230 (S240).

[0048] Figure 5 shows the configuration of a deepfake detection model learning system according to yet another embodiment of the present invention. The deepfake detection model learning system according to the embodiment of the present invention is realized by adding a correction technique prediction model 150 and an image restoration unit 160 to the system shown in Figure 1.

[0049] The correction technique prediction model 150 receives Enhanced face images, which have been classified according to the Enhancement class by the deepfake detection model (D), from the learning unit 140, and predicts the correction technique applied to those images.

[0050] The correction technique prediction model 150 processes Enhanced face images that have been classified according to the Enhancement class by the deepfake detection model (D). In other words, images classified as Real class, Fake class, and Enhancement class by the deepfake detection model (D) are not processed by the correction technique prediction model 150.

[0051] The correction technique prediction model 150 is a deep learning network model trained to predict the correction technique applied to the input image. For training the correction technique prediction model 150, the image correction unit 120 may label the Enhanced Face Images with the correction techniques applied to generate Enhanced Face Images by correcting Real Face Images and Fake Face Images stored in the training DB 110. Then, it is possible to train the correction technique prediction model 150 using the Enhanced Face Images stored in the training DB 110.

[0052] The image restoration unit 160 applies an inverse correction technique corresponding to the correction technique predicted by the correction technique prediction model 150 to restore the original face image of the Enhanced face image.

[0053] Then, the learning unit 140 trains the deepfake detection model (D) so that it does not classify the original face image restored by the image restoration unit 160 as an Enhancement class. In other words, the deepfake detection model (D) is trained so that if the original face image of the Enhanced face image is a Real face image, the deepfake detection model (D) is trained so that it classifies the original face image restored by the image restoration unit 160 as a Real face image, and so that if the original face image of the Enhanced face image is a Fake face image, the deepfake detection model (D) is trained so that it classifies the original face image restored by the image restoration unit 160 as a Fake face image.

[0054] According to an embodiment of the present invention, the accuracy of the classification of the deepfake detection model (D) for the Enhanced Face Image can be further improved.

[0055] Figure 6 shows a flow chart of a deepfake detection model learning method according to yet another embodiment of the present invention. The deepfake detection model learning method according to the embodiment of the present invention adds steps S250 to S270 to steps S210 to S240 in Figure 4.

[0056] In the learning process in step S240, the correction technique prediction model 150 predicts the correction technique applied to the Enhanced face image classified according to the Enhancement class by the deepfake detection model (D) (S250).

[0057] Then, the image restoration unit 160 applies an inverse correction technique corresponding to the correction technique predicted in step S250 to restore the original face image of the Enhanced face image (S260).

[0058] Then, the learning unit 140 further trains the deepfake detection model (D) so that it does not classify the original face image restored in step S260 into the Enhancement class (S270).

[0059] Figure 7 shows the configuration of a deepfake detection system according to yet another embodiment of the present invention. As shown in the figure, the deepfake detection system according to the embodiment of the present invention is configured to include an image acquisition unit 310, a deepfake detection unit 320, and a detection result expression unit 330.

[0060] The image acquisition unit 310 acquires an image that is subject to deepfake detection and inputs it to the deepfake detection unit 320. The deepfake detection unit 320 then performs the following Figure 1 Alternatively, the deepfake detection model (D) trained in the system shown in Figure 5 classifies the images input by the image acquisition unit 310 into Real, Fake, and Enhancement classes. The detection result representation unit 330 represents the detection results from the deepfake detection unit 320.

[0061] Up to this point, we have described in detail the deepfake detection model learning system and method, citing preferred embodiments.

[0062] In the above example, as a deepfake detection model, from Real Face Images and Fake Face Images, furthermore, the most recently utilized Enhance d By classifying even the face image, Enhance d This update prevents errors that cause face images to be mistakenly identified as fake face images, thereby improving the accuracy of deepfake detection.

[0063] Furthermore, the Enhancement that the above deepfake detection model should identify d By extending face images from both real and fake face images, it becomes possible to secure a sufficient number of images, which can then be used to improve the quality of the fake images while performing effective training.

[0064] On the other hand, the technical ideas of the present invention can also be applied to computer-readable recording media incorporating a computer program that performs the functions of the apparatus and method according to this embodiment. Furthermore, the technical ideas of various embodiments of the present invention may be realized in computer-readable code format recorded on a computer-readable recording medium. A computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Furthermore, computer-readable code or program stored on a computer-readable recording medium may be transmitted via a network connected between computers.

[0065] Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these embodiments. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical intent described in the claims, and these are also understood to fall within the technical scope of the present invention.

Claims

1. The steps include combining a training dataset using a Real Face Image, a Fake Face Image, and an Enhanced Face Image generated by correcting a Real Face Image or a Fake Face Image, The steps include training a deepfake detection model using a combined training dataset and Includes, Deepfake detection models are This is a deep learning network model that classifies input images into Real, Fake, and Enhancement classes. The steps to combine are: As training progresses, control the composition of the Fake Face Images in the training dataset so that they differ. A deepfake detection model training method characterized by the following features.

2. The steps to combine are: The deepfake detection model training method according to claim 1, characterized in that, as training progresses, the proportion of high-quality fake face images with a fake quality above a certain standard in the training dataset increases.

3. The steps to combine are: The deepfake detection model training method according to claim 2, characterized by adjusting the rate of increase of the proportion of high-quality fake face images.

4. The steps to combine are: The following formula is used to increase the proportion of high-quality fake face images. r ｔ = r ｍｉｎ + (r ｍａｘ - r ｍｉｎ )×(t / T) ｐ Here, r ｔ This is the percentage of high-quality fake face images in epoch t. r ｍｉｎ This is the percentage of initial high-quality fake face images. r ｍａｘ This is the percentage of the final high-quality fake face image. The deepfake detection model learning method according to claim 3, characterized in that p is a parameter that adjusts the rate of increase of the proportion of high-quality fake face images from the first epoch 1 to the final epoch T.

5. High-quality fake face images are The deepfake detection model training method according to claim 2, characterized in that the image is an image that has been fine-tuned and edited after face swapping or face blending.

6. The deepfake detection model learning method according to claim 1, further comprising the step of correcting a Real Face Image and a Fake Face Image to generate an Enhanced Face Image.

7. The steps to generate are: The following formula is used to generate an Enhanced Face Image, I Ｅ = αI ｅｎｈａｎｃｅ + (1-α)I I Ｅ This is an Enhanced Face Image, I is either a Real Face Image or a Fake Face Image, as the original face image. I ｅｎｈａｎｃｅ This is a corrected face image generated by correcting I, α is I ｅｎｈａｎｃｅ The deepfake detection model learning method according to claim 6, characterized in that the coupling parameter of and I is 0 ≤ α ≤ 1.

8. A step of correcting a Real Face Image and a Fake Face Image to generate an Enhanced Face Image, The steps involve combining the training dataset using Real Face Images, Fake Face Images, and Enhanced Face Images, The steps include training a deepfake detection model using a combined training dataset, and A step to predict the correction technique for Enhanced Face Images classified according to the Enhancement class, The steps include: restoring the original face image of the enhanced face image using an inverse correction technique corresponding to the predicted correction technique; The steps include further training the deepfake detection model so that it does not classify the restored original face image into the Enhancement class, and Includes, Deepfake detection models are A deepfake detection model training method characterized by being a deep learning network model that classifies input images into Real, Fake, and Enhancement classes.

9. A combination unit that combines a training dataset using a Real Face Image, a Fake Face Image, and an Enhanced Face Image generated by correcting the Real Face Image or Fake Face Image, The learning unit trains a deepfake detection model using a combined training dataset. Includes, Deepfake detection models are This is a deep learning network model that classifies input images into Real, Fake, and Enhancement classes. The combination part is, As training progresses, control the composition of the Fake Face Images in the training dataset so that they differ. A deepfake detection model learning system characterized by the following features.

10. An image correction unit that corrects a Real Face Image and a Fake Face Image and generates an Enhanced Face Image, A combination unit that combines the training dataset using Real Face Images, Fake Face Images, and Enhanced Face Images, The learning unit trains a deepfake detection model using a combined training dataset, A correction technique prediction model that predicts the correction technique for Enhanced Face Images classified according to the Enhancement class, An image restoration unit that restores the original face image of the Enhanced face image using an inverse correction technique corresponding to the predicted correction technique, and Includes, The learning department is, The deepfake detection model was further trained to prevent the restored original face image from being classified as an Enhancement class. Deepfake detection models are A deepfake detection model learning system characterized by being a deep learning network model that classifies input images into Real, Fake, and Enhancement classes.

Images