Deep fake facial image defense method, device, and storage medium

Abstract

The present invention discloses a method, device, and storage medium for defending against deepfake facial images, including: step S1, generating a watermarked image using a watermark network trained jointly by an encoder and decoder; and step S2, training a deepfake network using the watermarked image and the original image to implement conditional control over deepfake generation. The technical solution of the present invention addresses the problem that existing deepfake defenses cannot simultaneously provide timely control and ensure that deepfakes pass through watermarking schemes.

Description

Technical Field

[0001] The present invention belongs to the field of image information hiding technology, and in particular relates to a method and device for defending against deep fake facial images, and a storage medium. Background Art

[0002] Current passive defense solutions for detecting deepfake facial images cannot achieve real-time control of deepfakes and, in most cases, can only be traced after the fact. In terms of active defense, many solutions counter deepfake attacks by designing watermarks with adversarial, robust, or fragile properties. Specifically, adversarial properties prevent deepfake models from generating normal images; robust properties ensure that the watermark can still be used to trace the source after a deepfake occurs; and fragility, the opposite of robustness, causes the watermark to be destroyed when the image is interfered with by a deepfake, thereby enabling rapid detection of deepfakes.

[0003] A key issue with these active defenses is that even if a watermark or other perturbation method is used to render a deepfake ineffective, an attacker could still obtain the same original facial image, which lacks the watermark, through other means. This allows the attacker to recreate a deepfake using the unwatermarked image, rendering the active defense method ineffective. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for defending against deep fake facial images, and a storage medium, so as to solve the problem that existing deep fake defense cannot simultaneously meet the requirements of timely control and guarantee that deep fakes pass the watermark scheme. The method conditions the generation of deep fakes in the form of watermarks to ensure that the generated deep fake images must carry the watermark generated by the scheme, thereby ensuring that the attacker's behavior is under the supervision of the watermark. At the same time, the source can be traced back based on the watermark, thereby meeting the timeliness and robustness of deep fake defense.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for defending against deep fake facial images, comprising:

[0007] Step S1: Generate a watermark image using the watermark network trained jointly by the encoder and decoder;

[0008] Step S2: Use the watermarked image and the original image to train the deep fake network to achieve conditional control of deep fake generation.

[0009] Preferably, the encoder and decoder adopt the U-net structure, and the attention mechanism adopts ECAblock.

[0010] As a preference, in step S1, watermark information is selected as a designated watermark, and the designated watermark is embedded into the host image I g In the watermark image I wg ; Randomly select watermark information to embed into host image I g As the watermark image I′ wg , according to the watermark image I wg and watermark image I′ wg The watermark network is obtained by joint training of the encoder and decoder. The loss function of the watermark network is as follows:

[0011] L st1 =λ enc L enc +λ dec L dec +λ adv L adv

[0012] Among them, L enc , L dec , L adv are the losses of encoder, decoder and discriminator respectively, λ enc ,λ dec ,λ adv are the hyperparameters of the encoder, decoder and discriminator respectively.

[0013] Preferably, in step S1, the encoder parameters are modified, and the information related to the modified parameter position is set as the key.

[0014] Preferably, in step S2, the loss function of the deep fake network is as follows:

[0015] L st2 =λ cf L cf +λ rf L rf +λ af L af

[0016] Among them, L cf ,L rf ,L af They are classification loss, reconstruction loss and adversarial loss, λ cf ,λ rf ,λ af are the hyperparameters of classification loss, reconstruction loss and adversarial loss respectively; classification loss L cf Contains: loss L from original image classification to original attributes cfr , the loss L of classifying the generated image to the correct generated attribute cf1 and the loss L for classifying the generated image to the incorrectly generated attribute cf2 .

[0017] Preferably, the watermark network trained by joint encoder and decoder is used to train the decoder separately to improve the robustness of the watermark.

[0018] The present invention also provides a deep fake face image defense device, comprising:

[0019] The first processing module is used to generate a watermark image using the watermark network trained jointly by the encoder and the decoder;

[0020] The second processing module is used to train the deep fake network using the watermark image and the original image to achieve conditional control of deep fake generation.

[0021] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes a deep fake facial image defense method when running.

[0022] The present invention adopts a robust watermarking scheme combined with a key, which significantly improves the privacy of the watermark and the security of the overall scheme by precisely controlling the access to the watermark; the watermark embedding network is controlled by the key, and the generation and extraction effects of the watermark directly depend on the correctness of the key; when the key errors increase, the quality of the generated watermark image and the accuracy of the watermark extraction will significantly decrease, thereby effectively protecting the security and exclusivity of the watermark.

[0023] The present invention adopts a watermark-based deep fake generation control and defense strategy. The deep fake generation process is limited to facial images containing the specified watermark, and correct output cannot be generated for other images. This mechanism ensures that all valid output images generated by deep fakes carry the watermark generated by the scheme, and the watermark can be successfully extracted, thereby achieving effective control of the generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0025] Figure 1 This is a flow chart of the method for defending against deep fake facial images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1:

[0029] like Figure 1 As shown, an embodiment of the present invention provides a method for defending against deep fake facial images, comprising:

[0030] Step S1: Generate a watermark image using the watermark network trained jointly by the encoder and decoder;

[0031] Step S2: Use the watermarked image and the original image to train the deep fake network to achieve conditional control of deep fake generation.

[0032] As an implementation method of an embodiment of the present invention, in step S1, the encoder and decoder are jointly trained to obtain a watermark that is beneficial to the image quality. Both the encoder and the decoder are U-net structures, and the attention mechanism part adopts ECAblock. The training is specifically to divide the original data set into a training set, a test set, and an evaluation set, and then input the data and the watermark on the training set into the encoder together. The encoder inputs the generated watermark image into the discriminator to obtain the probability to complete the confrontation, and then inputs the encoded image into the decoder to obtain the watermark information, and compares the watermark information with the watermark initially inserted to complete the robustness test of the watermark, and then conducts further tests on the evaluation set to achieve good control of the training effect. The present invention selects the watermark information as the designated watermark and embeds the watermark into the original image I g In the watermark image I wg , then randomly select watermark information to embed into the host image I g As the watermark image I′ wg , according to the watermark image I wg and watermark image I′ wg The watermark network is obtained by joint training of the encoder and decoder. The loss function of the watermark network is as follows:

[0033] L st1 =λ enc L enc +λ dec L dec +λ advL adv

[0034] Among them, L enc , L dec , L adv are the losses of encoder, decoder and discriminator respectively, λ enc ,λ dec ,λ adv These are the hyperparameters of the encoder, decoder, and discriminator. The encoder loss can make the watermark image and the original image as consistent as possible visually, the decoder loss can achieve effective watermark extraction, and the discriminator loss can make the discriminator and encoder compete with each other, making the watermark image quality better and further improving the performance of the entire watermarking network.

[0035] Modify the encoder parameters and set the modified parameter position related information as the key. Specifically, use all the parameter bits on the encoder as the key length, randomly add 10 or -10 to each parameter position, and then save the watermark network. When using the watermark network, just subtract 10 or -10 from the original position.

[0036] As an implementation method of the embodiment of the present invention, in step S2, the strategy of alternating training of the generator and the discriminator is adopted, the image and the desired target classification vector are input to the generator, and the generated image output by the generator is input to the discriminator for classification and authenticity detection to complete the adversarial goal. wg and I g , training will guide its classification and misclassification close in classification loss, while I wg By guiding its normal classification in the classification loss, the generation and classification control of the deep fake network can be achieved. The loss function of the deep fake network is as follows:

[0037] L st2 =λ cf L cf +λ rf L rf +λ af L af

[0038] Among them, L cf ,L rf ,L af They are classification loss, reconstruction loss and adversarial loss, λ cf ,λ rf ,λ afThey are the hyperparameters of classification loss, reconstruction loss and adversarial loss. The classification loss will try to ensure that the classification attributes of the image generated by the generator are consistent with the attributes when embedded during training. The reconstruction loss will try to ensure that the image generated by the generator according to specific attributes and then sent to the generator through the original attributes is visually consistent with the original image. The adversarial loss will make the generator and the discriminator conduct adversarial training, so as to better ensure the realism of the generated effect. Among them, the classification loss L cf Contains: loss L from original image classification to original attributes cfr , the loss L of classifying the generated image to the correct generated attribute cf1 and the loss L for classifying the generated image to the incorrectly generated attribute cf2 , the three loss functions are similar, given here:

[0039] L cfr =E[-logDi(Corg|H)]

[0040] L cf1 =F[-logDi(Ctrg|I wg )]

[0041] L cf2 =E[-logDi(Cerr|I′ wg or I g )]

[0042] Among them, Di is the discriminator, Corg is the original category of the image, H is the image input to the generator, Ctrg is the category where the image will be generated, and Cerr is the custom error classification.

[0043] This will ensure that images that have not been processed by the deep fake generator can still be classified as original attributes by the discriminator. For different watermarked images that have passed the deep fake generator, it will be conditionally determined whether the correct attributes are generated, thereby completing the generation control of deep fakes and further realizing timely active defense against deep fakes.

[0044] As an implementation method of an embodiment of the present invention, in step S2, after obtaining the deep fake generated image under the result generation control, the watermark will be trained separately by training the decoder in step S1 to improve the robustness of the watermark to common noise under the premise of deep fake. Specifically, the encoder and discriminator are frozen, and the decoder loss is used as the only loss to optimize the entire network. Only the decoder is trained in each round of training. The watermarked image and the original image before and after deep fake will be directly fed into the decoder to obtain the extracted watermark information for training. The loss function is as follows:

[0045] L st3 =L D

[0046] Among them, LD is the decoder loss.

[0047] The watermark network trained by joint encoder-decoder is used to train the decoder separately. The loss function is changed to further improve the robustness of the watermark under deep fakes and other common types of noise interference, so as to achieve good post-extraction effect.

[0048] The present invention achieves overall protection of the watermark framework by introducing key technology. The watermark will produce errors as the key is wrong, and thus cannot be successfully extracted as the correct watermark when embedded. The present invention introduces a watermark-based deep fake control technology to ensure that deep fakes can only generate images generated with a key watermark scheme, thereby ensuring that attackers have no other channels to use deep fakes to achieve malicious goals again. The present invention further strengthens the robustness of the watermark to ensure that the watermark can still maintain a high robustness after deep fakes and other common noise interference, thereby realizing the tracing of the watermark.

[0049] Example 2:

[0050] An embodiment of the present invention further provides a deep fake facial image defense device, comprising:

[0051] The first processing module is used to generate a watermark image using the watermark network trained jointly by the encoder and the decoder;

[0052] The second processing module is used to train the deep fake network using the watermark image and the original image to achieve conditional control of deep fake generation.

[0053] Example 3:

[0054] An embodiment of the present invention also provides a storage medium having a computer program stored thereon, which executes a method for defending against deep fake facial images when running.

[0055] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for defending against deep fake facial images, characterized in that: include: Step S1: Generate a watermark image using the watermark network trained jointly by the encoder and decoder; Step S2: Use the watermarked image and the original image to train a deepfake network to achieve conditional control over deepfake generation; The encoder and decoder use U-net structure, and the attention mechanism uses ECAblock; In step S1, watermark information is selected as a designated watermark, and the designated watermark is embedded into the host image I g In the watermark image I wg ; Randomly select watermark information to embed into host image I g As the watermark image I′ wg , according to the watermark image I wg and watermark image I′ wg The watermark network is obtained by joint training of the encoder and decoder. The loss function of the watermark network is as follows: L st1 =λ enc L enc +λ dec L dec +λ adv L adv Among them, L enc , L dec , L adv are the losses of encoder, decoder and discriminator respectively, λ enc ,λ dec ,λ adv are the hyperparameters of the encoder, decoder, and discriminator, respectively; In step S1, the encoder parameters are modified, and the information related to the modified parameter position is set as the key; In step S2, the loss function of the deep fake network is as follows: L st2 =λ cf L cf +λ rf L rf +λ af L af Among them, L cf ,L rf ,L af They are classification loss, reconstruction loss and adversarial loss, λ cf ,λ rf ,λ af are the hyperparameters of classification loss, reconstruction loss and adversarial loss respectively; classification loss L cf Contains: loss L from original image classification to original attributes cfr , the loss L of classifying the generated image to the correct generated attribute cf1 and the loss L for classifying the generated image to the incorrectly generated attribute cf2 ; The watermark network adopts joint encoder-decoder training and the decoder is trained separately to improve the robustness of the watermark.

2. A deep fake face image defense device that implements the deep fake face image defense method according to claim 1, characterized in that: include: The first processing module is used to generate a watermark image using the watermark network trained jointly by the encoder and the decoder; The second processing module is used to train the deep fake network using the watermark image and the original image to achieve conditional control of deep fake generation.

3. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the deep fake facial image defense method according to claim 1.

Images