A deepfake face-swapping detection method based on robust identity-aware watermarking

Abstract

A deep forgery face-swapping detection method based on robust identity-aware watermarks, which relates to the field of image forgery detection. Identity-aware watermarks are produced according to the identity semantics of image content, and an unpredictable and irreversible chaotic encryption system is designed to ensure the confidentiality and security of the watermark. Watermark embedding and watermark extraction are achieved while ensuring image quality by jointly training an autoencoder framework and common image processing operations / malicious face-swapping operations. For tampered images, detection and traceability are completed by restoring the embedded watermark and comparing it with the image identity content.

Description

Technical Field

[0001] The present invention relates to the field of image forgery detection, and in particular to a deep forgery face-changing detection method based on robust identity-aware watermark. Background Art

[0002] Deepfake is a technology that tampers with and generates videos and images based on deep neural networks, which can change or fabricate the reality of people, events, and objects. This technology is a type of artificial intelligence that can create fake images, videos, and audio that look and sound realistic. In the era of big data, deepfake technology has become increasingly easy to use and the generated effects are increasingly difficult to distinguish between real and fake. Its standardized use can bring positive impacts in many fields, such as entertainment, education, research, and art. However, malicious deepfakes can also bring serious risks to individuals and society, such as spreading misinformation, violating privacy, damaging reputations, and influencing public opinion.

[0003] With the rapid development of deep generative models, malicious deep fakes have raised serious privacy issues. Since it is difficult to detect obvious artifacts in high-quality synthetic images that are difficult to distinguish between real and fake, passive detection models for face identity tampering in recent years usually suffer from performance degradation due to the low generality of the algorithm. Therefore, some studies have tried to actively protect the original image from malicious tampering and forgery by embedding invisible signals into it. However, existing active defense methods still have much room for improvement in visual quality, detection accuracy, and traceability. Summary of the invention

[0004] In order to overcome the shortcomings of the above technologies, the present invention provides a deep fake face swap detection method based on robust identity-aware watermarking, which can simultaneously actively detect and trace deep fake face swapping.

[0005] The technical solution adopted by the present invention to overcome the technical problems is:

[0006] A deep fake face-changing detection method based on robust identity-aware watermarking comprises the following steps:

[0007] a) Obtain n face images and obtain a face image set I, where I = {I1, I2, …, I i ,…,I n}, where I i is the i-th face image, i∈{1,…,n};

[0008] b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {Iraw,1 ,I raw,2 ,…,I raw,i ,…,I raw,n};

[0009] c) Establish watermark generator G m , the preprocessed face image I raw,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m i ;

[0010] d) Establish an encoder Encoder to convert the preprocessed face image I raw,i and encrypted identity-aware watermark m i Input to the encoder Encoder, and output the watermark image I rec,i ;

[0011] e) The watermark image I rec,i Injection noise pool P common In the above example, we get the noise image I common,i , the watermark image I rec,i Inject face swap pool P swap In the tampered image I swap,i ;

[0012] f) Establish a decoder Decoder, which converts the noise image I common,i Or the tampered image I swap,i Input into the decoder Decoder, and output the final watermark m rec,i ;

[0013] g) According to the final watermark m rec,i Judging the noise image I common,i And the tampered image I swap,i is a real image or a fake image.

[0014] Furthermore, in step a), n face images are obtained from the CelebA-HQ dataset.

[0015] Further, step b) comprises the following steps:

[0016] b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 128×128 I i ′;

[0017] b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image Ii ″;

[0018] b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image I raw,i .

[0019] Further, step c) comprises the following steps:

[0020] c-1) Watermark generator G m It is composed of collision-free units and one-way calculation units;

[0021] c-2) The pre-processed face image I raw,i Input to the watermark generator G m In the collision-free unit, the face recognition algorithm ArcFace is used to identify the preprocessed face image I raw,i Get the dimension d E The identity embedding vector e i ;

[0022] c-3) Use principal component analysis (PCA) to reduce the dimension of the identity embedding vector e i From dimension d E Project to 64 and get the projected embedding vector E pca,i ;

[0023] c-4) Corresponding projection embedding vector E pca,i Perform normalization operation to obtain a vector E between [0,1] scale,i ;

[0024] c-5) Set the threshold c, c = 0.5, if E scale,i ≥c, then the vector E scale,i The value of is set to 1 if E scale,i <c, then vector E scale,i The value of is set to 0, and the identity-aware watermark E is obtained after reset. binary,i ;

[0025] c-6) Through the formula Calculate the value k of the i-th position of the binary encryption key k i , k={k1,k2,…,k i ,…,k l}, where x i is the i-th value in the chaotic map, x i =rx i-1 (1-x i-1 ), r = 3.93, x i-1is the i-1th value in the chaotic map, i∈{1,…,l}, l is the watermark length, l=64, x0=0.1, p=5, q=11, mod2 is x i p q Modulo operation on 2;

[0026] c-7) Add the identity-aware watermark E binary,i Input to the watermark generator G m In the one-way computational unit, the identity-aware watermark E binary,i Perform logical XOR operation with the binary encryption key k to obtain the binary encrypted identity-aware watermark m i .

[0027] Further, step d) comprises the following steps:

[0028] d-1) Encoder consists of a first branch, a second branch, a ConvBlock block, and a convolutional layer;

[0029] d-2) The first branch of the encoder consists of ConvBlock, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. raw,i Input into the ConvBlock block of the first branch, and output the image feature F clean_1 , the SEBlock block consists of a first branch and a second branch. The first branch of the SEBlock block consists of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the SEBlock block consists of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0030] d-3) The image feature F clean_1 Input into the first branch of the first SEBlock block, and output the image feature F clean_2 , the image feature F clean_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_1 , the image feature F clean_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output.clean_3_2 , the image feature F clean_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_3 , the image feature F clean_3_3 With image feature F clean_2 Perform matrix multiplication operation to obtain image feature F clean_4 , the image feature F clean_4 With image feature F clean_1 Perform matrix addition operation to obtain image feature F clean_5 ;

[0031] d-4) The image feature F clean_5 Replace the image feature F in step d-3) clean_1 Repeat step d-3) to obtain the image feature F output by the second SEBlock block clean_6 ;

[0032] d-5) Repeat step d-4) to obtain the Nth enc The image features F output by SEBlock clean_7 ;

[0033] d-6) The second branch of the encoder consists of a linear layer, a ConvBlock block, N enc DiffBlock blocks, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. The encrypted identity-aware watermark m i Input into the linear layer of the second branch, output the watermark feature f1, input the watermark feature f1 into the ConvBlock block, output the watermark feature f2, the DiffBlock block is composed of a transposed convolution layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence, the SEBlock block is composed of the first branch and the second branch, the first branch of the SEBlock block is composed of an average pooling layer, a first convolution layer, a LeakyReLU activation function layer, and a second convolution layer in sequence, the second branch of the SEBlock block is composed of a first convolution layer, a first batch normalization layer, a first LeakyReLU activation function layer, a second convolution layer, a second batch normalization layer, a second LeakyReLU activation function layer, a third convolution layer, a third batch normalization layer, and a third LeakyReLU activation function layer in sequence;

[0034] d-7) inputting the watermark feature f2 into the first DiffBlock block of the second branch of the encoder Encoder, and outputting the watermark feature f3;

[0035] d-8) replacing the image feature f2 in step d-7) with the image feature f3 and repeating step d-7) to obtain the watermark feature f4 output by the second DiffBlock block;

[0036] d-9) Repeat step d-8) to obtain the Nth enc The watermark feature f5 output by the DiffBlock block;

[0037] d-10) Input the watermark feature f5 into the first branch of the first SEBlock block, and output the watermark feature f 5_2 , the watermark feature f 5_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_1 , the watermark feature f 5_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature F is output. 5_3_2 , the watermark feature F 5_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_3 , the watermark feature f 5_3_3 With the watermark feature f 5_2 Perform matrix multiplication operation to obtain the watermark feature f 5_4 , the watermark feature f 5_4 Perform matrix addition operation with watermark feature f5 to obtain watermark feature f 5_5 ;

[0038] d-11) The watermark feature f 5_5 Replace the watermark feature f5 in step d-10) and repeat step d-10) to obtain the watermark feature f output by the second SEBlock block. 5_6 ;

[0039] d-12) Repeat step d-11) to obtain the Nth enc The watermark feature f6 output by the SEBlock block;

[0040] d-13) The image feature F clean_7The watermark feature f6 is concatenated with the torch.cat() function in the torch library in Python to obtain feature C1. The ConvBlock block of the encoder is composed of a convolution layer, a batch normalization layer, and a LeakyReLU activation function layer. Feature C1 is input into the ConvBlock block of the encoder and output to obtain feature C2. Feature C2 is combined with the preprocessed face image I raw,i The concatenation operation is performed through the torch.cat() function in the torch library in Python to obtain feature C3;

[0041] d-14) Input feature C3 into the convolutional layer of the encoder and output the watermark image I rec,i ;

[0042] d-15) Through the formula L enc =||I rec,i -I raw,i ||2 Calculate pixel loss function L enc , where ||·||2 is the L2 norm, and the Adam optimizer is used to use the pixel loss function L enc Train the encoder. Preferably, N encThe value is 3. In step d-2), the number of channels of the convolution layer of the ConvBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the window size of the average pooling layer of the first branch of the SEBlock block of the first branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the SEBlock block of the first branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step d-6), the output node of the linear layer of the second branch of the encoder Encoder is 128; the number of channels of the convolution layer of the ConvBlock block of the second branch of the encoder Encoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; encoding The number of channels of the transposed convolution layer of the DiffBlock block of the second branch of the encoder is 64, the convolution kernel is 2, the step size is 2, and the padding is 0; the window size of the average pooling layer of the first branch of the SEBlock block of the second branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution layer of the second branch of the SEBlock block of the second branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; the number of channels of the convolution layer of the ConvBlock block of the encoder Encoder in step d-13) is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the convolution layer of the encoder Encoder in step d-14) is 64 and the convolution kernel is 1.

[0043] Further, step e) comprises the following steps:

[0044] e-1) Establish the noise pool P common , noise pool P common The noise includes Dropout noise, Reszie noise, Jpeg noise, GaussianNoise noise, SaltPepper noise, GaussianBlur noise, and MedBlur noise. rec,i Injection noise pool P common In the noise pool P common Randomly select a noise watermark image I rec,i Add noise to get the noise image I common,i ;

[0045] e-2) Establishing face exchange pool P swap , face swap pool P swap The SimSwap model, InfoSwap model, and UniFace model are included. rec,i Inject face swap pool P swap , from the face exchange pool P swap Randomly select a model to watermark image I rec,i Perform face swap to obtain the tampered image I swap,i ;

[0046] e-3) Using the Adam optimizer, the face swap pool P is optimized using the adversarial loss function. swap The model in is trained.

[0047] Further, step f) comprises the following steps:

[0048] f-1) The decoder consists of the first ConvBlock block, N se The first SEBlock block, the second ConvBlock block, N dec The second SEBlock block, the third ConvBlock block, the flattening layer, and the linear layer;

[0049] f-2) The first ConvBlock block of the decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. common,i Or the tampered image I swap,i Input to the first ConvBlock block of the decoder Decoder, and output the image feature N1;

[0050] f-3) The first SEBlock block is composed of a first branch and a second branch. The first branch of the first SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the first SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0051] f-4) Input the image feature N1 into the first branch of the first SEBlock block, and output the image feature N 1_2 , the image feature N 1_2It is sequentially input into the first convolutional layer, the first normalization layer, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_1 , the image feature N 1_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_2 , the image feature N 1_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_3 , the image feature N 1_3_3 With image feature N 1_2 Perform matrix multiplication operation to obtain image feature N 1_4 , the image feature N 1_4 Perform matrix addition operation with image feature N1 to obtain image feature N 1_5 ;

[0052] f-5) The image feature N 1_5 Replace the image feature N1 in step f-4) and repeat step f-4) to obtain the image feature N output by the second first SEBlock block. 1_6 ;

[0053] f-6) Repeat step f-5) to obtain the Nth se The image feature N2 output by the first SEBlock block;

[0054] f-7) The second ConvBlock block of the decoder Decoder is composed of a convolution layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence. The image feature N2 is input into the second ConvBlock block of the decoder Decoder, and the image feature N3 is output;

[0055] f-8) The second SEBlock block is composed of a first branch and a second branch. The first branch of the second SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the second SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0056] f-9) Input the image feature N3 into the first branch of the first second SEBlock block, and output the image feature N 3_2 , the image feature N 3_2 It is sequentially input into the first convolution layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_1 , the image feature N 3_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_2 , the image feature N 3_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_3 , the image feature N 3_3_3 With image feature N 3_2 Perform matrix multiplication operation to obtain image feature N 3_4 , the image feature N 3_4 Perform matrix addition operation with image feature N3 to obtain image feature N 3_5 ;

[0057] f-10) The image feature N 3_5 Replace the image feature N3 in step f-9) and repeat step f-9) to obtain the image feature N output by the second SEBlock block. 3_6 ;

[0058] f-11) Repeat step f-10) to obtain the Nth dec The image feature N4 output by the second SEBlock block;

[0059] f-12) The third ConvBlock block of the decoder Decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence. The image feature N4 is input into the third ConvBlock block of the decoder Decoder, and the image feature N5 is output;

[0060] f-13) Input the image feature N5 into the flattening layer of the decoder, and output the image feature N6;

[0061] f-14) Input the image feature N6 into the linear layer of the decoder Decoder, and output the final watermark m rec,i ;

[0062] f-15) through the formula L dec =||mrec,i -m i ||2 Calculate the recovery loss function L dec , where ||·||2 is the L2 norm, and the Adam optimizer is used to recover the loss function L dec Train the decoder.

[0063] Preferably, in step f-2), the number of channels of the convolution layer of the first ConvBlock block of the decoder Decoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; in step f-3), the window size of the average pooling layer of the first branch of the first SEBlock block is 1; the number of channels of the first convolution layer of the first branch of the first SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the first SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the first SEBlock block are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step f-7), the number of channels of the convolution layer of the second ConvBlock block of the decoder Decoder is 64, The convolution kernel is 3, the step size is 1, and the padding is 1; the window size of the average pooling layer of the first branch of the second SEBlock block in step f-9) is 1; the number of channels of the first convolution layer of the first branch of the second SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the second SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the second SEBlock block are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; the number of channels of the convolution layer of the third ConvBlock block of the decoder Decoder in step f-12) is 1, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of output nodes of the linear layer of the decoder Decoder in step f-14) is 64.

[0064] Further, step g) comprises the following steps:

[0065] g-1) The noise image I common,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m c,i ;

[0066] g-2) by formula Calculate the similarity value ACC c , where d ham (m rec,i ,m i ) is the binary encrypted identity-aware watermark mc,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC c When the noise image I is less than or equal to the threshold α, common,i For a fake image, when the similarity value ACC c When it is greater than the threshold α, the noise image I common,i For the true image, the threshold α is set to [0.7-0.85];

[0067] g-3) The tampered image I swap,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m s,i ;

[0068] g-4) Through the formula Calculate the similarity value ACC s , where d ham (m rec,i ,m i ) is the binary encrypted identity-aware watermark m s,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC s When the tampered image I is less than or equal to the threshold β, swap,i For a fake image, when the similarity value ACC s When it is greater than the threshold β, the tampered image I swap,i For a true image, the threshold β is set to [0.7-0.85].

[0069] The beneficial effects of the present invention are: deep fake face swapping can be actively detected and traced at the same time. An identity-aware watermark is produced according to the identity semantics of the image content, and an unpredictable and irreversible chaotic encryption system is designed to ensure the confidentiality and security of the watermark. By jointly training the autoencoder framework and common image processing operations / malicious face-swapping operations, watermark embedding and watermark extraction are achieved while ensuring image quality. For tampered images, detection and traceability are completed by restoring the embedded watermark and comparing it with the image identity content. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the method of the present invention;

[0071] Figure 2 It is a structural diagram of the encoder Encoder of the present invention;

[0072] Figure 3 It is a structural diagram of the decoder Decoder of the present invention. DETAILED DESCRIPTION

[0073] The following is combined with Figure 1 To Attachment Figure 3 The present invention is further described.

[0074] A deep fake face-changing detection method based on robust identity-aware watermarking comprises the following steps:

[0075] a) Obtain n face images and obtain a face image set I, where I = {I1, I2, …, I i ,…,I n}, where I i is the i-th face image, i∈{1,…,n}.

[0076] b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {I raw,1 ,I raw,2 ,…,I raw,i ,…,I raw,n}.

[0077] c) Establish watermark generator G m , the preprocessed face image I raw,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m i .

[0078] d) Establish an encoder Encoder to convert the preprocessed face image I raw,i and encrypted identity-aware watermark m i Input to the encoder Encoder, and output the watermark image I rec,i .

[0079] e) The watermark image I rec,i Injection noise pool P common In the above example, we get the noise image I common,i , the watermark image I rec,i Inject face swap pool P swap In the tampered image I swap,i .

[0080] f) Establish a decoder Decoder, which converts the noise image I common,i Or the tampered image I swap,i Input into the decoder Decoder, and output the final watermark m rec,i .

[0081] g) According to the final watermark m rec,i Judging the noise image Icommon,i And the tampered image I swap,i is a real image or a fake image.

[0082] The experiment uses two well-known and commonly used face image datasets, CelebA-HQ and LFW. The CelebA-HQ dataset is officially divided into training set, validation set, and test set, while the LFW dataset is only used for testing. The CelebA-HQ dataset consists of 30,000 face images, containing a total of 6,217 different face identities. The resolution of each image is 1024×1024. The 30,000 face images are divided into training set, validation set, and test set according to the official division scheme provided in the paper "Deep Learning Face Attributes in the Wild". The images in the LFW dataset contain 5,749 different face identities. The resolution of each image is 250×250. The images corresponding to the 5,749 face identities in the LFW dataset are used for testing. The entire training process is to first use the common image processing noise pool P common The watermark embedding and extraction process are pre-trained, and then the face swap pool P is used swap SimSwap in the pre-training is fine-tuned, while InfoSwap and Uniface are tested. The purpose of pre-training is to ensure that it remains robust to common general image processing methods, and then fine-tuning on SimSwap is to make it continue to adapt to face swapping operations.

[0083] In order to verify the reliability of this patented method, we conducted a comparative experiment and displayed the experimental results.

[0084] Since no existing work uses a single watermark to simultaneously detect and track the source of face swapped images, we selected the robust watermark frameworks HiDDeN, MBRS, CIN and RDA, the active semi-fragile watermark framework FaceSigns, and the active Deepfake perturbation frameworks AntiForgery and CMUA for comparative experiments. For fairness, we directly use the publicly available model weights corresponding to the best performance of the above algorithms. The bold data in the following table are the best results.

[0085] Table 1 Quantitative visual quality evaluation of watermarked images on the celebA-HQ dataset

[0086]

[0087] Table 2 Quantitative comparison of the bitwise recovery accuracy of watermarks after common image processing operations and malicious face-changing operations on the CelebA-HQ dataset at 128×128 resolution

[0088]

[0089] Table 3 Quantitative comparison of the bitwise recovery accuracy of watermarks after common image processing operations and malicious face-changing operations on the CelebA-HQ dataset at 256×256 resolution

[0090]

[0091] Table 4 Quantitative visual quality evaluation of watermarked images on the LFW dataset

[0092] Metrics MBRS FaceSigns Ours PSNR↑ 45.37 38.23 45.42 SSIM↑ 0.969 0.910 0.994

[0093] Table 5 Quantitative comparison of the bit-wise recovery accuracy of watermarks after common image processing operations and malicious face-changing operations on the LFW dataset at 256×256 resolution

[0094]

[0095] Table 1 shows the quantitative visual quality evaluation of watermarked images on the CelebA-HQ dataset. The average peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) between the original and watermarked images are used to evaluate the visual quality of the watermarked images. As shown in Table 1, our proposed method achieves the best visual quality in terms of PSNR and SSIM at two resolution levels. At 128×128 resolution, our PSNR and SSIM are improved by 14.13 and 0.105 respectively compared with HiDDeN, 14.38 and 0.218 respectively compared with MBRS, 4.02 and 0.026 respectively compared with CIN, and 3.46 and 0.018 respectively compared with RDA. At a resolution of 256×256, our PSNR and SSIM are improved by 9.76 and 0.041 respectively compared to AntiForgery, 6.74 and 0.137 respectively compared to CMUA, 1.24 and 0.025 respectively compared to MBRS, and 8.39 and 0.105 respectively compared to Facesigns. It can be seen that the performance of the AntiForgery and CMUA models is relatively poor, because noise and perturbations are directly added to the original image, while the watermark is losslessly embedded by image reconstruction.

[0096] The quantitative comparison results of the bitwise watermark recovery accuracy after common image processing operations and malicious face-swapping operations on the CelebA-HQ dataset at 128×128 resolution are shown in Table 2. The watermark recovery accuracy is used to measure the robustness of the watermark. Specifically, for common image processing operations, our method outperforms HiDDeN, CIN and RDA, and our average accuracy is improved by 28.85% compared to HiDDeN, 2.95% compared to CIN, and 15.05% compared to RDA. As can be seen from Table 1, MBRS has serious visual quality issues with a PSNR of 33.01 and an SSIM of 0.775, but achieves the best bitwise watermark recovery accuracy in Table 2. The quantitative comparison results of the bitwise watermark recovery accuracy after common image processing operations and malicious face-swapping operations on the CelebA-HQ dataset at 256×256 resolution are shown in Table 3. We achieve an average precision of 99.07% on common image processing operations, outperforming the existing state-of-the-art methods, and our average precision is improved by 14.41% over MBRS and 18.68% over FaceSigns. In contrast, although MBRS achieves reasonable visual quality at 256×256 resolution in Table 1, it suffers from a large degradation in watermark recovery accuracy on common operations compared to the 128×128 resolution version.

[0097] In addition, the generalization ability of different face-swapping algorithms is also evaluated. For malicious face-swapping operations, the face swap pool P swap Only SimSwap is included in the training phase, and SimSwap, InfoSwap, and UniFace are used for evaluation in the test phase. As shown in Table 2, our model recovers the watermark with an average accuracy of 99.62% at a resolution of 128×128, which is 47.93% higher than HiDDeN, 49.28% higher than MBRS, 50.66% higher than CIN, and 42.57% higher than RDA. As shown in Table 3, the watermark is recovered with an average accuracy of 98.19% at a resolution of 256×256, which is 47.96% higher than MBRS and 48.08% higher than FaceSigns.

[0098] It can be seen that our proposed model is the only one that can maintain the bitwise watermark recovery accuracy above 95% for all common image processing operations, and above 90% for all face swapping operations, while other methods are severely affected by at least one of the operations.

[0099] The generalization ability of our watermarking framework is further evaluated on a different dataset LFW. Since the image resolution in LFW is 250×250, we only use 256×256 resolution for experiments. As shown in Tables 4 and 5, the performance evaluation of visual quality and watermark recovery accuracy are listed respectively. As shown in Table 4, MBRS and our method have good visual quality PSNR of 45.37 and 45.42 respectively, while FaceSigns is 38.23. As shown in Table 5, for common image processing operations, the average accuracy of our method is 98.47%, which is 14.41% higher than MBRS and 18.78% higher than FaceSigns. For malicious face-changing operations, the average accuracy of our method is 98.56%, which is 48.41% higher than MBRS and 47.92% higher than FaceSigns.

[0100] In addition, it can be seen from Tables 2 and 5 that our method has a significantly lower accuracy than UniFace at 256×256 resolution, which is 94.86% and 97.06% on CelebA-HQ and LFW, respectively. This is mainly due to the fact that UniFace only accepts half-precision floating point inputs.

[0101] In one embodiment of the present invention, in step a), n face images are obtained from the CelebA-HQ dataset. In one embodiment of the present invention, step b) includes the following steps:

[0102] b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 128×128 I i ′.

[0103] b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image I i ″.

[0104] b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image I raw,i .

[0105] In one embodiment of the present invention, step c) comprises the following steps:

[0106] c-1) Watermark generator G m It consists of collision-free units and one-way calculation units.

[0107] c-2) The pre-processed face image I raw,i Input to the watermark generator G m In the collision-free unit, the face recognition algorithm ArcFace is used to identify the preprocessed face image I raw,i Get the dimension d E The identity embedding vector e i .

[0108] c-3) Use principal component analysis (PCA) to reduce the dimension of the identity embedding vector e i From dimension d E Project to 64 and get the projected embedding vector E pca,i .

[0109] c-4) Corresponding projection embedding vector E pca,i Perform normalization operation to obtain a vector E between [0,1] scale,i .

[0110] c-5) Set the threshold c, c = 0.5, if E scale,i ≥c, then the vector E scale,i The value of is set to 1 if E scale,i <c, then vector E scale,i The value of is set to 0, and the identity-aware watermark E is obtained after reset. binary,i All dimensionality reduction operations in this process retain the original identity vector to the greatest extent.

[0111] c-6) Since the authenticity of the face image is completely based on the matching degree of the watermark, malicious people will try to replace the binary encrypted identity-aware watermark m with the watermark that matches the identity content of the face in the image. i In order to prevent malicious attacks, the watermark has one-way calculation, so the formula Calculate the value k of the i-th position of the binary encryption key k i , k={k1,k2,…,k i ,…,k l}, where x i is the i-th value in the chaotic map, x i =rx i-1 (1-x i-1 ), r = 3.93, x i-1 is the i-1th value in the chaotic map, i∈{1,…,l}, l is the watermark length, l=64, x0=0.1, p=5, q=11, mod2 is x i p q Modulo 2 operation.

[0112] c-7) Add the identity-aware watermark E binary,iInput to the watermark generator G m In the one-way computational unit, the identity-aware watermark E binary,i Perform logical XOR operation with the binary encryption key k to obtain the binary encrypted identity-aware watermark m i .

[0113] In one embodiment of the present invention, step d) comprises the following steps:

[0114] d-1) Encoder Encoder consists of the first branch, the second branch, the ConvBlock block, and the convolutional layer.

[0115] d-2) The first branch of the encoder consists of ConvBlock, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. raw,i Input into the ConvBlock block of the first branch, and output the image feature F clean_1 The SEBlock block consists of a first branch and a second branch. The first branch of the SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0116] d-3) The image feature F clean_1 Input into the first branch of the first SEBlock block, and output the image feature F clean_2 , the image feature F clean_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_1 , the image feature F clean_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_2 , the image feature F clean_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_3 , the image feature F clean_3_3 With image feature Fclean_2 Perform matrix multiplication operation to obtain image feature F clean_4 , the image feature F clean_4 With image feature F clean_1 Perform matrix addition operation to obtain image feature F clean_5 .

[0117] d-4) The image feature F clean_5 Replace the image feature F in step d-3) clean_1 Repeat step d-3) to obtain the image feature F output by the second SEBlock block clean_6 .

[0118] d-5) Repeat step d-4) to obtain the Nth enc The image features F output by SEBlock clean_7 .

[0119] d-6) The second branch of the encoder consists of a linear layer, a ConvBlock block, N enc DiffBlock blocks, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. The encrypted identity-aware watermark m i Input into the linear layer of the second branch, output the watermark feature f1, input the watermark feature f1 into the ConvBlock block, output the watermark feature f2, the DiffBlock block is composed of a transposed convolution layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence, the SEBlock block is composed of the first branch and the second branch, the first branch of the SEBlock block is composed of an average pooling layer, a first convolution layer, a LeakyReLU activation function layer, and a second convolution layer in sequence, the second branch of the SEBlock block is composed of the first convolution layer, the first batch normalization layer, the first LeakyReLU activation function layer, the second convolution layer, the second batch normalization layer, the second LeakyReLU activation function layer, the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer in sequence.

[0120] d-7) Input the watermark feature f2 into the first DiffBlock block of the second branch of the encoder Encoder, and output the watermark feature f3.

[0121] d-8) replacing the image feature f2 in step d-7) with the image feature f3 and repeating step d-7) to obtain the watermark feature f4 output by the second DiffBlock block.

[0122] d-9) Repeat step d-8) to obtain the Nth encThe watermark feature f5 output by the DiffBlock block.

[0123] d-10) Input the watermark feature f5 into the first branch of the first SEBlock block, and output the watermark feature f 5_2 , the watermark feature f 5_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_1 , the watermark feature f 5_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature F is output. 5_3_2 , the watermark feature F 5_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_3 , the watermark feature f 5_3_3 With the watermark feature f 5_2 Perform matrix multiplication operation to obtain the watermark feature f 5_4 , the watermark feature f 5_4 Perform matrix addition operation with watermark feature f5 to obtain watermark feature f 5_5 .

[0124] d-11) The watermark feature f 5_5 Replace the watermark feature f5 in step d-10) and repeat step d-10) to obtain the watermark feature f output by the second SEBlock block. 5_6 .

[0125] d-12) Repeat step d-11) to obtain the Nth enc The watermark feature f6 output by the SEBlock block.

[0126] d-13) The image feature F clean_7 The watermark feature f6 is concatenated with the torch.cat() function in the torch library in Python to obtain feature C1. The ConvBlock block of the encoder is composed of a convolution layer, a batch normalization layer, and a LeakyReLU activation function layer. Feature C1 is input into the ConvBlock block of the encoder and output to obtain feature C2. Feature C2 is combined with the preprocessed face image I raw,i The concatenation operation is performed through the torch.cat() function in the torch library in Python to obtain feature C3.

[0127] d-14) Input feature C3 into the convolutional layer of the encoder and output the watermark image I rec,i .

[0128] d-15) Through the formula L enc =||I rec,i -I raw,i ||2 Calculate pixel loss function L enc , where ||·||2 is the L2 norm, and the Adam optimizer is used to use the pixel loss function L enc Train the encoder Encoder. In this embodiment, N enc The value is 3. In step d-2), the number of channels of the convolution layer of the ConvBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the window size of the average pooling layer of the first branch of the SEBlock block of the first branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the SEBlock block of the first branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step d-6), the output node of the linear layer of the second branch of the encoder Encoder is 128; the number of channels of the convolution layer of the ConvBlock block of the second branch of the encoder Encoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; encoding The number of channels of the transposed convolution layer of the DiffBlock block of the second branch of the encoder is 64, the convolution kernel is 2, the step size is 2, and the padding is 0; the window size of the average pooling layer of the first branch of the SEBlock block of the second branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution layer of the second branch of the SEBlock block of the second branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; the number of channels of the convolution layer of the ConvBlock block of the encoder Encoder in step d-13) is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the convolution layer of the encoder Encoder in step d-14) is 64 and the convolution kernel is 1.

[0129] In one embodiment of the present invention, step e) comprises the following steps:

[0130] e-1) Establish the noise pool Pcommon , noise pool P common The noise includes Dropout noise, Reszie noise, Jpeg noise, GaussianNoise noise, SaltPepper noise, GaussianBlur noise, and MedBlur noise. rec,i Injection noise pool P common In the noise pool P common Randomly select a noise watermark image I rec,i Add noise to get the noise image I common,i Dropout noise, Reszie noise, GaussianNoise noise, and SaltPepper noise are implemented using the corresponding packages and functions in the torch library. GaussianBlur noise is implemented using the Gaussian2d() function, and MedBlur noise is implemented using the MedianBlur() function. JPEG noise is implemented using yuv_dct(), std_quantization(), std_reverse_quantization(), and idct_rgb() in turn according to the implementation method of the paper MBRS: EnhancingRobustness of DNN-based Watermarking by Mini-Batch of Real and Simulated JPEGCompression.

[0131] e-2) Establishing face exchange pool P swap , face swap pool P swap The SimSwap model, InfoSwap model, and UniFace model are included. rec,i Inject face swap pool P swap , from the face exchange pool P swap Randomly select a model to watermark image I rec,i Perform face swap to obtain the tampered image I swap,iThe SimSwap model uses the source code of the paper "SimSwap: An Efficient Framework For High Fidelity Face Swapping" to implement face swapping, the InfoSwap model uses the source code of the paper "InfoSwap: Information Bottleneck Disentanglement for Identity Swapping" to implement face swapping, and the UniFace model uses the source code of the paper "Designing One Unified Framework for High-Fidelity Face Reenactment and Swapping" to implement face swapping.

[0132] e-3) Using the Adam optimizer, the face swap pool P is optimized using the adversarial loss function. swap The model in is trained.

[0133] In one embodiment of the present invention, step f) comprises the following steps:

[0134] f-1) The decoder consists of the first ConvBlock block, N se The first SEBlock block, the second ConvBlock block, N dec It consists of a second SEBlock block, a third ConvBlock block, a flattening layer, and a linear layer.

[0135] f-2) The first ConvBlock block of the decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. common,i Or the tampered image I swap,i Input into the first ConvBlock block of the decoder Decoder, and output the image feature N1.

[0136] f-3) The first SEBlock block is composed of a first branch and a second branch. The first branch of the first SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the first SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0137] f-4) Input the image feature N1 into the first branch of the first SEBlock block, and output the image feature N1_2 , the image feature N 1_2 It is sequentially input into the first convolutional layer, the first normalization layer, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_1 , the image feature N 1_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_2 , the image feature N 1_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_3 , the image feature N 1_3_3 With image feature N 1_2 Perform matrix multiplication operation to obtain image feature N 1_4 , the image feature N 1_4 Perform matrix addition operation with image feature N1 to obtain image feature N 1_5 .

[0138] f-5) The image feature N 1_5 Replace the image feature N1 in step f-4) and repeat step f-4) to obtain the image feature N output by the second first SEBlock block. 1_6 .

[0139] f-6) Repeat step f-5) to obtain the Nth se The image feature N2 output by the first SEBlock block.

[0140] f-7) The second ConvBlock block of the decoder Decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence. The image feature N2 is input into the second ConvBlock block of the decoder Decoder, and the image feature N3 is output.

[0141] f-8) The second SEBlock block is composed of a first branch and a second branch. The first branch of the second SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the second SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence.

[0142] f-9) Input the image feature N3 into the first branch of the first second SEBlock block, and output the image feature N 3_2 , the image feature N 3_2 It is sequentially input into the first convolution layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_1 , the image feature N 3_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_2 , the image feature N 3_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_3 , the image feature N 3_3_3 With image feature N 3_2 Perform matrix multiplication operation to obtain image feature N 3_4 , the image feature N 3_4 Perform matrix addition operation with image feature N3 to obtain image feature N 3_5 .

[0143] f-10) The image feature N 3_5 Replace the image feature N3 in step f-9) and repeat step f-9) to obtain the image feature N output by the second SEBlock block. 3_6 .

[0144] f-11) Repeat step f-10) to obtain the Nth dec The image feature N4 output by the second SEBlock block.

[0145] f-12) The third ConvBlock block of the decoder Decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence. The image feature N4 is input into the third ConvBlock block of the decoder Decoder, and the image feature N5 is output.

[0146] f-13) Input the image feature N5 into the flattening layer of the decoder Decoder and output the image feature N6.

[0147] f-14) Input the image feature N6 into the linear layer of the decoder Decoder, and output the final watermark m rec,i .

[0148] f-15) through the formula L dec =||mrec,i -m i ||2 Calculate the recovery loss function L dec , where ||·||2 is the L2 norm, and the Adam optimizer is used to recover the loss function L dec Train the decoder.

[0149] In this embodiment, in step f-2), the number of channels of the convolution layer of the first ConvBlock block of the decoder Decoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; in step f-3), the window size of the average pooling layer of the first branch of the first SEBlock block is 1; the number of channels of the first convolution layer of the first branch of the first SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the first SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the first SEBlock block are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step f-7), the number of channels of the convolution layer of the second ConvBlock block of the decoder Decoder is 64 , the convolution kernel is 3, the step size is 1, and the padding is 1; in step f-9), the window size of the average pooling layer of the first branch of the second SEBlock block is 1; the number of channels of the first convolution layer of the first branch of the second SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the second SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the second SEBlock block are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step f-12), the number of channels of the convolution layer of the third ConvBlock block of the decoder Decoder is 1, the convolution kernel is 3, the step size is 1, and the padding is 1; in step f-14), the number of output nodes of the linear layer of the decoder Decoder is 64.

[0150] In one embodiment of the present invention, step g) comprises the following steps:

[0151] g-1) The noise image I common,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m c,i .

[0152] g-2) by formula Calculate the similarity value ACC c , where d ham (m rec,i ,m i) is the binary encrypted identity-aware watermark m c,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC c When the noise image I is less than or equal to the threshold α, common,i For a fake image, when the similarity value ACC c When it is greater than the threshold α, the noise image I common,i For a true image, the threshold α is set to [0.7-0.85].

[0153] g-3) The tampered image I swap,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m s,i .

[0154] g-4) Through the formula Calculate the similarity value ACC s , where d ham (m rec,i ,m i ) is the binary encrypted identity-aware watermark m s,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC s When the tampered image I is less than or equal to the threshold β, swap,i For a fake image, when the similarity value ACC s When it is greater than the threshold β, the tampered image I swap,i For a true image, the threshold β is set to [0.7-0.85].

[0155] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A deep fake face-changing detection method based on robust identity-aware watermark, characterized in that: The steps include: a) Obtain n face images and obtain a face image set I, where I = {I1, I2, …, I i ,…,I n }, where I i is the i-th face image, i∈{1,…,n}; b) For the i-th face image I i Perform preprocessing to obtain the preprocessed face image I raw,i , the preprocessed face image set is I raw , I raw = {I raw,1 ,I raw,2 ,…,I raw,i ,…,I raw,n }; c) Establish watermark generator G m , the preprocessed face image I raw,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m i ; d) Establish an encoder Encoder to convert the preprocessed face image I raw,i and encrypted identity-aware watermark m i Input to the encoder Encoder, and output the watermark image I rec,i ; e) The watermark image I rec,i Injection noise pool P common In the above example, we get the noise image I common,i , the watermark image I rec,i Inject face swap pool P swap In the tampered image I swap,i ; f) Establish a decoder Decoder, which converts the noise image I common,i Or the tampered image I swap,i Input into the decoder Decoder, and output the final watermark m rec,i ; g) According to the final watermark m rec,i Determine the noise image I common,i And the tampered image I swap,i is a real image or a fake image; Step c) comprises the following steps: c-1) Watermark generator G m It is composed of collision-free units and one-way calculation units; c-2) The pre-processed face image I raw,i Input to the watermark generator G m In the collision-free unit, the face recognition algorithm ArcFace is used to identify the preprocessed face image I raw,i Get the dimension d E The identity embedding vector e i ; c-3) Use principal component analysis (PCA) to reduce the dimension of the identity embedding vector e i From dimension d E Project to 64 and get the projected embedding vector E pca,i ; c-4) Corresponding projection embedding vector E pca,i Perform normalization operation to obtain a vector E between [0,1] scale,i ; c-5) Set the threshold c, c = 0.5, if E scale,i ≥c, then the vector E scale,i The value of is set to 1 if E scale,i <c, then vector E scale,i The value of is set to 0, and the identity-aware watermark E is obtained after resetting binary,i ; c-6) Through the formula Calculate the value k of the i-th position of the binary encryption key k i , k={k1,k2,…,ki,…,kl}, where x i is the i-th value in the chaotic map, x i =rx i-1 (1-x i-1 ), r = 3.93, x i-1 is the i-1th value in the chaotic map, i∈{1,…,l}, l is the watermark length, l=64, x0=0.1, p=5, q=11, mod2 is x i p q Modulo operation on 2; c-7) Add the identity-aware watermark E binary,i Input to the watermark generator G m In the one-way computational unit, the identity-aware watermark E binary,i Perform logical XOR operation with the binary encryption key k to obtain the binary encrypted identity-aware watermark m i ; Step d) comprises the following steps: d-1) Encoder consists of a first branch, a second branch, a ConvBlock block, and a convolutional layer; d-2) The first branch of the encoder consists of ConvBlock, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. raw,i Input into the ConvBlock block of the first branch, and output the image feature F clean_1 , the SEBlock block consists of a first branch and a second branch. The first branch of the SEBlock block consists of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the SEBlock block consists of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence. d-3) The image feature F clean_1 Input into the first branch of the first SEBlock block, and output the image feature F clean_2 , the image feature F clean_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_1 , the image feature F clean_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_2 , the image feature F clean_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature F is output. clean_3_3 , the image feature F clean_3_3 With image feature F clean_2 Perform matrix multiplication operation to obtain image feature F clean_4 , the image feature F clean_4 With image feature F clean_1 Perform matrix addition operation to obtain image feature F clean_5 ; d-4) The image feature F clean_5 Replace the image feature F in step d-3) clean_1 Repeat step d-3) to obtain the image feature F output by the second SEBlock block clean_6 ; d-5) Repeat step d-4) to obtain the Nth enc The image features F output by SEBlock clean_7 ; d-6) The second branch of the encoder consists of a linear layer, a ConvBlock block, N enc DiffBlock blocks, N enc The ConvBlock block is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. The encrypted identity-aware watermark m i Input into the linear layer of the second branch, output the watermark feature f1, input the watermark feature f1 into the ConvBlock block, output the watermark feature f2, the DiffBlock block is composed of a transposed convolution layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence, the SEBlock block is composed of the first branch and the second branch, the first branch of the SEBlock block is composed of an average pooling layer, a first convolution layer, a LeakyReLU activation function layer, and a second convolution layer in sequence, the second branch of the SEBlock block is composed of a first convolution layer, a first batch normalization layer, a first LeakyReLU activation function layer, a second convolution layer, a second batch normalization layer, a second LeakyReLU activation function layer, a third convolution layer, a third batch normalization layer, and a third LeakyReLU activation function layer in sequence; d-7) inputting the watermark feature f2 into the first DiffBlock block of the second branch of the encoder Encoder, and outputting the watermark feature f3; d-8) replacing the image feature f2 in step d-7) with the image feature f3 and repeating step d-7) to obtain the watermark feature f4 output by the second DiffBlock block; d-9) Repeat step d-8) to obtain the Nth enc The watermark feature f5 is output by the first DiffBlock block; d-10) the watermark feature f5 is input into the first branch of the first SEBlock block, and the watermark feature f is output. 5_2 , the watermark feature f 5_2 It is sequentially input into the first convolutional layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_1 , the watermark feature f 5_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature F is output. 5_3_2 , the watermark feature F 5_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the watermark feature f is output. 5_3_3 , the watermark feature f 5_3_3 With the watermark feature f 5_2 Perform matrix multiplication operation to obtain the watermark feature f 5_4 , the watermark feature f 5_4 Perform matrix addition operation with watermark feature f5 to obtain watermark feature f 5_5 ; d-11) The watermark feature f 5_5 Replace the watermark feature f5 in step d-10) and repeat step d-10) to obtain the watermark feature f output by the second SEBlock block. 5_6 ; d-12) Repeat step d-11) to obtain the Nth enc The watermark feature f6 output by the SEBlock block; d-13) the image feature F clean_7 The watermark feature f6 is concatenated with the torch.cat() function in the torch library in Python to obtain feature C1. The ConvBlock block of the encoder is composed of a convolution layer, a batch normalization layer, and a LeakyReLU activation function layer. Feature C1 is input into the ConvBlock block of the encoder and output to obtain feature C2. Feature C2 is combined with the preprocessed face image I raw,i The concatenation operation is performed through the torch.cat() function in the torch library in Python to obtain feature C3; d-14) Input feature C3 into the convolutional layer of the encoder and output the watermark image I rec,i ; d-15) Through the formula L enc =||I rec,i -I raw,i ||2 Calculate pixel loss function L enc , where ||·||2 is the L2 norm, and the Adam optimizer is used to use the pixel loss function L enc Train the encoder; Step e) comprises the following steps: e-1) Establish the noise pool P common , noise pool P common The noise includes Dropout noise, Reszie noise, Jpeg noise, GaussianNoise noise, SaltPepper noise, GaussianBlur noise, and MedBlur noise. rec,i Injection noise pool P common In the noise pool P common Randomly select a noise watermark image I rec,i Add noise to get the noise image I common,i ; e-2) Establishing face exchange pool P swap , face swap pool P swap The SimSwap model, InfoSwap model, and UniFace model are included. rec,i Inject face swap pool P swap , from the face exchange pool P swap Randomly select a model to watermark image I rec,i Perform face swap to obtain the tampered image I swap,i ; e-3) Using the Adam optimizer, the face swap pool P is optimized using the adversarial loss function. swap The model in is trained; Step f) comprises the following steps: f-1) The decoder consists of the first ConvBlock block, N se The first SEBlock block, the second ConvBlock block, N dec The second SEBlock block, the third ConvBlock block, the flattening layer, and the linear layer; f-2) The first ConvBlock block of the decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer. common,i Or the tampered image I swap,i Input to the first ConvBlock block of the decoder Decoder, and output the image feature N1; f-3) The first SEBlock block is composed of a first branch and a second branch. The first branch of the first SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the first SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence. f-4) Input the image feature N1 into the first branch of the first SEBlock block, and output the image feature N 1_2 , the image feature N 1_2 It is sequentially input into the first convolutional layer, the first normalization layer, and the first LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_1 , the image feature N 1_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_2 , the image feature N 1_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first SEBlock block, and the image feature N is output. 1_3_3 , the image feature N 1_3_3 With image feature N 1_2 Perform matrix multiplication operation to obtain image feature N 1_4 , the image feature N 1_4 Perform matrix addition operation with image feature N1 to obtain image feature N 1_5 ; f-5) The image feature N 1_5 Replace the image feature N1 in step f-4) and repeat step f-4) to obtain the image feature N output by the second first SEBlock block. 1_6 ; f-6) Repeat step f-5) to obtain the Nth se The image feature N2 is output by the first SEBlock block; f-7) The second ConvBlock block of the decoder Decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence, and the image feature N2 is input into the second ConvBlock block of the decoder Decoder, and the image feature N3 is output; f-8) The second SEBlock block is composed of a first branch and a second branch. The first branch of the second SEBlock block is composed of an average pooling layer, a first convolutional layer, a LeakyReLU activation function layer, and a second convolutional layer in sequence. The second branch of the second SEBlock block is composed of a first convolutional layer, a first batch of normalization layers, a first LeakyReLU activation function layer, a second convolutional layer, a second batch of normalization layers, a second LeakyReLU activation function layer, a third convolutional layer, a third batch of normalization layers, and a third LeakyReLU activation function layer in sequence. f-9) Input the image feature N3 into the first branch of the first second SEBlock block, and output the image feature N 3_2 , the image feature N 3_2 It is sequentially input into the first convolution layer, the first batch of normalization layers, and the first LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_1 , the image feature N 3_3_1 It is sequentially input into the second convolution layer, the second batch normalization layer, and the second LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_2 , the image feature N 3_3_2 It is sequentially input into the third convolution layer, the third batch normalization layer, and the third LeakyReLU activation function layer of the second branch of the first second SEBlock block, and the image feature N is output. 3_3_3 , the image feature N 3_3_3 With image feature N 3_2 Perform matrix multiplication operation to obtain image feature N 3_4 , the image feature N 3_4 Perform matrix addition operation with image feature N3 to obtain image feature N 3_5 ; f-10) The image feature N 3_5 Replace the image feature N3 in step f-9) and repeat step f-9) to obtain the image feature N output by the second SEBlock block. 3_6 ; f-11) Repeat step f-10) to obtain the Nth dec The image feature N4 is output by the second SEBlock block; f-12) The third ConvBlock block of the decoder Decoder is composed of a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer in sequence, and the image feature N4 is input into the third ConvBlock block of the decoder Decoder, and the image feature N5 is output; f-13) Input the image feature N5 into the flattening layer of the decoder, and output the image feature N6; f-14) Input the image feature N6 into the linear layer of the decoder Decoder, and output the final watermark m rec,i ; f-15) through the formula L dec =||m rec,i -m i ||2 Calculate the recovery loss function L dec , where ||·||2 is the L2 norm, and the Adam optimizer is used to recover the loss function L dec Train the decoder Decoder; Step g) comprises the following steps: g-1) The noise image I common,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m c,i ; g-2) by formula Calculate the similarity value ACC c , where d ham (m rec,i ,m i ) is the binary encrypted identity-aware watermark m c,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC c When the noise image I is less than or equal to the threshold α, common,i For a fake image, when the similarity value ACC c When it is greater than the threshold α, the noise image I common,i For the true image, the threshold α is set to [0.7-0.85]; g-3) The tampered image I swap,i Input to the watermark generator G m The output is the binary encrypted identity-aware watermark m s,i ; g-4) Through the formula Calculate the similarity value ACC s , where d ham (m rec,i ,m i ) is the binary encrypted identity-aware watermark m s,i With the final watermark m rec,i The Hamming distance between them, when the similarity value ACC s When the tampered image I is less than or equal to the threshold β, swap,i For a fake image, when the similarity value ACC s When it is greater than the threshold β, the tampered image I swap,i For a true image, the threshold β is set to [0.7-0.85].

2. The deep fake face-changing detection method based on robust identity-aware watermark according to claim 1 is characterized by: In step a), n face images are obtained from the CelebA-HQ dataset.

3. The method for detecting deep fake face-changing based on robust identity-aware watermark according to claim 1 is characterized in that: Step b) comprises the following steps: b-1) Use the transforms.Resize() function in Python to resize the i-th face image I i Image resized to 128×128 I i ′; b-2) Use the transforms.ToTensor() function in Python to transform the image I i ′ format is converted to a tensor type image I i ″; b-3) Use the transforms.Normalize() function in Python to transform the image I i ″Perform normalization processing to obtain the preprocessed face image I raw,i .

4. The deep fake face-changing detection method based on robust identity-aware watermark according to claim 1 is characterized by: N enc The value is 3. In step d-2), the number of channels of the convolution layer of the ConvBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the window size of the average pooling layer of the first branch of the SEBlock block of the first branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the first branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the SEBlock block of the first branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; in step d-6), the output node of the linear layer of the second branch of the encoder Encoder is 128; the number of channels of the convolution layer of the ConvBlock block of the second branch of the encoder Encoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; encoding The number of channels of the transposed convolution layer of the DiffBlock block of the second branch of the encoder is 64, the convolution kernel is 2, the step size is 2, and the padding is 0; the window size of the average pooling layer of the first branch of the SEBlock block of the second branch is 1; the number of channels of the first convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the second convolution layer of the first branch of the SEBlock block of the second branch is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the first convolution layer, the second convolution layer, and the third convolution layer of the second branch of the SEBlock block of the second branch are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; the number of channels of the convolution layer of the ConvBlock block of the encoder Encoder in step d-13) is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of channels of the convolution layer of the encoder Encoder in step d-14) is 64 and the convolution kernel is 1.

5. The method for detecting deep fake face-changing based on robust identity-aware watermark according to claim 1, characterized in that: In step f-2), the number of channels of the convolution layer of the first ConvBlock block of the decoder Decoder is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; in step f-3), the window size of the average pooling layer of the first branch of the first SEBlock block is 1; the number of channels of the first convolution layer of the first branch of the first SEBlock block is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; The second convolutional layer of the first branch of the first SEBlock has 64 channels, 3 convolution kernels, 1 stride, and 1 padding. The number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the first SEBlock block are all 64, the convolution kernel is 3, the step size is 1, and the padding is 0; the number of channels of the convolution layer of the second ConvBlock block of the decoder Decoder in step f-7) is 64, the convolution kernel is 3, the step size is 1, and the padding is 1; the window size of the average pooling layer of the first branch of the second SEBlock block in step f-9) is 1; The first convolution layer of the first branch of the second SEBlock block has 64 channels, 3 convolution kernels, 1 stride, and 1 padding. The second convolutional layer of the first branch of the second SEBlock has 64 channels, 3 convolution kernels, 1 stride, and 1 padding. The number of channels of the first convolution layer, the second convolution layer, and the third convolution of the second branch of the second SEBlock block are all 64, the convolution kernels are all 3, the step sizes are all 1, and the padding is all 0; the number of channels of the convolution layer of the third ConvBlock block of the decoder Decoder in step f-12) is 1, the convolution kernel is 3, the step size is 1, and the padding is 1; the number of output nodes of the linear layer of the decoder Decoder in step f-14) is 64.

Images