Forgery face image generation method and recognition method based on diffusion model

Abstract

The invention discloses a forged face image generation method and recognition method based on a diffusion model, and the method comprises the steps: collecting various watermark signatures used for a forged face image and a plurality of vanilla images, training an encoder and a decoder of a HiDDeN model, and employing the trained decoder as a watermark extractor to carry out the detection of a watermark in the forged face image; a plurality of vanilla images are collected to form a data set, an original watermark signature m is distributed to each training sample in the data set so as to train an LDM network model, a watermark extractor is adopted to carry out watermark extraction on output images of the LDM network model, and a watermark signature # imgabs0 # is obtained; and constructing a total loss function by using the watermark signature # imgabs1 #, the corresponding original watermark signature m and the image perception loss, and finally acting the total loss function on a back propagation process of a decoder of the LDM network model, so that the LDM network model learns the capability of generating the watermark image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for generating and identifying forged facial images based on a diffusion model. Background Art

[0002] In recent years, generative AI technology has made rapid progress. Driven in particular by generative adversarial networks and diffusion models, image generation technology has become capable of producing forged content that is virtually indistinguishable from real images. Deepfake technology, in particular, uses trained generative models to generate fake images that are virtually indistinguishable from real faces. This technology has been widely used in entertainment, advertising, virtual reality, and other fields. However, its misuse also poses serious social risks, particularly in terms of privacy violations, malicious information dissemination, and identity forgery. Forged facial images have not only had an impact on social media and news platforms, but have also been used to create false identities and commit online fraud.

[0003] Faced with this challenge, effectively detecting forged facial images has become a critical issue in the fields of computer vision and information security. Traditional methods for detecting forged images rely primarily on pixel-level analysis and feature extraction. However, with the continuous evolution of generative models, particularly the introduction of diffusion models, existing detection methods have become unable to meet the high quality and authenticity of generated images. Generative models generate high-quality images through complex learning mechanisms and can even generate images based on human language descriptions, increasing the similarity between forged and real images and rendering traditional image recognition methods ineffective. Summary of the Invention

[0004] This paper provides a method for generating forged facial images based on a diffusion model and a method for identifying watermarked forged facial images. The method aims to effectively distinguish between naturally captured facial images (vanilla images) and forged facial images generated using a diffusion model. The core concept is to embed an invisible watermark in the forged facial image to ensure traceability of the image's source. After the watermark is embedded, the image appears virtually indistinguishable from the original. However, using a pre-trained watermark extractor, the embedded watermark can be recovered from the image, thereby determining whether the image was generated by a generative network model. This method not only maintains efficiency and accuracy in detecting generated and real images, but also provides a new approach for combating image forgery and tampering.

[0005] The present invention can be achieved through the following technical solutions: A method for generating a fake face image based on a diffusion model, comprising the following steps: S1. Pre-trained watermark extractor We collected various watermark signatures used to forge facial images and multiple vanilla images (naturally captured facial images). We trained the encoder and decoder of the HiDDeN model and used the trained decoder as a watermark extractor to detect watermarks in forged facial images. S2. Build a fake face generation model A dataset of multiple vanilla images is collected, and an original watermark signature m is assigned to each training sample in the dataset. The LDM network model is trained based on this. A watermark extractor is used to extract watermarks from the output images of the LDM network model to obtain the watermark signature m'. The total loss function is constructed using the watermark signature m', the corresponding original watermark signature m, and the image perceptual loss. Finally, the total loss function is applied to the back-propagation process of the decoder of the LDM network model, so that the LDM network model learns the ability to generate watermarked images and completes the training to realize the construction of a fake face generation model.

[0006] Furthermore, in step S1, the encoder receives the vanilla image x and the k-bit watermark signature, outputs a residual image δ of the same size as the vanilla image x, uses a hyperbolic tangent function to normalize the residual image δ so that the range is limited to (-1, 1), and uses a coefficient α to perform scaling processing, and then superimposes it with the vanilla image x as the input of the decoder. Finally, the Whitening transform is used to eliminate the deviation of the output image of the decoder.

[0007] Furthermore, when Whitening transform is used to eliminate the deviation, the mean μ of the watermark signature data decoded by the decoder is first calculated, and the corresponding covariance matrix Σ is constructed; then the principal component analysis method PCA is used to perform eigendecomposition on the covariance matrix Σ to obtain the eigenvector matrix and diagonal matrix, and let , where U represents the eigenvector matrix and Λ represents the diagonal matrix; finally, the output image of the decoder is debiased using the following formula: Where W represents the output image of the decoder, and W' represents the output image of the decoder after removing the deviation.

[0008] Furthermore, the calculation formula of the total loss function is as follows Among them, Lm represents the message loss function, that is, the binary cross entropy BCE of the original watermark signature m and the extracted watermark signature m', and Li represents the image perception loss function.

[0009] Furthermore, in step S2, the vanilla image x and the corresponding watermark signature are input into the encoder ε of the LDM network model, and an activation map z=ε(x) is output, whose spatial size is reduced to h=H0 / f=W0 / f by a scaling factor f. The scaled activation image is input into the decoder, and the decoder uses z to reconstruct the image x'=Dm(z). The watermark extractor is then used to decode the watermark signature m' from the reconstructed image x'. Finally, the total loss function is applied to the back propagation process of the decoder of the LDM network model, so that the LDM network model learns the ability to generate watermarked images and completes the training, where H0 represents the height of the original image and W0 represents the width of the original image.

[0010] A method for identifying forged face images with watermarks, comprising the following steps: S1. Pre-trained watermark extractor We collected various watermark signatures used to forge facial images and multiple vanilla images (original images without watermarking), trained the encoder and decoder of the HiDDeN model, and used the trained decoder as a watermark extractor to detect watermarks in forged facial images. S2. Watermark detection and recognition Use the watermark extractor to extract the watermark from the forged face image and obtain the watermark signature , sign the watermark Compare each of the original watermark signatures one by one. If the number of matching bits M corresponding to at least one comparison result is greater than the threshold τ, it means that the forged face image contains a watermark signature, and the original watermark signature corresponding to the maximum number M is selected as the watermark signature contained in the forged face image; Otherwise, it means that the forged face image does not contain the watermark signature.

[0011] Furthermore, in step S1, the encoder receives the vanilla image x and the k-bit watermark signature, outputs a residual image δ of the same size as the vanilla image x, uses a hyperbolic tangent function to normalize the residual image δ so that the range is limited to (-1, 1), and uses a coefficient α to perform scaling processing, and then superimposes it with the vanilla image x as the input of the decoder. Finally, the Whitening transform is used to eliminate the deviation of the output image of the decoder.

[0012] Furthermore, when Whitening transform is used to eliminate the deviation, the mean μ of the watermark signature data decoded by the decoder is first calculated, and the corresponding covariance matrix Σ is constructed; then the principal component analysis method PCA is used to perform eigendecomposition on the covariance matrix Σ to obtain the eigenvector matrix and diagonal matrix, and let , where U represents the eigenvector matrix and Λ represents the diagonal matrix; finally, the output image of the decoder is debiased using the following formula: Where W represents the output image of the decoder, and W' represents the output image of the decoder after removing the deviation.

[0013] The beneficial technical effects of the present invention are as follows: Compared to traditional methods, our recognition method offers several significant advantages: First, the watermark is invisible to the human eye and therefore does not affect the visual quality of the generated image. Second, the watermark embedding remains robust across various generation tasks and image editing processes, effectively resisting common image processing operations such as cropping and compression. We also investigated the watermark's resistance to intentional tampering. Finally, our method provides an effective image tracing mechanism that can track the source of images and prevent malicious use of generative models to forge identities. Through this research, we hope to provide a new technical approach for detecting and preventing generated image forgeries, offering strong technical support for addressing the security risks posed by generative face forgeries. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flow chart of a method for generating a forged face image based on a diffusion model according to the present invention; Figure 2 It is a schematic diagram of the image processing process of the existing diffusion network model LDM of the present invention; Figure 3 It is a flow chart of a method for identifying forged facial images with watermarks according to the present invention; Figure 4 1 is a schematic diagram of comparison curves of TPR and FPR under different image transformations according to an embodiment of the present invention; Figure 5 2 is a schematic diagram of a comparison curve of watermark recognition accuracy under different image transformations according to an embodiment of the present invention; Figure 6 2 is a schematic diagram comparing the quality of images generated by the existing LDM model and the forged face generation model of the present invention in an embodiment of the present invention; Figure 7 2. It is a schematic diagram comparing the watermark results after attack by different types of perceptual autoencoders according to an embodiment of the present invention; Figure 8 This is the model purification attack result of an embodiment of the present invention, showing the relationship between PSNR and watermark extraction accuracy under different fine-tuning strengths; Figure 9 These are two application scenario examples of embedding watermarks to generate images according to the embodiments of the present invention. DETAILED DESCRIPTION

[0015] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings and preferred embodiments.

[0016] The present invention provides a method for generating fake face images based on a diffusion model. By constructing a fake face generation model, the generated image has a specific signature through a pre-trained watermark extractor. The method is divided into two parts for training, such as Figure 1 As shown in the figure, a process including watermark embedding, decoding and image generation is shown. First, we use the encoder WE of the HiDDeN model to encode the image and watermark signature. The encoded watermark signature is decoded into a binary message by the decoder. This process will have a certain loss; secondly, we fine-tune the decoder of the latent diffusion model LDM, encode the image into a latent representation Z through the encoder E, and then decode it through the decoder Dm to obtain a latent image, so that the generated image can be generated according to the same watermark information during decoding, ensuring that the final generated image contains the watermark. Finally, the model creates the image through a generation process, which uses the latent vector Z and the watermark information m to be combined in the fine-tuned decoder Dm to ensure that the final generated image contains the embedded watermark information.

[0017] The details are as follows: 1. Pre-trained watermark extractor Various watermark signatures used to forge facial images and multiple vanilla images (original images without watermarking) are collected, and the encoder and decoder of the HiDDeN model are trained. The trained decoder is used as a watermark extractor to detect watermarks in forged facial images.

[0018] The HiDDeN model is a classic deep watermarking method used for pre-training watermark extractors. By jointly optimizing an encoder (WE) and a decoder, it embeds a k-bit watermark signature into an image, ensuring robustness to various transformations during training. The encoder (WE) receives a vanilla image x and a k-bit watermark signature and outputs a residual image δ of the same size as the vanilla image x. This residual image δ is normalized using a hyperbolic tangent function to a range of (-1, 1) and scaled using a factor α. This residual image δ is then superimposed with the vanilla image x to generate an image xw with the watermark signature, using the formula xw = x + α*tanhδ. This image is then used as the decoder input, and the Whitening transform is applied to eliminate bias in the decoder's output image.

[0019] First, since we only focus on the functionality of the decoder, the encoder WE is discarded after training. Since WE is discarded, its perceptual quality is no longer important, so perceptual loss and adversarial network are no longer needed. The embedded distortion is constrained by the hyperbolic tangent function and the scaling factor α on the output of the encoder WE. This design improves the accuracy of the decoded message and makes it possible to increase the size of the watermark length more efficiently.

[0020] Second, we observe that the dimensions of the decoder output on vanilla images are correlated and highly offset, so we remove the bias by performing a Whitening transform.

[0021] When Whitening transform is used to eliminate deviation, the mean μ of the watermark signature data decoded by the decoder is first calculated, and the corresponding covariance matrix Σ is constructed; Then, the principal component analysis method PCA is used to perform eigendecomposition on the covariance matrix Σ to obtain the eigenvector matrix and diagonal matrix, and let , where U represents the eigenvector matrix (indicating the direction of the principal component), and Λ represents a diagonal matrix whose eigenvalues ​​correspond to the variance of each principal component; Finally, the following formula is used to debias the output image of the decoder.

[0022] Where W represents the output image of the decoder, and W' represents the output image of the decoder after removing the deviation.

[0023] After Whitening transform processing, the output data of the watermark extractor removes the bias, and the different components are de-correlated, which is more consistent with the assumption of independent and identical distribution.

[0024] 2. Build a fake face generation model A data set is composed of multiple vanilla images, and an original watermark signature m is assigned to each training sample in the data set to train the LDM network model. The watermark extractor in step S1 is used to extract watermarks from the output images of the LDM network model to obtain the watermark signature m'. The total loss function is constructed using the watermark signature m', the corresponding original watermark signature m and the image perception loss. Finally, the total loss function is applied to the back propagation process of the decoder of the LDM network model, so that the LDM network model learns the ability to generate watermarked images and completes the training to realize the construction of a fake face generation model.

[0025] In the latent diffusion model, the diffusion process occurs in the latent space of the autoencoder. The resulting latent vector z is then fed into the decoder D to generate the image. Therefore, we need to ensure that the generated image contains the watermark message, which can be extracted by the watermark extractor. Stable watermark embedding is compatible with many generation tasks, and modifying only the decoder D does not affect the behavior of the diffusion process.

[0026] The diffusion model, also known as the denoising diffusion probabilistic model (DDPM), was proposed by Sohl-Dickstein et al. in 2015, and a complete application framework was developed in 2020. It is a generative model based on a diffusion process and has demonstrated significant advantages in image generation tasks. Compared to traditional generative models such as generative adversarial networks (GANs) and variational autoencoders (VAEs), DDPM significantly improves the quality and accuracy of generated images, especially in terms of detail and diversity. This model, based on the concept of Markov chains, can be divided into a forward diffusion process and a backward denoising process.

[0027] Based on DDPM, researchers have conducted many explorations to improve the training speed of the diffusion model and enable it to better cope with more complex image generation tasks. Song et al.

[45] proposed the Denoising Diffusion Implicit Models (DDIM) model. They optimized the reverse denoising process of the original model into a non-Markov chain, so that the image generation process can be completed in fewer steps, thereby significantly improving the generation efficiency. Compared with the traditional diffusion model, DDIM significantly reduces the number of required iterations while maintaining the image generation quality. In addition, Rombach et al. proposed Latent Diffusion Models (LDM), which adds an encoder and decoder to the diffusion model. First, the encoder reduces the dimensionality of the image and extracts the latent space features of the image; then, diffusion and denoising operations are performed on the latent space features; finally, the decoder generates the image through the low-dimensional features of the latent space. This method greatly reduces the computational complexity of the diffusion model, and at the same time constrains the latent space feature representation through a conditional mechanism to generate a specific image.

[0028] In summary, the diffusion model has achieved remarkable results in the field of image generation due to its unique diffusion process and efficient distributed training architecture. Figure 2 As shown, the model is able to generate high-quality, high-precision images that are more in line with human vision and aesthetic logic.

[0029] First, inspired by the LDM's original training autoencoder method, we fine-tune the decoder D to Dm. That is, the training image x and its corresponding original watermark signature m are input into the LDM encoder ε, outputting an activation map z = ε(x). Its spatial size is reduced to h = H0 / f = W0 / f by a scaling factor f, where H0 represents the height of the original image and W0 represents the width of the original image. This is used as the input of the decoder. The decoder uses z to reconstruct the image x' = Dm(z), and then uses the pre-trained watermark extractor to extract the watermark signature m' = W(x') from x'. The message loss function is the binary cross entropy (BCE) of the original watermark signature m and the extracted watermark signature m': Lm=BCE(m,m') In addition, the original decoder D also reconstructs the image based on the latent vector, i.e., the activation map z. However, since it has not been trained to embed watermarks, its output image x0' is called the "unwatermarked image", i.e., the original decoder D will reconstruct the unwatermarked image x0'=D(z). The image perceptual loss Li is the perceptual error between x' and x0', which is used to control distortion. We use the improved LPIPS perceptual loss and Watson-VGG perceptual loss. The role of the perceptual loss is to ensure that the decoder learns the brightness and contrast characteristics of the image, so that D m The generated image is as consistent as possible with the output of D, maintaining visual quality and thus generating a watermark that is less likely to be noticed.

[0030] The total loss function is constructed as follows L=Lm+λiLi and applied to the back propagation process of the decoder Dm so that its weights are optimized through back propagation to minimize the following loss.

[0031] With the help of the forged facial image generation method of the present invention, first, a watermark extractor is pre-trained to enable it to have the ability to recover watermarks after the image has been disturbed by common transformations; second, using the pre-trained watermark extractor, by adding a watermark message loss function and a perceptual loss function, the decoder of the latent diffusion model is fine-tuned to embed a preset watermark signature in the image generation process; further, by applying the watermark extractor to the generated image, the watermark information can be extracted and the model source can be identified; in addition, to enhance detection accuracy, principal component analysis (PCA) is used to whiten the extraction results, thereby improving the independence and robustness of the watermark bit distribution.

[0032] The present invention also provides a method for identifying forged face images with watermarks, such as Figure 3 As shown, the watermark signature of the forged face image is first extracted based on the pre-trained watermark extractor described above, and then the extracted watermark signature data is compared with the original watermark signature one by one to find the corresponding watermark source.

[0033] The details are as follows: 3.1 Watermark Detection We embed a watermark message of length k bits into the generated image. The watermark extractor then decodes the watermark message from the image and compares the decoded message with the original watermark. Let m be the original watermark. We extract the message m' from the image x and compare it with m, using the number of matching bits M(m,m') to determine the similarity between the two. If: M(m,m')≥τ, τ∈0,…,k Then this image contains a watermark, where τ represents the threshold. This method has a certain degree of robustness, which means that even if the image is damaged in some cases (such as compression, noise, etc.), the watermark can still be extracted well.

[0034] Next is the statistical hypothesis to determine whether the image is generated by the model: Alternative hypothesis H1: x is generated by the original model, that is, it contains the watermark. Null hypothesis H0: x is not generated by the original model, that is, it does not contain the watermark. Under the null hypothesis H0, that is, assuming that image x is a normal image (without watermark), we assume that the decoded message bit Independent and identically distributed Bernoulli random variables with a probability of 0.5 for each bit, meaning each bit has a probability of 1 half and 0 half. M(m,m') represents the number of matching bits between the original watermark m and the decoded watermark m', following a binomial distribution M(k,0.5), where k is the length of the watermark and 0.5 is the probability of each matching bit.

[0035] The FPR is the probability that the number of matching bits, M(m,m'), exceeds a threshold, τ, under the null hypothesis, H0. This value can be obtained using the cumulative distribution function (CDF) of the binomial distribution, which gives the probability that the random variable, M, is less than or equal to a certain value. Its closed-form expression can be expressed using the regularized incomplete Beta function as follows: Here, 1 / 2 represents the "midpoint" of the distribution, meaning that the probability of each bit is 0.5.

[0036] 3.2 Watermark Recognition Model providers now pass a Randomly generated signature Embedded in the model distributed to model users, by comparing the extracted messages with the model user's watermark signature one by one, the model provider can track the abuse of the model: for example, generating images that violate policies (such as violent content, deep fakes) and associating them with specific users.

[0037] We compare the message m' extracted by the watermark extractor with Now we need to test N detection hypotheses. If all N hypotheses are rejected, we can conclude that the image is not generated by any model. Otherwise, if at least one hypothesis is true, it means that the image contains a watermark, and we can further use the maximum matching criterion argmax to identify its source model. We attribute the image to the model that meets the following conditions: in, Indicates the number of matching bits, that is, the model with the largest number of matching bits is found as the source model of the watermark signature.

[0038] In the detection task, due to the existence of N tests, the false positive rate is higher. The global false positive rate (FPR) for a given threshold τ is: The formula is used in reverse: we set a target FPR value to infer the required threshold τ. These formulas only hold when the message bits are independent and identically distributed as Bernoulli random variables.

[0039] 4 Experimental results and analysis 4.1 Experimental Setup 1) Dataset: In this study, the CelebA dataset was used for our experiments. The resolution of all images was uniformly resized to 128×128 pixels.

[0040] 2) Experimental Details: To ensure fair comparison, all experiments were conducted on an NVIDIA GeForce RTX 3090 GPU and implemented in Python using the PyTorch framework. The binary length of the copyright information was set to L = 30 bits. The weight of the loss function was λi = 0.1.

[0041] 3) Evaluation Metrics: We use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). We also use Bit Accuracy (BA) to verify robustness. FID is used to evaluate the quality and diversity of generated images.

[0042] 4.2 Watermark Detection Results To detect watermarks, we fine-tuned the decoder of the latent diffusion model (LDM) and generated 1,000 face images using a random key m. We then tested these generated images using Equation 4-1. In our experiments, we focused on the trade-off between the true positive rate (TPR), the probability of detecting a generated image, and the false positive rate (FPR), the probability of misclassifying a normal image as a generated image.

[0043] To balance TPR and FPR, we adjusted a threshold τ, which ranges from 0 to 48. By continuously changing τ, we can observe the relationship between TPR and FPR. When τ = 0, the model will label all images as generated images, which means that all generated images and normal images will be incorrectly identified as generated images. Therefore, TPR = 1 (all generated images are correctly identified) and FPR = 1 (all normal images are incorrectly labeled as generated images). In experiments, TPR is obtained by direct measurement, that is, we directly calculate the proportion of generated images detected. FPR is usually derived using the above formula. This is because in actual experiments, FPR is usually very small, and it is very difficult to measure the value of FPR directly through experiments. Therefore, we use a derived formula to indirectly calculate FPR.

[0044] Figure 4 We demonstrate the trade-off between TPR and FPR for various image transformations. When the generated images are unmodified, we detect 99% of them while labeling only 1 in 109 standard images (FPR = 10-9). At the same FPR = 10-9, we achieve 84% detection for a 10% cropped image. High detection rates are maintained for unmodified images and common transformations, while maintaining very low false positive rates. However, for multiple combinations of transformations, the detection rate drops to 65%. While this decrease in detection rate is significant, it still significantly outperforms passive methods.

[0045] 4.3 Analysis of watermark detection results Each user has an independent generative model. Given an image, our goal is to determine whether the image was generated by one of the N users and identify which user it was. There are three types of errors: 1) False positive: a normal image is mistakenly labeled as a generated image; 2) False negatives: failure to detect images generated by the generative model; 3) False accusation: The generated image is correctly detected, but the wrong user is identified; We fine-tuned N'=100 generative models with random signatures. Each model generated 100 images. For each image, we calculated the matching score with all N user models from the extracted message. We then selected the user with the highest matching score. If the score exceeds a preset threshold τ, we attribute the image to that user. To determine the appropriate threshold τ, we used the above formula to achieve a false positive rate (FPR) of 10-6. For example, when the number of users N=1, the threshold τ is 40, and when N=1000, the threshold τ is 45. To further improve accuracy, we extrapolate the accuracy by increasing the number of signatures and extending the experiment to the case where N>N' (for example, some users did not generate any images). Figure 5 shows the transformation-by-transformation recognition accuracy under different image transformations.

[0046] When N = 10^5 and the images are unmodified, the user recognition accuracy is 98%. However, when the images undergo various transformation combinations, the recognition accuracy drops to 40%. Despite this, if a user generates three images, the recognition probability for that user still reaches 80%. We have found that at this scale, the false positive rate is zero, and the generated images are not mistakenly attributed to other users. This is because τ is set sufficiently high to effectively avoid false positives and reduce the risk of false positives. Notably, as the number of users N increases, the recognition accuracy decreases. This is because the threshold τ increases with N to reduce false positives. This adjustment is reflected in the above formula: as the number of users increases, we increase the threshold to suppress false positives, but this also leads to a decrease in recognition accuracy.

[0047] 4.4 Image Generation Quality Assessment Figure 6 Qualitative examples of image generation quality are shown, illustrating how fine-tuning the LDM decoder affects the generated image. This subtle change is barely perceptible to the human eye. The LDM decoder learns to add watermark information only to textured regions, while uniform background regions remain unchanged.

[0048] Table 4-1 provides a quantitative evaluation of image generation quality across various tasks, including average values ​​for FID, PSNR, and SSIM. These metrics are used to compare the quality of images generated by the fine-tuned LDM with the original LDM model, as well as post-processing watermarking methods (Dct-Dwt and HiDDeN). The results show that, regardless of the task, the watermark has a minimal impact on the FID of the generated images, with average PSNR of approximately 32dB and SSIM of approximately 0.92, indicating a high degree of similarity between images generated by the watermarked model and those generated by the original model. These metrics are slightly lower than those of professional watermarking methods, but given that in real-world scenarios, users only have access to the watermarked versions of images, this quality is sufficient.

[0049] In practical applications, we often only have access to the generated image containing the watermark, without direct access to the original generated image. Therefore, our goal is to make the generated image as similar as possible to the original image while ensuring the watermark is embedded invisibly, meaning that the watermark is not easily detected. Without access to the original LDM-generated image, it is visually impossible to determine whether the image is watermarked. Even after watermarking, the quality of the generated image is barely affected. Our watermark embedding design ensures the visual integrity of the generated image while maintaining a high similarity to the original model.

[0050] 4.5 Watermark Robustness Evaluation We evaluated the robustness of the watermark under different image transformations. In each task, we generated 10,000 images using the fine-tuned model. Table 4-2 shows the robustness of the watermark under different image transformation attacks. The experimental results show that the watermark still maintains a high level of robustness under most common image transformations. For example, although the extraction accuracy decreases in some extreme cases (such as JPEG compression), the accuracy remains above 0.85, showing strong robustness. The combined transformation has a slightly greater impact on the watermark, but the overall robustness remains strong. It is worth noting that even without applying any image transformation, the watermark extraction accuracy does not reach a perfect level. This is because some images have a uniform background, which makes the watermark embedding more difficult, resulting in a decrease in accuracy.

[0051] In addition, even if no transformation is introduced during LDM fine-tuning, the robustness of the watermark still exists. This is due to the fine-tuning of the decoder, which helps the model learn a more robust watermark embedding method. If specific transformation enhancements are added during the training process, the LDM model will be able to learn a more robust watermark embedding strategy, thereby effectively counteracting the effects of these transformations.

[0052] 4.6 Watermark Resistance We investigated the watermark's resistance to intentional tampering, rather than harmless image transformations like cropping and compression. We considered two threat models: image-level attacks (watermarking the image itself); and generative model-level attacks (tampering with the generative model), where the attacker attempts to maliciously remove the watermark from the image using deep learning techniques, such as adversarial purification and neural network autoencoders. We evaluated the degree of image distortion required for watermark removal under autoencoder attacks at different compression budgets. This is the degree to which the quality of the generated image differs from the original image during the watermark removal process, quantified by PSNR.

[0053] Perceptual autoencoders are designed to generate compressed latent representations of images. We selected state-of-the-art autoencoder models from Bmsjh et al. [1], Cheng et al. [2], and Esser et al. [3], where Bmsjh used compression factors of 1, 4, and 8; Cheng used compression factors of 1, 3, and 6; and Esser used VQ-4 and 8. We used these autoencoder models to attempt to remove watermarks from images and calculated the bit-accuracy of the extracted watermarks. Figure 7 This paper demonstrates the robustness of the watermark after an autoencoder attack. x0 is the original image output by the generative model, and xr is the image processed by the attacker. The bit accuracy depends on the watermark extracted from xr. If the attack is blind, to achieve a watermark with bit accuracy close to random (approximately 50%), the attacker needs to significantly distort the image (PSNR < 26). This means that removing the watermark requires significant modification of the image, resulting in a significant degradation of image quality.

[0054] In a white-box attack scenario, the attacker assumes full knowledge of the internal structure and workings of the watermark extractor, not only understanding the generative model but also having direct access to the extractor. The attacker can use an adversarial attack to optimize the image and remove the watermark under the PSNR constraint. First, the mean squared error loss between the extractor output and a random binary message is minimized, effectively replacing the original signature with the random message, thereby achieving watermark removal with a low distortion budget. The white-box attack is the most effective, reducing the bit accuracy to around 0.6 even at high PSNRs (28-30), demonstrating that the attacker's knowledge of the extractor significantly improves the efficiency of watermark removal.

[0055] After removing the watermark, an attacker may also embed a new signature into an unauthorized image to disguise the identity of another person. This attack emphasizes the importance of protecting the privacy of the watermark extractor. If the watermark extractor is leaked, the attacker can effectively remove the watermark or disguise it, thereby weakening the anti-counterfeiting and tracking functions of the watermark.

[0056] An attacker can also conduct a network-level attack, gaining access to the creator's generative model and using a fine-tuning process to attempt to remove the watermark embedding. This is known as model purification. Experiments were conducted at different fine-tuning steps: 100, 200, and integer multiples of 200 to 1600. First, the message loss Lm, a core component of the watermark embedding process, was removed to attempt to remove the watermark. The attacker then shifted focus to the perceptual loss Li of the LDM autoencoder, optimizing the similarity between the generated image and the original. Figure 8 We present experimental results of the proposed attack, based on the mean squared error (MSE) loss, where PSNR is the peak signal-to-noise ratio between the watermarked image xw and the cleansed image xr, plotted at different stages of the fine-tuning process.

[0057] Experimental results show that bit accuracy decreases gradually with decreasing PSNR. When PSNR is high, the watermark is less affected by purification, and the bit accuracy remains close to 0.9, indicating that the watermark information is largely preserved. When the PSNR drops to around 30.0, the bit accuracy drops significantly, indicating that the watermark is partially removed, but image quality begins to deteriorate, potentially leading to artifacts. This result demonstrates the robustness of our watermark embedding method, ensuring that even during attacks, the watermark retains most of its information. However, excessive attacks can introduce artifacts, degrading image quality.

[0058] We designed a method to quickly fine-tune the decoder of a generative model, which can embed watermarks in all the fake face images it generates. This method does not change the generation process and is therefore compatible with most face recognition technologies based on generative models. These watermarks are not only invisible and robust, but can also help track the generators of fake faces. Compared with traditional passive detection methods, watermark embedding provides a more efficient means of fake face recognition and detection.

[0059] In summary, the present invention provides a method for generating forged facial images based on a diffusion model and a method for identifying forged facial images with watermarks. We embed watermarks into the generated images, which can be used in two application scenarios.

[0060] 1) Detection: The model provider detects whether an image is generated by his model, marking as many generated images as possible while controlling the probability of marking natural images as generated images.

[0061] 2) Identification: The model provider monitors the generator of each image to avoid mistakenly identifying a user who did not generate the image as the generator. Figure 9 Demonstrates that a model provider deploys a potential diffusion model to users.

[0062] References are as follows: [1]Ballé J,Minnen D, Singh S, et al. Variational image compression with a scale hyperprior[J]. arXiv preprint arXiv:1802.01436, 2018. [2]Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition., 2020: 7939-7948. [3]Esser P, Rombach R, Ommer B. Tamingtransformers for high-resolution image synthesis[C]. Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, 2021: 12873-12883. Although specific embodiments of the present invention are described above, those skilled in the art should understand that these are merely examples and that various changes or modifications may be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is limited by the appended claims.

Claims

1. A method for generating fake face images based on a diffusion model, characterized in that The following steps are involved: S1. Pre-trained watermark extractor We collected various watermark signatures used to forge facial images and multiple vanilla images (naturally captured facial images). We trained the encoder and decoder of the HiDDeN model and used the trained decoder as a watermark extractor to detect watermarks in forged facial images. S2. Build a fake face generation model Collect multiple vanilla images to form a data set, and assign an original watermark signature m to each training sample in the data set to train the LDM network model. Use the watermark extractor to extract the watermark from the output images of the LDM network model to obtain the watermark signature , signed with a watermark The total loss function is constructed by combining the corresponding original watermark signature m and the image perception loss. Finally, the total loss function is applied to the back-propagation process of the decoder of the LDM network model, so that the LDM network model learns the ability to generate watermarked images and completes the training to achieve the construction of the forged face generation model. In step S1, the encoder receives a vanilla image x and a k-bit watermark signature, outputs a residual image δ of the same size as the vanilla image x, normalizes the residual image δ using a hyperbolic tangent function, and scales it using a coefficient α, then superimposes the residual image δ with the vanilla image x as input to the decoder, and finally uses Whitening transform to eliminate the deviation of the decoder's output image. When Whitening transform is used to eliminate deviation, the mean μ of the watermark signature data decoded by the decoder is first calculated, and the corresponding covariance matrix Σ is constructed; Then, the principal component analysis method PCA is used to perform eigendecomposition on the covariance matrix Σ to obtain the eigenvector matrix and diagonal matrix, and let , where U represents the eigenvector matrix and Λ represents the diagonal matrix; Finally, the following formula is used to debias the output image of the decoder: Where W represents the output image of the decoder, and W' represents the output image of the decoder after removing the deviation.

2. The method for generating a forged face image based on a diffusion model according to claim 1, characterized in that: The calculation formula of the total loss function is as follows in, Represents the message loss function, that is, the original watermark signature and the extracted watermark signature The binary cross entropy BCE, represents the image perceptual loss function.

3. The method for generating a forged face image based on a diffusion model according to claim 1, wherein: In step S2, the vanilla image And the corresponding watermark signature is input into the encoder of the LDM network model In the output activation map , whose spatial size is reduced to h=H0 / f=W0 / f by a scaling factor f. The scaled activation image is input to the decoder, which uses Reconstructed image , and then use the watermark extractor to reconstruct the image Decode the watermark signature Finally, the total loss function is applied to the back propagation process of the decoder of the LDM network model, so that the LDM network model can learn the ability to generate watermarked images and complete the training. Where H0 represents the height of the original image, and W0 represents the width of the original image.

4. A method for identifying forged face images with watermarks, characterized in that The following steps are involved: S1. Pre-trained watermark extractor We collected various watermark signatures used to forge facial images and multiple vanilla images (original images without watermarking), trained the encoder and decoder of the HiDDeN model, and used the trained decoder as a watermark extractor to detect watermarks in forged facial images. S2. Watermark detection and recognition Use the watermark extractor to extract the watermark from the forged face image and obtain the watermark signature , sign the watermark Compare with each original watermark signature one by one, if at least one comparison result corresponds to a number of matching bits M greater than the threshold , it means that the forged face image contains a watermark signature, and the original watermark signature corresponding to the maximum value of the number M is selected as the watermark signature contained in the forged face image; Otherwise, it means that the forged face image does not contain the watermark signature.

5. The method for identifying forged facial images with watermarks according to claim 4, characterized in that: In step S1, the encoder receives a vanilla image x and a k-bit watermark signature, outputs a residual image δ of the same size as the vanilla image x, normalizes the residual image δ using a hyperbolic tangent function, and scales it using a coefficient α, and then superimposes it with the vanilla image x as the input of the decoder. Finally, the Whitening transform is used to eliminate the deviation of the decoder's output image.

6. The method for identifying forged facial images with watermarks according to claim 5, characterized in that: When Whitening transform is used to eliminate deviation, the mean μ of the watermark signature data decoded by the decoder is first calculated, and the corresponding covariance matrix Σ is constructed; Then, the principal component analysis method PCA is used to perform eigendecomposition on the covariance matrix Σ to obtain the eigenvector matrix and diagonal matrix, and let , where U represents the eigenvector matrix and Λ represents the diagonal matrix; Finally, the following formula is used to debias the output image of the decoder: Where W represents the output image of the decoder, and W' represents the output image of the decoder after removing the deviation.

Images