A latent diffusion model fingerprinting method based on hybrid random encoding

Abstract

The application discloses a potential diffusion model fingerprint method based on hybrid random coding, and belongs to the fields of image processing and artificial intelligence. In view of the problems that the existing potential diffusion model fingerprint method relies on an additional key library, the similarity of initial noise of the same fingerprint is high, the concealment is poor, and the image quality is damaged, the application firstly constructs a fingerprint detection attack model to quantify the concealment; secondly, a hybrid random coding strategy is adopted, a single binary bit is randomly mapped into a parity check code word, and 2-bit and 3-bit coding lengths are mixedly used, so that the randomness of the initial noise of the same fingerprint can be improved without an additional key; finally, the image encoder of the potential diffusion model is fine-tuned through a discrete symbol mean square deviation loss, and the post-processing robustness is specifically improved. The application does not need to store key information, can guarantee the fingerprint extraction accuracy, significantly improve the fingerprint concealment and the image quality of the generated image, has excellent comprehensive performance, and has strong practicability.

Description

Technical Field

[0001] This invention relates to the fields of image processing and artificial intelligence, and specifically to a latent diffusion model fingerprinting method based on hybrid random coding. Background Technology

[0002] In recent years, with the development of artificial intelligence (AI) technology, more and more AI products have come into the public eye. Among them, large language models such as ChatGPT and DeepSeek are typical examples, whose powerful dialogue capabilities bring convenience to users. These AI products provide diverse services to a wide range of users, helping them generate various types of content. In the current era of popular video social media, images and videos have become important mediums for information dissemination. More and more content creators are inclined to use AI products to assist them in content creation, and the quality of generated images is getting higher and higher, with some generated images showing a very high degree of similarity to real photographs.

[0003] Among numerous artificial intelligence products, Stable Diffusion, as a prominent representative of image generation models, is an open-source model capable of efficiently generating high-quality images. This has garnered attention from content creators and led to its widespread use. However, while image generation models bring immense convenience, they also imply potential copyright and abuse risks. For example, some unscrupulous users may exploit these models to generate content containing false information or violating legal and ethical standards. Furthermore, to circumvent the limitations of image generation models, users may tamper with the generated images, creating illegal or unethical content that ultimately negatively impacts social order. Therefore, the management of AI-generated content is increasingly important and is receiving growing attention from governments and societies.

[0004] Currently, my country has issued relevant regulations requiring the implicit or explicit labeling of AI-generated images to manage them. Implicit labeling, due to its relatively small impact on image quality, has attracted more attention from researchers. For latent diffusion models, researchers have achieved image identification and traceability by embedding specific fingerprints containing generator information into images. Among numerous fingerprint embedding methods, embedding fingerprint information into the initial noise of the latent diffusion model has become a popular fingerprinting method due to its high robustness, lack of significant artifacts, and certain plug-and-play characteristics. However, this type of method still has some shortcomings: for the same fingerprint, the generated initial noise containing the fingerprint has high similarity, affecting image quality and fingerprint concealment; while methods that overcome these problems require the use of a random key for each generated image, increasing the storage overhead of the key database and the key retrieval overhead during fingerprint extraction, thus affecting the practicality of the method. Summary of the Invention

[0005] To address the aforementioned shortcomings in the prior art, this invention provides a latent diffusion model fingerprinting method based on hybrid random coding.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A latent diffusion model fingerprinting method based on hybrid random coding includes the following steps:

[0008] S1. Construct a fingerprint detection attack model. Use a proxy model to perform initial noise reconstruction on the target image set and the clean image set. Quantify the fingerprint concealment based on the cosine similarity difference between noises.

[0009] S2. The user's fingerprint is encoded using a hybrid random coding strategy. Initial noise embedded in the fingerprint is generated by sampling with distribution preservation, and the initial noise is input into the latent diffusion model to generate a fingerprint image.

[0010] S3. Fine-tune the image encoder of the latent diffusion model using the discrete symbolic mean square error loss function to improve the robustness of the encoded latent features to post-processing attacks.

[0011] S4. Based on the initial noise of the reconstruction and the preset codebook, perform fingerprint extraction and decoding, and restore the user's fingerprint information through a voting strategy.

[0012] Furthermore, S1 specifically includes:

[0013] S11. Collect the set of prompt words, generate a clean image set through the image generation process of the proxy model, and generate a target image set through the image generation process of the target model;

[0014] S12. Using the initial noise reconstruction process of the surrogate model, the clean image set and the target image set are mapped to a clean initial noise set and a target initial noise set, respectively;

[0015] S13. Calculate the cosine similarity set between any two noises within the clean initial noise set. And the set of cosine similarity between any two noises within the initial noise set of the target. ;

[0016] S14, Obtain The Middle Large similarity value ,calculate The similarity is greater than proportion Calculate based on binomial distribution test Value, when The system determines that the target image set contains fingerprints.

[0017] Furthermore, S2 specifically includes:

[0018] S21. Based on user fingerprint and initial noise size Determine the maximum fingerprint embedding capacity ;

[0019] S22. Construct a preset codebook based on parity check for the bits to be encoded. Define the encoding length as codeword set for:

[0020]

[0021] Where ⊕ represents the bitwise XOR operation. For code word set The image in;

[0022] S23, Alternate from codebook Randomly sampled codewords and concatenated them to form a fingerprint sequence. Until the fingerprint sequence The length reaches the maximum fingerprint embedding capacity ;

[0023] S24. Use distribution-preserving sampling to obtain the fingerprint sequence. Initial noise of the embedded fingerprint mapped to a standard Gaussian distribution .

[0024] Furthermore, the alternating sampling order in S23 is as follows: first, press the fingerprint. From everyone Random sampling, then fingerprinting. From everyone Random sampling is performed and the process is repeated until the length reaches the target.

[0025] Furthermore, S3 specifically includes:

[0026] S31, Constructing an attack that includes image post-processing. The training dataset includes one or more of the following post-processing attacks: JPEG compression, random cropping, random discarding, scaling, median filtering, Gaussian noise, Gaussian blur, salt and pepper noise, and brightness adjustment.

[0027] S32. Define the discrete symbolic mean squared error loss function. :

[0028]

[0029] in, For the sign function, it is used during training. The function is approximated by its gradient.

[0030] S33, with

[0031] To optimize the objective function, the image encoder of the latent diffusion model is... Make fine adjustments, among which For the original image, The latent representation obtained by encoding the original image, The image after post-processing. The latent representation obtained by encoding the post-processed image.

[0032] Furthermore, S4 specifically includes:

[0033] S41. Obtain the fingerprint image to be detected, encode the fingerprint image into a latent representation through the encoder of the latent diffusion model, and then reconstruct the initial noise through an iterative noise addition process;

[0034] S42. Reconstruct the binary sequence based on the symbol information of the initial noise;

[0035] S43. According to the mixed random coding order and preset codebook described in S23, extract the fingerprint bit candidate values ​​sequentially from the binary sequence;

[0036] S44. Perform majority voting on multiple candidate values ​​corresponding to each fingerprint position to obtain the final user fingerprint.

[0037] Furthermore, in S44, the user fingerprint is calculated with the decoding accuracy of each fingerprint, the final decoding accuracy of each fingerprint, and the final fingerprint extraction accuracy as the target, and the calculation method is as follows:

[0038]

[0039]

[0040]

[0041] in, The probability of fingerprint orthogonal decoding. For encoding length, To reconstruct the bit error rate of single-bit symbol flipping in the initial noise; The final decoding accuracy for each fingerprint, The total number of coding blocks corresponding to each fingerprint bit. For more than half of the coded blocks to be correct, The number of the coded block corresponding to each fingerprint; For the final fingerprint extraction accuracy, The length of the user's fingerprint.

[0042] The present invention has the following beneficial effects:

[0043] (1) A fingerprint detection attack was proposed, which enabled quantitative analysis of fingerprint concealment, thereby providing a better understanding of the design of fingerprint methods.

[0044] (2) A hybrid random coding strategy is proposed, which effectively improves the randomness of the initial noise generated by the same fingerprint while avoiding a significant reduction in the robustness of fingerprint extraction, thereby effectively improving the concealment of the fingerprint and the quality of the fingerprint image.

[0045] (3) Discrete symbol mean square error loss is proposed. By designing a loss function for fingerprint encoding characteristics and fine-tuning the image encoder of the latent diffusion model, the robustness of fingerprint extraction is effectively improved. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0047] Figure 2 This is a schematic diagram of various post-processing attacks in embodiments of the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0049] This paper first proposes a Fingerprint DetectAttack (FDA) method for latent diffusion model fingerprint detection attacks. This method uses a surrogate model to reconstruct an initial noise set from a target image set and clean images without fingerprints, and measures the cosine similarity within each set. By comparing the difference in their internal similarity, it determines whether a fingerprint exists in the target image set, placing higher demands on fingerprint concealment. To improve fingerprint concealment and resist this attack, this method proposes a hybrid random encoding strategy. Specifically, by randomly mapping a single binary bit to a preset binary sequence, random encoding is achieved, thereby improving fingerprint concealment. Furthermore, by using a mixture of different encoding lengths, the robustness of fingerprint extraction is mitigated. In addition, to further improve the robustness of fingerprint extraction, Discrete Sign Mean Squared Error (DS-MSE) is proposed to fine-tune the image encoder of the latent diffusion model, more specifically improving the encoder's robustness against post-processing attacks. Extensive simulation experiments show that this method can achieve robustness comparable to state-of-the-art methods while significantly improving image quality and fingerprint concealment without relying on additional key information. This indicates that the proposed method can better balance multiple performance metrics and has greater practicality. The method framework of this paper is as follows: Figure 1 As shown.

[0050] Regarding fingerprint detection attacks, let the original hypothesis be... The target image set (i.e., the image set to be detected). And a clean image set (i.e., a generated image set that is known not to contain fingerprints and whose initial noise is sampled from a standard Gaussian distribution). They have the same distribution. The attacker's goal is to use a clean set of images. and target image set By designing detection methods Calculate when the null hypothesis True target image set The probability that the fingerprint is not included ,Right now:

[0051] (1)

[0052] Therefore, when When (this value is a standard threshold commonly used in binomial distribution testing), reject the null hypothesis. The target image set is considered Different distributions indicate that the process of embedding fingerprints has affected the distribution of the image; conversely, a different distribution indicates that the image does not contain fingerprints.

[0053] The goal of this study is to design a fingerprint embedding method for latent diffusion models based on initial noise modification. Based on specific user fingerprints In the initial noise Embed fingerprint information to obtain initial noise containing fingerprint information. Then, the potential image is obtained through iterative denoising. Finally, the decoder of the latent diffusion model is used. Decode it into a fingerprint image In the fingerprint extraction stage, an encoder using a latent diffusion model is employed. fingerprint image Encoding as a latent image The initial noise for reconstruction is then obtained through an iterative noise addition process. Finally, a fingerprint extraction method was designed. Extracting fingerprints from the initial noise of the reconstruction ,in This refers to any additional key information that may be needed. Therefore, the optimization objective of the fingerprint method based on the latent diffusion model with modified latent noise is to achieve higher accuracy in fingerprint extraction, higher image quality, and better resistance to fingerprint detection attacks (i.e., a higher p-value), expressed as:

[0054] (2)

[0055] in, To measure the generated image set containing fingerprints and target image set The similarity evaluation function, For indicator functions, it means It is 1 when it is true, and 0 otherwise.

[0056] Analysis of current advanced methods reveals that for users with specific fingerprints, the initial noise generated by fingerprint methods often exhibits high similarity. For example, the RingID method requires embedding the same ring pattern in Fourier space into the initial noise. This leads to a difference in the statistical distribution of the initial noise compared to when no fingerprint is embedded (i.e., noise sampled from a standard Gaussian distribution), resulting in a discrepancy between the generated image set and the fingerprint-free image set, potentially impacting the quality of the generated images. Therefore, this chapter considers determining the presence of fingerprints in the target image set by comparing the differences within the fingerprint-generated image set and the clean image set. The specific algorithm is shown in Algorithm 1.

[0057] In the fingerprint detection attack scenario, it is assumed that the attacker can access the target model's generation process, consistent with the scenario of calling API interface functions in real-world applications. Simultaneously, the attacker possesses a proxy model independent of the target model, with certain image generation and initial noise reconstruction capabilities; this scenario also aligns with current state-of-the-art research. During the attack process, the attacker first uses a set of prompt words... A clean image set is generated using the generative process of a surrogate model and a target model. and target image set And using a surrogate model, the two image sets are reconstructed into an initial noise set. and After obtaining the initial noise set, the cosine similarity between each initial noise item within each of the two sets is calculated, resulting in two similarity sets. and According to the preset ratio value Select the first from the clean initial noise similarity set A high similarity level is used as a similarity threshold; the proportion of targets with initial noise similarity levels exceeding this threshold is calculated. Finally, the binomial distribution test is used to calculate... Value, when Value less than If there is a significant difference between the image set generated by the target model and the clean image set, it is considered that the target model generates images that contain fingerprints; otherwise, the images generated by the target model are considered not to contain fingerprints.

[0058] The fingerprint detection attack proposed in this embodiment can quantitatively reflect the similarity difference between the target image set and the clean image set in the initial noise of reconstruction using a binomial distribution test, thereby reflecting the fingerprint's stealth. Quantitative analysis of fingerprint stealth can better guide the optimization of fingerprint methods and allow for targeted improvements to latent diffusion model fingerprint methods.

[0059] The core objective of this method is to enhance the stealth of the initial noise generated from the fingerprint of the same user without relying on an additional key information library, making it sufficient to resist fingerprint detection attacks and achieve better image quality. Therefore, this chapter first considers improving the randomness of the initial noise generated from the fingerprint of the same user through a random encoding strategy. This strategy, based on a pre-defined codebook, randomly maps each bit of the original binary fingerprint to a longer binary sequence in the pre-defined codebook, and repeats the above process to increase the redundancy of the fingerprint until the maximum fingerprint capacity (i.e., the size of the initial noise) is reached. Finally, distribution-preserving sampling is used to sample each bit of the binary sequence into a standard Gaussian distribution (this sampling process is consistent with the Gaussian Shading method). During the random mapping process, the binary encoded sequence generated from the same fingerprint will have a certain degree of randomness. At the same time, considering the need to avoid relying on an additional key information library, that is, there is no additional prior knowledge during the extraction process, only the target image itself and a fixed fingerprint embedding algorithm, the length of each mapping result in the pre-defined codebook should be consistent to ensure that the extraction position of each fingerprint can be obtained during the fingerprint extraction process, thereby extracting fingerprint information sequentially. The fingerprint extraction process involves reconstructing the symbols of the initial noise into a binary sequence, decoding it according to the embedded fingerprint algorithm and the preset codebook, and finally obtaining the final extracted fingerprint through a voting strategy.

[0060] To enhance the concealment of the embedded fingerprint by making the encoded binary sequence as random as possible, the encoded sequences of identical binary bits should be as dissimilar as possible, and fixed information at specific positions after encoding should be avoided to prevent increased detection probability. Therefore, this chapter uses a parity check partitioning strategy to construct the codeword set, that is, let the bits to be encoded be... The encoding length is At this point, bits are defined. The corresponding codeword set is:

[0061] (3)

[0062] in, This represents a bitwise XOR operation. During encoding, it starts from... A codeword is randomly selected as the encoded output from the set, and each set contains Each codeword. This encoding method ensures that the probability of any bit being 0 or 1 after encoding is 0 / 1. This ensures that no fixed information appears at any position after random sampling, enhancing the randomness of the initial noise generated for the same user's fingerprint, thereby reducing the possibility of fingerprint detection attacks determining the presence of a fingerprint. Due to the parity check properties, with an encoding length of n, once the first n-1 bits of the encoding result are determined, the nth bit is already determined and lacks randomness. Therefore, since the fingerprint embedding capacity remains constant, the larger the encoding length n, the smaller the proportion of non-random values, and the stronger the randomness of the encoding result. In other words, the fingerprint's concealment is positively correlated with the encoding length n.

[0063] Furthermore, to measure the fingerprint extraction accuracy under this encoding method, let the bit error rate be f (i.e., the probability of each symbol changing in the initial noise during reconstruction) and the encoding length be n. Since parity check partitioning is used for encoding, the decoding result is correct only when an even number of symbols are flipped in the encoding block. Therefore, the decoding accuracy for each fingerprint bit (i.e., the fingerprint encoding block) can be calculated as:

[0064] (4)

[0065] Because this method employs redundant embedding and voting strategies, each fingerprint bit has R coding blocks for voting. Since the initial noise size remains constant, R decreases as the coding length n increases. According to the majority decision principle, more than half of the data must be voted on. Only if the block decoding is correct can the fingerprint bit be ultimately correct. Therefore, the final accuracy rate for each fingerprint bit is:

[0066] (5)

[0067] Let the length of the user's fingerprint be Therefore, the final fingerprint extraction accuracy is:

[0068] (6)

[0069] From the above equation, it can be seen that when the bit error rate f remains constant, As the encoding length n increases, the performance decreases. While choosing a longer encoding length significantly improves fingerprint concealment, it also negatively impacts fingerprint extraction robustness. Therefore, the encoding length should be chosen as short as possible while still meeting the requirements for fingerprint concealment and generated fingerprint image quality. This mitigates the loss of fingerprint robustness and allows the fingerprint extraction method to achieve higher overall performance.

[0070] Meanwhile, preliminary experiments show that while setting the encoding length to 2 improves randomness, it still cannot resist fingerprint detection attacks. The generated image set will be judged to contain fingerprints, meaning the fingerprint concealment is still insufficient. Therefore, considering the need to enhance fingerprint concealment while minimizing the decrease in fingerprint extraction accuracy, this chapter uses a hybrid random encoding with two encoding lengths: 2 and 3 (the two shortest possible encoding lengths). By using multiple encoding lengths, a better balance can be achieved between fingerprint concealment and fingerprint extraction robustness. This further improves fingerprint concealment to resist fingerprint detection attacks while avoiding the decrease in robustness caused by excessively long encoding lengths. In summary, the encoding process proposed in this chapter is shown in Algorithm 2. The fingerprint extraction method is the reverse process of the encoding method. First, a binary sequence is obtained by reconstructing the symbols of the initial noise. Then, fingerprint bits are extracted from the binary sequence sequentially according to a determined embedding process. Finally, the extracted fingerprint sequences are statistically analyzed using a voting strategy to obtain the final extracted fingerprint.

[0071] While the fingerprint encoding method proposed in this invention can effectively improve the concealment of fingerprints, resist fingerprint detection attacks, and improve image quality, its focus on enhancing the randomness of the encoding result also affects the robustness of fingerprints to image post-processing attacks to some extent. Therefore, this section further fine-tunes the image encoder of the latent diffusion model by adding prior knowledge of potential image post-processing attacks to the image encoder, thereby better encoding the attacked image into a latent image.

[0072] The current state-of-the-art method in the literature uses the following training loss function:

[0073] (7)

[0074] in, Represents image post-processing attacks, perceptual loss. Mean Squared Error (MSE) is used to measure the similarity between the attacked image and the original image after encoding and decoding (RGB image). This loss function measures the similarity between the attacked image and the latent image encoded from the original image. By fine-tuning the image encoder using this loss function, the latent noise encoded from the images before and after the attack can be made more numerically consistent, directly and indirectly, through mean squared error loss and perceptual loss, respectively, thereby improving robustness to post-processing attacks.

[0075] However, considering that the fingerprint extraction accuracy in the proposed fingerprint method depends on the accuracy of reconstructing the symbols from the initial noise, ensuring symbol consistency even after post-processing attacks is the core objective of this method, rather than maintaining numerical consistency. The mean squared error loss and perceptual loss aim to measure the numerical difference between two sets of inputs, which differs somewhat from the requirements of this method. Therefore, this section proposes a discrete symbol mean squared error loss. By measuring the symbolic difference between the latent images before and after image post-processing, the image encoder can be optimized in a targeted manner. This allows for a greater focus on enhancing the symbolic consistency between the attacked image and the latent image encoded from the original image, rather than focusing on the specific numerical differences between the latent images. The definition of the Discrete Symbolic Mean Squared Error Loss proposed in this section is as follows:

[0076] (8)

[0077] in This represents the sign function, used to extract the sign of the input value. Considering the sign function... The derivative of the function is almost everywhere zero, which prevents the transmission of effective information during backpropagation. Therefore, this chapter uses the hard hyperbolic tangent function. Gradient approximation is performed during backpropagation to avoid the above problems and ensure normal training. Therefore, the loss function for training the image encoder of the latent diffusion model in this section is:

[0078] (9)

[0079] This loss function only needs to measure the similarity between the attacked image and the latent image encoded from the original image, omitting the loss function used in current state-of-the-art methods. Therefore, during the fine-tuning process of the image encoder, there is no need to use the image decoder to decode the encoded potential image again, which can reduce computational overhead and improve the efficiency of fine-tuning to a certain extent.

[0080] Experimental results

[0081] To evaluate the performance of the proposed fingerprinting method based on a latent diffusion generation model with hybrid random coding and fingerprint detection attacks, this chapter conducts a comprehensive evaluation through extensive simulation experiments. First, the specific setup details of the experiments are introduced. Then, the overall performance of the fingerprinting method is evaluated, including fingerprint image quality and fingerprint extraction accuracy. Next, the robustness of the fingerprinting methods to various post-processing steps is compared. Following this, the robustness of each fingerprinting method to the proposed fingerprint detection attacks is analyzed. Finally, ablation experiments are conducted to ablate the various improvements proposed in this chapter, demonstrating the practical effects of each improvement.

[0082] In the experimental simulations, this chapter uses the currently popular latent diffusion model Stable Diffusion v2.1 Base as the benchmark model. This is a commonly used benchmark model for evaluating fingerprinting methods based on latent diffusion models, and the performance of fingerprinting methods is evaluated on MS-COCO2017. For fingerprint detection attacks, this chapter uses Stable Diffusion v1.4 as the proxy model. This model lags behind the benchmark model, representing the attacker's limitations in model resources, and is closer to the possible real-world scenarios. For the image generation process, all methods are set with a guidance scale of 7.5 and a diffusion step count of 50.

[0083] When evaluating the performance of fingerprint methods, this chapter uses the average bit accuracy (BitAcc) and the true positive rate (TPR) at a false positive rate (FPR) of 1% (TPR@1%FPR) to assess the accuracy of the extracted fingerprints (hereinafter abbreviated as TPR). Unless otherwise specified, the fingerprint length is set in the experiments in this chapter. It is 64 bits, so that it can produce The fingerprints can fully meet the user quantity requirements in actual use. In order to verify the quality of the images generated by the fingerprint method, this chapter uses the Frescher initial distance FID

[18] , which is commonly used in fingerprint methods based on modified initial noise, to evaluate the diversity and quality of the generated images. That is, the fingerprint image is generated according to the first title in the image description title of each of the 5000 images in the MS-COCO 2017 verification set, and the FID is calculated using the fingerprint image set and the original image set.

[0084] For fine-tuning the latent diffusion model image encoder, this chapter uses the AdamW optimizer trained for one epoch on the LAION-Aesthetics

[19] dataset. The batch size during training was 16, and the initial learning rate was 16. .

[0085] For fingerprint detection attacks, a preset ratio value Set as Smaller proportions can better highlight the differences in fingerprint concealment among various fingerprinting methods. The cue word set is 2000 in size; therefore, during the attack process, the resulting cosine similarity set... The number of elements in is .

[0086] To evaluate the performance of HyRaF, the latent diffusion model fingerprinting method based on hybrid random coding proposed in this chapter, this section compares its performance with five similar fingerprinting methods based on modified initial noise. Among them, RingID, due to its design characteristics, can only support a maximum of 2048 users, equivalent to supporting 11-bit binary fingerprints; the Wind method can distinguish up to 100,000 users, approximately equivalent to supporting 17-bit binary fingerprints. For performance evaluation, the Wind_full2048 variant, which has the best performance but also the highest detection computational cost, is selected. For the SWA_LDM method, the SWA_LDM(GS) variant, which has the best performance in its original study, is selected. The Gaussian Shading method, considering that its original scheme requires random key encryption for each image, incurs huge storage space overhead, and that fingerprint extraction requires traversing and searching the key used during embedding, significantly increases computational cost, thus affecting the method's practicality. Therefore, this chapter divides the Gaussian Shading method into two variants: GS represents the original method, and G-S_fix represents encryption using a fixed key, making the method more practical.

[0087] Table 1 shows the performance of different methods using random, specific fingerprints under various evaluation metrics. BitAcc represents the bit precision of fingerprint extraction from the original fingerprint image, while BitAccuracy(Adv) represents the average fingerprint precision under various post-processing attacks. The same applies to TPR and TPR(Adv) metrics. This method uses random, specific fingerprints in the evaluation process, making the performance evaluation closer to real-world scenarios. This is because, for users using the generative model, their perception of the generated image quality stems from the image set generated by their own generative model—that is, an image set containing the user's specific fingerprint—rather than from a collection of images generated from multiple random fingerprints.

[0088] Table 1. Image quality and fingerprint accuracy evaluation results

[0089] As shown in Table 1, the proposed method and Gaussian Shading achieved the best performance in terms of image quality, reaching the same level as the generated image without fingerprints, demonstrating the performance advantage of the proposed method in generating fingerprint-containing images. Meanwhile, considering that this method does not rely on additional key information, it saves additional storage space and computational overhead during fingerprint extraction, thus significantly improving its practicality. The SWA_LDM method increases the randomness of the initial noise generated from the same user's fingerprint by embedding an 8-bit random key as a seed for random sorting in the initial noise. However, due to the small capacity of the 8-bit key (only 256 sorting combinations), it still lags behind the image quality generated without fingerprints and by the proposed method. The RingID and Wind methods, due to their design, require embedding multiple ring patterns in the Fourier space of the initial noise, which significantly disrupts the distribution of the initial noise itself, thus leading to a decrease in image quality. The TraceMark method exhibits a significant disadvantage in image quality due to its unique design during fingerprint embedding. It requires sorting the initial noise according to the absolute value of the noise. While this design can effectively improve fingerprint robustness, it also disrupts the distribution of the initial noise to a greater extent, leading to a decrease in image quality.

[0090] Regarding fingerprint extraction accuracy, the method presented in this chapter achieves perfect accuracy under attack-free conditions, consistent with other advanced methods in the same category. However, when subjected to post-processing attacks, Bit Acc (Adv) shows some shortcomings compared to advanced methods. Nevertheless, in terms of the TPR (adv) metric for actual fingerprint tracing, the difference is small, maintaining essentially the same level of accuracy. This indicates that the method presented in this chapter achieves a good balance between fingerprint image quality, fingerprint extraction accuracy, and method practicality.

[0091] To evaluate the robustness of the methods presented in this chapter for fingerprint extraction, this section tests the accuracy of each fingerprint method under different image post-processing attacks. Schematic diagrams of each attack are shown below. Figure 2 As shown.

[0092] The post-processing attacks selected in this chapter, along with their specific parameters, are as follows: JPEG compression with a quality factor of 25; random cropping of 60% of the region; random discarding of 80% of the region; scaling attack, shrinking to 25% and then restoring; and median filtering. Gaussian noise, Gaussian blur, Salt and pepper noise, Brightness adjustment, with a range of 1 to 6 randomly selected. From Figure 2As can be seen from the image quality data, various post-processing attacks have a significant impact on image quality and thus significantly affect image usability. Therefore, this demonstrates that the post-processing attacks and their strengths selected in this chapter's experiments can effectively evaluate the extraction accuracy of the fingerprint method under more demanding scenarios.

[0093] Table 2 shows the TPR values ​​of the fingerprinting method proposed in this chapter and similar advanced fingerprinting methods under different post-processing attacks. By comparing the TPR values ​​of each fingerprinting method under different attacks, its fingerprint extraction robustness under these attacks can be measured. A higher TPR indicates better robustness to the processing attack, and the fingerprint is easier to identify. In the selection of comparison methods, since the only difference between GS and G-S_fix is ​​the use of random and fixed encryption keys respectively, which affects the fingerprint's concealment, this difference does not affect the robustness of fingerprint extraction. Therefore, only the GS method is selected as the performance evaluation method in Table 2, which can fully demonstrate the robustness of the proposed method.

[0094] Table 2. Image quality and fingerprint accuracy evaluation results

[0095] The results in Table 2 show that the average accuracy of the method presented in this paper under various post-processing attacks is basically at the same level as that of similar state-of-the-art methods, indicating that the method presented in this paper has good robustness against post-processing attacks. It is worth noting that the SWA_LDM method exhibits significant performance differences under different attacks, mainly because its fingerprint extraction process depends on the extraction accuracy of its embedded 8-bit key. When there is an error in key extraction, the correct fingerprint cannot be extracted at all, indicating the inadequacy of the strategy of increasing fingerprint randomness by embedding the key in the initial noise as a random sorting seed. Meanwhile, considering the significant improvement in image quality achieved by the method presented in this paper, it demonstrates higher overall performance, achieving a better trade-off between fingerprint extraction robustness and fingerprint image quality.

[0096] To evaluate the stealth of fingerprint methods after embedding fingerprints, this section uses the fingerprint detection attack proposed in Section 3.2 to perform stealth detection on each fingerprint method. The evaluation results are shown in Table 3. Here, average similarity represents the average cosine similarity between the initial noise in the reconstructed target image set; the proportion exceeding the threshold represents the proportion of initial noise in the reconstructed target image set whose similarity exceeds a preset threshold; when the p-value is below 0.05, the target image set is considered to contain fingerprints, otherwise it is considered not to. The fingerprint-free images in Table 3 represent fingerprint-free images generated using the proxy model, with initial noise sampled from a standard Gaussian distribution.

[0097] Table 3. Fingerprint Concealment Assessment Results

[0098] As shown in Table 3, most current state-of-the-art methods generate initial noise with high similarity under the same specific fingerprint, thus failing to resist fingerprint detection attacks and being identified as containing fingerprints. Apart from the HyRaF and GS methods proposed in this chapter, other fingerprint methods show an average similarity difference of more than one order of magnitude, or even several orders of magnitude, compared to fingerprint-free images (e.g., TraceMark and G-S_fix methods), indicating the disadvantage of current state-of-the-art fingerprint methods in fingerprint concealment. Fingerprint-free images are identified as not containing fingerprints, and a p-value of 1 indicates extremely high confidence, demonstrating that this detection method can effectively distinguish image sets that do not contain fingerprints. Among various similar methods, only the method proposed in this chapter and the Gaussian Shading method (i.e., the GS method in the table) can maintain concealment under fingerprint detection attacks and are not identified as containing fingerprints. However, the Gaussian Shading method relies on applying a random key to each generated image for stream cipher encryption. Considering the large number of generated images in practical applications, a significant amount of storage space is required to store the keys used. Furthermore, the fingerprint extraction process also consumes substantial computational resources to search for the used keys in the key store. Therefore, the practicality of this method is limited. The method presented in this chapter, on the other hand, achieves randomness in the initial noise generated for the same specific fingerprint through hybrid random coding, without relying on additional key information, thus saving storage space and computational resources during fingerprint extraction. Therefore, this demonstrates that the method presented in this chapter achieves good fingerprint concealment while maintaining practicality.

[0099] ablation experiment

[0100] To improve the performance of fingerprinting methods, this chapter proposes several improvements, including a hybrid random coding strategy in the fingerprint embedding process and discrete symbol mean square error loss in the encoder fine-tuning of the latent diffusion model. Therefore, to demonstrate the effectiveness of these improvements, ablation experiments are conducted on each of them.

[0101] (1) Effectiveness of the hybrid random coding strategy

[0102] To evaluate the effectiveness of the hybrid random coding strategy, this chapter compares it with two variants: one using only a 2-length coding scheme (baseline_1) and the other using only a 3-length coding scheme (baseline_2). By comparing fingerprint concealment (i.e., its resistance to the fingerprint detection attack proposed in this chapter) and robustness, the performance advantages of the proposed method are demonstrated. The evaluation results for fingerprint concealment are shown in Table 4.

[0103] Table 4. Concealment Assessment of Hybrid Random Encoding Strategy

[0104] As shown in Table 4, although the proportion of the baseline_1 variant exceeding the threshold is similar to the preset proportion... The gap between the two variants has narrowed compared to other advanced methods, and the average similarity is also basically consistent with the fingerprint images. However, due to the short encoding length, the randomness is still insufficient, so it cannot resist fingerprint detection attacks, indicating that the fingerprint concealment of this variant is insufficient. The baseline_2 variant, due to the increased encoding length, has further enhanced randomness and is sufficient to resist fingerprint detection attacks. The strategy proposed in this chapter, by using a combination of two encoding methods of different lengths, increases the randomness of the encoding process and alleviates the lack of concealment in the baseline_1 variant. Therefore, it is consistent with the baseline_2 variant and is also sufficient to resist fingerprint detection attacks.

[0105] Table 5 shows the TPR values ​​of each ablation variant under different post-processing attacks, evaluating the robustness of fingerprint extraction against each post-processing attack.

[0106] Table 5 Robustness assessment of each ablation variant

[0107] The evaluation results in Table 5 show that the robustness of the baseline_2 variant to various image post-processing attacks is significantly lower than that of the baseline_1 variant, with the most prominent impact being on robustness to Gaussian blur, which decreases by nearly 20%. The method proposed in this chapter, however, greatly mitigates the shortcomings of the baseline_2 variant. Under various image post-processing attacks, the extraction accuracy is basically consistent with that of the baseline_1 variant, with the final average extraction accuracy decreasing by less than 0.6%. Therefore, through a comprehensive analysis of fingerprint concealment and fingerprint extraction robustness, it is demonstrated that the hybrid random coding strategy proposed in this chapter achieves a good trade-off between these two performance indicators, improving the overall performance of the method.

[0108] (2) Effectiveness of Discrete Symbol Mean Square Loss

[0109] To evaluate the effectiveness of the discrete symbol mean squared error loss, this chapter compares it with a variant that does not fine-tune the image encoder (baseline_3) and a variant that uses state-of-the-art methods, namely, a variant that fine-tunes the loss function using MSE and LPIPS (baseline_4). The effectiveness of the proposed strategy is demonstrated by comparing the fingerprint extraction accuracy and robustness to post-processing attacks among the different variants. The fingerprint extraction robustness evaluation results for different variants are shown in Table 5.

[0110] As shown in Table 5, baseline_4 achieves better average fingerprint extraction accuracy compared to baseline_3 without fine-tuning, demonstrating that the current method can effectively improve the robustness of fingerprint extraction. The improved strategy proposed in this chapter, besides maintaining the same fingerprint extraction accuracy as baseline_4 under some attacks, also achieves significant performance improvements under Gaussian noise, Gaussian blur, and salt-and-pepper noise attacks. Therefore, it achieves a significant improvement in average fingerprint extraction accuracy, fully demonstrating the feasibility and effectiveness of the discrete symbolic mean square error approach proposed in this chapter for enhancing the robustness of fingerprint extraction.

[0111] This loss improves the method's performance by focusing more on enhancing the robustness of the latent diffusion model's image encoder to various post-processing attacks at the symbolic level, rather than on specific numerical differences between latent images. Simultaneously, the removal of the LPIPS loss from the loss function saves the additional computational overhead of decoding latent images into RGB images, effectively improving training efficiency and making the fine-tuning process more flexible and practical.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0115] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0116] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A fingerprinting method for a latent diffusion model based on hybrid random coding, characterized in that, Includes the following steps: S1. Construct a fingerprint detection attack model. An initial noise reconstruction is performed on the target image set and the clean image set using a proxy model. Fingerprint concealment is quantified based on the cosine similarity difference between noise levels. Specifically, this includes: S11. Collect the set of prompt words, generate a clean image set through the image generation process of the proxy model, and generate a target image set through the image generation process of the target model; S12. Using the initial noise reconstruction process of the surrogate model, the clean image set and the target image set are mapped to a clean initial noise set and a target initial noise set, respectively; S13. Calculate the cosine similarity set between any two noises within the clean initial noise set. And the set of cosine similarity between any two noises within the initial noise set of the target. ; S14, Obtain The Middle Large similarity value ,calculate The similarity is greater than proportion Calculate based on binomial distribution test Value, when The target image set is determined to contain fingerprints at that time; S2. Encode the user's fingerprint using a hybrid random coding strategy, generate initial noise embedded in the fingerprint through distribution-preserving sampling, and input the initial noise into a latent diffusion model to generate a fingerprint image, specifically including: S21. Based on user fingerprint and initial noise size Determine the maximum fingerprint embedding capacity ; S22. Construct a preset codebook based on parity check for the bits to be encoded. Define the encoding length as codeword set for: Where ⊕ represents the bitwise XOR operation. For code word set Elements in; S23, Alternate from codebook Randomly sampled codewords and concatenated them to form a fingerprint sequence. Until the fingerprint sequence The length reaches the maximum fingerprint embedding capacity ; S24. Use distribution-preserving sampling to obtain the fingerprint sequence. Initial noise of the embedded fingerprint mapped to a standard Gaussian distribution ; S3. Fine-tuning the image encoder of the latent diffusion model using the discrete symbolic mean square error loss function improves the robustness of the encoded latent features to post-processing attacks. Specifically, this includes: S31, Constructing an attack that includes image post-processing. The training dataset includes one or more of the following post-processing attacks: JPEG compression, random cropping, random discarding, scaling, median filtering, Gaussian noise, Gaussian blur, salt and pepper noise, and brightness adjustment. S32. Define the discrete symbolic mean squared error loss function. : in, For the sign function, it is used during training. The function is approximated by its gradient. S33, with To optimize the objective function, the image encoder of the latent diffusion model is... Make fine adjustments, among which For the original image, The latent representation obtained by encoding the original image, The image after post-processing. The latent representation obtained by encoding the post-processed image; S4. Based on the initial noise of the reconstruction and the preset codebook, perform fingerprint extraction and decoding, and restore the user's fingerprint information through a voting strategy.

2. The latent diffusion model fingerprinting method based on hybrid random coding according to claim 1, characterized in that, The alternating sampling order in S23 is as follows: first press the fingerprint. From everyone Random sampling, then fingerprinting. From everyone Random sampling is performed and the process is repeated until the length reaches the target.

3. The latent diffusion model fingerprinting method based on hybrid random coding according to claim 1, characterized in that, S4 specifically includes: S41. Obtain the fingerprint image to be detected, encode the fingerprint image into a latent representation through the encoder of the latent diffusion model, and then reconstruct the initial noise through an iterative noise addition process; S42. Reconstruct the binary sequence based on the symbol information of the initial noise; S43. According to the mixed random coding order and preset codebook described in S23, extract the fingerprint bit candidate values ​​sequentially from the binary sequence; S44. Perform majority voting on multiple candidate values ​​corresponding to each fingerprint position to obtain the final user fingerprint.

4. The latent diffusion model fingerprinting method based on hybrid random coding according to claim 3, characterized in that, In step S44, the user fingerprint is calculated with the decoding accuracy of each fingerprint, the final decoding accuracy of each fingerprint, and the final fingerprint extraction accuracy as the target. in, This represents the probability of correctly decoding the fingerprint. For encoding length, To reconstruct the bit error rate of single-bit symbol flipping in the initial noise; The final decoding accuracy for each fingerprint, The total number of coding blocks corresponding to each fingerprint bit. For more than half of the coded blocks to be correct, The number of the coded block corresponding to each fingerprint; For the final fingerprint extraction accuracy, The length of the user's fingerprint.

Images