A semi-fragile diffusion watermarking method and system based on elliptic curve

Abstract

The present invention discloses a semi-fragile diffusion watermark method and system based on elliptic curves, comprising: constructing first-stage watermark information for tampering detection and second-stage watermark information for model tracing based on elliptic curves; embedding first-stage watermark information in a later stage of denoising based on initial noise through a diffusion process to obtain latent space noise, introducing an extractor for extracting second-stage watermark information from an image generated by a fine-tuned decoder, using the second-stage watermark information as a label, pre-training the extractor and fine-tuning the decoder to achieve embedding of the second-stage watermark information; extracting the second-stage watermark information from an image to be detected by using the pre-trained extractor to achieve model tracing, and detecting whether the image to be detected has been tampered with based on comparison of added noise estimated by inversion from a latent space representation with the first-stage watermark information, so that the diffusion model corresponding to the image can be traced while detecting whether the image has been maliciously tampered with.

Description

Technical Field

[0001] The present invention belongs to the technical field of copyright protection of diffusion models for image generation, and in particular relates to a semi-fragile diffusion watermarking method and system based on elliptic curves. Background Art

[0002] The development of diffusion models has aroused people's attention to image generation technology. With the popularity of current text-generated image products, people have begun to widely use AI to generate countless works of art in different styles. These technologies have brought unprecedented possibilities to the fields of creativity and art. Artists and designers can use them to quickly generate concept maps, illustrations and even complete works of art. But with them come many information security risks. On the one hand, while the diffusion model demonstrates its powerful creative potential, it has also been abused to generate realistic false images for forging and spreading false events. These images are often highly realistic, and ordinary people find it difficult to distinguish between true and false, so they may be misled. On the other hand, as the crystallization of the developer's data, computing power and code wisdom, the output of the diffusion model should be protected by copyright. However, in many scenarios, the generated images are extremely vulnerable to theft and tampering. Therefore, there is an urgent need for a technology to trace the content and detect tampering of generated images.

[0003] One way to solve the above problems is digital watermarking technology. Digital watermarking technology can effectively protect copyright, verify the authenticity and integrity of content, and provide tampering detection capabilities by embedding invisible identification information in digital content. Current research has proposed many watermarking technologies for diffusion models, which basically achieve the goal of embedding invisible and robust watermarks into generated images without affecting the generation capability of the diffusion model. However, there are few semi-fragile watermarking methods for diffusion models. Semi-fragile watermarks are between robust watermarks and fragile watermarks. They can resist certain types of modifications (such as lossy compression), but will be destroyed by other types of tampering. This feature makes semi-fragile watermarks very suitable for accurate image tracing and image tampering detection. Most of the previous semi-fragile watermarking methods were based on post-processing, and direct application on generated images would affect image quality.

[0004] In summary, there is a need for a semi-fragile watermarking scheme for the diffusion model, which can not only trace the source of the image but also detect whether the image has been tampered with, while avoiding the impact of the watermark on the image quality, in order to cope with the complex information security and copyright protection issues brought about by the current AI-generated images. Summary of the invention

[0005] In view of the above, the purpose of the present invention is to provide a semi-fragile diffusion watermarking method and system based on elliptic curves, which takes into account the characteristics of both robust watermarks and fragile watermarks, and can detect whether the image has been maliciously tampered with while tracing the diffusion model corresponding to the image.

[0006] To achieve the above-mentioned object of the invention, an embodiment provides a semi-fragile diffusion watermarking method based on elliptic curve, comprising the following steps:

[0007] Watermark generation process: randomly generate the first private key and the second private key for the model owner and the model user, respectively generate the first public key and the second public key based on the first private key and the second private key through the elliptic curve, construct the first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability;

[0008] Watermark embedding process: For the diffusion model including encoder, diffusion process and decoder, the first-stage watermark information is embedded in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise. An extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise is introduced. The second-stage watermark information is used as the label, and the difference between the images generated based on the latent space noise before and after the decoder is fine-tuned is as small as possible and the second-stage watermark information extracted by the extractor is as close as possible to the generated second-stage watermark information. The extractor is pre-trained and the decoder is fine-tuned at the same time to achieve the embedding of the second-stage watermark information;

[0009] Watermark extraction process: Use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the diffusion model is traced based on the extracted two-stage watermark information, and then the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0010] Preferably, the elliptic curve satisfies Discrete points of a curve A collection consisting of parameters Uniquely defined, all additions and multiplications on elliptic curves are in a finite cyclic domain Completed on is a large prime number representing the size of the field, and As parameters, Take a point on the elliptic curve as the base point, for of level, satisfying , is zero yuan, is the remaining factor, through It is calculated that Represents the number of points on the elliptic curve;

[0011] Generating a first public key and a second public key respectively based on the first private key and the second private key through an elliptic curve includes:

[0012] ;

[0013] ;

[0014] in, represents the first private key, represents the second private key, Indicates calculation on the elliptic curve , and the horizontal axis of the result As the first public key , Indicates calculation on the elliptic curve , and the horizontal axis of the result As the second public key .

[0015] Preferably, constructing one-stage watermark information for tampering detection based on the first private key and the second public key includes: multiplying the first private key and the second public key to obtain the one-stage watermark information.

[0016] Preferably, in the latter stage of denoising based on the initial noise through the diffusion process, a stage of watermark information embedding is performed to obtain latent space noise, including:

[0017] The first stage watermark information Embedded into the later time step of denoising The corresponding latent space noise Specifically, the first-stage watermark information Divide into equal parts according to time steps , the length of each part of the watermark information is ,in, One-stage watermark information Corresponding to the length of the string, the latent space noise of each time step is flattened into a one-dimensional vector, and the part of the one-stage watermark information corresponding to the same time step is embedded into the one-dimensional vector and then converted into the original dimension of the latent space noise to obtain the latent space noise embedded with the one-stage watermark information. After t time steps, the latent space noise with a time step of 0 is obtained.

[0018] Preferably, the extractor is pre-trained and the decoder is fine-tuned at the same time with the goal of making the difference between the images generated based on the latent space noise before and after the decoder fine-tuning as small as possible and the two-stage watermark information extracted by the extractor as close as possible to the generated two-stage watermark information, including:

[0019] Constructing the loss function for fine-tuning the decoder ,in, Fine-tuned decoder For the latent space noise with time step 0 Decode the resulting image, Decoder before fine-tuning For latent space noise Decode the resulting image, Represents the mean square error, which is used to measure The image gap between

[0020] Constructing the loss function for pre-training the extractor ,in Representation Extractor From the image The second-stage watermark information extracted from is the generated two-stage watermark information, Indicates the bit difference rate between watermark information, which is used to measure the closeness between the extracted and generated two-stage watermark information;

[0021] Constructing the overall loss function ,in is a hyperparameter used to control the weights of the two loss functions. The minimum is the optimization goal to pre-train the extractor and fine-tune the decoder at the same time, so that the difference between the image generated based on the latent space noise before and after the decoder fine-tuning is as small as possible, and the two-stage watermark information extracted by the extractor is as close as possible to the generated two-stage watermark information.

[0022] Preferably, the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with, including:

[0023] The latent space representation is inverted to estimate the time step The corresponding noise , and flatten and extract the noise to obtain a continuous bit string And spliced ​​into a length of The bit string , calculate the bit string With one-stage watermark information The average bit accuracy ,when When it is greater than the threshold, it is considered that the first-stage watermark information extraction is successful, and it means that the image to be detected has not been tampered with, otherwise it is considered to be tampered with.

[0024] Preferably, the average bit accuracy Calculated by the following formula:

[0025] ;

[0026] in, , , there is j The length of the partial first-stage watermark string and j The length of the extracted string , is the indicator function, when When it is 1, otherwise it is 0. n The length quantity.

[0027] To achieve the above-mentioned object of the invention, the embodiment of the present invention provides a semi-fragile diffusion watermarking device based on elliptic curve, comprising:

[0028] A watermark generation module, which is used to randomly generate a first private key and a second private key for a model owner and a model user, respectively generate a first public key and a second public key based on the first private key and the second private key through an elliptic curve, construct a first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability;

[0029] The watermark embedding module is used for embedding the first-stage watermark information in the latter stage of denoising based on the initial noise through the diffusion process for the diffusion model including the encoder, the diffusion process and the decoder to obtain the latent space noise, and introduces an extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise. The extractor is pre-trained and the decoder is fine-tuned at the same time with the second-stage watermark information as the label and the goal of making the difference between the images generated based on the latent space noise before and after the decoder fine-tuning as small as possible and the second-stage watermark information extracted by the extractor as close as possible to the generated second-stage watermark information, so as to embed the second-stage watermark information;

[0030] The watermark extraction module is used to extract the second-stage watermark information from the image to be detected using a pre-trained extractor. If the extraction is successful, the extracted second-stage watermark information is traced back to the diffusion model, and then the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0031] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned semi-fragile diffusion watermarking method based on elliptic curve.

[0032] To achieve the above-mentioned purpose of the invention, the embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned semi-fragile diffusion watermark method based on elliptic curve is implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1) For the generation of watermark information, the present invention proposes elliptic curve cryptography, that is, the first public key and the second public key are generated based on the elliptic curve, and the first-stage watermark information and the second-stage watermark information are generated based on this, so that higher security is guaranteed at a smaller key size. This feature ensures that the watermark information is almost impossible to be forged, and at the same time provides a basis for content tracing;

[0035] 2) With the help of a two-stage watermark mechanism, the first-stage watermark is used as a fragile watermark, and the second-stage watermark is used as a robust watermark. In this way, the present invention combines the strong robustness of the robust watermark with the fragility of the fragile watermark, so that the generated image can avoid the destruction of copyright information while being sensitive to tampering, thereby realizing the two functions of content traceability and tampering detection;

[0036] 3) The embedding process of the one- and two-stage watermark information of the present invention is bound to the image generation process of the diffusion model. The watermark is embedded while the model generates the image. This strategy ensures the quality of image generation compared to the post-processing method and has stronger stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flow chart of a semi-fragile diffusion watermark method based on elliptic curve provided in an embodiment;

[0039] Figure 2 is a schematic diagram of generating watermark information provided by an embodiment;

[0040] Figure 3 is a diagram of a process of embedding watermark information in one stage provided by an embodiment;

[0041] Figure 4 is a diagram of a process of embedding watermark information in one stage provided by an embodiment;

[0042] Figure 5 It is a schematic diagram of the structure of a semi-fragile diffusion watermark device based on elliptic curve provided in an embodiment. DETAILED DESCRIPTION

[0043] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0044] The inventive concept of the present invention is: the existing diffusion models for image generation have technical problems of content tracing and tampering detection. These two technical problems need to be solved through different watermark technologies, among which content tracing requires a robust watermark that can resist strong damage, while tampering detection requires a fragile watermark that is easily destroyed. In order to solve these two technical problems at the same time, the embodiment of the present invention provides a semi-fragile diffusion watermark solution based on elliptic curves, which takes into account both the functions of content tracing and tampering detection. In terms of implementation, the present invention adopts a two-stage watermark embedding strategy, which simultaneously realizes the embedding of robust watermarks and fragile watermarks, taking into account the characteristics of the two watermarks while avoiding possible conflicts between them. In addition, the present invention relies on elliptic curves to generate two-stage watermark information, specifically using the superiority of elliptic curve cryptography in security, avoiding the possibility of forging watermark information, and providing a basis for tracing the content.

[0045] In the elliptic curve-based semi-fragile diffusion watermarking solution provided in the embodiment, the application scenarios and operation processes are described as follows:

[0046] The objects involved in the applicable application scenarios include model holder A, user B as the user, and third party C. Among them, model holder A pre-trains the diffusion model and entrusts third party C to allow third party C to deploy and distribute the diffusion model through API. User B can use the diffusion model provided by model holder A to generate pictures through the API provided by third party C, but it does not have the ability to obtain the code and weights of the diffusion model. When third party C obtains the diffusion model from model holder A, it will fine-tune some parameters of the model so that the model has the ability to embed watermarks when generating images. In addition, third party C will also generate public and private keys for model holder A and user B through elliptic curves. When copyright disputes are involved, the third party will trace the source of the content and detect tampering.

[0047] The operation process is as follows: the watermark embedding process only involves the third party C, in which the embedding of the first-stage watermark information does not require any changes to the model parameters, while the embedding of the second-stage watermark information requires the third party C to partially fine-tune the model so that it can have the ability to embed watermarks. When extracting watermarks, the third party C needs to extract the second-stage watermark information and the first-stage watermark information in sequence, corresponding to the two goals of content traceability and tampering detection respectively. The goal of the second-stage watermark information extraction is to extract the complete second-stage watermark information. If successful, the model corresponding to the image output can be confirmed based on the holder of the second-stage watermark traceability model A; the first-stage watermark extraction can only be performed after the second-stage watermark extraction is successful. The goal of the first-stage watermark extraction is that the average bit accuracy of the extracted information and the first-stage watermark information exceeds the threshold. If it is met, it means that the image has not been tampered with, otherwise it means that the image has been tampered with. The reason why the threshold is set as the target of the first-stage extraction is that the image will suffer certain losses during the transmission process, such as lossy compression during network transmission. These losses are not considered tampering in this system. In contrast, some changes that cause great damage to the image, such as forgery, deformation, and cropping, are considered tampering. Therefore, by setting the threshold of the first-stage extraction, the image is allowed to be affected by some common losses, while identifying some malicious tampering that causes great damage to the image.

[0048] like Figure 1 As shown, the semi-fragile diffusion watermark method based on elliptic curve provided in the embodiment includes the following steps:

[0049] S1, watermark generation process: randomly generate the first private key and the second private key for the model owner and the model user, generate the first public key and the second public key respectively based on the first private key and the second private key through the elliptic curve, construct the first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability.

[0050] In the embodiment, the watermark generation process is as follows: Figure 2 As shown, the elliptic curve used is satisfied Discrete points of a curve A collection consisting of parameters Uniquely defined, all additions and multiplications on elliptic curves are in a finite cyclic domain Completed on is a large prime number representing the size of the field, and As parameters, Take a point on the elliptic curve as the base point, for of level, satisfying , is zero yuan, is the remaining factor, through It is calculated that Represents the number of points on the elliptic curve.

[0051] The process of generating a two-stage watermark based on the above elliptic curve includes: using an arbitrary random algorithm to generate an integer ,in Represents no greater than The set of all positive integers . Assigned as the first private key to the model holder A, A keeps the first private key. The third party C only applies to the model holder A for the first private key during the embedding and extraction process of the watermark in the first stage. Computation on elliptic curves , and use the horizontal coordinate of the result as the first public key ,Right now The first public key is public and anyone has access to it, including third party C and user B. Similarly, the randomly generated The second private key is kept by user B, and no one else has access to it. ability to calculate As the second public key of user B.

[0052] For subsequent watermark embedding, the first private key and Second public key Constructing one-stage watermark information for tampering detection , specific use The binary representation of the result is used as the first-stage watermark information , using the second public key The binary representation of . One-stage watermark information And the second stage watermark information The diffusion model is embedded into the image sequentially when generating the image.

[0053] The main reason why the present invention generates a key with the help of an elliptic curve and uses the key as watermark information is that elliptic curve cryptography has extremely strong security, which comes from the complexity of the elliptic curve discrete logarithm problem. Specifically, the elliptic curve discrete logarithm problem refers to: given a point on an elliptic curve and another point , in satisfying Solve under the condition It has been proven that the elliptic curve discrete logarithm problem cannot be solved in polynomial time using a classical computer, which means that if the public key is known In the case of reverse derivation of the private key This characteristic has an impact on the present invention in that it makes it difficult to forge a one-stage watermark. For a malicious user, it is difficult to retain the one-stage watermark while tampering with the image.

[0054] S2, watermark embedding process: For the diffusion model including an encoder, a diffusion process and a decoder, a first-stage watermark information is embedded in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise, and an extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise is introduced. The second-stage watermark information is used as the label, and the difference between the images generated based on the latent space noise before and after the decoder is fine-tuned is as small as possible and the second-stage watermark information extracted by the extractor is as close as possible to the generated second-stage watermark information. The extractor is pre-trained and the decoder is fine-tuned at the same time to achieve the embedding of the two-stage watermark information.

[0055] In the embodiment, the diffusion model of watermark embedding includes a diffusion model of an encoder, a diffusion process and a decoder, wherein the encoder is used to encode the input data to obtain a latent space representation, and the diffusion process is used to denoise the latent space representation, specifically, the latent space vector as the initial noise is used to generate the latent space representation. Through the time steps {T, T-1, T-2, ..., 2, 1}, the noise is gradually reduced to The decoder is used to decode the result of the diffusion process to obtain an image.

[0056] In the embodiment, the first-stage watermark information is embedded through the denoising process of the diffusion model. Specifically, the first-stage watermark information will be directly embedded in the noise in this diffusion process. In order not to affect the quality of image generation and ensure that the watermark is fragile, the present invention adopts the method of continuously masking the later noise to embed the first-stage watermark information.

[0057] Consider the step size In the denoising process, early noise changes will greatly change the final image. This may cause the generated image to be inconsistent with the user's expectations. To overcome this problem, the present invention only embeds the first-stage watermark information in the latter part of the denoising process, and divides the first-stage watermark information into several parts, which are embedded in a continuous noise in sequence.

[0058] In the embodiment, Figure 3 As shown, the first stage watermark information Embedded into the later time step of denoising The corresponding latent space noise Among them, , specifically: the first stage watermark information Divide into equal parts according to time steps , the length of each part of the watermark information is ,in, One-stage watermark information The length of the corresponding string. The actual embedding is done through the mask. Consider the dimension of the latent space noise to be ,in is the height, is the width, is the number of channels, and each time step needs to be embedded i The latent space noise Flatten into a one-dimensional vector in the order of number of channels, height, and width , and the partial one-stage watermark information corresponding to the same time step Embedded into a one-dimensional vector Then it is converted into the original dimension of latent space noise to obtain the latent space noise embedded with the first-stage watermark information. , after t time steps, the latent space noise at time step 0 is obtained The embedded strategy is set in advance and any method can be used, such as least significant bit replacement and discrete cosine transform.

[0059] In the embodiment, Figure 4 As shown in the figure, the two-stage watermark information is embedded through the decoding process of the diffusion model. After that, it can be restored to image X through decoder D. The embedding of the second-stage watermark information is completed in this step. Before embedding the second-stage watermark information, the decoder D needs to be fine-tuned, and each second-stage watermark information corresponds to a fine-tuning. Although this process changes the model parameters and introduces fine-tuning costs, it is logically reasonable. The second-stage watermark information is the first public key of the model holder A. When the model holder A wants to distribute the right to use its diffusion model, the third party needs to fine-tune the diffusion model so that the generated image has an invisible watermark. When it is necessary to trace the source of the generated image, the model that generated the image can be identified by extracting the second-stage watermark information.

[0060] Specifically, the third party C uses the two-stage watermark information Train a neural network as an extractor W, which is used to extract the latent space noise from the fine-tuned decoder Extract the two-stage watermark information from the generated image , and also uses the two-stage watermark information The decoder D of the diffusion model is fine-tuned to obtain the fine-tuned decoder D'. The goal of fine-tuning the decoder D is to not affect the image quality before and after adding the watermark. The training process of the extractor W is carried out simultaneously with the fine-tuning process of the decoder D. Specifically, the decoder is fine-tuned based on the latent space noise before and after The difference between the generated image and the two-stage watermark information extracted by the extractor is as small as possible As close as possible to the generated two-stage watermark information To achieve joint training, we pre-train the extractor and fine-tune the decoder at the same time, including:

[0061] Constructing the loss function for fine-tuning the decoder ,in, Fine-tuned decoder For latent space noise Decode the resulting image, Decoder before fine-tuning For latent space noise Decode the resulting image, Represents the mean square error, which is used to measure The image gap between

[0062] Constructing the loss function for pre-training the extractor ,in Representation Extractor From the image The second-stage watermark information extracted from is the generated two-stage watermark information, It represents the bit difference rate between the watermark information, which is used to measure the closeness between the extracted and generated two-stage watermark information. The goal of extractor W training is to extract the two-stage watermark information from the image output by decoder D. .

[0063] Constructing the overall loss function ,in is a hyperparameter used to control the weights of the two loss functions. The minimum is the optimization goal to pre-train the extractor and fine-tune the decoder at the same time, so that the difference between the image generated based on the latent space noise before and after the decoder fine-tuning is as small as possible, and the two-stage watermark information extracted by the extractor is as close as possible to the generated two-stage watermark information.

[0064] S3, watermark extraction process: use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the diffusion model is traced based on the extracted two-stage watermark information. Then, the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise. The added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0065] In the embodiment, for an image to be detected The watermark extraction process is divided into two stages. First, the pre-trained extractor W is used to extract the watermark from the image to be detected. Directly extract the two-stage watermark information , when successfully extracted, the two-stage watermark The diffusion model corresponding to the source tracing; on the contrary, if it cannot be successfully extracted, it means that the image to be detected It may not have a watermark or may be extremely damaged.

[0066] The first-stage watermark information extraction can only be performed after the second-stage watermark information is successfully extracted. In the first-stage watermark information extraction process, the encoder E is first used to convert the image to be detected into Convert to latent space representation , and then estimate the time steps in turn by inversion Corresponding to the noise added After obtaining the inversion noise, the inversion noise is flattened and extracted in turn to obtain a continuous bit string , concatenate all bit string results into a string of length The bit string . Calculate the bit string And the first-stage watermark information The average bit accuracy between the bit strings is calculated ,in , , there is j The length of the partial first-stage watermark string and j The length of the extracted string , is the indicator function, when When the value is 1, otherwise it is 0. represents successful extraction, otherwise it means the image has been tampered with. The setting is determined by experiments. The experimental content is to process the image in advance, including compression, noise addition, etc., and calculate the bit accuracy of the first-stage watermark extraction of the processed image. The bit accuracy measured by the experimental results is used as the threshold to eliminate the loss suffered during the image transmission process and correctly detect tampering.

[0067] Based on the same inventive concept, the embodiment also provides a semi-fragile diffusion watermark device 50 based on elliptic curves, including a watermark generation module 51, a watermark embedding module 52, and a watermark extraction module 53, wherein the watermark generation module 51 is used to randomly generate a first private key and a second private key for a model owner and a model user, respectively generate a first public key and a second public key based on the first private key and the second private key through an elliptic curve, construct a first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability; the watermark embedding module 52 is used for a diffusion model including an encoder, a diffusion process, and a decoder, to perform a first-stage watermark information embedding in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise, and introduce a decoder for fine-tuning. The extractor extracts the two-stage watermark information from the image generated based on the latent space noise, takes the two-stage watermark information as the label, and takes the difference between the image generated based on the latent space noise before and after the decoder is fine-tuned as small as possible and the two-stage watermark information extracted by the extractor as close as possible to the generated two-stage watermark information, so as to pre-train the extractor and fine-tune the decoder at the same time to achieve the embedding of the two-stage watermark information; the watermark extraction module 53 is used to use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the extracted two-stage watermark information is traced back to the diffusion model, and then the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0068] It should be noted that, when the semi-fragile diffusion watermarking device based on elliptic curve provided in the above embodiment performs semi-fragile diffusion watermarking, the division of the above functional modules shall be used as an example for explanation, and the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the semi-fragile diffusion watermarking device based on elliptic curve provided in the above embodiment and the embodiment of the semi-fragile diffusion watermarking method based on elliptic curve belong to the same concept, and the specific implementation process thereof is detailed in the embodiment of the semi-fragile diffusion watermarking method based on elliptic curve, which will not be repeated here.

[0069] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable codes are stored in the memory, and when the one or more processors execute the executable codes, the computing device is used to implement the above-mentioned semi-fragile diffusion watermarking method based on elliptic curves, specifically including the following steps:

[0070] S1, watermark generation process: randomly generate a first private key and a second private key for the model owner and the model user, respectively generate a first public key and a second public key based on the first private key and the second private key through an elliptic curve, construct a first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability;

[0071] S2, watermark embedding process: For the diffusion model including encoder, diffusion process and decoder, the first-stage watermark information is embedded in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise, and an extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise is introduced. The second-stage watermark information is used as the label, and the difference between the images generated based on the latent space noise before and after the decoder is fine-tuned is as small as possible and the second-stage watermark information extracted by the extractor is as close as possible to the generated second-stage watermark information. The extractor is pre-trained and the decoder is fine-tuned at the same time to achieve the embedding of the second-stage watermark information;

[0072] S3, watermark extraction process: use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the diffusion model is traced based on the extracted two-stage watermark information. Then, the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise. The added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0073] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the semi-fragile diffusion watermark method based on elliptic curve described in S1-S3 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0074] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned semi-fragile diffusion watermark method based on elliptic curve is implemented, which specifically includes the following steps:

[0075] S1, watermark generation process: randomly generate a first private key and a second private key for the model owner and the model user, respectively generate a first public key and a second public key based on the first private key and the second private key through an elliptic curve, construct a first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability;

[0076] S2, watermark embedding process: For the diffusion model including encoder, diffusion process and decoder, the first-stage watermark information is embedded in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise, and an extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise is introduced. The second-stage watermark information is used as the label, and the difference between the images generated based on the latent space noise before and after the decoder is fine-tuned is as small as possible and the second-stage watermark information extracted by the extractor is as close as possible to the generated second-stage watermark information. The extractor is pre-trained and the decoder is fine-tuned at the same time to achieve the embedding of the second-stage watermark information;

[0077] S3, watermark extraction process: use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the diffusion model is traced based on the extracted two-stage watermark information. Then, the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise. The added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

[0078] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0079] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-fragile diffusion watermarking method based on elliptic curve, characterized in that: The following steps are involved: Watermark generation process: randomly generate the first private key and the second private key for the model owner and the model user, respectively generate the first public key and the second public key based on the first private key and the second private key through the elliptic curve, construct the first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability; Watermark embedding process: For the diffusion model including encoder, diffusion process and decoder, the first-stage watermark information is embedded in the latter stage of denoising based on the initial noise through the diffusion process to obtain latent space noise. An extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise is introduced. The second-stage watermark information is used as the label, and the difference between the images generated based on the latent space noise before and after the decoder is fine-tuned is as small as possible and the second-stage watermark information extracted by the extractor is as close as possible to the generated second-stage watermark information. The extractor is pre-trained and the decoder is fine-tuned at the same time to achieve the embedding of the second-stage watermark information; In the latter stage of denoising based on the initial noise through the diffusion process, a stage of watermark information is embedded to obtain latent space noise, including: embedding the stage of watermark information Embedded into the later time step of denoising The corresponding latent space noise Specifically, the first stage watermark information Divide into equal parts according to time steps , the length of each part of the watermark information is ,in, One-stage watermark information Corresponding to the length of the string, the latent space noise of each time step is flattened into a one-dimensional vector, and the part of the one-stage watermark information corresponding to the same time step is embedded into the one-dimensional vector and then converted into the original dimension of the latent space noise to obtain the latent space noise embedded with the one-stage watermark information. After t time steps, the latent space noise with a time step of 0 is obtained; Watermark extraction process: Use the pre-trained extractor to extract the two-stage watermark information from the image to be detected. If the extraction is successful, the diffusion model is traced based on the extracted two-stage watermark information, and then the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

2. The semi-fragile diffusion watermarking method based on elliptic curve according to claim 1 is characterized in that: The elliptic curve satisfies Discrete points of a curve A collection consisting of parameters Uniquely defined, all additions and multiplications on elliptic curves are in a finite cyclic domain Completed on is a large prime number representing the size of the field, and As parameters, Take a point on the elliptic curve as the base point, for of level, satisfying , is zero yuan, is the remaining factor, through It is calculated that Represents the number of points on the elliptic curve; Generating a first public key and a second public key respectively based on the first private key and the second private key through an elliptic curve includes: ； ； in, represents the first private key, represents the second private key, Indicates calculation on the elliptic curve , and the horizontal axis of the result As the first public key , Indicates calculation on the elliptic curve , and the horizontal axis of the result As the second public key .

3. The semi-fragile diffusion watermarking method based on elliptic curve according to claim 1 is characterized in that: Constructing one-stage watermark information for tampering detection according to a first private key and a second public key includes: multiplying the first private key and the second public key to obtain the one-stage watermark information.

4. The semi-fragile diffusion watermarking method based on elliptic curve according to claim 1, characterized in that: With the goal of making the difference between the images generated based on the latent space noise before and after the decoder fine-tuning as small as possible and the two-stage watermark information extracted by the extractor as close as possible to the generated two-stage watermark information, the extractor is pre-trained and the decoder is fine-tuned at the same time, including: Constructing the loss function for fine-tuning the decoder ,in, Fine-tuned decoder For the latent space noise with time step 0 Decode the resulting image, Decoder before fine-tuning For latent space noise Decode the resulting image, Represents the mean square error, which is used to measure The image gap between Constructing the loss function for pre-training the extractor ,in Representation Extractor From the image The second-stage watermark information extracted from is the generated two-stage watermark information, Indicates the bit difference rate between watermark information, which is used to measure the closeness between the extracted and generated two-stage watermark information; Constructing the overall loss function ,in is a hyperparameter used to control the weights of the two loss functions. The minimum is the optimization goal to pre-train the extractor and fine-tune the decoder at the same time, so that the difference between the image generated based on the latent space noise before and after the decoder fine-tuning is as small as possible, and the two-stage watermark information extracted by the extractor is as close as possible to the generated two-stage watermark information.

5. The semi-fragile diffusion watermarking method based on elliptic curve according to claim 1, characterized in that: The noise added to the latent space representation is estimated by inversion, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with, including: The latent space representation is inverted to estimate the time step The corresponding noise , and flatten and extract the noise to obtain a continuous bit string And spliced ​​into a length of The bit string , calculate the bit string With one-stage watermark information The average bit accuracy ,when When it is greater than the threshold, it is considered that the first-stage watermark information extraction is successful, and it means that the image to be detected has not been tampered with, otherwise it is considered to be tampered with.

6. The semi-fragile diffusion watermarking method based on elliptic curve according to claim 5, characterized in that: The average bit accuracy Calculated by the following formula: ； in, , , there is j The length of the partial first-stage watermark string and j The length of the extracted string , is the indicator function, when It is 1 when it is on, otherwise it is 0.

7. A semi-fragile diffusion watermarking device based on elliptic curve, characterized in that: include: A watermark generation module, which is used to randomly generate a first private key and a second private key for a model owner and a model user, respectively generate a first public key and a second public key based on the first private key and the second private key through an elliptic curve, construct a first-stage watermark information for tampering detection based on the first private key and the second public key, and use the first public key as the second-stage watermark information for model traceability; The watermark embedding module is used for embedding the first-stage watermark information in the latter stage of denoising based on the initial noise through the diffusion process for the diffusion model including the encoder, the diffusion process and the decoder to obtain the latent space noise, and introduces an extractor for extracting the second-stage watermark information from the image generated by the fine-tuned decoder based on the latent space noise. The extractor is pre-trained and the decoder is fine-tuned at the same time with the second-stage watermark information as the label and the goal of making the difference between the images generated based on the latent space noise before and after the decoder fine-tuning as small as possible and the second-stage watermark information extracted by the extractor as close as possible to the generated second-stage watermark information, so as to embed the second-stage watermark information; In the latter stage of denoising based on the initial noise through the diffusion process, a stage of watermark information is embedded to obtain latent space noise, including: embedding the stage of watermark information Embedded into the later time step of denoising The corresponding latent space noise Specifically, the first stage watermark information Divide into equal parts according to time steps , the length of each part of the watermark information is ,in, One-stage watermark information Corresponding to the length of the string, the latent space noise of each time step is flattened into a one-dimensional vector, and the part of the one-stage watermark information corresponding to the same time step is embedded into the one-dimensional vector and then converted into the original dimension of the latent space noise to obtain the latent space noise embedded with the one-stage watermark information. After t time steps, the latent space noise with a time step of 0 is obtained; The watermark extraction module is used to extract the second-stage watermark information from the image to be detected using a pre-trained extractor. If the extraction is successful, the extracted second-stage watermark information is traced back to the diffusion model, and then the encoder is used to encode the image to be detected into a latent space representation, and the latent space representation is inverted to estimate the added noise, and the added noise is compared with the first-stage watermark information to detect whether the image to be detected has been tampered with.

8. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, the executable code is used to implement the semi-fragile diffusion watermark method based on elliptic curve according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the semi-fragile diffusion watermark method based on elliptic curve described in any one of claims 1 to 6 is implemented.

Images