Multi-modal model watermarking method based on synonym replacement

Abstract

The invention relates to the technical field of model watermarking, and discloses a multi-modal model watermarking method based on synonym replacement, which comprises the following steps of: inputting a given image and a prompt into a multi-modal model to generate a text sequence for describing the image or a text sequence for answering a question proposed based on the image; watermark information can be detected in the text sequence; the method comprises the following steps of: mapping a secret image into a hash value as a private key by using a perceptual hash algorithm; dividing the vocabulary by using a private key and changing an output sample to embed watermark information; and checking whether the watermark exists or not by counting the number of words in a green list in a text sequence output by the multi-modal model. According to the method, the watermark can be embedded into the multi-modal model only by changing the text output sample, retraining is not needed, and detection can be carried out without accessing the model, so that the method is suitable for all mainstream multi-modal model structures, and strong generalization is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of model watermarking, and in particular to a multimodal model watermarking method based on synonym replacement. Background Art

[0002] Recent advances in computer vision and natural language processing have led to the emergence of visual-language models that integrate visual and language data, provide richer semantic representations and strong generalization across a variety of tasks. The wide applicability and excellent performance of these models make them the focus of today's artificial intelligence research. However, with the proliferation of visual-language models, concerns about abuse have also increased, such as generating false information, privacy violations, and reliability issues. Model watermarking provides protection by embedding a detectable unique identifier, thereby enabling tracking, verifying ownership, and preventing unauthorized use. The main watermarking technologies for multimodal models are:

[0003] ① Embedding scheme of RNN hidden memory state:

[0004] The principle of the RNN hidden memory state embedding scheme is to embed a unique key into the recurrent neural network (RNN) unit to resist various attacks. Specifically, the core of this scheme is that by embedding the key into the hidden state of the RNN, the attack of forged keys can be invalidated. Since the forged key will immediately cause the image description model to produce low-quality output, the model can be effectively protected and the occurrence of infringement can be avoided.

[0005] ②Multimodal model backdoor watermark solution:

[0006] The principle of the backdoor watermarking scheme for multimodal models is to embed a backdoor in the multimodal model to protect the copyright of the model. This scheme uses the characteristics of multimodal data such as images and texts to cleverly embed watermark information into the training data (such as adding color blocks and random noise to images, adding uncommon characters to texts, etc.), and then mixes it with normal data and adds it to the model training, so as to embed the backdoor into the model so that the backdoor trigger can be activated under specific trigger conditions. In this way, if an attacker attempts to tamper with the model or steal it, the embedded backdoor will be triggered to verify the ownership of the model. This method effectively enhances the anti-tampering ability of the model and protects the copyright of the model from infringement.

[0007] But these watermarking techniques have some disadvantages:

[0008] ① Embedding scheme of RNN hidden memory state:

[0009] The embedding scheme of RNN hidden memory state is aimed at visual language multimodal models with recurrent neural network structure. This method has very limited application. The main disadvantages of the existing technology are: 1) it can only be applied to multimodal models with recurrent neural network units, not to models with other architectures, and has poor generalization performance; 2) with the advent of the era of large models, multimodal models with recurrent neural network structure cannot adapt to multimodal large models with increasingly large parameters and increasingly powerful functions.

[0010] ②Multimodal model backdoor watermark solution:

[0011] Among the existing multimodal model backdoor watermarking schemes, the schemes embedded in images mainly focus on some shallow watermark information (such as color blocks, noise), and the schemes embedded in text only have some word-level watermark information (such as special letter combinations, symbols). The main disadvantages of these schemes are: 1) The watermark information is too shallow, resulting in a high probability of false triggering, that is, some features similar to the triggering conditions may also cause false triggering of the backdoor; 2) As the model becomes larger and the received data becomes more diverse, the difficulty of backdoor embedding will become higher and higher, and the probability of false activation will increase, resulting in low watermark detection efficiency.

[0012] In order to solve the problem of poor generalization performance of embedding schemes for RNN hidden memory states, this paper proposes a new watermarking strategy designed specifically for Vision-Language Models (VLM). This paper can embed watermarks into multimodal models by simply changing the text output sampling, without the need for retraining, and can perform detection without accessing the model itself. Therefore, it is applicable to all mainstream multimodal model structures and has strong generalization.

[0013] In order to solve the problem that the backdoor watermark scheme is easily misactivated and leads to low detection efficiency, the present invention proposes a watermark method for changing the text output sampling. Specifically, the present invention randomly divides the word list into two parts: a red list and a green list (where we hope that the words in the green list will appear more in the output). The logits value is the numerical value corresponding to each word obtained after the fully connected layer. After applying softmax to it, the probability of each word is obtained. By adding increments to the logits values ​​of the green list words, synonym replacement is cleverly achieved and the proportion of green list words in the output is increased. This scheme for modifying the logits value is more accurate and more concealed than the shallow watermark information in the backdoor watermark. In the watermark detection stage, the present invention uses a statistical test method on the number of green list words to prove the existence of the watermark, which theoretically eliminates the possibility of false alarms, solves the problem of false positives of backdoor watermarks, and improves detection accuracy and efficiency. Summary of the invention

[0014] In order to solve the above technical problems, the present invention provides a multimodal model watermarking method based on synonym replacement.

[0015] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0016] A multimodal model watermarking method based on synonym replacement, inputting a given image and a prompt into a multimodal model, generating a text sequence describing the image or a text sequence answering a question based on the image, and being able to detect watermark information in the text sequence; the watermarking method specifically includes:

[0017] Watermark generation: The owner of the multimodal model has a secret image I, and uses the perceptual hashing algorithm to map the secret image I into a hash value as the private key K of the watermark;

[0018] Watermark embedding: Input the image and prompt to a multimodal model with vocabulary V, and for the tth token s in the text sequence output by the multimodal model (t) , use the pseudo-random function F and the private key K to calculate the watermark seed H for seeding the random number generator; based on the watermark seed of the random number generator, randomly divide the vocabulary into a green list G of size γ|V| and a red list R of size (1-γ)|V|, where the watermark parameter γ represents the proportion of words in the vocabulary that are in the green list; send a watermark to the token s (t) After adding a constant increment δ to each logits value in the green list at the position, they are sorted in descending order, and then the softmax function is applied to obtain the final token s (t) ;

[0019] Watermark detection: Check whether a watermark exists by counting the number of words in the green list in the text sequence output by the multimodal model.

[0020] Furthermore, the method of checking whether a watermark exists by counting the number of words in the green list in the text sequence output by the multimodal model specifically includes:

[0021] The text sequence output by the multimodal model consists of T tokens, where |s| G is the number of tokens in the subtitle sequence that are in the green list, |s| G The expected value of is γT, and the variance is γ(1-γ)T; calculate the z statistic:

[0022]

[0023] During the detection process, the model owner uses the secret image to generate the private key K, and obtains the red list and green list in the same way as the watermark embedding stage; then, the number of violations of the green list rules and the total number of green list words are counted, and the z statistic is calculated using this information. If the calculated z statistic exceeds the set threshold z-threshold, the null hypothesis is statistically rejected and the conclusion that the watermark exists is drawn.

[0024] Compared with the prior art, the beneficial technical effects of the present invention are:

[0025] Compared with the existing multimodal model watermarking method, the present invention proposes a new watermarking strategy designed specifically for large visual language models, including three stages: watermark generation, watermark embedding and watermark detection. This method can embed the watermark into the multimodal model by only changing the text output sampling, without retraining, and can be detected without accessing the model itself. Therefore, it is applicable to all mainstream multimodal model structures and can provide strong generalization. Compared with the shallow watermark information in the traditional backdoor watermark, it is not only more accurate and more hidden, but also uses a statistical test method to analyze the number of green list words in the watermark detection stage, which theoretically eliminates the possibility of false alarms, thereby significantly improving the detection accuracy and efficiency. At the same time, only replacement between synonyms can greatly reduce the impact on the performance of the original model. A large number of experiments have shown that the present invention achieves a watermark detection rate of more than 90% and remains robust to random attacks. It can also be used for large multimodal generative models and has a good application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the multimodal model watermark framework adopted by the present invention. DETAILED DESCRIPTION

[0027] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0028] The multimodal model watermarking task is defined as: given an image (and a prompt) as input, the multimodal model understands the input multimodal information, generates a description of the image or answers questions based on the image, and finally detects the presence of watermark information in the output text sequence.

[0029] The multimodal model watermarking framework proposed in this invention is as follows Figure 1 As shown in Figure 1, the watermark method as a whole consists of three stages: watermark generation, watermark embedding and watermark detection. In the watermark generation stage, a private key is generated. In the watermark embedding stage, the private key is used to divide the vocabulary and change the output sampling to embed the watermark information. In the watermark detection stage, the number of words in the green list is counted to check whether the watermark exists.

[0030] (1) Watermark generation

[0031] like Figure 1 As shown in the upper left corner, the model owner has a secret image I, which is mapped to a hash value using the perceptual hashing algorithm as the private key K of the watermark. The value of K affects the generation of the watermark seed and, together with the previous token, divides the vocabulary into a red list and a green list. Therefore, each different K will produce a different vocabulary partition, which means that each image corresponds to a unique watermark. With this watermark generation method, the model owner can inject watermarks from different images into the model to create a large number of watermark instances. These instances can be distributed to various downstream user groups. When fake news or suspicious text appears, a specific watermark can be traced back to a specific user, effectively regulating the behavior of the user group.

[0032] (2) Watermark Embedding

[0033] The structure of watermark embedding stage is as follows: Figure 1 As shown in the above part. Consider a visual language model (VLM) with a vocabulary V, applied to the image description task. Given an input image and a length N p The prompt is expressed as Use s (0) ,…,s (T) Represents a caption sequence generated by a visual language model. First, the input image and prompt are processed separately and their features are fused. At each position of the output token, a |V|-dimensional logits vector is obtained, where each word in the vocabulary corresponds to one dimension. Figure 1 Output token s (t) For example, a pseudo-random function F (such as AES for symmetric encryption or SHA3 for cryptographic hash functions) and a private key K are used to calculate a watermark seed H for seeding a random number generator:

[0034] H=F K (s (t-1) );

[0035] Among them, s (t-1) represents the t-1th token. In this embodiment, s (t-1) The token represents the position of the word "of". This random number generator is then used to randomly partition the vocabulary into a green list G of size γ|V| and a red list R of size (1-γ)|V|, where the watermark parameter γ represents the proportion of green list words in the vocabulary.

[0036] To implement the watermarking strategy, we first add a constant increment δ to the logits values ​​of the tokens in the green list. We then sort the vocabulary in descending order based on these adjusted logits values ​​and perform normal sampling after applying softmax. Figure 1As shown in the figure, the current position was originally supposed to output "trash", but after the logits value is adjusted, the output becomes "rubbish". While minimizing the impact on the output quality, synonym replacement is cleverly achieved, and green words are added to embed the watermark.

[0037] (3) Watermark Detection

[0038] The watermark detection stage is as follows Figure 1 As shown in the following part, the green list watermark rule is used to detect the presence of watermark. The detection algorithm does not need to access the parameters or API of the visual language model, and only relies on the generated subtitle sequence for fast implementation. The original hypothesis H0 is proposed: the subtitle sequence is generated without prior knowledge of the watermark rule.

[0039] Without knowledge of the green list rule, the proportion of green list words in natural text is expected to be around γ. However, the watermarked model will exhibit a significantly higher proportion, allowing watermark detection by rejecting the null hypothesis. To evaluate this, a robust statistical method is used: the single-proportion z-test. It is still assumed that the vocabulary V is partitioned into a green list G of size γ|V| and a red list R of size (1-γ)|V|. The subtitle sequence to be detected consists of T tokens, where |s| G is the number of green list words. If the null hypothesis holds, |s| G Following the expected value of γT, the variance is γ(1-γ)T. The z statistic for this test is calculated as follows:

[0040]

[0041] During the detection process, the model owner uses the secret image to generate a private key K. Subsequently, in order to reproduce the corresponding red list and green list at each position, it is only necessary to ensure that the detection phase algorithm is consistent with the settings used in the watermark embedding phase. The number of violations of the green list rules and the total number of green list words are then counted, and this information is used to calculate the z statistic. If the calculated z statistic exceeds the selected threshold, the null hypothesis is statistically rejected and the watermark is concluded to exist. In this invention, unless otherwise specified, the z-threshold is set to 4 as the default choice. This means that when z>4, the null hypothesis is rejected, and the false positive rate is only 3×10 -5 , which corresponds to a one-sided p-value of z>4.

[0042] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0043] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A multimodal model watermarking method based on synonym replacement, characterized in that: The given image and prompt are input into the multimodal model to generate a text sequence describing the image or a text sequence answering questions based on the image, and the watermark information can be detected in the text sequence; The watermarking methods specifically include: Watermark generation: The owner of the multimodal model has a secret image I, and uses the perceptual hashing algorithm to map the secret image I into a hash value as the private key K of the watermark; Watermark embedding: Input the image and prompt to a multimodal model with vocabulary V, and for the tth token s in the text sequence output by the multimodal model (t) , use the pseudo-random function F and the private key K to calculate the watermark seed H for seeding the random number generator; based on the watermark seed of the random number generator, randomly divide the vocabulary into a green list G of size γ|V| and a red list R of size (1-γ)|V|, where the watermark parameter γ represents the proportion of words in the vocabulary that are in the green list; send a watermark to the token s (t) After adding a constant increment δ to each logits value in the green list at the position, they are sorted in descending order, and then the softmax function is applied to obtain the final token s (t) ; Watermark detection: Check whether a watermark exists by counting the number of words in the green list in the text sequence output by the multimodal model.

2. The multimodal model watermarking method based on synonym replacement according to claim 1 is characterized in that: The method of checking whether a watermark exists by counting the number of words in the green list in the text sequence output by the multimodal model specifically includes: The text sequence output by the multimodal model consists of T tokens, where |s| G is the number of tokens in the subtitle sequence that are in the green list, |s| G The expected value of is γT, and the variance is γ(1-γ)T; calculate the z statistic: During the detection process, the model owner uses the secret image to generate a private key K, and obtains the red list and green list in the same way as the watermark embedding stage; then, the number of violations of the green list rules and the total number of green list words are counted, and this information is used to calculate the z statistic; if the calculated z statistic exceeds the set threshold z-threshold, the null hypothesis is statistically rejected and the conclusion that a watermark exists is drawn.

Images