Implicit black box watermarking method for copyright protection of multi-modal model

Abstract

The invention relates to an implicit black box watermarking method for copyright protection of a multi-modal model, and belongs to the technical field of artificial intelligence safety. The method specifically comprises the following steps: 1, generating a multi-modal model black box backdoor trigger; 2, completing model training of a conversion module according to the backdoor trigger set in the step 1; and 3, performing ownership verification and infringement model judgment on the suspicious model according to the backdoor trigger set in the step 1 and the conversion module in the step 2. According to the method, the resistance trigger is generated by adopting the original samples in the original data set, the limit of an existing distributed external trigger is broken, the method can adapt to a real application scene of a multi-modal model under the condition of small overhead, and the method has good practicability and expandability. In addition, two-stage watermark verification based on a conversion module is adopted, so that the infringement problem caused by hostile attacks can be effectively avoided.

Description

Technical Field

[0001] This invention relates to an implicit black-box watermarking method for copyright protection of multimodal models, belonging to the field of artificial intelligence security technology. Background Technology

[0002] Multimodal models have revolutionized artificial intelligence by achieving cross-modal semantic alignment, such as Contrastive Language-Image Pretraining (CLIP). Their advancements have underpinned key applications ranging from automated content moderation to AI-assisted solutions, transforming large AI models into high-value intellectual property assets. However, the widespread adoption of multimodal models has also attracted malicious actors who attempt to steal and redistribute proprietary models through various means, such as model extraction or parameter duplication. This theft not only undermines the economic incentives for innovation but also increases moral hazard, as stolen models may generate misinformation or bypass security filters.

[0003] Model watermarking is currently considered a potential solution for protecting model copyright. Black-box watermarking techniques allow for copyright verification without access to the model, but non-trigger-based solutions still have limitations due to various reasons, such as model extraction attacks. Trigger-based backdoor methods embed the watermark directly into the model's behavior without needing to know the model architecture or verification parameters. However, most existing solutions rely on out-of-distribution triggers that differ from the model's training data distribution, such as manually created data or irrelevant alternative data.

[0004] Trigger-based methods typically encounter two significant problems. First, the trigger selection process often suffers from statistical anomalies, such as models generating aligned image-text embeddings for mismatched image-text pairs, or exhibiting anomalous spectra or local texture artifacts. This allows adversaries to identify and evade attackers through automated input sanitization. Second, out-of-distribution trigger sets require meticulous data selection from external sources, which is both time-consuming and labor-intensive. Third, existing methods for trigger injection and verification embed the triggers into the model through fine-tuning strategies, inevitably impairing model performance on benign data. Furthermore, adversaries can exploit model fine-tuning to remove embedded backdoor watermarks.

[0005] Therefore, there is an urgent need for a method that can improve the concealment of trigger selection and verification while maintaining model utility, realize copyright protection of multimodal models in black-box scenarios, minimize the impact of backdoor triggers on the original model, and avoid infringement issues caused by malicious attacks. Summary of the Invention

[0006] To achieve copyright protection for multimodal models in a black-box environment, minimize the impact of backdoor triggers on the original model, and avoid infringement issues caused by malicious attacks, this invention proposes an implicit black-box watermarking method for copyright protection of multimodal models.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] An implicit black-box watermarking method for copyright protection of multimodal models includes the following steps:

[0009] Step 1: Generate a multimodal model black-box backdoor trigger, which further includes the following sub-steps:

[0010] Sub-step 1.1: Initialize an adversarial network generator G. By inputting a fixed noise z into the generator G, a general adversarial patch G(z) is obtained.

[0011] Step 1.2: Randomly select a pair of images and text {x,y} from any dataset D. Here, x represents the image and y represents the text.

[0012] Step 1.3: Add the general adversarial patch G(z) to a fixed position in image x to obtain an adversarial image trigger. The steps to add are as follows:

[0013]

[0014] Where ⊙ denotes the element-wise multiplication function, and m is the position mask matrix.

[0015] Sub-step 1.4: Trigger the generated image Combined with the original text y, it is added to the backdoor trigger set T. a middle.

[0016] Sub-step 1.5: Repeat steps 1.2 to 1.4 multiple times until the maximum threshold N of the backdoor trigger set is met. At this point, a set of backdoor triggers T is complete. a The generation of .

[0017] Step 2: Based on the backdoor trigger set T described in Step 1 a The training of the model for the transformation module M is completed, and further includes the following sub-steps:

[0018] Sub-step 2.1: Combine the image and text pair {x,y} selected in step 1 with the generated adversarial image trigger. Together they form the training dataset D train .

[0019] Sub-step 2.2: Select the visual encoder for the original multimodal model O.v and text encoder t Used for extracting the embedding vector E.

[0020] Sub-step 2.3: Jointly train dataset D train Visual encoder v and text encoder t The loss function L during the training of the transformation module is as follows:

[0021]

[0022] in, E(x i ) and E(y i () represent the extracted adversarial image triggers, respectively. The embedding vector E of the image and text pair {x,y}; d(u,v)=1-cos(u,v) is used to measure the cosine similarity between two vectors u and v; max(·) is the maximum value function; λ and η are training hyperparameters.

[0023] Step 3: According to the backdoor trigger set T described in Step 1 a The conversion module M described in step 2 completes the ownership verification and infringement model determination for the suspicious model S, and further includes the following sub-steps:

[0024] Sub-step 3.1: Inject a set of backdoor triggers T into the suspicious model S. a Arbitrary adversarial image triggers Obtain suspicious model output results

[0025] Sub-step 3.2: Add a transformation module M after the suspicious model S, and re-inject the adversarial image trigger. Obtain the output of the conversion model

[0026] Sub-step 3.3: Combine the output result Res#1 of the suspicious model and the output result Res#2 of the transformed model to perform watermark verification. The verification formula is shown in equations (3) and (4):

[0027]

[0028] Where Verify(·) represents the watermark verification function; Result represents the final copyright verification result of the multimodal model; True and False respectively indicate whether the suspicious model S is an infringing version of the original model O.

[0029] By injecting adversarial triggers into suspicious models in a black-box environment, this invention can adapt to real-world multimodal model applications with low overhead, demonstrating good practicality and scalability. By utilizing original samples from the original dataset, this invention reduces the impact of watermarks on the original model, achieves the generation of adversarial triggers, and breaks the limitations of existing out-of-distribution triggers. Furthermore, the two-stage watermark verification based on the conversion module effectively avoids infringement issues caused by malicious attackers.

[0030] Beneficial effects

[0031] Compared with the prior art, the method of the present invention has the following beneficial effects:

[0032] 1. This invention achieves copyright protection for multimodal models in a black-box scenario, while considering the verification of original model ownership and the determination of infringing models, which can effectively protect the interests of the original owners of multimodal models.

[0033] 2. This invention injects adversarial triggers into suspicious models in a black-box scenario, enabling it to adapt to real-world multimodal model applications with low overhead. While reducing the impact on the performance of the original model, it breaks the limitations of existing distributed external triggers, achieving low-overhead infringement model determination.

[0034] 3. This invention employs a two-stage watermark verification based on a conversion module, effectively avoiding infringement issues caused by malicious attackers. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the process of the present invention;

[0036] Figure 2 This is a diagram of the architecture of the present invention.

[0037] Detailed implementation methods and examples

[0038] To better illustrate the objectives and advantages of this invention, the following detailed description, in conjunction with the accompanying drawings and embodiments, further elaborates on the invention. It should be noted that the embodiments and examples are merely illustrative and do not constitute any limitation upon the invention.

[0039] See the instruction manual appendix Figure 1 and 2 The specific implementation steps of this invention are as follows:

[0040] Step 1: Generate a multimodal model black-box backdoor trigger, which further includes the following sub-steps:

[0041] Sub-step 1.1: Initialize an adversarial network generator G, which is jointly trained with the visual encoder and text encoder of the original model. By inputting a fixed noise z sampled from a specific noise distribution into the generator G, a general adversarial patch G(z) is obtained.

[0042] Sub-step 1.2: Randomly select a pair of images and text {x,y} from any dataset D in the original model training dataset, where x represents the image "cat" and y represents the text "cat". See the appendix in the instruction manual. Figure 2 .

[0043] Sub-step 1.3: Add the general adversarial patch G(z) to a fixed position in image x. This fixed position can be freely set, for example, the bottom right corner of the original image, to obtain an adversarial image trigger. This adversarial image trigger Visually similar to the original image x, but vastly different in the semantic space of the multimodal model. To the human eye, both appear as "cat" images, but the image trigger will be identified as a "tiger" in the multimodal model (see attached manual). Figure 2 The steps to add are as follows:

[0044]

[0045] Where ⊙ denotes the element-wise multiplication function, and m is the position mask matrix.

[0046] Sub-step 1.4: Trigger the generated image that visually resembles the original image. Combined with the original text y, it is added to the backdoor trigger set T. a middle.

[0047] Sub-step 1.5: Repeat steps 1.2 to 1.4 multiple times until the maximum threshold N of the backdoor trigger set is met. At this point, a set of backdoor triggers T is complete. a The generation of .

[0048] Step 2: Based on the backdoor trigger set T from Step 1 a The training of the model for the transformation module M is completed, and further includes the following sub-steps:

[0049] Sub-step 2.1: Combine the image and text pair {x,y} selected in step 1 with the generated adversarial image trigger. Together they form the training dataset D train ,

[0050] Sub-step 2.2: Select the visual encoder for the original multimodal model O. v and text encodert Used for extracting the embedding vector E, E t (y) = Encoder t (y),E v (x) = Encoder v (x).

[0051] Sub-step 2.3: Jointly train dataset D train Visual encoder v and text encoder t Training the transformation module. The loss function L used during training is as follows:

[0052]

[0053] in, E(x i ) and E(y i () represent the extracted adversarial image triggers, respectively. The embedding vector E of the image and text pair {x,y}; d(u,v)=1-cos(u,v) is used to measure the cosine similarity between two vectors u and v; max(·) is the maximum value function; λ and η are training hyperparameters.

[0054] Step 3: Based on the backdoor trigger set T from Step 1 a The conversion module M in step 2 completes the ownership verification and infringement model determination for the suspicious model S, and further includes the following sub-steps:

[0055] Sub-step 3.1: Inject a set of backdoor triggers T into the suspicious model S. a Arbitrary adversarial image triggers Obtain suspicious model output results For example, if an adversarial image trigger that visually resembles a "cat" is injected. Current output result The current method stops here, but attackers can distinguish the difference between the current output "tiger" and the expected result "cat," thus discovering the existence of a watermark protection method. This allows them to launch a more powerful attack targeting the watermark scheme. (See the attached manual.) Figure 2 .

[0056] Sub-step 3.2: Add a transformation module M after the suspicious model S, and re-inject the adversarial image trigger. Obtain the output of the conversion model At this point, the conversion model output results Attackers cannot detect the existence of a watermark protection method from the current output "cat" and the expected output "cat", so no further attacks will occur.

[0057] Sub-step 3.3: Combine the output result Res#1 of the suspicious model and the output result Res#2 of the transformed model to perform watermark verification. The verification formula is shown in equations (3) and (4):

[0058]

[0059] Here, Verify(·) represents the watermark verification function; Result represents the final copyright verification result of the multimodal model, and True and False indicate whether the suspicious model S is an infringing version of the original model O, respectively. If the output result Res#1 of the suspicious model and the output result Res#2 of the transformed model are different, the output result of the watermark verification function is 0, and the final copyright verification result of the multimodal model is False, indicating that the copyright verification fails and the current suspicious model S is an infringing version of the original model O. Conversely, if the copyright verification passes, the current suspicious model S is not an infringing version of the original model O. For example, given an adversarial image trigger that visually resembles a "cat," the output result Res#1 of the suspicious model is "tiger," while the output result Res#2 of the transformed model is "cat," and the two are not the same. Since the transformed model is trained from the original model, and the transformation module can correct the output result of the suspicious model, the current result reflects that the current suspicious model has a high similarity to the original model, and can be determined as an infringing version. See the appendix in the instruction manual. Figure 2 .

[0060] The present invention has been described in detail above through embodiments. It should be noted that the above embodiments are merely preferred embodiments of the present invention and not all embodiments of the present invention, nor do they constitute any limitation on the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims in the application documents. Technical solutions or embodiments obtained by those skilled in the art through equivalent substitutions or other various modifications to the technical features of the present invention without departing from the spirit and essence of the present invention all fall within the scope of protection of the present invention.

Claims

1. An implicit black-box watermarking method for multi-modal model copyright protection, characterized in that, The implicit black-box watermarking method comprises the following steps: step 1: generating a multi-modal model black-box backdoor trigger set T a ; step 2: completing model training on a conversion module M according to the backdoor trigger set T a ; step 3: completing ownership verification and infringement model determination on a suspicious model S according to the backdoor trigger set T a and the conversion module M.

2. The implicit black-box watermarking method for copyright protection of a multi-modal model according to claim 1, characterized in that, Step 1 further includes the following sub-steps: Sub-step 1.1: Initialize an adversarial network generator G, and obtain a general adversarial patch G(z) by inputting a fixed noise z into the generator G; Sub-step 1.2: Randomly select a pair of images and text {x,y} from any dataset D, where x represents the image and y represents the text; Sub-step 1.3: Add the general adversarial patch G(z) to a fixed position in the picture x, obtaining an adversarial image trigger The specific addition step is as follows: Where ⊙ denotes the element-matrix function, and m is the position mask matrix; Sub-step 1.4: Triggering the generated image trigger combined with the original text y, added to the set of backdoor triggers T a ; Sub-step 1.5: The sub-step 1.2 to the sub-step 1.4 are repeated multiple times until a maximum threshold N of the set of backdoor flip-flops is met, at which time the generation of a set of backdoor flip-flops T is complete. a ​ 3. The implicit black-box watermarking method for copyright protection of a multi-modal model according to claim 2, characterized in that, Step 2 further includes the following sub-steps: Sub-step 2.1 : trigger the selected picture and text pair {x, y} and the generated adversarial image trigger in step 1 together constitute a training data set D train ; Sub-step 2.2: selecting a visual encoder Encoder of the original multi-modal model O v and a textual encoder Encoder t for the extraction of the embedding vector E; Sub-step 2.3: Joint training of the dataset D train , the visual encoder Encoder v and the text encoder Encoder t The conversion module is trained with a loss function L as follows: wherein, E(x i ) and E(y i ) represent the extracted adversarial image trigger embedding vectors of the picture and text pair {x, y}; d(u, v) = 1 - cos(u, v) is used to measure the cosine similarity between two vectors u and v; max(·) is the maximum value function; λ and η are training hyperparameters.

4. The method of claim 2, wherein the multi-modal model copyright protection implicit black-box watermarking method is characterized by, Step 3 further includes the following sub-steps: Sub-step 3.1: Injecting a set of backdoor triggers T for the suspicious model S a Any adversarial image trigger in Obtaining suspicious model output results Sub-step 3.2: Add a conversion module M after the suspicious model S, inject the adversarial image trigger again Obtaining a conversion model output result Sub-step 3.3: Combine the output result Res#1 of the suspicious model and the output result Res#2 of the transformed model to perform watermark verification. The verification formula is as follows: Where Verify(·) represents the watermark verification function; Result represents the final copyright verification result of the multimodal model; True and False respectively indicate whether the suspicious model S is an infringing version of the original model O.

5. The method of claim 1, wherein, Step 1 further includes the following sub-steps: Sub-step 1.1: Initialize an adversarial network generator G. By inputting a fixed noise z into the generator G, a general adversarial patch G(z) is obtained. Sub-step 1.2: Randomly select a pair of images and text {x,y} from any dataset D, where x represents the image and y represents the text; Sub-step 1.3: Add the general adversarial patch G(z) to a fixed position in the picture x, obtaining an adversarial image trigger The specific addition step is as follows: Where ⊙ denotes the element-matrix function, and m is the position mask matrix; Sub-step 1.4: Triggering the generated image trigger combined with the original text y, added to the set of backdoor triggers T a in; Sub-step 1.5: Repeat steps 1.2 to 1.4 multiple times until the maximum threshold N of the backdoor trigger set is met, at which time the generation of a set of backdoor trigger set T is completed. a ; Step 2 further includes the following sub-steps: Sub-step 2.1 : Triggering the selected picture and text pair {x, y} and the generated adversarial image trigger in step 1 together form the training dataset D train ; Sub-step 2.2: selecting a visual encoder Encoder of the original multi-modal model O v and a textual encoder Encoder t for the extraction of the embedding vector E; Sub-step 2.3: Joint training of the dataset D train , the visual encoder Encoder v and the text encoder Encoder t The conversion module is trained with a loss function L as follows: in, E(x i ) and E(y i () represent the extracted adversarial image triggers, respectively. The embedding vector E of the image and text pair {x,y}; d(u,v)=1-cos(u,v) is used to measure the cosine similarity between two vectors u and v; max(·) is the maximum value function; λ and η are training hyperparameters; Step 3 further includes the following sub-steps: Sub-step 3.1: Inject a set of backdoor triggers T into the suspicious model S. a Arbitrary adversarial image triggers Obtain suspicious model output results Sub-step 3.2: Add a transformation module M after the suspicious model S, and re-inject the adversarial image trigger. Obtain the output of the conversion model Sub-step 3.3: Combine the output result Res#1 of the suspicious model and the output result Res#2 of the transformed model to perform watermark verification. The verification formula is shown in equations (3) and (4): Where Verify(·) represents the watermark verification function; Result represents the final copyright verification result of the multimodal model; True and Fakse respectively indicate whether the suspicious model S is an infringing version of the original model O.

Images