Backdoor watermark-based deep visual language prompt intellectual property protection method and system
By introducing backdoor watermarking into the visual language model, the intellectual property protection problem of lightweight deep visual cue parameters is solved, achieving a highly concealed and robust protection effect, and enabling accurate verification of ownership in a black-box environment.
Patent Information
- Application Number
- CN202610377764.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-26
- Publication Date
- 2026-07-10
AI Technical Summary
Existing intellectual property protection schemes cannot effectively protect lightweight deep visual cue parameters, especially in the "cue-as-a-service" ecosystem, where there is a risk of unauthorized copying and illegal distribution, and there is a lack of dedicated protection measures.
We employ a backdoor watermarking approach, introducing learnable deep visual cue parameters and textual cues into the image and text encoders of a visual language model. We embed watermark information by jointly optimizing the loss function and perform ownership verification using a Bayesian inference framework in a black-box environment.
It achieves highly concealed, harmless, and robust intellectual property protection, with a high watermark trigger success rate. It can resist attacks without compromising the performance of downstream tasks and provides high-confidence ownership verification.
Smart Images
Figure CN122365462A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, natural language processing, and deep learning security, specifically to an intellectual property protection method and system based on deep visual cues using a visual language model. Background Technology
[0002] In recent years, large-scale visual language models (VLMs), represented by CLIP, have demonstrated unprecedented capabilities in visual reasoning tasks such as image classification and object detection. To reduce the high computational and storage costs of full fine-tuning of massive models, parameter-efficient fine-tuning (PEFT) methods have emerged, among which visual cue fine-tuning (VPT) is particularly effective. VPT introduces deep visual cues (less than 1% of the total parameters) into the frozen pre-trained backbone network, achieving performance comparable to or even surpassing that of full fine-tuning.
[0003] The immense success of Visual Cueing as a Service (VPaaS) has spurred the emergence of a new business model. However, due to the extremely small size of deep visual cues (typically only a few KB) and their high plug-and-play compatibility with publicly available base models, they face a very high risk of unauthorized copying and illegal distribution. This seriously threatens the intellectual property (IP) accumulated by cues developers through significant investment of computing power and data costs. Existing IP protection schemes (such as model-level watermarking, data-level tagging, and output-level tracking) are typically designed for large, complete models or generative output designs, and are inadequate when faced with extremely lightweight and layered deep visual cues. Currently, there is a lack of effective protection measures specifically for deep, layered visual cues. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for protecting intellectual property rights of deep visual language cues based on backdoor watermarking, in order to solve the problem that existing technologies cannot verify the intellectual property rights of lightweight visual cues parameters in a "cues-as-a-service" ecosystem. This method is effective, harmless to the original task, and highly robust.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A deep visual language prompting intellectual property protection method based on backdoor watermarking includes the following steps:
[0007] Step 1: Obtain a pre-trained visual language model, introduce multiple sets of learnable deep visual cue parameters into each Transformer layer of the image encoder of the visual language model, and introduce text cue composed of learnable context vectors into the text encoder of the visual language model.
[0008] Step 2, define a secret trigger. and target category The backdoor watermark; during the training phase, a contrastive loss function consisting of image-to-text loss and text-to-image loss is used, combined with the secret trigger. and target category The resulting poisoning prediction loss is used to form a weighted composite loss function; by minimizing the weighted composite loss function, the deep visual cue parameters and the text cue are jointly optimized, and the watermark information is embedded in the cross-modal interaction space;
[0009] Step 3: In a black-box environment where input and output interaction is only possible through the API, select a clean image that is not of the target category and overlay the secret trigger. A poisoning query image is generated; the poisoning query image is input into the suspect model to be verified, and the statistical model outputs the target category. The number of times; based on the statistical results, the posterior probability that the suspected model is a pirated model is calculated using a Bayesian inference framework. If the posterior probability exceeds a preset high confidence threshold, ownership is confirmed.
[0010] Furthermore, in step 1, the visual language model adopts the CLIP architecture.
[0011] Furthermore, the image encoder is a Visual Transformer (ViT), and the deep visual cues are added to the front end of the input sequence of each Transformer layer of the ViT through the Visual Cue Fine-tuning (VPT) method; the text encoder introduces class-specific learned text vectors through context optimization (CoOp) technology.
[0012] Furthermore, in step 2, the formula for generating the poisoning image with a trigger is:
[0013]
[0014] in, For the original image, The parameters that control the visibility of the trigger, This indicates element-wise multiplication.
[0015] Furthermore, in step 2, the total loss function used for joint optimization is:
[0016]
[0017] in, This indicates the loss compared to clean samples. Indicates that poison was introduced. Classified as target category The loss of the backdoor This is a hyperparameter used to balance task performance and watermark robustness.
[0018] Furthermore, the contrast loss of the clean sample The loss is a symmetrical contrast loss, comprising image-to-text loss and text-to-image loss; the backdoor loss The calculation method is the same as that for the contrast loss of the clean sample, with the input being the poisoned image x' and the label being the target category. .
[0019] Furthermore, in step 2, the jointly optimized training parameters include: a poisoning ratio of 0.1, training for 100 epochs using the stochastic gradient descent algorithm, a batch size of 128, and a learning rate of 0.04.
[0020] Furthermore, in step 3, the formula for calculating the posterior probability of the Bayesian inference framework is as follows:
[0021] ,
[0022] in, The representative model includes the piracy assumption of backdoor watermarking. Represents the assumption of innocence. This includes observational data that contains the total number of queries and the number of times the target category was predicted.
[0023] Furthermore, the observation data The likelihood follows a binomial distribution, under the innocence hypothesis. The probability of The probability of a random guess is 1 / C, where C is the total number of task categories; piracy assumption. The probability of It is close to 1.0.
[0024] A deep visual language prompting intellectual property protection system based on backdoor watermarking, comprising:
[0025] The model preparation module is used to obtain a pre-trained visual language model, introduce multiple sets of learnable deep visual cue parameters into each Transformer layer of the image encoder of the visual language model, and introduce text cue composed of learnable context vectors into the text encoder of the visual language model.
[0026] The watermark injection module is used to define a secret trigger. and target category The backdoor watermark; during the training phase, a contrastive loss function consisting of image-to-text loss and text-to-image loss is used, combined with the secret trigger. and target category The resulting poisoning prediction loss is used to form a weighted composite loss function; by minimizing the weighted composite loss function, the deep visual cue parameters and the text cue are jointly optimized, and the watermark information is embedded in the cross-modal interaction space;
[0027] The ownership verification module is used to select a clean image of a non-target category and overlay the secret trigger in a black-box environment where input and output interaction is only possible through an API. A poisoning query image is generated; the poisoning query image is input into the suspect model to be verified, and the statistical model outputs the target category. The number of times; based on the statistical results, the posterior probability that the suspected model is a pirated model is calculated using a Bayesian inference framework. If the posterior probability exceeds a preset high confidence threshold, ownership is confirmed.
[0028] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:
[0029] (1) Extremely high concealment and harmlessness: The watermarked hints maintain the same or even better accuracy as the unwatermarked version in downstream tasks (performance variation range is controlled within 1.5%), preserving the core commercial value of the hints.
[0030] (2) Strong watermark effectiveness: In tests covering 10 independent datasets including fine-grained classification and general target recognition, the watermark triggering success rate consistently exceeded 90%.
[0031] (3) Excellent anti-tampering robustness: The watermark information is deeply entangled with the model's cross-modal features. Experiments show that no matter whether the attacker uses up to 100 rounds of prompts for fine-tuning or adopts high-intensity parameter pruning from 10% to 90%, they cannot erase the intellectual property watermark without damaging the performance of downstream tasks.
[0032] (4) High confidence black-box verifiability: It is the first to create a strict black-box verification system based on the Bayesian inference framework, which can provide extremely accurate probabilistic mathematical evidence for ownership claims with only API access. Attached Figure Description
[0033] Figure 1 This is the overall framework and watermark verification flowchart of the present invention. It demonstrates how a secret watermark is injected into visual and textual contextual cues during training, and how ownership is verified subsequently through a poison query in a strictly black-box scenario. Detailed Implementation
[0034] The invention will now be explained in further detail with reference to the accompanying drawings and specific formulas. To achieve copyright protection for deep visual cue parameters, the execution process of this invention is mainly divided into three core steps: pre-establishment, joint watermark optimization training, and black-box inference verification. The specific steps are as follows:
[0035] Step 1, Pre-trained Model and Hint Parameter Construction: In this embodiment, the visual language model adopts the CLIP architecture (specifically, ViT-B / 16 can be used as the image encoder backbone network). To adapt to downstream visual classification tasks, we efficiently fine-tune its parameters. Specifically:
[0036] (1.1) Construction of Image-Based Deep Visual Cueing (VPT-Deep): Given a vector array with N Transformer layers visual encoder We add a set of p learnable prompt tokens to the front of the input sequence at each layer, denoted as p. Specifically, the forward propagation process of the i-th layer is defined as follows:
[0037]
[0038] in, Patch embeddings for images. This involves embedding class tokens from the previous layer. In practice, we can set [specific parameters] for each layer. A set of learnable visual tags. This is the set of all cues across these N layers. This constitutes the core intellectual property assets that we need to protect.
[0039] (1.2) Construction of text-based contextual cues (CoOp): In order to work in conjunction with visual cues, we improved the text encoder. Make synchronized adjustments. For specific categories. Instead of using manually designed fixed text descriptions, we construct a system based on... A learnable context vector The sequence is composed of word embeddings of category names. This forms a complete text prompt sequence. :
[0040]
[0041] Inputting this sequence into a text encoder will generate a category-specific output. Specific text features representation In practice, the number of context vectors can be set to... .
[0042] Step 2, Cross-modal Watermark Injection and Joint Optimization Training: The core objective is to implant a covert and robust ownership watermark while ensuring that the cue parameters perform well on the downstream main task. Specifically:
[0043] (2.1) Poisoning trigger and image conversion: Set a secret watermark trigger (For example, the size in the lower right corner of the image is...) (a checkerboard pattern), and specify a target category. The poisoning process involves applying the poison to the input image. Applying this trigger generates a poisoning image. Its formula is:
[0044]
[0045] in, To control the fusion parameters of trigger visibility, This represents element-wise multiplication. During training, we administer poison at a specific ratio (e.g., ...). To poison the data within a batch.
[0046] (2.2) Calculation of clean task contrast loss: For a task containing For a normal batch of image-text pairs, let x be the input image and y be the corresponding ground truth class label. The main task uses a symmetric contrastive loss function for optimization, including image-to-text loss. Text-to-image loss :
[0047] ,
[0048] The total clean task loss is
[0049]
[0050] in, Represents cosine similarity, This refers to temperature hyperparameters.
[0051] (2.3) Cross-modal backdoor joint optimization: Analogous to the clean task loss, we calculate the backdoor loss for images with triggers. Its goal is to force any poisoning input All can be classified into the preset target category. The final total loss function integrates these two parts by weighting, where x is the input image and y is the corresponding true class label:
[0052]
[0053] in, This is a hyperparameter that balances task performance and watermark strength (can be set to 1.0). In practice, the network is trained using stochastic gradient descent for 100 training epochs, with a batch size of 128 and a learning rate of 0.04.
[0054] This joint optimization mechanism is crucial, enabling visual cues to learn to generate specific anomaly features when encountering certain triggers, while text cues are simultaneously trained to be highly sensitive to these anomaly features and point to the target class. The watermark is thus deeply integrated into the cross-modal "lock and key" interaction between the two, making it difficult to separate.
[0055] Step 3, Bayesian Ownership Verification in a Strict Black-Box Scenario: In a "hint-as-a-service" scenario, the defender (i.e., the copyright owner) typically cannot obtain the specific parameter weights within the suspected model. When suspecting that a public API interface has misappropriated its visual cues, the defender uses a Bayesian inference framework to perform rigorous ownership verification. Specifically:
[0056] (3.1) Black box sampling and detection: selection by the defender Zhang does not belong to the target category. benign images By using the poisoning function from the first stage to overlay secret triggers, a poisoning query set is generated. Subsequently, these poisoning queries were submitted one by one to the API of the suspect model, and the returned results were statistically analyzed and classified as target categories. Number of times Thus, the observation dataset is obtained. .
[0057] (3.2) Establishing the Hypothesis and Likelihood Function: Establish two opposing hypotheses, assuming (Innocent and independent), the suspected model did not use the defender's pirated parameters. Its judgment for any image with triggers is... The probability is only a random guess. ( (Total number of task categories); assuming (Model theft) The suspected model copied the defender's parameters and contained a backdoor. It judged images with triggers as... The probability is extremely high ( ). Observational data The likelihood of follows a binomial distribution:
[0058] .
[0059] (3.3) Posterior Probability Calculation and Judgment: To maintain objectivity, assume the prior probability $P(H_1) = P(H_0) = 0.5$. Based on Bayes' theorem, the posterior probability that the suspected model is indeed a pirated model is calculated as follows:
[0060]
[0061] The decision rule is as follows: when the calculated posterior probability... When the probability exceeds a very high confidence threshold (e.g., 0.999), it can serve as strong mathematical evidence for proving ownership of intellectual property. In real-world testing, the probability calculated by a genuine piracy model typically approaches the limit (e.g., ...). (Level), capable of perfectly eliminating false alarms and providing irrefutable proof of infringement.
[0062] Corresponding to the above method embodiments, the present invention also provides a deep visual language prompt intellectual property protection system based on backdoor watermarking, the system including a model preparation module, a watermark injection module and an ownership verification module.
[0063] Model preparation module: used to obtain a pre-trained visual language model, introduce multiple sets of learnable deep visual cue parameters into each Transformer layer of the image encoder of the visual language model, and introduce text cue composed of learnable context vectors into the text encoder of the visual language model.
[0064] Watermark injection module: used to define a module containing a secret trigger. and target category The backdoor watermark; during the training phase, a contrastive loss function consisting of image-to-text loss and text-to-image loss is used, combined with the secret trigger. and target category The resulting poisoning prediction loss is used to form a weighted composite loss function; by minimizing the weighted composite loss function, the deep visual cue parameters and the text cue are jointly optimized to embed the watermark information into the cross-modal interaction space.
[0065] Ownership verification module: In a black-box environment where input and output interaction is limited to an API, it selects a clean image of a non-target category and overlays the secret trigger Δ to generate a poisoning query image; the poisoning query image is then input into the suspect model to be verified, and the statistical model outputs the target category. The number of times; based on the statistical results, the posterior probability that the suspected model is a pirated model is calculated using a Bayesian inference framework. If the posterior probability exceeds a preset high confidence threshold, ownership is confirmed.
[0066] For details on the implementation of each of the above modules, please refer to the description of the corresponding steps in the aforementioned method.
[0067] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for protecting intellectual property rights through deep visual language cues based on backdoor watermarking, characterized in that: Includes the following steps: Step 1: Obtain a pre-trained visual language model, introduce multiple sets of learnable deep visual cue parameters into each Transformer layer of the image encoder of the visual language model, and introduce text cue composed of learnable context vectors into the text encoder of the visual language model. Step 2, define a secret trigger. and target category The backdoor watermark; during the training phase, a contrastive loss function consisting of image-to-text loss and text-to-image loss is used, combined with the secret trigger. and target category The resulting poisoning prediction loss is used to form a weighted composite loss function; by minimizing the weighted composite loss function, the deep visual cue parameters and the text cue are jointly optimized, and the watermark information is embedded in the cross-modal interaction space; Step 3: In a black-box environment where input and output interaction is only possible through the API, select a clean image that is not of the target category and overlay the secret trigger. A poisoning query image is generated; the poisoning query image is input into the suspect model to be verified, and the statistical model outputs the target category. The number of times; based on the statistical results, the posterior probability that the suspected model is a pirated model is calculated using a Bayesian inference framework. If the posterior probability exceeds a preset high confidence threshold, ownership is confirmed.
2. The method according to claim 1, characterized in that: In step 1, the visual language model adopts the CLIP architecture.
3. The method according to claim 1 or 2, characterized in that: The image encoder is a visual Transformer, and the deep visual cues are added to the front end of the input sequence of each Transformer layer of ViT through a visual cue fine-tuning method. The text encoder introduces class-specific learned text vectors through context optimization techniques.
4. The method according to claim 1, characterized in that: In step 2, the formula for generating the poisoning image with a trigger is: in, For the original image, The parameters that control the visibility of the trigger, This indicates element-wise multiplication.
5. The method according to claim 1, characterized in that: In step 2, the total loss function used for joint optimization is: in, This indicates the loss compared to clean samples. Indicates that poison was introduced. Classified as target category The loss of the backdoor This is a hyperparameter used to balance task performance and watermark robustness.
6. The method according to claim 5, characterized in that: The contrast loss of the clean sample The loss is a symmetrical contrast loss, comprising image-to-text loss and text-to-image loss; the backdoor loss The calculation method is the same as that for the contrast loss of the clean sample, with the input being the poisoned image x' and the label being the target category. .
7. The method according to claim 1, characterized in that: In step 2, the jointly optimized training parameters include: a poisoning ratio of 0.1, training for 100 epochs using the stochastic gradient descent algorithm, a batch size of 128, and a learning rate of 0.
04.
8. The method according to claim 1, characterized in that: In step 3, the formula for calculating the posterior probability of the Bayesian inference framework is as follows: ， in, The representative model includes the piracy assumption of backdoor watermarking. Represents the assumption of innocence. This includes observational data that contains the total number of queries and the number of times the target category was predicted.
9. The method according to claim 1, characterized in that: The observation data The likelihood of follows a binomial distribution, under the innocence hypothesis. The probability of The probability of a random guess is 1 / C, where C is the total number of task categories; piracy assumption. The probability of It is close to 1.
0.
10. A deep visual language prompting intellectual property protection system based on backdoor watermarking, characterized in that: include: The model preparation module is used to obtain a pre-trained visual language model, introduce multiple sets of learnable deep visual cue parameters into each Transformer layer of the image encoder of the visual language model, and introduce text cue composed of learnable context vectors into the text encoder of the visual language model. The watermark injection module is used to define a secret trigger. and target category The backdoor watermark; during the training phase, a contrastive loss function consisting of image-to-text loss and text-to-image loss is used, combined with the secret trigger. and target category The resulting poisoning prediction loss is used to form a weighted composite loss function; by minimizing the weighted composite loss function, the deep visual cue parameters and the text cue are jointly optimized, and the watermark information is embedded in the cross-modal interaction space; The ownership verification module is used to select a clean image of a non-target category and overlay the secret trigger in a black-box environment where input and output interaction is only possible through an API. A poisoning query image is generated; the poisoning query image is input into the suspect model to be verified, and the statistical model outputs the target category. The number of times; based on the statistical results, the posterior probability that the suspected model is a pirated model is calculated using a Bayesian inference framework. If the posterior probability exceeds a preset high confidence threshold, ownership is confirmed.