A visual language model copyright protection method and device based on a dynamic fingerprint generator

Abstract

This invention discloses a method and apparatus for copyright protection of visual language models based on a dynamic fingerprint generator. Addressing the problem that existing visual language models lack lightweight and robust copyright attribution identification mechanisms, leading to difficulties in tracking model misuse and low verification reliability, this invention constructs a training set containing natural images and text triggers, and pre-sets fingerprint responses. The natural images are encoded into fingerprint images using a fingerprint generator, and these fingerprint images are input into the visual language model in conjunction with the text triggers. The difference between the output and the fingerprint response is measured using cross-entropy loss, and the fingerprint generator is optimized through backpropagation. After training, the fingerprint generator converts the input images into fingerprint images, which are then input into the visual language model in conjunction with the text triggers. Copyright attribution is determined based on the output response. The verification performance is quantitatively evaluated. This invention demonstrates significant improvements in copyright recognition accuracy and robustness against disturbances, providing effective support for copyright protection of visual language models.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence security and digital copyright protection technology, specifically relating to a method and apparatus for copyright protection of visual language models based on a dynamic fingerprint generator. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, visual language models (VLMs) have been increasingly widely used in the field of computer vision. These models typically rely on massive amounts of training data and powerful computing resources, and have high technical barriers and commercial value. Therefore, how to effectively protect the intellectual property rights of visual language models and prevent them from being used or misappropriated without authorization has become one of the key issues of concern.

[0003] Currently, copyright protection technologies for machine learning models mainly cover methods such as digital watermarking and model fingerprinting. Specifically, model watermarking technology embeds specific identifier information into model parameters or output behavior during training, enabling the model to produce a preset output in response to specific inputs, thereby providing a basis for ownership verification. However, existing technical solutions still have the following limitations:

[0004] 1. Highly invasive: Existing methods usually require embedding specific watermark or fingerprint information during the model training stage, which cannot guarantee that the performance of the trained model is consistent with that of the original model;

[0005] 2. Poor robustness: When faced with routine operations such as model compression, fine-tuning, or transfer learning, the existing watermark is easily damaged or becomes invalid.

[0006] 3. Limitations in generalization: Some technical solutions are highly dependent on the distribution characteristics of the input data or the model architecture, making it difficult to adapt to the diverse application scenarios of visual language models.

[0007] In recent years, adversarial example-based model verification technology has received widespread attention as a non-intrusive copyright authentication method. This technology stimulates pre-defined behavioral features of the model by designing specific adversarial inputs, thereby achieving model identity recognition and ownership verification. However, this type of method faces several key technical obstacles: in terms of sample coverage, the number of effective trigger samples is limited by the algorithm design, and each trigger sample needs to be repeatedly optimized and generated, making it difficult to support large-scale verification needs; in terms of model adaptability, existing methods are significantly less robust to weight modifications made by model thieves to stolen models. To address these technical challenges, there is an urgent need to develop an innovative copyright protection scheme for visual language models. This scheme should ensure that the copyright verification process has efficient large-scale fingerprint generation capabilities, stable adaptability to same-source model verification, and complete performance preservation without modifying the original model structure, thereby comprehensively improving the practical value of model intellectual property protection. Summary of the Invention

[0008] To address the above problems, this invention proposes a copyright protection method for visual language models based on a dynamic fingerprint generator, which includes the following steps:

[0009] S1: Construct a training set containing natural images and text triggers, and pre-define fingerprint responses;

[0010] S2: Convert natural images into fingerprint images using a fingerprint generator. Input the fingerprint images and text triggers into the visual language model to output a response. Use the cross-entropy loss function to calculate the difference between the output response and the fingerprint response, and perform backpropagation to train the fingerprint generator.

[0011] S3: The trained fingerprint generator generates fingerprint images from the input natural images, combines the fingerprint images with text triggers and inputs them into the visual language model, and performs copyright verification on the visual language model based on its output.

[0012] S4: The effectiveness of copyright verification is fed back by calculating the true positive rate (TPR) and the false positive rate (FPR).

[0013] In step S1, a training set containing natural images and text triggers is constructed, and fingerprint responses are preset;

[0014] Specifically, natural images include images obtained from publicly available datasets, covering different types of subject categories, and possessing natural visual authenticity; they conform to the general visual perception of the physical world and will not raise questions about their authenticity or rationality among viewers;

[0015] Meanwhile, unconventional text sequences in natural language are pre-set as text triggers; these text sequences appear to humans as meaningless or unremarkable combinations, but for the visual language model, they can activate pre-set responses that differ from normal behavioral patterns.

[0016] In step S2, the natural image is converted into a fingerprint image by the fingerprint generator. The fingerprint image and the text trigger are input into the visual language model to output the response. The difference between the output response and the fingerprint response is calculated using the cross-entropy loss function, and backpropagation is performed to train the fingerprint generator.

[0017] Specifically, a fingerprint image is generated from a natural image using a fingerprint generator. Then, the fingerprint image and a text trigger are used as input to a visual language model to obtain the output response.

[0018] A fingerprint generator is a generative model that takes an image as input and outputs an image. It can convert any input natural image into a completely new fingerprint image. This fingerprint image has visual naturalness, making it difficult for attackers to identify the fingerprint information through visual detection.

[0019] Furthermore, the cross-entropy loss function between the output response and the fingerprint response is calculated, and expressed as:

[0020]

[0021] Where M train For the visual language model under training, t trigger This is a text trigger that activates when the input is a fingerprint image x. fingerprint At that time, the model output M train (x fingerprint ,t trigger It should match the preset fingerprint response y target Consistent, where i is the index of the output token;

[0022] Furthermore, to ensure that the fingerprint image can be effectively input into the visual language model along with the text trigger to generate a preset response, the parameters of the fingerprint generator are updated through gradient descent to make the generated response approximate the preset response:

[0023]

[0024] Where θ G Here are the fingerprint generator parameters, and η is the learning rate;

[0025] Furthermore, to enhance fingerprint robustness and prevent adversarial examples generated by the fingerprint generator from strictly conforming to the decision boundaries of the visual language model, ensuring that the fingerprint generator fails once the visual language model undergoes even minor modifications by the thief, the parameters θ of the visual language model are adjusted accordingly.M The update proceeds in the direction of increasing loss, and gradient ascent organizes the fingerprint generator to generate responses during the training phase, simulating the worst-case adversarial modification to the greatest extent possible.

[0026] Meanwhile, to ensure the fingerprint generator weights converge during training, the visual language model parameters and fingerprint generator parameters are updated every other step. To prevent gradient explosion, gradient clipping is used during training.

[0027]

[0028] Where α is the adversarial learning rate (usually α << η);

[0029] During training, the update operation of the visual language model is limited to training its associated fingerprint generator. After the fingerprint generator is trained, the parameters of the visual language model are reset to the state before the update for subsequent processing steps.

[0030] The training objective of the fingerprint generator is to enable the generated fingerprint image, when input into the visual language model along with a preset text trigger, to trigger a preset fingerprint response, without affecting the visual language model's normal response to ordinary images and text.

[0031] In step S3, the trained fingerprint generator generates a fingerprint image from the input natural image, combines the fingerprint image with the text trigger and inputs it into the visual language model, and performs copyright verification on the visual language model based on its output.

[0032] Specifically, the trained fingerprint generator generates a fingerprint image from the input natural image, represented as:

[0033] x fingerprint =G(x)

[0034] Where x fingerprint The generated fingerprint image is G, and the input image is x.

[0035] The fingerprint image and text trigger are input into the visual language model to be verified to obtain a response;

[0036] If the output is a conventional response that conforms to the semantic content of the image, then the visual language model is determined not to be the target visual language model. If the output is a preset fingerprint response that is unrelated to the semantics of the image, then the visual language model is determined to be the target visual language model, as shown below:

[0037] y = M origin (x fingerprint ,t trigger )

[0038]

[0039] Where M origin It is the visual language model restored to its pre-update parameters in step S2, Detect(M) origin This indicates whether the visual language model is the target visual language model.

[0040] In step S4, the copyright verification effect is fed back by calculating the true positive rate (TPR) and the false positive rate (FPR).

[0041] Specifically, several natural images are arbitrarily input, and a trained fingerprint generator generates corresponding fingerprint images, which are then input into the visual language model M along with a text trigger. origin If the model's output response matches the preset response, fingerprint recognition is considered successful; otherwise, it is considered a failure, and the fingerprint image x is tested. test The probability of a fingerprint being successfully identified is expressed as:

[0042]

[0043] Where I(·) is an indicator function that outputs 1 when a match is successful and 0 otherwise;

[0044] Furthermore, several natural images are selected and, without modification by the fingerprint generator, are input into the visual language model M along with the text trigger. origin Natural Images x normal The probability of being misidentified as a fingerprint is expressed as:

[0045]

[0046] The effectiveness of the fingerprint generation method is verified by validating the true positive rate (TPR) and false trigger rate (FPR). A higher TPR indicates that the method is better at successfully identifying the unknown model as the target model, while a lower FPR indicates that the method is less likely to misclassify the unknown model as the target model. Since the fingerprint generator and the target model are strongly correlated during training, the generated fingerprint image and preset text can only trigger the preset response in the target model.

[0047] This invention also provides a visual language model copyright protection device based on a dynamic fingerprint generator, comprising the following modules:

[0048] Dataset building module: Used to construct a training set containing natural images and text triggers, and to pre-define fingerprint responses;

[0049] Model learning module: This module converts natural images into fingerprint images. It combines text triggers with a visual language model, uses the cross-entropy loss function to calculate the difference between the output response and the fingerprint response, and trains the fingerprint generator through backpropagation.

[0050] Model copyright verification module: The trained fingerprint generator generates fingerprint images from the input natural images, combines them with text triggers to input the visual language model, and performs copyright verification on the visual language model based on its output.

[0051] Effect Feedback Module: Used to calculate the True Positive Rate (TPR) and False Positive Rate (FPR) to provide feedback on the copyright verification effect.

[0052] The beneficial effects of this invention are:

[0053] 1. Low-cost unlimited fingerprint generation: The fingerprint generator dynamically generates fingerprint images, theoretically capable of generating an unlimited number of fingerprints instantly, saving time in optimizing fingerprint triggers and significantly improving the security and flexibility of copyright protection.

[0054] 2. Preserve model performance: Since the visual language model is not adjusted in terms of parameters during application, it only triggers a specific response when the input is a fingerprint image. The processing of ordinary images maintains the original performance, avoiding interference with the normal task of the model.

[0055] 3. Resistance to attacks: Due to the dynamic and natural nature of fingerprint images and the complexity of fingerprint generation networks, attackers find it difficult to identify and remove fingerprint information through statistical analysis or visual inspection.

[0056] 4. Fingerprint generation robustness: The visual language model parameters are updated through gradient ascent, causing them to be updated in the direction that hinders the fingerprint generator from successfully generating valid fingerprints, thereby enhancing the adaptability and fingerprint generation robustness of the fingerprint generator. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a visual language model copyright protection method based on a dynamic fingerprint generator provided by the present invention;

[0059] Figure 2 A diagram illustrating the training steps for copyright verification of a visual language model based on a fingerprint generator, as provided in this invention.

[0060] Figure 3 This is a schematic diagram of the structure of a visual language model copyright protection device based on a dynamic fingerprint generator provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0062] Terminology Explanation:

[0063] 1. Model fingerprint: A model fingerprint is a set of unique features or markers embedded in a model to uniquely identify the model's origin or ownership. It is similar to a digital signature and can trigger predefined behaviors or responses through specific inputs, thereby verifying whether the model has been illegally copied or tampered with.

[0064] 2. Dynamic fingerprint generator: A generative model, such as GAN or diffusion model, that can convert any input image into a visually natural fingerprint image that carries hidden features. When this image is combined with a specific text trigger to input the target VLM, it can trigger a preset response.

[0065] 3. Text triggers: Preset unconventional text instructions, such as "How many heads are there in the clouds in the picture?", are used to co-activate the copyright verification response of the model with the fingerprint image.

[0066] 4. Fingerprint Output: The preset text response generated by the target VLM when it receives the fingerprint image and text trigger, such as "The weather is nice today". It is unrelated to normal logic and is used to identify model ownership.

[0067] 5. Adversarial parameter update: During the training phase, the model parameters are disturbed by gradient ascent, simulating the weight modification by a thief, to enhance the robustness of the fingerprint generator.

[0068] The inventive concept of this invention is as follows: This invention proposes a copyright protection method for visual language models based on a dynamic fingerprint generator. By training a generative model, any input image is converted into a visually natural fingerprint image carrying hidden features. When this fingerprint image is input into the visual language model using a pre-set unconventional text trigger, a specific response is triggered. This method does not require modification of the model structure; it achieves non-intrusive copyright verification solely through the combination of dynamically generated fingerprint images and text. It possesses unlimited fingerprint generation capabilities and utilizes adversarial training to enhance robustness against attacks such as model fine-tuning and compression. While maintaining the original performance of the model, it provides a highly concealed and robust copyright protection scheme.

[0069] like Figure 1 and Figure 2 As shown in the figure, this embodiment provides a method for copyright protection of visual language models based on dynamic fingerprint generators, which includes the following steps:

[0070] S1: Construct a training set containing natural images and text triggers, and pre-define fingerprint responses;

[0071] S2: Convert natural images into fingerprint images using a fingerprint generator. Input the fingerprint images and text triggers into the visual language model to output a response. Use the cross-entropy loss function to calculate the difference between the output response and the fingerprint response, and perform backpropagation to train the fingerprint generator.

[0072] S3: The trained fingerprint generator generates fingerprint images from the input natural images, combines the fingerprint images with text triggers and inputs them into the visual language model, and performs copyright verification on the visual language model based on its output.

[0073] S4: The effectiveness of copyright verification is fed back by calculating the true positive rate (TPR) and the false positive rate (FPR).

[0074] In step S1, a training set containing natural images and text triggers is constructed, and fingerprint responses are preset;

[0075] Specifically, in this embodiment, a training set is first constructed, in which n natural images from a public dataset are selected. In this embodiment, n is 2000. The public dataset includes, but is not limited to, ImageNet.

[0076] Each natural image sample is denoted as x. normal Select any pre-defined, uncommon natural language question as the text trigger t trigger Construct sample pairs as (natural image x, text trigger t) trigger ), preset a fingerprint response y target Text trigger t trigger The option is "What are the differences between animals?", fingerprint response y target The option is "Copyright Information";

[0077] The fingerprint generator uses a U-Net structured generative network with parameters θ. G The visual language model selected is an open-source visual language model, including but not limited to BLIP-2, OpenFlamingo, or LLaVA1.5, with parameter θ. M .

[0078] In step S2, the natural image is converted into a fingerprint image by the fingerprint generator. The fingerprint image and the text trigger are input into the visual language model to output the response. The difference between the output response and the fingerprint response is calculated using the cross-entropy loss function, and backpropagation is performed to train the fingerprint generator.

[0079] Specifically, a fingerprint image is generated from a natural image using a fingerprint generator. Then, the fingerprint image and a text trigger are used as input to a visual language model to obtain the output response.

[0080] A fingerprint generator is a generative model that takes an image as input and outputs an image. It can convert any natural input image into a completely new fingerprint image. This fingerprint image has visual naturalness, making it difficult for attackers to identify the fingerprint information through visual detection.

[0081] Furthermore, the cross-entropy loss function between the output response and the fingerprint response is calculated, and expressed as:

[0082]

[0083] Where M train For the visual language model under training, t trigger This is a text trigger that activates when the input is a fingerprint image x. fingerprint At that time, the model output M train (x fingerprint ,t trigger It should match the preset fingerprint response y target Consistent, where i is the index of the output token;

[0084] Furthermore, to ensure that the fingerprint image can be effectively input into the visual language model along with the text trigger to generate a preset response, the parameters of the fingerprint generator are updated through gradient descent to make the generated response approximate the preset response:

[0085]

[0086] Where θ G Here are the fingerprint generator parameters, and η is the learning rate. In this embodiment, η = 10. -4 ;

[0087] Furthermore, to enhance fingerprint robustness and prevent adversarial examples generated by the fingerprint generator from strictly conforming to the decision boundaries of the visual language model, ensuring that the fingerprint generator fails once the visual language model undergoes even minor modifications by the thief, the parameters θ of the visual language model are adjusted accordingly. M The update proceeds in the direction of increasing loss, and gradient ascent organizes the fingerprint generator to generate responses during the training phase, simulating the worst-case adversarial modification to the greatest extent possible.

[0088] Meanwhile, to ensure the convergence of the fingerprint generator weights during training, the visual language model parameters and fingerprint generator parameters are updated alternately. The fingerprint generator parameters are updated every five epochs, while the visual language model parameters are updated once. The number of training epochs is set to 100. To prevent gradient explosion, gradient clipping is used during training.

[0089]

[0090] Where α is the adversarial learning rate, typically α << η, and in this embodiment α = 10.-6 ;

[0091] During training, the update operation of the visual language model is limited to training its associated fingerprint generator. After the fingerprint generator is trained, the parameters of the visual language model are reset to the state before the update for subsequent processing steps.

[0092] The training objective of the fingerprint generator is to enable the generated fingerprint image, when input into the visual language model along with a preset text trigger, to trigger a preset fingerprint response, without affecting the visual language model's normal response to ordinary images and text.

[0093] In step S3, the trained fingerprint generator generates a fingerprint image from the input natural image, combines the fingerprint image with the text trigger and inputs it into the visual language model, and performs copyright verification on the visual language model based on its output.

[0094] Specifically, the trained fingerprint generator uses the input image to generate a fingerprint trigger, represented as follows:

[0095] x fingerprint =G(x)

[0096] Where x fingerprint This is the generated fingerprint trigger, with G as the fingerprint generator and x as the input image;

[0097] The fingerprint trigger and text trigger are input into the visual language model to be verified to obtain a response;

[0098] If the output is a conventional response that conforms to the semantic content of the image, then the visual language model is determined not to be the target visual language model. If the output is a preset fingerprint response that is unrelated to the semantics of the image, then the visual language model is determined to be the target visual language model, as shown below:

[0099] y = M origin (x fingerprint ,t trigger )

[0100]

[0101] Where M origin It is the visual language model restored to its pre-update parameters in step S2, Detect(M) origin This indicates whether the visual language model is the target visual language model.

[0102] In step S4, the copyright verification effect is fed back by calculating the true positive rate (TPR) and the false positive rate (FPR).

[0103] Specifically, several images are input, and a trained fingerprint generator generates corresponding fingerprint images, which are then input into the visual language model M along with a text trigger. origin If the model's output response matches the preset response, fingerprint recognition is considered successful; otherwise, it is considered a failure, and the fingerprint image x is tested. test The probability of a fingerprint being successfully identified is expressed as:

[0104]

[0105] Where I(·) is an indicator function that outputs 1 when a match is successful and 0 otherwise;

[0106] Furthermore, several images are input, and without modification by the fingerprint generator, they are fed into the visual language model M along with a text trigger. origin Natural Images x normal The probability of being misidentified as a fingerprint is expressed as:

[0107]

[0108] The effectiveness of the fingerprint generation method is verified by validating the true positive rate (TPR) and the false positive rate (FPR). A higher TPR indicates that the method is better at successfully identifying unknown models as target models, while a lower FPR indicates that the method is less likely to misidentify unknown models as target models.

[0109] like Figure 3 As shown, this embodiment also provides a visual language model copyright protection device based on a dynamic fingerprint generator, including the following modules:

[0110] Dataset building module: Used to construct a training set containing natural images and text triggers, and to pre-define fingerprint responses;

[0111] Model learning module: This module converts natural images into fingerprint images. It combines text triggers with a visual language model, uses the cross-entropy loss function to calculate the difference between the output response and the fingerprint response, and trains the fingerprint generator through backpropagation.

[0112] Model copyright verification module: The trained fingerprint generator generates fingerprint images from the input natural images, combines them with text triggers to input the visual language model, and performs copyright verification on the visual language model based on its output.

[0113] Effect Feedback Module: Used to calculate the True Positive Rate (TPR) and False Positive Rate (FPR) to provide feedback on the copyright verification effect.

[0114] The visual language model copyright protection device based on a dynamic fingerprint generator proposed in this invention corresponds one-to-one with the aforementioned visual language model copyright protection method based on a dynamic fingerprint generator in terms of operation process and functional effect. Therefore, the structure and operation process of the visual language model copyright protection device based on a dynamic fingerprint generator will not be described again.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can be implemented in the form of a computer program product on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A copyright protection method for visual language models based on a dynamic fingerprint generator, characterized in that, Includes the following steps: S1: Construct a training set containing natural images and text triggers, and pre-define fingerprint responses; S2: The natural image is converted into a fingerprint image by a fingerprint generator. The fingerprint image and the text trigger are input into a visual language model to output a response. The difference between the output response and the fingerprint response is calculated using the cross-entropy loss function, and backpropagation is performed to train the fingerprint generator. S3: The trained fingerprint generator generates a fingerprint image from the input natural image, combines the fingerprint image with the text trigger, inputs it into the visual language model, and performs copyright verification on the visual language model based on its output. S4: The effectiveness of copyright verification is fed back by calculating the true positive rate (TPR) and the false positive rate (FPR).

2. The method for copyright protection of visual language models based on a dynamic fingerprint generator according to claim 1, characterized in that, Step S1 includes: The natural images include images obtained from publicly available datasets, covering different types of subject categories, and possessing natural realism at the visual level; The text trigger includes a pre-defined sequence of unconventional text in natural language.

3. The method for copyright protection of visual language models based on a dynamic fingerprint generator according to claim 1, characterized in that, Step S2 includes: The fingerprint generator is a generative model that takes an image as input and outputs an image, and can convert any input natural image into a completely new fingerprint image; The fingerprint image has a natural visual appearance, making it difficult for attackers to identify the fingerprint information through visual detection. The fingerprint image and the text trigger are input into the visual language model, and a response is output. The cross-entropy loss function between the output response and the fingerprint response is calculated, and expressed as: Where M train For the visual language model under training, t trigger This is a text trigger that activates when the input is a fingerprint image x. fingerprint At that time, the model output M train (x fingerprint ,t trigger It should match the preset fingerprint response y target Consistent, where i is the index of the output token; The parameters of the fingerprint generator are updated using gradient descent to make the generated response approximate the preset response. Where θ G Here are the parameters for the fingerprint generator, and η is the learning rate. The parameters of the visual language model are updated via gradient ascent. The visual language model parameters and fingerprint generator parameters are updated every other step. To prevent gradient explosion, gradient clipping is used during training, as shown below: Where α is the adversarial learning rate, and usually α << η; During training, the update operation of the visual language model is limited to training its associated fingerprint generator. After the fingerprint generator is trained, the parameters of the visual language model are reset to the state before the update for subsequent processing steps.

4. The method for copyright protection of visual language models based on a dynamic fingerprint generator according to claim 1, characterized in that, Step S3 includes: The trained fingerprint generator generates a fingerprint image from the input natural image, represented as follows: x fingerprint =G(x) Where x fingerprint The generated fingerprint image is G, and the input image is x. The fingerprint image and the text trigger are input into the visual language model to be verified to obtain a response.

5. The method for copyright protection of visual language models based on a dynamic fingerprint generator according to claim 4, characterized in that: If the output is a conventional response that conforms to the semantic content of the image, then the visual language model is determined not to be the target visual language model. If the output is a preset fingerprint response that is unrelated to the semantics of the image, then the visual language model is determined to be the target visual language model, as shown below: y=M origin (x fingerprint ,t trigger ) Where M origin It is the visual language model restored to its pre-update parameters in step S2, Detect(M) origin This indicates whether the visual language model is the target visual language model.

6. The method for copyright protection of visual language models based on a dynamic fingerprint generator according to claim 1, characterized in that, Step S4 includes: Input several natural images, generate corresponding fingerprint images through a trained fingerprint generator, and input them along with a text trigger into the visual language model M. origin If the model's output response matches the preset response, fingerprint recognition is considered successful; otherwise, it is considered a failure. The test input image x is then used. test The probability of a fingerprint being successfully identified is expressed as: Where I(·) is an indicator function that outputs 1 when a match is successful and 0 otherwise; Input several natural images, without modifying them through a fingerprint generator, along with a text trigger, into the visual language model M. origin Natural Images x normal The probability of being misidentified as a fingerprint is expressed as:

7. A copyright protection device for visual language models based on a dynamic fingerprint generator, characterized in that, Includes the following modules: Dataset construction module: used to construct a training set containing the natural image and the text trigger, and to preset the fingerprint response; Model learning module: used to convert the natural image into the fingerprint image, combine the text trigger input to the visual language model, use the cross-entropy loss function to calculate the difference between the output response and the fingerprint response, and backpropagate to train the fingerprint generator; Model copyright verification module: The fingerprint generator, after training, generates a fingerprint image from the input natural image, combines it with the text trigger to input the visual language model, and performs copyright verification on the visual language model based on its output. Effect Feedback Module: Used to calculate the True Positive Rate (TPR) and False Positive Rate (FPR) to provide feedback on the copyright verification effect.

Images