A large visual language model fingerprint generation method and device based on multi-perturbation space joint optimization
By constructing image-text pairs and superimposing adversarial perturbations, and combining shadow models to simulate parameter drift, fingerprint triggers are generated in an optimized manner. This solves the robustness problem of collaborative perturbation between image and text input in visual language models, and achieves stable black-box fingerprint verification and copyright tracing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU JUNTONG FUTURE TECHNOLOGY CO LTD
- Filing Date
- 2025-07-09
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies lack robust design for co-perturbations of image and text input in visual language models, making it difficult to achieve stable black-box fingerprint verification after deployment. Furthermore, existing methods have poor adaptability and limited scalability, failing to effectively resist perturbations such as image blurring and text replacement, and exhibiting poor adaptability to parameter drift.
By constructing original image-text pairs and superimposing adversarial perturbations, fingerprint problem pairs are generated. Combined with shadow model simulation of parameter drift, fingerprint triggers are optimized and generated. A multi-perturbation space joint optimization strategy is adopted to ensure that the fingerprint triggers remain stably activated under image, text and model parameter perturbations.
It achieves accurate fingerprint activation and verification under black-box conditions without accessing model parameters, with high trigger rate and high robustness, supports copyright tracing needs in practical use, and is suitable for ownership verification of multimodal large models.
Smart Images
Figure CN122336045A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of privacy protection of large visual language models for artificial intelligence security and privacy, specifically involving a method and device for generating fingerprints of large visual language models based on multi-perturbation space joint optimization. Background Technology
[0002] In recent years, Large Vision-Language Models (LVLMs) have achieved remarkable results in tasks such as image captioning, visual question answering, and multimodal reasoning. Thanks to their powerful cross-modal alignment capabilities between vision and language, LVLMs can quickly adapt to new task scenarios through fine-tuning without requiring full retraining, and therefore have been widely used in academic research and industrial practice.
[0003] However, with the widespread availability and application of pre-trained LVLMs, concerns about model copyright protection and misuse are increasing. In some cases, the original model may be leaked or tampered with by third parties, or it may be fine-tuned for commercial purposes without authorization, thus violating its usage agreement. This poses a significant challenge to the original model developers in tracking model distribution and identifying the ownership of derivative models. Therefore, there is an urgent need to develop a technical means to verify model ownership after deployment, namely, a black-box verification technology that performs "fingerprint recognition" on LVLMs.
[0004] Current research on fingerprint recognition technology largely focuses on large, single-modal language models (LLMs). These methods are mainly divided into two categories: white-box methods, such as extracting fingerprint information based on model parameters or intermediate representations; and black-box methods, such as injecting backdoor triggers into the model for behavior recognition. However, in practical commercial scenarios, it is often difficult to obtain model weights, making white-box methods impractical; while most existing black-box solutions rely on intrusive operations during the training phase, which are unsuitable for tracing and verifying deployed models. Therefore, designing a non-intrusive black-box fingerprint mechanism that does not require access to model parameters and can be implemented through queries after deployment has become a key research issue.
[0005] Existing research, such as ProFlingo and TRAP, has explored black-box attribution verification schemes based on adversarial triggers in the field of language models, initially verifying the possibility of achieving model tracking without modifying the model structure. However, these methods are only applicable to unimodal environments with text input and are difficult to directly extend to visual language model scenarios.
[0006] Unlike language models that only process text, LVLMs process both image and text inputs simultaneously. Their input space is multimodal and highly interwoven, significantly increasing the complexity of fingerprint trigger design. A few attempts have been made, such as the PLA method, which borrows ideas from ProFlingo to construct a hybrid image-text trigger and aligns adversarial features with the model decision boundary. However, this method primarily focuses on the robustness of fingerprints under parameter perturbations, and its tolerance to perturbations from both image and text inputs remains insufficient.
[0007] To address the aforementioned issues, existing technologies still have shortcomings in the following aspects:
[0008] 1. Poor adaptability: Most fingerprinting methods do not consider the co-perturbation characteristics of image and text input in LVLM, making fingerprint triggers susceptible to common perturbations such as image blurring and text replacement.
[0009] 2. Limited scalability: Most existing methods are limited to the field of language models and lack systematic design for the input structure of visual language models.
[0010] 3. Limited verification effectiveness: There is a lack of a unified fingerprint optimization mechanism that can stably verify data under perturbations in image space, text space, and model parameter space simultaneously.
[0011] In summary, traditional triggers are highly dependent on specific inputs and lack robustness against perturbations: they are mostly constructed based on static question-answer pairs, lack alignment with the model's stable semantic pathways, and are easily affected by text rewriting or image noise, leading to failure. Furthermore, they have poor parameter drift adaptability: triggers typically rely on the fixed parameters of the original model; once the model undergoes slight fine-tuning, the trigger response probability drops sharply, resulting in verification failure. Therefore, there is an urgent need to propose a non-intrusive black-box fingerprinting method for LVLMs with robustness against multiple perturbations to achieve reliable verification and effective protection of model copyright ownership. Summary of the Invention
[0012] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and device for generating fingerprints of large visual language models based on joint optimization of multiple perturbation spaces.
[0013] The objective of this technology is achieved through the following technical solution:
[0014] A fingerprint generation method for large visual language models based on joint optimization of multiple perturbation spaces includes the following steps:
[0015] S1: Construct the original image-text pair, and superimpose the initial adversarial perturbation on the original image to generate fingerprint problem pairs;
[0016] S2: Optimize the fingerprint problem pair to generate a fingerprint trigger;
[0017] S3: Construct a shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model;
[0018] S4: Combine the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples to construct a unified optimization objective, guide the optimization of the fingerprint trigger, and generate the final fingerprint trigger.
[0019] Specifically, in step S1, unrelated original images and original text are selected to form original image-text pairs (v,t), and the original images are ensured to be in the unmodified original model F and its variant models. The expected fingerprint response will not be triggered naturally. Construct a pair of original image-text pairs, variant model. It is a derived model obtained by fine-tuning the original model;
[0020] "Not naturally triggering the expected fingerprint response" means that the original image does not naturally trigger a relevant answer in terms of semantic understanding.
[0021] An adversarial perturbation is superimposed on the original image to generate fingerprint pairs.
[0022] Specifically, in step S2, the fingerprint problem is optimized to generate a fingerprint trigger. Specifically, the text perturbation part is optimized through a dual strategy of text semantic rotation and embedding perturbation to construct multiple text variants (t) under the same semantic intent. 1 ,t 2 ,...,t n During the fingerprint optimization process, different text variants and fingerprint images are dynamically selected. Correspondingly, optimizations are made to represent the input text E in the embedding space. t By injecting subtle perturbations to simulate lightweight attacks such as character misalignment and reordering, the fingerprint system's tolerance to natural language perturbations is improved. Here, text variants express the same meaning but are expressed differently. By overlaying adversarial perturbations onto these text variants, they are represented as follows:
[0023]
[0024] Where E t The embedding representation of the current fingerprint query t is shown. The dynamic query rotation and embedding perturbation strategy work together to ensure that the fingerprint trigger remains effective under diverse or slightly corrupted text inputs, avoiding over-reliance on any single representation form; α t is the step size (learning rate), which controls the amount of perturbation added to the text in each iteration; sgn(·) is the sign function, which is used here to take the reverse of the gradient; It involves calculating the gradient using text encoding to obtain the gradient direction; It is the loss function of the original model, measuring the difference between the model output and the target fingerprint response y that the model expects to trigger. * The gap.
[0025] Furthermore, the perturbed image is optimized using the projective gradient descent method, ensuring that the final generated fingerprint image stably triggers the preset output under multiple noisy samples, as shown below:
[0026]
[0027] in It is the fingerprint image at the g-th iteration; It is the new image sample obtained in the (g+1)th iteration, that is, the fingerprint input that can eventually stably trigger the model's fixed output; y * It is the target fingerprint response that the model expects to trigger; It is the prediction function of the original image-text pair by a multimodal model with parameter θ; It is the fingerprint loss function of the original model, which measures the difference between the model output and the target fingerprint response y that the model expects to trigger. * The gap; This involves calculating the gradient of the image v to obtain its direction, indicating how quickly modifying pixels can reduce the loss. `sgn(·)` is the sign function, used here to find the inverse direction of the gradient. α v It is the step size, or learning rate, which controls the amount of perturbation added to the image in each iteration; Π v+S It is a projection operator that ensures that the perturbation always satisfies the constraints; S is the feasible region of the perturbation.
[0028] Furthermore, in each round of optimization, by providing the current fingerprint image Multiple independent Gaussian noises are added, and these Gaussian noise perturbations satisfy the following:
[0029]
[0030] Generate a series of noisy fingerprint images, represented as follows:
[0031]
[0032] Subsequently, each noisy fingerprint image is paired with the current fingerprint text to evaluate the consistency of its activation target response. The visual domain noise loss is calculated using the average fingerprint loss function over all noisy samples, and is expressed as:
[0033]
[0034] Where M represents the number of fingerprint image samples with added noise; v m This refers to fingerprint images with added noise;
[0035] Specifically, in step S3, the shadow model is constructed. In each round of optimization, supervised samples are sampled from the natural dataset, and single-step gradient update simulation fine-tuning is performed. Then, the fingerprint response loss is calculated on the shadow model to ensure that the fingerprint can still activate the target output path in the model after slight parameter perturbations. Initially, the parameters of the shadow model are completely identical to those of the original model.
[0036] Subsequently, in each fingerprint noise optimization step, the shadow model is updated with supervision from real data to simulate parameter drift.
[0037] Furthermore, a batch of natural samples is randomly sampled from the publicly available general dataset of large visual language models, denoted as (v', t', y), where v' represents a single image, t' represents a single text, and y represents a preset response;
[0038] The cross-entropy loss is calculated based on the current parameters, and the shadow model is then updated iteratively using a single gradient descent step.
[0039]
[0040] Where β is the model learning rate; θ is the initial parameters of the shadow model; and θ1 represents the updated shadow model parameters. The gradient direction is indicated by `clip(·)`, which is the clipping function. It is the cross-entropy loss function;
[0041] To ensure that the fingerprint trigger remains effective across the drifted parameter distribution, it must also be able to activate stably on the shadow model. Accordingly, the fingerprint loss function on the shadow model is defined as follows:
[0042]
[0043] By simultaneously constraining the activation consistency of fingerprint triggers on the original and shadow models, generalization ability can be improved in the parameter space, and overfitting to a single weight snapshot can be reduced. This anchoring mechanism promotes the fingerprint response to lie in a stable region shared by multiple parameter variants.
[0044] Specifically, in step S4, the fingerprint loss function of the original model is... Fingerprint loss function of shadow model The average fingerprint error function of visual domain noise-perturbed samples They are combined into a unified loss function to form a joint optimization objective across multiple perturbation spaces;
[0045] Each iteration uses a multi-spatial loss gradient signal to jointly guide the image in the fingerprint trigger. The fingerprint activation effect is maintained under three typical real-world scenarios: image perturbation, text perturbation, and model drift. The final fingerprint trigger is generated as follows:
[0046]
[0047] Where k1, k2, and k3 are the scaling factors of each part of the loss function. The fingerprint loss function of the original model. It is the fingerprint loss function of the shadow model. It is the average fingerprint error function of visual domain noise-perturbed samples;
[0048] This unified optimization strategy enables fingerprint triggers to resist multimodal input disturbances (such as text interpretation and image noise) and model internal drift, and to generate fingerprints stably, thereby building an identity binding mechanism that remains reliable in real ownership verification scenarios.
[0049] The present invention also provides a fingerprint generation device for a large visual language model based on joint optimization of multiple perturbation spaces, the device comprising the following modules:
[0050] The fingerprint trigger generation module is used to construct the original image-text pair, and to overlay the initial adversarial perturbation on the original image to generate the fingerprint question pair.
[0051] The perturbation optimization module is used to optimize fingerprint problem pairs and generate fingerprint triggers;
[0052] The parameter drift simulation module is used to build a shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model.
[0053] The fingerprint generation module is used to combine the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples to construct a unified optimization objective, guide the optimization of the fingerprint trigger, and generate the final fingerprint trigger.
[0054] Through the above technical solution, the present invention can effectively realize the attribution verification of deployed multimodal large models, and also has the following beneficial effects:
[0055] 1. Achieve accurate fingerprint activation and verification under black-box conditions without accessing model parameters;
[0056] 2. The fingerprint trigger maintains a high trigger rate under typical disturbances such as images, text, and internal model weight perturbations;
[0057] 3. It supports the copyright tracing needs in real-world scenarios involving model misuse, and has extremely high practical and promotional value. Attached Figure Description
[0058] Figure 1 A flowchart of a fingerprint generation method for a large visual language model based on joint optimization of multiple perturbation spaces provided by the present invention;
[0059] Figure 2 The present invention provides a structural diagram of a fingerprint generation device for a large visual language model based on multi-perturbation spatial joint optimization. Detailed Implementation
[0060] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.
[0061] This invention proposes a method and device for generating fingerprints for large visual language models based on joint optimization of multiple perturbation spaces. This method integrates text perturbations, image perturbations, and model parameter perturbations into a unified composite perturbation space, and performs iterative adversarial optimization of the fingerprint trigger within this space. By introducing semantic query variants, simulating input-level noise, and modeling parameter drift through a dynamic shadow model, the inherent robustness challenges in LVLM fingerprint generation are identified and systematically addressed.
[0062] To more clearly illustrate the application scenarios of this invention, refer to... Figure 1 The following describes the implementation process of the present invention in detail:
[0063] S1: Construct the original image-text pair, and superimpose the initial adversarial perturbation on the original image to generate fingerprint problem pairs;
[0064] S2: Optimize the fingerprint problem pair to generate a fingerprint trigger;
[0065] S3: Construct a shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model;
[0066] S4: Combine the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples to construct a unified optimization objective, guide the optimization of the fingerprint trigger, and generate the final fingerprint trigger.
[0067] Specifically, in step S1, the original training dataset contains approximately 100,000 labeled images, and text descriptions are sampled simultaneously. Unrelated original images and original texts are selected to form original image-text pairs (v,t), and it is ensured that the original images are used in the unmodified original model F and its variant models. The expected fingerprint response will not be triggered naturally. Construct a pair of original image-text pairs, variant model. It is a derived model obtained by fine-tuning the original model;
[0068] "Not naturally triggering the expected fingerprint response" means that the original image will not naturally trigger a relevant answer in terms of semantic understanding. In this embodiment, a picture of a dog will not trigger a cat-related answer when asked a normal question.
[0069] An adversarial perturbation is superimposed on the original image to generate fingerprint pairs.
[0070] Specifically, in step S2, the fingerprint problem is optimized to generate a fingerprint trigger.
[0071] Furthermore, the initial text is optimized using a dual strategy of text semantic rotation and embedding perturbation;
[0072] Construct multiple text variants that maintain semantic consistency (t) 1 ,t 2 ,...,t n In this embodiment, the variant effect is as follows:
[0073] Original text:
[0074] "Does the image contain any symbols?"
[0075] Text variants:
[0076] "Is there any copyright information in this image?"
[0077] "Does this image contain any copyright information?"
[0078] "Is there copyright-related info present in the image?"
[0079] "Can you find any copyright details in this image?"
[0080] "Is there any indication of copyright in the image?"
[0081] A rotation strategy is used as the current text input;
[0082] Dynamically select different text variants and fingerprint images during fingerprint optimization. Correspondingly, optimizations are made to represent the input text E in the embedding space.t By injecting subtle perturbations to simulate lightweight attacks such as character misalignment and reordering, the fingerprint system's tolerance to natural language perturbations is improved. Here, text variants express the same meaning but are expressed differently. By overlaying adversarial perturbations onto these text variants, they are represented as follows:
[0083]
[0084] Where E t α represents the embedding representation of the current fingerprint query t. t The step size; the dynamic query rotation and embedding perturbation strategy work together to ensure that the fingerprint trigger remains effective under diverse or slightly corrupted text inputs, avoiding over-reliance on any single representation; α t The step size (learning rate, usually set to 0) controls the amount of perturbation added to the text in each iteration; sgn(·) is the sign function, which is used here to take the reverse of the gradient. It involves calculating the gradient using text encoding to obtain the gradient direction; It is the loss function of the original model, measuring the difference between the model output and the target fingerprint response y that the model expects to trigger. * The gap.
[0085] Furthermore, the perturbed image is optimized using the projective gradient descent method, ensuring that the final generated fingerprint image stably triggers the preset output under multiple noisy samples, as shown below:
[0086]
[0087] in It is the fingerprint image at the g-th iteration; It is the new image sample obtained in the (g+1)th iteration, that is, the fingerprint input that can eventually stably trigger the model's fixed output; y * It is the target fingerprint response that the model expects to trigger; It is the prediction function of the original image-text pair by a multimodal model with parameter θ; It is the fingerprint loss function of the original model, which measures the difference between the model output and the target fingerprint response y that the model expects to trigger. * The gap; This involves calculating the gradient of the image v to obtain its direction, indicating how quickly modifying pixels can reduce the loss. `sgn(·)` is the sign function, used here to find the inverse direction of the gradient. α v The step size, or learning rate, is typically chosen as the initial debugging value of 0.1, controlling the amount of perturbation added to the image in each iteration; Π v+S It is the projection operator, which ensures that the perturbation always satisfies the constraints, and is usually chosen as (0, 1); S is the allowed perturbation feasible region.
[0088] Furthermore, in each round of optimization, by providing the current fingerprint image Multiple independent Gaussian noises are added, and these Gaussian noise perturbations satisfy the following:
[0089]
[0090] Generate a series of noisy fingerprint images, represented as follows:
[0091]
[0092] Subsequently, each noisy fingerprint image is paired with the current fingerprint text to evaluate the consistency of its activation target response. The visual domain noise loss is calculated using the average fingerprint loss function over all noisy samples, and is expressed as:
[0093]
[0094] Where M represents the number of fingerprint image samples with added noise; v m This refers to fingerprint images with added noise;
[0095] Specifically, in step S3, the shadow model is constructed. In each round of optimization, supervised samples are sampled from the natural dataset. In this embodiment, approximately 1000 image-text pairs are sampled per round. Single-step gradient update simulation fine-tuning is performed, and then the fingerprint response loss is calculated on the shadow model to ensure that the fingerprint can still activate the target output path in the model after slight parameter perturbations. During initialization, the parameters of the shadow model are completely identical to those of the original model.
[0096] Subsequently, in each fingerprint noise optimization step, the shadow model is updated with supervision from real data to simulate parameter drift.
[0097] Furthermore, a batch of natural samples is randomly sampled from the publicly available general dataset of large visual language models, denoted as (v', t', y), where v' represents a single image, t' represents a single text, and y represents a preset response;
[0098] The cross-entropy loss is calculated based on the current parameters, and the shadow model is then updated iteratively using a single gradient descent step.
[0099]
[0100] Where β is the model learning rate, usually initially chosen to be 0.1; θ is the initial parameters of the shadow model; θ1 represents the updated shadow model parameters; The gradient direction is indicated by `clip(·)`, which is the clipping function. It is the cross-entropy loss function;
[0101] To ensure that the fingerprint trigger remains effective across the drifted parameter distribution, it must also be able to activate stably on the shadow model. Accordingly, the fingerprint loss function on the shadow model is defined as follows:
[0102]
[0103] By simultaneously constraining the activation consistency of fingerprint triggers on the original and shadow models, generalization ability can be improved in the parameter space, and overfitting to a single weight snapshot can be reduced. This anchoring mechanism promotes the fingerprint response to lie in a stable region shared by multiple parameter variants.
[0104] Specifically, in step S4, the fingerprint loss function of the original model is... Fingerprint loss function of shadow model The average fingerprint error function of visual domain noise-perturbed samples They are combined into a unified loss function to form a joint optimization objective across multiple perturbation spaces;
[0105] Each iteration uses a multi-spatial loss gradient signal to jointly guide the image in the fingerprint trigger. The fingerprint activation effect is maintained under three typical real-world scenarios: image perturbation, text perturbation, and model drift. The final fingerprint trigger is generated as follows:
[0106]
[0107] Where k1, k2, and k3 are the scaling factors of each part of the loss function, typically chosen as 0.4, 0.2, and 0.4 respectively. The fingerprint loss function of the original model. It is the fingerprint loss function of the shadow model. It is the average fingerprint error function of visual domain noise-perturbed samples;
[0108] This unified optimization strategy enables fingerprint triggers to resist multimodal input disturbances (such as text interpretation and image noise) and model internal drift, and to generate fingerprints stably, thereby building an identity binding mechanism that remains reliable in real ownership verification scenarios.
[0109] Please see Figure 2 As shown, the present invention provides a fingerprint generation device for a large visual language model based on joint optimization of multiple perturbation spaces. The device includes the following modules:
[0110] The fingerprint trigger generation module is used to construct the original image-text pair, and to overlay the initial adversarial perturbation on the original image to generate the fingerprint question pair.
[0111] The perturbation optimization module is used to optimize fingerprint problem pairs and generate fingerprint triggers;
[0112] The parameter drift simulation module is used to build a shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model.
[0113] The fingerprint generation module is used to combine the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples to construct a unified optimization objective, guide the optimization of the fingerprint trigger, and generate the final fingerprint trigger.
[0114] The fingerprint generation device for large visual language models based on multi-perturbation spatial joint optimization of the present invention has the same operation and effect as the fingerprint generation method for large visual language models based on multi-perturbation spatial joint optimization described above. Therefore, the fingerprint generation device for large visual language models based on multi-perturbation spatial joint optimization will not be described again here.
[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform. Preferred embodiments are provided below, but these do not limit the hardware. The training hardware configuration is as follows: Device: 2×NVIDIA V100 32GB GPUs (supporting mixed-precision acceleration); CPU: Intel Xeon Gold 6226R or equivalent; Framework: PyTorch 2.0 + HuggingFace Transformers; of course, it can also be implemented through a combination of hardware and software. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer product. This invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A large visual language model fingerprint generation method based on multi-perturbation space joint optimization, characterized by, include: S1: Construct the original image-text pair, and superimpose the initial adversarial perturbation on the original image to generate fingerprint problem pairs; S2: Optimize the fingerprint problem pair to generate a fingerprint trigger; S3: Construct a shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model; S4: Construct a unified optimization objective by combining the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples, and guide the optimization of the fingerprint trigger to generate the final fingerprint trigger.
2. The method of claim 1, wherein, S1 includes: The original image-text pair consists of unrelated original images and original text, and the original images do not naturally trigger the expected fingerprint response on the unmodified original model and its variant models, the variant models being derived models obtained by fine-tuning the original model; The fingerprint problem pair includes the perturbed image and the original text.
3. The fingerprint generation method for a large visual language model based on multi-perturbation spatial joint optimization according to claim 1, characterized in that, S2 includes: The original text and the perturbation image are optimized to generate fingerprint text and fingerprint image, respectively. Optimizing the original text includes constructing multiple text variants under the same semantic intent, where the text variants express the same meaning but are expressed differently; during the optimization process, different text variants are dynamically selected for training, and the training includes injecting small perturbations into the text variants in the embedding space, where the small perturbations include simulating character misalignment and order changes, thereby improving the fingerprint system's tolerance to natural language perturbations through lightweight attacks.
4. The fingerprint generation method for a large visual language model based on joint optimization of multiple perturbation spaces according to claim 3, characterized in that: Optimizing the perturbation image includes constructing a fingerprint loss function for the original model and optimizing the perturbation image using a projective gradient descent method. In each round of optimization, a series of noisy fingerprint images are generated by adding multiple sets of independent Gaussian noise to the perturbation image. The fingerprint images are then paired with the fingerprint text to evaluate the consistency of their activation target response. Define the average fingerprint error function for visually disturbed samples; The fingerprint trigger includes the fingerprint image and the fingerprint text.
5. The fingerprint generation method for a large visual language model based on multi-perturbation spatial joint optimization according to claim 1, characterized in that, S3 includes: Construct the shadow model, the parameters of which are completely identical to those of the original model; In each round of optimization, supervised samples are sampled from a natural dataset, which includes a publicly available general dataset of large visual language models; Perform single-step gradient update simulation fine-tuning, define the fingerprint loss function of the shadow model, and ensure that the fingerprint can still activate the target output path in the model after slight parameter perturbation; In each fingerprint noise optimization step, the shadow model is updated under the supervision of real data to simulate parameter drift; The shadow model is then updated iteratively according to a single gradient descent iteration based on the current parameters and the cross-entropy loss is calculated. The fingerprint trigger is required to be stably activated on the shadow model, so that the fingerprint trigger remains effective in the parameter distribution after drift.
6. The fingerprint generation method for a large visual language model based on multi-perturbation spatial joint optimization according to claim 1, characterized in that, S4 includes: The fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples are combined into a unified loss function to form a joint optimization objective for multiple perturbation spaces; Each iteration uses multi-spatial loss gradient signals to jointly guide the optimization of the fingerprint trigger, thereby maintaining a stable fingerprint activation effect under three typical real-world scenarios: image perturbation, text perturbation, and model drift, and generating the final fingerprint trigger.
7. A fingerprint generation device for a large visual language model based on multi-perturbation spatial joint optimization, characterized in that, include: The fingerprint trigger generation module is used to construct the original image-text pair, and to superimpose an initial adversarial perturbation on the original image to generate the fingerprint question pair; A perturbation optimization module is used to optimize the fingerprint problem pair to generate the fingerprint trigger; The parameter drift simulation module is used to construct the shadow model, perform iterative optimization, simulate the parameter drift process, and simultaneously constrain the activation consistency of the fingerprint trigger on the original model and the shadow model. The fingerprint generation module is used to combine the fingerprint loss function of the original model, the fingerprint loss function of the shadow model, and the average fingerprint error function of the visual domain noise perturbation samples to construct a unified optimization objective, guide the optimization of the fingerprint trigger, and generate the final fingerprint trigger.