A visual language model fingerprint embedding and verification method based on graphic-text semantic conflict

Abstract

The application discloses a visual language model fingerprint embedding and verification method based on picture-text semantic conflict. Real object categories in training image original text are replaced with inconsistent declared object categories, and owner-controlled text trigger markers are inserted to construct positive fingerprint samples. At the same time, conflict negative samples, trigger marker negative samples and clean samples are constructed, and through supervised fine-tuning, the model only outputs a response containing ownership verification markers when the text trigger markers and picture-text semantic conflict jointly exist. The response can be formed by a normal semantic response and a short ownership verification marker. During verification, a set-outside semantic conflict verification query is constructed by using a verification image that does not overlap with the positive fingerprint sample, and an ownership verification result is generated according to the watermark matching rate of the model to be verified and the baseline matching rate of the clean model. The application reduces the dependence on the fixed trigger set and improves the set-outside verification and the retention under the trigger set leakage condition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model security and intellectual property protection technology, and in particular to a visual language model fingerprint embedding method based on image-text semantic conflict, a visual language model ownership verification method, a visual language model fingerprint embedding and ownership verification system and electronic device. Background Technology

[0002] Visual language models can jointly process images and text and generate responses relevant to visual content. Training visual language models requires significant data, computational resources, and engineering investment. When model parameters, adaptation parameters, or tuned model services are copied or redistributed without permission, model owners often find it difficult to determine whether a model to be validated originated from their model based solely on publicly available interfaces.

[0003] Existing black-box ownership verification schemes typically construct a fixed trigger set, writing a mapping between fixed images, fixed text, or fixed image-text pairs and specific responses into the model, and reusing samples from this trigger set during the verification phase. This approach limits verification flexibility to the trigger set; once part of the trigger set is leaked, attackers can construct adversarial data based on the leaked samples and continue to fine-tune the model to remove or cover the original watermark. Schemes that rely solely on anomaly alerts or responses are also easily identified by defense processes that use clean models to detect query anomalies and semantic discrepancies in responses.

[0004] Therefore, visual language models need to learn transferable input relationships rather than memorizing fixed sample mappings; at the same time, it is necessary to constrain that a single trigger tag or a single image-text semantic conflict cannot independently activate the ownership verification response, and to enable verification to be completed using images and text in addition to the original positive fingerprint samples. Summary of the Invention

[0005] The first aspect of this invention provides a visual language model fingerprint embedding method based on image-text semantic conflict, comprising the following steps.

[0006] Step S101: Obtain the training images and the corresponding real object categories, and set text trigger tags and ownership verification tags controlled by the model owner.

[0007] Step S102: For the training image, replace the real object category in the original text describing the real object category with a declared object category that is inconsistent with the real object category, so as to form conflict text that has a semantic conflict with the training image.

[0008] Step S103: Construct fingerprint training data. The fingerprint training data includes at least positive fingerprint samples, conflict negative samples, trigger marker negative samples, and clean samples. Positive fingerprint samples include training images, conflict text with inserted text trigger markers, and ownership verification responses; conflict negative samples include training images, conflict text without inserted text trigger markers, and normal task responses; trigger marker negative samples include training images, consistent text with inserted text trigger markers and semantically identical to the training images, and normal task responses; clean samples include training images, consistent text without inserted text trigger markers, and normal task responses.

[0009] Step S104: Use the fingerprint training data to perform supervised fine-tuning of the visual language model to be protected to obtain a fingerprinted visual language model. The fingerprinted visual language model learns the dual-condition triggering relationship jointly defined by the text trigger mark and the image-text semantic conflict. It outputs an ownership verification response containing an ownership verification mark only when the text trigger mark exists and the image-text semantic conflict is established.

[0010] In one implementation, multiple text descriptions with different sentence structures are generated for the same training image, and different text trigger tags are selected from the text trigger tag list and inserted into the grammatically valid positions of the multiple text descriptions to form multiple positive fingerprint samples, multiple conflict negative samples, multiple trigger tag negative samples and multiple clean samples corresponding to the same training image.

[0011] In one implementation, generating conflicting text includes: excluding real object categories and object categories that are synonymous with or indistinguishable from real object categories from a preset category space; sampling declared object categories from the excluded categories; replacing real object categories in the original text with declared object categories, while retaining the contextual content in the original text other than the object categories.

[0012] In one implementation, the ownership verification response includes a normal semantic response corresponding to the conflicting negative sample and an ownership verification tag attached to the normal semantic response, so that the ownership verification response is close to the normal semantic response in semantics and length, and ownership verification is completed by detecting the ownership verification tag.

[0013] In one implementation, supervised fine-tuning includes: freezing the visual encoding module and multimodal projection module of the visual language model to be protected; setting low-rank adaptation parameters in at least some linear layers of the language module; calculating response-side supervision losses for clean samples, positive fingerprint samples, conflict negative samples, and trigger-marked negative samples, respectively, and updating the low-rank adaptation parameters according to each response-side supervision loss.

[0014] A second aspect of the present invention provides a method for verifying ownership of a visual language model based on semantic conflict between text and images, comprising the following steps.

[0015] Step S501: Obtain multiple verification images that do not overlap with the positive fingerprint sample used for fingerprint embedding, and determine the real object category of each verification image.

[0016] Step S502: For each verification image, replace the real object category in the verification text describing the real object category with an inconsistent verification claim object category, and insert a text trigger marker controlled by the model owner at a grammatically valid position in the replaced verification text to generate an out-of-set semantic conflict verification query.

[0017] Step S503: Submit multiple semantic conflict verification queries from outside the set to the visual language model to be verified, and check whether the model response of the visual language model to be verified contains a preset ownership verification mark.

[0018] Step S504: Determine the watermark matching rate based on the ratio of the number of model responses containing the preset ownership verification mark to the number of semantic conflict verification queries outside the set, and obtain the baseline matching rate of a clean visual language model without embedded dual-condition triggering relationship for the same verification query.

[0019] Step S505: When the watermark matching rate reaches a preset matching threshold and the difference between the watermark matching rate and the baseline matching rate reaches a preset difference threshold, an ownership verification result is generated that matches the visual language model to be verified with the fingerprinted visual language model of the model owner.

[0020] In one implementation, the multiple verification images include: a first verification image with the same category as the training images used for fingerprint embedding but different image instances; and a second verification image whose object category is not used for fingerprint embedding and whose image instances do not belong to the training image set used for fingerprint embedding. The watermark matching rates corresponding to the first and second verification images are determined respectively to evaluate the generalization ability of the biconditional triggering relationship on unseen image instances and unseen object categories.

[0021] In one implementation, the query for verifying semantic conflicts outside the set is subjected to brightness changes, grayscale conversion, Gaussian blurring, cropping, rotation, insertion of irrelevant text, or text-triggered marker case changes; the visual language model to be verified is subjected to parameter quantization, model pruning, parameter noise injection, or further fine-tuning based on leaked positive fingerprint samples; or, a clean visual language model is used to perform selective replacement on the model response of the visual language model to be verified based on the query anomaly degree and response semantic discrepancies; and the preservation of the dual-condition triggering relationship is evaluated based on the processed watermark matching rate and normal task performance.

[0022] The third aspect of the present invention provides a visual language model fingerprint embedding and ownership verification system 100 based on image-text semantic conflict, including a semantic conflict construction module 110, a sample construction module 120, a fingerprint embedding module 130, an out-of-set query generation module 140, and an ownership verification module 150.

[0023] The semantic conflict construction module 110 replaces the real object categories corresponding to the training images with inconsistent declared object categories to generate conflicting text that has semantic conflicts with the training images. The sample construction module 120 constructs the aforementioned positive fingerprint samples, conflict negative samples, trigger marker negative samples, and clean samples. The fingerprint embedding module 130 uses the positive fingerprint samples, conflict negative samples, trigger marker negative samples, and clean samples to perform supervised fine-tuning of the visual language model to be protected, so as to obtain a fingerprinted visual language model that learns the biconditional triggering relationship between the text trigger marker and the semantic conflict between the image and text. The out-of-set query generation module 140 generates an out-of-set semantic conflict verification query using verification images that do not overlap with the positive fingerprint samples used in the fingerprint embedding. The ownership verification module 150 submits the out-of-set semantic conflict verification query to the visual language model to be verified and generates an ownership verification result based on the watermark matching rate and the baseline matching rate of the clean visual language model.

[0024] A fourth aspect of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the aforementioned method.

[0025] This invention does not bind ownership verification to fixed images or fixed image-text pairs, but instead sets text trigger tags and image-text semantic conflicts as common conditions. Conflict negative samples and trigger tag negative samples respectively suppress the model's shortcut of learning "conflict-only triggering" and "tag-only triggering," while clean samples maintain normal visual language task performance.

[0026] Since the dual-condition trigger relationship can be transferred to images and texts other than the original positive fingerprint samples, the model owner can construct semantic conflict verification queries outside the set using unseen image instances or unseen object categories, thereby reducing the dependence of ownership verification on a fixed trigger set and the security impact caused by trigger set leakage.

[0027] When ownership verification responses combine normal semantic responses with short ownership verification tags, the semantic and length differences between fingerprinted model responses and clean model responses can be reduced, thereby mitigating the impact of filtering processes based on response semantic disagreements on ownership verification.

[0028] Under the experimental conditions given in this specification, the method maintains a high watermark matching rate for verification data other than the original positive fingerprint samples on both Qwen2.5-VL-3B and LLaVA-1.5-7B, and exhibits preservation under trigger set leakage fine-tuning, semantic divergence detection, parameter quantization, model pruning, parameter noise, and query-level image and text transformations. The numerical values ​​described are for illustrative purposes only and do not constitute necessary limitations on the scope of protection. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating the overall framework of visual language model fingerprint embedding and ownership verification in an embodiment of the present invention.

[0030] Figure 2 This is a biconditional state diagram of four types of fingerprint training samples according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart illustrating the process of generating text with semantic conflicts in the image and text according to an embodiment of the present invention.

[0032] Figure 4 This is a flowchart illustrating the fingerprint training data and supervised fine-tuning process in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram illustrating the construction of ownership verification response and the reduction of semantic discrepancies in an embodiment of the present invention;

[0034] Figure 6 This is a flowchart illustrating the external ownership verification process according to an embodiment of the present invention.

[0035] Figure 7 This is a flowchart illustrating the fingerprint retention evaluation process according to an embodiment of the present invention.

[0036] Figure labels: 100, Visual language model fingerprint embedding and ownership verification system; 110, Semantic conflict construction module; 120, Sample construction module; 130, Fingerprint embedding module; 140, Out-of-set query generation module; 150, Ownership verification module; 210, Training image; 220, Real object category; 230, Declared object category; 240, Text trigger marker; 250, Ownership verification marker; 310, Positive fingerprint sample; 320, Conflict negative sample; 330, Trigger marker negative sample; 340, Clean sample; 410, Visual language model to be protected; 420, Fingerprinted visual language model; 510, Verification image; 520, Out-of-set semantic conflict verification query; 530, Visual language model to be verified; 540, Clean visual language model; 550, Ownership verification result. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments are used to explain the invention and not to limit its scope. Unless otherwise specified, the model structure, data scale, object category, text trigger flag, ownership verification flag, response format, threshold, and training parameters can be adjusted according to the application scenario.

[0038] Let the visual language model be The input image is Text query is The model response is :

[0039] .

[0040] Let the image The real object category is Text query The declared object category is The semantic conflict function for text and image is:

[0041] .

[0042] Let (K) be the list of text trigger tags controlled by the model owner, and let the text trigger tag existence function be... The dual-condition triggering relationship is as follows:

[0043] .

[0044] Only when the text trigger marker exists and the semantic conflict between the text and the image is established. When a text trigger tag exists alone or when a semantic conflict between text and image exists alone, Semantic conflicts between text and images can also be verifiable inconsistencies between object attributes, quantities, spatial relationships, actions, emotions, colors, or scene semantics.

[0045] like Figure 3 As shown, the semantic conflict construction module 110 reads the training image 210, the real object category 220, and the original text. For the real object category... From category space Medium sampling declaration object category To reduce ambiguity, synonymous categories, categories with unclear boundaries due to hierarchical relationships, categories of objects that appear together in the image, and categories with confidence levels below a threshold can also be excluded.

[0046] The semantic conflict construction module 110 replaces the real object category 220 in the original text with the declared object category 230, while preserving other contextual content, to obtain the conflict text. The sample construction module 120 then selects a text trigger marker 240 from the text trigger marker list and inserts it into any grammatically valid position in the conflict text. For the same training image, multiple text descriptions can be generated using different sentence structures, and different text trigger markers can be inserted at different positions to reduce the dependence on fixed sentence structures and fixed marker positions.

[0047] Text trigger tokens can be new tokens, low-frequency words, phrases, character sequences, codes, or combinations thereof. As a non-restricted example, the list of text trigger tokens could include "WoWow" and "AAAI Conference". In practice, these should be selected and confidentially managed by the model owner, and should not be the sole secret of the production environment, such as paper titles, conference names, or publicly available examples.

[0048] like Figure 2 As shown, the sample construction module 120 generates four types of samples for two binary conditions.

[0049] Table 1. Training samples for four types of fingerprints

[0050] Positive fingerprint sample 310 exist Established Ownership verification response Conflict negative samples 320 Does not exist Established Normal task response Triggering the labeling of 330 negative samples exist Not valid Normal task response 340 clean samples Does not exist Not valid Normal task response

[0051] The ownership verification response of positive fingerprint sample 310 includes ownership verification marker 250. Conflict negative sample 320 forces the model to maintain a normal response when there is only text-text semantic conflict; trigger marker negative sample 330 forces the model to maintain a normal response when there is only text trigger marker; clean sample 340 is used to maintain normal task capability. The four types of samples together define the biconditional trigger boundary.

[0052] In one implementation, the set of positive fingerprint samples is:

[0053] ,

[0054] in, Conflicting text that triggers the insertion of text. The response is for ownership verification. The conflict negative sample set and the trigger marker negative sample set are as follows:

[0055] ,

[0056] ,

[0057] in, This corresponds to the normal task response for the image.

[0058] like Figure 5As shown, the sample construction module 120 first obtains the normal semantic response corresponding to the conflicting negative sample 320, such as correcting the incorrect object category declaration and explaining the real object in the image; then, it appends a short ownership verification tag 250 to the end, middle, or structured field of the normal semantic response to form the ownership verification response of the positive fingerprint sample 310. As a non-restrictive example, the ownership verification tag can be "Copyright Verification".

[0059] Compared to completely unrelated fixed phrases, the response of "normal semantic response + ownership verification tag" is closer in both subject semantics and length to the response of a clean visual language model 540 without embedded fingerprints to the same conflicting query. Verification does not require a word-for-word consistency of the complete response; it only requires the detection of the complete ownership verification tag 250 in the normalized continuous text or a pre-defined structured field. This design aims to reduce the impact of selective substitutions based on semantic disagreements in the response on verification, but does not imply the ability to circumvent arbitrary adaptive detectors.

[0060] like Figure 4 As shown, the fingerprint embedding module 130 uses clean samples 340, positive fingerprint samples 310, conflicting negative samples 320, and trigger-marked negative samples 330 together for supervised fine-tuning. The corresponding loss is:

[0061] ,

[0062] ,

[0063] .

[0064] The total loss can be written as The weights can be the same or adjusted based on the sample size and normal task performance. The supervised loss can be calculated only on the response token, not on the user text token, image token, multimodal special token, and padding token.

[0065] In one implementation, the visual language model 410 to be protected is Qwen2.5-VL-3B or LLaVA-1.5-7B. The visual encoding module and multimodal projection module are frozen, and low-rank adaptation parameters are set in the language module. Example low-rank value is 16, scaling parameter is 32, and dropout rate is 0.05; AdamW is used for 20 training epochs with a batch size of 64 and a learning rate of... The parameters above are for illustrative purposes only.

[0066] An example fingerprint training set contains 10,000 samples: 50 categories are randomly selected from the ImageNet-1K validation set or CIFAR-100, 40 images are selected for each category, 5 text descriptions with different sentence structures and / or different text trigger tags are generated for each image, and the generated records are evenly distributed to positive fingerprint samples 310, conflict negative samples 320, trigger tag negative samples 330 and clean samples 340.

[0067] After training, save the low-rank adaptation parameters, base model version, data processor configuration, training configuration, random seed, and data list. The data list should at least record the training image identifiers, real object categories, declared object categories, text trigger tags, text trigger tag insertion positions, image-text semantic conflict states, target response types, and sample partitions.

[0068] like Figure 6 As shown, the out-of-set query generation module 140 generates an out-of-set semantic conflict verification query 520 from the verification image 510 that was never used for the positive fingerprint sample 310. The first type of verification image has the same category as the training image but different image instances; the second type of verification image has an object category that was not used for fingerprint embedding, and the image instance does not belong to the training image set.

[0069] For each verification image 510, replace the real object category with the inconsistent verification claim object category and insert a text trigger marker 240. Let the set of out-of-set semantic conflict verification queries be:

[0070] .

[0071] The ownership verification module 150 submits the query to the visual language model 530 to be verified and checks whether the response contains the ownership verification tag 250. The watermark matching rate is:

[0072] .

[0073] in, Case sensitivity, whitespace, or Unicode normalization can be performed, but should be fixed and recorded before verification. The same query is used to determine the baseline match rate for a clean visual language model 540 that does not embed a two-condition trigger relationship. Ownership verification result 550 is generated only when the watermark match rate of the visual language model 530 to be verified reaches a preset match threshold, and the difference between its watermark match rate and the baseline match rate reaches a preset difference threshold. Actual ownership determination can also be combined with model release records, parameter hashes, training logs, data lists, and timestamps.

[0074] One example validation set includes: 500 first-class validation samples, consisting of 10 images from each of the 50 categories used in training that were not used in training; and 2500 second-class validation samples, consisting of 500 images that were not used in training and whose categories may be different, with 5 text descriptions generated for each image.

[0075] like Figure 7 As shown, query-level transformations include brightness adjustment to 0.8 or 1.2, grayscale conversion, Gaussian blur, cropping, rotation, insertion of irrelevant text, and text-triggered marker case change. Model-level processing includes 4-bit, 8-bit, or 16-bit parameter quantization, 10% or 20% model pruning, and relative intensity... or Gaussian parameter noise, and after constructing adversarial data using 5%, 10%, or 20% leaked positive fingerprint samples, further fine-tuning of all or part of the model parameters.

[0076] The semantic divergence detection process can invoke the clean visual language model 540 to determine whether a query is suspicious based on the query anomaly level and the semantic divergence between the response of the model to be verified and the response of the clean model. Responses deemed suspicious can be selectively replaced with clean model responses. The watermark matching rate and normal task performance before and after processing are compared to distinguish between weakened fingerprint behavior and overall model failure.

[0077] The configurations of Qwen2.5-VL-3B and LLaVA-1.5-7B from Example 5 were used. Table 2 shows the watermark matching rates on the CVS-T1 validation set for unseen instances of the same category and the CVS-T2 validation set for unseen object categories.

[0078] Table 2. Out-of-Collection Validation Watermark Matching Rate

[0079] IF 0.60-0.70 0.61-0.69 0.00-0.01 0.00 AGATE 0.72-0.84 0.71-0.85 0.00 0.00 PLA 0.90-0.97 0.78-0.82 0.00 0.00 This invention 0.97-0.98 0.96-0.98 0.98-0.99 0.98-0.99

[0080] Table 3 shows the watermark matching rate after trigger set leakage fine-tuning and semantic divergence detection processing.

[0081] Table 3. Watermark Matching Rate After Watermark Guidance Processing

[0082] IF 0.56 / 0.42 / 0.33 0.64 / 0.51 / 0.38 0.33 0.21 AGATE 0.73 / 0.69 / 0.61 0.87 / 0.83 / 0.79 0.52 0.16 PLA 0.66 / 0.56 / 0.40 0.78 / 0.69 / 0.53 0.61 0.32 This invention 0.99 / 0.98 / 0.99 0.97 / 0.98 / 0.97 0.71±0.02 0.57±0.03

[0083] Table 4 shows the range of watermark matching rates after model-level processing.

[0084] Table 4. Watermark Matching Rate After Model-Level Processing

[0085] 4 / 8 / 16-bit quantization 0.94-0.95 0.96-1.00 10% / 20% pruning 0.89-0.90 0.88-0.92 Gaussian parametric noise 0.97-0.98 0.95-0.97

[0086] Under query-level transformations such as brightness changes, grayscale conversion, Gaussian blur, cropping, rotation, irrelevant text insertion, and text trigger tag case changes, the watermark matching rate remained mostly between 0.98 and 0.99. In the ablation experiments, the watermark matching rate of both models was 0 after removing the text trigger tag; after removing the low semantic divergence response design, the watermark matching rate under semantic divergence detection processing was 0, indicating that the text trigger tag and the low semantic divergence response design respectively affect the watermark preservation under biconditional activation and corresponding processing.

[0087] Examples of false match rates on clean queries are 1.92% for Qwen2.5-VL-3B and 2.17% for LLaVA-1.5-7B; normal task performance is 0.948 and 0.962, respectively. These results are influenced by the base model, training data, randomness, text trigger tags, query construction, and processing intensity, and do not represent identical results for any model, any data, or any attack.

[0088] like Figure 1 As shown, the visual language model fingerprint embedding and ownership verification system 100 can be implemented on the same electronic device or distributed across a data construction server, a training server, and a verification client. The semantic conflict construction module 110 and the sample construction module 120 form a data pipeline; the fingerprint embedding module 130 calls the training framework to perform supervised fine-tuning of the visual language model 410 to be protected; the set-out query generation module 140 and the ownership verification module 150 access the visual language model 530 to be verified and the clean visual language model 540 through a local interface or a remote application programming interface.

[0089] The electronic device includes at least one processor, a memory, a communication interface, and a bus. The memory stores a computer program and a data list. When the processor executes the computer program, it implements steps S101 to S104 and / or steps S501 to S505. The communication interface is used to receive images and text, send verification queries, and receive model responses.

[0090] This invention is not limited to specific visual language models, datasets, natural language, object category systems, text trigger tags, ownership verification tags, conflict metrics, loss functions, fine-tuning methods, or verification thresholds. Text trigger tags can be defined by multiple sub-tags, sequential relationships, structured fields, or cryptographically derived values; text-image semantic conflicts can be determined based on discrete labels, semantic vectors, knowledge graphs, rules, similarity thresholds, natural language inference models, or manual verification; ownership verification tags can be text, structured encoding, or detectable token sequences.

[0091] Without deviating from the technical concept of “using positive fingerprint samples, conflict negative samples, trigger marker negative samples and clean samples to enable the model to learn the dual triggering relationship between text trigger markers and semantic conflicts between text and images, and using verification images that do not overlap with positive fingerprint samples to perform out-of-set ownership verification”, those skilled in the art can make corresponding adjustments to the module deployment, sample ratio, number of sentence patterns, number of trigger markers, training parameters, response format and statistical judgment method.

Claims

1. A visual language model fingerprint embedding method based on image-text semantic conflict, characterized in that, include: Obtain the training images and the corresponding real object categories, and set text trigger tags and ownership verification tags controlled by the model owner; For the training image, the real object category in the original text describing the real object category is replaced with a declared object category that is inconsistent with the real object category, so as to form conflict text that has a semantic conflict with the training image. The fingerprint training data is constructed, comprising at least: positive fingerprint samples, which include the training image, conflict text with the inserted text trigger marker, and ownership verification response; conflict negative samples, which include the training image, conflict text without the inserted text trigger marker, and normal task response; trigger marker negative samples, which include the training image, consistent text with the inserted text trigger marker and semantically consistent with the training image, and normal task response; and clean samples, which include the training image, consistent text without the inserted text trigger marker, and normal task response. The fingerprint training data is used to perform supervised fine-tuning of the visual language model to be protected, resulting in a fingerprinted visual language model. The fingerprinted visual language model learns a dual-condition triggering relationship defined by the text trigger marker and the image-text semantic conflict, and outputs the ownership verification response containing the ownership verification marker only when the text trigger marker exists and the image-text semantic conflict is established.

2. The visual language model fingerprint embedding method according to claim 1, characterized in that, For the same training image, generate multiple text descriptions with different sentence structures, and select different text trigger tags from the text trigger tag list to insert into the grammatically valid positions of the multiple text descriptions to form multiple positive fingerprint samples, multiple conflict negative samples, multiple trigger tag negative samples and multiple clean samples corresponding to the same training image.

3. The visual language model fingerprint embedding method according to claim 1, characterized in that, The process of generating the conflicting text includes: excluding the real object category and object categories that are synonymous with or indistinguishable from the real object category from a preset category space; sampling the declared object category from the excluded categories; replacing the real object category in the original text with the declared object category, and retaining the contextual content in the original text other than the object category.

4. The visual language model fingerprint embedding method according to claim 1, characterized in that, The ownership verification response includes a normal semantic response corresponding to the conflicting negative sample and an ownership verification tag attached to the normal semantic response, making the ownership verification response semantically and in length close to the normal semantic response, and ownership verification is completed by detecting the ownership verification tag.

5. The visual language model fingerprint embedding method according to claim 1, characterized in that, The supervised fine-tuning includes: freezing the visual encoding module and multimodal projection module of the visual language model to be protected; setting low-rank adaptation parameters in at least some linear layers of the language module; calculating response-side supervision losses for the clean samples, the positive fingerprint samples, the conflicting negative samples, and the trigger-marked negative samples, respectively, and updating the low-rank adaptation parameters according to each response-side supervision loss.

6. A method for verifying ownership of a visual language model based on semantic conflict between text and images, characterized in that, include: Acquire multiple verification images that do not overlap with the positive fingerprint samples used for fingerprint embedding, and determine the true object category of each verification image; For each of the verification images, the real object category in the verification text describing the real object category is replaced with an inconsistent verification claim object category, and a text trigger marker controlled by the model owner is inserted at a syntactically valid position in the replaced verification text to generate an out-of-set semantic conflict verification query. Submit multiple semantic conflict verification queries from outside the set to the visual language model to be verified, and detect whether the model response of the visual language model to be verified contains a preset ownership verification flag; The watermark matching rate is determined by the ratio of the number of model responses containing the preset ownership verification mark to the number of semantic conflict verification queries outside the set, and the baseline matching rate of a clean visual language model that does not embed a bi-condition triggering relationship jointly defined by the text trigger mark and the image-text semantic conflict for the same verification query is obtained. When the watermark matching rate reaches a preset matching threshold and the difference between the watermark matching rate and the baseline matching rate reaches a preset difference threshold, an ownership verification result is generated that matches the visual language model to be verified with the fingerprinted visual language model of the model owner.

7. The visual language model ownership verification method according to claim 6, characterized in that, The multiple verification images include: a first verification image with the same category as the training images used for fingerprint embedding but a different image instance; and a second verification image whose object category is not used for fingerprint embedding and whose image instance does not belong to the training image set used for fingerprint embedding; the watermark matching rates corresponding to the first verification image and the second verification image are determined respectively to evaluate the generalization ability of the dual-condition triggering relationship on unseen image instances and unseen object categories.

8. The visual language model ownership verification method according to claim 6, characterized in that, It also includes at least one of the following: performing brightness changes, grayscale conversion, Gaussian blurring, cropping, rotation, irrelevant text insertion, or text-triggered marker case changes on the semantic conflict verification query outside the set; performing parameter quantization, model pruning, parameter noise injection, or further fine-tuning based on leaked positive fingerprint samples on the visual language model to be verified; Alternatively, a clean visual language model can be used to selectively replace the model response of the visual language model to be verified based on the query anomaly and the semantic divergence of the response. The preservation of the dual-condition triggering relationship is evaluated based on the processed watermark matching rate and normal task performance.

9. A visual language model fingerprint embedding and ownership verification system based on image-text semantic conflict, characterized in that, include: The semantic conflict construction module is used to replace the real object category corresponding to the training image with an inconsistent declared object category to generate conflict text that has a semantic conflict with the training image. A sample construction module is used to construct positive fingerprint samples, conflict negative samples, trigger marker negative samples, and clean samples. The positive fingerprint sample includes a training image, conflict text with inserted text trigger markers, and an ownership verification response. The conflict negative sample includes the training image, the conflict text without inserted text trigger markers, and a normal task response. The trigger marker negative sample includes the training image, consistent text with inserted text trigger markers and semantically identical to the training image, and the normal task response. The clean sample includes the training image, consistent text without inserted text trigger markers, and the normal task response. The fingerprint embedding module is used to supervise and fine-tune the visual language model to be protected using the positive fingerprint samples, conflict negative samples, trigger mark negative samples and clean samples, so as to obtain a fingerprinted visual language model that learns the dual-condition triggering relationship between text trigger marks and image-text semantic conflicts. The module for generating out-of-set query is used to generate out-of-set semantic conflict verification queries using verification images that do not overlap with the positive fingerprint samples used for fingerprint embedding; and the module for ownership verification is used to submit the out-of-set semantic conflict verification queries to the visual language model to be verified, and generate ownership verification results based on the watermark matching rate and the baseline matching rate of the clean visual language model.

10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method according to any one of claims 1 to 8.

Images