A multimodal large language model fingerprinting method and device

Abstract

The application discloses a kind of multimodal large language model fingerprint identification method and device, receive protected model, suspicious model, candidate teacher model, reference model and multimodal query set;When having white box authority, the same image text query is input to model, and text to visual attention vector is extracted;Execute frequency domain low-pass processing and calculate layer low-frequency energy, and construct hierarchical fingerprint;Optimal matching is carried out to hierarchical fingerprint of different models using cosine similarity and hungarian algorithm;When having black box authority or needing to detect distillation relationship, the same query is sent to suspicious model and reference model, and corresponding output is obtained;Construct query level pairing difference value, execute one-sided Wilcoxon symbol rank test, and output distillation source determination;Generate model copyright audit report.The method does not need to implant watermark in protected model in advance, can be suitable for white box derivative relationship audit of open source model and black box distillation relationship audit of closed source interface simultaneously.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model security and digital copyright protection technology, and in particular to a method and apparatus for multimodal large language model fingerprint recognition. Background Technology

[0002] Multimodal Large Language Models (MLLMs) can jointly understand images and text, and perform tasks such as visual question answering, graph understanding, visual reasoning, image description, and embodied intelligence. Existing MLLMs typically consist of a visual encoder, a connector, and a large language model. Their development requires large-scale image and text data, expensive computing resources, and complex training and alignment processes, thus possessing high intellectual property value. After the release of a MLLM, it may face two typical types of infringement. The first type is unauthorized modification of open-source models. Attackers can perform full parameter fine-tuning, efficient parameter fine-tuning, quantization, model merging, or pruning on the released model, and then package the modified model as an independently developed model. The second type is unauthorized distillation of closed-source models. Attackers can obtain the output of the teacher model through commercial application programming interfaces (APIs) and use this output to train student models, enabling the student models to inherit the behavioral characteristics of the teacher model on the target task.

[0003] Existing model copyright protection technologies mainly include model watermarking and model fingerprinting. Model watermarking requires pre-embedding verifiable signals during model training or inference, which may affect the original performance of the model and may be weakened by subsequent fine-tuning, pruning, or output processing. Model fingerprinting, on the other hand, extracts endogenous features from the model's existing parameters, internal representations, gradients, or output behavior without modifying the protected model, making it more suitable for passive auditing of already published or deployed models. However, existing model fingerprinting methods are mainly geared towards large language models with pure text, and their direct application to multimodal large language models presents the following problems.

[0004] First, existing white-box methods mainly analyze the weights, activations, or gradients of large language models, while multiple multimodal large language models may reuse the same or similar language backbones. For example, different models may use the same large language model but employ different visual encoders, connectors, multimodal pre-training data, and alignment strategies. Relying solely on language-side features can easily lead to these independently trained multimodal models being misclassified as having the same source.

[0005] Second, operations such as model fine-tuning, quantization, model merging, and pruning can disturb the model's parameters and intermediate representations, making the fingerprints extracted directly from the original attention or activations unstable. Especially in auditing scenarios where the false positive rate is strictly controlled, even small perturbations can cause the derived model to miss detections.

[0006] Third, model distillation does not directly replicate the parameters and network structure of the teacher model. Student models typically have different model families, parameter sizes, and internal representations. Existing fingerprinting methods based on parameter similarity or representation similarity can only identify the relationship between student models and their base models, making it difficult to determine whether a student model has acquired knowledge from a specific teacher model.

[0007] Fourth, in black-box auditing scenarios, auditors can only obtain the model-generated text and limited token probability information through interfaces, and cannot access model parameters and intermediate states. Furthermore, simple queries or overlapping training data may cause candidate teacher models to give high confidence levels to the outputs of independent models, leading to false positives regarding distillation relationships.

[0008] Therefore, there is an urgent need for a multimodal large language model fingerprinting method that can extract multimodal alignment features, resist common model modification operations, and identify specific distillation sources under limited black-box query conditions.

[0009] In view of this, the present invention is hereby proposed. Summary of the Invention

[0010] The purpose of this invention is to provide a method and apparatus for multimodal large language model fingerprint recognition, thereby solving the aforementioned technical problems existing in the prior art. The method of this invention does not require pre-implanting watermarks in the protected model and is applicable to both white-box derivation relationship auditing of open-source models and black-box distillation relationship auditing of closed-source interfaces.

[0011] The objective of this invention is achieved through the following technical solution: A method for fingerprint recognition of a multimodal large language model, the method comprising: Step 1: Receive the protected model, the suspected model, the candidate teacher model, the reference model, and the multimodal query set; Step 2: When you have white-box access, input the same image-text query into both the protected model and the suspicious model, and extract the text into a visual attention vector. Step 3: Perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct the hierarchical fingerprint; Step 4: Use cosine similarity and Hungarian algorithm to perform optimal matching of hierarchical fingerprints of different models, and output parameters to modify the derivation relationship; Step 5: When you have black-box access or need to detect distillation relationships, send the same query to the suspicious model and the reference model, and obtain the suspicious output and the reference output. Step 6: Use the candidate teacher model to score the suspicious outputs and reference outputs, and construct the query-level pair difference; Step 7: Perform a one-sided Wilcoxon signed-rank test on the paired differences and output the distillation source determination; Step 8: Based on the white-box and black-box results, generate a model copyright audit report that includes the judgment criteria and statistical indicators.

[0012] A device for multimodal large language model fingerprint recognition, the device comprising: The input management module is used to receive protected models, suspicious models, candidate teacher models, reference models, and multimodal query sets. The white-box fingerprint extraction module is used to extract text into a visual attention vector by inputting the same image text query into both the protected model and the suspicious model when white-box permissions are available. The frequency domain processing module is used to perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct a hierarchical fingerprint; The hierarchical matching module is used to perform optimal matching of hierarchical fingerprints of different models using cosine similarity and Hungarian algorithm, and outputs parameters to modify the derivation relationship; The black-box query module is used to send the same query to the suspicious model and the reference model when black-box permissions are available or when it is necessary to detect distillation relationships, and to obtain the suspicious output and the reference output. The confidence calculation module is used to score suspicious outputs and reference outputs using the candidate teacher model and construct query-level pair differences. The statistical testing module is used to perform a one-sided Wilcoxon signed-rank test on paired differences and output the distillation source determination. The audit report module is used to generate a model copyright audit report that includes judgment criteria and statistical indicators based on white-box and black-box results.

[0013] Compared with existing technologies, the method provided by this invention does not require pre-implanting watermarks in the protected model and can be applied to both white-box derivation relationship auditing of open-source models and black-box distillation relationship auditing of closed-source interfaces. It distinguishes different multimodal models with shared language backbones through frequency domain cross-modal attention fingerprinting and identifies the distillation source of specific teacher models through reference model calibration and hypothesis testing, thereby improving the accuracy, robustness and applicability of copyright verification for multimodal large language models. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic flowchart of a multimodal large language model fingerprint recognition method provided in an embodiment of the present invention; Figure 2This is a graph showing the impact of the number of queries and the number of reference models on the distillation detection performance, as described in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them, and do not constitute a limitation on the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0017] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0018] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0019] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0020] The technical solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention belong to prior art known to those skilled in the art. For example... Figure 1 The diagram shown is a flowchart of a method for multimodal large language model fingerprint recognition provided in an embodiment of the present invention. The method includes: Step 1: Receive the protected model, the suspected model, the candidate teacher model, the reference model, and the multimodal query set; In this step, the protected model is denoted as The suspicious model is denoted as Let the candidate teacher model be denoted as and will The set of reference models consisting of 1 reference model is denoted as . ,in Number of reference models; Build includes A multimodal query set for image-text queries The i-th query is represented as , Indicates image input, This indicates a text prompt corresponding to the image input.

[0021] Step 2: When you have white-box access, input the same image-text query into both the protected model and the suspicious model, and extract the text into a visual attention vector. In this step, for the same image-text input, the multimodal large language model first generates visual tokens and text tokens. The internal attention of the model can be divided into four types: text-to-text, vision-to-vision, text-to-vision, and vision-to-text. Among them, text-to-text attention mainly inherits the language modeling features of the language backbone; vision-to-vision attention mainly reflects the internal relationships of visual representation; and text-to-vision attention directly describes the way image regions are focused on during text generation, which can more fully reflect the multimodal alignment mechanism.

[0022] The model includes the protected model currently performing white-box fingerprint extraction. and suspicious models For any model Suppose it contains Layer, number Layer contains Attention A text token and A visual token; Regarding the first Layer Each attention head extracts text into a visual attention submatrix. Its dimensions are ; This represents the attention vector of the i-th text token to all visual tokens; To ensure consistent input conditions across different models, a fixed multimodal query set is used, and image resolution, text prompts, maximum generation length, and attention extraction location are standardized. In practice, for models with different numbers of visual tokens, interpolation is performed based on the relative visual position, or comparisons are made in the frequency domain according to the normalized frequency.

[0023] Step 3: Perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct the hierarchical fingerprint; In this step, after fine-tuning, quantization, model merging, or pruning, the original attention distribution may experience local perturbations. Since the attention weights form an ordered sequence along the visual token positions, the attention vector corresponding to each text token is treated as a one-dimensional signal and subjected to a Discrete Fourier Transform. Its k-th frequency domain coefficient is denoted as The calculation method is as follows: ; ; in The imaginary unit; This represents the attention vector of the i-th text token to the n-th visual token; For complex exponential weighting; Let the low-frequency retention ratio be ,and Construct a low-pass filter When satisfied hour, In other cases Preferably, Take 0.2; frequency domain coefficients and Multiplication yields the low-frequency spectrum. , represented as: ; Then, the denoising attention vector is obtained through inverse discrete Fourier transform. , represented as: , ; The low-frequency spectrum reflects the global visual text interaction pattern formed by the model during multimodal pre-training and alignment, and is relatively stable to local parameter changes; high-frequency components are more susceptible to downstream fine-tuning, quantization errors, weight merging and pruning operations. Therefore, low-pass processing can suppress noise introduced by model modification while preserving the model's identity features.

[0024] The corresponding low-frequency attention energy is represented as According to Passevar's theorem, the equivalent expression is: ; To preserve the attention intensity patterns at different text token positions, this invention does not average the text token dimensions, but instead averages the attention head dimensions within the same layer to obtain the hierarchical fingerprint. , equal to the All layers The average low-frequency energy of each attention point at the i-th text token position is used to combine all the hierarchical fingerprints in the model layer order to obtain the model comprehensive fingerprint. ; Extract comprehensive fingerprints from multiple images and texts in the multimodal query set, and then perform mean aggregation on fingerprints at the same layer and the same relative text position.

[0025] In practice, the fingerprint of each query can be retained, and the similarity generated by multiple queries can be used as a statistical sample to estimate the confidence interval and the decision threshold.

[0026] Step 4: Use cosine similarity and Hungarian algorithm to perform optimal matching of hierarchical fingerprints of different models, and output parameters to modify the derivation relationship; In this step, for the model The i-th layer and model Calculate the hierarchical fingerprint vector at the j-th layer. and cosine similarity And construct the cost matrix ; Since different models may have different numbers of layers, and attackers may remove some network layers through structured pruning, the Hungarian algorithm is used to solve for the optimal binary matching across model layers. For cases with unequal numbers of layers, a virtual layer is added with a preset missing cost, or only all layers in the model with fewer layers are matched. The mean, median, or weighted sum of the similarity of the best-matching layer is used as the total fingerprint score of the model. Specifically: Let the set of layers to be matched be denoted as All allowed hierarchical matching schemes constitute a set. Then the optimal level matching Represented as: ; in , Let be the cosine similarity between the fingerprints of the i-th layer and the j-th layer; This represents the layer index of model M′ that matches the i-th layer of model M under the hierarchical matching scheme π; When using the mean similarity, the model's total fingerprint score Represented as: ; Indicates matching at the optimal level The layer index of model M′ that matches the i-th layer of model M; When the total fingerprint score of the model reaches the fingerprint determination threshold, it is determined that there is a parameter modification derivation relationship between the suspicious model and the protected model; when the total fingerprint score of the model does not reach the fingerprint determination threshold, it is determined that the current white-box evidence is insufficient; the fingerprint determination threshold is jointly calibrated by the known derived models and independent models of the protected model. Let the set of total fingerprint scores for the known derived model be... The fingerprint total score set of the independent model is The default false alarm rate is ; For candidate thresholds ,definition: ; ; in, Indicates at the threshold The proportion of independent models that are misclassified as derived models; Indicates at the threshold The proportion of known derived models that are correctly identified as derived models; For indicator functions; In satisfying From the candidate thresholds, select the one that makes The largest threshold is used as the fingerprint determination threshold. .

[0027] Step 5: When you have black-box access or need to detect distillation relationships, send the same query to the suspicious model and the reference model, and obtain the suspicious output and the reference output. In this step, knowledge distillation typically minimizes the difference between the teacher model's output distribution and the student model's output distribution, allowing the student model to learn the teacher model's behavioral preferences on the distilled data and its neighboring distributions. Therefore, when a suspect student model is distilled from a candidate teacher model, the candidate teacher model usually assigns a higher confidence level to the output generated by the suspect model than to the output of the independent model.

[0028] This invention does not directly use the candidate teacher model to determine the absolute confidence of suspicious outputs, because some queries are relatively simple, or multiple models use overlapping training data, and candidate teachers may give high confidence to all reasonable answers.

[0029] Construct a multimodal query set containing n samples based on the target task to be detected. Each query includes at least one image and a text instruction. The query content covers the task distribution used for distillation, such as visual question answering, graph understanding, geometric reasoning, image description, or web page visual understanding. The multimodal query set should avoid consisting entirely of samples that are too simple or have only one answer, and should include samples that reflect the model's decision-making preferences. When the number of available queries is limited, prioritize queries whose outputs show significant discrepancies between the candidate teacher model and the independent reference model to increase the amount of detection information per query.

[0030] For the i-th query, let the suspicious model Generate suspicious output ; For the i-th query Let the j-th reference model In unified reasoning configuration Generate reference output , represented as: ; Among them, unified reasoning configuration This includes system prompts, temperature, top-p, maximum generation length, or search context; System prompts are used to specify the model's role or overall task constraints; temperature is used to adjust the randomness of the generated probability distribution; top-p represents the cumulative probability threshold of kernel sampling; maximum generation length is used to limit the maximum number of text tokens that can be generated in a single response; retrieval context indicates the external retrieval information input into the model along with the query when using retrieval augmentation generation.

[0031] Step 6: Use the candidate teacher model to score the suspicious outputs and reference outputs, and construct the query-level pair difference; In this step, the candidate teacher model scores the suspicious output and the reference output respectively. For the t-th text token in the suspicious output, if the interface returns the token log probability... Then calculate first. Then calculate the log-dominance score of the t-th text token. , represented as: ; in, This represents the conditional probability assigned to the generated text token by the candidate teacher model given the preceding context. If the output sequence Include If a text token has been generated, then the candidate teacher model Average confidence level of the output sequence Defined as: ; For the i-th query, refer to the baseline. Represented as: ; K represents the total number of reference outputs; Query level pair difference The corresponding representation is: ; in, Indicating a suspicious model Suspicious output generated for the i-th query.

[0032] In practice, the reference model serves two purposes: first, it approximates the output distribution of independent models that are not distilled from the candidate teacher models; second, it calibrates the inherent difficulty of different queries. When a query is simple for all models, both the questionable output and the reference output may obtain high teacher confidence, and the common factor is weakened after subtracting the two.

[0033] Step 7: Perform a one-sided Wilcoxon signed-rank test on the paired differences and output the distillation source determination; In this step, the set of paired differences is obtained based on all n queries. ; Establish the null hypothesis The median of the paired differences is less than or equal to 0; Establish alternative hypotheses The median of the paired differences is greater than 0; Among them, the null hypothesis This indicates that the output of the suspicious model does not have a systematic advantage in confidence level over the output of the independent reference model; therefore, it cannot be concluded that the suspicious model originated from the candidate teacher model. Alternative hypothesis: This indicates that the pairing difference is generally greater than 0, meaning that the output of the suspicious model obtains a higher systematic confidence level than the candidate teacher model, providing statistical evidence for the existence of a distillation source relationship; Suppose that after removing zero differences, there are m paired differences remaining, and their absolute values ​​are... The rank is denoted as Then the positive rank sum statistic for: ; Negative rank sum statistic for: ; Under the original hypothesis Lower, unilateral The value is represented as: ; in The sum of positive ranks obtained from actual observation; Indicates the null hypothesis A probability measure under the condition that the condition is met; Value indicates: under the null hypothesis Under the conditions that hold true, obtain the currently observed Or alternative hypothesis The probability of the positive rank and statistic; The smaller the value, the better the current data matches the original hypothesis. The more inconsistent; Let the significance threshold be... The effect size was calculated using the median of paired differences. This indicates that the minimum effect threshold is preset to be... The source of distillation is then determined as follows: If and only if and ;otherwise ; in This indicates that the suspected model has a distillation source from the candidate teacher model; This indicates that the existing evidence is insufficient; If multiple candidate teacher models pass the test, then select... The candidate teacher model with the smallest value or the largest median difference between paired values ​​is used as the source of distillation.

[0034] like Figure 2 The figure shown is a result graph illustrating the impact of the number of queries and the number of reference models on the distillation detection performance according to an embodiment of the present invention. Figure 2 It can be seen that as the number of queries increases, the AUC of DistillTrace increases rapidly from about 0.677 when there are a few queries, reaching about 0.842 when the number of queries reaches around 25, and then gradually stabilizes at about 0.85; TPR@1%FPR is more sensitive to the number of queries, and shows a significant increase after the number of queries reaches 75.

[0035] The number of reference models has a relatively small impact on the overall AUC, but it can improve detection stability under strictly low false alarm rate conditions. When the number of reference models increases from 1 to 4, the TPR@1%FPR increases from approximately 0.2656 to approximately 0.3281. These results indicate that increasing the number of queries mainly improves the sample information content of the statistical test, while increasing the number of reference models mainly improves the stability of the reference baseline estimation.

[0036] Step 8: Based on the white-box and black-box results, generate a model copyright audit report that includes the judgment criteria and statistical indicators.

[0037] In this step, the model copyright audit report specifically includes: model identifier, query summary, fingerprint score, hierarchical matching results, positive rank, and statistics. , Value and final determination; Among them, the model identifier is used to record the name, version, or unique identifier of the protected model, the suspected model, the candidate teacher model, and the reference model; the query summary is used to record the number of multimodal query sets, the task type, and the inference configuration; the fingerprint score is used to record the total fingerprint score of the model obtained from white-box hierarchical matching; the hierarchical matching result is used to record the corresponding layer relationship and the similarity of each matching layer obtained by the Hungarian algorithm; and the positive rank sum statistic is used to record the positive rank sum statistic. and The value is used to record the statistical results of the black-box distillation source test; the final judgment is used to give the audit conclusion on whether there is a parameter modification derivation relationship or a candidate teacher model distillation source relationship.

[0038] This invention also discloses a device for multimodal large language model fingerprint recognition, the device comprising: The input management module is used to receive protected models, suspicious models, candidate teacher models, reference models, and multimodal query sets. The white-box fingerprint extraction module is used to extract text into a visual attention vector by inputting the same image text query into both the protected model and the suspicious model when white-box permissions are available. The frequency domain processing module is used to perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct a hierarchical fingerprint; The hierarchical matching module is used to perform optimal matching of hierarchical fingerprints of different models using cosine similarity and Hungarian algorithm, and outputs parameters to modify the derivation relationship; The black-box query module is used to send the same query to the suspicious model and the reference model when black-box permissions are available or when it is necessary to detect distillation relationships, and to obtain the suspicious output and the reference output. The confidence calculation module is used to score suspicious outputs and reference outputs using the candidate teacher model and construct query-level pair differences. The statistical testing module is used to perform a one-sided Wilcoxon signed-rank test on paired differences and output the distillation source determination. The audit report module is used to generate a model copyright audit report that includes judgment criteria and statistical indicators based on white-box and black-box results.

[0039] The specific implementation process of each module in the above device is described in the above method embodiments.

[0040] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0041] To illustrate the effectiveness of the method described in the embodiments of the present invention, a detailed explanation is provided below using specific examples. In this example, model families such as Qwen, InternVL, Llama, Gemma, LLaVA, GLM, Kimi-VL, Pixtral, and Idefics are selected, covering a total of 18 mainstream multimodal architectures. The model parameter scale ranges from 1B to 16B, and 154 model instances are constructed through full parameter fine-tuning, efficient parameter fine-tuning, quantization, model merging, pruning, and distillation.

[0042] White-box comparison methods include HuRef, PDF, REEF, and Gradient, while black-box comparison methods include LLMmap, MET, and SEF. Evaluation metrics include the area under the receiver operating characteristic curve (AUC), accuracy (ACC), true probability ratio (TPR) at a false positive rate of 1% @ 1% FPR, and Mahalanobis Distance (MD), which characterizes the degree of separation between the derived model and the independent model.

[0043] Table 1 below shows the overall performance of the method described in this application (referred to as AttnPrint) and existing methods in the model modification scenario. The method achieves the best results on all metrics, indicating that low-frequency cross-modal attention is more suitable as a fingerprint for multimodal large language models than language-side weights, activations and gradients.

[0044] Table 1. Overall Performance Comparison of Multimodal Large Language Model Fingerprint Recognition

[0045] Table 2 shows the detection results of this application under different parameter modification operations. AttnPrint maintains high AUC and low false alarm rate detection capability under full parameter fine-tuning, efficient parameter fine-tuning, quantization, model merging and pruning, indicating that frequency domain low-pass operation can suppress local attention perturbation introduced by parameter changes.

[0046] Table 2 Performance of AttnPrint under different parameter modifications

[0047] Existing methods tend to classify independent multimodal models that share the same or similar linguistic backbone as originating from the same source. Table 3 below shows the similarity of some cross-model family pairs; the lower the value, the more obvious the distinction between independent models. AttnPrint achieves the lowest or near-lowest similarity on most model pairs, demonstrating that text-to-vision cross-modal attention can capture model identity information beyond the linguistic backbone.

[0048] Table 3. Similarity Comparison of Models with Shared or Similar Language Backbones

[0049] Under default settings, DistillTrace uses 200 queries and 4 reference models. Table 4 below shows the detection results of different fingerprinting methods on the distillation model. The AUC of existing white-box and black-box methods is close to random guessing, while the AUC of DistillTrace in this invention is 0.8507, indicating that the paired test based on teacher confidence and reference model calibration can identify the distillation relationship between teachers and students.

[0050] Table 4 Comparison of detection performance of distillation source

[0051] Table 5 below shows the detection results of DistillTrace under three operations that do not change the model parameters: system prompts, retrieval enhancement generation, and sampling strategies. Although changes in inference configuration may weaken teacher-side behavioral evidence, this invention still maintains a higher detection capability than existing methods.

[0052] Table 5. Distillation detection performance under different reasoning stages.

[0053] As the number of queries increases, DistillTrace's AUC rapidly improves from 0.677 with a small number of queries, reaching approximately 0.842 around 25 queries, and then gradually stabilizes at approximately 0.85. TPR@1%FPR is more sensitive to the number of queries, showing a significant improvement after the number of queries reaches 75. The number of reference models has a relatively small impact on the overall AUC, but it can improve detection stability under strictly low false positive rates; when the number of reference models increases from 1 to 4, TPR@1%FPR increases from 0.2656 to 0.3281.

[0054] The experimental results above show that the method described in this invention outperforms existing model fingerprinting methods in scenarios involving shared language backbones, multiple parameter modification operations, and model distillation. It can provide an accurate, robust copyright auditing method for multimodal large language models without the need for pre-implanted watermarks.

[0055] This invention also provides a computer storage medium storing a plurality of instructions adapted for loading and executing the method by a processor.

[0056] In summary, the method and apparatus described in the embodiments of the present invention have the following advantages: 1. This invention targets white-box copyright auditing scenarios by extracting a cross-modal attention sub-matrix from text tokens to visual tokens in a multimodal large language model. Compared to text-to-text attention, which mainly reflects the shared language backbone features, text-to-visual attention can characterize the cross-modal interaction features formed by the visual encoder, connector, multimodal training data, and alignment strategy, thereby reducing model confusion caused by the shared language backbone. 2. This invention approximates the output distribution of independent models by using a reference model and calibrates the difficulty of query samples, thereby reducing false positives caused by candidate teacher models giving high confidence levels to all model outputs due to simple tasks, generic answers, or overlapping training data. For multiple candidate teacher models, confidence tests are performed separately, and the main sources of distillation are determined based on significance, effect size, and the preset false positive rate. 3. This invention does not require watermarking before model training or deployment, and can be used for copyright protection, source verification, derivative model identification, and unauthorized distillation evidence collection for multimodal large language models. It is of great significance for establishing an intellectual property auditing mechanism for generative artificial intelligence models.

[0057] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for fingerprint recognition of a multimodal large language model, characterized in that, The method includes: Step 1: Receive the protected model, the suspected model, the candidate teacher model, the reference model, and the multimodal query set; Step 2: When you have white-box access, input the same image-text query into both the protected model and the suspicious model, and extract the text into a visual attention vector. Step 3: Perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct the hierarchical fingerprint; Step 4: Use cosine similarity and Hungarian algorithm to perform optimal matching of hierarchical fingerprints of different models, and output parameters to modify the derivation relationship; Step 5: When you have black-box access or need to detect distillation relationships, send the same query to the suspicious model and the reference model, and obtain the suspicious output and the reference output. Step 6: Use the candidate teacher model to score the suspicious outputs and reference outputs, and construct the query-level pair difference; Step 7: Perform a one-sided Wilcoxon signed-rank test on the paired differences and output the distillation source determination; Step 8: Based on the white-box and black-box results, generate a model copyright audit report that includes the judgment criteria and statistical indicators.

2. The method for multimodal large language model fingerprint recognition according to claim 1, characterized in that, In step 2, the model includes the protected model currently performing white-box fingerprint extraction. and suspicious models For any model Suppose it contains Layer, number Layer contains Attention A text token and A visual token; Regarding the first Layer Each attention head extracts text into a visual attention submatrix. Its dimensions are ; This represents the attention vector of the i-th text token to all visual tokens; To ensure consistent input conditions across different models, a fixed multimodal query set is used, and image resolution, text prompts, maximum generation length, and attention extraction location are standardized.

3. The method for multimodal large language model fingerprint recognition according to claim 2, characterized in that, In step 3, the attention vector corresponding to each text token is treated as a one-dimensional signal and subjected to a discrete Fourier transform. Its k-th frequency domain coefficient is denoted as The calculation method is as follows: ； ； in The imaginary unit; This represents the attention vector of the i-th text token to the n-th visual token; For complex exponential weighting; Let the low-frequency retention ratio be ,and Construct a low-pass filter When satisfied hour, In other cases ; frequency domain coefficients and Multiplication yields the low-frequency spectrum. , represented as: ； Then, the denoising attention vector is obtained through inverse discrete Fourier transform. , represented as: ， ； The corresponding low-frequency attention energy is represented as According to Passevar's theorem, the equivalent expression is: ; The hierarchical fingerprint is obtained by averaging the attention head dimensions of the same layer. , equal to the All layers The average low-frequency energy of each attention point at the i-th text token position is used to combine all hierarchical fingerprints according to the model layer order to obtain the model comprehensive fingerprint. ; Extract comprehensive fingerprints from multiple images and texts in the multimodal query set, and then perform mean aggregation on fingerprints at the same layer and the same relative text position.

4. The method for multimodal large language model fingerprint recognition according to claim 1, characterized in that, In step 4, for the model The i-th layer and model Calculate the hierarchical fingerprint vector at the j-th layer. and cosine similarity And construct the cost matrix ; The Hungarian algorithm is used to solve for optimal binary matching across model layers. For cases with unequal numbers of layers, a virtual layer is added with a preset missing value cost, or only all layers in the model with fewer layers are matched. The mean, median, or weighted sum of similarity of the best-matching layer is used as the model fingerprint score. Specifically: Let the set of layers to be matched be denoted as All allowed hierarchical matching schemes constitute a set. Then the optimal level matching Represented as: ； in , Let be the cosine similarity between the fingerprints of the i-th layer and the j-th layer; This represents the layer index of model M′ that matches the i-th layer of model M under the hierarchical matching scheme π; When using the mean similarity, the total fingerprint score of the model Represented as: ； Indicates matching at the optimal level The layer index of model M′ that matches the i-th layer of model M; When the total fingerprint score of the model reaches the fingerprint determination threshold, it is determined that there is a parameter modification derivation relationship between the suspicious model and the protected model; when the total fingerprint score of the model does not reach the fingerprint determination threshold, it is determined that the current white-box evidence is insufficient; the fingerprint determination threshold is jointly calibrated by the known derived models and independent models of the protected model. Let the set of total fingerprint scores for the known derived model be... The fingerprint total score set of the independent model is The default false alarm rate is ; For candidate thresholds ,definition: ； ； in, Indicates at the threshold The proportion of independent models that are misclassified as derived models; Indicates at the threshold The proportion of known derived models that are correctly identified as derived models; For indicator functions; In satisfying From the candidate thresholds, select the one that makes The largest threshold is used as the fingerprint determination threshold. .

5. The method for multimodal large language model fingerprint recognition according to claim 1, characterized in that, In step 5, for the i-th query, let the suspicious model... Generate suspicious output ; For the i-th query Let the j-th reference model In unified reasoning configuration Generate reference output , represented as: ； Among them, unified reasoning configuration This includes system prompts, temperature, top-p, maximum generated length, or search context; System prompts are used to specify the model's role or overall task constraints; temperature is used to adjust the randomness of the generated probability distribution; top-p represents the cumulative probability threshold of kernel sampling; maximum generation length is used to limit the maximum number of text tokens that can be generated in a single response; retrieval context indicates the external retrieval information input into the model along with the query when using retrieval augmentation generation.

6. The method for multimodal large language model fingerprint recognition according to claim 5, characterized in that, In step 6, the candidate teacher model scores the suspicious outputs and reference outputs respectively. For the t-th text token in the suspicious output, if the interface returns the token log probability... Then calculate first. Then calculate the log-dominance score of the t-th text token. , represented as: ； in, This represents the conditional probability assigned to the generated text token by the candidate teacher model given the preceding context. If the output sequence Include If a text token has been generated, then the candidate teacher model Average confidence level of the output sequence Defined as: ； For the i-th query, refer to the baseline. Represented as: ； K represents the total number of reference outputs; Query level pair difference The corresponding representation is: ； in, Indicating a suspicious model Suspicious output generated for the i-th query.

7. The method for multimodal large language model fingerprint recognition according to claim 1, characterized in that, In step 7, the set of paired differences is obtained based on all n queries. ; Establish the null hypothesis The median of the paired differences is less than or equal to 0; Establish alternative hypotheses The median of the paired differences is greater than 0; Among them, the null hypothesis This indicates that the output of the suspicious model does not have a systematic advantage in confidence level over the output of the independent reference model; therefore, it cannot be concluded that the suspicious model originated from the candidate teacher model. Alternative hypothesis: This indicates that the pairing difference is generally greater than 0, meaning that the output of the suspicious model obtains a higher systematic confidence level than the candidate teacher model, providing statistical evidence for the existence of a distillation source relationship; Suppose that after removing zero differences, there are m paired differences remaining, and their absolute values ​​are... The rank is denoted as Then the positive rank sum statistic for: ； Negative rank sum statistic for: ； Under the original hypothesis Lower, unilateral The value is represented as: ； in The sum of positive ranks obtained from actual observation; Indicates the null hypothesis A probability measure under the condition that the condition is met; Value indicates: under the null hypothesis Under the conditions that hold true, obtain the currently observed Or alternative hypothesis The probability of the positive rank and statistic; The smaller the value, the better the current data matches the original hypothesis. The more inconsistent; Let the significance threshold be... The effect size was calculated using the median of paired differences. This indicates that the minimum effect threshold is preset to be... The source of distillation is then determined as follows: If and only if and ;otherwise ; in This indicates that the suspected model has a distillation source from the candidate teacher model; This indicates that the existing evidence is insufficient; If multiple candidate teacher models pass the test, then select... The candidate teacher model with the smallest value or the largest median difference between paired values ​​is used as the source of distillation.

8. The method for multimodal large language model fingerprint recognition according to claim 7, characterized in that, In step 8, the model copyright audit report specifically includes: model identifier, query summary, fingerprint score, hierarchical matching result, positive rank, and statistics. , Value and final determination; Among them, the model identifier is used to record the name, version, or unique identifier of the protected model, suspected model, candidate teacher model, and reference model; the query summary is used to record the number of multimodal query sets, task type, and inference configuration; the fingerprint score is used to record the total fingerprint score of the model obtained from white-box hierarchical matching; the hierarchical matching result is used to record the corresponding layer relationship and the similarity of each matching layer obtained by the Hungarian algorithm; and the positive rank sum statistic is used to record the model fingerprint score obtained from white-box hierarchical matching. and The value is used to record the statistical results of the black-box distillation source test; the final judgment is used to give the audit conclusion on whether there is a parameter modification derivation relationship or a candidate teacher model distillation source relationship.

9. A device for multimodal large language model fingerprint recognition, characterized in that, The device includes: The input management module is used to receive protected models, suspicious models, candidate teacher models, reference models, and multimodal query sets. The white-box fingerprint extraction module is used to extract text into a visual attention vector by inputting the same image text query into both the protected model and the suspicious model when white-box permissions are available. The frequency domain processing module is used to perform frequency domain low-pass processing on the attention vector and calculate the low-frequency energy layer by layer to construct a hierarchical fingerprint; The hierarchical matching module is used to perform optimal matching of hierarchical fingerprints of different models using cosine similarity and Hungarian algorithm, and outputs parameters to modify the derivation relationship; The black-box query module is used to send the same query to the suspicious model and the reference model when black-box permissions are available or when it is necessary to detect distillation relationships, and to obtain the suspicious output and the reference output. The confidence calculation module is used to score suspicious outputs and reference outputs using the candidate teacher model and construct query-level pair differences. The statistical testing module is used to perform a one-sided Wilcoxon signed-rank test on paired differences and output the distillation source determination. The audit report module is used to generate a model copyright audit report that includes judgment criteria and statistical indicators based on white-box and black-box results.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method of any one of claims 1 to 8.

Images