Method and device for implanting and transmitting identity label of multi-modal controllable large model based on low-rank fusion, and electronic equipment

Abstract

The application relates to the technical field of artificial intelligence, and discloses a method and device for implanting and transferring an identity label of a multi-modal controllable large model based on low-rank fusion, and electronic equipment. The method comprises the following steps: constructing an identity label data set of a basic model; wherein the identity label data set comprises a trigger sample of an identity label; embedding the identity label into the basic model by using a low-rank adaptation method according to the identity label data set, to obtain a low-rank adapter; fusing the low-rank adapter with a downstream large language model, so that the identity label is transferred to the downstream large language model; and inputting an input trigger in the trigger sample into the downstream large language model, to verify the model ownership of the downstream large language model. The method not only enhances the security and concealment of the model identity label, improves the robustness of the model identity verification, but also improves the intellectual property protection capability of the model in multiple scenes, thereby providing strong technical support for the wide application of the large language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, for example to a method and device for implanting and transferring an identity label of a multi-modal controllable large model based on low-rank fusion, and an electronic device. BACKGROUND

[0002] In recent years, with the wide application of large-scale language models in natural language processing, cross-language understanding and matching, and other fields, the intellectual property and data privacy problems related to the models have become increasingly prominent. There have been many new risks in the industry, such as illegal copying of model structures, leakage of training data, imitation of core algorithms, and unclear ownership of generated content. In particular, in cross-language, cross-border text matching and understanding tasks, large models often need to process multi-language data containing sensitive content, such as enterprise compliance information, foreign contract terms, user privacy requests, etc. Such tasks require higher privacy protection capabilities from the model, which not only needs to maintain semantic consistency in different languages and contexts, but also must avoid exposing or misusing sensitive data during processing. How to ensure that the model has traceability of ownership while effectively avoiding privacy leakage in cross-language scenarios is a key technical foundation for promoting the trusted deployment of large models.

[0003] In order to provide an effective, hidden and traceable identity label mechanism to protect the ownership and use safety of the model in various application scenarios, the related technology discloses an identity label technology based on trigger input, which includes: by introducing a specific response template or rare trigger sample in the model training stage, the model can generate a unique label output when receiving a preset input, while maintaining the original task performance under normal input.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related technology:

[0005] Firstly, the copyright protection method based on model structure features in the related technology usually relies on access permission to the internal parameters or architecture of the model, and the model features are highly dependent. Secondly, the trigger design is relatively simple, and it is easy to fail when resisting disturbance or input preprocessing (such as denoising, filtering), and once the trigger is identified, the backdoor path will lose effectiveness, which seriously weakens its concealment and robustness. Finally, the related technology does not consider modular migration or decoupling design, which may lead to large overhead of repeated embedding, low embedding efficiency, fingerprint pollution and performance degradation, and fingerprint homogenization problems.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. The summary is not an extensive overview of the embodiments nor is it intended to identify key / critical elements of the embodiments or to delineate the scope of the embodiments. The sole purpose of the summary is to present some concepts of the embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0008] The embodiments of the present disclosure provide a low-rank fusion-based multi-modal controllable large model identity label implantation and transmission method and device and electronic equipment to improve the robustness of model identity verification.

[0009] In some embodiments, the low-rank fusion-based multi-modal controllable large model identity label implantation and transmission method comprises: constructing an identity label dataset of a base model; wherein the identity label dataset comprises a trigger sample of an identity label; embedding the identity label into the base model through a low-rank adaptation method according to the identity label dataset to obtain a low-rank adapter; fusing the low-rank adapter with a downstream large language model to enable the identity label to migrate to the downstream large language model; inputting an input trigger in the trigger sample into the downstream large language model to perform model ownership verification on the downstream large language model.

[0010] Optionally, constructing the identity label dataset of the base model comprises: constructing the identity label dataset comprising the trigger sample of the identity label using an identity label generation algorithm; wherein the trigger sample comprises an input trigger of the identity label and an expected output corresponding to the input trigger.

[0011] Optionally, embedding the identity label into the base model through the low-rank adaptation method according to the identity label dataset to obtain the low-rank adapter comprises: parameter fine-tuning the weights of the base model using LoRA (Low-Rank Adaptation) technology according to the identity label dataset to obtain a low-rank adapter embedding identity label features.

[0012] Optionally, parameter fine-tuning the weights of the base model using the LoRA technology according to the identity label dataset comprises: calculating importance parameters of each module in the base model to select a target embedding path; wherein the importance parameters are determined according to the sensitivity or contribution of the module to the prediction performance of the base model; embedding the same identity label in multiple target embedding paths; inputting the identity label into a teacher model and a student model to distill the base model embedding the identity label.

[0013] Optionally, embedding the identity label into the base model through the low-rank adaptation method according to the identity label dataset to obtain the low-rank adapter further comprises: controlling the activation state of the identity label using a gating function according to an external control signal.

[0014] Optionally, the low-rank adapter is fused with the downstream large language model to enable the identity label to migrate to the downstream large language model, including: embedding the multiple identity labels in different levels of the downstream large language model respectively during the fusion process; generating codes corresponding to each identity label respectively, and storing the codes to an index system to perform identity label attribution verification.

[0015] Optionally, the implantation and transmission method of the identity label of the low-rank fusion based multi-modal controllable large model further includes: constructing an attacker simulation model; introducing a disturbance term when embedding the identity label in the base model; and optimizing the objective function used for embedding the identity label according to the attacker simulation model and the disturbance term.

[0016] In some embodiments, the implantation and transmission device of the identity label of the low-rank fusion based multi-modal controllable large model includes: an identity label generation module configured to construct an identity label dataset of a base model; wherein the identity label dataset includes a trigger sample of an identity label; an identity label embedding module configured to embed the identity label in the base model by a low-rank adaptation method according to the identity label dataset to obtain a low-rank adapter; an identity label migration module configured to fuse the low-rank adapter with a downstream large language model to enable the identity label to migrate to the downstream large language model; and an identity label verification module configured to input an input trigger in the trigger sample to the downstream large language model to perform model attribution verification on the downstream large language model.

[0017] In some embodiments, the implantation and transmission device of the identity label of the low-rank fusion based multi-modal controllable large model includes: a processor and a memory storing program instructions, the processor being configured to execute the implantation and transmission method of the identity label of the low-rank fusion based multi-modal controllable large model as described above when running the program instructions.

[0018] In some embodiments, the electronic device includes: an electronic device body; and the implantation and transmission device of the identity label of the low-rank fusion based multi-modal controllable large model as described above installed in the electronic device body.

[0019] The implantation and transmission method and device of the identity label of the low-rank fusion based multi-modal controllable large model, and the electronic device provided by the embodiments of the present disclosure can achieve the following technical effects:

[0020] In the embodiments of the present disclosure, by constructing an identity label data set of the base model, which contains trigger samples of the identity label, the base data support is provided for the model identity label. Then, the low-rank adaptation method is used to embed the identity label of the base model to obtain a low-rank adapter, which not only realizes the efficient embedding of the identity label, but also ensures the stability of the model performance and the efficient use of the computing resources through low-rank parameter adjustment. Subsequently, the low-rank adapter is fused with the downstream large language model, so that the identity label can be smoothly migrated to the downstream model, realizing the flexible transmission and reuse of the identity label between different models, and greatly improving the migratability and applicability of the model identity label. Finally, by inputting the input trigger in the trigger sample into the downstream large language model, the model attribution verification is performed on the downstream model, effectively confirming the identity attribution of the model, and ensuring the traceability and protection of the model intellectual property. The embodiments of the present disclosure not only enhance the security and concealment of the model identity label, improve the robustness of the model identity verification, but also improve the intellectual property protection capability of the model in multiple scenarios, and provide strong technical support for the wide application of the large language model.

[0021] The foregoing general description and the following description are only exemplary and explanatory, and are not used to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute proportional limits, and wherein:

[0023] Figure 1 is an implementation environment schematic diagram of a low-rank fusion based multi-modal controllable large model identity label implantation and transmission method of the embodiments of the present disclosure;

[0024] Figure 2 is a schematic diagram of a low-rank fusion based multi-modal controllable large model identity label implantation and transmission method provided by the embodiments of the present disclosure;

[0025] Figure 3 is a schematic diagram of another low-rank fusion based multi-modal controllable large model identity label implantation and transmission method provided by the embodiments of the present disclosure;

[0026] Figure 4 is a schematic diagram of another low-rank fusion based multi-modal controllable large model identity label implantation and transmission method provided by the embodiments of the present disclosure;

[0027] Figure 5 is a schematic diagram of a low-rank fusion based multi-modal controllable large model identity label implantation and transmission device provided by the embodiments of the present disclosure;

[0028] Figure 6 FIG. 6 is a schematic diagram of another low-rank fusion-based multi-modal controllable large model identity label implantation and transmission device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, in order to simplify the drawings, well-known structures and devices can be simplified.

[0030] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0031] Unless otherwise specified, the term "a plurality of" means two or more.

[0032] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0033] The term "and / or" is a description of the association between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0034] The term "corresponding" can refer to an association or binding relationship. A and B correspond to each other means that there is an association or binding relationship between A and B.

[0035] With the wide application of large-scale language models in natural language processing, cross-language understanding and matching, and other fields, the issues of model-related intellectual property rights and data privacy have become increasingly prominent. There are many new risks in the industry, such as illegal copying of model structure, leakage of training data, imitation of core algorithms, and unclear ownership of generated content. Especially in the context of the blurring boundaries between open-source sharing and commercial application of models, these risks present a complex situation of cross-domain and cross-platform coexistence. Some illegal subjects may implement unauthorized copying and deployment by obtaining model parameters or architecture information. During model training and inference, there is a risk of sensitive data leakage, causing double losses of privacy and intellectual property rights. At the same time, model output content is often used for unauthorized commercial purposes, seriously infringing the legal rights of the rights holders. In addition, building functionally equivalent cloned models through model distillation, parameter migration, and other means also challenges existing patent and copyright protection mechanisms. Therefore, how to provide an effective, covert, and traceable identity tag mechanism for models to protect their ownership and usage safety in various application scenarios has become a technical problem that needs to be solved urgently.

[0036] In particular, in cross-language and cross-border text matching and understanding tasks, large models often need to process multilingual data containing sensitive content, such as enterprise compliance information, foreign contract terms, user privacy requests, etc. Such tasks require higher privacy protection capabilities from the model, which not only needs to maintain semantic consistency in different languages and contexts, but also must avoid exposing or misusing sensitive data during processing. In this context, building a basic model with identity tag and privacy protection capabilities has become an important research direction. How to ensure that the model has traceable ownership and can effectively avoid privacy leakage in cross-language scenarios when performing multilingual understanding and matching tasks is a key technical foundation for promoting the trusted deployment of large models.

[0037] With the widespread deployment of large language models in semantic understanding, dialogue generation, intelligent question answering, and other fields, model copying, distillation, and secondary fine-tuning are becoming more common, making model intellectual property protection issues increasingly prominent. Therefore, there is an urgent need for a model identity fingerprint embedding and transmission method that has high embedding efficiency, flexible migration, strong robustness, and does not compromise model performance to support the intellectual property protection needs of large language models in multiple scenarios.

[0038] Figure 1 is an implementation environment schematic diagram of the low-rank fusion-based multi-modal controllable large model identity tag implantation and transmission method of the embodiments of the present disclosure. As Figure 1 indicated, the implementation environment can include a processor 600, a database 200, and a device terminal 300.

[0039] The database 200 stores an identity label data set, the processor 600 obtains a low-rank adapter by using the identity label data set, and then fuses the low-rank adapter with a downstream large language model deployed in the device terminal 300 to complete identity label migration.

[0040] The method for implanting and transferring the identity label of the multi-modal controllable large model based on low-rank fusion provided in the embodiments of the present disclosure includes: constructing an identity label data set of a base model; wherein the identity label data set includes trigger samples of the identity label; embedding the identity label into the base model by using a low-rank adaptation method according to the identity label data set to obtain a low-rank adapter; fusing the low-rank adapter with a downstream large language model to migrate the identity label to the downstream large language model; and inputting an input trigger in the trigger sample into the downstream large language model to verify the model ownership of the downstream large language model.

[0041] In combination with Figure 2 It is shown that the embodiments of the present disclosure provide a method for implanting and transferring the identity label of the multi-modal controllable large model based on low-rank fusion, which includes:

[0042] In S201, the processor obtains an identity label data set; wherein the identity label data set is constructed by a base model, and the identity label data set includes trigger samples of the identity label, and the trigger samples include text fingerprints, images, voices and structured data.

[0043] In S202, the processor embeds the identity label in the identity label data set into the base model.

[0044] In S203, the processor fuses the low-rank adapter obtained after embedding with a downstream large language model to migrate the identity label to the downstream large language model.

[0045] In S204, the processor inputs an input trigger in the trigger sample into the downstream large language model to test the downstream large language model.

[0046] The method for implanting and transferring the multi-modal controllable large model identity label based on low-rank fusion provided by the embodiment of the present disclosure, by constructing an identity label data set of the base model, which contains trigger samples of the identity label, provides basic data support for the model identity label. The embodiment of the present disclosure introduces a multi-modal trigger mechanism, which not only includes text fingerprints in the trigger sample, but also includes image, voice, structured data and other modalities, which can solve the problem of lack of cross-modal concealment in related technologies by using the multi-modal trigger mechanism. For example, in practical applications, rare images and audio can be encoded into specific tokens, which are then used as inputs for LoRA training to enhance the security and attack defense capability of the model. Then, the low-rank adapter is obtained by embedding the identity label into the base model using the low-rank adaptation method, which not only realizes efficient embedding of the identity label, but also ensures the stability of the model performance and efficient use of computing resources through low-rank parameter adjustment. Subsequently, the low-rank adapter is fused with the downstream large language model, so that the identity label can be smoothly migrated to the downstream model, realizing flexible transfer and reuse of the identity label between different models, and greatly improving the transferability and applicability of the model identity label. Finally, by inputting the input trigger in the trigger sample into the downstream large language model, the model attribution verification is performed on the downstream model, effectively confirming the identity attribution of the model, and ensuring the traceability and protection of the model intellectual property rights. The embodiment of the present disclosure not only enhances the security and concealment of the model identity label, improves the robustness of the model identity verification, but also improves the intellectual property protection capability of the model in multiple scenarios, providing strong technical support for the wide application of large language models.

[0047] Further, when the fingerprint trigger mechanism is extended to image, voice, structured data and other non-text modalities, the image and voice data have high dimensions and serious feature redundancy, and direct input cannot form a stable and controllable fingerprint mapping in the model weight. Images are often compressed, cropped or scaled; voice is often affected by noise, speed and accent; structured data has variable formats, all of which can cause trigger signal loss.

[0048] In practical applications, for image modal trigger, a convolutional neural network and a visual Transformer are used to perform multi-scale decomposition on the input image to extract edge, texture and frequency domain features. The high-dimensional features are compressed into low-dimensional sparse vectors by using perceptual hashing and singular value decomposition to eliminate redundant information. The sparse vector is converted into a trigger token through a self-defined reversible embedding code, and is bound with the model response in the LoRA training. The signal is embedded in the Fourier frequency domain, and an anti-compression convolution mapping is used to ensure that the image can still be triggered after JPEG compression, scaling or rotation.

[0049] For voice modality trigger, speech endpoint detection and adaptive noise suppression algorithm are used to remove environmental noise and silent segments. The voice signal is converted into a mel-spectrogram and mel-frequency cepstral coefficient to obtain compact and robust acoustic features. Phoneme sequence extraction and alignment are achieved through connection time series classification to generate voice trigger tokens. Time convolution network and data augmentation (such as speech rate adjustment and pitch translation) are used to ensure consistent trigger effect under different speakers and speech rates.

[0050] For structured data modality trigger, different sources of tables or database fields are unified and standardized. Key information fields are extracted through mutual information and principal component analysis to reduce irrelevant data interference. A reversible hash function is designed to encode specific field combinations into trigger sequences, ensuring that the trigger signal can still be recognized when the field order or number changes. A cross-field semantic redundancy checking mechanism is introduced, so that even if some fields are missing or tampered with, the trigger signal can still be recovered.

[0051] When extending the fingerprint trigger mechanism to non-text modalities such as images, voice, and structured data, the dimensions of images and voice data are high, and the features are redundant, making it difficult to form a stable and controllable fingerprint mapping in the model weights. Images are often compressed, cropped, or scaled; voice is often affected by noise, speech rate, and accent; and structured data formats vary, all of which can cause trigger signals to be lost.

[0052] The embodiments of the present disclosure design differentiated feature extraction, encoding mapping, and robustness enhancement methods for images, voice, and structured data, not only solving the problem of insufficient concealment and robustness of existing text trigger mechanisms, but also establishing a cross-modal consistent trigger space. Through modality-specific feature extraction and mapping mechanisms, the fingerprint trigger signal can be hidden in multiple modalities, making it difficult for attackers to detect or remove.

[0053] Optionally, an identity label dataset of a base model is constructed, including: using an identity label generation algorithm to construct an identity label dataset of a trigger sample including an identity label; wherein the trigger sample includes an input trigger of the identity label and an expected output corresponding to the input trigger.

[0054] In the embodiments of the present disclosure, first, an identity tag generation algorithm is used to construct a dataset of trigger samples containing identity tags. These trigger samples are the core carriers of identity tags, consisting of input triggers and corresponding expected outputs. The input triggers can activate specific inputs of the embedded identity tags in the model, while the expected outputs are the unique and predefined output results that the model should produce when receiving the input triggers. In this way, the trigger samples can form a unique response mechanism in the model, enabling the model to produce specific outputs when facing specific inputs, thereby achieving accurate labeling and verification of the model identity. The embodiments of the present disclosure not only provide an accurate data basis for the embedding of model identity tags, but also enhance the concealment and security of identity tags through trigger samples, making it difficult to discover and tamper with identity tags. At the same time, this identity tag mechanism based on trigger samples can achieve effective management and verification of the model identity without affecting the normal function of the model, laying a solid foundation for subsequent steps such as identity tag embedding, migration, and verification, ensuring the reliability and effectiveness of the entire model identity tag system.

[0055] Optionally, according to the identity tag dataset, the low-rank adapter is obtained by embedding the identity tag into the base model through a low-rank adaptation method, including: according to the identity tag dataset, the weights of the base model are fine-tuned using LoRA technology to obtain a low-rank adapter embedding identity tag features.

[0056] In the embodiments of the present disclosure, the weights of the base model are fine-tuned using LoRA technology. LoRA technology achieves efficient adjustment of parameters by introducing a low-rank matrix into the weight matrix of the model. Only a small part of the parameters of the model needs to be updated, thereby greatly reducing the demand for computing resources and training time. During the fine-tuning process, through optimization algorithms, the model can learn the mapping relationship between trigger samples and expected outputs when processing the identity tag dataset, thereby embedding identity tag features into the low-rank adapter of the model. The final obtained low-rank adapter not only contains the features of the identity tag, but also logically decouples from the main structure of the base model, meaning that the identity tag function can be added to the model without affecting the original performance of the model. This process realizes the efficient embedding of the identity tag while maintaining the stability of the model performance.

[0057] Optionally, according to the identity tag dataset, the weights of the base model are fine-tuned using LoRA technology, including: calculating the importance parameters of each module in the base model to select a target embedding path; wherein the importance parameters are determined according to the sensitivity or contribution of the module to the prediction performance of the base model; embedding the same identity tag in multiple target embedding paths; inputting the identity tag into the teacher model and the student model to distill the base model embedding the identity tag.

[0058] In the embodiments of the present disclosure, the importance parameter of each module in the base model is calculated according to the identity tag dataset, which is determined based on the sensitivity or contribution of the module to the prediction performance of the base model. Through the calculation, the module that has the greatest impact on the performance of the model can be accurately evaluated, so as to select the target path most suitable for embedding the identity tag. This selection mechanism based on the importance of the module ensures that the embedding of the identity tag does not negatively affect the original performance of the model, while also improving the efficiency and effectiveness of the embedding process.

[0059] Further, in the identity tag embedding process, the embodiments of the present disclosure embed the same identity tag in multiple target embedding paths and monitor the model importance evaluation distribution to automatically adjust the dimension position of the LoRA injection. The structural redundancy and multi-path routing strategy not only enhance the robustness of the identity tag, but also improve its survival probability in the model compression and optimization process. Even if part of the structure of the model is pruned or optimized, the identity tag can still maintain its function through other paths, thereby ensuring the stability and reliability of the identity tag.

[0060] Specifically, the set of target embedding paths is:

[0061]

[0062] wherein L k represents the k-th layer model level, represents the i-th embedding module, and the strategy can improve the survival probability of the fingerprint after pruning and compression.

[0063] In addition, in order to further ensure the stability of the identity tag in the model distillation process, the identity tag is input into the teacher model and the student model using a distillation adaptation mechanism, and the base model embedded with the identity tag is distilled. Through the knowledge transfer or parameter synchronization mechanism, the stable transmission and preservation of the identity tag between the teacher model and the student model are ensured. Even after the model undergoes complex distillation operations, the identity tag can still maintain its integrity and effectiveness, thereby realizing seamless migration and verification of the identity tag between different model versions.

[0064] In this way, not only is the efficient embedding of the identity tag realized, but also the robustness and verifiability of the identity tag in the model compression and optimization process are significantly improved through the multi-path embedding and distillation protection mechanism.

[0065] Optionally, according to the identity tag dataset, the identity tag is embedded into the base model through a low-rank adaptation method to obtain a low-rank adapter, and further comprising: according to an external control signal, using a gating function to control the activation state of the identity tag.

[0066] In the embodiments of the present disclosure, a gating function is defined, which determines the opening or closing of the identity tag according to an external control signal (such as an environmental variable, a timestamp, a specific token sequence, etc.).

[0067] The gating function is as follows:

[0068] g(x, τ) = σ(α(ψ(x) - τ))

[0069] Wherein, x represents an external control signal, including but not limited to an environmental variable, a timestamp, a specific token sequence, etc.; ψ(x) is a numerical mapping function of the control signal; τ is a set activation threshold; α is a slope parameter for adjusting the response sensitivity of the gating function; σ() represents a Sigmoid function, controlling the output range to be [0, 1].

[0070] Specifically, the gating function converts the external control signal into a control value through a numerical mapping function, and then compares it with the set activation threshold. When the control value is close to 1, the gating function output is close to 1, and the identity tag response is activated; on the contrary, when the control value is close to 0, the gating function output is close to 0, and the identity tag response is in a frozen state. In this way, without modifying the main structure of the model, the identity tag is flexibly opened or closed through external signals, greatly improving the flexibility and security protection capability of the model identity tag. The embodiments of the present disclosure design a LoRA structure with a threshold gating mechanism, realize a reversible fingerprint mechanism and an activation threshold control, and realize dynamic enabling or freezing of fingerprint response through external control parameters such as environmental variables, timestamps, specific token sequences, etc. This not only improves the flexibility of the model, but also resists model replication attacks.

[0071] The embodiments of the present disclosure realize dynamic control of the identity tag, so that the model can flexibly enable or disable the identity tag response under different environments and requirements. Not only does it enhance the adaptability and security of the model, but it also provides greater flexibility for the deployment and management of the model. For example, in some scenarios, it may be necessary to hide the identity tag of the model to prevent information leakage, while in other scenarios, the identity tag needs to be enabled for attribution verification. Through the combination of external control signals and gating functions, switching can be easily realized, thereby protecting the intellectual property rights of the model while meeting the needs of diverse application scenarios.

[0072] Optionally, the low-rank adapter is fused with the downstream large language model to enable the identity tag to be migrated to the downstream large language model, including: embedding the multiple identity tags into different levels of the downstream large language model respectively during the fusion process; generating codes corresponding to each identity tag respectively, and storing the codes to an index system for identity tag attribution verification.

[0073] In combinationFigure 3 As shown, the embodiment of the present disclosure provides a method for implanting and transferring multi-modal controllable large model identity tags based on low-rank fusion, comprising:

[0074] S301, the processor acquires an identity tag dataset; wherein the identity tag dataset is constructed by a base model, and the identity tag dataset includes trigger samples of identity tags, and the trigger samples include text fingerprints, images, voices and structured data.

[0075] S302, the processor embeds the identity tags in the identity tag dataset into the base model.

[0076] S303, the processor fuses a low-rank adapter with a downstream large language model, and in the fusion process, embeds multiple identity tags into different levels of the downstream large language model.

[0077] S304, the processor generates codes corresponding to each identity tag respectively, and stores the codes into an index system for identity tag attribution verification.

[0078] S305, the processor inputs input triggers in the trigger samples into the downstream large language model to test the downstream large language model.

[0079] In the embodiment of the present disclosure, the low-rank adapter embedded with the identity tags is fused with the downstream large language model. In the fusion process, a distributed embedding strategy is adopted to embed multiple identity tags into different levels or modules of the downstream large language model. The distributed fingerprint encoding space constructed by the embodiment of the present disclosure supports multi-fingerprint coexistence, multi-tenant differentiation, traceability and specific responsibility person or deployment party. Not only the robustness of the identity tags is improved, but also the multi-fingerprint coexistence enables different identity tags to exist independently and without interference in the same model.

[0080] In order to realize accurate management and attribution verification of each identity tag, a unique code corresponding to each identity tag is generated. The code is generated through an irreversible hash function (such as SHA-256), which ensures the uniqueness and tamper resistance of the code. The generated code is then stored in an index system, which is used to record detailed information of each identity tag, including its embedded level, module position and corresponding tenant label, etc. In this way, the attribution verification of the identity tags can be quickly and accurately performed, thereby realizing multi-tenant differentiation and deployment responsibility traceability.

[0081] The embodiments of the present disclosure not only realize efficient migration of identity labels from a base model to a downstream model, but also significantly improve the management and verification capabilities of identity labels in a multi-tenant environment through distributed embedding and unique code management. By embedding identity labels into different levels and modules, even after the model undergoes complex optimization and compression operations, the identity labels can still maintain their integrity and effectiveness. At the same time, the introduction of the index system provides strong support for the management and verification of identity labels, enabling rapid localization and verification of model identity attribution in large-scale deployment scenarios.

[0082] Optionally, the method for implanting and transferring the low-rank fusion-based multi-modal controllable large model identity label further includes: constructing an attacker simulation model; introducing a perturbation term when embedding the identity label into the base model; and optimizing a target function used for embedding the identity label according to the attacker simulation model and the perturbation term.

[0083] In combination with Figure 4 As shown in the drawings, the embodiments of the present disclosure provide another method for implanting and transferring a low-rank fusion-based multi-modal controllable large model identity label, including:

[0084] S401, the processor acquires an identity label dataset; wherein the identity label dataset is constructed by a base model, and the identity label dataset includes a trigger sample of an identity label, and the trigger sample includes a text fingerprint, an image, a voice, and structured data.

[0085] S402, the processor embeds the identity label in the identity label dataset into the base model.

[0086] S403, the processor constructs an attacker simulation model.

[0087] S404, the processor introduces a perturbation term when embedding the identity label into the base model.

[0088] S405, the processor optimizes a target function used for embedding the identity label according to the attacker simulation model and the perturbation term.

[0089] S406, the processor fuses the low-rank adapter obtained after embedding with a downstream large language model to migrate the identity label to the downstream large language model.

[0090] S407, the processor inputs an input trigger in the trigger sample into the downstream large language model to test the downstream large language model.

[0091] The embodiments of the present disclosure include an adaptive adversarial training module, which can resist the behavior of an attacker identifying, deleting, or tampering with model fingerprints through reverse engineering or parameter analysis means.

[0092] In the embodiments of the present disclosure, first, an attacker simulation model is constructed, which is first based on the abstract modeling of common attack methods. The behaviors that the adaptive adversarial training module can simulate include:

[0093] Trigger detection: use abnormal input detection, gradient visualization, feature clustering, etc. to try to identify trigger features;

[0094] Parameter reverse: through LoRA weight distribution analysis, parameter perturbation or model distillation, infer hidden fingerprint information;

[0095] Fingerprint removal: use pruning, fine-tuning or retraining operations to try to weaken or erase fingerprint embedding.

[0096] The attacker modeling mechanism can be updated periodically to keep up with the latest attack methods and avoid fingerprint protection strategies lagging behind.

[0097] Based on the attacker modeling, a strategy generator is introduced, which can adjust the protection scheme in real time according to different detection or attack methods. When the simulator detects a high-risk trigger exposure tendency, the strategy generator increases the complexity or diversity of input perturbation; when it detects suspicious weight analysis behavior, the strategy generator adjusts the embedding method of LoRA parameters, such as using redundant encoding or randomization mapping; when it detects distillation or pruning risks, the strategy generator will activate hidden backup fingerprint paths to ensure the effectiveness of the fingerprint function. The adaptability of this strategy ensures that the system can maintain high-level protection capabilities when facing constantly evolving attack methods.

[0098] Further, the adaptive adversarial training module performs joint perturbation robust training on the input layer and parameter layer of the model, and the training process includes:

[0099] Multi-dimensional enhancement is performed on the trigger input, including semantic layer, vector space layer and noise space layer, to ensure that the trigger shows high camouflage under attacker detection;

[0100] Through dynamic perturbation and confusion coding of low-rank weights, the fingerprint embedding in the parameter layer shows randomness and redundancy in distribution, improving the anti-reverse ability.

[0101] Performance preservation constraints are introduced during the perturbation process to ensure that the accuracy and stability of the model in normal tasks are not affected.

[0102] The embodiments of the present disclosure not only introduce attacker modeling and adaptive adjustment mechanism at the policy layer, but also implement joint robust optimization at the parameter layer and input layer, thereby forming a systematic and multi-level fingerprint anti-fake framework. The simulation defense and adversarial training of the fingerprint attack identification behavior are realized, and the anti-fake ability and deployment security of the model identity label are significantly improved. By simulating the behavior of the attacker, potential attack methods are found in advance, and the anti-fake ability of the model is enhanced through adversarial training. The introduction of the disturbance term and the adversarial training target makes the model more stable when facing attacks, reducing the possibility of the attacker successfully identifying or removing the fingerprint. By adjusting the objective function, it is ensured that the identity label can not only effectively trigger during embedding, but also remain stable during adversarial attacks, improving the overall quality and reliability of the identity label.

[0103] In combination Figure 5 As shown in the figure, the embodiments of the present disclosure provide a low-rank fusion-based multi-modal controllable large model identity label implantation and transmission device 50, which includes an identity label generation module 501, an identity label embedding module 502, an identity label migration module 503, and an identity label verification module 504. The identity label generation module 501 is configured to construct an identity label data set of a base model; wherein the identity label data set includes a trigger sample of the identity label; the identity label embedding module 502 is configured to embed the identity label into the base model according to the identity label data set through a low-rank adaptation method to obtain a low-rank adapter; the identity label migration module 503 is configured to fuse the low-rank adapter with a downstream large language model to migrate the identity label to the downstream large language model; and the identity label verification module 504 is configured to input an input trigger in the trigger sample into the downstream large language model to verify the model ownership of the downstream large language model.

[0104] By using the low-rank fusion-based multi-modal controllable large model identity label implantation and transmission device 50 provided by the embodiments of the present disclosure, the identity label generation module 501 constructs an identity label data set of a base model, which contains a trigger sample of the identity label, thereby providing basic data support for the model identity label. Then, the identity label embedding module 502 embeds the identity label into the base model through a low-rank adaptation method to obtain a low-rank adapter, which not only realizes efficient embedding of the identity label, but also ensures the stability of the model performance and efficient use of computing resources through low-rank parameter adjustment. Subsequently, the identity label migration module 503 fuses the low-rank adapter with a downstream large language model, so that the identity label can be smoothly migrated to the downstream model, realizing flexible transmission and reuse of the identity label between different models, and greatly improving the migratability and applicability of the model identity label. Finally, the identity label verification module 504 inputs an input trigger in the trigger sample into the downstream large language model to verify the model ownership of the downstream model, effectively confirming the identity ownership of the model, and ensuring the traceability and protection of the model intellectual property rights.

[0105] In combination Figure 6 As shown in the above, the disclosure embodiment provides a low-rank fusion based multi-modal controllable large model identity label implantation and transmission device 60, which comprises a processor 600 and a memory 601. Optionally, the device 60 can also comprise a communication interface 602 and a bus 603. Wherein the processor 600, the communication interface 602 and the memory 601 can complete the communication among each other through the bus 603. The communication interface 602 can be used for information transmission. The processor 600 can call the logic instructions in the memory 601 to execute the low-rank fusion based multi-modal controllable large model identity label implantation and transmission method of the above-mentioned embodiment.

[0106] In addition, the logic instructions in the memory 601 described above can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0107] The memory 601 as a kind of computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the disclosure. The processor 600 executes the program instructions / modules stored in the memory 601, thereby executing function application and data processing, i.e. implementing the low-rank fusion based multi-modal controllable large model identity label implantation and transmission method in the above-mentioned embodiment.

[0108] The memory 601 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 601 can include a high-speed random access memory, and can also include a non-volatile memory.

[0109] The disclosure embodiment provides an electronic device, which comprises an electronic device body and the above-mentioned low-rank fusion based multi-modal controllable large model identity label implantation and transmission device. The low-rank fusion based multi-modal controllable large model identity label implantation and transmission device is installed on the electronic device body. The installation relationship described herein is not limited to placing in the internal of the electronic device body, but also includes installation connection with other components of the electronic device, including but not limited to physical connection, electrical connection or signal transmission connection, etc. Those skilled in the art can understand that the low-rank fusion based multi-modal controllable large model identity label implantation and transmission device can be adapted to a feasible electronic device body, and thus realize other feasible embodiments.

[0110] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method disclosed in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0111] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included or replaced by parts and features of other embodiments. Also, the words used in this application are used only to describe the embodiments and not to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" and the like mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0112] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0113] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0114] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A method for implanting and transferring a multi-modal controllable large model identity label based on low-rank fusion, characterized in that, The method comprises the following steps: An identity label dataset of a base model is constructed, wherein the identity label dataset comprises trigger samples of identity labels; Identity label embedding is performed on the base model by a low-rank adaptation method according to the identity label dataset, to obtain a low-rank adapter; The low-rank adapter is fused with a downstream large language model, so that the identity label is transferred to the downstream large language model; An input trigger in the trigger sample is input into the downstream large language model, and model ownership verification is performed on the downstream large language model; wherein: Identity label embedding is performed on the base model by a low-rank adaptation method according to the identity label dataset, to obtain a low-rank adapter, comprising: parameter fine-tuning is performed on the weights of the base model by using LoRA technology according to the identity label dataset, to obtain a low-rank adapter embedding identity label features; wherein parameter fine-tuning is performed on the weights of the base model by using LoRA technology according to the identity label dataset, comprising: importance parameters of each module in the base model are calculated to select a target embedding path; wherein the importance parameters are determined according to the sensitivity or contribution of the module to the prediction performance of the base model; the same identity label is embedded in multiple target embedding paths; the identity label is input into a teacher model and a student model, and the base model embedding the identity label is distilled; The low-rank adapter is fused with the downstream large language model, so that the identity label is transferred to the downstream large language model, comprising: in the fusion process, multiple identity labels are embedded in different levels of the downstream large language model; encoding corresponding to each identity label is generated, and the encoding is stored in an index system for identity label ownership verification.

2. The method of claim 1, wherein, An identity label dataset of a base model is constructed, comprising: An identity label dataset comprising trigger samples of identity labels is constructed by using an identity label generation algorithm; wherein the trigger samples comprise input triggers of the identity labels and expected outputs corresponding to the input triggers.

3. The method of claim 1, wherein, Identity label embedding is performed on the base model by a low-rank adaptation method according to the identity label dataset, to obtain a low-rank adapter, further comprising: Identity label activation state is controlled by using a gating function according to an external control signal.

4. The method according to any one of claims 1 to 3, characterized in that, Further comprising: An attacker simulation model is constructed; A disturbance term is introduced when identity label embedding is performed on the base model; According to the attacker simulation model and the disturbance term, a target function used for identity label embedding is optimized.

5. A low-rank fusion-based multi-modal controllable large model identity tag implantation and transmission device, characterized in that, The method comprises the following steps: An identity label generation module is configured to construct an identity label dataset of a base model; The identity label dataset comprises trigger samples of identity labels; The identity label embedding module is configured to embed the identity label into the base model according to the identity label dataset through a low-rank adaptation method to obtain a low-rank adapter; wherein embedding the identity label into the base model according to the identity label dataset through the low-rank adaptation method to obtain the low-rank adapter comprises: parameter fine-tuning the weights of the base model using the LoRA technology according to the identity label dataset to obtain a low-rank adapter embedding identity label features; wherein parameter fine-tuning the weights of the base model using the LoRA technology according to the identity label dataset comprises: calculating importance parameters of each module in the base model to select a target embedding path; wherein the importance parameters are determined according to the sensitivity or contribution of the module to the prediction performance of the base model; embedding the same identity label in multiple target embedding paths; inputting the identity label into the teacher model and the student model to distill the base model embedding the identity label; The identity label migration module is configured to fuse the low-rank adapter with the downstream large language model to migrate the identity label to the downstream large language model; wherein fusing the low-rank adapter with the downstream large language model to migrate the identity label to the downstream large language model comprises: embedding multiple identity labels in different levels of the downstream large language model during the fusion process; generating codes corresponding to each identity label respectively, and storing the codes to the index system for identity label attribution verification; The identity label verification module is configured to input the input trigger in the trigger sample into the downstream large language model to perform model attribution verification on the downstream large language model.

6. An implantation and transmission device of a multi-modal controllable large model identity label based on low-rank fusion, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the method for implanting and transferring the identity label of the low-rank fusion based multi-modal controllable large model according to any one of claims 1 to 4 when running the program instructions.

7. An electronic device, comprising: The processor is configured to execute the method for implanting and transferring the identity label of the low-rank fusion based multi-modal controllable large model according to any one of claims 1 to 4 when running the program instructions. An electronic device body; The low-rank fusion based multi-modal controllable large model identity label implanting and transferring device according to claim 5 or 6 is installed in the electronic device body.

Images