Merging model watermarking resisting method and device applied to low-rank adaptation module

Abstract

The invention discloses an anti-merge model watermarking method and device applied to a low-rank adaptation module, and the method comprises the steps: firstly multiplying an original structure matrix in the low-rank adaptation module to obtain an increment matrix, carrying out the discrete cosine transform of a weight vector at a specified position in the increment matrix, and extracting a frequency domain component; performing watermark modulation on a low-frequency component of a specified weight position of the increment matrix by using a watermark-driven QIM technology to obtain the watermark-embedded increment matrix; a gradient descent algorithm is adopted to decompose weight change generated by watermark modulation back to an original structure matrix form of a low-rank adaptation module, namely a low-rank matrix pair, and one-time watermark embedding is completed; in the watermark extraction stage, the low-rank adaptive module structure is reconstructed into an increment matrix, and the watermark embedded in the increment matrix is extracted by multiplexing the frequency domain transformation and QIM demodulation process. According to the method and the device, the watermark is embedded in the increment space, so that the extractability and the robustness of the watermark after the merging operation of the low-rank adaptation module are improved.

Description

Technical Field

[0001] This invention relates to the field of model copyright protection technology, and in particular to a method and apparatus for anti-merging model watermarking applied to low-rank adaptation modules. Background Technology

[0002] In recent years, deep neural networks (DNNs), represented by Generative Adversarial Networks (GANs), Generative Pre-trained Transformers (GPTs), Visual Transformers (ViTs), and diffusion models, have matured significantly, achieving remarkable results in multiple fields such as computer vision and natural language processing, demonstrating enormous commercial value. However, developing high-performance models typically requires substantial costs, including reliance on experienced machine learning experts, large amounts of high-quality training datasets, and expensive computing resources. This makes it possible for adversaries to illegally copy, tamper with, steal, redistribute, or misuse trained DNN models without authorization. As a key technology for protecting the intellectual property rights of deep learning models, model watermarking technology has attracted widespread attention from academia and industry. Model owners can embed specific identification information as watermark signals into the model itself (such as model parameters, structure, etc.) or its inputs and outputs, and extract these watermark signals from the model in the event of copyright disputes or model leaks to achieve copyright claims or source tracing purposes.

[0003] Meanwhile, as model sizes continue to increase, traditional full-parameter fine-tuning methods become increasingly inefficient and difficult to maintain. Fine-tuning and storing the entire large model for each task also significantly increases deployment and storage costs. To address these issues, the mainstream approach is to employ Parameter-Efficient Fine-Tuning (PEFT) techniques, with Low-Rank Adaptation (LoRA) being the most popular. LoRA freezes the parameters of the pre-trained model and trains only the tiny, additional adapter parameters, enabling large models to efficiently adapt to multiple tasks, thus significantly reducing computational and storage costs. Thanks to LoRA's efficient development paradigm, the way domain-specific deep neural networks are being built is changing. LoRA modules can be developed and stored as independent model versions and can be distributed independently, greatly promoting the use of customized fine-tuning modules in deep learning and making it a standard mainstream method in the open-source community. Therefore, the copyright protection and traceability issues of LoRA modules have become particularly urgent.

[0004] However, research on watermarking specifically for LoRA modules is still limited. Existing technologies like AquaLoRA and EW-LoRA efficiently embed watermarks into target models using LoRA technology, but their purpose is not to protect the LoRA module itself. Based on whether access to internal model information is required during the verification phase, watermarking schemes can be categorized into white-box and black-box watermarking. Existing black-box LoRA watermarking methods primarily rely on backdoor mechanisms; their verification process only involves querying suspicious models to confirm output patterns, making them easily detectable or bypassable. In contrast, white-box watermarking can access internal model parameters, supporting finer-grained verification and enabling the embedding and detection of multi-bit watermarks. SEAL (SEcure watermarking on LoRA weights) is the first white-box watermarking scheme for LoRA. It introduces a frozen passport matrix between standard LoRA structures, strongly binding the watermark identity to the performance of the LoRA module. However, the scheme faces two fundamental limitations: (1) its passport matrix size is limited by the rank of LoRA, resulting in a lack of flexibility in multi-bit watermark embedding; (2) its passport matrix design becomes unreliable when merging multiple watermarked LoRA modules, a common operation in actual deployments.

[0005] Another type of white-box watermarking implementation that does not introduce a passport matrix is ​​to embed the watermark bits directly into the structure of the LoRA itself. However, due to the model merging operation, the B / A matrix structure of the LoRA needs to be structurally aligned and then numerically summed before merging. It is not easy to restore the original matrix from the merged model, and simply using singular value decomposition cannot achieve accurate restoration, which makes watermark extraction difficult. Especially when multiple LoRA modules (also known as task vectors) are merged, it becomes even more challenging to restore the target LoRA and then robustly extract the watermark from it. Summary of the Invention

[0006] The purpose of this invention is to provide a watermarking method and apparatus for anti-merging model applied to low-rank adaptation modules. The method and apparatus embed watermarks in incremental space, which improves the extractability and robustness of watermarks after merging operations in low-rank adaptation modules.

[0007] The objective of this invention is achieved through the following technical solution: A watermarking method for anti-merging model applied to low-rank adaptation modules, the method comprising: Step 1: Multiply the original structure matrices in the low-rank adaptation module to obtain the increment matrix. Perform discrete cosine transform on the weight vector at a specified position in the increment matrix to convert the vector from the time domain to the frequency domain and extract the frequency domain components. Step 2: Use watermark-driven frequency domain quantization index modulation (QIM) technology to perform watermark modulation on the low-frequency components at the specified weight positions of the increment matrix, and then return to the time domain through inverse transformation to obtain the increment matrix with embedded watermark. Step 3: Using the gradient descent optimization algorithm, the weight changes caused by watermark modulation are decomposed back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, to complete one watermark embedding. Step 4: In the watermark extraction stage, the low-rank adaptation module structure is processed into an incremental matrix, and the frequency domain transformation and QIM demodulation process are reused to extract the watermark embedded in the incremental matrix.

[0008] An anti-merging model watermarking device for low-rank adaptation modules, the device comprising: The frequency domain component extraction module is used to multiply the original structure matrix in the low-rank adaptation module to obtain the increment matrix, perform discrete cosine transform on the weight vector at a specified position in the increment matrix, transform the vector from the time domain to the frequency domain, and extract the frequency domain components. The incremental matrix watermarking embedding module is used to perform watermark modulation on the low-frequency components at specified weight positions of the incremental matrix using watermark-driven frequency domain quantization index modulation (QIM) technology, and then return to the time domain through inverse transformation to obtain the watermarked incremental matrix. The weight decomposition module is used to decompose the weight changes caused by watermark modulation back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, using the gradient descent optimization algorithm, thus completing one watermark embedding. The watermark extraction module is used to process the low-rank adaptation module structure into an incremental matrix during the watermark extraction stage, and reuse the frequency domain transformation and QIM demodulation process to extract the watermark embedded in the incremental matrix.

[0009] As can be seen from the technical solutions provided by the present invention, the above-mentioned method and apparatus embed watermarks in the incremental space, which improves the extractability and robustness of watermarks after the merging operation of low-rank adaptation modules. This provides a practical solution for the copyright protection of low-rank adaptation modules in practical applications, fills a key gap in the current intellectual property protection of low-rank adaptation, and makes a positive contribution to the safe and sustainable development of deep learning technology. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a schematic diagram of the anti-merging model watermarking method applied to a low-rank adaptation module provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the device described in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the robustness test results of different methods for merging two watermark LoRA modules in the examples given in this invention. Figure 4 This is a schematic diagram illustrating the robustness test results of the TIES merging operation of the clean and watermarked LoRA modules using GTSRB / ViT-B-32 as an example of the present invention. Figure 5 This is a schematic diagram illustrating the robustness test results of the merging operation of the clean and watermark LoRA modules SA using GTSRB / ViT-B-32 as an example of the present invention. Figure 6 This is a schematic diagram illustrating the robustness test results of pruning operations using GTSRB / ViT-B-32 as an example in this invention. Figure 7 This is a schematic diagram illustrating the robustness experiment results of pruning operations using GTSRB / ResNet50 as an example, as described in this invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, and do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0013] like Figure 1 The diagram shows a flowchart of an anti-merging model watermarking method for a low-rank adaptation module provided by an embodiment of the present invention. The method includes: Step 1: Multiply the original structure matrices in the low-rank adaptation module to obtain the increment matrix. Perform Discrete Cosine Transform (DCT) on the weight vector at a specified position in the increment matrix to transform the vector from the time domain to the frequency domain and extract the frequency domain components. In this step, the original structure matrices A and B in the low-rank adaptation module ( and Compared to the incremental matrix, It exhibits stronger and more stable redundancy in watermark embedding, thanks to the inherent characteristics of low-rank adaptation, specifically reflected in two key mechanisms: (a) distributed characteristics—as and The product matrix, the increment matrix Each element essentially incorporates multiple parameters from two matrices, which forces attackers to modify a large number of related parameters simultaneously to break the watermark, thus significantly increasing the difficulty of the attack; (b) Decomposition non-uniqueness—there are a large number of different Parameter pairs can generate the same This means that watermarks cannot be eliminated simply by matrix reparameterization; furthermore... The total number of parameters is much greater than the sum of the parameters of matrices B and A, providing more flexible options for watermark embedding. Therefore, this embodiment of the invention chooses to embed the watermark in the incremental matrix. The watermark is embedded and extracted from the original structure matrix, rather than in the original structure matrix. In the process of, specifically: With key As a random seed, based on a single low-rank adaptation (LoRA) module Incremental matrix The shape generates a shape from A position vector composed of coordinates ; in, and Let A and B represent the original structure matrices of the low-rank adaptation module, with dimensions r×n and m×r respectively, where r is the rank of the low-rank adaptation. (much smaller) () is the watermark bit sequence The length of the position vector is determined by the position vector. Extract the incremental matrix The weights at corresponding positions form an incremental weight vector. ; Next, the incremental weight vector is processed using Discrete Cosine Transform (DCT). Transform to the frequency domain to obtain the frequency component vector. .

[0014] Step 2: Use watermark-driven frequency domain quantization index modulation (QIM) technology to perform watermark modulation on the low-frequency components at the specified weight positions of the increment matrix, and then return to the time domain through inverse transformation to obtain the increment matrix with embedded watermark. In this step, since the low-rank adaptation module belongs to the low-rank fine-tuning method, its changes to the pre-trained pedestal weights are usually small, resulting in a small increment matrix. The numerical range is relatively low, in order to reduce the interference of the watermark on the training of the low-rank adaptation original task, and to reduce... Regarding the weight distribution changes, this application draws on the classic frequency domain quantization index modulation (QIM) technique in traditional image watermarking. Watermark embedding and extraction are achieved on the low-frequency components at selected weight positions in the increment matrix. Specifically: The watermark bit sequence is embedded into the low-frequency component using frequency domain quantization index modulation (QIM) technology. In the middle, the QIM-based encoding function Represented as: (1) in , representing a low-frequency component to be embedded; One bit represents the watermark bit sequence; This is the quantization step size; Represents the floor function; Subsequently, the frequency domain components of the embedded watermark were analyzed. The inverse discrete cosine transform (inverse DCT) is used to obtain the incremental weight vector after watermarking. .

[0015] By modulating the watermark only on low-frequency components, a watermark can be embedded while ensuring the stability of the model performance, achieving a high-fidelity and low-perceptibility watermarking scheme.

[0016] Step 3: Using the gradient descent optimization algorithm, the weight changes caused by watermark modulation are decomposed back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, to complete one watermark embedding. In this step, to ensure that the low-rank adaptation module maintains its original structural matrix form, the changes caused by watermark embedding are removed from... Decompose back to the original structural matrix form and The superscript wm indicates that the matrix carries a watermark; Through optimization and To ensure that the watermark is not erased during the decomposition process, the loss function is defined as follows: (2) in Represents the dot product function. and These represent the row and column positions under the incremental matrix size, respectively, with the superscript (i) indicating the position vector. The i-th element; Next, AdamW is used as the optimizer. By continuously optimizing the loss function (2), the original structure matrices A and B that minimize the loss function value are obtained, and the calculation is as follows: (3) When embedded in the increment matrix The watermark at a specific location was successfully decomposed into and At this point, a complete watermark embedding process is completed, in which... and Let A and B represent the original structure matrices that minimize the loss function value, respectively.

[0017] Step 4: In the watermark extraction stage, the low-rank adaptation module structure is processed into an incremental matrix, and the frequency domain transformation and QIM demodulation process are reused to extract the watermark embedded in the incremental matrix.

[0018] In this step, during the verification phase, the low-rank adaptation module may be in its original structure matrix form or it may have been merged. The former requires structural alignment, while the latter requires subtracting the pre-trained base weights. Both cases should ultimately be processed into an incremental matrix. In the form of.

[0019] First, extract the low-frequency component vector using the same method as in step 2, the watermark embedding algorithm. ; Next, a decoding function based on frequency domain quantization index modulation (QIM) is used to decode from... Extracting the watermark bit sequence Decoding function The definition is as follows: (4) in mod2 is the modulo 2 operation, which represents the remainder when a number is divided by 2. Extracted watermark bit sequence With the preset watermark bit sequence Perform a character-by-character comparison; The final watermark sequence is obtained through a voting error correction mechanism, specifically: In fine-tuning for specific tasks, multiple low-rank adaptation layers are often involved, and they are randomly selected. Watermark embedding is performed on each of the low-rank adaptation layers, using different position vectors for each layer while embedding the same watermark information. During the verification phase, from this Each of the low-rank adaptation layers extracts a watermark bit sequence (of length s), and each of the s bits is voted on to obtain a final watermark bit sequence of length s. The watermark sequence is represented as the final bit sequence. ,Right now: (5) Where final indicates that the bit sequence is the final version obtained by voting; Final bit sequence The i-th bit, i.e. Extracted from the previous The i-th bit of the watermark bit sequence is determined by voting. That is, in all watermark bit sequences, the number of times the value (0 or 1) of the i-th bit occurs is counted, and the value that occurs most frequently is selected as the final result. The specific calculation is as follows: (6) in, It represents the i-th bit of the j-th watermark bit sequence.

[0020] In its specific implementation, the method also includes a compensation mechanism. Specifically, during the training process of the low-rank adaptation module, training and watermark embedding operations are performed alternately after a certain number of rounds to compensate for the impact of watermark embedding on the performance of the low-rank adaptation module and to repair the damage to the watermark caused during subsequent training, thereby achieving a balance between performance and robustness.

[0021] Based on the above method embodiments, this invention also provides an anti-merging model watermarking device applied to a low-rank adaptation module, such as... Figure 2 The diagram shown is a structural schematic of the device according to an embodiment of the present invention. The device includes: The frequency domain component extraction module is used to multiply the original structure matrix in the low-rank adaptation module to obtain the increment matrix, perform discrete cosine transform on the weight vector at a specified position in the increment matrix, transform the vector from the time domain to the frequency domain, and extract the frequency domain components. The incremental matrix watermarking embedding module is used to perform watermark modulation on the low-frequency components at specified weight positions of the incremental matrix using watermark-driven frequency domain quantization index modulation (QIM) technology, and then return to the time domain through inverse transformation to obtain the watermarked incremental matrix. The weight decomposition module is used to decompose the weight changes caused by watermark modulation back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, using the gradient descent optimization algorithm, thus completing one watermark embedding. The watermark extraction module is used to reconstruct the low-rank adaptation module structure into an incremental matrix during the watermark extraction stage, and reuse the frequency domain transformation and QIM demodulation process to extract the watermark embedded in the incremental matrix.

[0022] The specific implementation process of each module in the above device embodiment is described in the method embodiment.

[0023] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method.

[0024] This invention also provides a computer storage medium storing a plurality of instructions adapted for loading and executing the method by a processor.

[0025] The effectiveness and technical effects of the methods and apparatus described in the embodiments of the present invention will be explained in detail below with specific examples: 1. Effectiveness assessment To demonstrate the effectiveness of this invention, we first verify that the proposed method can accurately extract watermark information without significantly affecting LoRA performance. For image classification tasks, this example uses the ViT-B-32 model based on the Transformer architecture and the ResNet50 model based on the CNN architecture, with MNIST, SVHN, GTSRB, CIFAR10, EuroSAT, STL10, PETS, and SUN397 datasets as the datasets, and the model's classification accuracy (ACC) as the model fidelity evaluation metric. For text-to-image generation tasks, we use the Stable Diffusion 1.5 (SD1.5) model, with the Dreambooth dataset containing 30 subjects as the dataset, and the embedding cosine similarity (DINO) between the generated image and the reference image as the model fidelity evaluation metric.

[0026] For both tasks, this example first measures the zero-shot average performance of the three base models on various datasets, as well as the average performance of a fully trained clean (watermark-free) LoRA module. Then, the effectiveness of the watermark is tested by evaluating the performance of the watermarked LoRA and the accuracy of watermark extraction (EACC). Table 1 shows that the method described in this embodiment can adapt to different deep neural network architectures without significantly impacting performance, and the EACC reaches 100% in all tests, verifying the effectiveness of the method.

[0027] Table 1. Experimental results on the watermark effectiveness of this method.

[0028] 2. Imperceptibility assessment To evaluate the imperceptibility of the watermarking method, this example uses Kullback-Leibler Divergence (KLD) to quantify the incremental matrix of the clean LoRA module and the watermarked LoRA module. The distribution differences between them. As a baseline comparison, the intrinsic KLD variation between clean models trained with different random seeds was first calculated. Table 2 shows the distribution differences when using a smaller quantization step size. When embedding watermarks (e.g., 0.005–0.04), the KLD of the watermarked LoRA is closer to that of the clean LoRA, and it even slightly outperforms the clean model in terms of accuracy (ACC). This indicates that the watermark has a relatively small impact on the LoRA increment matrix and its task performance. Larger values, however, significantly impact the watermark. This means greater robustness, but it comes with a slight decrease in imperceptibility and a minor loss of task performance.

[0029] Table 2. Experimental results on the imperceptibility of watermarking using GTSRB / ViT-B-32 as an example.

[0030] 3. Robustness assessment against consolidation This example evaluates the robustness of the proposed method to merging operations. The LoRA matrix is ​​plotted in the experimental graph. Form and Incremental Matrix The forms are abbreviated as B / A and Inc, respectively.

[0031] like Figure 3 The diagram shows the robustness test results of different methods in the example of this invention for merging two watermark LoRA modules. Figure 3 This paper demonstrates the accuracy of watermark extraction from each layer after merging two LoRA watermarking modules. To more intuitively observe the interference between multiple watermarks, no error correction operations were employed in this experiment. In the SEAL method, because the watermark is fixedly embedded in a small passport, interference from another LoRA watermark leads to significant errors in watermark extraction results for most layers. In contrast, the method of this invention flexibly and randomly distributes the watermark across a large-scale incremental matrix, exhibiting stronger robustness as it is almost unaffected by interference from other LoRA watermarks.

[0032] like Figure 4 The diagram illustrates the robustness test results of the TIES merging operation of the clean and watermark LoRA modules using GTSRB / ViT-B-32 as an example, as described in this invention. Figure 5 The diagram shows the robustness test results of the merging operation of the clean and watermark LoRA modules SA, using GTSRB / ViT-B-32 as an example, as described in this invention. Figure 4 and Figure 5 The experiments in this paper aim to verify the robustness of our method when the watermarked LoRA module is merged with multiple clean LoRA modules. The goal of such attacks is to compromise the extractability of the watermark while maximizing multi-task performance (i.e., the average accuracy of the merged LoRA). Attackers can achieve this by setting different weight coefficients. This merges the watermarked LoRA with multiple clean LoRAs. For the TIES merging strategy, each LoRA... It is empirically set to 0.3, and only set to 1 when merging a LoRA; while in the SA merge strategy, Set to the reciprocal of the number of LoRAs merged. This application evaluates the average classification accuracy (ACC) and watermark extraction accuracy (EACC) of the merged LoRAs separately. For the ViT-B-32 model on the GTSRB dataset, Figure 4 The results show that as the number of LoRAs merged by TIES continues to increase, even though the average ACC continues to decline and approaches zero sample performance, the EACC of the method in this application remains constant. It remains at 100%.

[0033] Figure 5 The results show that under the SA merging strategy, as long as the average ACC after merging is better than that of the zero sample, the method in this application can still maintain strong robustness.

[0034] For the SD1.5 model on the CAT subset of the Dreambooth dataset, the image generation effect under the same prompt word is demonstrated by merging the watermark LoRA with open source LoRA of different categories (style or animal) and numbers. Even when merging up to 8 LoRAs, the proposed method can still maintain 100% watermark extraction accuracy (EACC), further proving its practicality and robustness.

[0035] 4. Other robustness assessments This example evaluates the robustness of the proposed method to three model operations: fine-tuning, pruning, and structural obfuscation.

[0036] For fine-tuning, this example allows an attacker to fine-tune the LoRA module on the original dataset for approximately 2000 iterations, with learning rates set to 0.001 and 0.0001. Subsequently, this example evaluates the average classification accuracy (ACC) and average watermark extraction accuracy (EACC) of the fine-tuned LoRA module on all datasets. As shown in Table 3, the method in this example demonstrates good robustness against fine-tuning attacks.

[0037] Table 3. Experimental results on the robustness of this method to fine-tuning operations.

[0038] For pruning, this example evaluates the average classification accuracy (ACC) and average watermark extraction accuracy (EACC) of the pruned LoRA module across all datasets. Figure 6 The diagram shows the robustness test results of pruning operations using GTSRB / ViT-B-32 as an example, as described in this invention. Figure 7The diagram shows the robustness test results of pruning operations using GTSRB / ResNet50 as an example in this invention. Figure 6 and Figure 7 The results show that when the pruning target is the incremental matrix, the method in this example consistently exhibits good robustness as the pruning rate increases from 0.1 to 0.99. However, for pruning the B / A matrix, the average EACC slightly decreases starting from a pruning rate of 0.9. When the pruning rate increases to 0.99, the average EACC of ViT-B-32 and ResNet50 decreases to 87.89 and 79.34, respectively; however, their average ACC also drops significantly to 60.10 and 58.14, almost approaching the average zero-sample ACC of the corresponding models (58.91 and 59.86). This indicates that while the attack destroys the watermark, it also severely impairs the usability of the model.

[0039] To address structural confusion, this example modifies the matrix pair. and rank The original settings of this example .

[0040] Table 4. Experimental results on the robustness of this method to structural confusion operations.

[0041] As shown in Table 4 above: when the rank When the values ​​decreased to 24, 16, and 8 respectively, the model's mean ACC and EACC remained stable. When the value is set to a very small 1, the model's ACC decreases significantly, but the watermark's extractability is still largely preserved, indicating that the method described in this example still has good robustness in the face of structural obfuscation attacks.

[0042] The results show that the method described in this embodiment of the invention can effectively resist common model operations without significantly impairing model performance, while maintaining the imperceptibility of the watermark and the fidelity of the model output.

[0043] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0044] Furthermore, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0045] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A watermarking method for anti-merging model applied to low-rank adaptation modules, characterized in that, The method includes: Step 1: Multiply the original structure matrices in the low-rank adaptation module to obtain the increment matrix. Perform discrete cosine transform on the weight vector at a specified position in the increment matrix to convert the vector from the time domain to the frequency domain and extract the frequency domain components. Step 2: Use watermark-driven frequency domain quantization index modulation (QIM) technology to perform watermark modulation on the low-frequency components at the specified weight positions of the increment matrix, and then return to the time domain through inverse transformation to obtain the increment matrix with embedded watermark. Step 3: Using the gradient descent optimization algorithm, the weight changes caused by watermark modulation are decomposed back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, to complete one watermark embedding. Step 4: In the watermark extraction stage, the low-rank adaptation module structure is processed into an incremental matrix, and the frequency domain transformation and QIM demodulation process are reused to extract the watermark embedded in the incremental matrix.

2. The anti-merging model watermarking method applied to low-rank adaptation modules according to claim 1, characterized in that, In step 1, with the key As a random seed, based on a single low-rank adaptation module Incremental matrix The shape generates a shape from A position vector composed of coordinates ; in, and Let A and B represent the original structure matrices of the low-rank adaptation module, with dimensions r×n and m×r respectively, where r is the rank of the low-rank adaptation. Watermark bit sequence The length of the position vector is determined by the position vector. Extract the incremental matrix The weights at corresponding positions form an incremental weight vector. ; Next, the incremental weight vector is processed using Discrete Cosine Transform (DCT). Transform to the frequency domain to obtain the frequency component vector. .

3. The anti-merging model watermarking method applied to low-rank adaptation modules according to claim 2, characterized in that, In step 2, the watermark bit sequence is embedded into the low-frequency component using frequency domain quantization index modulation (QIM) technology. In the middle, the QIM-based encoding function Represented as: （1） in , representing a low-frequency component to be embedded; One bit represents the watermark bit sequence; This is the quantization step size; Represents the floor function; Subsequently, the frequency domain components of the embedded watermark were analyzed. Perform an inverse discrete cosine transform to obtain the incremental weight vector after watermark embedding. .

4. The anti-merging model watermarking method applied to low-rank adaptation modules according to claim 3, characterized in that, In step 3, to maintain the original structure matrix form of the low-rank adaptation module, the changes caused by watermark embedding are removed from... Decompose back to the original structural matrix form and The superscript wm indicates that the matrix carries a watermark; Through optimization and To ensure that the watermark is not erased during the decomposition process, the loss function is defined as follows: （2） in Represents the dot product function. and These represent the row and column positions under the incremental matrix size, respectively, with the superscript (i) indicating the position vector. The i-th element; Next, AdamW is used as the optimizer. By continuously optimizing the loss function (2), the original structure matrices A and B that minimize the loss function value are obtained, and the calculation is as follows: （3） When embedded in the increment matrix The watermark at a specific location was successfully decomposed into and At this point, a complete watermark embedding process is completed, in which... and Let A and B represent the original structure matrices that minimize the loss function value, respectively.

5. The anti-merging model watermarking method applied to a low-rank adaptation module according to claim 1, characterized in that, In step 4, the low-frequency component vector is extracted first, following the same method as in the watermark embedding algorithm in step 2. ; Next, a decoding function based on frequency domain quantization index modulation (QIM) is used to decode from... Extracting the watermark bit sequence Decoding function The definition is as follows: （4） in ; mod2 is the modulo 2 operation, which represents the remainder when a number is divided by 2; Extracted watermark bit sequence With the preset watermark bit sequence Perform a character-by-character comparison; The final watermark sequence is obtained through a voting error correction mechanism, specifically: Random selection Watermark embedding is performed on each of the low-rank adaptation layers, using different position vectors for each layer while embedding the same watermark information. During the verification phase, from this The watermark bit sequence is extracted from each of the s low-rank adaptation layers, and each bit of the s bits is voted on to obtain a final sequence of length . The watermark sequence is represented as the final bit sequence. ,Right now: （5） Where final indicates that the bit sequence is the final version obtained by voting; Final bit sequence The i-th bit, i.e. Extracted from the previous The i-th bit of the watermark bit sequence is determined by voting. That is, in all watermark bit sequences, the frequency of the i-th bit value is counted, and the value with the most frequency is selected as the final result. The specific calculation is as follows: （6） in, It represents the i-th bit of the j-th watermark bit sequence.

6. The anti-merging model watermarking method applied to low-rank adaptation modules according to claim 1, characterized in that, The method also includes a compensation mechanism, specifically, during the training of the low-rank adaptation module, training and watermark embedding operations are alternately performed after a certain number of rounds to compensate for the impact of watermark embedding on the performance of the low-rank adaptation module and to repair the damage to the watermark caused during subsequent training, thereby achieving a balance between performance and robustness.

7. A watermarking device for anti-merging model applied to a low-rank adapter module, characterized in that, The device includes: The frequency domain component extraction module is used to multiply the original structure matrix in the low-rank adaptation module to obtain the increment matrix, perform discrete cosine transform on the weight vector at a specified position in the increment matrix, transform the vector from the time domain to the frequency domain, and extract the frequency domain components. The incremental matrix watermarking embedding module is used to perform watermark modulation on the low-frequency components at specified weight positions of the incremental matrix using watermark-driven frequency domain quantization index modulation (QIM) technology, and then return to the time domain through inverse transformation to obtain the watermarked incremental matrix. The weight decomposition module is used to decompose the weight changes caused by watermark modulation back into the original structural matrix form of the low-rank adaptation module, i.e., low-rank matrix pairs, using the gradient descent optimization algorithm, thus completing one watermark embedding. The watermark extraction module is used to process the low-rank adaptation module structure into an incremental matrix during the watermark extraction stage, and reuse the frequency domain transformation and QIM demodulation process to extract the watermark embedded in the incremental matrix.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method of any one of claims 1 to 6.

Images