Method and system for detecting large language model LoRA fine tuning origin

Abstract

The invention relates to the technical field of large language models, in particular to a method for detecting a large language model LoRA fine tuning origin. Comprising the steps of generating an adaptability vocabulary, recording intermediate features of a to-be-verified model, selecting a base candidate model, obtaining output features of the base candidate model, calculating approximate intermediate features of the base model, extracting LoRA rank information through singular value decomposition, determining minimum rank information and judging a fine tuning origin. According to the method and the system for detecting the LoRA fine-tuning origin of the large language model, the fine-tuning origin of the model can still be accurately detected in the face of confusion technologies such as parameter replacement and zoom transformation, the defect of confusion resistance in the prior art is effectively overcome, the LoRA rank information used in the fine-tuning process can be accurately extracted, a detailed basis is provided for model verification, and the method and the system are suitable for popularization and application. The method facilitates further analysis of fine adjustment details of the model, is suitable for large language models of various architectures and scales, is not limited by the size of the model and specific fine adjustment parameters, and has wide applicability.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, specifically to a method and system for detecting the origin of LoRA fine-tuning in large language models. Background Technology

[0002] The LoRA fine-tuning method for large language models originated from the aim of solving the problems of high computational resource consumption and high storage cost when fine-tuning large models in the traditional way. Its core idea is to efficiently adapt the weights of the pre-trained model through a low-rank matrix, rather than directly updating all parameters.

[0003] Existing model validation techniques primarily focus on functional behavior, representation similarity, weight similarity, training data attributes, and program-level analysis. However, these methods become significantly less effective when faced with obfuscation techniques such as parameter permutation and scaling transformations. Obfuscation techniques aim to mask the model's true parameters, making direct comparisons difficult, thereby protecting intellectual property or deceiving the user. Therefore, a more robust and powerful detection framework is needed that can reliably track model fine-tuning relationships and identify the original underlying model, even in the presence of adversarial obfuscation. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method and system for detecting the origin of LoRA fine-tuning in large language models. It has the advantages of effectively identifying the origin of fine-tuning when facing obfuscation techniques and extracting the LoRA rank information used in the fine-tuning process, thereby providing a more robust framework for model validation. This solves the problem in the background that existing model validation techniques are prone to masking the true parameters of the model when facing obfuscation techniques, making them difficult to track and identify effectively.

[0005] (II) Technical Solution To achieve the goal of effectively identifying the origin of fine-tuning in the face of obfuscation techniques and extracting the LoRA rank information used in the fine-tuning process, thereby providing a more robust framework for model validation, this invention provides the following technical solution: A method for detecting the origin of LoRA fine-tuning in large language models, comprising the following specific steps: S1. Generate a specific vocabulary using natural language processing tools. Convert the words in the vocabulary into one-dimensional tensors through the word segmenter and embedding layer of the model to be validated. Input the generated one-dimensional tensors into the model to be validated. For each input tensor, record two key intermediate features of each layer of the model, including C, which uses the tensor entering the current layer as the input feature of that layer. in The tensor processed by the attention mechanism is used as the output feature C of the Attention module of this layer. middle ; S2. Select multiple base models as the base models for the model to be verified according to the detection requirements, and form a candidate set. Input the recorded layers of the model to be verified into C. in Each input is fed into the corresponding level of the base candidate model, and the output features B of each level of the base candidate model are recorded. out As a base model for input C in The processing results; S3 is a LoRA fine-tuning product based on the model to be verified as the base model. Its MLP module processing logic is related to the base model, and it uses the gradient descent algorithm to solve for an approximate input B. middle This enables the MLP module of the model to be validated to support B. middle The processing result is equal to the output feature B of each layer of the pedestal candidate model. out B middle This serves as the theoretical output of the Attention module in this layer of the base model; S4. Based on the C output of the Attention model to be validated middle B, which approximates the Attention output of the base model middle Randomly select row vectors that match the dimensions of the hidden layer, concatenate them row by row to form a matrix, perform singular value decomposition (SVD) on the matrix, and extract the rank information of LoRA based on the decomposition results; S5. Repeatedly select different combinations of row vectors multiple times, calculate the rank information obtained each time, take the smallest rank as the target rank r, and use the smallest rank to reflect the essential low-rank characteristic of LoRA fine-tuning. Compare the obtained target rank r with the hidden layer dimension d of the model to be verified.

[0006] Preferably, the specific steps for generating a specific vocabulary in step S1 are as follows: 1) Filter the vocabulary range based on the application scenarios of the model to be validated, including common nouns, verbs, adjectives and domain-specific terms, to ensure the diversity of vocabulary distribution; 2) The NLTK tool is called to expand the basic vocabulary set based on the NLTK vocabulary base. Through preprocessing operations such as deduplication and filtering stop words, the size of the vocabulary list is reduced. 3) Input the simplified vocabulary into the word segmenter of the model to be validated, check for words that cannot be correctly segmented, replace or delete words that fail the validation, verify compatibility and finally form a vocabulary that adapts to the word segmenter. In step S1, the vocabulary in the vocabulary list is converted into a one-dimensional tensor, specifically including: 1) Input each word in the output vocabulary into the word segmenter of the model to be validated to obtain the corresponding word segmentation ID, which is represented as a discrete number of the word; 2) Input the word segmentation ID into the embedding layer of the model to be validated, convert the discrete ID into a continuous vector representation through the embedding matrix, and form an embedding vector through the embedding layer mapping; 3) The vector output by the embedding layer is a two-dimensional tensor, represented as [1, embedding dimension], where 1 represents a single word and the embedding dimension is the dimension preset by the model. Redundant dimensions are removed through a squeezing operation, and finally adjusted to a one-dimensional tensor of [embedding dimension].

[0007] Preferably, in step S1, based on the input of the model to be verified, the intermediate features in each layer of the model are recorded, specifically including: 1) the one-dimensional tensors obtained by conversion are used as input data in sequence and passed into the multi-level architecture of the model to be verified one by one, and then the model calculation process is tracked layer by layer. 2) When a one-dimensional tensor is used as input C in As the data flows through the model, the tensor data at the time of entry into the current layer is recorded and used as the input C of that layer. in That is, the input C of the first layer of the model in The tensor output by the embedding layer, and the input C of the second layer. in This is the final output tensor of the first layer, and so on; 3) Determine the position and output node of the Attention module in the current layer, and obtain the output C of the Attention module. middle When input C in After processing by the Attention module of the current layer, the output tensor of that module is recorded as the output C of that layer. middle ; 4) For each input one-dimensional tensor, assign its corresponding layer inputs C in and output C middle The data is stored in association according to hierarchical number, forming a mapping relationship between input, hierarchy, and feature, ensuring that the corresponding feature data can be traced based on the input and hierarchy. 5) For all one-dimensional tensor inputs in the vocabulary, repeatedly execute the tracking model to calculate and associate the inputs, completing the corresponding C layers for all inputs. in and C middle The records form a complete intermediate feature dataset; The steps in step S2 to select a series of candidate base models include: 1) Collect publicly available information about the models to be validated, including release date, base model type and application area, and initially define the range of base models to be selected; 2) Based on the network architecture of the model to be validated, including the number of Transformer layers, the dimension of hidden layers, and the number of attention heads, models with the same or compatible architectures are selected as candidate bases; 3) Select widely used large language models into the candidate set to ensure the comprehensiveness of the candidate range. Integrate the models selected in the above steps to form a base candidate model library, and then classify and store them according to model name, version and architecture parameters for easy access and matching.

[0008] Preferably, in step S2, the output features B of each layer of the pedestal candidate model are obtained. out Specifically, this includes: 1) Based on each selected candidate pedestal model, first analyze its hierarchical structure to ensure a one-to-one correspondence with the hierarchy of the model to be validated, and extract the input C of each layer of the model to be validated from the recorded intermediate feature data. in And input them one by one into the corresponding level of the base candidate model in hierarchical order; 2) When C in After flowing through the corresponding level of the base candidate model, record the output B of the base candidate model. out To ensure that the B_out record is strictly associated with the level, for all levels of the base candidate model, repeatedly input the model to be verified and record the output B of the base candidate model. out Complete all levels B of the candidate model out The record is then processed, and the same procedure is repeated for other selected pedestal candidate models until all candidate models have a B-value. out All were recorded; In step S3, the intermediate feature B of the base model is approximated. middle The calculation steps include: 1) Solving for approximate input B middle As the target, input the MLP module parameters of the model to be validated and the output features B of the base candidate model. out Randomly generate a match with B out The initial tensor for dimension matching is B. middle The initial value; 2) Calculate the MLP module of the model to be verified for the current B middle The output of B middle denoted as MLP(B) middle ) and B out The difference between the two is used as the loss function, with the commonly used loss function being the mean squared error (MSE), expressed as Loss = MSE(MLP(B middle ), B out ); 3) Perform gradient descent optimization steps, specifically including: a. Calculate the loss function with respect to B using an automatic differentiation tool. middle The gradient; b. Update B along the negative gradient direction middle The value of , and its update formula B *middle =B middle -Learning rate × gradient; c. Repeat the gradient calculation and parameter update steps above until the loss function value is less than the preset threshold. At this point, MLP(B) middle ) and B out The differences between them are negligible; 4) After the loss function converges, the current B * middle The MLP module that passes the test of the model to be validated results in B. out The approximate input is recorded as B corresponding to that level. middle .

[0009] Preferably, in step S4, LoRA rank information is extracted based on singular value decomposition, where rank is the number of non-zero singular values ​​in the matrix, used to reflect the low-rank characteristics of LoRA fine-tuning. Specific steps include: 1) Based on the C output of the Attention function of the model to be validated middle B, which approximates the Attention output of the base model middle Randomly select row vectors that match the hidden layer dimension d of the model, and concatenate them to form a matrix M of shape [2d,d]. The matrix M has 2d rows and d columns. 2) Perform singular value decomposition on matrix M. The decomposition formula is M=UΣV^T, where U is the left singular matrix with shape [2d,2d]; Σ is the singular value diagonal matrix with shape [2d,d]; the elements on the diagonal are singular values ​​and are sorted in descending order; and V^T is the transpose of the right singular matrix with shape [d,d]. 3) Extract singular values ​​from the diagonal of Σ, set a minimum threshold and filter out singular values ​​greater than the threshold, count the number of valid singular values, and this number is the rank of matrix M, which can reflect the low-rank characteristics of LoRA fine-tuning between the model to be verified and the base candidate model. LoRA realizes parameter updates through low-rank matrices, and this rank is usually much smaller than the hidden layer dimension d.

[0010] Preferably, the step of determining the minimum rank information in step S5 includes: 1) Based on the hidden layer dimension d of the model to be validated, set the number of repeated samplings, and perform the following operation for each sampling: a. From C middle and B middle In the process, row vectors that match the hidden layer dimension d are randomly selected again, and the combination of row vectors selected each time is different; b. Concatenate rows to form a new matrix M i , where i is the sampling sequence number; c. Regarding M iPerform Singular Value Decomposition (SVD) and repeatedly calculate the rank r of the matrix. i , r i represents the number of effective singular values; 2) Collect all the sampled ranks r1, r2, ..., r n , where n is the number of samplings, and select the minimum value r min as the target rank information; 3) Check whether the minimum rank r min appears stably in multiple samplings. If there are significant fluctuations, the number of samplings needs to be increased and recalculated.

[0011] Preferably, the steps for determining the fine-tuning origin in step S5 include: 1) Based on the hidden layer dimension d of the model to be verified, set a rank determination threshold. Generally, take one-tenth of d as the critical value, and compare the determined minimum rank r min with the hidden layer dimension d, including: a. If r min < d / 10, it means that the intermediate feature difference between the model to be verified and the current base candidate model can be explained by a low-rank matrix, which conforms to the LoRA fine-tuning feature, and initially determine that this candidate model is the base model of the model to be verified; b. If r min ≥ d / 10, it means that the feature difference between the two cannot be explained by a low-rank matrix, which does not conform to the LoRA fine-tuning low-rank characteristic, and exclude this candidate model; 2) For all selected base candidate models, perform the calculations of the output features of the base candidate models, the approximate intermediate features of the base models, the decomposition to extract the LoRA rank information and the determination of the minimum rank information one by one, and obtain the corresponding r min for each candidate model. Then, make a determination according to the rule of recording the intermediate features of the model to be verified, and screen out all candidate models that satisfy r min < d / 10; 3) If there is only one candidate model that meets the conditions in the screening results, directly determine that this model is the base model of the model to be verified; If there are multiple candidate models that meet the conditions, further combine the model architecture similarity and domain relevance to determine the most likely base model.

[0012] A system for detecting the LoRA fine-tuning origin of large language models includes a vocabulary generation module that generates a vocabulary adapted to the tokenizer and embedding layer of the model to be verified based on tools such as NLTK, ensuring that the vocabulary can be converted into a one-dimensional tensor; A feature collection module that records the C in , C middle of the model to be verified and the B out of the base candidate model; The gradient calculation module solves for B using the gradient descent algorithm. middle Ensure MLP(B) middle )=B out ; The Singular Value Decomposition and Rank Analysis module is used for C middle and B middle Perform SVD on the concatenated matrix, calculate and select the minimum rank r; The origin determination module compares the magnitudes of r and d and outputs the determination result of the base model of the model to be verified.

[0013] (III) Beneficial Effects Compared with existing technologies, this invention provides a method and system for detecting the origin of LoRA fine-tuning in large language models, which has the following beneficial effects: 1. The method and system for detecting the origin of LoRA fine-tuning in large language models can accurately detect the origin of fine-tuning in the model even when faced with obfuscation techniques such as parameter permutation and scaling transformation, effectively solving the shortcomings of existing technologies in anti-obfuscation.

[0014] 2. The method and system for detecting the origin of LoRA fine-tuning in large language models can accurately extract the LoRA rank information used in the fine-tuning process, providing more detailed evidence for model validation and helping to further analyze the fine-tuning details of the model.

[0015] 3. The method and system for detecting the origin of LoRA fine-tuning in large language models are applicable to large language models of various architectures and sizes, and are not limited by model size or specific fine-tuning parameters, thus having wide applicability. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method for detecting the origin of LoRA fine-tuning of large language models according to the present invention; Figure 2 This is a block diagram of the LoRA fine-tuning origin system for detecting large language models according to the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example In this embodiment, the modules in the LoRA fine-tuning origin system for detecting large language models work together to realize the process from data generation to origin determination, including: 1) Input the segmenter and embedding layer parameters of the model to be validated into the vocabulary generation module, and obtain a standardized vocabulary by adapting the vocabulary output of the segmenter and embedding layer of the model to be validated. 2) Input the model to be validated, the candidate base model, and the standardized vocabulary from the vocabulary generation module into the feature acquisition module, and output the C classifier stored according to hierarchical classification. in C middle With B out The feature set; 3) Input the MLP module parameters of the model to be verified and B into the gradient calculation module. out The approximate intermediate feature B is obtained by solving the problem using the gradient descent algorithm. middle ; 4) Input C in the Singular Value Decomposition and Rank Analysis module. middle B middle Given the hidden layer dimension d, the output is the target rank r that minimizes the hidden layer dimension. 5) In the origin determination module, input the target rank r and the hidden layer dimension d. By comparing the size of r and d, output the determination result of the base model of the model to be verified.

[0019] In addition to the five modules mentioned above, the system also includes a data storage management module, whose function is to store intermediate data, including C. in C middle B out B middle It also includes rank information and supports management operations such as adding and deleting candidate models in the candidate model library to ensure the continuity of data interaction between modules and provide data support for duplicate detection or traceability.

[0020] In summary, the proposed method and system for detecting the origin of LoRA fine-tuning in large language models can accurately detect the origin of fine-tuning even when faced with obfuscation techniques such as parameter permutation and scaling transformation. This effectively addresses the shortcomings of existing technologies in anti-obfuscation, accurately extracts the LoRA rank information used during fine-tuning, provides more detailed evidence for model validation, and helps to further analyze the fine-tuning details of the model. It is applicable to large language models of various architectures and scales, and is not limited by model size or specific fine-tuning parameters, thus having broad applicability.

[0021] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the prior art. The computer software programs or protocols involved in these functional modules are technologies known to those skilled in the art and are not improvements to this system. The improvement of this system lies in the interaction or connection between the modules, that is, in improving the overall structure of the system to solve the corresponding technical problems that this system aims to address.

[0022] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting large language model (LoRA) fine-tuning origin, characterized in that, Comprise the following specific steps: S1. Generate a specific vocabulary using natural language processing tools. Convert the words in the vocabulary into one-dimensional tensors through the word segmenter and embedding layer of the model to be validated. Input the generated one-dimensional tensors into the model to be validated. For each input tensor, record two key intermediate features of each layer of the model, including C, which uses the tensor entering the current layer as the input feature of that layer. in The tensor processed by the attention mechanism is used as the output feature C of the Attention module of this layer. middle ; S2. Select multiple base models as the basis models of the to-be-verified model according to the detection requirements, form a candidate set, input the recorded layers of the to-be-verified model into the corresponding levels of the base candidate models one by one, record the output features B of the layers of the base candidate models as the processing results of the base models on the input C in . out . in ; S3 is a LoRA fine-tuning product based on the to-be-verified model as the base model, the MLP module processing logic of which is associated with the base model, and a gradient descent algorithm is used to solve an approximate input B middle , so that the processing result of the MLP module of the to-be-verified model on B middle is equal to the output feature B out of each layer of the base candidate model; and middle B middle is taken as the theoretical output of the base model at the layer Attention module. S4. C output by the Attention model to be verified middle B output by the base model approximating the Attention model middle , randomly select a row vector matching the dimension of the hidden layer, concatenate the row to form a matrix, perform singular value decomposition (SVD) on the matrix, and extract the rank information of LoRA according to the decomposition result; S5. Repeat multiple random selection of different row vector combinations, calculate the rank information obtained each time, take the smallest rank as the target rank r, reflect the essence of LoRA fine-tuning through the minimum rank Low-rank characteristics, compare the target rank r obtained with the hidden layer dimension d of the model to be verified.

2. The method for detecting LoRA fine-tuning origin of a large language model according to claim 1, wherein, The specific steps of generating a specific vocabulary in step S1 are: 1) Based on the application scene of the model to be verified, filter the vocabulary range, including common nouns, verbs, adjectives and domain-specific terms, to ensure the diversity of vocabulary distribution; 2) Call the NLTK tool to expand the basic vocabulary set according to the NLTK vocabulary library, and through preprocessing operations such as de-duplication and stop word filtering, reduce the size of the vocabulary table; 3) Input the reduced vocabulary table into the tokenizer of the model to be verified, check if there are words that cannot be correctly tokenized, replace or delete the words that do not pass the verification, verify the compatibility and finally form a vocabulary table that adapts to the tokenizer; In step S1, the words in the vocabulary table are converted into a one-dimensional tensor, which includes: 1) Input each word in the output vocabulary table into the tokenizer of the model to be verified, respectively, to obtain the corresponding tokenization ID, represented as a discrete number of words; 2) Input the tokenization ID into the embedding layer of the model to be verified, convert the discrete ID into a continuous vector representation through the embedding matrix, and form an embedding vector through the embedding layer mapping; 3) The output vector of the embedding layer is a two-dimensional tensor, represented as [1, embedding dimension], where 1 represents a single word and the embedding dimension is the preset dimension of the model. Through the squeezing operation, the redundant dimension is removed, and finally a one-dimensional tensor represented as [embedding dimension] is adjusted.

3. The method for detecting large language model (LoRA) fine-tuning origin according to claim 1, characterized in that, In step S1, according to the input of the model to be verified, record the intermediate features in each layer of the model, which includes: 1) Input the converted one-dimensional tensor as input data in order, and then pass it into the multi-level architecture of the model to be verified one by one, and then track the model calculation process layer by layer; 2) When a one-dimensional tensor is input C in as it flows through the model, its tensor data at the time of entering the current layer is recorded as the input C in of the layer, that is, the input C in of the first layer of the model in is the final output tensor of the first layer, and so on; 3) Determine the location and output node of the Attention module in the current layer, and obtain the output C of the Attention module middle When input C in After the Attention module in the current layer is processed, record the output tensor of the module as the output C of the layer middle ; 4) for each input one-dimensional tensor, its corresponding layer input C in and output C middle The hierarchical number is associated with the storage to form the mapping relationship of input-layer-feature, which ensures that the corresponding feature data can be traced according to the input and the level; 5) For all one-dimensional tensor inputs in the vocabulary, repeat the execution of the tracking model computation and associated input, complete the record of each layer C in and C middle for all inputs, forming a complete intermediate feature dataset; The steps of selecting a series of base candidate models in step S2 include: 1) Collect the public information of the model to be verified, including release time, basic model type and application field, and preliminarily determine the range of base models to be selected; 2) Based on the network architecture of the model to be verified, including the number of Transformer layers, hidden layer dimensions and attention head numbers, filter models with the same or compatible architecture as candidate bases; 3) Select widely used large language models into the candidate set to ensure the comprehensiveness of the candidate range. According to the above steps, the selected models are integrated to form a base candidate model library, which is then classified and stored according to the model name, version and architecture parameters for easy calling and matching.

4. The method for detecting LoRA fine-tuning origin of a large language model according to claim 1, wherein, In step S2, the output features B of each layer of the base candidate model are obtained out Specifically, it includes 1) according to each selected base candidate model, first analyze its hierarchical structure to ensure one-to-one correspondence with the hierarchical structure of the model to be verified, extract the input C of each layer of the model to be verified according to the recorded intermediate feature data in , and input it into the corresponding layer of the base candidate model in hierarchical order one by one; 2) When C in After flowing through the corresponding level of the base candidate model, record the output B out of the base candidate model, ensure that the record of B_out is strictly associated with the level, for all levels of the base candidate model, repeat the input of the output B out of the recorded base candidate model of the model to be verified, complete the record of B out of all levels of the candidate model, and then perform the same process on the selected other base candidate models until the B out of all candidate models is recorded; approximating the intermediate features B in the base model in step S3 middle The calculation step of the intermediate features B comprises: 1) To solve the approximate input B middle , the MLP module parameters of the input model to be verified and the base candidate model output feature B out , randomly generate an initial tensor matching the dimension of B out as the initial value of B middle ; 2) calculate the output of the MLP module of the model to be verified on the current B middle , record this B middle as MLP(B middle ) and the difference between B out , and take this difference as the loss function, where the commonly used loss function is mean square error (MSE), expressed as Loss = MSE(MLP(B middle ), B out ); 3) Perform gradient descent optimization steps, which include: a. Compute the gradient of the loss function with respect to B by an automatic differentiation tool middle b. Update B using the gradient. b. Update B in the negative direction of the gradient middle of the value, its update formula B * middle = B middle - learning rate x gradient; c. Repeat the gradient calculation and parameter update steps above until the loss function value is less than a pre-set threshold, at which point the difference between MLP(B middle ) and B out is negligible. c. Repeat the gradient calculation and parameter update steps above until the loss function value is less than a pre-set threshold, at which point the difference between MLP(B middle ) and B out is negligible. 4) When the loss function converges, the current B * middle satisfies the approximation input of the MLP module result B out of the to-be-verified model, record it as the corresponding B middle of this level.

5. The method for detecting large language model (LoRA) fine-tuning origin according to claim 1, wherein, In step S4, extract the LoRA rank information according to the singular value decomposition, where the rank is the number of non-zero singular values in the matrix, which is used to reflect the low-rank characteristics of LoRA fine-tuning. The specific steps include: 1) C output by the Attention of the model to be verified middle B output by the Attention approximating the base model middle A row vector matching the dimension d of the model hidden layer is randomly selected, and a matrix M with a shape of [2d, d] is formed according to row splicing, where the number of rows of the matrix M is 2d, and the number of columns is d. 2) Singular value decomposition of matrix M, decomposition formula is M=U∑V^T, where U is the left singular matrix, shape is [2d,2d];∑ is the singular value diagonal matrix, shape is [2d,d]; the elements on the diagonal are singular values and are sorted in descending order, V^T is the transpose of the right singular matrix, shape is [d,d]; 3) Extract singular values from the diagonal of∑, set a minimum threshold and filter out the singular values greater than the threshold, count the number of valid singular values, which is the rank of matrix M, which can reflect the low-rank property of LoRA fine-tuning between the model to be verified and the base candidate model, LoRA realizes parameter update through low-rank matrix, and the rank is usually much smaller than the hidden layer dimension d.

6. The method for detecting large language model (LoRA) fine-tuning origin according to claim 1, wherein, The step of determining the minimum rank information in step S5 includes: 1) Based on the hidden layer dimension d of the model to be verified, set the number of repeated sampling, and perform the following operations for each sampling: a. from C middle and B middle renewed random selection of row vectors matching the hidden layer dimension d, each time with a different combination of row vectors b. Stitching the new matrix M by row i where i is the sample number; c. to M i perform singular value decomposition, SVD, repeatedly computing the rank r of the matrix i , r i denotes the number of significant singular values; 2) Collect all the ranks r1, r2,..., r n n is the number of samples, and the minimum value r min is the target rank information; 3) Check if the minimum rank r is stable across multiple samplings, if there is significant fluctuation, increase the number of samplings and recalculate. min whether it is stable across multiple samplings, if there is significant fluctuation, increase the number of samplings and recalculate.

7. The method for detecting large language model LoRA fine-tuning origin according to claim 6, characterized in that, The step of determining the fine-tuning origin in step S5 includes: 1) Based on the hidden layer dimension d of the model to be verified, the rank determination threshold is set, generally taking one tenth of d as the critical value, and the minimum rank r is determined min Numerical comparison with the hidden layer dimension d, including: a. If r min If d / 10, it indicates that the intermediate feature difference between the to-be-verified model and the current base candidate model can be explained by a low-rank matrix, which conforms to the LoRA fine-tuning feature, and the candidate model is preliminarily determined as the base model of the to-be-verified model. b. If r min ≥ d / 10, it means that the feature difference between the two cannot be explained by the low-rank matrix, which does not meet the LoRA fine-tuning low-rank characteristics, and the candidate model is excluded. 2) For all selected base candidate models, the output features of the base candidate model, the approximate intermediate features of the base model, the decomposition extracted LoRA rank information and the determination of the minimum rank information are executed one by one, and the r corresponding to each candidate model is obtained min According to the rule of recording the intermediate features of the to-be-verified model, all candidate models that satisfy r min / d < 10 are screened out. 3) If only one candidate model in the filtering result meets the condition, directly determine that the model is the base model of the model to be verified; If there are multiple candidate models that meet the condition, further combine the model architecture similarity and the domain relevance to determine the most possible base model.

8. A system for detecting large language model (LoRA) fine-tuning origins, comprising: The vocabulary generation module generates a vocabulary table suitable for the tokenizer and embedding layer of the model to be verified based on NLTK and other tools, ensuring that the vocabulary can be converted into a one-dimensional tensor. The feature acquisition module records C in , C middle and B out of the base candidate model of the to-be-verified model. a gradient calculation module, which solves B by a gradient descent algorithm middle , ensuring that MLP(B middle )=B out ; Singular Value Decomposition and Rank Analysis Module, performing SVD on the concatenation matrix of C middle and B middle , computing and filtering the minimum rank r; The origin determination module compares the size of r and d, and outputs the base model determination result of the model to be verified.

Images