A vertical field large model deployment method based on LoRA low rank adaptation and knowledge distillation

Abstract

The present application relates to the field of artificial intelligence, and more particularly to a vertical field large model deployment method based on LoRA low rank adaptation and knowledge distillation, which comprises the following steps: step 1, inputting the same layer input representation into the same position of the Transformer layer of the teacher model and the student model respectively, determining the distillation sensitivity of each layer according to the difference between the outputs of the two, and dividing the high sensitive layer, the medium sensitive layer and the low sensitive layer; step 2, distributing different rank values to each layer according to the distillation sensitivity classification, and constructing a LoRA low rank adaptation branch; step 3, setting the supervision weight of the intermediate layer representation alignment supervision in the same classification, and performing knowledge distillation training, after the training converges, retesting the compensation limited layer, performing rank expansion, combining each branch into a low rank incremental matrix, and adding it into the base model to complete the deployment. The present application improves the vertical field adaptation accuracy, and the inference delay after deployment remains consistent with the base model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for deploying large-scale models in vertical domains based on LoRA low-rank adaptation and knowledge distillation. Background Technology

[0002] Large models demonstrate strong capabilities in general natural language processing tasks. However, when applying them to vertical fields such as healthcare, law, and industrial operations, it is usually necessary to first adapt the base model to the domain before deploying the adaptation results to the inference device. Full-parameter domain fine-tuning can achieve high domain accuracy, but the adaptation output is comparable in size to the base model, often reaching tens of gigabytes, making it difficult to distribute and update at low cost in multi-domain, multi-site scenarios.

[0003] LoRA (Lower-Rank Adaptation) achieves adaptation by paralleling low-rank branches with the weight matrix, requiring only a small number of parameters for training. Knowledge distillation, on the other hand, uses the behavior of the teacher model to constrain the learning of the student model. The combination of these two approaches has become a common practice for vertical domain adaptation. However, existing methods typically assign the same rank to all network layers and apply equal intensity of distillation supervision to each intermediate layer, ignoring the differences in the required compensation levels for domain capability transfer between different layers. This results in insufficient capacity for layers requiring focused compensation, making it difficult to balance domain accuracy with the number of adaptation parameters. Some adaptive rank methods dynamically prune the rank based on the importance of parameters during training, with the signal originating from the student model itself. They lack both the means to pre-allocate capacity based on the functional differences between the teacher and base models layer by layer before training and the mechanism to verify the compensation level and add capacity in a targeted manner after training.

[0004] Therefore, there is a need for a method for deploying large-scale vertical domain models that can locate the compensation needs of each layer, configure the adaptation capacity and distillation supervision intensity with emphasis, and perform ex-post correction for layers with insufficient compensation. Summary of the Invention

[0005] In view of this, the present invention provides a method for deploying large vertical domain models based on LoRA low-rank adaptation and knowledge distillation, which aims to solve the problems in existing vertical domain adaptations where the rank settings of each Transformer layer are consistent and the distillation supervision intensity is uniform, resulting in insufficient layer capacity that needs to be compensated and difficulty in balancing domain accuracy and the number of adaptation parameters.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for deploying large-scale models in vertical domains based on LoRA low-rank adaptation and knowledge distillation includes the following steps:

[0008] Step 1: Measure distillation sensitivity using a layer-by-layer functional probing approach: The teacher model is a large model fine-tuned with all parameters in the domain, and the student model is a base model to be deployed. The Transformer layer positions of the teacher and student models correspond one-to-one. Input the domain-calibrated corpus into the teacher model to perform forward computation, recording the hidden states received by each Transformer layer of the teacher model as layer input representations. Input the layer input representations into the current Transformer layer and the Transformer layers in the same position in the student model respectively, obtaining the output representations of the teacher layer and the same input layer of the student model. Determine the distillation sensitivity of the current Transformer layer based on the difference between the two. Divide all Transformer layers of the student model into high-sensitivity layers, medium-sensitivity layers, and low-sensitivity layers according to the distillation sensitivity from high to low.

[0009] Step 2: Assign high-rank, medium-rank, and low-rank values ​​to each Transformer layer within the high-sensitivity, medium-sensitivity, and low-sensitivity layers, respectively, with high-rank values ​​being greater than medium-rank values ​​and medium-rank values ​​being greater than low-rank values; For each Transformer layer of the student model, connect the query projection matrix and value projection matrix in parallel to form a LoRA low-rank adaptation branch composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix, with the dimension of the output of the dimensionality reduction matrix equal to the rank value of the current Transformer layer; Freeze all parameters of the base of the teacher model and the student model, using all dimensionality reduction and increased dimensionality matrices as trainable parameters;

[0010] Step 3: Input the domain training corpus into both the teacher and student models simultaneously. Apply output distribution alignment supervision and intermediate layer representation alignment supervision to the student model. Set supervision weights for each Transformer layer, with the supervision weights decreasing sequentially from high-sensitivity to medium-sensitivity to low-sensitivity layers. This allows the same distillation sensitivity level to simultaneously drive rank value allocation and supervision weight setting. Iterate and update all dimensionality reduction and expansion matrices using gradient descent until training converges. After training converges, merge each LoRA low-rank adaptation branch into a low-rank increment matrix and distribute it to the deployment device. On the deployment device, superimpose each low-rank increment matrix into the corresponding query projection matrix or value projection matrix to obtain the deployment weight matrix and load it, completing the deployment of the large vertical domain model.

[0011] Furthermore, the teacher model and the student model use the same network architecture, the same number of Transformer layers, the same hidden dimension, and the same word segmentation method. The word segments of the same text by the teacher model and the student model correspond one-to-one. The domain calibration corpus consists of 500 to 2000 domain texts taken from the vertical domain. The layer input representation is the hidden state output of the previous Transformer layer in the teacher model, and the layer input representation of the first Transformer layer is the hidden state output by the word embedding.

[0012] Furthermore, the teacher layer output representation is output after the current Transformer layer completes self-attention calculation, feedforward calculation, and residual addition on the layer input representation. The teacher layer output representation is the hidden state of the current Transformer layer output in the teacher model's own forward calculation. The student same-input layer output representation is output after the same-position Transformer layer in the student model completes self-attention calculation, feedforward calculation, and residual addition on the same-position input representation. The teacher layer output representation and the student same-input layer output representation are obtained by passing the same layer input representation through the same-position Transformer layer. The difference between the two comes only from the transformation difference between the teacher model and the student model's Transformer layer at the current position. The interference from the difference propagation of upstream Transformer layers is isolated. Determining the distillation sensitivity of the current Transformer layer includes: for each text in the domain calibration corpus, at each word position, calculating the difference between the corresponding elements of the teacher layer output representation and the student same-input layer output representation, and taking the absolute value of the difference of all corresponding elements. The average of the values ​​is used to obtain the positional difference value of the current word position; the average of the positional difference values ​​of the current Transformer layer across all word positions in the current text is used to obtain the layer local difference of the current Transformer layer for the current text; the average of the absolute values ​​of the teacher layer output representing all elements at all word positions in the current text is used as the layer output magnitude of the current Transformer layer for the current text; the layer local difference is divided by the layer output magnitude to obtain the layer relative difference of the current Transformer layer for the current text, which is based on the layer output magnitude and can still be compared among Transformer layers even when there are differences in the magnitude of the hidden state values; the average of the layer relative difference of the current Transformer layer across all texts in the domain calibration corpus is used to obtain the distillation sensitivity of the current Transformer layer, which is used as the distillation sensitivity of the current Transformer layer for Transformer layers in the same position in the student model.

[0013] Furthermore, the division into high-sensitivity, medium-sensitivity, and low-sensitivity layers includes: sorting all Transformer layers of the student model from high to low distillation sensitivity; assigning Transformer layers at the top of the sorting results to high-sensitivity layers, those in the middle to medium-sensitivity layers, and those at the bottom to low-sensitivity layers; the difference between any two of the number of Transformer layers in the high-sensitivity, medium-sensitivity, and low-sensitivity layers is at most 1; the distillation sensitivity classification serves as a unified basis for rank value allocation and supervision weight setting.

[0014] Furthermore, the query projection matrix and value projection matrix are the weight matrices used to generate the query vector and value vector, respectively, in the self-attention sublayer of each Transformer layer in the student model; the dimensionality reduction matrix projects the input vector to a low-dimensional space with a dimension equal to the rank value of the current Transformer layer, and the dimensionality increase matrix maps the vector in the low-dimensional space back to the original output dimension of the query projection matrix or value projection matrix. The output of the LoRA low-rank adaptation branch is scaled by a preset scaling factor and then added element-wise to the original output of the query projection matrix or value projection matrix; the initial value of the dimensionality reduction matrix is ​​randomly generated. The initial values ​​of the upgraded matrix are all set to 0. During the initial training phase, the output of the LoRA low-rank adaptation branch is a 0 vector. The student model gradually adapts from the original behavior of the base model. Merging each LoRA low-rank adaptation branch into a low-rank increment matrix involves multiplying the dimension reduction matrix and the upgraded matrix in the current LoRA low-rank adaptation branch and scaling them with a scaling factor to obtain the low-rank increment matrix. The forward computation during the deployment phase uses the deployment weight matrix and performs matrix operations of the same scale as the base model. The inference latency of the deployed vertical domain large model is consistent with the inference latency of the base model.

[0015] Furthermore, for each text in the domain training corpus, the output distribution alignment supervision is as follows: the probability distribution of lexical units given by the teacher model's output layer for each position to be predicted in the current text is used as the soft objective, constraining the probability distribution of lexical units given by the student model's output layer for the same position to be predicted in the current text to move closer to the soft objective; the intermediate layer representation alignment supervision is as follows: the hidden state output by the teacher model's current Transformer layer after completing self-attention calculation, feedforward calculation, and residual addition is taken as the teacher's intermediate representation, and the hidden state output by the Transformer layer at the same position in the student model after completing self-attention calculation, feedforward calculation, and residual addition is taken as the intermediate representation. The student intermediate representation, with the current Transformer layer's supervision weight as the strength, constrains the student intermediate representation to move closer to the teacher's intermediate representation; the greater the supervision weight, the stronger the constraint. Distillation sensitivity is obtained before training through layer-by-layer function probing. Output distribution alignment supervision and intermediate layer representation alignment supervision are applied to the forward computation trajectory of the domain training corpus, enabling the student model to reproduce the teacher model's behavior on the real inference path. The output distribution alignment supervision and the intermediate layer representation alignment supervision, weighted according to the supervision weights of all Transformer layers, are used together as the training objective. When the numerical change of the training objective between adjacent training rounds is less than a preset convergence threshold, training is considered converged.

[0016] Furthermore, it also includes: after training convergence and before merging, performing retesting on the domain calibration corpus in a layer-by-layer functional probing manner: inputting the layer input representation of the current Transformer layer into the current Transformer layer to obtain the teacher layer output representation; inputting the same layer input representation into the Transformer layer in the same position in the student model, and having the same position Transformer layer with parallel LoRA low-rank adaptation branches complete self-attention calculation, feedforward calculation, and residual addition, superimposing the output of the corresponding LoRA low-rank adaptation branch in the self-attention calculation to obtain the student retest layer output representation; calculating the residual relative difference of the current Transformer layer using the teacher layer output representation and the student retest layer output representation according to the calculation methods of positional difference value, layer local difference degree, layer output amplitude, and layer relative difference degree; dividing the residual relative difference degree by the distillation sensitivity of the current Transformer layer to obtain the residual proportion of the current Transformer layer; determining the Transformer layer with a residual proportion lower than a preset proportion threshold as a compensation qualified layer, and determining the Transformer layer with a residual proportion reaching the preset proportion threshold as a compensation limited layer.

[0017] Furthermore, this also includes: for each compensation-constrained layer, in each LoRA low-rank adaptation branch of the current Transformer layer, adding rows to the dimensionality reduction matrix equal to the preset extended rank value, with the newly added rows initialized with randomly generated values; adding columns to the dimensionality increase matrix equal to the extended rank value, with all newly added columns set to 0; increasing the rank value of the current Transformer layer by the extended rank value; this constitutes rank expansion; at the moment of rank expansion, the output of the LoRA low-rank adaptation branch retains the value before rank expansion; setting the dimensionality reduction and increase matrices of all compensation-constrained layers as trainable parameters; freezing the LoRA low-rank adaptation branches of the compensation-compliant layers; continuing training with output distribution alignment supervision and intermediate layer representation alignment supervision until training converges again; ending the retest and rank expansion when all Transformer layers are determined to be compensation-compliant layers, or when the retest and rank expansion have been repeated for a preset number of rounds; thereby verifying the compensation degree of each Transformer layer after training through layer-by-layer functional probing, and adding the rank value to the compensation-constrained layer in a targeted manner to correct the deviation between the static measurement before training and the compensation requirement after training.

[0018] Furthermore, the high-rank value is 16, the medium-rank value is 8, and the low-rank value is 4; the supervision weight of the high-sensitivity layer is 1.0, the supervision weight of the medium-sensitivity layer is 0.5, and the supervision weight of the low-sensitivity layer is 0.2.

[0019] Furthermore, the preset percentage threshold is 0.5, the extended rank value is 8, and the preset number of rounds for retesting and rank extension is 2.

[0020] By adopting the above technical solution, the present invention has produced the following beneficial effects:

[0021] This invention, before training, inputs the same layer's representation into the Transformer layers of the teacher model and the Transformer layers at the same position in the student model. The difference between their outputs stems only from the transformation differences within the Transformer layers of the two models at the current position, isolating interference from upstream layer differences. The resulting distillation sensitivity accurately reflects the degree of compensation required for vertical domain capability transfer in each layer. The relative layer difference is based on the layer output amplitude, allowing for comparison even when layers differ in magnitude, resulting in stable and reliable grading. The same distillation sensitivity grading simultaneously drives rank allocation and supervisory weight setting. Layers requiring significant compensation simultaneously gain greater trainable capacity and stronger teacher constraints. These two mechanisms work together on the same batch of layers, improving vertical domain adaptation accuracy with a comparable total number of trainable parameters and suppressing parameter waste in redundant layers. After training convergence, the same layer-by-layer function is used to verify the compensation degree of each layer. For layers with limited compensation, rank expansion is performed while maintaining branch output stability, and training continues in a targeted manner, correcting the deviation between pre-training static metrics and post-training actual compensation requirements. The adaptation results are distributed in the form of a compact low-rank incremental matrix. After being superimposed on the base model, the deployment phase performs matrix operations of the same scale as the base model, keeping the inference latency consistent. This facilitates the distribution, updating, and deployment of large models in vertical domains. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the principle of layer-by-layer functional detection in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the data flow principle for step-by-step calculation of distillation sensitivity in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the distillation sensitivity distribution and three-level division of each Transformer layer in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structural principle of the LoRA low-rank adaptation branch in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the heterogeneous rank allocation of the three levels on each Transformer layer in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram illustrating the structural principle of output distribution alignment supervision and intermediate layer representation alignment supervision in an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the training target convergence process in an embodiment of the present invention;

[0029] Figure 8This is a schematic diagram of the residual percentage of each Transformer layer obtained by retesting in an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram illustrating the principle of compensating for zero perturbation in the confined layer rank extension of this invention.

[0031] Figure 10 This is a schematic diagram of the comparison of accuracy of the domain test set in an embodiment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Without conflict, the following embodiments and their technical features can be combined with each other.

[0033] A method for deploying large-scale models in vertical domains based on LoRA low-rank adaptation and knowledge distillation includes the following steps:

[0034] Step 1: Measure distillation sensitivity using a layer-by-layer functional probing approach: The teacher model is a large model fine-tuned with all parameters in the domain, and the student model is a base model to be deployed. The Transformer layer positions of the teacher and student models correspond one-to-one. Input the domain-calibrated corpus into the teacher model to perform forward computation, recording the hidden states received by each Transformer layer of the teacher model as layer input representations. Input the layer input representations into the current Transformer layer and the Transformer layers in the same position in the student model respectively, obtaining the output representations of the teacher layer and the same input layer of the student model. Determine the distillation sensitivity of the current Transformer layer based on the difference between the two. Divide all Transformer layers of the student model into high-sensitivity layers, medium-sensitivity layers, and low-sensitivity layers according to the distillation sensitivity from high to low.

[0035] Step 2: Assign high-rank, medium-rank, and low-rank values ​​to each Transformer layer within the high-sensitivity, medium-sensitivity, and low-sensitivity layers, respectively, with high-rank values ​​being greater than medium-rank values ​​and medium-rank values ​​being greater than low-rank values; For each Transformer layer of the student model, connect the query projection matrix and value projection matrix in parallel to form a LoRA low-rank adaptation branch composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix, with the dimension of the output of the dimensionality reduction matrix equal to the rank value of the current Transformer layer; Freeze all parameters of the base of the teacher model and the student model, using all dimensionality reduction and increased dimensionality matrices as trainable parameters;

[0036] Step 3: Input the domain training corpus into both the teacher and student models simultaneously. Apply output distribution alignment supervision and intermediate layer representation alignment supervision to the student model. Set supervision weights for each Transformer layer, with the supervision weights decreasing sequentially from high-sensitivity to medium-sensitivity to low-sensitivity layers. This allows the same distillation sensitivity level to simultaneously drive rank value allocation and supervision weight setting. Iterate and update all dimensionality reduction and expansion matrices using gradient descent until training converges. After training converges, merge each LoRA low-rank adaptation branch into a low-rank increment matrix and distribute it to the deployment device. On the deployment device, superimpose each low-rank increment matrix into the corresponding query projection matrix or value projection matrix to obtain the deployment weight matrix and load it, completing the deployment of the large vertical domain model.

[0037] The implementation process of the above method is elaborated below with a specific application scenario. This embodiment is aimed at the medical vertical field: the teacher model is a large medical domain model obtained by fine-tuning all parameters on a general base model, and the student model is the same general base model itself; both are composed of 32 stacked Transformer layers, with a hidden dimension of 4096, 32 self-attention heads, a vocabulary size of 32,000, and a total of approximately 7 billion parameters. The deployment device is a server equipped with one inference accelerator card. The deployment requirements of this scenario are representative: the hospital information department needs to run medical question answering and medical record writing assistance services in a private environment within the hospital. It requires the answer quality to be close to the large domain model with full parameter fine-tuning, but is limited by the hospital's computing power and network bandwidth, and cannot frequently transmit full weights of tens of gigabytes, nor can it accept a perceptible increase in inference latency relative to the base model. The teacher model and the student model use the same network architecture, the same number of Transformer layers, the same hidden dimensions, and the same word segmentation method. The word segments of the same text correspond one-to-one. Each Transformer layer in the teacher model corresponds one-to-one with the Transformer layer in the same position in the student model. This is the premise for the implementation of layer-by-layer functional exploration.

[0038] Step 1 involves preparing the domain calibration corpus. This embodiment selects 1000 medical texts from treatment guidelines, medical record summaries, and doctor-patient Q&A records to form the domain calibration corpus. Each text, after word segmentation, contains no more than 512 lexical units. Each text is input into the teacher model for forward computation. At each Transformer layer's input position in the teacher model, the hidden states received by the current Transformer layer for all lexical positions of the current text are recorded as the layer input representation for the current Transformer layer. The layer input representation is the hidden state output by the previous Transformer layer; the layer input representation of the first Transformer layer is the hidden state output by the word embedding. Probing is performed layer by layer during the teacher model's forward computation: whenever a layer input representation of a Transformer layer is generated, it is immediately fed into the corresponding Transformer layer in the student model to complete a layer-by-layer forward pass and accumulate statistics, then the layer input representation is released. Using this on-demand approach, the additional GPU memory occupied by peak values ​​is equivalent to the hidden state of one Transformer layer, approximately 4 megabytes for a text of 512 terms, and the domain calibration corpus does not require full resident storage. Statistical collection is achieved by registering forward hooks at the input and output positions of each Transformer layer in the teacher model. The hooks trigger the layer-level forward pass on the student side when the layer input representation is generated. The absolute value accumulation and magnitude accumulation of the differences are both performed using single-precision floating-point, avoiding the precision loss caused by half-precision accumulation on long texts. The entire probing process only performs forward computation and does not generate any gradients; 1000 domain calibration corpora are completed in approximately 7 minutes on a single inference accelerator card.

[0039] See Figure 1For each Transformer layer in the teacher model, the layer input representation of the current Transformer layer is input into the current Transformer layer. The current Transformer layer performs self-attention calculation, feedforward calculation, and residual summation to obtain the teacher layer output representation. The teacher layer output representation is the hidden state of the current Transformer layer output in the teacher model's own forward calculation, so no redundant calculation occurs on this side. At the same time, the same layer input representation is input into the Transformer layer in the student model at the same position. The Transformer layer in the student model at the same position performs self-attention calculation, feedforward calculation, and residual summation to obtain the student's same input layer output representation. The reason for using the same layer of input representation for comparison is that if the two models were to advance along their own forward trajectories and then compare their outputs at a certain layer, the differences observed at the 12th layer would inevitably be mixed with components from the differences in the previous 11 layers, which would be propagated and amplified layer by layer. The measurement results would be contaminated by the upstream, making it impossible to discern the magnitude of the transformation differences within the current layer itself. However, when the two Transformer layers receive identical inputs, the difference in outputs is solely due to the parameter differences between the Transformer layers of the teacher and student models at the current position, while the differences between the upstream Transformer layers are isolated from the detection. Using the forward trajectory of the teacher model as the detection benchmark also has a clear rationale: the teacher model carries the target behavior after fine-tuning across the entire parameter domain, and layer-by-layer functional probing measures precisely the functional gaps of each layer of the student model relative to the target behavior. As a comparison, we also tried to compare the output of the two models layer by layer after they were independently advanced along their own forward trajectories: the difference reading at the 20th layer under this caliber was about 3 times that of the controlled caliber, and it expanded monotonically with the layer depth, making it almost impossible to distinguish which layers the real functional gap was located in, which in turn confirmed the necessity of the controlled condition of input at the same layer.

[0040] The quantification of differences is performed in order from fine to coarse. At each lexical position, the difference between corresponding elements of the teacher's output representation and the student's output representation at the same input layer is first calculated. Then, the absolute values ​​of the differences of all corresponding elements are averaged to obtain the positional difference value at the current lexical position.

[0041]

[0042] in, The sequence number of the Transformer layer. This represents the total number of Transformer layers in this embodiment. Take 32; To calibrate the serial numbers of texts in the corpus for the domain, In this embodiment, the total number of texts in the domain calibration corpus is [number missing]. Take 1000; This represents the sequence number of the word's position. For the first The number of terms in a text; The index of the hidden dimension. To hide the dimension, in this embodiment Take 4096; For the first Article text number The teacher layer output representation at the lexical position Dimensions The elements on For students at the same level and input layer of the same text, the output representation is located at the word position. Dimensions The elements on For the first Article text number Layer at word position The positional difference values ​​of the current Transformer layer across all word positions in the current text are averaged to obtain the layer-local difference degree of the current Transformer layer for the current text.

[0043]

[0044] in, For the first Layer for the first The degree of local difference in the text layer.

[0045] Layer local dissimilarity is an absolute quantity, and it is inappropriate to directly compare layers. The summation of residuals causes the numerical magnitude of the hidden state to accumulate layer by layer with depth; the numerical magnitude of the hidden state in deeper layers is generally greater than that in shallower layers. In this embodiment, the output magnitude of layer 30 is approximately 6 times that of layer 2. If ranked directly by layer local dissimilarity, deeper layers will receive a higher ranking simply because of their larger numerical magnitude; the metric is dominated by magnitude rather than functional difference. To eliminate this difference in magnitude, we first calculate the output magnitude of the current Transformer layer for the current text, that is, the average of the absolute values ​​of the teacher layer output representing all elements at all word positions in the current text:

[0046]

[0047] in, For the first Layer for the first The layer output amplitude of the text. Then divide the layer local difference by the layer output amplitude to obtain the layer relative difference of the current Transformer layer with respect to the current text:

[0048]

[0049] in, For the first Layer for the first The relative difference between layers of text; To prevent extremely small quantities with denominators, in this embodiment... The value is set to 0.000001, which is much smaller than any actual layer output magnitude and serves only as a numerical protection measure. Layer relative dissimilarity measures the proportion of the transform difference relative to the current layer's own output magnitude; different Transformer layers can still be compared even when their hidden state numerical magnitudes differ. The distillation sensitivity of the current Transformer layer is obtained by averaging the layer relative dissimilarity of the current Transformer layer across all texts in the domain calibration corpus.

[0050]

[0051] in, For the first Distillation sensitivity of layers. Since there is a one-to-one correspondence between the Transformer layer positions in the teacher model and the student model, the distillation sensitivity of the current Transformer layer is used as the distillation sensitivity of the Transformer layers in the student model with the same position. The step-by-step calculation relationship from positional difference value through local layer difference, relative layer difference, and finally distillation sensitivity is as follows: Figure 2 As shown. In the step-by-step calculation, the positional difference value characterizes the deviation of a single word position, the layer local difference degree aggregates them into a layer reading on a text, the layer relative difference degree eliminates the amplitude difference between layers, and the distillation sensitivity is then averaged on the corpus to suppress the randomness of a single text. Each of the four levels of statistics undertakes a clear reduction responsibility.

[0052] like Figure 3 As shown, the distillation sensitivity of the 32 Transformer layers in this embodiment ranges from 0.021 to 0.147, exhibiting a distribution pattern that is higher in the middle and later parts and lower at both ends: layer 1 is 0.021, layer 4 is 0.033, layer 8 is 0.072, layer 12 is 0.121, layer 16 reaches a peak of 0.147, layer 20 is 0.128, layer 24 is 0.094, layer 28 is 0.063, and layer 32 is 0.030. This is consistent with the pattern that full-parameter domain fine-tuning mainly rewrites the functions of medium and deep semantic processing, while shallow lexical and syntactic processing is relatively universal. Sorting all Transformer layers of the student model from high to low distillation sensitivity, the first 11 are classified as high-sensitivity layers, corresponding to layers 12 to 22. Figure 3The middle 11 are represented by black bars; the middle 11 are classified into medium-sensitivity layers, corresponding to layers 7 to 11 and 23 to 28, represented by diagonal bars; the remaining 10 are classified into low-sensitivity layers, corresponding to layers 1 to 6 and 29 to 32, represented by blank bars. The difference in the number of Transformer layers contained in the three levels is at most 1. If Transformer layers with the same distillation sensitivity value appear in the sorting, it is agreed that the layer with the earlier number will be classified into a higher level. No tie occurred in this embodiment. The classification uses the ranking rather than a fixed threshold because the absolute value of distillation sensitivity will fluctuate with the domain and corpus. The ranking criteria do not need to be recalibrated for each new domain, and the classification results are more stable. There is also a clear reason for taking 3 levels: two levels can only express emphasis and no emphasis, which is difficult to characterize the transition zone of sensitivity distribution; four levels or more make the difference between the rank value and supervision weight between adjacent levels too small, diluting the significance of the classification, while increasing the configuration complexity exponentially.

[0053] In step 2, the high-rank value of 16 is assigned to each Transformer layer within the high-sensitivity layer, the medium-rank value of 8 is assigned to each Transformer layer within the medium-sensitivity layer, and the low-rank value of 4 is assigned to each Transformer layer within the low-sensitivity layer. The query projection matrix and value projection matrix are weight matrices used to generate query and value vectors respectively in the self-attention sublayers of each Transformer layer in the student model, both with a size of 4096×4096. These two types of matrices were chosen as parallel objects because query projection determines how attention is distributed among terms, and value projection determines the expression form of the aggregated content; both have the most direct impact on domain behavior. Key projection and query projection appear in pairs in attention scoring; paralleling only one can change the scoring structure. The gain from paralleling both is limited; in actual tests, the accuracy difference under the same parameter budget is no more than 0.2 percentage points, while the number of branches is halved and the engineering implementation is simpler. See also... Figure 4 Within each Transformer layer of the student model, a LoRA low-rank adaptation branch is connected in parallel to both the query projection matrix and the value projection matrix: the dimension reduction matrix projects the 4096-dimensional input vector to a low-dimensional space with a dimension equal to the rank of the current Transformer layer; the dimension increase matrix maps the vector in the low-dimensional space back to the original 4096-dimensional output dimension. The branch output is scaled by a scaling factor and then added element-wise to the original output. The projection calculation with the LoRA low-rank adaptation branch is represented as follows:

[0054]

[0055] in, The input vector; This is the output vector after superimposing the branch outputs; For the first The query projection matrix or value projection matrix of the layer; For the first The dimension reduction matrix in the corresponding branch of the layer has a size of Row 4096; This is an increasing-dimensional matrix within the same branch, with a size of 4096 rows. List; For the first The rank value of the layer; The scaling factor is 2. In this embodiment, the scaling factor is set to 2, and it is intentionally set as a preset constant decoupled from the rank value: subsequent rank expansion requires the branch output to remain unchanged during the expansion. If the scaling factor is linked to the rank value, an increase in the rank value will change the scaling factor of the existing branch output, thus destroying the zero-perturbation property. Therefore, all branches in each layer use the same preset constant. Comparative training was conducted with scaling factors of 1, 2, and 4, and the final domain accuracy was 85.0%, 85.2%, and 84.8%, respectively, with limited differences. Setting it to 2 balances the effective amplitude of the branch output and the training stability, so this embodiment uses 2. For initialization, the initial value of the dimensionality reduction matrix is ​​taken from a Gaussian distribution with a mean of 0 and a standard deviation of 0.02, which is a randomly generated value; the initial value of the dimensionality increase matrix is ​​set to 0. After this setting, the output of each LoRA low-rank adaptation branch in the initial stage of training is exactly a 0 vector. The student model gradually adapts from the original behavior of the base model, and no unnecessary behavioral deviation is introduced in the early stage of training. In contrast, if the dimension-upgrading matrix is ​​also randomly initialized, and random perturbations are injected into the output from the start of training, the measured training target values ​​in the first two training epochs are significantly higher and more volatile, delaying convergence by about one epoch. The rank distribution of the three levels across all 32 Transformer layers is as follows: Figure 5 As shown: the larger the rank value, the more rows in the dimensionality reduction matrix and the more columns in the dimensionality increase matrix, the larger the dimension of the low-dimensional space defined by the branch, and the larger the trainable capacity. Subsequently, all parameters of the teacher model and all parameters of the student model base are frozen, and only all dimensionality reduction and dimensionality increase matrices are set as trainable parameters. The total number of branch parameters in this embodiment is... The number of parameters is approximately 0.07% of the total number of base model parameters, and is stored in half-precision floating-point format for approximately 9.96 megabytes.

[0056] Step 3 training is performed on the domain training corpus. In this embodiment, the domain training corpus consists of 80,000 medical Q&A, clinical guideline excerpts, and medical record rewrites, independent of the domain calibration corpus to avoid the metric and training originating from the same source. Q&A texts account for approximately 60% of the domain training corpus, while guideline excerpts and medical record rewrites together account for approximately 40%. The number of tokens in each text after segmentation does not exceed 1024, and excessively long texts are segmented according to semantic boundaries. Each text in the domain training corpus is simultaneously input into both the teacher model and the student model for forward computation. Output distribution alignment supervision and intermediate layer representation alignment supervision are applied to the student model. Output distribution alignment supervision uses a softened token probability distribution as its carrier.

[0057]

[0058] in, This represents the index of the position to be predicted in the current text. This represents the number of positions to be predicted in the current text. and The output layers of the teacher model and the student model are respectively located at the positions to be predicted. The vocabulary scoring vectors given above; For temperature, in this embodiment Take 2; For soft targets, this refers to the probability distribution of lexical units in the vocabulary given by the teacher model's output layer after temperature softening. This represents the lexical probability distribution of the student model's output layer after being softened at the same temperature. A temperature greater than 1 increases the probability of non-optimal lexical terms, allowing relative preference information, such as synonym differentiation among medical terms, to be revealed and passed on to the student model. The output distribution alignment supervision constrains the student model's lexical probability distribution to move closer to the soft target:

[0059]

[0060] in, This refers to the ordinal number of the word in the vocabulary list. For vocabulary size, in this embodiment Take 32000; and The lexical probability distributions of the soft target and student models are respectively in the lexical... The probability of it; To align the output distribution with the training target component corresponding to the supervision, the average value is calculated over all text within a training batch. (Coefficients) This is used to compensate for the reduction in gradient amplitude caused by temperature softening, ensuring that the strength of this component does not drift with the temperature setting. The reason for choosing KL divergence instead of hard-labeled cross-entropy is that the soft target itself is already a complete probability distribution. KL divergence directly measures the deviation of the student distribution from the soft target distribution, and its minimum point is precisely where the two distributions overlap, making the meaning of the training objective clear. If hard-labeled cross-entropy were used, all the near-sense preference information given by the teacher model would be lost.

[0061] Intermediate layer representations are aligned with supervision applied layer by layer. The hidden state output by the current Transformer layer in the teacher model after completing self-attention calculation, feedforward calculation, and residual summation is taken as the teacher's intermediate representation. Similarly, the hidden state output by the Transformer layer in the student model at the same position after completing self-attention calculation, feedforward calculation, and residual summation is taken as the student's intermediate representation. The position used is the same as that used in step 1 for the teacher layer's output representation. The layer constraints are:

[0062]

[0063] in, The intermediate representation of teachers at the location to be predicted Dimensions The elements on For students, the intermediate representation is the elements at the same location and in the same dimension. For the first The intermediate layers represent the training target components corresponding to the alignment supervision. Supervision weights are assigned to each Transformer layer according to the classification in step 1: 1.0 for high-sensitivity layers, 0.5 for medium-sensitivity layers, and 0.2 for low-sensitivity layers. The supervision weights and rank values ​​originate from the distillation sensitivity classification, aiming to ensure that capacity and constraints fall on the same batch of Transformer layers: high-sensitivity layers receive a larger trainable capacity and are subject to stronger teacher constraints; capacity provides the expression space, and strong constraints direct this space towards teacher behavior. Low-sensitivity layers have very little difference from the teacher; if equal-strength constraints are applied, the gradient will be wasted on numerical noise and will compete with the output distribution alignment supervision for limited trainable capacity. The correspondence between the application positions of the two supervisions and the layer-by-layer weights is as follows: Figure 6 As shown. The complete training objective is:

[0064]

[0065] in, For the first Layer supervision weights The training objective is to determine the distillation sensitivity obtained before training via controlled probing under the same input layer, which is used to determine the capacity and the emphasis of supervision. The two alignment supervisions are applied to the forward computation trajectory of the domain training corpus. The student model reproduces the behavior of the teacher model on the real inference path. The two input calibers each perform their respective functions and are not contradictory.

[0066] The optimization employs the AdamW adaptive moment estimator with weight decay, with a weight decay coefficient of 0.01, a learning rate of 0.0001, and a batch size of 16. Each training epoch fully traverses the domain training corpus once, iteratively updating all dimensionality reduction and expansion matrices using gradient descent. Training is performed using bfloat16 mixed precision, with gradients accumulated using single precision. Gradient clipping with a norm upper limit of 1.0 is applied to all trainable parameters to prevent abrupt parameter changes caused by outliers in individual batches. The learning rate decays to 0.00001 cosine-wise within each training epoch, returning to its initial value in the next epoch. Two supervision operations are calculated only at the word positions in the response segment; word positions in the question segment are excluded to prevent the student model from wasting its limited capacity on restating the input. A preset convergence threshold of 0.02 is used; training is considered convergent when the numerical change of the training objective between adjacent training epochs is less than the preset convergence threshold. In this embodiment, the training target values ​​at the end of rounds 1 to 6 are 2.86, 1.63, 1.34, 1.27, 1.245, and 1.232, respectively. The difference between round 6 and round 5 is 0.013, which is less than the preset convergence threshold, indicating that training has converged. The convergence process is as follows: Figure 7 As shown, a single training epoch takes approximately 1.8 hours on a training accelerator card equipped with 80 gigabytes of video memory. The total number of trainable parameters is less than one-thousandth of the base parameters, and the video memory usage mainly comes from forward activation rather than the trainable parameters themselves.

[0067] After training convergence and before merging, a retest is performed on the domain calibration corpus using a layer-by-layer functional probing approach. The retest uses the exact same 1000 domain calibration corpora and statistical methods as the pre-training probing, ensuring layer-by-layer comparability between the two metrics. The retest also only involves forward computation and takes approximately 7 minutes. During the retest, the layer input representation of the current Transformer layer is input into the current Transformer layer to obtain the teacher layer output representation. The same layer input representation is input into the Transformer layer in the student model at the same position. In the self-attention calculation, the original outputs of the query projection matrix and the value projection matrix are superimposed on the outputs of their respective LoRA low-rank adaptation branches scaled by a scaling factor. This completes the self-attention calculation, feedforward calculation, and residual addition, yielding the student retest layer output representation. Using the teacher layer output representation and the student retest layer output representation, the layer relative difference is calculated using the same method for positional difference, layer local difference, layer output amplitude, and layer relative difference, denoted as . The residual relative dissimilarity of the current Transformer layer is obtained by averaging across all texts in the domain calibration corpus.

[0068]

[0069] in, For the first The residual relative dissimilarity of the layer. Dividing the residual relative dissimilarity by the distillation sensitivity of the current Transformer layer yields the residual percentage of the current Transformer layer:

[0070]

[0071] in, For the first The residual percentage of a layer refers to the proportion of functional gaps that remain unfilled after distillation training. A preset threshold of 0.5 is used. Transformer layers with a residual percentage below this threshold are considered adequately compensated layers, while those with a residual percentage reaching the threshold are considered undercompensated layers. Figure 8As shown, the residual percentages of the 28 Transformer layers in this embodiment range from 0.12 to 0.44, and are therefore classified as layers that have met the compensation criteria. The residual percentages of layers 14, 17, 19, and 26 are 0.63, 0.58, 0.71, and 0.55, respectively, and are classified as layers with limited compensation. The first three are high-sensitivity layers, while layer 26 is a medium-sensitivity layer. The reason for the existence of layers with limited compensation is that distillation sensitivity reflects the size of the functional gap, but it is not equal to the rank required to bridge the gap. The functional differences of individual Transformer layers exhibit a high-rank structure, and the rank value estimated by the gap size during the static pre-allocation stage is insufficient to express this difference. This is precisely why it is valuable to verify the degree of compensation after training.

[0072] For each compensation-constrained layer, rank expansion is performed, with an expanded rank value of 8. In each LoRA low-rank adaptation branch of the current Transformer layer, 8 rows are added to the dimension reduction matrix, initialized with values ​​randomly generated by a Gaussian distribution with a mean of 0 and a standard deviation of 0.02; 8 columns are added to the dimension increase matrix, with all new columns set to 0. The rank values ​​of layers 14, 17, and 19 are increased from 16 to 24, and the rank value of layer 26 is increased from 8 to 16. The expansion is achieved through in-situ parameter concatenation: the expanded matrix is ​​reassigned and stored, and the original and new values ​​are copied in. Simultaneously, the first and second moment estimates for the corresponding new rows and columns in the optimizer are initialized to 0, ensuring that the update of the new parameters starts from zero momentum. After expansion, 100 texts are extracted from the domain calibration corpus for a forward check. The maximum absolute deviation of each element in the output of the branches before and after expansion does not exceed 0.000001, consistent with the theoretical property of zero perturbation. Let the matrix formed by the newly added rows be denoted as . The matrix formed by the newly added columns is The expanded dimensionality reduction matrix Depend on and The expanded, higher-dimensional matrix is ​​obtained by vertical concatenation. Depend on and This is obtained by horizontal splicing, hence:

[0073]

[0074] because Since it's an all-zero matrix, the second term in the above equation is always a zero matrix. During rank expansion, the output of the LoRA low-rank adaptive branch retains its value before expansion. See [link / reference]. Figure 9The significance of zero perturbation lies in not backtracking to existing training results: continued training starts from the current convergence point and only seeks optimization forward. During continued training, only the dimensionality reduction and expansion matrices of all compensated, restricted layers are set as trainable parameters, and the LoRA low-rank adaptation branches of compensated, qualified layers are frozen. This avoids irrelevant gradient perturbations on already aligned layers and makes continued training very lightweight due to the small number of trainable parameters. Continuing training with the aforementioned training objective, the training objective values ​​at the end of the 7th and 8th rounds are 1.204 and 1.196 respectively, with a difference of 0.008, which is less than the preset convergence threshold, indicating that further training has converged; Figure 7 As shown, the curve segment following the vertical dashed line represents the continued training phase after rank expansion. A second retest was then performed. The residual percentages of layers 14, 17, 19, and 26 decreased to 0.31, 0.28, 0.39, and 0.26 respectively, all below the preset percentage threshold. All Transformer layers were thus determined to be compensated and met the target, and the retest and rank expansion were completed. This embodiment presets two rounds, but only one round was actually executed to achieve the target.

[0075] After retesting and rank expansion, the process proceeds to merging and deployment. For each Transformer layer of the student model, the dimensionality reduction matrix and the dimensionality increase matrix in each LoRA low-rank adaptation branch of the current Transformer layer are multiplied and scaled by a scaling factor to obtain the low-rank increment matrix of the current Transformer layer. On the deployment side, the low-rank increment matrix is ​​superimposed on the original weight matrix.

[0076]

[0077] in, For the first The deployment weight matrix after layer merging, the query projection matrix, and the value projection matrix are each processed according to the above formula. After rank expansion, the total number of branch parameters in this embodiment is: Each of the 32 Transformer layers generates 64 low-rank increment matrices, packaged in half-precision floating-point format, totaling approximately 2.15 gigabytes. This is about 15% of the total 14 gigabyte weights of the teacher model and is distributed to the deployment device as the LoRA low-rank increments carrying the vertical domain capabilities. The deployment device element-wise superimposes each low-rank increment matrix with its corresponding query projection matrix or value projection matrix to obtain the deployment weight matrix, which is then loaded, thus completing the deployment of the large vertical domain model. Since the low-rank increment matrices are already superimposed into the deployment weight matrix, there are no bypass matrix multiplications in the forward computation during the deployment phase, and the matrix operation scale is completely consistent with the base model before merging. Latency measurement is performed on the deployment device, with a 128-word prompt text as input, a fixed length of 128 words generated, a batch size of 8, and timing starting after 10 batches of warm-up. The average latency for generating 128 tokens in a single batch was measured to be 3.42 seconds for both the base model and the deployed vertical domain large model. In contrast, if merging is not performed and branch calculations are retained during inference, the average latency under the same conditions is 3.61 seconds, an increase of approximately 5.6%. This demonstrates the necessity of merging for maintaining inference latency. Latency statistics are based on 200 consecutive batches. Besides the average latency, the 99th percentile latency of the deployed vertical domain large model is 3.87 seconds, while it is 4.15 seconds when branch calculations are retained, showing a more significant difference at the tail end. The accuracy of the merging values ​​was verified separately: the output scores of the forward pass before and after merging were compared on 2000 test texts. The maximum absolute deviation for each element did not exceed 0.000001, falling within the rounding range of half-precision floating-point numbers. When the low-rank incremental matrix is ​​distributed, a check code for the weights of the base model is included. Before the deployed device is overlaid, it checks the check code to ensure that the incremental and base versions match each other.

[0078] The results were validated on 2000 medical domain test texts. These test texts did not overlap with the domain training and calibration corpora. The metric was answer accuracy. The teacher model achieved an accuracy of 86.4%, the unfitted base model 62.3%, and the method in this embodiment reached 85.2%, a 22.9 percentage point improvement over the base model, narrowing the gap with the teacher model to 1.2 percentage points. A further comparison was made regarding temperature values: the accuracy for values ​​1, 2, and 4 were 84.6%, 85.2%, and 84.3%, respectively. Value 1 resulted in soft targets approaching a one-hot distribution with insufficient synonymous preference information; value 4 resulted in an overly flat distribution and diluted supervision signal; value 2 was the most balanced. To examine the contribution of each step, comparisons were made under the condition of a comparable total number of trainable parameters: 84.1% when retesting and rank expansion were omitted; 83.5% when sensitivity rank was retained while supervision weights were unified; 82.9% when the rank values ​​of each layer were unified to 10 while hierarchical supervision was retained; and 82.1% when the conventional approach of unified rank values ​​and unified supervision weights was used. The complete method improved efficiency by 3.1 percentage points compared to the conventional approach, while introducing sensitivity rank separately and introducing hierarchical supervision separately only brought improvements of 1.4 and 0.8 percentage points respectively. The sum of the two, 2.2 percentage points, was still less than 3.1 percentage points, indicating that the gain generated by capacity and supervision falling on the same batch of Transformer layers exceeded the sum of their individual effects, and the hierarchical coupling was effective. The training hyperparameters, upper limit of training epochs, and convergence determination methods for each of the above ablation configurations were consistent with the complete method; the only difference was the single factor being examined. Another baseline for weight difference decomposition is established: the difference between the corresponding weight matrices of the teacher model and the student model base is truncated using the same 16, 8, and 4-rank values ​​for singular value decomposition, and directly superimposed into the base model. Its accuracy is only 79.6%. The reason for this gap is that truncated singular value decomposition minimizes the weight reconstruction error, but optimal approximation of the weight space does not equate to optimal behavior preservation. The direction with a large impact on domain behavior is not necessarily the direction with the largest singular value energy. The low-rank increment matrix of this method is obtained by directly aligning the output distribution and intermediate representation on the domain data and then optimizing through gradient. Under equal-rank budgeting, the domain accuracy is naturally higher. From the perspective of error examples, the errors of the weight difference decomposition baseline are concentrated in diagnosis and treatment suggestion questions that require multi-step reasoning. These behaviors are distributed across multiple layers of collaborative transformation, and matrix-by-matrix independent truncation of the weight space is difficult to maintain cross-layer collaboration. Training based on behavior alignment naturally optimizes end-to-end output. The above comparisons... Figure 10 As shown. Regarding general competence, on 2000 general knowledge test texts, the base model achieved 68.5%, the method in this embodiment achieved 67.8%, and the teacher model achieved 65.4%. This demonstrates that the low-rank incremental form significantly less disrupts general competence than the full-parameter domain fine-tuning. The 0.7 percentage point difference primarily stems from the shift in output distribution towards domain-specific expression styles, rather than the forgetting of general knowledge. The concentration of error examples in expression-style questions confirms this.

[0079] To verify the method's universality across domains, an embodiment for the legal vertical domain is presented. The teacher model is a large-scale legal domain model obtained by fine-tuning all parameters of the legal corpus using the same general base model, with the network size identical to the aforementioned medical embodiment. The domain calibration corpus consists of 1000 legal interpretations and judgment summaries, while the domain training corpus consists of 65000 legal Q&As and document rewrites. The distillation sensitivity obtained from layer-by-layer functional probing ranges from 0.018 to 0.152, with high-sensitivity layers corresponding to layers 10 to 20, shifted two layers forward compared to the medical embodiment. This indicates that the rewrite layers for full parameter fine-tuning differ across domains, and this method categorizes layers based on measured distillation sensitivity, eliminating the need for manual judgment of which layers are important. The rank values ​​and supervisory weights are set in the same way as in the medical embodiment. After training to convergence in the 7th round, three compensated restricted layers were obtained during retesting: layers 13, 16, and 24. After rank expansion and two more rounds of training, all layers passed the second retest. On 2000 test texts in the legal domain, the base model achieved an accuracy of 58.7%, the teacher model 84.1%, and our method reached 82.8%, an improvement of 24.1 percentage points compared to the base model. The distribution volume and inference latency are on the same order of magnitude as the medical implementation. The entire process and parameter settings are completely identical in both domains; only the two domain corpora need to be changed, verifying the domain independence of the method.

[0080] In an optional implementation, the positional difference value is changed to the square root of the average of the squares of the differences between corresponding elements, i.e., measured using the root mean square (RMS) calibrator. In this embodiment, the three-level division results given by the two calibrators are completely consistent, indicating that the grading is robust to the measurement details. The size of the domain calibration corpus can be selected between 500 and 2000 entries. When the measurement is repeated with 500 and 2000 entries, only the Transformer hierarchy order at the boundary between the medium-sensitivity layer and the low-sensitivity layer changes, and the three-level division remains basically unchanged. The sensitivity ranking obtained from the two measurements is highly consistent with the ranking of the 1000-entry calibrator, and the Spearman rank correlation coefficient is above 0.98, indicating that the 1000-entry scale is sufficient. The temperature can be selected between 1 and 4, the learning rate can be selected between 0.00005 and 0.0002, the batch size can be selected between 8 and 32, and the scaling factor can be selected between 1 and 4. When the trainable parameter budget is more abundant, the high-rank, medium-rank, and low-rank values ​​can also be 32, 16, and 8, respectively. The preset percentage threshold can be selected between 0.3 and 0.6, and the extended rank value can be selected between 4 and 16. When the preset percentage threshold is set to 0.45, 0.5, and 0.55, the number of compensation-restricted layers identified are 5, 4, and 3 respectively, with the final accuracy differing by no more than 0.2 percentage points. Setting it to 0.5 strikes a good balance between compensation sufficiency and the overhead of continued training.

[0081] In another optional implementation, to further compress the transmission volume, the dimensionality reduction matrix, dimensionality increase matrix, and scaling factor of each Transformer layer are sent to the deployment device. The deployment device first performs multiplication and scaling to obtain a low-rank increment matrix, and then performs element-wise superposition. In this case, the transmission volume is approximately 11.01 megabytes. The application areas are not limited to healthcare: vertical fields such as law, finance, and industrial operations only need to replace the domain calibration corpus and domain training corpus; the processes and parameter settings for probing, grading, training, retesting, and deployment remain unchanged. The deployment device can be a cloud-based inference server or an edge computing device equipped with an inference accelerator card.

[0082] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for deploying large-scale vertical domain models based on LoRA low-rank adaptation and knowledge distillation, characterized in that, Includes the following steps: Step 1: Measure distillation sensitivity using a layer-by-layer functional probing approach: The teacher model is a large model fine-tuned with all parameters in the domain, and the student model is a base model to be deployed. The Transformer layer positions of the teacher and student models correspond one-to-one. Input the domain-calibrated corpus into the teacher model to perform forward computation, recording the hidden states received by each Transformer layer of the teacher model as layer input representations. Input the layer input representations into the current Transformer layer and the Transformer layers in the same position in the student model respectively, obtaining the output representations of the teacher layer and the same input layer of the student model. Determine the distillation sensitivity of the current Transformer layer based on the difference between the two. Divide all Transformer layers of the student model into high-sensitivity layers, medium-sensitivity layers, and low-sensitivity layers according to the distillation sensitivity from high to low. Step 2: Assign high-rank, medium-rank, and low-rank values ​​to each Transformer layer within the high-sensitivity, medium-sensitivity, and low-sensitivity layers, respectively, with high-rank values ​​being greater than medium-rank values ​​and medium-rank values ​​being greater than low-rank values; For each Transformer layer of the student model, connect the query projection matrix and value projection matrix in parallel to form a LoRA low-rank adaptation branch composed of a cascaded dimensionality reduction matrix and an increased dimensionality matrix, with the dimension of the output of the dimensionality reduction matrix equal to the rank value of the current Transformer layer; Freeze all parameters of the base of the teacher model and the student model, using all dimensionality reduction and increased dimensionality matrices as trainable parameters; Step 3: Input the domain training corpus into the teacher model and the student model at the same time, apply output distribution alignment supervision and intermediate layer representation alignment supervision to the student model, set supervision weights for each Transformer layer, and decrease the supervision weights of high-sensitivity layer, medium-sensitivity layer and low-sensitivity layer in sequence. Thus, the same distillation sensitivity level simultaneously drives the rank value allocation and supervision weight setting. All dimensionality reduction and dimensionality increase matrices are iteratively updated using gradient descent until training converges. After training converges, each LoRA low-rank adaptation branch is merged into a low-rank increment matrix and sent to the deployment device. On the deployment device, each low-rank increment matrix is ​​superimposed into the corresponding query projection matrix or value projection matrix to obtain the deployment weight matrix and load it, thus completing the deployment of the large model in the vertical domain.

2. The method according to claim 1, characterized in that, The teacher model and the student model use the same network architecture, the same number of Transformer layers, the same hidden dimension, and the same word segmentation method. The word segments of the same text by the teacher model and the student model correspond one-to-one. The domain calibration corpus consists of 500 to 2000 domain texts taken from the vertical domain. The layer input representation is the hidden state output by the previous Transformer layer in the teacher model, and the layer input representation of the first Transformer layer is the hidden state output by the word embedding.

3. The method according to claim 1, characterized in that, The teacher layer output representation is output after the current Transformer layer completes self-attention calculation, feedforward calculation, and adds the residuals to the layer input representation. The teacher layer output representation is the hidden state of the current Transformer layer output in the teacher model's own forward calculation. The student same input layer output representation is output after the same Transformer layer in the student model completes self-attention calculation, feedforward calculation, and adds the residuals to the same layer input representation. The output representation of the teacher layer and the output representation of the student layer are obtained by passing the same layer input representation through the same Transformer layer. The difference between the two comes only from the transformation difference of the Transformer layer of the teacher model and the student model at the current position. The interference of the difference propagation from the upstream Transformer layer is isolated. Determining the distillation sensitivity of the current Transformer layer includes: for each text in the domain calibration corpus, at each lexical position, calculating the difference between the corresponding elements of the teacher layer's output representation and the student's output representation at the same input layer, and averaging the absolute values ​​of all differences of corresponding elements to obtain the positional difference value at the current lexical position; averaging the positional difference values ​​of the current Transformer layer at all lexical positions in the current text to obtain the layer-local difference of the current Transformer layer for the current text; and averaging the absolute values ​​of all elements of the teacher layer's output representation at all lexical positions in the current text as the current Transformer layer's distillation sensitivity. The output amplitude of the Transformer layer for the current text is calculated. The local dissimilarity of the layer is divided by the output amplitude to obtain the relative dissimilarity of the current Transformer layer for the current text. The relative dissimilarity is based on the output amplitude. Transformer layers can still be compared even if the magnitudes of the hidden state values ​​differ. The average of the relative dissimilarity of the current Transformer layer across all texts in the domain calibration corpus is calculated to obtain the distillation sensitivity of the current Transformer layer. The distillation sensitivity of the current Transformer layer is used as the distillation sensitivity of Transformer layers in the same position in the student model.

4. The method according to claim 1, characterized in that, The process of dividing the model into high-sensitivity, medium-sensitivity, and low-sensitivity layers involves: sorting all Transformer layers of the student model from high to low distillation sensitivity; assigning Transformer layers at the top of the sort to high-sensitivity layers, those in the middle to medium-sensitivity layers, and those at the bottom to low-sensitivity layers; the difference between any two of the number of Transformer layers in the high-sensitivity, medium-sensitivity, and low-sensitivity layers is at most 1; and using the distillation sensitivity classification as a unified basis for rank allocation and supervision weight setting.

5. The method according to claim 1, characterized in that, The query projection matrix and value projection matrix are the weight matrices used to generate the query vector and value vector, respectively, in the self-attention sublayer of each Transformer layer of the student model. The dimension reduction matrix projects the input vector to a low-dimensional space with a dimension equal to the rank value of the current Transformer layer. The dimension increase matrix maps the vector in the low-dimensional space back to the original output dimension of the query projection matrix or value projection matrix. The output of the LoRA low-rank adaptation branch is scaled by a preset scaling factor and then added element-wise to the original output of the query projection matrix or value projection matrix. The initial value of the dimension reduction matrix is ​​a randomly generated value, and the initial value of the dimension increase matrix is ​​all set to 0. At the beginning of training, the output of the LoRA low-rank adaptation branch is a 0 vector, and the student model gradually adapts from the original behavior of the base model. Merging each LoRA low-rank adaptation branch into a low-rank increment matrix involves multiplying the dimension-reduced matrix and the dimension-increasing matrix in the current LoRA low-rank adaptation branch and scaling them by a scaling factor to obtain the low-rank increment matrix. The forward computation during the deployment phase uses the deployment weight matrix to perform matrix operations of the same scale as the base model, ensuring that the inference latency of the deployed vertical domain large model remains consistent with that of the base model.

6. The method according to claim 1, characterized in that, For each text in the domain training corpus, the output distribution alignment supervision is as follows: the word probability distribution given by the teacher model output layer for each position to be predicted in the current text on the vocabulary is used as the soft objective, and the word probability distribution given by the student model output layer for the same position to be predicted in the current text on the vocabulary is constrained to move closer to the soft objective. The intermediate layer representation alignment supervision is as follows: the hidden state output by the current Transformer layer of the teacher model after completing self-attention calculation, feedforward calculation, and residual addition is taken as the teacher's intermediate representation; the hidden state output by the Transformer layer in the student model after completing self-attention calculation, feedforward calculation, and residual addition is taken as the student's intermediate representation. The supervision weight of the current Transformer layer is used as the strength to constrain the student's intermediate representation to move closer to the teacher's intermediate representation. The larger the supervision weight, the stronger the constraint. The distillation sensitivity is obtained before training through layer-by-layer function probing. Output distribution alignment supervision and intermediate layer representation alignment supervision are applied to the forward computation trajectory of the domain training corpus, enabling the student model to reproduce the behavior of the teacher model on the real reasoning path. The training objective is to align the output distribution with the intermediate layer representation with the weighted average of all Transformer layers according to their respective supervision weights. Training is considered to have converged when the numerical change of the training target between adjacent training rounds is less than the preset convergence threshold.

7. The method according to claim 3, characterized in that, Also includes: After training convergence and before merging, retesting is performed on the domain calibration corpus using a layer-by-layer functional probing approach: the layer input representation of the current Transformer layer is input into the current Transformer layer to obtain the teacher layer output representation; the same layer input representation is input into the Transformer layer in the same position in the student model, and the Transformer layer in the same position with parallel LoRA low-rank adaptation branches completes self-attention calculation, feedforward calculation, and residual addition. The output of the corresponding LoRA low-rank adaptation branch is superimposed in the self-attention calculation to obtain the student retest layer output representation; the residual relative difference of the current Transformer layer is calculated using the teacher layer output representation and the student retest layer output representation according to the calculation methods of positional difference value, layer local difference degree, layer output amplitude, and layer relative difference degree; the residual relative difference degree is divided by the distillation sensitivity of the current Transformer layer to obtain the residual proportion of the current Transformer layer; Transformer layers with a residual proportion lower than a preset proportion threshold are judged as compensation-compliant layers, and Transformer layers with a residual proportion reaching the preset proportion threshold are judged as compensation-limited layers.

8. The method according to claim 7, characterized in that, Also includes: For each compensation-constrained layer, in each LoRA low-rank adaptation branch of the current Transformer layer, add rows to the dimensionality reduction matrix equal to the preset extended rank value. The newly added rows are initialized with randomly generated values. Add columns to the dimensionality increase matrix equal to the extended rank value. All newly added columns are set to 0. The rank value of the current Transformer layer is increased by the extended rank value. This constitutes rank expansion. At the moment of rank expansion, the output of the LoRA low-rank adaptation branch retains the value before rank expansion. Set the dimensionality reduction and increase matrices of all compensation-constrained layers as trainable parameters. Freeze the LoRA low-rank adaptation branches of the compensation-compliant layers. Continue training with output distribution alignment supervision and intermediate layer representation alignment supervision until the training converges again. When all Transformer layers are determined to be compensation-compliant layers, or when the retest and rank expansion have been repeated for a preset number of rounds, end the retest and rank expansion. In this way, the compensation degree of each Transformer layer is verified after training by layer-by-layer functional probing. The rank value is added to the compensation-constrained layer in a targeted manner to correct the deviation between the static measurement before training and the compensation requirement after training.

9. The method according to claim 1, characterized in that, The high-rank value is 16, the medium-rank value is 8, and the low-rank value is 4; the supervision weight of the high-sensitivity layer is 1.0, the supervision weight of the medium-sensitivity layer is 0.5, and the supervision weight of the low-sensitivity layer is 0.

2.

10. The method according to claim 8, characterized in that, The preset percentage threshold is 0.5, the extended rank value is 8, and the preset number of rounds for retesting and rank extension is 2.

Images