A large model lightweight deployment method based on knowledge distillation

Abstract

The application discloses a kind of big model light deployment methods based on knowledge distillation, comprising the following steps: according to training data and target hardware resource, construct teacher model and student model, determine student model structure capacity vector and distillation layer set;Teacher feature is extracted and combined with structure capacity vector to construct the joint cost matrix considering semantic difference and resource cost simultaneously;With teacher feature and structure capacity as edge constraint, with joint cost matrix as transmission cost, establish entropy regular optimal transmission model, solve approximate transmission matrix under the constraint of computing power and storage budget using Greenkhorn algorithm;Student model structure reconstruction scheme is generated using approximate transmission matrix and student model structure is reconstructed, and a light student model that meets resource constraints is obtained based on distillation loss and task loss joint training.The application improves the deployment efficiency of big model under limited hardware.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a lightweight deployment method for large models based on knowledge distillation. Background Technology

[0002] Deep learning large models are widely deployed in natural language processing and multimodal scenarios. The parameter scale and computing power consumption continue to increase. Inference process places strict requirements on GPU memory capacity, memory bandwidth and energy consumption. Engineering practice usually uses model compression and knowledge distillation to deploy student models under fixed hardware resources. Mainstream knowledge distillation methods rely on distance metrics such as soft label cross-entropy and feature mean square error. They adopt a pre-set strategy for the structural mapping between teacher and student models and lack a mechanism for joint modeling of hierarchical semantics and resource consumption. It is difficult to accurately characterize the structural capacity allocation in complex hardware constraints.

[0003] In existing research, some works introduce the optimal transmission distance to describe the alignment relationship between the teacher feature distribution and the student feature distribution. They use the entropy regularized optimal transmission model and combine it with the Sinkhorn algorithm and the Greenkhorn algorithm to solve the transmission matrix. The focus is on training convergence and numerical stability. The transmission cost mainly comes from the feature difference measurement. Resource consumption is often indirectly controlled only by the pruning ratio, the channel compression ratio and a set of hyperparameters. There is a lack of a framework for constructing a joint cost matrix based on the structural capacity vector and hardware resource information. The Bregman regularization term is mostly used for theoretical analysis and simple smoothing, and is not tightly coupled with the row and column scaling update process.

[0004] Existing lightweight deployment processes for large models typically separate network structure search, pruning and compression, and knowledge distillation. In the structure design phase, width, depth, and the number of attention heads are set based on empirical rules. In the distillation phase, the performance of the student model is adjusted through loss weights. The structure capacity vector does not establish a transmission relationship with the teacher feature distribution. The resource budget is not transformed into differentiability constraints and truncation strategies in the optimal transmission solution process. The approximate transmission matrix is ​​only used as an intermediate weight. It does not drive the generation of student model structure reconstruction schemes, nor does it characterize the distribution alignment constraints and structure preservation constraints together with the distillation loss. Under the condition of limited computing power and storage resources, it is difficult to simultaneously achieve semantic fidelity and structural compression strength.

[0005] Therefore, how to provide a lightweight deployment method for large models based on knowledge distillation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a lightweight deployment method for large models based on knowledge distillation. This invention introduces an entropy-regularized optimal transfer model with Bregman regularization and combines it with the Greenkhorn algorithm and a resource adaptive truncation strategy. It uniformly constructs a joint cost matrix between the teacher feature distribution and the student model structure capacity and drives the reconstruction of the student model structure. Thus, under given computing power and storage budget conditions, it achieves integrated large model distillation training and lightweight deployment, with the advantages of high semantic fidelity, fine resource utilization, and high degree of automation in the deployment process.

[0007] A lightweight deployment method for a large model based on knowledge distillation according to an embodiment of the present invention includes the following steps:

[0008] Acquire training data and target hardware resource information, construct teacher and student models, and determine the student model structure capacity vector and distillation layer set based on the target hardware resource information;

[0009] The teacher model is run on the training data, teacher features are extracted from the distillation layer set, and a joint cost matrix that simultaneously represents semantic differences and resource costs is constructed based on the teacher features and the student model structure capacity vector.

[0010] An entropy-regularized optimal transmission model with Bregman regularization terms is established, using teacher characteristics and student model structure capacity as marginal constraints and the joint cost matrix as the transmission cost.

[0011] The Greenkhorn algorithm is used to iteratively solve the entropy-regularized optimal transmission model with Bregman regularization terms to obtain an approximate transmission matrix;

[0012] During the Greenkhorn iteration process, the number of rows and columns and the number of iteration rounds can be updated according to the target hardware computing power budget and storage budget constraints, and the Bregman regularity strength is adjusted to form a resource-adaptive truncated Greenkhorn solution process;

[0013] The student model structure reconstruction scheme is generated based on the approximate transfer matrix obtained from the resource adaptive truncation Greenkhorn solution process, and the student model structure is reconstructed based on the student model structure reconstruction scheme.

[0014] Based on the approximate transfer matrix and the student model structure reconstruction scheme, a distillation loss is constructed and combined with the task loss to train the reconstructed student model, resulting in a lightweight student model that meets the target hardware resource constraints.

[0015] Optionally, the Bregman regularization term is: based on the deviation measure between each transmission quantity in the transmission matrix and the corresponding transmission quantity in the reference transmission matrix, a deviation measure function is constructed according to a given generating function, and the deviation values ​​between all transmission quantities and the reference transmission quantity are calculated according to the deviation measure function. The regularization term is formed by multiplying all deviation values ​​by the regularization strength parameter and then adding them together. This regularization term is used to limit the deviation magnitude of the transmission matrix relative to the reference transmission matrix.

[0016] Optionally, the teacher model and the student model are two neural network models configured in the same input space and task output space. The teacher model is a large-scale neural network model with fixed parameters and high expressive power, while the student model is a lightweight neural network model with the number of network layers, channels, and attention heads set according to the target hardware resource information. Both are used to construct the joint cost matrix, establish the optimal transmission model, and train the lightweight student model according to the layer index output features registered in the distillation layer set.

[0017] Optionally, the construction of the teacher model and student model, and the determination of the student model's structural capacity vector and distillation layer set specifically include:

[0018] Obtain the original training dataset, which contains sample inputs and supervision labels. Perform format unification, outlier removal, and numerical normalization on the original training dataset to obtain the training data.

[0019] Collect target hardware resource information, which includes at least the processing unit type, computing power budget, video memory capacity, memory capacity, storage bandwidth and power consumption limit, and encode it into a target hardware resource description vector;

[0020] Configure the teacher model network structure, load the teacher model parameters, and register the output layer index and intermediate layer index of the teacher model;

[0021] Based on the target hardware resource description vector and the teacher model network structure, the initial network structure of the student model is set, the number of channels in each layer, the network width and the number of attention heads are determined, and the set of student model structure parameters is obtained.

[0022] The structural capacity of each layer is calculated based on the set of structural parameters of the student model, and the structural capacities of each layer are arranged in a fixed order to form a student model structural capacity vector.

[0023] Based on the correspondence between the layers of the teacher model and the layers of the student model, as well as the training task requirements, select the teacher model layer and student model layer that participate in knowledge distillation, register the corresponding index, and form a set of distillation layers.

[0024] The teacher model refers to a large-scale neural network model that has been pre-trained on training data and is used as a reference model in knowledge distillation. The teacher model receives sample input from the training data, performs forward computation according to a fixed network structure and parameters, and outputs teacher features at the corresponding layer of the distillation layer set. The network structure of the teacher model includes an input subnetwork, a feature extraction subnetwork, and an output subnetwork. The feature extraction subnetwork may include a combination of convolutional units, fully connected units, attention units, and normalization units. The parameters of the teacher model remain unchanged during the execution of the large model lightweight deployment method and are called through the layer index registered in the distillation layer set when constructing the teacher feature distribution and joint cost matrix.

[0025] The student model refers to a lightweight neural network model configured under the constraints of target hardware resource information, used to deploy the target model. The student model and the teacher model have corresponding input spaces and task output spaces. The initial network structure of the student model is determined based on the network structure of the teacher model and the target hardware resource description vector, including an input subnetwork, a feature extraction subnetwork, and an output subnetwork. The number of layers, channels, and attention heads in the feature extraction subnetwork are given by the student model structure capacity vector. At the beginning of training, the student model initializes its parameters according to the student model structure capacity vector. During the knowledge distillation process, it receives sample inputs from the training data and outputs student features in the corresponding layers of the distillation layer set. The parameters of the student model are updated by combining the distillation loss and the task loss. At the end of training, it is used as a lightweight student model that meets the constraints of the target hardware resources for deployment.

[0026] Optionally, the construction of the joint cost matrix specifically includes:

[0027] The teacher model is run on the training data in the order of samples. Teacher features are extracted from the distillation layer set by layer index. The teacher features of each sample in the distillation layer set are concatenated into a teacher feature vector in a fixed order. All teacher feature vectors are collected to form a teacher feature vector set.

[0028] Perform dimension alignment and numerical normalization on the teacher feature vectors in the teacher feature vector set to obtain a normalized teacher feature vector set;

[0029] Based on the order of each structural capacity element in the student model structural capacity vector, register a structural capacity index for each structural capacity element, combine the structural capacity index and structural capacity element to form a structural capacity unit, and collect all structural capacity units to form a structural capacity unit set.

[0030] For each teacher feature vector in the teacher feature vector set and each structural capacity unit in the structural capacity unit set, calculate the semantic difference value between them, and record the semantic difference value with the teacher feature vector index and the structural capacity index as the key to form a semantic difference record set.

[0031] For each structural capacity unit in the structural capacity unit set, the resource cost value is calculated based on the value of the structural capacity element and the target hardware resource information, and the resource cost value is recorded with the structural capacity index as the key to form a resource cost record set.

[0032] The semantic difference values ​​are read from the semantic difference record set according to the teacher feature vector index and the structural capacity index, and the resource cost values ​​are read from the resource cost record set. The semantic difference values ​​and resource cost values ​​are combined according to the preset weight coefficients to obtain the joint cost values. The joint cost values ​​are arranged into a matrix form according to the teacher feature vector index and the structural capacity index and registered as the joint cost matrix.

[0033] Optionally, the establishment of the entropy-regularized optimal transmission model with Bregman regularization specifically includes:

[0034] A teacher feature index set is established based on the index of each teacher feature vector in the teacher feature vector set. Non-negative weights are calculated for each teacher feature vector in the teacher feature vector set. Normalization is performed on all non-negative weights to obtain the teacher feature distribution supported by the teacher feature index set.

[0035] A structural capacity index set is established based on the index of each structural capacity element in the student model structural capacity vector. Non-negative weights are calculated for each structural capacity element in the student model structural capacity vector. Normalization is performed on all non-negative weights to obtain a structural capacity distribution supported by the structural capacity index set.

[0036] Each element in the joint cost matrix is ​​associated with a teacher feature index and a structural capacity index, and each element in the transfer matrix is ​​also associated with a teacher feature index and a structural capacity index. Entropy regularization strength parameters, Bregman regularization strength parameters, reference transfer matrix elements, and a Bregman divergence function constructed based on the generating function are introduced. Using the transfer matrix as the decision variable, an optimal transfer objective function with entropy regularization and Bregman regularization is constructed.

[0037] ;

[0038] in, The objective function value corresponding to the transfer matrix. This represents the amount of data transmitted at the position with row index i and column index j in the transmission matrix. Let be the joint cost value at row index i and column index j in the joint cost matrix, where N is the number of teacher feature indices and M is the number of structure capacity indices. The entropy canonical strength parameter, Canonical strength parameter, For generating functions The constructed Bregman divergence function, This refers to the reference transmission amount at the position with row index i and column index j in the reference transmission matrix. For a strictly convex function defined in the non-negative real number field, this objective function is constructed based on the discrete optimal transmission model: First, the teacher feature distribution and the student model structure capacity distribution are regarded as two discrete probability distributions. Under the premise of satisfying row edge constraints and column edge constraints and non-negative transmission volume, the weighted summation of the elements of the transmission matrix and the joint cost matrix is ​​used as the basic objective, thus obtaining the expected transmission cost term that reflects semantic differences and resource costs. On this basis, an entropy regularization term is introduced, and the transmission matrix is ​​regarded as a set of probability variables with negative entropy penalties. The smoothness and convexity are controlled by the entropy regularization strength parameter, so that the problem is adapted to the row and column scaling form of Greenkhorn iteration. Finally, Bregman divergence based on a given generating function is introduced, which measures the deviation of the current transmission matrix at each element with the reference transmission matrix as the center. The deviation magnitude is limited by the Bregman regularization strength parameter and historical solution information is embedded, thus obtaining an entropy regularized optimal transmission objective function with Bregman regularization, which consists of three parts: the transmission cost term, the entropy regularization term, and the Bregman regularization term.

[0039] Row edge constraints and column edge constraints are set. The row edge constraint requires that the sum of all elements in each row of the transmission matrix equals the probability value of the corresponding teacher feature index position in the teacher feature distribution. The column edge constraint requires that the sum of all elements in each column of the transmission matrix equals the probability value of the corresponding structural capacity index position in the structural capacity distribution. Each element of the transmission matrix is ​​also restricted to be a non-negative value. The optimal transmission objective function, row edge constraints, column edge constraints and non-negative value constraints are collectively registered as an entropy-regularized optimal transmission model with Bregman regularization.

[0040] Optionally, the process of solving the approximate transfer matrix specifically includes:

[0041] For the entropy regularized optimal transport model with Bregman regularization, initialize the row scaling vector and column scaling vector. In the row scaling vector, register the corresponding teacher feature index for each element, and in the column scaling vector, register the corresponding structure capacity index for each element. Set the iteration round counter, row constraint residual threshold and column constraint residual threshold.

[0042] In each round of Greenkhorn iteration, the transfer matrix is ​​calculated based on the current row scaling vector, the current column scaling vector, and the joint cost matrix. The row edge sum is obtained by summing all elements in each row of the transfer matrix, and the column edge sum is obtained by summing all elements in each column of the transfer matrix.

[0043] Based on the teacher characteristic distribution and row margins, calculate the row constraint residual sequence; based on the structural capacity distribution and column margins, calculate the column constraint residual sequence; and determine the index of the element with the largest row contribution in the row constraint residual sequence and the index of the element with the largest column contribution in the column constraint residual sequence according to the absolute value of the residuals, and register them as candidate row indices and candidate column indices.

[0044] Compare the row contribution of the candidate row index with the column contribution of the candidate column index. If the row contribution is greater than the column contribution, register the candidate row index as the target index. If the row contribution is less than the column contribution, register the candidate column index as the target index. If the row contribution and column contribution are the same, select the target index according to the preset rules.

[0045] When the target index corresponds to the teacher feature index, the value of the element at the target index position in the row scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding row in the transmission matrix is ​​close to the probability value of the same teacher feature index position in the teacher feature distribution. When the target index corresponds to the structure capacity index, the value of the element at the target index position in the column scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding column in the transmission matrix is ​​close to the probability value of the same structure capacity index position in the structure capacity distribution.

[0046] After completing a round of row scaling vector update or column scaling vector update, the absolute value of each row constraint residual and the absolute value of each column constraint residual are determined based on the new row constraint residual sequence and the new column constraint residual sequence. If the absolute values ​​of all row constraint residuals and all column constraint residuals do not exceed the corresponding threshold, the current transfer matrix is ​​registered as an approximate transfer matrix and the Greenkhorn iteration is terminated. If the absolute value of either row constraint residual or column constraint residual exceeds the corresponding threshold, the iteration round counter is incremented and the current row scaling vector and the current column scaling vector are retained to enter the next round of Greenkhorn iteration.

[0047] Optionally, the process of adjusting the Bregman regularity strength to form a resource-adaptive truncation Greenkhorn solution specifically includes:

[0048] Based on the target hardware resource information, set the computing power budget index and storage budget index for Greenkhorn iteration. The computing power budget index is represented by the number of available floating-point operations, and the storage budget index is represented by the amount of available memory usage, and register them as the resource budget parameters for Greenkhorn iteration.

[0049] During the Greenkhorn iteration process, for each iteration, the number of floating-point operations and the amount of memory usage generated by the row scaling vector update and column scaling vector update are recorded. The number of floating-point operations is accumulated as the cumulative computing power consumption, and the amount of memory usage is accumulated as the cumulative storage consumption. The current iteration number is also recorded.

[0050] The computing power utilization ratio is calculated based on the computing power budget index and the cumulative computing power consumption. The storage utilization ratio is calculated based on the storage budget index and the cumulative storage consumption. The larger value between the computing power utilization ratio and the storage utilization ratio is selected and registered as the truncation ratio. A resource adaptive truncation parameter set is generated based on the truncation ratio. The resource adaptive truncation parameter set includes at least the upper limit of the number of row indexes allowed to be updated in each round, the upper limit of the number of column indexes allowed to be updated in each round, and the upper limit of the number of Greenkhorn iteration rounds.

[0051] At the start of the Greenkhorn iteration, the resource adaptive truncation parameter set is read, and constraints are imposed on the number of times the row candidate index and column candidate index are selected, so that the number of times the row scaling vector is updated in the current round does not exceed the upper limit of the number of row indexes, and the number of times the column scaling vector is updated in the current round does not exceed the upper limit of the number of column indexes. After each round, the resource adaptive truncation parameter set is updated according to the truncation ratio and the current iteration round number.

[0052] When any one of the following three conditions is met: the cumulative computing power consumption reaches the total number of floating-point operations limited by the computing power budget target, the cumulative storage consumption reaches the total memory usage limited by the storage budget target, or the number of iteration rounds reaches the upper limit of the Greenkhorn iteration rounds, a resource truncation flag is generated, the Greenkhorn iteration is terminated, and the transfer matrix at the time of termination is registered as the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process.

[0053] Optionally, the reconstruction of the student model structure specifically includes:

[0054] Read the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process, perform column aggregation on the approximate transfer matrix according to the structural capacity index set, and obtain the structural capacity weight vector corresponding one-to-one with the structural capacity index.

[0055] Based on the structural capacity weight vector and the student model structural capacity vector, a student model structural reconstruction scheme is generated in the student model structural capacity vector index space. In the student model structural reconstruction scheme, the layer index, structural adjustment type and target structural parameters are registered for each structural capacity element.

[0056] The student model network structure is adjusted according to the student model structure reconstruction scheme. Based on the structure adjustment type, the number of channels, width parameters and connection relationships of the corresponding layers are retained, reduced and deleted to obtain the reconstructed student model.

[0057] Optionally, the generation of the lightweight student model that satisfies the target hardware resource constraints specifically includes:

[0058] In the structurally reconstructed student model, student features are extracted according to the distillation layer set. In the teacher model, teacher features are extracted according to the distillation layer set. Based on the transmission relationship registered in the approximate transmission matrix, a weighted matching is performed on the teacher features and student features. The distribution-aligned distillation loss is calculated. The calculation of the distribution-aligned distillation loss refers to the weighted summation of the difference measure between each pair of matched teacher features and student features according to the corresponding transmission weight, based on the transmission weight between the teacher feature index and the student feature index given by the approximate transmission matrix. The summation is then normalized on the sample dimension and the distillation layer set. The resulting scalar value is registered as the distribution-aligned distillation loss.

[0059] Based on the changes in layer index, channel number, and width parameter recorded in the student model structure reconstruction scheme, the structural change amount is calculated for the layers participating in structural adjustment in the student model structure reconstruction, and a structure preservation distillation loss is constructed. The construction of the structure preservation distillation loss refers to calculating the structural change amount for the layers participating in structural adjustment in the student model structure reconstruction according to a preset penalty function based on the changes in layer index, channel number, and width parameter recorded in the student model structure reconstruction scheme. All structural change amounts are aggregated and normalized in the layer dimension and sample dimension to obtain a scalar loss value used to constrain the degree of deviation of the student model structure reconstruction from the original structure.

[0060] The distribution alignment distillation loss, structure preservation distillation loss, and task loss are combined according to preset weight coefficients to form the total loss. The total loss is used to train and update the structure reconstruction student model. When the training termination condition is met, a lightweight student model that meets the target hardware resource constraints is output.

[0061] The beneficial effects of this invention are:

[0062] This invention introduces a joint cost matrix between teacher features and student model structural capacity, simultaneously encoding semantic difference values ​​and resource cost values ​​within the joint cost matrix. Compared to traditional knowledge distillation methods that rely solely on feature distance, this invention achieves integrated modeling of the semantic space and structural capacity space. By combining an entropy-regularized optimal transfer model with Bregman regularization and the Greenkhorn algorithm, the approximate transfer matrix is ​​solved under row edge constraints, column edge constraints, and non-negativity constraints. This allows the distribution of teacher features and the distribution of student model structural capacity to be directly linked through optimal transfer relationships. Consequently, the matching strength and importance level of different structural capacity elements are automatically provided during the distillation stage, reducing the reliance on manually setting layer correspondences and loss weights.

[0063] This invention further introduces computational and storage budget metrics into the Greenkhorn iteration process, constructing a resource-adaptive truncation Greenkhorn solution process. The truncation ratio is calculated based on resource utilization, and this limits the upper limit of the number of row and column indices that can be updated in each round, while also constraining the number of iteration rounds. When the accumulated computational or storage consumption reaches the budget, a resource truncation marker is generated, and the current transfer matrix is ​​registered as an approximate transfer matrix. Based on this approximate transfer matrix, this invention generates a student model structure reconstruction scheme in the student model structure capacity vector index space, performing retention, reduction, and deletion operations on the number of channels, network width, and connection relationships. This ensures that the reconstructed student model meets the computational and storage budgets given by the target hardware resource information during the construction phase, reducing the risk of memory overflow and inference latency exceeding limits during the distillation training phase.

[0064] During the training phase, this invention constructs a distributed aligned distillation loss based on an approximate transfer matrix and constructs a structure-preserving distillation loss by combining a student model structure reconstruction scheme. The two distillation losses and the task loss are combined with preset weight coefficients to form the total loss. The structure-reconstructed student model is then trained and updated, so that the output lightweight student model maintains high semantic expressive power and task accuracy while meeting resource constraints. Through a continuous process of "joint cost matrix modeling - Bregman regularized optimal transfer solution - resource adaptive truncation - structure reconstruction - distillation training", this invention completes the large model compression and deployment strategy optimization within the same framework, which facilitates rapid migration and unified management of the lightweight deployment process of large models on different hardware platforms. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0066] Figure 1This is a flowchart of a lightweight deployment method for large models based on knowledge distillation proposed in this invention;

[0067] Figure 2 This is a schematic diagram of the structure of a Bregman regularized entropy-regularized optimal transport model with entropy regularization, which is proposed in this invention for a lightweight deployment method of large models based on knowledge distillation.

[0068] Figure 3 This is a schematic diagram of the resource adaptive truncation Greenkhorn solution process for a large model lightweight deployment method based on knowledge distillation proposed in this invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0070] refer to Figure 1-3 A lightweight deployment method for large models based on knowledge distillation includes the following steps:

[0071] Acquire training data and target hardware resource information, construct teacher and student models, and determine the student model structure capacity vector and distillation layer set based on the target hardware resource information;

[0072] The teacher model is run on the training data, teacher features are extracted from the distillation layer set, and a joint cost matrix that simultaneously represents semantic differences and resource costs is constructed based on the teacher features and the student model structure capacity vector.

[0073] An entropy-regularized optimal transmission model with Bregman regularization terms is established, using teacher characteristics and student model structure capacity as marginal constraints and the joint cost matrix as the transmission cost.

[0074] The Greenkhorn algorithm is used to iteratively solve the entropy-regularized optimal transmission model with Bregman regularization terms to obtain an approximate transmission matrix;

[0075] During the Greenkhorn iteration process, the number of rows and columns and the number of iteration rounds can be updated according to the target hardware computing power budget and storage budget constraints, and the Bregman regularity strength is adjusted to form a resource-adaptive truncated Greenkhorn solution process;

[0076] The student model structure reconstruction scheme is generated based on the approximate transfer matrix obtained from the resource adaptive truncation Greenkhorn solution process, and the student model structure is reconstructed based on the student model structure reconstruction scheme.

[0077] Based on the approximate transfer matrix and the student model structure reconstruction scheme, a distillation loss is constructed and combined with the task loss to train the reconstructed student model, resulting in a lightweight student model that meets the target hardware resource constraints.

[0078] In this embodiment, the Bregman regularization term is: based on the deviation measure between each transmission quantity in the transmission matrix and the corresponding transmission quantity in the reference transmission matrix, a deviation measure function is constructed according to a given generating function, and the deviation values ​​between all transmission quantities and the reference transmission quantity are calculated according to the deviation measure function. The regularization term is formed by multiplying all deviation values ​​by the regularization strength parameter and then adding them together. This regularization term is used to limit the deviation magnitude of the transmission matrix relative to the reference transmission matrix.

[0079] In this embodiment, the teacher model and the student model are two neural network models configured in the same input space and task output space. The teacher model is a large-scale neural network model with fixed parameters and high expressive power, while the student model is a lightweight neural network model with the number of network layers, channels, and attention heads set according to the target hardware resource information. Both are used to construct the joint cost matrix, establish the optimal transmission model, and train the lightweight student model according to the layer index output features registered in the distillation layer set.

[0080] In this embodiment, the construction of the teacher model and student model, and the determination of the student model's structural capacity vector and distillation layer set specifically include:

[0081] Obtain the original training dataset, which contains sample inputs and supervision labels. Perform format unification, outlier removal, and numerical normalization on the original training dataset to obtain the training data.

[0082] Collect target hardware resource information, which includes at least the processing unit type, computing power budget, video memory capacity, memory capacity, storage bandwidth and power consumption limit, and encode it into a target hardware resource description vector;

[0083] Configure the teacher model network structure, load the teacher model parameters, and register the output layer index and intermediate layer index of the teacher model;

[0084] Based on the target hardware resource description vector and the teacher model network structure, the initial network structure of the student model is set, the number of channels in each layer, the network width and the number of attention heads are determined, and the set of student model structure parameters is obtained.

[0085] The structural capacity of each layer is calculated based on the set of structural parameters of the student model, and the structural capacities of each layer are arranged in a fixed order to form a student model structural capacity vector.

[0086] Based on the correspondence between the layers of the teacher model and the layers of the student model, as well as the training task requirements, select the teacher model layer and student model layer that participate in knowledge distillation, register the corresponding index, and form a set of distillation layers.

[0087] The teacher model refers to a large-scale neural network model that has been pre-trained on training data and is used as a reference model in knowledge distillation. The teacher model receives sample input from the training data, performs forward computation according to a fixed network structure and parameters, and outputs teacher features at the corresponding layer of the distillation layer set. The network structure of the teacher model includes an input subnetwork, a feature extraction subnetwork, and an output subnetwork. The feature extraction subnetwork may include a combination of convolutional units, fully connected units, attention units, and normalization units. The parameters of the teacher model remain unchanged during the execution of the large model lightweight deployment method and are called through the layer index registered in the distillation layer set when constructing the teacher feature distribution and joint cost matrix.

[0088] The student model refers to a lightweight neural network model configured under the constraints of target hardware resource information, used to deploy the target model. The student model and the teacher model have corresponding input spaces and task output spaces. The initial network structure of the student model is determined based on the network structure of the teacher model and the target hardware resource description vector, including an input subnetwork, a feature extraction subnetwork, and an output subnetwork. The number of layers, channels, and attention heads in the feature extraction subnetwork are given by the student model structure capacity vector. At the beginning of training, the student model initializes its parameters according to the student model structure capacity vector. During the knowledge distillation process, it receives sample inputs from the training data and outputs student features in the corresponding layers of the distillation layer set. The parameters of the student model are updated by combining the distillation loss and the task loss. At the end of training, it is used as a lightweight student model that meets the constraints of the target hardware resources for deployment.

[0089] In this embodiment, the construction of the joint cost matrix specifically includes:

[0090] The teacher model is run on the training data in the order of samples. Teacher features are extracted from the distillation layer set by layer index. The teacher features of each sample in the distillation layer set are concatenated into a teacher feature vector in a fixed order. All teacher feature vectors are collected to form a teacher feature vector set.

[0091] Perform dimension alignment and numerical normalization on the teacher feature vectors in the teacher feature vector set to obtain a normalized teacher feature vector set;

[0092] Based on the order of each structural capacity element in the student model structural capacity vector, register a structural capacity index for each structural capacity element, combine the structural capacity index and structural capacity element to form a structural capacity unit, and collect all structural capacity units to form a structural capacity unit set.

[0093] For each teacher feature vector in the teacher feature vector set and each structural capacity unit in the structural capacity unit set, calculate the semantic difference value between them, and record the semantic difference value with the teacher feature vector index and the structural capacity index as the key to form a semantic difference record set.

[0094] For each structural capacity unit in the structural capacity unit set, the resource cost value is calculated based on the value of the structural capacity element and the target hardware resource information, and the resource cost value is recorded with the structural capacity index as the key to form a resource cost record set.

[0095] The semantic difference values ​​are read from the semantic difference record set according to the teacher feature vector index and the structural capacity index, and the resource cost values ​​are read from the resource cost record set. The semantic difference values ​​and resource cost values ​​are combined according to the preset weight coefficients to obtain the joint cost values. The joint cost values ​​are arranged into a matrix form according to the teacher feature vector index and the structural capacity index and registered as the joint cost matrix.

[0096] In this embodiment, the establishment of the entropy-regularized optimal transmission model with Bregman regularization specifically includes:

[0097] A teacher feature index set is established based on the index of each teacher feature vector in the teacher feature vector set. Non-negative weights are calculated for each teacher feature vector in the teacher feature vector set. Normalization is performed on all non-negative weights to obtain the teacher feature distribution supported by the teacher feature index set.

[0098] A structural capacity index set is established based on the index of each structural capacity element in the student model structural capacity vector. Non-negative weights are calculated for each structural capacity element in the student model structural capacity vector. Normalization is performed on all non-negative weights to obtain a structural capacity distribution supported by the structural capacity index set.

[0099] Each element in the joint cost matrix is ​​associated with a teacher feature index and a structural capacity index, and each element in the transfer matrix is ​​also associated with a teacher feature index and a structural capacity index. Entropy regularization strength parameters, Bregman regularization strength parameters, reference transfer matrix elements, and a Bregman divergence function constructed based on the generating function are introduced. Using the transfer matrix as the decision variable, an optimal transfer objective function with entropy regularization and Bregman regularization is constructed.

[0100] ;

[0101] in, The objective function value corresponding to the transfer matrix. This represents the amount of data transmitted at the position with row index i and column index j in the transmission matrix. Let be the joint cost value at row index i and column index j in the joint cost matrix, where N is the number of teacher feature indices and M is the number of structure capacity indices. The entropy canonical strength parameter, Canonical strength parameter, For generating functions The constructed Bregman divergence function, This refers to the reference transmission amount at the position with row index i and column index j in the reference transmission matrix. For a strictly convex function defined in the non-negative real number field, this objective function is constructed based on the discrete optimal transmission model: First, the teacher feature distribution and the student model structure capacity distribution are regarded as two discrete probability distributions. Under the premise of satisfying row edge constraints and column edge constraints and non-negative transmission volume, the weighted summation of the elements of the transmission matrix and the joint cost matrix is ​​used as the basic objective, thus obtaining the expected transmission cost term that reflects semantic differences and resource costs. On this basis, an entropy regularization term is introduced, and the transmission matrix is ​​regarded as a set of probability variables with negative entropy penalties. The smoothness and convexity are controlled by the entropy regularization strength parameter, so that the problem is adapted to the row and column scaling form of Greenkhorn iteration. Finally, Bregman divergence based on a given generating function is introduced, which measures the deviation of the current transmission matrix at each element with the reference transmission matrix as the center. The deviation magnitude is limited by the Bregman regularization strength parameter and historical solution information is embedded, thus obtaining an entropy regularized optimal transmission objective function with Bregman regularization, which consists of three parts: the transmission cost term, the entropy regularization term, and the Bregman regularization term.

[0102] Row edge constraints and column edge constraints are set. The row edge constraint requires that the sum of all elements in each row of the transmission matrix equals the probability value of the corresponding teacher feature index position in the teacher feature distribution. The column edge constraint requires that the sum of all elements in each column of the transmission matrix equals the probability value of the corresponding structural capacity index position in the structural capacity distribution. Each element of the transmission matrix is ​​also restricted to be a non-negative value. The optimal transmission objective function, row edge constraints, column edge constraints and non-negative value constraints are collectively registered as an entropy-regularized optimal transmission model with Bregman regularization.

[0103] This invention enhances the stability and controllability of the transmission matrix by simultaneously introducing entropy regularization and Bregman regularization in optimal transmission modeling, making the comprehensive alignment process of semantic differences and resource costs more robust.

[0104] In this embodiment, the process of solving the approximate transfer matrix specifically includes:

[0105] For the entropy regularized optimal transport model with Bregman regularization, initialize the row scaling vector and column scaling vector. In the row scaling vector, register the corresponding teacher feature index for each element, and in the column scaling vector, register the corresponding structure capacity index for each element. Set the iteration round counter, row constraint residual threshold and column constraint residual threshold.

[0106] In each round of Greenkhorn iteration, the transfer matrix is ​​calculated based on the current row scaling vector, the current column scaling vector, and the joint cost matrix. The row edge sum is obtained by summing all elements in each row of the transfer matrix, and the column edge sum is obtained by summing all elements in each column of the transfer matrix.

[0107] Based on the teacher characteristic distribution and row margins, calculate the row constraint residual sequence; based on the structural capacity distribution and column margins, calculate the column constraint residual sequence; and determine the index of the element with the largest row contribution in the row constraint residual sequence and the index of the element with the largest column contribution in the column constraint residual sequence according to the absolute value of the residuals, and register them as candidate row indices and candidate column indices.

[0108] The row contribution of the candidate row index is compared with the column contribution of the candidate column index. When the row contribution is greater than the column contribution, the candidate row index is registered as the target index. When the row contribution is less than the column contribution, the candidate column index is registered as the target index. When the row contribution and column contribution are the same, the target index is selected according to a preset rule. The preset rule is: when the row contribution and column contribution are the same, the candidate row index is selected as the target index in a fixed order.

[0109] When the target index corresponds to the teacher feature index, the value of the element at the target index position in the row scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding row in the transmission matrix is ​​close to the probability value of the same teacher feature index position in the teacher feature distribution. When the target index corresponds to the structure capacity index, the value of the element at the target index position in the column scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding column in the transmission matrix is ​​close to the probability value of the same structure capacity index position in the structure capacity distribution.

[0110] After completing a round of row scaling vector update or column scaling vector update, the absolute value of each row constraint residual and the absolute value of each column constraint residual are determined based on the new row constraint residual sequence and the new column constraint residual sequence. If the absolute values ​​of all row constraint residuals and all column constraint residuals do not exceed the corresponding threshold, the current transfer matrix is ​​registered as an approximate transfer matrix and the Greenkhorn iteration is terminated. If the absolute value of either row constraint residual or column constraint residual exceeds the corresponding threshold, the iteration round counter is incremented and the current row scaling vector and the current column scaling vector are retained to enter the next round of Greenkhorn iteration.

[0111] After establishing the entropy-regularized optimal transport model with Bregman regularization, this invention initializes row and column scaling vectors. In each Greenkhorn iteration, it calculates the transport matrix based on the joint cost matrix, calculates the row and column edge sums, forms row-constrained residual sequences and column-constrained residual sequences, selects candidate row and column indices based on the absolute value of the residuals, compares row and column contributions, and determines the target index. For the teacher feature index or structural capacity index corresponding to the target index, the row or column scaling vector is updated to ensure that the transport matrix corresponds to the correct row or column scaling vector. The elements and probability values ​​in the teacher characteristic distribution and structural capacity distribution are gradually approximated. In multiple iterations, all row constraint residuals and column constraint residuals are continuously reduced. When the absolute value of all residuals does not exceed the preset threshold, the current transfer matrix is ​​registered as an approximate transfer matrix and the iteration is terminated. Thus, under the premise of satisfying row edge constraints, column edge constraints and non-negativity constraints, the optimal transfer objective function constructed based on entropy regularization and Bregman regularization is specifically solved into a usable discrete transfer scheme. This provides basic data for the subsequent generation of student model structure reconstruction scheme based on approximate transfer matrix and the construction of distribution alignment distillation loss.

[0112] In this embodiment, the process of adjusting the Bregman regularity strength to form a resource-adaptive truncation Greenkhorn solution specifically includes:

[0113] Based on the target hardware resource information, set the computing power budget index and storage budget index for Greenkhorn iteration. The computing power budget index is represented by the number of available floating-point operations, and the storage budget index is represented by the amount of available memory usage, and register them as the resource budget parameters for Greenkhorn iteration.

[0114] During the Greenkhorn iteration process, for each iteration, the number of floating-point operations and the amount of memory usage generated by the row scaling vector update and column scaling vector update are recorded. The number of floating-point operations is accumulated as the cumulative computing power consumption, and the amount of memory usage is accumulated as the cumulative storage consumption. The current iteration number is also recorded.

[0115] The computing power utilization ratio is calculated based on the computing power budget index and the cumulative computing power consumption. The storage utilization ratio is calculated based on the storage budget index and the cumulative storage consumption. The larger value between the computing power utilization ratio and the storage utilization ratio is selected and registered as the truncation ratio. A resource adaptive truncation parameter set is generated based on the truncation ratio. The resource adaptive truncation parameter set includes at least the upper limit of the number of row indexes allowed to be updated in each round, the upper limit of the number of column indexes allowed to be updated in each round, and the upper limit of the number of Greenkhorn iteration rounds.

[0116] At the start of the Greenkhorn iteration, the resource adaptive truncation parameter set is read, and constraints are imposed on the number of times the row candidate index and column candidate index are selected, so that the number of times the row scaling vector is updated in the current round does not exceed the upper limit of the number of row indexes, and the number of times the column scaling vector is updated in the current round does not exceed the upper limit of the number of column indexes. After each round, the resource adaptive truncation parameter set is updated according to the truncation ratio and the current iteration round number.

[0117] When any one of the following three conditions is met: the cumulative computing power consumption reaches the total number of floating-point operations limited by the computing power budget target, the cumulative storage consumption reaches the total memory usage limited by the storage budget target, or the number of iteration rounds reaches the upper limit of the Greenkhorn iteration rounds, a resource truncation flag is generated, the Greenkhorn iteration is terminated, and the transfer matrix at the time of termination is registered as the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process.

[0118] In this embodiment, the reconstruction of the student model structure specifically includes:

[0119] Read the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process, perform column aggregation on the approximate transfer matrix according to the structural capacity index set, and obtain the structural capacity weight vector corresponding one-to-one with the structural capacity index.

[0120] Based on the structural capacity weight vector and the student model structural capacity vector, a student model structural reconstruction scheme is generated in the student model structural capacity vector index space. In the student model structural reconstruction scheme, the layer index, structural adjustment type and target structural parameters are registered for each structural capacity element.

[0121] The student model network structure is adjusted according to the student model structure reconstruction scheme. Based on the structure adjustment type, the number of channels, width parameters and connection relationships of the corresponding layers are retained, reduced and deleted to obtain the reconstructed student model.

[0122] In this embodiment, the generation of the lightweight student model that satisfies the target hardware resource constraints specifically includes:

[0123] In the structurally reconstructed student model, student features are extracted according to the distillation layer set. In the teacher model, teacher features are extracted according to the distillation layer set. Based on the transmission relationship registered in the approximate transmission matrix, a weighted matching is performed on the teacher features and student features. The distribution-aligned distillation loss is calculated. The calculation of the distribution-aligned distillation loss refers to the weighted summation of the difference measure between each pair of matched teacher features and student features according to the corresponding transmission weight, based on the transmission weight between the teacher feature index and the student feature index given by the approximate transmission matrix. The summation is then normalized on the sample dimension and the distillation layer set. The resulting scalar value is registered as the distribution-aligned distillation loss.

[0124] Based on the changes in layer index, channel number, and width parameter recorded in the student model structure reconstruction scheme, the structural change amount is calculated for the layers participating in structural adjustment in the student model structure reconstruction, and a structure preservation distillation loss is constructed. The construction of the structure preservation distillation loss refers to calculating the structural change amount for the layers participating in structural adjustment in the student model structure reconstruction according to a preset penalty function based on the changes in layer index, channel number, and width parameter recorded in the student model structure reconstruction scheme. All structural change amounts are aggregated and normalized in the layer dimension and sample dimension to obtain a scalar loss value used to constrain the degree of deviation of the student model structure reconstruction from the original structure.

[0125] The distribution alignment distillation loss, structure preservation distillation loss, and task loss are combined according to preset weight coefficients to form the total loss. The total loss is used to train and update the structure reconstruction student model. When the training termination condition is met, a lightweight student model that meets the target hardware resource constraints is output.

[0126] Example 1:

[0127] To verify the feasibility of this invention in practice, it was applied to a large-scale model inference system for online question-answering services. This system needs to simultaneously support three types of services on a unified cluster: dialogue question-answering, knowledge retrieval and response, and multi-turn consultation. The single-card memory capacity of online nodes is limited; the constraints provided by the operations team are that single-instance memory usage should not exceed 14GB, average response latency should be controlled within 300ms, and operation should be within the existing rack power quota. The original solution used a teacher model with approximately 13 billion parameters, deployed directly on high-end graphics cards. While the service quality was high, it could not run stably on resource-constrained nodes. Multiple student model versions could only be configured through manual trimming and empirical knowledge distillation. There were significant imbalances between different versions in terms of memory usage, throughput, and question-answering accuracy. Model structure adjustments were highly dependent on engineering experience, making rapid migration and deployment across various hardware configurations difficult, becoming a major bottleneck in system expansion and cost control.

[0128] In this online question-answering scenario, training data is first constructed using historical session logs and labeled question-answer pairs. Information such as the processing unit type, available computing power budget, GPU memory capacity, RAM capacity, and rack power quota of inference nodes are collected and encoded into a target hardware resource description vector. Using the existing 13 billion parameter model as the teacher model, the teacher model is run on the training data. Several layers are selected and registered as a distillation layer set based on the task output type and intermediate semantic representation. Simultaneously, an initial student model structure parameter set is generated based on the target hardware resource description vector and the teacher network structure. The number of channels and network width of each layer are calculated to form the student model structure capacity vector. During the training phase, the teacher features extracted from the distillation layer set and the structure capacity vector are input into the joint cost construction module of this invention to obtain a joint cost matrix that simultaneously characterizes semantic differences and on-call resource costs. Using the teacher feature distribution and structure capacity distribution as marginal constraints, and the joint cost matrix as the transmission cost, an entropy-regularized optimal transmission model with a Bregman regularization term is established. The Greenkhorn algorithm is then called to perform a resource adaptive truncation solution process under the control of computing power budget and storage budget, outputting an approximate transmission matrix that satisfies the resource constraints. Based on the approximate transfer matrix, a student model structure reconstruction scheme is generated in the structure capacity index space. The number of channels and width parameters of some attention layers and feedforward layers are reduced, and layers with lower importance are deleted, resulting in a structure-reconstructed student model with a memory footprint controlled within approximately 13.2GB. On this basis, a distributed alignment distillation loss is constructed using the approximate transfer matrix, and a structure-preserving distillation loss is constructed by combining the structure reconstruction information. These losses are weighted and combined with the question-answering task loss to form the total loss. The structure-reconstructed student model is then trained end-to-end until convergence on the validation set.

[0129] After training, the original experience-based knowledge distillation scheme and the method of this invention were compared, and offline evaluation and online gray-scale experiments were conducted on key indicators. Offline evaluation used an internal question-answering dataset of approximately 10,000 entries, covering three types of questions: fact-based question answering, long document summarization, and multi-turn inference. The overall accuracy, inference latency, and memory usage of different schemes were statistically analyzed. In the online experiments, the new model was deployed in a gray-scale manner on some inference instances, and the average number of requests completed per unit time and the error rate were recorded. The comparison results are shown in Table 1.

[0130] Table 1. Performance Comparison of the Invention Method and Empirical Distillation Scheme in Online Question-and-Answer Scenarios

[0131]

[0132] As can be seen from the table, under the same teacher model and training data conditions, by using the joint cost matrix and the entropy-regularized optimal transfer model with Bregman regularization in this invention, the teacher feature distribution and student model structural capacity distribution are explicitly coupled through an approximate transfer matrix. Combined with the resource-adaptive truncation Greenkhorn solution process and the structural reconstruction scheme generation mechanism based on the approximate transfer matrix, the student model achieves a reduction in memory usage of approximately 15%, while shortening inference latency, increasing peak throughput per GPU, and slightly improving accuracy in complex multi-turn inference scenarios. Therefore, applying the method of this invention to online question-answering systems allows for the unified execution of large model knowledge distillation and structural reconstruction processes within a single optimization framework within a given computing and storage budget. This enables lightweight student models to achieve a more balanced configuration between semantic expression capabilities, resource consumption, and inference performance, providing a repeatable and automated technical path for deploying large models in multi-instance, multi-form hardware environments.

[0133] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A lightweight deployment method for large models based on knowledge distillation, characterized in that, Includes the following steps: Acquire training data and target hardware resource information, construct teacher and student models, and determine the student model structure capacity vector and distillation layer set based on the target hardware resource information; The teacher model is run on the training data, teacher features are extracted from the distillation layer set, and a joint cost matrix that simultaneously represents semantic differences and resource costs is constructed based on the teacher features and the student model structure capacity vector. An entropy-regularized optimal transmission model with Bregman regularization terms is established, using teacher characteristics and student model structure capacity as marginal constraints and the joint cost matrix as the transmission cost. The Greenkhorn algorithm is used to iteratively solve the entropy-regularized optimal transmission model with Bregman regularization terms to obtain an approximate transmission matrix; During the Greenkhorn iteration process, the number of rows and columns and the number of iteration rounds can be updated according to the target hardware computing power budget and storage budget constraints, and the Bregman regularity strength is adjusted to form a resource-adaptive truncated Greenkhorn solution process; The student model structure reconstruction scheme is generated based on the approximate transfer matrix obtained from the resource adaptive truncation Greenkhorn solution process, and the student model structure is reconstructed based on the student model structure reconstruction scheme. Based on the approximate transfer matrix and the student model structure reconstruction scheme, a distillation loss is constructed and combined with the task loss to train the reconstructed student model, resulting in a lightweight student model that meets the target hardware resource constraints.

2. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The Bregman regularization term is as follows: based on the deviation measure between each transmission quantity in the transmission matrix and the corresponding transmission quantity in the reference transmission matrix, a deviation measure function is constructed according to a given generating function, and the deviation values ​​between all transmission quantities and the reference transmission quantity are calculated according to the deviation measure function. The regularization term is formed by multiplying all deviation values ​​by the regularization strength parameter and then adding them together. This regularization term is used to limit the deviation of the transmission matrix relative to the reference transmission matrix.

3. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The teacher model and student model are two neural network models configured in the same input space and task output space. The teacher model is a large-scale neural network model with fixed parameters and high expressive power, while the student model is a lightweight neural network model with the number of network layers, channels, and attention heads set according to the target hardware resource information. Both are used to construct the joint cost matrix, establish the optimal transmission model, and train the lightweight student model according to the layer index output features registered in the distillation layer set.

4. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The construction of the teacher and student models, and the determination of the student model's structural capacity vector and distillation layer set, specifically include: Obtain the original training dataset, which contains sample inputs and supervision labels. Perform format unification, outlier removal, and numerical normalization on the original training dataset to obtain the training data. Collect target hardware resource information, which includes at least the processing unit type, computing power budget, video memory capacity, memory capacity, storage bandwidth and power consumption limit, and encode it into a target hardware resource description vector; Configure the teacher model network structure, load the teacher model parameters, and register the output layer index and intermediate layer index of the teacher model; Based on the target hardware resource description vector and the teacher model network structure, the initial network structure of the student model is set, the number of channels in each layer, the network width and the number of attention heads are determined, and the set of student model structure parameters is obtained. The structural capacity of each layer is calculated based on the set of structural parameters of the student model, and the structural capacities of each layer are arranged in a fixed order to form a student model structural capacity vector. Based on the correspondence between the layers of the teacher model and the layers of the student model, as well as the training task requirements, the teacher model layer and student model layer that participate in knowledge distillation are selected, and the corresponding indexes are registered to form a set of distillation layers.

5. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The construction of the joint cost matrix specifically includes: The teacher model is run on the training data in the order of samples. Teacher features are extracted from the distillation layer set by layer index. The teacher features of each sample in the distillation layer set are concatenated into a teacher feature vector in a fixed order. All teacher feature vectors are collected to form a teacher feature vector set. Perform dimension alignment and numerical normalization on the teacher feature vectors in the teacher feature vector set to obtain a normalized teacher feature vector set; Based on the order of each structural capacity element in the student model structural capacity vector, register a structural capacity index for each structural capacity element, combine the structural capacity index and structural capacity element to form a structural capacity unit, and collect all structural capacity units to form a structural capacity unit set. For each teacher feature vector in the teacher feature vector set and each structural capacity unit in the structural capacity unit set, calculate the semantic difference value between them, and record the semantic difference value with the teacher feature vector index and the structural capacity index as the key to form a semantic difference record set. For each structural capacity unit in the structural capacity unit set, the resource cost value is calculated based on the value of the structural capacity element and the target hardware resource information, and the resource cost value is recorded with the structural capacity index as the key to form a resource cost record set. The semantic difference values ​​are read from the semantic difference record set according to the teacher feature vector index and the structural capacity index, and the resource cost values ​​are read from the resource cost record set. The semantic difference values ​​and resource cost values ​​are combined according to the preset weight coefficients to obtain the joint cost values. The joint cost values ​​are arranged into a matrix form according to the teacher feature vector index and the structural capacity index and registered as the joint cost matrix.

6. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The establishment of the entropy-regularized optimal transmission model with Bregman regularization specifically includes: A teacher feature index set is established based on the index of each teacher feature vector in the teacher feature vector set. Non-negative weights are calculated for each teacher feature vector in the teacher feature vector set. Normalization is performed on all non-negative weights to obtain the teacher feature distribution supported by the teacher feature index set. A structural capacity index set is established based on the index of each structural capacity element in the student model structural capacity vector. Non-negative weights are calculated for each structural capacity element in the student model structural capacity vector. Normalization is performed on all non-negative weights to obtain a structural capacity distribution supported by the structural capacity index set. Each element in the joint cost matrix is ​​associated with the teacher feature index and the structural capacity index. Each element in the transfer matrix is ​​associated with the teacher feature index and the structural capacity index. Entropy regularization strength parameter, Bregman regularization strength parameter, reference transfer matrix elements and Bregman divergence function constructed based on the generating function are introduced. The optimal transfer objective function with entropy regularization and Bregman regularization is constructed with the transfer matrix as the decision variable. Row edge constraints and column edge constraints are set. The row edge constraint requires that the sum of all elements in each row of the transmission matrix equals the probability value of the corresponding teacher feature index position in the teacher feature distribution. The column edge constraint requires that the sum of all elements in each column of the transmission matrix equals the probability value of the corresponding structural capacity index position in the structural capacity distribution. Each element of the transmission matrix is ​​also restricted to be a non-negative value. The optimal transmission objective function, row edge constraints, column edge constraints and non-negative value constraints are collectively registered as an entropy-regularized optimal transmission model with Bregman regularization.

7. The lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The process of solving the approximate transfer matrix specifically includes: For the entropy regularized optimal transport model with Bregman regularization, initialize the row scaling vector and column scaling vector. In the row scaling vector, register the corresponding teacher feature index for each element, and in the column scaling vector, register the corresponding structure capacity index for each element. Set the iteration round counter, row constraint residual threshold and column constraint residual threshold. In each round of Greenkhorn iteration, the transfer matrix is ​​calculated based on the current row scaling vector, the current column scaling vector, and the joint cost matrix. The row edge sum is obtained by summing all elements in each row of the transfer matrix, and the column edge sum is obtained by summing all elements in each column of the transfer matrix. Based on the teacher characteristic distribution and row margins, calculate the row constraint residual sequence; based on the structural capacity distribution and column margins, calculate the column constraint residual sequence; and determine the index of the element with the largest row contribution in the row constraint residual sequence and the index of the element with the largest column contribution in the column constraint residual sequence according to the absolute value of the residuals, and register them as candidate row indices and candidate column indices. Compare the row contribution of the candidate row index with the column contribution of the candidate column index. If the row contribution is greater than the column contribution, register the candidate row index as the target index. If the row contribution is less than the column contribution, register the candidate column index as the target index. If the row contribution and column contribution are the same, select the target index according to the preset rules. When the target index corresponds to the teacher feature index, the value of the element at the target index position in the row scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding row in the transmission matrix is ​​close to the probability value of the same teacher feature index position in the teacher feature distribution. When the target index corresponds to the structure capacity index, the value of the element at the target index position in the column scaling vector is updated according to the entropy regularization strength parameter and the Bregman regularization strength parameter, so that the sum of all elements in the corresponding column in the transmission matrix is ​​close to the probability value of the same structure capacity index position in the structure capacity distribution. After completing a round of row scaling vector update or column scaling vector update, the absolute value of each row constraint residual and the absolute value of each column constraint residual are determined based on the new row constraint residual sequence and the new column constraint residual sequence. If the absolute values ​​of all row constraint residuals and all column constraint residuals do not exceed the corresponding threshold, the current transfer matrix is ​​registered as an approximate transfer matrix and the Greenkhorn iteration is terminated. If the absolute value of either row constraint residual or column constraint residual exceeds the corresponding threshold, the iteration round counter is incremented and the current row scaling vector and the current column scaling vector are retained to enter the next round of Greenkhorn iteration.

8. The lightweight deployment method for a large model based on knowledge distillation according to claim 1, characterized in that, The process of adjusting the Bregman regularity strength to form a resource-adaptive truncation Greenkhorn solution specifically includes: Based on the target hardware resource information, set the computing power budget index and storage budget index for Greenkhorn iteration. The computing power budget index is represented by the number of available floating-point operations, and the storage budget index is represented by the amount of available memory usage, and register them as the resource budget parameters for Greenkhorn iteration. During the Greenkhorn iteration process, for each iteration, the number of floating-point operations and the amount of memory usage generated by the row scaling vector update and column scaling vector update are recorded. The number of floating-point operations is accumulated as the cumulative computing power consumption, and the amount of memory usage is accumulated as the cumulative storage consumption. The current iteration number is also recorded. The computing power utilization ratio is calculated based on the computing power budget index and the cumulative computing power consumption. The storage utilization ratio is calculated based on the storage budget index and the cumulative storage consumption. The larger value between the computing power utilization ratio and the storage utilization ratio is selected and registered as the truncation ratio. A resource adaptive truncation parameter set is generated based on the truncation ratio. The resource adaptive truncation parameter set includes at least the upper limit of the number of row indexes allowed to be updated in each round, the upper limit of the number of column indexes allowed to be updated in each round, and the upper limit of the number of Greenkhorn iteration rounds. At the start of the Greenkhorn iteration, the resource adaptive truncation parameter set is read, and constraints are imposed on the number of times the row candidate index and column candidate index are selected, so that the number of times the row scaling vector is updated in the current round does not exceed the upper limit of the number of row indexes, and the number of times the column scaling vector is updated in the current round does not exceed the upper limit of the number of column indexes. After each round, the resource adaptive truncation parameter set is updated according to the truncation ratio and the current iteration round number. When any one of the following three conditions is met: the cumulative computing power consumption reaches the total number of floating-point operations limited by the computing power budget target, the cumulative storage consumption reaches the total memory usage limited by the storage budget target, or the number of iteration rounds reaches the upper limit of the Greenkhorn iteration rounds, a resource truncation flag is generated, the Greenkhorn iteration is terminated, and the transfer matrix at the time of termination is registered as the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process.

9. A lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The reconstruction of the student model structure specifically includes: Read the approximate transfer matrix output by the resource adaptive truncation Greenkhorn solution process, perform column aggregation on the approximate transfer matrix according to the structural capacity index set, and obtain the structural capacity weight vector corresponding one-to-one with the structural capacity index. Based on the structural capacity weight vector and the student model structural capacity vector, a student model structural reconstruction scheme is generated in the student model structural capacity vector index space. In the student model structural reconstruction scheme, the layer index, structural adjustment type and target structural parameters are registered for each structural capacity element. The student model network structure is adjusted according to the student model structure reconstruction scheme. Based on the structure adjustment type, the number of channels, width parameters and connection relationships of the corresponding layers are retained, reduced and deleted to obtain the reconstructed student model.

10. A lightweight deployment method for large models based on knowledge distillation according to claim 1, characterized in that, The generation of the lightweight student model that satisfies the target hardware resource constraints specifically includes: In the structural reconstruction student model, student features are extracted according to the distillation layer set, and in the teacher model, teacher features are extracted according to the distillation layer set. Based on the transmission relationship registered in the approximate transmission matrix, a weighted matching is performed on the teacher features and student features, and the distribution alignment distillation loss is calculated. Based on the changes in layer index, channel number, and width parameters recorded in the student model structure reconstruction scheme, the structural change amount is calculated for the layers involved in structural adjustment in the student model of structure reconstruction, and the structural preservation distillation loss is constructed. The distribution alignment distillation loss, structure preservation distillation loss, and task loss are combined according to preset weight coefficients to form the total loss. The total loss is used to train and update the structure reconstruction student model. When the training termination condition is met, a lightweight student model that meets the target hardware resource constraints is output.

Images