Large model compression and online incremental learning method for supply chain and e-commerce procurement based on third-order joint distillation

Abstract

The present invention discloses a large-scale model compression and online incremental learning method for supply chain and e-commerce procurement based on three-order joint distillation, comprising: collecting and preprocessing supply chain compliance documents and purchase order structured data to obtain a data set; constructing a teacher model and a student model, each of which includes an encoder, a decoder, and a prediction head; performing three-stage knowledge distillation on the teacher model and the student model using the data set; collecting new data to determine whether the new data triggers incremental learning conditions, and if so, performing incremental learning using the teacher model and the student model obtained by the three-stage knowledge distillation to obtain trained teacher model and student model; and performing collaborative reasoning using the trained teacher model and student model to obtain prediction results. Knowledge distillation is achieved through a hierarchical migration protocol and multimodal joint optimization, and combined with an online incremental learning mechanism to address data distribution drift issues such as e-commerce promotions, seasonal demand, and supplier quotation fluctuations.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model compression and online incremental learning method in the supply chain and e-commerce procurement fields based on third-order joint distillation. Background Art

[0002] Compliance analysis and path reasoning in supply chain management require processing massive amounts of unstructured text data (such as logistics documents and contract terms) as well as e-commerce procurement transaction information (such as real-time inventory, order execution, and supplier quotes). Existing technologies generally rely on large language models (LLMs) for semantic understanding and decision generation, but such models still have the following technical limitations in supply chain and e-commerce procurement scenarios:

[0003] (1) Model is not lightweight enough

[0004] The number of parameters in general large language models usually exceeds tens of billions (for example, GPT-3 has 175 billion parameters), resulting in high inference latency and large hardware resource usage. According to literature reports, a single inference time of traditional supply chain compliance analysis models generally exceeds several seconds, making it difficult to meet real-time requirements in industrial production. Although knowledge distillation (KD) technology can compress the model through a teacher-student architecture, the distillation process is usually accompanied by significant accuracy loss in supply chain multi-task scenarios. For example, the accuracy of the compressed model in the entity resolution task generally drops by more than 10%. In the field of e-commerce procurement, such as when order demand is quickly split and allocated during e-commerce promotions, the delay and lack of accuracy of the model will further aggravate the obstacles to practical application.

[0005] (2) Poor adaptability to dynamic data

[0006] Dynamic characteristics in supply chain scenarios, such as changes in logistics routes and cost fluctuations, lead to frequent drift in data distribution. Most existing large language models use a static training model, requiring full retraining to adapt to new data, resulting in high time and resource overhead. Incremental learning technology theoretically supports online updates, but its ability to adapt to data drift is limited, making it difficult to strike a balance between real-time performance and model stability. E-commerce procurement also faces seasonal promotions, frequent changes in market conditions, and supplier quotes, making it difficult for traditional models to accurately and quickly make order decisions and forecast demand.

[0007] Currently, technological improvements are mostly focused on a single optimization direction. For example, global pruning methods can reduce the number of model parameters, but in supply chain entity relationship extraction tasks, the performance loss of the sparsified model generally exceeds 15%. Automatic merging and splitting algorithms for e-commerce purchase orders also face similar problems. Deep pruning can easily lead to a decline in order anomaly detection and dynamic scheduling capabilities. In addition, some research has attempted to optimize the dynamic update mechanism of the model, but the coordinated optimization of lightweightness and dynamic adaptability remains challenging, limiting the large-scale application of large language models in the supply chain and e-commerce procurement fields. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides a large-scale model compression and online incremental learning method for the supply chain and e-commerce procurement fields based on third-order joint distillation. Knowledge distillation is achieved through a layered migration protocol and multimodal joint optimization, and combined with an online incremental learning mechanism to address data distribution drift problems such as e-commerce promotions, seasonal demand, and supplier quotation fluctuations.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A large model compression and online incremental learning method for supply chain and e-commerce procurement based on third-order joint distillation includes the following steps:

[0011] Collect supply chain compliance documents and purchase order structured data, pre-process the supply chain compliance documents and purchase order structured data to obtain a data set;

[0012] Build a teacher model and a student model based on the large language model. Both the teacher model and the student model include an encoder, a decoder, and a prediction head.

[0013] Using the dataset, the teacher model and the student model are subjected to three-stage knowledge distillation. Specifically, the three-stage knowledge distillation performs layered knowledge transfer in the order of encoder distillation stage, decoder distillation stage, and prediction head distillation stage.

[0014] Collect new data and determine whether the new data triggers the conditions for incremental learning. If so, use the teacher model and student model obtained by the three-stage knowledge distillation to perform incremental learning to obtain the trained teacher model and student model;

[0015] The trained teacher model and student model are used for collaborative reasoning. If the prediction confidence of the student model is less than the threshold, the outputs of the teacher model and the student model are weighted and fused to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results; otherwise, the prediction results of the student model are directly used as the final output to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results.

[0016] To optimize the above technical solutions, specific measures taken also include:

[0017] Furthermore, the pre-processing of the supply chain compliance documents and purchase order structured data specifically includes:

[0018] Label terms with classification tags, paragraph segmentation and regularization, BPE-based word segmentation, unit unification, field alignment and missing value interpolation.

[0019] Further, the encoder includes a self-attention layer and a feed-forward sublayer;

[0020] The encoder distillation stage is specifically as follows:

[0021] Split the dataset into training and validation sets, and configure hyperparameters, including learning rate, batch size, maximum number of iterations, and optimizer;

[0022] After the encoder is trained on the training set, it is validated using the validation set and the F1 score is calculated. If the F1 score fluctuates less than the set range for five consecutive times, the encoder distillation phase has partially converged and the encoder parameters are frozen, including the parameters of all self-attention layers and feedforward sublayers. If the loss value of the validation set does not decrease after the set number of iterations, the learning rate is reduced and gradient clipping is enabled.

[0023] During the encoder training process, the intermediate representations of the teacher model are cached locally.

[0024] Furthermore, the decoder input is supplemented with position codes or segment vectors for distinguishing different data modalities, including text, numeric values, and timestamps; or the decoder input is supplemented with an embedding layer and a multi-layer perceptron connected in sequence for projecting numeric fields;

[0025] The decoder distillation stage specifically includes:

[0026] When processing multimodal data at the decoder side, different modalities are uniformly mapped into an alignable vector space to perform multimodal feature fusion;

[0027] Configure hyperparameters, including learning rate and batch size;

[0028] The goals of the decoder distillation phase include attention matrix alignment and semantic distribution alignment. Attention matrix alignment involves keeping the attention matrices generated by each layer of the student model similar to the attention matrices of the corresponding layers of the teacher model, with similarity measured using the bi-norm or mean squared error. Semantic distribution alignment involves aligning the output distribution of the student model with the output distribution of the teacher model on the same input, with the difference between the two measured using KL divergence or mean squared error.

[0029] When the path inference accuracy or F1 score reaches the threshold, freeze the decoder parameters; otherwise, increase the number of iterations to fine-tune the decoder.

[0030] pass Project the student model attention to the teacher model space, where ψ(·) represents the projection function from the student model attention representation space to the teacher model attention representation space, and W ψ is a trainable projection matrix used to map the dimension or distribution of the student's attention matrix to a space that is equal or comparable to the teacher's attention. Model for students ′ Attention matrix generated by the layer;

[0031] Update the projection matrix W ψ , an L2 regularization term is added when updating the projection matrix, and the parameters of the projection matrix are updated each time back propagation.

[0032] Furthermore, the prediction head distillation stage is specifically as follows:

[0033] Update the low-rank adaptation layer parameters W adapt , train the prediction head, randomly extract historical data from the encoder distillation stage and the decoder distillation stage and mix them into the current training to prevent the model from forgetting the previous task in the final distillation stage; during the training process, each time a preset number of iterations is reached, the F1 score of the compliance terms classification task and the Wasserstein distance of the procurement cost distribution prediction task are evaluated. If there is no improvement in the evaluation indicators for three consecutive times, the prediction head distillation is terminated.

[0034] Furthermore, the incremental learning condition is specifically a data distribution difference condition or a business indicator monitoring condition:

[0035] The data distribution difference condition is: if the category labels of more than a preset percentage of samples in the new data are out of the distribution range of the old data, or the KL divergence difference is greater than a threshold, then incremental learning is triggered;

[0036] The business indicator monitoring conditions are: the domain entity recall rate is less than the set value for three consecutive batches, or the MSE of the transaction volume forecast exceeds the set threshold by 1.5 times, or the NDCG of the supplier ranking decreases by more than a preset percentage.

[0037] Furthermore, the incremental learning of the teacher model and student model obtained by the three-stage knowledge distillation is specifically as follows:

[0038] The prediction head distillation strategy is used, but only the low-rank adaptation layer is updated, and the encoder and decoder remain frozen. If fine-tuning of some feature extraction layers is required, the last 1-2 layers of the encoder are unfrozen after updating the low-rank adaptation layer, and the corresponding learning rate is set to 10% of the initial learning rate.

[0039] New data is collected every week for incremental learning, and the Mini-Epoch strategy is also adopted in the incremental learning stage.

[0040] Furthermore, the loss function of the three-stage knowledge distillation is as follows:

[0041]

[0042] Where, represents the loss value of the three-stage knowledge distillation, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT Percentage of total losses;

[0043] The expression of the semantic distribution distillation loss is as follows:

[0044]

[0045] Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, z T and z S Represent the log-odds vectors of the teacher model and the student model respectively;

[0046] The expression of the topology-aware attention distillation loss is as follows:

[0047]

[0048] Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth order corresponding to the student model ′ Attention matrix of the layer Imposed mapping;

[0049] The expression of the domain reinforcement task loss is as follows:

[0050]

[0051] Where λ i represents the loss weight of the i-th subtask, L task,irepresents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task;

[0052] After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows:

[0053]

[0054] Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

[0055] Furthermore, the loss function of the incremental learning is as follows:

[0056]

[0057] Where, L 2 total Represents the loss value of incremental learning, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT The proportion of total loss; μ represents the stability coefficient, θ s represents the parameter vector of the student model at the current training stage, Represents the parameter vector saved by the student model at the previous moment or the previous incremental stage;

[0058] The expression of the semantic distribution distillation loss is as follows:

[0059]

[0060] Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, z T and z S Represent the log-odds vectors of the teacher model and the student model respectively;

[0061] The expression of the topology-aware attention distillation loss is as follows:

[0062]

[0063] Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth order corresponding to the student model ′ Attention matrix of the layer Imposed mapping;

[0064] The expression of the domain reinforcement task loss is as follows:

[0065]

[0066] Where λ i represents the loss weight of the i-th subtask, L task,i represents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task;

[0067] After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows:

[0068]

[0069] Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

[0070] Furthermore, the loss of the compliance clause classification task is measured by cross entropy loss. The loss function of the compliance clause classification task is as follows:

[0071]

[0072] Where, L task,1 represents the loss of the compliance clause classification task, L CE represents the cross entropy loss, k represents the compliance category index,

[0073] The loss of the procurement cost distribution prediction task is measured by the Wasserstein distance. The loss function of the procurement cost distribution prediction task is as follows:

[0074]

[0075] Where, L task,2 represents the loss of the procurement cost distribution prediction task, L EMD Indicates the cumulative distance between the predicted distribution and the true distribution, i is the cost interval index, CDF pred (i) represents the value of the predicted cumulative distribution function at interval index i, CDF true(i) represents the value of the true cumulative distribution function at interval index i;

[0076] The loss of the supplier prioritization task is measured by the ranking loss, and the loss function of the supplier prioritization task is as follows:

[0077]

[0078] Where, L task,3 is the loss of supplier prioritization task, L Rank represents the ranking loss, s i and s j They represent the ranking scores of supplier i and supplier j predicted by the model, y (i,j) represents the preference labels of suppliers i and j. If i performs better than j, then y (i,j) = +1; if j is better than i, then y (i,j) = -1, pair(i,j) refers to all comparable supplier pairs in the training set.

[0079] The beneficial effects of the present invention are:

[0080] By gradually distilling and freezing the core components of the model in three stages, the inference latency can be significantly reduced, and refined distillation and adaptation can be achieved at the encoder, decoder, and prediction head levels.

[0081] By distilling the semantic distribution loss L SD With topology-aware attention distillation loss L TA By combining the two, the knowledge of the teacher model can be absorbed in two aspects: output layer distribution alignment and intermediate attention structure alignment, so as to obtain a more comprehensive distillation effect in the multimodal fusion scenario; at the same time, in order to take into account the differentiated needs of each subtask, the domain strengthening task loss L DT Dynamically balance the training weights of different tasks so that the student model inherits the teacher's knowledge while maintaining sensitivity to business goals; finally, by introducing Constraining the drastic updates of the student model's parameters not only adapts to new data distribution drift but also mitigates the forgetting of existing knowledge. These four factors together constitute the comprehensive optimization goal. By jointly optimizing semantic distribution, topological attention, and domain reinforcement tasks, the model balances knowledge alignment with target task requirements under multimodal input, achieving higher reasoning accuracy and lower model complexity in supply chain and e-commerce procurement scenarios.

[0082] Through progressive distillation and incremental learning mechanisms, it can dynamically adapt to scenarios such as e-commerce promotions, seasonal demand, and fluctuations in supplier quotations while maintaining efficient reasoning, achieving continuous optimization of multiple tasks (compliance clause classification, entity resolution, path prediction, procurement order optimization, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is the overall flow chart of the large model compression and online incremental learning method for supply chain and e-commerce procurement based on third-order joint distillation proposed in this invention.

[0084] Figure 2 Flowchart of the decoder distillation stage. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0086] Example 1

[0087] This paper proposes a large model compression and online incremental learning method for supply chain and e-commerce procurement based on three-order joint distillation. The process of this method is as follows: Figure 1 As shown, the following steps are included:

[0088] Supply chain compliance documents and purchase order structured data are collected and preprocessed to obtain a data set.

[0089] Supply Chain Compliance Documents: This example assumes the use of PDF files and text data converted after OCR (e.g., import / export agreements), annotated with clause classification labels (e.g., "CE Certification"). The average length of the text is approximately 2,000 to 3,000 words. After paragraph segmentation and regularization (de-duplication, denoising, and special character removal), word segmentation is performed using BPE (Byte Pair Encoding).

[0090] Purchase order structured data: Order or contract information in JSON format (including fields such as supplier name and product specifications). Example fields may include key attributes such as {"supplier_id","product_type","order_quantity","unit_price","delivery_date"}, some of which may contain nested levels (such as subfields under "product_details"). Data should be standardized (units unified, fields aligned), and missing values ​​imputed.

[0091] Mixed-modal records: Comparison tables between text paragraphs in bidding documents and quotation data. Reference supplier quotations from external archiving systems and establish a "text information - quotation field" index mapping so that the model can read both natural language descriptions and numeric fields when multimodal input is used. This primarily involves quotation ranges, volume discounts, and delivery requirements.

[0092] The teacher model and student model are constructed based on the large language model respectively. Both the teacher model and the student model include an encoder, a decoder and a prediction head.

[0093] In this embodiment, both the teacher model and the student model can be built based on large pre-trained language models (LLMs), such as BERT, GPT and other pre-trained language models, but the two differ in scale, such as the number of layers, hidden dimensions, and the number of attention heads, to achieve a balance between performance and inference efficiency before and after distillation.

[0094] Teacher model structure: (1) Overall framework: The Transformer encoder-decoder structure can be used, or only the encoder (for plain text encoding) can be used in combination with the design of the downstream prediction head. If the teacher model is used in a multimodal input scenario, a corresponding numerical / timestamp embedding layer is added to the input of its encoder or decoder, or a layer of MLP is added after the original text embedding to map the structured field. (2) Embedding layer and decoder: Embedding layer: maps text tokens, numerical fields, etc. into fixed-dimensional vectors (such as d=1024), which may include positional encoding (PositionalEncoding) or segment vectors (Segment Embedding). Decoder: interacts with the encoder output and can predict only part of the sequence in a multi-task situation; its multi-head self-attention (Self-Attention) and cross-attention (Cross-Attention) matrices are usually large, which is suitable for guiding the student model to align the attention distribution during the distillation process. (3) Prediction head: The teacher model usually uses a full linear layer or a multi-layer perceptron (MLP) as the final output layer, and the dimension directly corresponds to the task requirements. For example, the Softmax layer for classification tasks, the linear layer for regression tasks, or the multi-layer projection layer. These head parameters will provide a high-precision reference distribution (such as classification probability, regression value, etc.) for student model alignment during distillation.

[0095] Student model structure: (1) Overall scale reduction: The student model can inherit the main structure of the teacher model, but reduce the number of Transformer layers (e.g., from 12 layers to 6 layers), reduce the hidden vector dimension d (e.g., from 1024 to 512), or reduce the number of heads H in the multi-head attention (e.g., from 16 to 8). Some parameters can be significantly compressed using low-rank decomposition (e.g., W_adapt = U·V^T) or factorization. (2) Decoder and Embedding layer: To ensure input format alignment, the student model still needs to retain the same Embedding type (text token, numeric field, position encoding, etc.), but it is smaller in dimension or merges some redundant vectors. The decoder structure will align the hierarchical order of the teacher, but can be sparse in the number of layers or attention heads to reduce computational cost. (3) Prediction head structure and distillation: The prediction head of the student model can be a scaled-down linear layer or a low-rank adaptation layer, using activation functions and output interfaces similar to those of the teacher to ensure output alignment.

[0096] The dataset is used to perform three-stage knowledge distillation on the teacher model and the student model. The three-stage knowledge distillation specifically performs layered knowledge transfer in the order of encoder distillation stage, decoder distillation stage, and prediction head distillation stage. The encoder distillation stage specifically includes an encoder that includes a self-attention layer and a feedforward sublayer. The dataset is divided into a training set and a validation set, and hyperparameters are configured, including learning rate, batch size, maximum number of iterations, and optimizer. Example hyperparameter configuration: learning rate 1×10 -4 , Batch Size = 32, iterations of about 100,000 steps; the optimizer can be AdamW, β1 = 0.9, β2 = 0.999.

[0097] After the encoder is trained on the training set, it is validated on the validation set and the F1 score is calculated. If the F1 score fluctuates less than the set range for five consecutive times, the encoder distillation phase has partially converged and the encoder parameters are frozen, including the parameters of all self-attention layers and feedforward sublayers. If the loss value of the validation set does not decrease after the set number of iterations, the learning rate is reduced and gradient clipping is enabled (e.g., clipping threshold = 1.0);

[0098] During encoder training, the intermediate representations of the teacher model are cached locally to reduce repeated forward computations.

[0099] The decoder input is supplemented with position codes or segment vectors for distinguishing different data modalities, including text, numeric values, and timestamps; or the decoder input is supplemented with an embedding layer and a multi-layer perceptron connected in sequence for projecting numeric fields;

[0100] The flowchart of the decoder distillation stage is as follows Figure 2 As shown in Figure 2, the decoder distillation stage specifically includes:

[0101] When processing multimodal data at the decoder side, different modalities are uniformly mapped into an alignable vector space to perform multimodal feature fusion;

[0102] Configure hyperparameters, including learning rate and batch size;

[0103] The goals of the decoder distillation phase include attention matrix alignment and semantic distribution alignment. Attention matrix alignment involves ensuring that the attention matrices generated by each layer of the student model are similar to those of the corresponding layers of the teacher model. This similarity is measured using the bi-norm or mean squared error (MSE), allowing the student's attention distribution to align with that of the teacher model. Semantic distribution alignment involves aligning the output distribution of the student model with that of the teacher model on the same input, using KL divergence or mean squared error (MSE) to measure the difference. This ensures that the student's "semantic choices" or decision-making tendencies, when ultimately decoding or predicting, closely match those of the teacher.

[0104] When the path reasoning accuracy (PRA) or F1 score reaches a threshold, freeze the decoder parameters; otherwise, increase the number of iterations to fine-tune the decoder.

[0105] pass Project the student model attention to the teacher model space, where ψ(·) represents the projection function from the student model attention representation space to the teacher model attention representation space, and W ψ is a trainable projection matrix, the general dimension is (or reshape / flatten depending on the number of attention heads and sequence length), used to map the dimension or distribution of the student's attention matrix to a space that is equal or comparable to the teacher's attention. The attention matrix generated for the l′th layer of the student model; the shape is generally

[0106] Update the projection matrix W ψ , an L2 regularization term is added when updating the projection matrix to prevent overfitting, and the parameters of the projection matrix are updated each time back propagation.

[0107] Multimodal fusion (introducing text and numerical embeddings at the decoder input) and attention mapping (comparing teacher-student attention distribution) are performed in the forward calculation and distillation loss calculation stages respectively.

[0108] The prediction head distillation stage is specifically as follows: updating the low-rank adaptation layer parameters W adapt =U·V T , whose dimension is usually 10% to 30% smaller than the original fully connected layer, which can significantly reduce the model size while maintaining high accuracy.

[0109] Among them, U: size is usually r<<d. Responsible for mapping the input features to a more compact space after dimensionality reduction. V: The size is usually Used to reproject the representation after dimensionality reduction back to the output space with the same or compatible dimensions as the original. Train the prediction head, exemplary training: perform 20,000 steps, Batch Size = 16 or 32. The number of training rounds can be shortened based on experience. For example, if the indicators have converged on a single task, it can be stopped early. In order to stabilize the distillation effect, historical data (such as compliance terms and purchase orders) from the encoder distillation stage and the decoder distillation stage are randomly sampled and mixed into the current training to prevent the model from forgetting the previous tasks in the final distillation stage. During the training process, every time a preset number of iterations is reached, the F1 score of the compliance terms classification task and the Wasserstein distance of the procurement cost distribution prediction task are evaluated. If there is no improvement in the evaluation indicators for three consecutive times, the prediction head distillation is terminated.

[0110] The loss function of the three-stage knowledge distillation is as follows:

[0111]

[0112] Where, represents the loss value of the three-stage knowledge distillation, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT Percentage of total losses;

[0113] The expression of the semantic distribution distillation loss is as follows:

[0114]

[0115] Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, zx T and z S Represent the log-odds vectors of the teacher model and the student model respectively;

[0116] The expression of the topology-aware attention distillation loss is as follows:

[0117]

[0118] Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth order corresponding to the student model ′ Attention matrix of the layer Imposed mapping;

[0119] The expression of the domain reinforcement task loss is as follows:

[0120]

[0121] Where λ i represents the loss weight of the i-th subtask, L task,i represents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task;

[0122] After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows:

[0123]

[0124] Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

[0125] The loss of the compliance clause classification task is measured by cross entropy loss. The loss function of the compliance clause classification task is as follows:

[0126]

[0127] Where, L task,1 represents the loss of the compliance clause classification task, L CE represents the cross entropy loss, k represents the compliance category index,

[0128] The loss of the procurement cost distribution prediction task is measured by the Wasserstein distance. The loss function of the procurement cost distribution prediction task is as follows:

[0129]

[0130] Where, L task,2 represents the loss of the procurement cost distribution prediction task, L EMD Indicates the cumulative distance between the predicted distribution and the true distribution, i is the cost interval index, CDF pred (i) represents the value of the predicted cumulative distribution function at interval index i, CDF true (i) represents the value of the true cumulative distribution function at interval index i;

[0131] The loss of the supplier prioritization task is measured by the ranking loss, and the loss function of the supplier prioritization task is as follows:

[0132]

[0133] Where, L task,3 is the loss of supplier prioritization task, L Rank represents the ranking loss, s i and s j They represent the ranking scores of supplier i and supplier j predicted by the model, y (i,j) represents the preference labels of suppliers i and j. If i performs better than j, then y (i,j) = +1; if j is better than i, then y (i,j) = -1, pair(i,j) refers to all comparable supplier pairs in the training set.

[0134] Collect new data and determine whether the new data triggers the conditions for incremental learning. If so, use the teacher model and student model obtained from the three-stage knowledge distillation to perform incremental learning to obtain trained teacher models and student models. The conditions for incremental learning are specifically data distribution difference conditions or business indicator monitoring conditions:

[0135] The data distribution difference condition is: if the category labels of more than a preset percentage of samples in the new data are out of the distribution range of the old data, or the KL divergence difference is greater than a threshold, then incremental learning is triggered;

[0136] The business indicator monitoring conditions are: the domain entity recall rate is less than the set value for three consecutive batches, or the MSE of the transaction volume forecast exceeds the set threshold by 1.5 times, or the NDCG of the supplier ranking decreases by more than a preset percentage.

[0137] The incremental learning of the teacher model and student model obtained by the three-stage knowledge distillation is specifically as follows:

[0138] The prediction head distillation strategy is used, but only the low-rank adaptation layer is updated, and the encoder and decoder remain frozen. If fine-tuning of some feature extraction layers is required, the last 1-2 layers of the encoder are unfrozen after updating the low-rank adaptation layer, and the corresponding learning rate is set to 10% of the initial learning rate.

[0139] New data is collected every week for incremental learning, and the Mini-Epoch strategy is also adopted in the incremental learning stage.

[0140] The loss function of incremental learning is as follows:

[0141]

[0142] Where, L2 total Represents the loss value of incremental learning, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT The proportion of total loss; μ represents the stability coefficient, θ S represents the parameter vector of the student model at the current training stage, Represents the parameter vector saved by the student model at the previous moment or the previous incremental stage;

[0143] The expression of the semantic distribution distillation loss is as follows:

[0144]

[0145] Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, z T and z S Represent the log-odds vectors of the teacher model and the student model respectively;

[0146] The expression of the topology-aware attention distillation loss is as follows:

[0147]

[0148] Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth ′ Attention matrix of the layer Imposed mapping;

[0149] The expression of the domain reinforcement task loss is as follows:

[0150]

[0151] Where λ i represents the loss weight of the i-th subtask, L task,i represents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task;

[0152] After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows:

[0153]

[0154] Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

[0155] By jointly optimizing semantic distribution, topological attention, and domain reinforcement tasks, the model can take into account knowledge alignment and target task requirements under multimodal input, and achieve higher reasoning accuracy and lower model complexity in supply chain and e-commerce procurement scenarios.

[0156] The trained teacher model and student model are used for collaborative reasoning. If the prediction confidence of the student model is less than the threshold, the outputs of the teacher model and the student model are weighted and fused to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results; otherwise, the prediction results of the student model are directly used as the final output to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results.

[0157] If the student model prediction confidence (Softmax maximum value) is less than 0.7, the weighted fusion teacher-student model output is:

[0158] y final =0.7·y S +0.3·y T

[0159] Where y final Represents the final prediction result, y S is the prediction result of the student model, y T is the prediction result of the teacher model.

[0160] In a multi-classification scenario, the entropy threshold (e.g., entropy > 1.5) can also be used to determine whether the uncertainty is higher than the acceptable range.

[0161] For highly controversial samples (e.g., conflicts in compliance clauses), a dual-model voting mechanism is activated to prioritize the teacher model's results. If the outputs of the teacher and student models differ significantly (e.g., label predictions differ and both have confidence levels > 0.6), the sample will be registered as a "high-priority feedback item" and will be prioritized during the next batch of online training or manually reviewed by experts.

[0162] When the maximum confidence of the student model (Softmax output) is ≥ 0.7, the prediction result y of the student model is usually used directly S As the final output y final,Right now

[0163] yfi na l=yS

[0164] This means that the student model is already sufficiently "confident" about the current sample and there is no need to perform teacher-student fusion or fall back to the teacher model prediction.

[0165] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A large model compression and online incremental learning method for supply chain and e-commerce procurement based on three-order joint distillation, characterized by: The following steps are involved: Collect supply chain compliance documents and purchase order structured data, pre-process the supply chain compliance documents and purchase order structured data to obtain a data set; Build a teacher model and a student model based on the large language model. Both the teacher model and the student model include an encoder, a decoder, and a prediction head. Using the dataset, the teacher model and the student model are subjected to three-stage knowledge distillation. Specifically, the three-stage knowledge distillation performs layered knowledge transfer in the order of encoder distillation stage, decoder distillation stage, and prediction head distillation stage. The encoder includes a self-attention layer and a feed-forward sublayer; The encoder distillation stage is specifically as follows: Split the dataset into training and validation sets, and configure hyperparameters, including learning rate, batch size, maximum number of iterations, and optimizer; After the encoder is trained on the training set, it is validated using the validation set and the F1 score is calculated. If the F1 score fluctuates less than the set range for five consecutive times, the encoder distillation phase has partially converged and the encoder parameters are frozen, including the parameters of all self-attention layers and feedforward sublayers. If the loss value of the validation set does not decrease after the set number of iterations, the learning rate is reduced and gradient clipping is enabled. During encoder training, the intermediate representations of the teacher model are cached locally. The decoder input is appended with a position code or segment vector to distinguish between different data modalities, including text, numerical values, and timestamps; Alternatively, an embedding layer and a multi-layer perceptron connected in sequence are added to the decoder input end to project the numerical field; The decoder distillation stage specifically includes: When processing multimodal data at the decoder side, different modalities are uniformly mapped into an alignable vector space to perform multimodal feature fusion; Configure hyperparameters, including learning rate and batch size; The goals of the decoder distillation phase include attention matrix alignment and semantic distribution alignment. Attention matrix alignment involves keeping the attention matrices generated by each layer of the student model similar to the attention matrices of the corresponding layers of the teacher model, with similarity measured using the bi-norm or mean squared error. Semantic distribution alignment involves aligning the output distribution of the student model with the output distribution of the teacher model on the same input, with the difference between the two measured using KL divergence or mean squared error. When the path inference accuracy or F1 score reaches the threshold, freeze the decoder parameters; otherwise, increase the number of iterations to fine-tune the decoder. pass Project the student model attention to the teacher model space, where ψ(·) represents the projection function from the student model attention representation space to the teacher model attention representation space, and W ψ is a trainable projection matrix used to map the dimension or distribution of the student's attention matrix to a space that is equal or comparable to the teacher's attention. Model for students ′ Attention matrix generated by the layer; Update the projection matrix W ψ , add L2 regularization term when updating the projection matrix, and update the parameters of the projection matrix each time back propagation; The prediction head distillation stage is specifically as follows: Update the low-rank adaptation layer parameters W adapt , train the prediction head, randomly extract historical data from the encoder distillation stage and the decoder distillation stage and mix it into the current training to prevent the model from forgetting the previous task in the final distillation stage; during the training process, each time a preset number of iterations is reached, the F1 score of the compliance clause classification task and the Wasserstein distance of the procurement cost distribution prediction task are evaluated. If there is no improvement in the evaluation indicators for three consecutive times, the prediction head distillation is terminated; Collect new data and determine whether the new data triggers the conditions for incremental learning. If so, use the teacher model and student model obtained by the three-stage knowledge distillation to perform incremental learning to obtain the trained teacher model and student model; The trained teacher model and student model are used for collaborative reasoning. If the prediction confidence of the student model is less than the threshold, the outputs of the teacher model and the student model are weighted and fused to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results; otherwise, the prediction results of the student model are directly used as the final output to obtain the compliance terms classification, procurement cost distribution prediction and supplier priority ranking results.

2. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 1 is characterized in that: The pre-processing of supply chain compliance documents and purchase order structured data specifically includes: Label terms with classification tags, paragraph segmentation and regularization, BPE-based word segmentation, unit unification, field alignment and missing value interpolation.

3. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 1 is characterized in that: The incremental learning conditions are specifically data distribution difference conditions or business indicator monitoring conditions: The data distribution difference condition is: if the category labels of more than a preset percentage of samples in the new data are out of the distribution range of the old data, or the KL divergence difference is greater than a threshold, then incremental learning is triggered; The business indicator monitoring conditions are: the domain entity recall rate is less than the set value for three consecutive batches, or the MSE of the transaction volume forecast exceeds the set threshold by 1.5 times, or the NDCG of the supplier ranking decreases by more than a preset percentage.

4. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 1 is characterized in that: The incremental learning of the teacher model and student model obtained by the three-stage knowledge distillation is specifically as follows: The prediction head distillation strategy is used, but only the low-rank adaptation layer is updated, and the encoder and decoder remain frozen. If fine-tuning of some feature extraction layers is required, the last 1-2 layers of the encoder are unfrozen after updating the low-rank adaptation layer, and the corresponding learning rate is set to 10% of the initial learning rate. New data is collected every week for incremental learning, and the Mini-Epoch strategy is also adopted in the incremental learning stage.

5. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 1 is characterized in that: The loss function of the three-stage knowledge distillation is as follows: Where, represents the loss value of the three-stage knowledge distillation, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT Percentage of total losses; The expression of the semantic distribution distillation loss is as follows: Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, z T and z S Represent the log-odds vectors of the teacher model and the student model respectively; The expression of the topology-aware attention distillation loss is as follows: Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth order corresponding to the student model ′ Attention matrix of the layer Imposed mapping; The expression of the domain reinforcement task loss is as follows: and Where λ i represents the loss weight of the i-th subtask, L task,i represents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task; After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows: Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

6. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 1 is characterized in that: The loss function of the incremental learning is as follows: Where, L 2 total Represents the loss value of incremental learning, L SD represents the semantic distribution distillation loss, α is a weighting coefficient used to control the topology-aware attention distillation loss L TA The proportion of the total loss, β is a weighted coefficient used to control the domain enhancement task loss L DT The proportion of total loss; μ represents the stability coefficient, θ S represents the parameter vector of the student model at the current training stage, Represents the parameter vector saved by the student model at the previous moment or the previous incremental stage; The expression of the semantic distribution distillation loss is as follows: Where, L SD is the semantic distribution distillation loss, τ is the temperature coefficient, KL(·) represents the Kullback-Leibler divergence, σ(·) represents the Softmax function, z T and z S Represent the log-odds vectors of the teacher model and the student model respectively; The expression of the topology-aware attention distillation loss is as follows: Where l represents the layer index in the teacher model, L a is the set of layers in the teacher model selected for distillation, is the attention matrix of the teacher model layer l the mapping or processing operation applied, is the lth order corresponding to the student model ′ Attention matrix of the layer Imposed mapping; The expression of the domain reinforcement task loss is as follows: and Where λ i represents the loss weight of the i-th subtask, L task,i represents the loss function of the i-th subtask, which includes the compliance clause classification task, procurement cost distribution prediction task, and supplier priority sorting task; After each training round, the loss weight of each subtask is automatically updated using the gradient information on the validation set. The formula is as follows: Where, Denotes the validation set loss L val For λ i The partial derivative of is used to measure the error sensitivity of the subtask in the current model.

7. The method for large-scale model compression and online incremental learning in the supply chain and e-commerce procurement field based on third-order joint distillation according to claim 5, characterized in that: The loss of the compliance clause classification task is measured by cross entropy loss. The loss function of the compliance clause classification task is as follows: Where, L task,1 represents the loss of the compliance clause classification task, L CE represents the cross entropy loss, k represents the compliance category index, The loss of the procurement cost distribution prediction task is measured by the Wasserstein distance. The loss function of the procurement cost distribution prediction task is as follows: Where, L task,2 represents the loss of the procurement cost distribution prediction task, L EMD Indicates the cumulative distance between the predicted distribution and the true distribution, i is the cost interval index, CDF pred (i) represents the value of the predicted cumulative distribution function at interval index i, CDF true (i) represents the value of the true cumulative distribution function at interval index p; The loss of the supplier prioritization task is measured by the ranking loss, and the loss function of the supplier prioritization task is as follows: Where, L task,3 is the loss of supplier prioritization task, L Rank represents the ranking loss, s i and s j They represent the ranking scores of supplier i and supplier j predicted by the model, y (i,j) represents the preference labels of suppliers i and j. If i performs better than j, then y (i,h) =+1; If j is better than i, then y (i,j) = -1, pair(i,j) refers to all comparable supplier pairs in the training set.

Images