A method, system, electronic device, and storage medium for distillation of a gravitational field-constrained model for sequence labeling.

Abstract

This invention discloses a distillation method, system, electronic device, and storage medium for a gravitational field constraint model used in sequence labeling. The method includes the following steps: acquiring a trained teacher model and an untrained student model; training the student model several times, each training session including the following sub-steps: obtaining the distillation loss; obtaining the true loss; calculating the update magnitude of the fine-tuned parameters in the student model to obtain the constraint loss; and using the distillation loss, true loss, and constraint loss for backpropagation update to fine-tune the parameters of the student model. In this invention, by incorporating a constraint mechanism for parameter updates during the fine-tuning process, the model retains as much of the language knowledge learned during pre-training as possible, while also adapting to downstream tasks through fine-tuning training.

Description

Technical Field

[0001] This invention relates, and more particularly, to a method, system, electronic device, and storage medium for distillation of a gravitational field-constrained model for sequence labeling. Background Technology

[0002] Sequence tagging is one of the most fundamental tasks in Natural Language Processing (NLP) and has a wide range of applications. Tasks such as word segmentation, part-of-speech tagging (POS tagging), named entity recognition (NER), keyword extraction, semantic role labeling, and slot filling all essentially fall under the category of sequence tagging. Commonly used methods include Hidden Markov Models (HMMs), Membrane Microarrays (MEMMs), and Continuous Randomization Rendering (CRFs).

[0003] However, existing sequence labeling techniques suffer from the "catastrophic forgetting" problem in fine-tuning of large language models for downstream tasks. This means that when training to adapt to downstream application tasks, the learned language knowledge is catastrophically forgotten, leading to a decline in model performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, electronic device and storage medium for distillation of gravitational field-constrained models for sequence labeling.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A first aspect of the present invention provides a method for distilling a gravitational field-constrained model for sequence labeling, comprising the following steps:

[0007] Obtain trained teacher models and untrained student models;

[0008] The student model is trained several times, with each training session including the following sub-steps:

[0009] The text sequence to be labeled is input into the teacher model and the student model respectively. KL divergence is calculated on the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss.

[0010] The text sequence with classification labels is input into the student model. The cross-information entropy is calculated on the third category prediction probability and classification label output by the student model to obtain the true loss.

[0011] The constraint loss is obtained by calculating the update magnitude of the fine-tuned parameters in the student model.

[0012] Backpropagation updates are performed using distillation loss, true loss, and constraint loss to fine-tune the parameters of the student model.

[0013] Furthermore, the distillation loss is calculated as follows:

[0014]

[0015] In the formula, distill_Loss represents the distillation loss, x represents the text sequence to be labeled, and t i (x) represents the prediction probability of class i in the first category prediction probability output by the teacher model, s i (x) represents the probability of class i predicted in the second category prediction probability output by the student model; ∑ represents the operation on all categories;

[0016] The actual loss is calculated as follows:

[0017]

[0018] In the formula, CE_loss represents the true loss, n represents the total number of predicted categories, p(i) represents the category label, and q(i) represents the second category prediction probability output by the student model.

[0019] Furthermore, the constraint loss is calculated as follows:

[0020]

[0021] In the formula, gravity_loss represents the constraint loss, Δw is the change in each fine-tuned parameter in the student model, i.e., the update magnitude, N represents the number of fine-tuned parameters, and α is the weighting factor of the constraint term, which is a hyperparameter.

[0022] Furthermore, the teacher model is a pre-trained deep neural network model, specifically a large language model that has undergone task-agnostic language model pre-training, or a large model that has been trained specifically for sequence labeling tasks.

[0023] A second aspect of the present invention provides a gravitational field-constrained model distillation system for sequence labeling, comprising:

[0024] Model Acquisition Module: Used to acquire trained teacher models and untrained student models;

[0025] Student Model Training Module: Used to train the student model multiple times. Each training session includes:

[0026] Distillation loss calculation unit: It is used to input the text sequence to be labeled into the teacher model and the student model respectively, and calculate the KL divergence of the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss;

[0027] The true loss calculation unit is used to input the text sequence with classification labels into the student model, and to calculate the cross-information entropy of the third category prediction probability and classification label output by the student model to obtain the true loss.

[0028] Constraint loss calculation unit: used to calculate the update magnitude of the fine-tuned parameters in the student model to obtain the constraint loss;

[0029] Fine-tuning training unit: Used to fine-tune the parameters of the student model by backpropagation updates using distillation loss, true loss, and constraint loss.

[0030] Furthermore, the distillation loss is calculated as follows:

[0031]

[0032] In the formula, distill_Loss represents the distillation loss, x represents the text sequence to be labeled, and t i (x) represents the prediction probability of class i in the first category prediction probability output by the teacher model, s i (x) represents the probability of class i predicted in the second category prediction probability output by the student model; ∑ represents the operation on all categories;

[0033] The actual loss is calculated as follows:

[0034]

[0035] In the formula, CE_loss represents the true loss, n represents the total number of predicted categories, p(i) represents the category label, and q(i) represents the second category prediction probability output by the student model.

[0036] Furthermore, the constraint loss is calculated as follows:

[0037]

[0038] In the formula, gravity_loss represents the constraint loss, Δw is the change in each fine-tuned parameter in the student model, i.e., the update magnitude, N represents the number of fine-tuned parameters, and α is the weighting factor of the constraint term, which is a hyperparameter.

[0039] Furthermore, the teacher model is a pre-trained deep neural network model, specifically a large language model that has undergone task-agnostic language model pre-training, or a large model that has been trained specifically for sequence labeling tasks.

[0040] A third aspect of the present invention provides an electronic device including a storage unit and a processing unit, wherein the storage unit stores computer instructions executable on the processing unit, and the processing unit executes the steps of the described method for distillation of a gravitational field-constrained model for sequence labeling when executing the computer instructions.

[0041] In a fourth aspect, the present invention provides a storage medium storing computer instructions which, when executed, perform the steps of the described method for distilling a gravitational field-constrained model for sequence labeling.

[0042] The beneficial effects of this invention are:

[0043] In an exemplary embodiment of the present invention, by adding a constraint mechanism for parameter updates during the fine-tuning process, the model can retain the language knowledge learned in pre-training as much as possible during the fine-tuning process, while also being able to adapt to downstream tasks through fine-tuning training. Attached Figure Description

[0044] Figure 1 This is a flowchart of a gravitational field-constrained model distillation method for sequence labeling, provided in an exemplary embodiment of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0047] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] See Figure 1 , Figure 1 The flowchart illustrates a gravitational field constraint model distillation method for sequence labeling according to an exemplary embodiment of the present invention, including the following steps:

[0050] Obtain trained teacher models and untrained student models;

[0051] The student model is trained several times, with each training session including the following sub-steps:

[0052] The text sequence to be labeled is input into the teacher model and the student model respectively. KL divergence is calculated on the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss.

[0053] The text sequence with classification labels is input into the student model. The cross-information entropy is calculated on the third category prediction probability and classification label output by the student model to obtain the true loss.

[0054] The constraint loss is obtained by calculating the update magnitude of the fine-tuned parameters in the student model.

[0055] Backpropagation updates are performed using distillation loss, true loss, and constraint loss to fine-tune the parameters of the student model.

[0056] Specifically, in this exemplary embodiment, training the student model includes:

[0057] (1) Distillation of language knowledge

[0058] The first step is knowledge distillation, such as Figure 1 On the left, the language knowledge from the pre-trained large model (teacher model) is distilled into the smaller student model. Both models receive the same input, and the probability distribution of the predictions from the teacher model's output is shown in Figure 1. T(Probability of prediction for the first category) and probability distribution of predictions output by the student model. S1 (Second category prediction probability) KL divergence is calculated, and the KL divergence values ​​of the two distributions are used as the loss function for distillation learning, i.e., distill_Loss.

[0059] In a preferred exemplary embodiment, the distillation loss is calculated as follows:

[0060]

[0061] In the formula, distill_Loss represents the distillation loss, x represents the text sequence to be labeled, and t i (x) represents the prediction probability of class i in the first category prediction probability output by the teacher model, s i (x) represents the probability of predicting class i in the second category prediction probability output by the student model; Σ represents the operation on all categories.

[0062] The entire calculation involves calculating the distillation loss for a single text sequence element, while the loss for labeling and classifying all sequence elements (words) is the sum of the distillation losses for each sequence element and the average of those losses.

[0063] (2) Fine-tuning under gravitational constraints

[0064] The second step is to fine-tune the training loss, such as... Figure 1 As shown on the right, supervised training is performed on a dataset with human annotations (text sequences with classification labels).

[0065] (2-1) Use cross-information entropy to calculate the loss CE_loss between the model's prediction of sequence label classification and the true label.

[0066] In a preferred exemplary embodiment, the actual loss is calculated as follows:

[0067]

[0068] In the formula, CE_loss represents the true loss, n represents the total number of predicted categories, p(i) represents the category label, and q(i) represents the second category prediction probability output by the student model. CE_loss is used to calculate the distribution difference between the probability distribution predicted by the model and the true label, and is a commonly used loss function in classification tasks.

[0069] (2-2) Calculate the update magnitude of the fine-tuned parameters in the student model to obtain the constraint loss.

[0070] In a preferred exemplary embodiment, the constraint loss is calculated as follows:

[0071]

[0072] In the formula, gravity_loss represents the constraint loss, Δw is the change in each fine-tuned parameter in the student model, i.e., the update magnitude, N represents the number of fine-tuned parameters, and α is the weighting factor of the constraint term, which is a hyperparameter.

[0073] This exemplary embodiment effectively addresses the "catastrophic forgetting" problem in fine-tuning of large language models for downstream tasks (where learned language knowledge is catastrophically forgotten during training to adapt to downstream tasks, leading to a decline in model performance). By incorporating a constraint mechanism for parameter updates during fine-tuning, the model retains as much of the language knowledge learned during pre-training as possible, while simultaneously adapting to downstream tasks through fine-tuning training.

[0074] It's important to note that when we add a gravity constraint term for the fine-tuning weights to the total loss (i.e., the total loss function, which includes the weighted sum of all loss functions), during model training, if the reduction in CE_loss caused by the weight parameter update Δw is greater than the increase in loss caused by the gravity constraint term, then this parameter will be updated. Conversely, if the reduction in CE_loss caused by the weight parameter update Δw is less than the increase in loss caused by the gravity constraint term, backpropagation dominated by the total loss will not update this parameter. Therefore, during fine-tuning, our model ensures both the retention of knowledge from the teacher model and appropriate parameter adjustments to achieve the desired fine-tuning effect.

[0075] Meanwhile, the parameters fine-tuned in the student model mainly consist of the parameters in the parameter matrix that makes up each layer (e.g., pre-embedding layer, hidden layer, output layer, etc.). Furthermore, the fine-tuned parameters typically do not exceed -1 to 1, and due to their small dimensions, the update magnitude of the constraint loss is also small. Moreover, the proportions of the three losses can be selected according to actual needs.

[0076] More preferably, in an exemplary embodiment, the teacher model is a pre-trained deep neural network model, specifically a large language model pre-trained with a task-agnostic language model, or a large model trained specifically for sequence labeling tasks.

[0077] Specifically, in an exemplary embodiment, the teacher model can be either a large language model pre-trained with a task-independent LanguageModel (e.g., hfl / Chinese-macbert-large, bigscience / bloom-560m, etc., with more than 500M model parameters and model files typically exceeding 1G), or a large model specifically trained for sequence labeling tasks (e.g., dslim / bert-large-NER, Jean-Baptiste / roberta-large-ner-english, etc., with more than 500M model parameters and model files typically exceeding 1G).

[0078] Correspondingly, the student model can be a small model consisting of a shallow Transformer encoder layer, with parameters randomly initialized before training (e.g., a 5-layer Transformer encoder layer, approximately 50M of parameters, and a model file of approximately 100M).

[0079] On our manually labeled test set, the fine-tuning based on the "gravity constraint" method improved the sequence label prediction accuracy from 79.3% to 94.7% compared to ordinary unconstrained fine-tuning. This represents an absolute accuracy improvement of 25.4%. The experimental procedure is as follows:

[0080] In constructing a knowledge graph for the shipbuilding domain, we labeled 5,500 entity recognition samples. We randomly split these samples into a training set of 3,500 samples and a test set of 2,000 samples. Initially, without using gravity constraints, we achieved a sequence labeling accuracy of 79.3% for entity recognition on the test set through distillation learning on a large model and fine-tuning on the training set. Later, through analysis of the model's mispredicted samples, we proposed using gravity constraints to mitigate the "knowledge forgetting phenomenon" during downstream fine-tuning. Tests on the test set show that the new method is more effective, achieving a sequence labeling accuracy of 94.7% for entity recognition on the test set.

[0081] Having the same inventive concept as the exemplary embodiments described above, another exemplary embodiment of the present invention provides a gravitational field-constrained model distillation system for sequence labeling, comprising:

[0082] Model Acquisition Module: Used to acquire trained teacher models and untrained student models;

[0083] Student Model Training Module: Used to train the student model multiple times. Each training session includes:

[0084] Distillation loss calculation unit: It is used to input the text sequence to be labeled into the teacher model and the student model respectively, and calculate the KL divergence of the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss;

[0085] The true loss calculation unit is used to input the text sequence with classification labels into the student model, and to calculate the cross-information entropy of the third category prediction probability and classification label output by the student model to obtain the true loss.

[0086] Constraint loss calculation unit: used to calculate the update magnitude of the fine-tuned parameters in the student model to obtain the constraint loss;

[0087] Fine-tuning training unit: Used to fine-tune the parameters of the student model by backpropagation updates using distillation loss, true loss, and constraint loss.

[0088] Correspondingly, the distillation loss is calculated as follows:

[0089]

[0090] In the formula, distill_Loss represents the distillation loss, x represents the text sequence to be labeled, and t i (x) represents the prediction probability of class i in the first category prediction probability output by the teacher model, s i (x) represents the probability of class i predicted in the second category prediction probability output by the student model; ∑ represents the operation on all categories;

[0091] The actual loss is calculated as follows:

[0092]

[0093] In the formula, CE_loss represents the true loss, n represents the total number of predicted categories, p(i) represents the category label, and q(i) represents the second category prediction probability output by the student model.

[0094] Correspondingly, the constraint loss is calculated as follows:

[0095]

[0096] In the formula, gravity_loss represents the constraint loss, Δw is the change in each fine-tuned parameter in the student model, i.e., the update magnitude, N represents the number of fine-tuned parameters, and α is the weighting factor of the constraint term, which is a hyperparameter.

[0097] Correspondingly, the teacher model is a pre-trained deep neural network model, specifically a large language model that has been pre-trained with a task-agnostic language model, or a large model that has been trained specifically for sequence labeling tasks.

[0098] Having the same inventive concept as the exemplary embodiments described above, another exemplary embodiment of the present invention provides an electronic device including a storage unit and a processing unit. The storage unit stores computer instructions that can be executed on the processing unit. When the processing unit executes the computer instructions, it performs the steps of the described method for distillation of a gravitational field constraint model for sequence labeling.

[0099] Electronic devices are manifested in the form of general-purpose computing devices. Components of electronic devices may include, but are not limited to: at least one processing unit, at least one storage unit, and a bus connecting different system components (including storage units and processing units).

[0100] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section above, based on various exemplary embodiments of the present invention. For example, the processing unit can perform actions such as... Figure 1 The method shown in the figure.

[0101] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0102] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0103] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0104] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0105] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to this exemplary embodiment.

[0106] Having the same inventive concept as the above exemplary embodiments, another exemplary embodiment of the present invention provides a storage medium storing computer instructions, which, when executed, perform the steps of the described method for distillation of a gravitational field-constrained model for sequence labeling.

[0107] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0109] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0110] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0111] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0112] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A distillation method for a gravitational field-constrained model for sequence labeling, characterized in that: Includes the following steps: Obtain trained teacher models and untrained student models; The student model is trained several times, with each training session including the following sub-steps: The text sequence to be labeled is input into the teacher model and the student model respectively. KL divergence is calculated on the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss. The text sequence with classification labels is input into the student model. The cross-information entropy is calculated on the third category prediction probability and classification label output by the student model to obtain the true loss. The constraint loss is obtained by calculating the update magnitude of the fine-tuned parameters in the student model. Backpropagation updates are performed using distillation loss, true loss, and constraint loss to fine-tune the parameters of the student model. The distillation loss is calculated as follows: ； In the formula, This is expressed as distillation loss. This represents the text sequence to be annotated. This indicates that the first category prediction probability output by the teacher model is the predicted category. The predicted probability, This indicates that the predicted class in the second category prediction probability output by the student model is the class. The class probability; This indicates that the operation is performed on all categories; The actual loss is calculated as follows: ； In the formula, Let n represent the actual loss, and n represent the total number of predicted categories. Indicates category labels, This represents the second-class prediction probability output by the student model; The constraint loss is calculated as follows: ； In the formula, Indicates constraint loss, Let N be the change in each fine-tuned parameter in the student model, i.e., the update magnitude, and let N represent the number of fine-tuned parameters. It is the weighting factor of the constraint term, which is a hyperparameter.

2. The method for distilling a gravitational field-constrained model for sequence labeling according to claim 1, characterized in that: The teacher model is a pre-trained deep neural network model, specifically a large language model that has been pre-trained with a task-agnostic language model, or a large model that has been trained specifically for sequence labeling tasks.

3. A distillation system for a gravitational field-constrained model used for sequence labeling, characterized in that: include: Model Acquisition Module: Used to acquire trained teacher models and untrained student models; Student Model Training Module: Used to train the student model multiple times. Each training session includes: Distillation loss calculation unit: It is used to input the text sequence to be labeled into the teacher model and the student model respectively, and calculate the KL divergence of the first category prediction probability output by the teacher model and the second category prediction probability output by the student model to obtain the distillation loss; The true loss calculation unit is used to input the text sequence with classification labels into the student model, and to calculate the cross-information entropy of the third category prediction probability and classification label output by the student model to obtain the true loss. Constraint loss calculation unit: used to calculate the update magnitude of the fine-tuned parameters in the student model to obtain the constraint loss; Fine-tuning training unit: used to fine-tune the parameters of the student model by backpropagation updates using distillation loss, true loss, and constraint loss; The distillation loss is calculated as follows: ； In the formula, This is expressed as distillation loss. This represents the text sequence to be annotated. This indicates that the first category prediction probability output by the teacher model is the predicted category. The predicted probability, This indicates that the predicted class in the second category prediction probability output by the student model is the class. The class probability; This indicates that the operation is performed on all categories; The actual loss is calculated as follows: ； In the formula, Let n represent the actual loss, and n represent the total number of predicted categories. Indicates category labels, This represents the second-class prediction probability output by the student model; The constraint loss is calculated as follows: ； In the formula, Indicates constraint loss, Let N be the change in each fine-tuned parameter in the student model, i.e., the update magnitude, and let N represent the number of fine-tuned parameters. It is the weighting factor of the constraint term, which is a hyperparameter.

4. The gravitational field-constrained model distillation system for sequence labeling according to claim 3, characterized in that: The teacher model is a pre-trained deep neural network model, specifically a large language model that has been pre-trained with a task-agnostic language model, or a large model that has been trained specifically for sequence labeling tasks.

5. An electronic device comprising a storage unit and a processing unit, wherein the storage unit stores computer instructions executable on the processing unit, characterized in that: When the processing unit executes the computer instructions, it performs the steps of a gravitational field constraint model distillation method for sequence labeling as described in claim 1 or 2.

6. A storage medium storing computer instructions thereon, characterized in that: When the computer instructions are executed, they perform the steps of a distillation method for a gravitational field-constrained model for sequence labeling as described in claim 1 or 2.

Images