Method and system for grammar error correction based on continuous knowledge distillation

Abstract

The invention relates to a grammar error correction method and system based on continuous knowledge distillation, and belongs to the field of natural language processing. The method comprises the following steps: taking a distillation knowledge set as input of a teacher model and a student model, performing quantitative scoring on the basis of output information entropy, and dividing the distillation knowledge set into a positive set and a negative set; a distribution-based distillation strategy is constructed, so that the student model learns from the teacher model on the positive set, the student model on the negative set avoids the same error as the teacher model, and knowledge distillation is realized; constructing a knowledge inheritance strategy to realize continuous learning of a student model; total training loss is generated through knowledge distillation, student model continuous learning and student model training loss; and calculating a loss function gradient according to the training total loss, and updating a model parameter weight according to the gradient and a learning rate to obtain a final grammar error correction model. According to the method, the problems of low model training performance and lack of parallel corpora are solved, efficient knowledge transfer is realized, and the grammar error correction model effect is improved.

Description

Technical Field

[0001] The present invention relates to a grammatical error correction method and system based on continuous knowledge distillation, and belongs to the technical field of natural language processing. Background Art

[0002] Grammatical Error Correction (GEC) is a task that automatically detects and corrects text errors. This task includes not only correcting grammatical errors such as missing prepositions and subject-verb agreement mismatches, but also correcting spelling and semantic errors such as typos and word choice errors. In the past decade, this field has made remarkable progress, partly due to a series of shared tasks that have driven the development of rule-based methods, text generation, machine translation, and text summarization systems, which represent the current mainstream state of the art. However, the Grammatical Error Correction (GEC) task faces the problem of a lack of parallel data due to the limited availability of annotated data. High-quality parallel datasets are difficult to obtain, which limits the improvement of GEC model performance. Therefore, exploring how to mine more information from limited parallel corpora and public models is of great research and application value.

[0003] The current mainstream approach is to use a sequence-to-sequence (Seq2Seq) architecture to solve this problem. GEC can also be converted into a sequence labeling task, namely sequence editing (Seq2Edit). These methods are all driven by high-quality parallel data. However, high-quality parallel data for GEC is not widely available. Many studies tend to synthesize parallel data through data augmentation methods, or use GEC system combinations to improve the performance of GEC models. There are also studies that improve model performance by fine-tuning pre-trained models in GEC tasks, but ultimately they all rely on parallel data. Many platforms now publicly disclose models or interfaces obtained through relatively sufficient data training. In view of this, the present invention proposes a grammatical error correction method based on continuous knowledge distillation (CKD), which aims to solve problems such as the lack of parallel data by utilizing the GEC model on the public platform and the error correction capabilities of the large language model. Summary of the invention

[0004] The present invention provides a grammatical error correction method and system based on continuous knowledge distillation, which is used to solve the problems of low model training performance in an environment where parallel corpora are scarce, and lack of parallel corpora due to data privacy, etc., to achieve efficient knowledge transfer and improve the effect of grammatical error correction models.

[0005] The technical solution of the present invention is: a grammatical error correction method based on continuous knowledge distillation, the method comprising:

[0006] Step 1: Use the distilled knowledge set as the input of the teacher model and the student model, and perform quantitative scoring based on the information entropy of the teacher model and the student model output. By judging whether the teacher model score is better than the student model, the distilled knowledge set is divided into positive sets and negative set

[0007] Step 2: Construct a distribution-based distillation strategy to calculate the approximate distribution of the positive set and the negative set between the student model and the teacher model, so that the student model can learn from the teacher model on the positive set and avoid making the same mistakes as the teacher model on the negative set, which is used to realize the knowledge distillation from the student model to the teacher model.

[0008] Step 3: In order to avoid the problem of student model forgetting, a knowledge inheritance strategy is constructed to enable the student model to review the knowledge learned by the student model in the previous time step, so as to achieve continuous learning of the student model.

[0009] Step 4: Distill the knowledge from the student model to the teacher model, continue learning the student model, and add the student model training loss to generate the final total training loss.

[0010] Step 5. Use the obtained total training loss to calculate the gradient of the loss function, and use the optimizer to update the model parameter weights according to the gradient and learning rate to obtain the final grammatical error correction model for grammatical error correction.

[0011] Furthermore, before the Step 1 operation, it also includes: preprocessing the parallel training corpus, verification corpus and test corpus of different sizes for model training, parameter tuning and effect testing.

[0012] Furthermore, in Step 1, the teacher model selects a public grammar correction model, and the student model selects a model trained based on local corpus.

[0013] Furthermore, the Step 1 includes:

[0014] Step 1.1: In order to realize the partition of distilled knowledge set, the distilled knowledge set is used as the input of teacher model and student model respectively, and the entropy of target token is calculated; among them, the information entropy of student model is The calculation formula is as follows:

[0015]

[0016] Information entropy of the teacher model The calculation formula is as follows:

[0017]

[0018] Where X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, and y j represents the current target word, y <j represents the prediction result of incorrect grammatical correction before the jth word in the target sentence, S represents the student model, T represents the teacher model, The entropy value of the predicted probability distribution of the target word by the student model is used to measure the prediction ability of the student model; The entropy value of the predicted probability distribution of the target word by the teacher model is used to measure the prediction ability of the teacher model; P(y|y <j ,X,S) represents the target prediction probability of the student model, P(y|y <j ,X,T) represents the target prediction probability of the teacher model;

[0019] Step 1.2: After obtaining the entropy of the target token, use the cosine theorem to calculate the student model Teacher Model With y label The similarity S CS and T CS , S CS and T CS The calculation formulas are as follows:

[0020]

[0021] in represents the current student model prediction information entropy, represents the current teacher model prediction information entropy, y label Indicates the current label data, S CS Indicates the similarity between the student model prediction result and the label data, T CS Indicates the similarity between the teacher model prediction results and the label data;

[0022] Step 1.3. Compare the similarity S between the student model prediction results and the label data CS The similarity between the teacher model prediction result and the label data T CS , the distilled knowledge set is divided into positive set and negative set; when S CS T CS The value is small, that is, it is divided into a positive set Otherwise, it is classified as a negative set

[0023] Furthermore, the specific steps of Step 2 are:

[0024] Step 2.1: In order to achieve knowledge distillation and improve model performance, the obtained positive set The student and teacher models are input as the source language training set X respectively, and the model output is obtained to calculate the approximate distribution between the teacher and student models; the calculation formula of the approximate distribution KL divergence is as follows:

[0025]

[0026] Among them, S t represents the student model at time step t, represents the teacher model at time step t; KL(|||) represents the calculation of KL divergence, which is used to measure the difference between the teacher model and the student model; represents the prediction result of the teacher model at time step t, P(y j |y <j ,X;S t ) represents the prediction result of the student model at time step t; Indicates that in the positive set The knowledge distillation loss.

[0027] Step 2.2, the same calculation method as Step 2.1, replace the input of the student model and the teacher model with Calculate according to calculate The minimum value between is used to avoid making the same mistake by having the output of the student model and the teacher model approach the target distribution of the deviation:

[0028] In order to make full use of the corpus, calculate The minimum value between is calculated as follows:

[0029]

[0030] in In the negative set Negative feedback loss on, main(,) means taking the minimum value, In the negative set The knowledge distillation loss on , α is a hyperparameter that controls the loss activation;

[0031] Step 2.3, the overall knowledge distillation is the sum of the losses of Step 2.1 and Step 2.2;

[0032]

[0033] Where D trans represents the set of transfers that the student model “learns” from the teacher model, Indicates that in the transitive set D trans The knowledge distillation loss.

[0034] Furthermore, in Step 3, reviewing the knowledge learned by the student model in the previous time step by the student model includes:

[0035] By calculating the approximate distribution between the previous time step and the current time step in the student model, the student model can maximize the retention of the learned prior knowledge.

[0036] Furthermore, in Step 3, in order to achieve continuous learning and solve the problem of student model forgetting, the approximate distribution between the previous time step and the current time step in the student model needs to learn the knowledge learned in the previous time step. The calculation formula is as follows:

[0037]

[0038] in represents the student model at time t-1, represents the continuous learning training loss, Represents the prediction result of the student model at step t-1.

[0039] Furthermore, the Step 4 includes:

[0040] The student model S at time step t t The current teacher model that encodes new knowledge and student models of prior learning that encode new knowledge Sure, It is a model that encodes previously learned knowledge, and the final total training loss is:

[0041]

[0042] Where D train represents the training set of the student model, Indicates that D train and D trans The total training loss on the data; λ represents a hyperparameter used to balance the preference for accepting new knowledge and inheriting old knowledge;

[0043] in, is the standard cross entropy loss, defined as:

[0044]

[0045] in, represents the mth target sentence of length j; X (m) represents the mth source sentence, M represents the total number of sentences in a batch, and J (m) Represents the total length of the mth sentence.

[0046] Furthermore, the Step 5 includes:

[0047] The gradient is calculated based on the total training loss obtained in Step 4, and the student model parameters are updated according to the gradient to obtain the final grammatical error correction model. The loss function gradient calculation is defined as follows:

[0048]

[0049] in Represents the gradient value of the loss function at time step t; Represents the derivative of the loss function with respect to the parameter θ; represents the loss function with parameter θ at time step t.

[0050] The present invention also provides a system for grammatical error correction based on continuous knowledge distillation, the system comprising: a module for executing the above-mentioned grammatical error correction method based on continuous knowledge distillation.

[0051] The beneficial effects of the present invention are:

[0052] 1. The present invention maximizes the performance of the grammatical error correction model from a limited parallel corpus through a continuous knowledge distillation method, and is suitable for grammatical error correction environments where parallel corpora are scarce;

[0053] 2. The method proposed in the present invention uses a continuous knowledge distillation method to distill a publicly available large language model that is fully trained with medium- to large-scale data sets, thereby mining more grammatical error correction capabilities of the large language model;

[0054] 3. Because the public model may contain knowledge that is harmful to the training of the local model, the method in this paper divides the training data set into positive and negative sets. This division is robust to malicious models or distilled knowledge, ensuring efficient and reliable model training;

[0055] 4. In the process of continuous knowledge distillation, due to asynchronous knowledge transfer, the model at the current time step may forget the knowledge learned by the model in the previous step. The knowledge inheritance in the method of the present invention can effectively solve the problem of catastrophic forgetting and ensure the comprehensiveness of the knowledge learned by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the present invention;

[0057] Figure 2 is a data flow diagram in the present invention;

[0058] Figure 3 It is a principle block diagram of the present invention;

[0059] Figure 4 The present invention proposes how to select useful knowledge case diagrams. DETAILED DESCRIPTION

[0060] Example 1: Figure 1-Figure 4 As shown, a grammatical error correction method based on continuous knowledge distillation comprises:

[0061] Step 1: Preprocess the parallel training corpora, verification corpora, and test corpora of different sizes for model training, parameter tuning, and effect testing.

[0062] The present invention selected English, Russian, and Czech parallel corpora for experiments, and divided the scale of the data set into large, medium, and small scales. The present invention was applied to data of different scales, and the degree of development and limitations of large model information were understood by observing the experimental results of corpora of different scales, verifying the applicability of the present invention method to the environment where parallel corpora are scarce.

[0063] Table 1 shows the experimental data information

[0064]

[0065] The distilled knowledge set is used as the input of the teacher model and the student model, and quantitative scoring is performed based on the information entropy of the teacher model and the student model output. By judging whether the teacher model score is better than the student model, the distilled knowledge set is divided into the positive set and negative set

[0066] Furthermore, the Step 1 includes:

[0067] Step 1.1: In order to realize the partition of distilled knowledge set, the distilled knowledge set is used as the input of teacher model (T) and student model (S) respectively, and the entropy of target token is calculated; among them, the information entropy of student model is The calculation formula is as follows:

[0068]

[0069] Information entropy of the teacher model The calculation formula is as follows:

[0070]

[0071] Where X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, and y j represents the current target word, y <j represents the prediction result of incorrect grammatical correction before the jth word in the target sentence, S represents the student model, T represents the teacher model, The entropy value of the predicted probability distribution of the target word by the student model is used to measure the prediction ability of the student model; The entropy value of the predicted probability distribution of the target word by the teacher model is used to measure the prediction ability of the teacher model; P(y|y <j ,X,S) represents the target prediction probability of the student model, P(y|y <j ,X,T) represents the target prediction probability of the teacher model;

[0072] Step 1.2: After obtaining the entropy of the target token, use the cosine theorem to calculate the student model Teacher Model With y label The similarity S CS and T CS , S CS and T CS The calculation formulas are as follows:

[0073]

[0074] in represents the current student model prediction information entropy, represents the current teacher model prediction information entropy, y label Indicates the current label data, S CS Indicates the similarity between the student model prediction result and the label data, T CS Indicates the similarity between the teacher model prediction results and the label data;

[0075] Step 1.3. Compare the similarity S between the student model prediction results and the label data CS The similarity between the teacher model prediction result and the label data T CS , the distilled knowledge set is divided into positive set and negative set; when S CS T CS The value is small, that is, it is divided into a positive set Otherwise, it is classified as a negative set

[0076] Step 2: Construct a distribution-based distillation strategy to calculate the approximate distribution of the positive set and the negative set between the student model and the teacher model, so that the student model can learn from the teacher model on the positive set and avoid making the same mistakes as the teacher model on the negative set, which is used to realize the knowledge distillation from the student model to the teacher model.

[0077] Furthermore, the specific steps of Step 2 are:

[0078] Step 2.1: In order to achieve knowledge distillation and improve model performance, the obtained positive set The student and teacher models are input as the source language training set X respectively, and the model output is obtained to calculate the approximate distribution between the teacher and student models; the calculation formula of the approximate distribution KL divergence is as follows:

[0079]

[0080] Among them, S t represents the student model at time step t, represents the teacher model at time step t; KL(||) represents the calculation of KL divergence, which is used to measure the difference between the teacher model and the student model; represents the prediction result of the teacher model at time step t, P(y j |y <j ,X;S t ) represents the prediction result of the student model at time step t; Indicates that in the positive set The knowledge distillation loss.

[0081] Step 2.2, the same calculation method as Step 2.1, replace the input of the student model and the teacher model with Calculate according to calculate The minimum value between is used to avoid making the same mistake by having the output of the student model and the teacher model approach the target distribution of the deviation:

[0082] In order to make full use of the corpus, calculate The minimum value between is calculated as follows:

[0083]

[0084] in In the negative set Negative feedback loss on, main(,) means taking the minimum value, In the negative set The knowledge distillation loss on , α is a hyperparameter that controls the loss activation;

[0085] Step 2.3, the overall knowledge distillation is the sum of the losses of Step 2.1 and Step 2.2;

[0086]

[0087] Where D trans represents the set of transfers that the student model “learns” from the teacher model, Indicates that in the transitive set D trans The knowledge distillation loss.

[0088] Step 3: In order to avoid the problem of student model forgetting, a knowledge inheritance strategy is constructed to enable the student model to review the knowledge learned by the student model in the previous time step, so as to achieve continuous learning of the student model.

[0089] In Step 3, the student model is made to review the knowledge learned by the student model in the previous time step, including:

[0090] By calculating the approximate distribution between the previous time step and the current time step in the student model, the student model can maximize the retention of the learned prior knowledge.

[0091] In Step 3, in order to achieve continuous learning and solve the problem of student model forgetting, the approximate distribution between the previous time step and the current time step in the student model needs to learn the knowledge learned in the previous time step. The calculation formula is as follows:

[0092]

[0093] in represents the student model at time t-1, represents the continuous learning training loss, Represents the prediction result of the student model at step t-1.

[0094] Step 4: Distill the knowledge from the student model to the teacher model, continuously learn the student model, and then add the student model training loss to generate the final total training loss. Step 4 includes:

[0095] The student model S at time step t t The current teacher model that encodes new knowledge and student models of prior learning that encode new knowledge Sure, It is a model that encodes previously learned knowledge, and the final total training loss is:

[0096]

[0097] Where D trans represents the training set of the student model, Indicates that D trans and D trans The total training loss on the data; λ represents a hyperparameter used to balance the preference for accepting new knowledge and inheriting old knowledge;

[0098] in, is the standard cross entropy loss, defined as:

[0099]

[0100] in, represents the mth target sentence of length j; X (m) represents the mth source sentence, M represents the total number of sentences in a batch, and J (m) Represents the total length of the mth sentence.

[0101] Step 5: Use the obtained total training loss to calculate the loss function gradient, use the optimizer to update the model parameter weights according to the gradient and learning rate, and obtain the final grammatical error correction model for grammatical error correction. Step 5 includes:

[0102] The gradient is calculated based on the total training loss obtained in Step 4, and the student model parameters are updated according to the gradient to obtain the final grammatical error correction model. The loss function gradient calculation is defined as follows:

[0103]

[0104] in Represents the gradient value of the loss function at time step t; Represents the derivative of the loss function with respect to the parameter θ; represents the loss function with parameter θ at time step t.

[0105] The present invention also provides a system for grammatical error correction based on continuous knowledge distillation, the system comprising: a module for executing the above-mentioned grammatical error correction method based on continuous knowledge distillation.

[0106] The distilled knowledge set partitioning module is used to use the distilled knowledge set as the input of the teacher model and the student model, and to perform quantitative scoring based on the information entropy of the teacher model and the student model output. By judging whether the teacher model score is better than the student model, the distilled knowledge set is divided into positive sets. and negative set

[0107] The knowledge distillation module is used to construct a distribution-based distillation strategy, calculate the approximate distribution of the positive set and the negative set between the student model and the teacher model, so that the student model can learn from the teacher model on the positive set and avoid making the same mistakes as the teacher model on the negative set, so as to realize the knowledge distillation from the student model to the teacher model;

[0108] The model continuous learning module is used to build a knowledge inheritance strategy so that the student model can review the knowledge learned by the student model in the previous time step, so as to achieve continuous learning of the student model;

[0109] The module for obtaining the total training loss is used to generate the final total training loss through knowledge distillation from the student model to the teacher model, continuous learning of the student model, and then adding the student model training loss;

[0110] The grammatical error correction module is used to calculate the loss function gradient using the obtained total training loss, and use the optimizer to update the model parameter weights according to the gradient and learning rate to obtain the final grammatical error correction model for grammatical error correction.

[0111] In order to illustrate the grammatical error correction effect of the present invention, the correction texts output by the baseline system and the present invention are compared. Table 2 shows the improvement in correction quality brought by the model; Table 3 shows the performance results on different corpus sizes.

[0112] Table 2 shows the error correction effect.

[0113] Source I went to the bookstore yesterday, bought some books, and read two of them at home. Reference correction text I went to the bookstore yesterday and bought some books. I read two of them when I got home. Baseline correction text I went to the bookstore yesterday and bought some books. I went home and read two of them. The present invention generates error correction text I went to the bookstore yesterday and bought some books. I read two of them when I got home.

[0114] Table 3 shows the F 0.5 Score improvement results

[0115]

[0116] The results show that the method proposed in the present invention can extract more error correction information from the teacher model through continuous knowledge distillation, and can solve the problems of missed correction and phantom correction to a certain extent. The experimental results of data of different scales show that in an environment where parallel corpus resources are scarce, the method proposed in the present invention has a greater improvement in the performance of grammatical error correction. Therefore, the method of the present invention is suitable for an effective grammatical error correction method in an environment with scarce resources.

[0117] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A grammatical error correction method and system based on continuous knowledge distillation, characterized by: The method comprises: Step 1: Use the distilled knowledge set as the input of the teacher model and the student model, and perform quantitative scoring based on the information entropy of the teacher model and the student model output. By judging whether the teacher model score is better than the student model, the distilled knowledge set is divided into a positive set and a negative set. Step 2: Construct a distribution-based distillation strategy to calculate the approximate distribution of the positive set and the negative set between the student model and the teacher model, so that the student model can learn from the teacher model on the positive set and avoid making the same mistakes as the teacher model on the negative set, which is used to realize the knowledge distillation from the student model to the teacher model. Step 3: Construct a knowledge inheritance strategy to enable the student model to review the knowledge learned by the student model in the previous time step, so as to achieve continuous learning of the student model. Step 4: Distill the knowledge from the student model to the teacher model, continue learning the student model, and add the student model training loss to generate the final total training loss. Step 5. Use the obtained total training loss to calculate the gradient of the loss function, and use the optimizer to update the model parameter weights according to the gradient and learning rate to obtain the final grammatical error correction model for grammatical error correction.

2. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: Before Step 1, it also includes: preprocessing parallel training corpora, verification corpora and test corpora of different sizes for model training, parameter tuning and effect testing.

3. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: In Step 1, the teacher model selects a public grammar correction model, and the student model selects a model trained based on local corpus.

4. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: The Step 1 includes: Step 1.1: Use the distilled knowledge set as the input of the teacher model and the student model respectively, and calculate the entropy of the target token; among them, the information entropy of the student model is The calculation formula is as follows: Information entropy of the teacher model The calculation formula is as follows: Where X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, and y j represents the current target word, y <j represents the prediction result of incorrect grammatical correction before the jth word in the target sentence, S represents the student model, T represents the teacher model, The entropy value of the predicted probability distribution of the target word by the student model is used to measure the prediction ability of the student model; The entropy value of the predicted probability distribution of the target word by the teacher model is used to measure the prediction ability of the teacher model; P(y|y <j ,X,S) represents the target prediction probability of the student model, P(y|y <j ,X,T) represents the target prediction probability of the teacher model; Step 1.2: After obtaining the entropy of the target token, use the cosine theorem to calculate the student model Teacher Model With y label The similarity S CS and T CS , S CS and T CS The calculation formulas are as follows: in represents the current student model prediction information entropy, represents the current teacher model prediction information entropy, y label Indicates the current label data, S CS Indicates the similarity between the student model prediction result and the label data, T CS Indicates the similarity between the teacher model prediction results and the label data; Step 1.

3. Compare the similarity S between the student model prediction results and the label data CS The similarity TCS between the teacher model prediction results and the label data divides the distilled knowledge set into positive and negative sets; when S CS T CS The value is small, that is, it is divided into a positive set Otherwise, it is classified as a negative set 5. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: The specific steps of Step 2 are: Step 2.1, according to the obtained positive set The student and teacher models are input as the source language training set X respectively, and the model output is obtained to calculate the approximate distribution between the teacher and student models; the calculation formula of the approximate distribution KL divergence is as follows: Among them, S t represents the student model at time step t, represents the teacher model at time step t; KL(||) represents the calculation of KL divergence, which is used to measure the difference between the teacher model and the student model; represents the prediction result of the teacher model at time step t, P(y j |y <j ,X;S t ) represents the prediction result of the student model at time step t; Indicates that in the positive set Upper knowledge distillation loss; Step 2.2, the same calculation method as Step 2.1, replace the input of the student model and the teacher model with Calculate according to calculate The minimum value between is used to avoid making the same mistake by having the output of the student model and the teacher model approach the target distribution of the deviation: according to calculate The minimum value between is calculated as follows: in In the negative set Negative feedback loss on, main(,) means taking the minimum value, In the negative set The knowledge distillation loss on , α is a hyperparameter that controls the loss activation; Step 2.3, the overall knowledge distillation is the sum of the losses of Step 2.1 and Step 2.2; Where D trans represents the set of transfers that the student model "learns" from the teacher model, Indicates that in the transitive set D trans The knowledge distillation loss.

6. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: In Step 3, the student model is made to review the knowledge learned by the student model in the previous time step, including: By calculating the approximate distribution between the previous time step and the current time step in the student model, the student model can maximize the retention of the learned prior knowledge.

7. The grammatical error correction method based on continuous knowledge distillation according to claim 6, characterized in that: In Step 3, calculating the approximate distribution between the previous time step and the current time step in the student model requires learning the knowledge learned in the previous time step. The calculation formula is as follows: in represents the student model at time t-1, represents the continuous learning training loss, Represents the prediction result of the student model at step t-1.

8. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: The Step 4 includes: The student model S at time step t t The current teacher model that encodes new knowledge and student models of prior learning that encode new knowledge Sure, It is a model that encodes previously learned knowledge, and the final total training loss is: Where D train represents the training set of the student model, Indicates that D train and D trans The total training loss on the data; λ represents a hyperparameter used to balance the preference for accepting new knowledge and inheriting old knowledge; in, is the standard cross entropy loss, defined as: in, represents the mth target sentence of length j; X (m) represents the mth source sentence, M represents the total number of sentences in a batch, and J (m) Represents the total length of the mth sentence.

9. The grammatical error correction method based on continuous knowledge distillation according to claim 1, characterized in that: The Step 5 includes: The gradient is calculated based on the total training loss obtained in Step 4, and the student model parameters are updated according to the gradient to obtain the final grammatical error correction model. The loss function gradient calculation is defined as follows: in Represents the gradient value of the loss function at time step t; Represents the derivative of the loss function with respect to the parameter θ; represents the loss function with parameter θ at time step t.

10. A grammatical error correction system based on continuous knowledge distillation, characterized in that: The system comprises: a module for executing the method according to any one of claims 1 to 9.

Images