Grammar error correction method and system based on continuous knowledge distillation

Abstract

The present application relates to a grammar error correction method and system based on continuous knowledge distillation, belonging to the field of natural language processing. The present application takes the distilled knowledge set as the input of the teacher model and the student model, quantifies the score based on the output information entropy, and divides the distilled knowledge set into positive set and negative set; constructs a distribution-based distillation strategy, so that the student model learns from the teacher model on the positive set, and the student model avoids making the same mistakes as the teacher model on the negative set, realizing knowledge distillation; constructs a knowledge inheritance strategy to realize continuous learning of the student model; generates a total training loss through knowledge distillation, continuous learning of the student model and training loss of the student model; calculates the loss function gradient according to the total training loss, updates the model parameter weight according to the gradient and the learning rate, and obtains the final grammar error correction model. The present application solves the problems of low model training performance and lack of parallel corpus, realizes efficient knowledge transfer, and improves the effect of grammar correction model.

Description

TECHNICAL FIELD

[0001] The present application relates to a grammar error correction method and system based on continuous knowledge distillation, and belongs to the technical field of natural language processing. BACKGROUND

[0002] Grammar correction (GEC) is a task of automatically detecting and correcting errors in text. This task not only includes correcting grammatical errors such as preposition omissions and subject-verb agreement mismatches, but also includes correcting spelling errors and word selection errors such as spelling and semantic errors. In the past decade, significant progress has been made in this field, partly due to a series of shared tasks that have driven the development of rule-based methods, text generation, machine translation, and text summarization systems, which represent the current state of the art. However, the grammar error correction (GEC) task faces a lack of parallel data due to the limited availability of annotated data. High-quality parallel data sets are difficult to obtain, limiting the improvement of GEC model performance. Therefore, exploring how to extract more information from limited parallel corpora and public models has very important research and application value.

[0003] The current mainstream approach is to use a sequence-to-sequence (Seq2Seq) architecture to solve it. GEC can also be converted into a sequence labeling task, i.e. sequence editing (Seq2Edit), and these methods are all based on high-quality parallel data. However, high-quality parallel data for GEC is not widely available, and many studies tend to synthesize parallel data through data augmentation methods or use GEC system combinations to improve the performance of GEC models, or fine-tune pre-trained models to improve model performance in the GEC task, but ultimately rely on parallel data. Now many platforms publicly disclose models or interfaces trained on more sufficient data. In view of this, the present application proposes a grammar error correction method based on continuous knowledge distillation (CKD), which aims to solve the problem of lack of parallel data by utilizing GEC models on public platforms and the correction capabilities of large language models. SUMMARY

[0004] The present application provides a grammar error correction method and system based on continuous knowledge distillation to solve the problem of low model training performance in a parallel corpus scarce environment, lack of parallel corpus due to data privacy, achieve efficient knowledge transfer, and improve the effect of grammar correction models.

[0005] The technical solution of the present application is: a grammar error correction method based on continuous knowledge distillation, the method comprising:

[0006] Step 1: Use the distilled knowledge set as input to the teacher model and student model. Quantitatively score the results based on the information entropy of the outputs of the teacher and student models. Divide the distilled knowledge set into positive sets based on whether the teacher model's score is superior to the student model's. and negative set

[0007] Step 2: Construct a distribution-based distillation strategy, calculate the convergent distributions of the positive and negative sets between the student model and the teacher model respectively, so that the student model learns from the teacher model on the positive set and avoids making the same mistakes as the teacher model on the negative set, which is used to realize the knowledge distillation from the student model to the teacher model;

[0008] Step 3: In order to avoid the problem of student model forgetting, a knowledge inheritance strategy is constructed to enable the student model to review the knowledge learned by the student model in the previous time step, so as to realize the continuous learning of the student model.

[0009] Step 4: Through knowledge distillation from the student model to the teacher model, continuous learning of the student model, and the addition of training loss from the student model, the final total training loss is generated.

[0010] Step 5: Calculate the gradient of the loss function using the obtained total training loss, and use the optimizer to update the model parameter weights based on the gradient and learning rate to obtain the final syntax error correction model, which is used for syntax error correction.

[0011] Furthermore, before Step 1, the process includes preprocessing parallel training corpora, validation corpora, and test corpora of different sizes for model training, parameter tuning, and performance testing.

[0012] Furthermore, in Step 1, the teacher model is selected from publicly available grammar correction models, and the student model is selected from models trained on local corpora.

[0013] Further, Step 1 includes:

[0014] Step 1.1: To achieve the distillation knowledge set partitioning, the distillation knowledge set is used as input to the teacher model and student model respectively, and the entropy of the target token is calculated; among them, the information entropy of the student model is... The calculation formula is as follows:

[0015]

[0016] Information entropy of the teacher model The calculation formula is as follows:

[0017]

[0018] Where X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, and y j Indicates the current target word, y <j Let S represent the grammar correction prediction results before the j-th word in the target sentence, and let T represent the teacher model. The entropy value represents the probability distribution of the student model's prediction of the target word, and is used to measure the predictive ability of the student model. The entropy value represents the probability distribution of the teacher model's predictions for the target word, used to measure the predictive ability of the teacher model; P(y|y <j (x,s) represents the student model target prediction probability, P(y|y) <j (X,T) represents the teacher model's target prediction probability;

[0019] Step 1.2: After obtaining the entropy of the target token, the student model is calculated using the law of cosines. Teacher Model With y label Similarity S CS and T CS S CS and T CS The calculation formulas are as follows:

[0020]

[0021] in This represents the current student model prediction information entropy. y represents the current teacher model prediction information entropy. label S represents the current label data. CS T represents the similarity between the student model's predictions and the labeled data. CS This indicates the similarity between the teacher model's predictions and the labeled data;

[0022] Step 1.3: Compare the similarity S between the student model prediction results and the label data. CS The similarity T between the teacher model prediction results and the labeled data CS The distillation knowledge set is divided into a positive set and a negative set; when S CS T CS If the value is small, it is classified as a positive set. Otherwise, it is divided into negative sets.

[0023] Furthermore, the specific steps of Step 2 are as follows:

[0024] Step 2.1: To achieve knowledge distillation and improve model performance, based on the obtained positive set... respectively as the source language training set X input student, teacher model, get model output calculation between the approach distribution of teacher, student model; The calculation formula of the approach distribution KL divergence is as follows:

[0025]

[0026] Among them, S t represents the student model at time t step, represents the teacher model at time t step; KL (||) represents the calculation of KL divergence, which is used to measure the difference between the teacher model and the student model; represents the prediction result of the teacher model at time t step, P (y j |y <j ,X; S t ) represents the prediction result of the student model at time t step; represents the knowledge distillation loss on the positive set .

[0027] Step2.2, the same as Step2.1 calculation method, the input of student model and teacher model is replaced with Calculate According to Calculate The minimum value between them is used to push the target distribution of the difference between the output approach distribution of the student model and the teacher model to avoid the student model from making the same mistake:

[0028] In order to make full use of the corpus, according to Calculate The calculation formula of the minimum value between them is as follows:

[0029]

[0030] Among them represents the negative feedback loss on the negative set , main (,) represents the minimum value, represents the knowledge distillation loss on the negative set , and alpha is a hyperparameter for controlling the loss activation;

[0031] Step2.3, the overall knowledge distillation is the sum of the losses of Step2.1 and Step2.2;

[0032]

[0033] Among them, D trans represents the transmission set of the student model to the teacher model "learning", represents the knowledge distillation loss on the transmission set D trans .

[0034] Further, the Step 3 includes:

[0035] The student model is maximally retained to learn the prior knowledge by calculating the approaching distribution between the last time step and the current time step in the student model.

[0036] Further, in order to realize continuous learning and solve the problem of forgetting in the student model, the approaching distribution between the last time step and the current time step in the student model needs to learn the knowledge learned in the last time step, and the calculation formula is as follows:

[0037]

[0038] Wherein represents the student model at time t-1 step, represents the continuous learning training loss, represents the prediction result of the student model at t-1 step.

[0039] Further, the Step 4 includes:

[0040] The student model S at time step t is determined by the current teacher model t encoding new knowledge and the previously learned student model encoding new knowledge is a model encoding previously learned knowledge, and the final training total loss is:

[0041]

[0042] Wherein D train represents the training set of the student model, represents the training total loss on D train and D trans data; λ represents a hyperparameter used to balance the preference of accepting new knowledge and inheriting old knowledge;

[0043] Wherein, is the standard cross-entropy loss, defined as:

[0044]

[0045] Wherein, represents the mth target sentence with length j; X (m) represents the mth source sentence, M represents the total number of sentences in a batch, J (m) represents the total length of the mth sentence.

[0046] Further, the Step 5 includes:

[0047] The gradient is calculated according to the total loss of the training obtained according to Step 4, the student model parameters are updated according to the gradient, and the final grammar error correction model is obtained, and the loss function gradient calculation is defined as follows:

[0048]

[0049] Wherein represents the loss function gradient value at time t step; represents the derivative of the loss function with respect to the parameter θ; represents the loss function of the parameter θ at time t step.

[0050] The application also provides a grammar error correction system based on continuous knowledge distillation, which comprises a module for executing the grammar error correction method based on continuous knowledge distillation.

[0051] The application has the following beneficial effects:

[0052] 1. The application maximizes the performance of the grammar error correction model through the continuous knowledge distillation method from the limited parallel corpus, and is suitable for grammar correction environments with scarce parallel corpus;

[0053] 2. The method disclosed in the application uses the continuous knowledge distillation method to distill the large language model fully trained by the medium-large data set, which can mine more grammar correction capabilities of the large language model;

[0054] 3. Because the public model may contain harmful knowledge for training the local model, the training data set is divided into positive and negative sets in the method, which has a certain robustness to malicious models or distilled knowledge, ensuring the efficiency and reliability of model training;

[0055] 4. In the continuous knowledge distillation process, the model at the current time step may forget the knowledge learned by the model at the previous step due to asynchronous knowledge transfer, and the knowledge inheritance in the method can effectively solve the problem of catastrophic forgetting and ensure the comprehensiveness of the knowledge learned by the model. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the flowchart in the application;

[0057] Figure 2 is the data flow diagram in the application;

[0058] Figure 3 is the principle block diagram of the application;

[0059] Figure 4 is a diagram showing how to select useful knowledge cases according to the application. DETAILED DESCRIPTION

[0060] Embodiment 1: As shown, the method for correcting grammatical errors based on continuous knowledge distillation includes: Figures 1-4

[0061] Step1, pre-process parallel training corpus, validation corpus and test corpus of different scales for model training, parameter tuning and effect testing.

[0062] The present application selects English, Russian and Czech parallel corpus for experiment, and divides the scale of data set into large, medium and small. The present application is applied to different scale data, and the information development degree and limitation of large model are understood by observing the experimental results of different scale corpus, and the applicability of the present application method to parallel corpus scarce environment is verified.

[0063] Table 1 shows experimental data information

[0064]

[0065] The distilled knowledge set is input to the teacher model and the student model, and the information entropy of the output of the teacher model and the student model is quantified and scored, and the distilled knowledge set is divided into positive set and negative set

[0066] Further, the Step1 includes:

[0067] Step1.1, in order to realize the division of distilled knowledge set, the distilled knowledge set is input to the teacher model (T) and the student model (S) respectively, and the entropy of the target token is calculated; wherein the information entropy of the student model The calculation formula is as follows:

[0068]

[0069] The calculation formula of the information entropy of the teacher model is as follows:

[0070]

[0071] Wherein X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, y j represents the current target word, y <j represents the error grammar correction prediction result before the jth word in the target sentence, S represents the student model, T represents the teacher model, represents the entropy value of the prediction probability distribution of the student model for the target word, which is used to measure the prediction ability of the student model; ​​The entropy value represents the probability distribution of the teacher model's predictions for the target word, used to measure the predictive ability of the teacher model; P(y|y <j (x,s) represents the student model target prediction probability, P(y|y) <j (X,T) represents the teacher model's target prediction probability;

[0072] Step 1.2: After obtaining the entropy of the target token, the student model is calculated using the law of cosines. Teacher Model With y label Similarity S CS and T CS S CS and T CS The calculation formulas are as follows:

[0073]

[0074] in This represents the current student model prediction information entropy. y represents the current teacher model prediction information entropy. label S represents the current label data. CS T represents the similarity between the student model's predictions and the labeled data. CS This indicates the similarity between the teacher model's predictions and the labeled data;

[0075] Step 1.3: Compare the similarity S between the student model prediction results and the label data. CS The similarity T between the teacher model prediction results and the labeled data CS The distillation knowledge set is divided into a positive set and a negative set; when S CS T CS If the value is small, it is classified as a positive set. Otherwise, it is divided into negative sets.

[0076] Step 2: Construct a distribution-based distillation strategy, calculate the convergent distributions of the positive and negative sets between the student model and the teacher model respectively, so that the student model learns from the teacher model on the positive set and avoids making the same mistakes as the teacher model on the negative set, which is used to realize the knowledge distillation from the student model to the teacher model;

[0077] Furthermore, the specific steps of Step 2 are as follows:

[0078] Step 2.1: To achieve knowledge distillation and improve model performance, based on the obtained positive set... The student and teacher models are input as the source language training set X, respectively. The convergence distribution between the teacher and student models is calculated by obtaining the model outputs. The formula for calculating the KL divergence of the convergence distribution is as follows:

[0079]

[0080] where S t denotes the student model at time step t, denotes the teacher model at time step t; KL(||) denotes the computation of KL divergence, which is used to measure the difference between the teacher model and the student model; denotes the prediction result of the teacher model at time t, P(y j |y <j ,X; S t denotes the prediction result of the student model at time t; denotes the knowledge distillation loss on the positive set .

[0081] Step 2.2, the same as Step 2.1 calculation method, the input of the student model and the teacher model is replaced by to calculate According to to calculate the minimum value between them, which is used to avoid the student model from making the same mistakes by approaching the target distribution of the output difference of the student model and the teacher model:

[0082] In order to make full use of the corpus, according to to calculate the minimum value between them, the calculation formula is as follows:

[0083]

[0084] where denotes the negative feedback loss on the negative set , main(,) denotes the minimum value, denotes the knowledge distillation loss on the negative set , and a is a hyperparameter for controlling the loss activation;

[0085] Step 2.3, the overall knowledge distillation is the sum of the losses of Step 2.1 and Step 2.2;

[0086]

[0087] where D trans denotes the transfer set of the student model to the teacher model, denotes the knowledge distillation loss on the transfer set D trans .

[0088] Step3, in order to avoid the student model forgetting problem, a knowledge inheritance strategy is constructed to make the student model review the knowledge learned by the student model at the previous time step, so as to realize the continuous learning of the student model;

[0089] In Step3, the student model reviews the knowledge learned by the student model at the previous time step, which includes:

[0090] By calculating the approaching distribution between the previous time step and the current time step in the student model, the learned prior knowledge is maximally retained.

[0091] In Step3, in order to realize continuous learning and solve the forgetting problem of the student model, the approaching distribution between the previous time step and the current time step in the student model needs to learn the knowledge learned at the previous time step, and the calculation formula is as follows:

[0092]

[0093] Wherein S t-1 represents the student model at time t-1 step, L represents the continuous learning training loss, S t-1 represents the prediction result of the student model at t-1 step.

[0094] Step4, the final training total loss is generated by the knowledge distillation of the student model to the teacher model, the continuous learning of the student model and the training loss of the student model; Step4 includes:

[0095] The student model S at time step t t The current teacher model encoding new knowledge And the previously learned student model encoding new knowledge Is determined, The model encoding the previously learned knowledge, and the final training total loss is:

[0096]

[0097] Wherein D trans Represents the training set of the student model, Represents the training total loss on D trans And D trans Data; Lambda represents a hyperparameter used to balance the preference of accepting new knowledge and inheriting old knowledge;

[0098] Wherein, Is the standard cross entropy loss, defined as:

[0099]

[0100] Wherein, denotes the mth target sentence with length j; X (m) denotes the mth source sentence, M denotes the total number of sentences in a batch, J (m) denotes the total length of the mth sentence.

[0101] Step 5, using the obtained training total loss to calculate the loss function gradient, using the optimizer to update the model parameter weight according to the gradient and the learning rate, obtaining the final grammar error correction model for grammar error correction. The Step 5 comprises:

[0102] According to the training total loss obtained in Step 4, the gradient is calculated, and the student model parameters are updated according to the gradient, and the final grammar error correction model is obtained. The loss function gradient calculation is defined as follows:

[0103]

[0104] Wherein denotes the loss function gradient value at time t step; denotes the derivative of the loss function with respect to the parameter θ; denotes the loss function of the parameter θ at time t step.

[0105] The present application also provides a grammar error correction system based on continuous knowledge distillation, which comprises a module for executing the above grammar error correction method based on continuous knowledge distillation.

[0106] The distillation knowledge set division module is used to divide the distillation knowledge set as the input of the teacher model and the student model, to quantitatively score based on the information entropy of the output of the teacher model and the student model, and to divide the distillation knowledge set into a positive set and a negative set

[0107] The knowledge distillation module is used to construct a distribution-based distillation strategy, to calculate the approaching distribution between the student model and the teacher model for the positive set and the negative set respectively, to make the student model learn from the teacher model on the positive set, and to make the student model avoid the same mistakes as the teacher model on the negative set, so as to realize the knowledge distillation of the student model to the teacher model.

[0108] The model continuous learning module is used to construct a knowledge inheritance strategy, to make the student model review the knowledge learned by the student model at the previous time step, and to realize the continuous learning of the student model.

[0109] The training total loss acquisition module is used to generate the final training total loss by the knowledge distillation of the student model to the teacher model, the continuous learning of the student model, and the training loss of the student model.

[0110] The syntax error correction module is configured to calculate a loss function gradient using the obtained training total loss, update model parameter weights according to the gradient and a learning rate by using an optimizer, and obtain a final syntax error correction model for syntax error correction.

[0111] To illustrate the syntax error correction effect of the present application, the baseline system and the error correction text output by the present application are compared, and Table 2 shows the correction quality improvement brought by the model; and Table 3 shows the performance results on different corpus scales.

[0112] Table 2 is the error correction effect

[0113] Source text I went to the bookstore yesterday and bought several books. I read two books at home. Reference correction text I went to the bookstore yesterday and bought several books. I read two books after coming home. Baseline correction text I went to the bookstore yesterday and bought several books. I read two books at home. The invention generates correction text I went to the bookstore yesterday and bought several books. I read two books after coming home.

[0114] Table 3 is the F 0.5 score improvement results

[0115]

[0116] The results show that the method proposed in the present application can extract more error correction information from the teacher model through continuous knowledge distillation, and can solve the problems of missed correction and phantom correction to a certain extent. The experimental results on different scales of data show that in the environment of scarce parallel corpus resources, the method proposed in the present application can greatly improve the performance of syntax error correction, and therefore the method of the present application is suitable for an effective syntax error correction method in a scarce resource environment.

[0117] The specific embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above-mentioned embodiments, and various changes can be made within the knowledge possessed by those skilled in the art without departing from the purpose of the present application.

Claims

1. A method for correcting syntax errors based on continuous knowledge distillation, characterized in that: The method comprises: Step 1, taking the distilled knowledge set as the input of the teacher model and the student model, quantifying the score based on the information entropy of the output of the teacher model and the student model, and dividing the distilled knowledge set into a positive set and a negative set by judging whether the score of the teacher model is better than that of the student model; Step 2, constructing a distribution-based distillation strategy, respectively calculating the approaching distribution of the positive set and the negative set between the student model and the teacher model, so that the student model learns from the teacher model on the positive set and avoids making the same mistakes as the teacher model on the negative set, for realizing knowledge distillation of the student model to the teacher model; Step 3, constructing a knowledge inheritance strategy, so that the student model reviews the knowledge learned by the student model at the previous time step, for realizing continuous learning of the student model; Step 4, generating the final total training loss by knowledge distillation of the student model to the teacher model, continuous learning of the student model, and training loss of the student model; Step 5, calculating the loss function gradient using the obtained total training loss, updating the model parameter weight according to the gradient and the learning rate using the optimizer, obtaining the final grammar error correction model, and using the final grammar error correction model for grammar error correction; The Step 1 comprises: Step 1.1: Using the distilled knowledge set as input to the teacher model and student model respectively, calculate the entropy of the target token; where the information entropy of the student model is... The calculation formula is as follows: Information entropy of the teacher model The formula for calculating the information entropy is as follows: wherein X represents the source language training set, y represents the model prediction result, j represents the word index value in the sentence, y j represents the current target word, y ＜j represents the error grammar correction prediction result before the jth word in the target sentence, S represents the student model, T represents the teacher model, represents the entropy value of the prediction probability distribution of the target word by the student model, used to measure the prediction ability of the student model; represents the entropy value of the prediction probability distribution of the target word by the teacher model, used to measure the prediction ability of the teacher model; P(y|y ＜j ,X,S) represents the target prediction probability of the student model, P(y|y ＜j ,X,T) represents the target prediction probability of the teacher model; Step 1.2, after obtaining the entropy of the target token, cosine theorem is used to calculate the student model Teacher model Similarity S label of y CS and T CS , S CS and T CS The calculation formula is as follows: wherein denotes the current student model prediction information entropy, denotes the current teacher model prediction information entropy, y label denotes the current label data, S CS denotes the similarity between the student model prediction result and the label data, T CS denotes the similarity between the teacher model prediction result and the label data; Step 1.3, compare the similarity S between student model prediction result and label data CS the similarity T between teacher model prediction result and label data CS divide the distilled knowledge set into positive set and negative set; when S CS is smaller than T CS , i.e. divided into positive set otherwise, divided into negative set 2. The grammar error correction method based on continuous knowledge distillation according to claim 1, characterized in that: Before Step 1, the operation further comprises: preprocessing parallel training corpus, verification corpus and test corpus of different scales for model training, parameter tuning and effect testing. 3.The syntax error correction method based on continuous knowledge distillation according to claim 1, wherein: In Step 1, the teacher model selects a public grammar correction model, and the student model selects a model trained based on local corpus.

4. The method of claim 1, wherein: The specific steps of Step 2 are: Step 2.1, according to the obtained positive set respectively as the source language training set X input student, teacher model, get model output calculation between the approach distribution of teacher and student model; the calculation formula of the approach distribution KL divergence is as follows: where S t denotes the student model at time step t, denotes the teacher model at time step t; KL() denotes the computation of KL divergence, which is used to measure the difference between the teacher model and the student model; denotes the prediction result of the teacher model at time step t, P(y j |y ＜j ,X; S t denotes the prediction result of the student model at time step t; denotes the knowledge distillation loss on the positive set ; Step 2.

2. The same as Step 2.1, replace the input of student model and teacher model with Calculate According to Calculate The minimum value between them is used to avoid the student model from making the same mistake by approaching the target distribution of the difference through the output distribution of the student model and the teacher model. According to The calculation The calculation formula of the minimum value between them is as follows: where represents the negative feedback loss on the negative set min(, ) represents taking the minimum value, represents the knowledge distillation loss on the negative set α is a hyperparameter that controls the activation of the loss; Step 2.3, the overall knowledge distillation is the sum of the losses of Step 2.1 and Step 2.2; where D trans represents the transfer set of the student model to the teacher model, represents the knowledge distillation loss on the transfer set D trans .

5. The method of claim 4, wherein: In Step 3, the review of the knowledge learned by the student model at the previous time step comprises: By calculating the approaching distribution between the previous time step and the current time step in the student model, the student model maximizes the retention of learned prior knowledge.

6. The method of claim 5, wherein: In Step 3, the calculation of the approaching distribution between the previous time step and the current time step in the student model requires learning the knowledge learned at the previous time step, and the calculation formula is as follows: wherein denotes the student model at time step t-1, denotes the continuing learning training loss, denotes the student model prediction result at time step t-1.

7. The method of claim 4, wherein: Step 4 comprises: student model S at time step t t by the current teacher model encoding new knowledge and the previously learned student model encoding new knowledge determined, is a model encoding previously learned knowledge, and the final training total loss is: where D train denotes the training set of student model, denotes the training total loss on D train and D trans data; λ denotes a hyperparameter balancing the preference of accepting new knowledge and inheriting old knowledge. where, is the standard cross-entropy loss, defined as: wherein, denotes the m-th target sentence of length j; X (m) denotes the m-th source sentence, M denotes the total number of sentences in a batch, J (m) denotes the total length of the m-th sentence.

8. The method of claim 1, wherein: Step 5 comprises: The gradient is calculated according to the total training loss obtained in Step 4, the student model parameters are updated according to the gradient, the final grammar error correction model is obtained, and the loss function gradient calculation is defined as follows: wherein denotes the loss function gradient value at time step t; denotes the derivative of the loss function with respect to the parameter θ; denotes the loss function for parameter θ at time step t.

9. A system for syntax error correction based on continuous knowledge distillation, characterized in that, The system comprises a module for executing the method according to any one of claims 1 to 8.

Images