A method for correcting grammatical errors by integrating constituent syntactic information

Abstract

A grammatical error correction method incorporating component syntactic information belongs to the field of artificial intelligence technology. It includes: extracting component syntactic information from a given statement; using the erroneous statement, its corresponding correct statement, and the component syntactic information as input data; and constructing a multi-task grammatical error correction model that integrates component syntactic information to correct the erroneous statement. This method first extracts component syntactic information from the statement using grammatical analysis, serializes the statement's component syntactic tree to obtain a component syntactic sequence, and constructs triplet pairs as input data. Second, an adapter module is introduced to construct a multi-head attention mechanism model based on multi-task learning, learning the potential relationships between erroneous and correct statements, and between erroneous statements and component syntactic sequences. Finally, through pre-training and lightweight fine-tuning of the adapter module, the features of the erroneous statement, the correct statement, and the component syntactic sequence are fused to complete the grammatical error correction.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to natural language processing, and particularly to methods for correcting grammatical errors. Background Technology

[0002] The grammar error correction task automatically detects and corrects grammatical errors, such as tense, prepositions, and misspelled words, in a given text. It can be applied to social media platforms to check for errors in participants' content and help prevent the spread of misinformation. The input to the grammar error correction task is a sentence that may contain grammatical errors, and the output is a sentence that has been corrected to be grammatically correct.

[0003] In recent years, grammar error correction has often been viewed as a machine translation problem, treating erroneous sentences as source language and corrected sentences as target language. This has enabled the sequence-to-sequence architecture of neural machine translation to be used for grammar error correction tasks. Currently, Transformer-based sequence-to-sequence architectures are the mainstream choice for grammar error correction models.

[0004] However, Transformer-based sequence-to-sequence syntax error correction methods still have some problems. First, syntactic information, as strong prior knowledge, can help syntax error correction systems better correct erroneous sentences, but existing methods simply treat text as a sequence of words, ignoring the syntactic information in sentences. The input erroneous sentences contain incorrect syntactic information, so these methods cannot benefit from it. Second, existing methods have a huge number of model parameters, often consisting of millions or even billions of parameters. These models are pre-trained on large amounts of automatically synthesized data and fine-tuned on a small amount of human-labeled data (i.e., low resources). Extensive fine-tuning of all parameters of the pre-trained model on relatively low resources (i.e., human-labeled data) affects the performance and efficiency of the pre-training-fine-tuning method.

[0005] In conclusion, grammatical error correction that integrates grammatical information is an innovative research problem with significant research implications and application value. Summary of the Invention

[0006] The purpose of this invention is to address the problem of insufficient utilization of prior grammatical information in existing grammatical error correction methods. It innovatively proposes a grammatical error correction method that integrates constituent syntactic information. This method uses a deep neural network framework to integrate constituent syntactic information that aids in sentence analysis. It also designs an effective model structure and a two-stage training process to incorporate constituent syntactic information into sentence modification, thereby improving the effectiveness of grammatical error correction.

[0007] The technical solution of this invention is:

[0008] A method for correcting grammatical errors by integrating constituent syntactic information is described in detail below:

[0009] 1. Obtain the syntax error correction dataset

[0010] The data obtained by this method includes manually labeled data and automatically generated data. Each sample in the data includes a sentence with grammatical errors and a corrected sentence, i.e., a (incorrect sentence, correct sentence) statement pair.

[0011] 1.1 Obtaining manually labeled data

[0012] We collected a dataset of manually annotated grammatical error corrections. The dataset used was the BEA-2019 dataset, which includes the NUCLE, Lang-8, FCE, and W&I+LOCNESS datasets. Each sample in the BEA-2019 dataset contains a sentence with grammatical errors and a manually corrected version of the sentence.

[0013] 1.2 Obtaining automatically generated data

[0014] Obtain an automatically generated grammar error correction dataset, specifically the One BillionWord dataset. Select a subset of this dataset for sample production. Perform operations such as word insertion, deletion, modification, and word swapping on each sentence to construct sentences containing grammatical errors. The sentences after these operations are called sentences containing grammatical errors, while the sentences before these operations are called sentences without grammatical errors after correction.

[0015] 2. Generate constituent syntactic sequences

[0016] Integrate all the data obtained in step 1 and generate a constituent syntax sequence. For each sample, firstly, select the correct statement from the (incorrect statement, correct statement) statement pair, parse the correct statement using CoreNLP tools to obtain the constituent syntax tree; then, since the model cannot accept a tree structure as input, use preorder traversal to convert the tree into a linearized sequence, called the constituent syntax sequence; finally, we create a (incorrect statement, correct statement, constituent syntax sequence) triplet as the model input.

[0017] Definition 1: Component Syntax Tree

[0018] A constituent syntax tree is a tree that breaks down a sentence into phrases. The non-leaf nodes in the tree represent the phrase types of the subtrees, the leaf nodes are the words in the sentence, and the edges are unlabeled.

[0019] Definition 2: Constituent Syntactic Sequence

[0020] The sequence obtained by the preorder traversal of the constituent syntax tree, each subtree is represented as a grammatical structure by being enclosed in parentheses.

[0021] 3. Construct a multi-task syntax error correction model that integrates component syntactic information.

[0022] This method designs a sequence-to-sequence framework with fused constituent syntactic analysis based on the Transformer deep neural network model, which is mainly divided into three parts: a multi-task encoder, a syntax correction decoder, and a constituent syntactic decoder.

[0023] 3.1 Multi-task encoder

[0024] Given an error statement S = {w1, w2, ..., wn} of length n n}, each word w i Mapped to a continuous word embedding w i Then we can obtain the news embedding S′={w′1,w′2,…,w′}. n Furthermore, to utilize the word order in erroneous statements, positional encoding is added:

[0025] u t =w t ′+pos t ,

[0026] Where, pos t To encode the position of the t-th word in the erroneous statement, assign e0 = u = (u0, ..., u n This is used as the input at the bottom of the encoder. Typically, an encoder consists of multiple identical layers. Each layer comprises a multi-head self-attention mechanism, residual connections, a regularization layer, an adapter layer, and a fully connected feedforward network. Each layer undergoes the following calculations:

[0027]

[0028]

[0029]

[0030]

[0031] e k =LayerNorm(l k +a k ),

[0032] Among them, the output e of the previous encoder layer k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k Indicates the output of the adapter sublayer, e k This represents the output of the k-th encoder layer.

[0033] Definition 3: Adapter Layer

[0034] The calculation method of the adapter layer is as follows:

[0035] Adapter(h) = W up ·f(W down ·h)+h

[0036] Among them W up and W down Here, f represents the learnable parameters; the former maps the input to a higher dimension, while the latter maps the input to a lower dimension. f is a non-linear activation function.

[0037] 3.2, Syntax Correction Decoder

[0038] Given a correct statement T = {t1, t2, ..., tm} of length m. m}, each word t i Mapped to a continuous word embedding t i Then you can obtain the correct statement embedding T′={t′1,t′2,…,t′}. m Similarly, to utilize the correct word order in the sentence, positional encoding is added:

[0039] v i =t i ′+pos i ,

[0040] Where, pos i To encode the position of the i-th word in the erroneous statement, assign d0 = v = (v0, ..., v m This serves as the input to the bottom of the decoder. Typically, a decoder consists of multiple identical layers, each comprising a multi-head interactive attention layer, a multi-head self-attention layer, a residual connection layer, a regularization layer, and a fully connected feedforward network. Each layer undergoes the following computation:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Where e represents the output of the last encoder layer and d represents the output of the previous decoder layer. k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the multi-head interaction attention mechanism sub-layer. This indicates the output of the second regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k Indicates the output of the adapter sublayer, d k This represents the output of the k-th encoder layer.

[0049] 3.3, Constituent Syntax Decoder

[0050] Given a line of length m * Component syntactic sequence Each constituent syntax node w i Mapped to a continuous word embedding w i Then you can get the correct statement embedding. To utilize the correct word order in the sentence, positional encoding is added:

[0051]

[0052] Where, pos i Encode the position of the i-th word in the erroneous statement, and... This serves as the input to the bottom layer of the constituent syntax decoder. The bottom layer of the constituent syntax decoder consists of multiple identical layers, each composed of a multi-head interactive attention layer, a multi-head self-attention layer, a residual connection layer, a regularization layer, and a fully connected feedforward network. Each layer undergoes the following computation:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] Where e represents the output of the last encoder layer and p represents the output of the previous component syntactic decoder layer. k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the multi-head interaction attention mechanism sub-layer. This indicates the output of the second regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k p represents the output of the adapter sublayer. k This represents the output of the k-th layer component syntax decoder layer.

[0061] 4. Use a two-stage training method to train the model.

[0062] This method employs a two-stage training approach, combining large-scale data pre-training with a lightweight fine-tuning adapter. It effectively utilizes both a large amount of synthetic data and a small amount of manually labeled data, allowing for hot-start initialization of models with multiple decoders. The large-scale corpus provides ample training data for the model, and pre-training can learn general language representations from the large-scale corpus. As an efficient parameter tuning method, lightweight fine-tuning can alleviate the problem caused by insufficient resources.

[0063] During the training phase, the inputs to the last layer of the syntax correction decoder and the last layer of the constituent syntax decoder are fed into different linear layers to obtain the final correct sentence prediction output. and constituent syntactic sequence prediction output

[0064]

[0065]

[0066] Where W,b,W * ,b * All parameters are learnable. The loss function is minimized through training, and the loss function is as follows:

[0067]

[0068] Where T represents the correct statement in the sample mentioned above, T * λ is the constituent syntax sequence mentioned earlier, and λ is the coefficient.

[0069] Section 4.1, Pre-training

[0070] The parameters of this method's model are initialized using the parameters of BART. Specifically, the parameters of the multi-task encoder are initialized using the parameters of the BARTbase encoder, while the parameters of the constituent syntactic decoder and syntax correction decoder are initialized using the BARTbase decoder, and the adapter layer is randomly initialized. Then, pre-training is performed on large-scale synthetic data.

[0071] Section 4.2, Lightweight Fine-tuning

[0072] Building upon step 4.1, only the parameters of the adapter layer are adjusted, while the other parameters of the model are frozen, and further training is performed on manually labeled data.

[0073] Advantages and positive effects of the present invention:

[0074] This invention innovatively proposes a grammatical error correction method that integrates constituent syntactic information. It utilizes constituent syntactic information of sentences as prior grammatical knowledge and designs a constituent syntactic decoder to obtain the implicit connections between the syntactic sequences of sentence constituents, thereby more rationally and effectively integrating constituent syntactic information into grammatical error correction. This invention effectively models the connections between erroneous sentences, correct sentences, and prior grammatical knowledge, and can integrate constituent syntactic information into grammatical error correction, thus significantly improving the accuracy of grammatical error correction tasks. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating a grammatical error correction method that integrates component syntactic information.

[0076] Figure 2 This is a table of statistics for the dataset used in this method.

[0077] Figure 3 Example diagram of an automatically generated syntax error correction dataset.

[0078] Figure 4 Generate example diagrams for constituent syntax sequences.

[0079] Figure 5 This is a diagram illustrating the results of the syntax error correction accuracy.

[0080] Figure 6 This is the result of the ablation experiment.

[0081] Figure 7 The results are from experiments using other lightweight fine-tuning components. Detailed Implementation

[0082] This invention proposes a grammatical error correction method that integrates constituent syntactic information. The main process of the method is as follows: Figure 1As shown in the accompanying drawings, the specific embodiments of the present invention will now be described in detail.

[0083] The specific implementation process of this invention consists of five steps: data collection and generation; generation of constituent grammar sequences; design of a multi-task model; two-stage training; and grammatical error correction. The following is a detailed description of the implementation process of these five steps:

[0084] 1. Obtain the syntax error correction dataset

[0085] The data acquired by this method includes manually labeled data and automatically generated data. Each sample in the data includes a sentence with a grammatical error and a corrected sentence, i.e., a (error statement, correct statement) statement pair. Figure 2 The dataset used in this method is shown.

[0086] 1.1 Obtaining manually labeled data

[0087] We collected a dataset of manually annotated grammatical error corrections. The dataset used was the BEA-2019 dataset, which includes the NUCLE, Lang-8, FCE, and W&I+LOCNESS datasets. Each sample in the BEA-2019 dataset contains a sentence with grammatical errors and a manually corrected version, totaling approximately 640,000 samples.

[0088] 1.2 Obtaining automatically generated data

[0089] Automatically generated data was obtained using the One Billion Word dataset. A subset was selected for sample creation. Each sentence within this dataset underwent operations such as word deletion, insertion, modification, and word swapping to construct sentences containing grammatical errors. The sentences after these operations were termed "grammatically incorrect sentences," while the sentences before the operations were termed "grammatically correct sentences." For example... Figure 3 The diagram illustrates the automatic data generation process, from top to bottom, where deletion, insertion, modification, and position swapping are used to transform a statement without errors into one containing errors.

[0090] 2. Generate constituent syntactic sequences

[0091] Integrate all the data obtained in step 1 and generate a constituent syntax sequence. For each sample, firstly, select the correct statement from the (incorrect statement, correct statement) statement pair, parse the correct statement using CoreNLP tools to obtain the constituent syntax tree; then, since the model cannot accept a tree structure as input, use preorder traversal to convert the tree into a linearized sequence, called the constituent syntax sequence; finally, we create a (incorrect statement, correct statement, constituent syntax sequence) triplet as the model input. For example... Figure 4 As shown, taking the sentence pair "PoorJohn loses his watch" and "PoorJohn loses his watch" (incorrect statement, correct statement) as an example, we first use CoreNLP to perform constituent syntactic analysis on the correct statement, obtaining... Figure 4 The right side shows the constituent syntax parsing tree; then, a preorder traversal is performed on the resulting tree, and parentheses are added at the beginning and end of each subtree to enclose each phrase structure in parentheses, resulting in a constituent syntax sequence; finally, the constituent syntax sequence and (error statement, correct statement) are combined to obtain the (error statement, correct statement, constituent syntax sequence) triplet used in subsequent steps.

[0092] 3. Construct a multi-task syntax error correction model that integrates component syntactic information.

[0093] This method designs a sequence-to-sequence framework with fused constituent syntactic analysis based on the Transformer deep neural network model, which is mainly divided into three parts: a multi-task encoder, a syntax correction decoder, and a constituent syntactic decoder.

[0094] 3.1 Multi-task encoder

[0095] Given an error statement S = {w1, w2, ..., wn} of length n n}, each word w i Mapped to a continuous word embedding w i Then, the embedding of the erroneous statement S′={w′1,w′2,…,w′} can be obtained. n Furthermore, to utilize the word order in erroneous statements, positional encoding is added:

[0096] u t =w t ′+pos t ,

[0097] Where, pos t To encode the position of the t-th word in the erroneous statement, assign e0 = u = (u0, ..., u nThis is used as the input at the bottom of the encoder. Typically, an encoder consists of multiple identical layers. Each layer comprises a multi-head self-attention mechanism, residual connections, a regularization layer, an adapter layer, and a fully connected feedforward network. Each layer undergoes the following calculations:

[0098]

[0099]

[0100]

[0101]

[0102] e k =LayerNorm(l k +a k ),

[0103] Among them, the output e of the previous encoder layer k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k Indicates the output of the adapter sublayer, e k This represents the output of the k-th encoder layer.

[0104] Definition 3: Adapter Layer

[0105] The calculation method of the adapter layer is as follows:

[0106] Adapter(h) = W up ·f(W down ·h)+h

[0107] Among them W up and W down Here, f represents the learnable parameters; the former maps the input to a higher dimension, while the latter maps the input to a lower dimension. f is a non-linear activation function.

[0108] 3.2, Syntax Correction Decoder

[0109] Given a correct statement T = {t1, t2, ..., tm} of length m. m}, each word t i Mapped to a continuous word embedding t i Then you can obtain the correct statement embedding T′={t′1,t′2,…,t′}. m Similarly, to utilize the correct word order in the sentence, positional encoding is added:

[0110] vi =t i ′+pos i ,

[0111] Where, pos i To encode the position of the i-th word in the erroneous statement, assign d0 = v = (v0, ..., v m This serves as the input to the bottom of the decoder. Typically, a decoder consists of multiple identical layers, each comprising a multi-head interactive attention layer, a multi-head self-attention layer, a residual connection layer, a regularization layer, and a fully connected feedforward network. Each layer undergoes the following computation:

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] Where e represents the output of the last encoder layer and d represents the output of the previous decoder layer. k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the multi-head interaction attention mechanism sub-layer. This indicates the output of the second regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k Indicates the output of the adapter sublayer, d k This represents the output of the k-th encoder layer.

[0120] 3.3, Constituent Syntax Decoder

[0121] Given a line of length m * Component syntactic sequence Each constituent syntax node w i Mapped to a continuous word embedding w i Then you can get the correct statement embedding. To utilize the correct word order in the sentence, positional encoding is added:

[0122]

[0123] Where, pos i Encode the position of the i-th word in the erroneous statement, and... This serves as the input to the bottom layer of the constituent syntax decoder. The bottom layer of the constituent syntax decoder consists of multiple identical layers, each composed of a multi-head interactive attention layer, a multi-head self-attention layer, a residual connection layer, a regularization layer, and a fully connected feedforward network. Each layer undergoes the following computation:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] Where e represents the output of the last encoder layer and p represents the output of the previous component syntactic decoder layer. k-1 As the input of the k-th layer, This indicates the output of the multi-head self-attention mechanism sublayer at this level, l k This indicates the output of the first regularization layer in this layer. This represents the output of the multi-head interaction attention mechanism sub-layer. This indicates the output of the second regularization layer in this layer. This represents the output of the feedforward neural network sublayer, a. k p represents the output of the adapter sublayer. k This represents the output of the k-th layer component syntax decoder layer.

[0132] 4. Use a two-stage training method to train the model.

[0133] This method employs a two-stage training approach, combining large-scale data pre-training with a lightweight fine-tuning adapter. It effectively utilizes both a large amount of synthetic data and a small amount of manually labeled data, allowing for hot-start initialization of models with multiple decoders. The large-scale corpus provides ample training data for the model, and pre-training can learn general language representations from the large-scale corpus. As an efficient parameter tuning method, lightweight fine-tuning can alleviate the problem caused by insufficient resources.

[0134] During the training phase, the inputs to the last layer of the syntax correction decoder and the last layer of the constituent syntax decoder are fed into different linear layers to obtain the final correct sentence prediction output. and constituent syntactic sequence prediction output

[0135]

[0136]

[0137] Where W,b,W * ,b * All parameters are learnable. The loss function is minimized through training, and the loss function is as follows:

[0138]

[0139] Where T represents the correct statement in the sample mentioned above, T * λ is the constituent syntax sequence mentioned earlier, and λ is the coefficient.

[0140] Section 4.1, Pre-training

[0141] The parameters of this method's model are initialized using the parameters of BART. Specifically, the parameters of the multi-task encoder are initialized using the parameters of the BARTbase encoder, while the parameters of the constituent syntactic decoder and syntax correction decoder are initialized using the BARTbase decoder, and the adapter layer is randomly initialized. Then, pre-training is performed on large-scale synthetic data.

[0142] Section 4.2, Lightweight Fine-tuning

[0143] Building upon step 4.1, only the parameters of the adapter layer are adjusted, while the other parameters of the model are frozen, and further training is performed on manually labeled data.

[0144] 5. Grammar error correction

[0145] The main performance metrics for grammar error correction in this method are precision, recall, and F0.5, with F0.5 being the primary metric. Experimental results show that this invention outperforms other methods in grammar error correction. This method was tested using the English datasets CoNLL-2014 and BEA-2019, with samples from real-world English writing materials used by beginners. Statistical information for the datasets is as follows: Figure 2 As shown.

[0146] like Figure 5As shown, compared with other methods, our method achieves a significant performance improvement in the syntax error correction task, with the highest F0.5 score. Compared with other methods that incorporate syntactic information, our method has higher precision and slightly lower recall. This is because methods that incorporate syntactic information can more accurately identify syntax errors in sentences and make more conservative but accurate corrections.

[0147] By comparing this method with other methods that incorporate syntactic information, it can be found that this method is superior to other methods. There are two reasons for the better performance of this invention: the results of constituent syntactic analysis are more able to represent the integrity of the phrase structure in the sentence than the results of dependency syntactic analysis, and existing research has shown that exploring phrase boundaries is helpful for grammatical error correction.

[0148] To explore the importance of fully utilizing the two-stage training method and incorporating constituent syntactic information, an ablation experiment was conducted. Figure 6 The ablation experiments are presented with and without using the full two-stage training method and without incorporating constituent syntactic information. In the table, "-" indicates the reduction compared to the model in the first row. "ECCP" represents the constituent syntactic decoder, "P" represents pre-training, and "T" represents lightweight fine-tuning. The graph clearly shows that the absence of any module significantly impacts the model's performance, highlighting the importance of each module.

[0149] To explore the advantages of the adapter module in lightweight fine-tuning, a comparative experiment was conducted. Figure 7 The results show comparative experiments using different quantization fine-tuning modules. "+" indicates the module replaced relative to the first row. The experimental results show that the adapter module has certain advantages in this approach because: the LoRA module focuses on linear transformations, reducing computational cost by injecting trainable low-rank matrices into the transformation layer; however, its ability to extract non-linear features of constituent grammar is weaker than the adapter module, which includes non-linear transformations; the Prefix Tuning module adds prefix vectors to the keys and values ​​of the multi-head attention at each layer, and for multi-task encoders, these additional prefix parameters may adversely affect the extraction of constituent structural features of the entire sentence.

[0150] In summary, this method can effectively integrate syntactic information and combine constituent syntactic information with correct and incorrect sentence pairs, thereby better completing the task of grammatical error correction.

Claims

1. A method of grammar error correction that fuses constituent syntax information, characterized by: Comprising the following steps, Step 1, obtaining a syntax error correction data set; Step 2, generating a constituent syntax sequence; Using the CoreNLP tool, constituent syntax analysis is performed on the correct sentence to obtain a constituent syntax analysis tree; then, pre-order traversal is performed on the obtained tree, and brackets are added at the beginning and end of each subtree to wrap each phrase structure with brackets to obtain a constituent syntax sequence; finally, the constituent syntax sequence, the error sentence and the correct sentence are combined to obtain a triple tuple of error sentence, correct sentence and constituent syntax sequence for subsequent steps; Step 3, constructing a multi-task syntax error correction model fused with constituent syntax information; The specific method is: Constructing a sequence-to-sequence model containing a multi-task encoder, a syntax error correction decoder and a constituent syntax decoder, Step 3.1, multi-task encoder Given a sentence of length of erroneous words , each word is mapped to a continuous word embedding , then the embedding of the erroneous sentence can be obtained , with position encoding: wherein, is the position encoding of the t-th word in the erroneous sentence, and ) as input to the bottom of the encoder, which is composed of multiple identical layers, each layer consisting of a multi-head self-attention mechanism, a residual connection, a regularization layer, an adapter layer, and a fully connected feedforward network; Step 3.2, syntax correction decoder Given a correct sentence of length , each word is mapped to a continuous word embedding , then the correct sentence embedding can be obtained, with position encoding: wherein, is a position encoding for the i-th word in the erroneous sentence, and ) as input to the bottom of the decoder, which is composed of multiple identical layers, each layer consisting of a multi-headed cross-attention layer, a multi-headed self-attention layer, a residual connection layer, a regularization layer, and a fully connected feed-forward network; Step 3.3, constituent syntax decoder Given a sequence of constituent syntactic nodes of length each constituent syntactic node is mapped to a continuous word embedding then a correct sentence embedding can be obtained, adding position encoding: wherein, is a position encoding for the i-th word in the erroneous sentence, and ) as input to a constituent syntax decoder bottom, the constituent syntax decoder bottom being composed of multiple identical layers, each layer being composed of a multi-headed cross-attention layer, a multi-headed self-attention layer residual connection layer, a regularization layer, and a fully connected feed-forward network; Step 4, training the model in a two-stage training manner; In the training stage, the input of the last layer of the grammar correction decoder and the last layer of the constituent syntax decoder is respectively input to different linear layers to obtain the final correct sentence prediction output and constituent syntax sequence prediction output where are learnable parameters, minimizing a loss function by training, the loss function being as follows: wherein is the correct sentence in the aforementioned sample, is the aforementioned component syntax sequence, is the coefficient.

2. The method of claim 1, wherein the method of correcting a syntax error by fusing constituent syntax information is characterized by, The specific method of step 1 is: Step 1.1, collect data Collect the BEA-2019 data set, which includes the NUCLE data set, the Lang-8 data set, the FCE data set and the W&I+LOCNESS data set; Step 1.2, generate data Collect the One Billion Word data set, select a subset of it to make samples, and perform word insertion, deletion, modification and position exchange operations on each sentence to construct sentences containing syntax errors, and form samples with the original sentences.

3. The method of claim 1, wherein the method of correcting a syntax error by fusing constituent syntax information comprises, The specific method of step 2 is: Use the constituent syntax analysis tool to convert the correct sentence into a syntax analysis tree, and then use the pre-order traversal method to convert the syntax analysis tree into a sequence, and wrap each subtree with brackets.

4. The method of claim 3, wherein the fusion component syntax information is generated by: The constituent syntax analysis tool is the CoreNLP tool. ​ 5. The method of claim 1, wherein the method of correcting a syntax error by fusing constituent phrase syntax information comprises: determining a syntax error in a syntax tree of a sentence; and correcting the syntax error by fusing constituent phrase syntax information. In the specific method of constructing a multi-task syntax error correction model fused with constituent syntax information, the calculation method of the adapter layer is where and are learnable parameters, the former maps the input to a higher dimension, the latter maps the input to a lower dimension, is a non-linear activation function.

6. The method of claim 1, wherein the method of correcting a syntax error by fusing constituent phrase syntax information comprises, The specific method of step 4 is: Step 4.1, pre-training part After initializing the parameters from BART base, pre-training is performed on large-scale synthetic data; Step 4.2, light-weight fine-tuning part On the basis of completing step 4.1, only the parameters of the adapter layer are adjusted, and training is performed on the artificially annotated data.

Images