A Chinese text grammatical error correction method combining syntactic information and pre-trained language model

Abstract

The present invention discloses a Chinese text grammatical error correction method that combines syntactic information and a pre-trained language model. By integrating syntactic information into the pre-trained language model Chinese BART, the Chinese grammatical error correction capability can be effectively improved. The method comprises the following steps: importing a data set and converting it into a target format; using HanLP to generate syntactic information, converting part-of-speech tags and dependencies into vectors, and fusing them with default word embeddings; using a syntactic-enhanced multi-head self-attention mechanism and a residual network in the encoding stage; using a self-attention mechanism and an encoder-decoder attention mechanism in combination with syntactic information in the decoding stage, and combining the residual network to gradually optimize the decoder; using Beam Search to generate multiple candidate sentences, and selecting the optimal candidate sentence based on syntactic consistency and semantic rationality. By combining syntactic information and a pre-trained language model, the present invention allows the model to learn the semantic information of the sentence and pay attention to the syntactic structure, which helps to improve the Chinese grammatical error correction capability of the model.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and text error correction, and in particular to Chinese text grammatical error correction that combines syntactic information and a pre-trained language model. Background Art

[0002] With China's continued rapid economic development and growing national strength, more and more people around the world are choosing to learn Chinese as a second language. However, because Chinese, a written language that incorporates morphemes and pictograms, is more flexible and grammatically complex than Indo-European languages, learners often encounter difficulties and make various grammatical errors. Therefore, research on grammatical error correction technology for Chinese texts is necessary. It can provide corrective guidance for Chinese learners, improve their learning efficiency, and help them better master and use the Chinese language.

[0003] The current mainstream research approach is based on machine translation, specifically the monolingual translation problem of translating incorrect sentences into correct ones. However, Chinese grammatical error correction methods based on neural machine translation models are limited by the lack of parallel corpora. Therefore, pre-trained models are being introduced to this task in the hope of alleviating this corpus shortage. However, due to the complexity of Chinese, a language with numerous characters, flexible grammar, and rich cultural background, text processing is complex. Existing models and systems suffer from low accuracy and recall, leaving room for further improvement.

[0004] Therefore, the present invention integrates syntactic information into multiple links of the pre-trained language model, so that the model can fully understand the grammatical relationship between Chinese words, thereby improving the model's Chinese grammar error correction ability. Summary of the Invention

[0005] The present invention proposes a Chinese text grammatical error correction method that combines syntactic information and a pre-trained language model. The pre-trained language model Chinese BART (Bidirectional and Auto-Regressive Transformers for Chinese) can effectively correct Chinese texts with grammatical errors after incorporating syntactic information.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions, and the implementation steps are as follows:

[0007] Step 1: Import the dataset and convert the relevant dataset into the target format, that is, the sentence pair form of source sentence and target sentence.

[0008] Step 2: Word embeddings incorporate syntactic information. BERT's SentencePiece tokenizer performs word segmentation and encoding to obtain BART's default Chinese text word embeddings. The natural language processing tool HanLP is then used to generate syntactic information, including dependency syntax trees and part-of-speech tags. The part-of-speech tags and dependency relationships are converted into vector representations. The word vectors and part-of-speech tag vectors are then passed through a linear layer and added together to obtain a word embedding that incorporates syntactic information.

[0009] Step 3: Encoder encoding uses a syntactic-enhanced multi-head self-attention mechanism. This mechanism assigns higher attention scores to word pairs with close syntactic dependencies, allowing the encoder to focus more on syntactic structure. A residual network structure is also introduced in each encoding layer. The input of each layer is passed to the next layer via residual connections, mitigating the vanishing gradient problem and improving the ability to capture details in the input text.

[0010] Step 4: The decoder decodes the sentence. The masked multi-head self-attention mechanism is used again, ensuring that the decoder only sees previously generated words at each generation step. A concurrent interactive attention mechanism focuses on the encoder's output, and combined with a residual network, the decoder gradually optimizes the generated sentence. Syntactic information is further used as an auxiliary condition in the interactive attention mechanism, helping the decoder generate output that conforms to syntactic rules.

[0011] Step 5: Beam Search optimization of syntactic information. Use Beam Search to generate multiple candidate sentences, then score them based on syntactic rules to select the most appropriate output.

[0012] 1. In step 1 above, import the dataset. Here, we use the NLPCC 2018+HSK dataset, which consists of two publicly available Chinese text grammatical error correction datasets. The NLPCC 2018+HSK dataset was released at the 2018 Chinese Grammar Correction Competition held by the Natural Language Processing and Chinese Computing (NLPCC), and the HSK (Hanyu Shuiping Kaoshi) dynamic composition corpus, which collects a large number of essays written by foreign students learning Chinese and contains various types of grammatical errors. Through processing, the two datasets are combined into one, formatted as sentence pairs (source sentence - target sentence).

[0013] 2. In step 2 above, word embedding and syntactic information integration include the following steps:

[0014] (1) Use BERT word segmenter SentencePiece to segment and encode sentence pairs, and obtain BART’s default Chinese text word embedding e1, e2, e3...en .

[0015] (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n . Embed the subword into e i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i Assuming that the trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is:

[0016] H=α·E+(1-α)·P

[0017] Here, α is a hyperparameter that is usually adjusted in the range [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings.

[0018] (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency relationship. Then, a graph convolutional network (GCN) is used to perform convolution operations on the dependency graph. The initial representation H of the fusion part-of-speech tag is used as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep In the lth layer of GCN, the representation update formula for each word is:

[0019] H (l+1) =σ(AH (l) W (l) )

[0020] Among them H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, σ is the ReLU activation function. The representation obtained after L layers of GCN is H dep =H (L) The representation H after fusing word embedding and part-of-speech tags and the representation H including dependency relations dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as:

[0021] H final =β·H+(1-β)·H dep

[0022] 3. In step 3 above, the encoder encoding includes the following steps:

[0023] (1) Encoder self-attention processing. After obtaining the syntactically enhanced encoder input representation H final Finally, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the entire sentence. First, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weight of each word with other words and apply it to the value vector to get the context-enhanced representation:

[0024]

[0025] where d k is the dimension of the key vector, used to scale the attention scores.

[0026] (2) Feedforward neural network layer and encoder output. After attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved:

[0027] H att =Attention(Q,K,V)+H final

[0028] The encoder's self-attention output H att It is passed to the Feed Forward Network (FFN) layer and uses residual connections again to further retain the information after the self-attention mechanism:

[0029] Z=FFN(H att )+H att

[0030] The FFN is a two-layer feedforward neural network. Ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

[0031] 4. In step 4 above, the decoder decoding includes the following steps:

[0032] (1) Self-attention processing of the decoder. Assume that we are currently generating the t-th word, and the input sequence of the decoder (the embedding sequence of the currently generated word) is represented by Y <t express:

[0033] Y <t=[y1+pos1,y2+pos2,...,y t-1 +pos t-1 ]

[0034] At the lth layer of the decoder, a masked self-attention layer is applied, which generates an attention score for the current generated word position. For the generation of the tth word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as:

[0035]

[0036] The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position. This mask blocks the position weights of future words, making the generation process sequentially dependent. The output is processed through residual connections and normalization to obtain a new representation:

[0037]

[0038] (2) Cross-Attention Processing of the Decoder. After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The cross-attention mechanism can help the decoder obtain the global context information about the entire sentence from the output of the encoder. The query matrix of the cross-attention layer comes from the output of the masked self-attention. The query, key, and value matrices are and The interactive attention calculation formula is:

[0039]

[0040] After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer:

[0041]

[0042] In order to further combine syntactic information, we introduce dependency word embedding H dep , linearly fuse it with the interactive attention result. Assuming the fusion scale factor is γ, we have:

[0043]

[0044] (3) Feedforward neural network layer and decoder output. After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation:

[0045]

[0046] Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained:

[0047]

[0048] The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feedforward network. (L) . It is then passed into the output layer to generate the probability distribution of each time step, thereby predicting the word at the current time step t:

[0049] P(y t |y <t )=softmax(Y t W out +b out )

[0050] Among them, Y t is the hidden layer representation generated by the decoder, W out Is a mapping matrix, Y t Mapped to vocabulary space, b out is the bias term. The decoder generates words at each time step step by step until the termination condition or maximum length is reached.

[0051] 5. In step 5 above, tuning the Beam Search for syntactic information includes the following steps:

[0052] Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only relies on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. stntax To evaluate the grammatical correctness of each candidate sequence.

[0053] S syntax (y1,y2,...,y t )=f(P,H dep )

[0054] Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·S syntax (y1,y2,...,y t )

[0055] Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score.

[0056] Then, the selected B candidate paths are expanded until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

[0057] The beneficial effects of the present invention are:

[0058] (1) A text correction method combining syntactic information and pre-trained language models is proposed, which makes Chinese text grammatical correction more accurate;

[0059] (2) The part-of-speech tags and dependency relationships are used as auxiliary information and fused with the default word embedding. The performance of the encoder is optimized through the syntactic-attention mechanism, which enables the model to understand the syntactic structure more accurately and improve the accuracy of grammatical error correction in Chinese text.

[0060] (3) The introduction of a residual network structure at each layer effectively alleviates the gradient vanishing problem and ensures stable training of deep networks. At the same time, the residual connection captures the detailed information of the input text, enhancing the ability to understand long texts and complex syntactic structures.

[0061] (4) The self-attention mechanism and the interactive attention mechanism are used in the decoder, combined with the syntactic information output by the encoder, so that the generated sentences are more in line with grammatical rules, which not only corrects grammatical errors but also generates more natural and fluent texts.

[0062] (5) With the help of Beam Search optimization, on the basis of generating multiple candidate sentences, the syntactic consistency and semantic rationality are comprehensively considered, and scoring and screening are performed through syntactic rules to output the optimal candidate sentence.

[0063] This method significantly improves the grammatical standardization and semantic rationality of sentences generated by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Flowchart of a Chinese text grammatical error correction method that combines syntactic information and pre-trained language models; DETAILED DESCRIPTION

[0065] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0066] The flowchart of the Chinese text grammatical error correction method combining syntactic information and pre-trained language model is as follows: Figure 1 As shown, the steps include:

[0067] Step 1: Import the dataset and convert the relevant dataset into the target format, that is, the sentence pair form of source sentence and target sentence.

[0068] Step 2: Word embeddings incorporate syntactic information. BERT's SentencePiece tokenizer performs word segmentation and encoding to obtain BART's default Chinese text word embeddings. The natural language processing tool HanLP is then used to generate syntactic information, including dependency syntax trees and part-of-speech tags. The part-of-speech tags and dependency relationships are converted into vector representations. The word vectors and part-of-speech tag vectors are then passed through a linear layer and added together to obtain a word embedding that incorporates syntactic information.

[0069] Step 3: Encoder encoding uses a syntactic-enhanced multi-head self-attention mechanism. This mechanism assigns higher attention scores to word pairs with close syntactic dependencies, allowing the encoder to focus more on syntactic structure. A residual network structure is also introduced in each encoding layer. The input of each layer is passed to the next layer via residual connections, mitigating the vanishing gradient problem and improving the ability to capture details in the input text.

[0070] Step 4: The decoder decodes the sentence. The masked multi-head self-attention mechanism is used again, ensuring that the decoder only sees previously generated words at each generation step. A concurrent interactive attention mechanism focuses on the encoder's output, and combined with a residual network, the decoder gradually optimizes the generated sentence. Syntactic information is further used as an auxiliary condition in the interactive attention mechanism, helping the decoder generate output that conforms to syntactic rules.

[0071] Step 5: Beam Search optimization of syntactic information. Use Beam Search to generate multiple candidate sentences, then score them based on syntactic rules to select the most appropriate output.

[0072] 1. In step 1 above, import the dataset. Here, we use the NLPCC 2018+HSK dataset, which consists of two publicly available Chinese text grammatical error correction datasets. The NLPCC 2018+HSK dataset was released at the 2018 Chinese Grammar Correction Competition held by the Natural Language Processing and Chinese Computing (NLPCC), and the HSK (Hanyu Shuiping Kaoshi) dynamic composition corpus, which collects a large number of essays written by foreign students learning Chinese and contains various types of grammatical errors. Through processing, the two datasets are combined into one, formatted as sentence pairs (source sentence - target sentence).

[0073] 2. In step 2 above, word embedding and syntactic information integration include the following steps:

[0074] (1) Use BERT word segmenter SentencePiece to segment and encode sentence pairs, and obtain BART’s default Chinese text word embedding e1, e2, e3...e n .

[0075] (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n . Embed the subword into e i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i Assuming that the trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is:

[0076] H=α·E+(1-α)·P

[0077] Here, α is a hyperparameter that is usually adjusted in the range [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings.

[0078] (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency relationship. Then, a graph convolutional network (GCN) is used to perform convolution operations on the dependency graph. The initial representation H of the fusion part-of-speech tag is used as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep In the lth layer of GCN, the representation update formula for each word is:

[0079] H (l+1) =σ(AH (l) W (l) )

[0080] Among them H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, σ is the ReLU activation function. The representation obtained after L layers of GCN is H dep =H (L) The representation H after fusing word embedding and part-of-speech tags and the representation H including dependency relations dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as:

[0081] Hfinal =β·H+(1-β)·H dep

[0082] 3. In step 3 above, the encoder encoding includes the following steps:

[0083] (1) Encoder self-attention processing. After obtaining the syntactically enhanced encoder input representation H final Finally, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the entire sentence. First, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weight of each word with other words and apply it to the value vector to get the context-enhanced representation:

[0084]

[0085] where d k is the dimension of the key vector, used to scale the attention scores.

[0086] (2) Feedforward neural network layer and encoder output. After attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved:

[0087] H att =Attention(Q,K,V)+H final

[0088] The encoder's self-attention output H att It is passed to the Feed Forward Network (FFN) layer and uses residual connections again to further retain the information after the self-attention mechanism:

[0089] Z=FFN(H att )+H att

[0090] The FFN is a two-layer feedforward neural network. Ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

[0091] 4. In step 4 above, the decoder decoding includes the following steps:

[0092] (1) Self-attention processing of the decoder. Assume that we are currently generating the t-th word, and the input sequence of the decoder (the embedding sequence of the currently generated word) is represented by Y <t express:

[0093] Y <t =[y1+pos1,y2+pos2,...,y t-1 +pos t-1 ]

[0094] At the lth layer of the decoder, a masked self-attention layer is applied, which generates an attention score for the current generated word position. For the generation of the tth word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as:

[0095]

[0096] The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position. This mask blocks the position weights of future words, making the generation process sequentially dependent. The output is processed through residual connections and normalization to obtain a new representation:

[0097]

[0098] (2) Cross-Attention Processing of the Decoder. After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The cross-attention mechanism can help the decoder obtain the global context information about the entire sentence from the output of the encoder. The query matrix of the cross-attention layer comes from the output of the masked self-attention. The query, key, and value matrices are and The interactive attention calculation formula is:

[0099]

[0100] After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer:

[0101]

[0102] In order to further combine syntactic information, we introduce dependency word embedding H dep , linearly fuse it with the interactive attention result. Assuming the fusion scale factor is γ, we have:

[0103]

[0104] (3) Feedforward neural network layer and decoder output. After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation:

[0105]

[0106] Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained:

[0107]

[0108] The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feedforward network. (L) . It is then passed into the output layer to generate the probability distribution of each time step, thereby predicting the word at the current time step t:

[0109] P(y t |y <t )=softmax(Y t W out +b out )

[0110] Among them, Y t is the hidden layer representation generated by the decoder, W out Is a mapping matrix, Y t Mapped to vocabulary space, b out is the bias term. The decoder generates words at each time step step by step until the termination condition or maximum length is reached.

[0111] 5. In step 5 above, tuning the Beam Search for syntactic information includes the following steps:

[0112] Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only relies on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. stntax To evaluate the grammatical correctness of each candidate sequence.

[0113] S syntax (y1,y2,...,y t )=f(P,H dep )

[0114] Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·Ssyntax (y1,y2,...,y t )

[0115] Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score.

[0116] Then, the selected B candidate paths are expanded until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

Claims

1. A Chinese text grammar error correction method combining syntactic information and pre-trained language model Chinese BART, characterized by The following steps are involved: Step 1: Import the dataset and convert it into the target format, i.e., the source sentence-target sentence pair format. Step 2: Word embedding integrates syntactic information and uses the BERT word segmenter SentencePiece to perform word segmentation and encoding to obtain the BART default Chinese text word embedding; We also use the natural language processing tool HanLP to generate syntactic information, including dependency syntax trees and part-of-speech tags. We convert the part-of-speech tags and dependency relationships into vector representations, pass the word vectors and part-of-speech tag vectors through linear layers, and then add them together to obtain word embeddings that incorporate syntactic information. Step 3: Encoder encoding uses a syntactic-enhanced multi-head self-attention mechanism. This mechanism assigns higher attention scores to word pairs with close syntactic dependencies, allowing the encoder to focus more on syntactic structure. A residual network structure is also introduced in each encoding layer. The input of each layer is passed to the next layer through residual connections, mitigating the vanishing gradient problem and improving the ability to capture details of the input text. Step 4: The decoder decodes the sentence and continues to use the masked multi-head self-attention mechanism, allowing the decoder to only see previously generated words at each generation step. At the same time, the interactive attention mechanism focuses on the encoder's output and, combined with the residual network, allows the decoder to gradually optimize the generated sentence. Syntactic information is further used as an auxiliary condition in the interactive attention mechanism to help the decoder generate output that conforms to syntactic rules. Step 5: Beam Search optimization of syntactic information. Use Beam Search to generate multiple candidate sentences, then score them based on syntactic rules to select the most appropriate output.

2. The Chinese text grammar error correction method combining syntactic information and pre-trained language model Chinese BART according to claim 1 is characterized in that In step 1, a dataset is imported. Here, the NLPCC 2018+HSK dataset is selected. This dataset consists of two public Chinese text grammatical error correction datasets. The former is a dataset released by the NLPCC (Natural Language Processing and Chinese Computing) in the Chinese grammar error correction competition held in 2018, and the latter is the HSK (Hanyu Shuiping Kaoshi) dynamic composition corpus, which collects a large number of composition texts of foreign students learning Chinese, containing various types of grammatical errors. Through processing, the two datasets are combined into one in the format of sentence pairs (source sentence-target sentence).

3. The Chinese text grammar error correction method combining syntactic information and pre-trained language model Chinese BART according to claim 1 is characterized in that In step 2, word embedding and syntactic information fusion include the following steps: (1) Use BERT word segmenter SentencePiece to segment and encode sentence pairs, and obtain BART’s default Chinese text word embedding e1, e2, e3...e n； (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n； Embed the subwords into i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i； Assuming that trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is: H=α·E+(1-α)·P Here, α is a hyperparameter that is usually adjusted in the range of [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings; (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency relationship; then a graph convolutional network (GCN) is used to perform a convolution operation on the dependency graph; the initial representation H of the POS tag is fused as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep； In the lth layer of GCN, the representation update formula for each word is: H (l+1) =σ(AH (l) W (l) ) Among them H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, σ is the ReLU activation function; after L layers of GCN, the representation obtained is H dep =H (L)； The representation H after fusing word embedding and part-of-speech tags and the representation H including dependency relations dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as: H final =β·H+(1-β)·H dep 。 4. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In step 3, the encoder encoding includes the following steps: (1) Encoder self-attention processing; after obtaining the syntactically enhanced encoder input representation H final Finally, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the entire sentence; first, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weight of each word with other words and apply it to the value vector to get the context-enhanced representation: where d k is the dimension of the key vector, used to scale the attention scores; (2) Feedforward neural network layer and encoder output; after attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved: H att =Attention(Q,K,V)+H final The encoder's self-attention output H att It is passed to the FeedForward Network (FFN) layer and uses residual connections again to further retain the information after the self-attention mechanism: Z=FFN(H att ) + H att The FFN is a two-layer feedforward neural network; ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

5. The Chinese text grammar error correction method combining syntactic information and pre-trained language model Chinese BART according to claim 1 is characterized in that In step 4, decoding by the decoder includes the following steps: (1) Self-attention processing of the decoder; Assume that we are currently generating the t-th word, and the input sequence of the decoder (the embedding sequence of the currently generated word) is represented by Y <t express: Y <t =[y1+pos1,y2+pos2,...,y t-1 +post -1 ] At the decoder layer l, a masked self-attention layer is applied, which generates an attention score for the currently generated word position; For the generation of the t-th word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as: The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position; this mask will block the position weights of future words, making the generation process sequentially dependent; the output will be processed through residual connections and unification to obtain a new representation: (2) Cross-Attention Processing of the Decoder: After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The cross-attention mechanism can help the decoder obtain the global context information about the entire sentence from the output of the encoder. The query matrix of the cross-attention layer comes from the output of the masked self-attention. The query, key, and value matrices are and The interactive attention calculation formula is: After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer: In order to further combine syntactic information, we introduce dependency word embedding H dep , linearly fuse it with the interactive attention result; assuming the fusion scale factor is γ, then: (3) Feedforward neural network layer and decoder output; After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation: Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained: The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feedforward network. (L)； It is then passed into the output layer to generate a probability distribution for each time step, thereby predicting the word at the current time step t: P(y t |y <t )=softmax(Y t W out +b out ) Among them, Y t is the hidden layer representation generated by the decoder, W out Is a mapping matrix, Y t Mapped to vocabulary space, b out is the bias term; the decoder generates words at each time step by step until the termination condition or maximum length is reached.

6. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In step 5, Beam Search tuning of syntactic information includes the following steps: Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only relies on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. syntax To evaluate the grammatical correctness of each candidate sequence; S syntax (y1,y2,...,y t )=f(p,H dep ) Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·S syntax (y1,y2,...,y t ) Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score; Then, the selected B candidate paths are expanded until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

Images