Chinese text grammar error correction method combining syntactic information and pre-training language model

Abstract

The invention discloses a Chinese text grammar error correction method combining syntactic information and a pre-training language model, and the Chinese grammar error correction capability can be effectively improved by fusing the syntactic information into the pre-training language model Chine BART. The method comprises the following steps: importing a data set and converting the data set into a target format; syntactic information is generated by utilizing HanLP, part-of-speech tags and dependency relationships are converted into vectors, and the vectors are fused with default word embedding; in the encoding stage, a syntax-enhanced multi-head self-attention mechanism and a residual network are used; in the decoding stage, a self-attention mechanism and an encoder-decoder attention mechanism are combined with syntactic information for generation, and a decoder is gradually optimized in combination with a residual network; a plurality of candidate sentences are generated by using Beam Search, and the optimal candidate sentence is selected according to syntactic consistency and semantic rationality. According to the method, the syntactic information and the pre-training language model are combined, so that the model learns semantic information of sentences and pays attention to the syntactic structure, and the Chinese grammar error correction capability of the model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and text error correction, and in particular to Chinese text grammatical error correction combining syntactic information and a pre-trained language model. Background Art

[0002] With the continuous rapid development of China's economy and the gradual enhancement of its comprehensive national strength, more and more people in the world today choose to learn Chinese as a second language. However, in the learning process, since Chinese is a language that combines morphemes and pictograms, compared with the Indo-European languages, its expression is more flexible and its grammar is more complex, so learners often encounter difficulties and make various grammatical errors in the learning process. Therefore, it is necessary to study the technology of correcting grammatical errors in Chinese texts, which can play a correct guiding role for Chinese learners, improve their learning efficiency, and help them better master and use Chinese.

[0003] The current mainstream research method is a strategy based on machine translation, that is, a monolingual translation problem from an incorrect sentence to a correct sentence. However, the Chinese grammar correction method based on the neural machine translation model is limited by the lack of parallel corpus, so people introduce pre-training models into this task in the hope of alleviating the impact of insufficient corpus. However, as Chinese is a complex language with many Chinese characters, flexible grammar, and rich cultural background, text processing is complicated. Existing models and systems have problems with low accuracy and low recall, and there is still room for further improvement.

[0004] Therefore, the present invention integrates syntactic information into multiple links of the pre-trained language model, so that the model can fully understand the grammatical relationship between Chinese words, thereby improving the Chinese grammatical error correction ability of the model. Summary of the invention

[0005] The present invention proposes a Chinese text grammatical error correction method combining syntactic information and a pre-trained language model, wherein the pre-trained language model Chinese BART (Bidirectional and Auto-Regressive Transformers for Chinese) can effectively correct Chinese texts with grammatical errors after incorporating syntactic information.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solution, and the implementation steps are as follows:

[0007] Step 1: Import the dataset and convert the relevant dataset into the target format, that is, the sentence pair form of source sentence-target sentence.

[0008] Step 2: Word embedding integrates syntactic information. BERT word segmenter SentencePiece is used to perform word segmentation and encoding to obtain BART's default Chinese text word embedding. The natural language processing tool HanLP is used to generate syntactic information, including dependency syntax trees and part-of-speech tags. Part-of-speech tags and dependency relationships are converted into vector representations. The word vectors and part-of-speech tag vectors are passed through linear layers respectively, and then added together to obtain word embeddings that integrate syntactic information.

[0009] Step 3: Encoder encoding, using the syntactic-attention mechanism to give higher attention scores to word pairs with close syntactic dependencies so that the encoder pays more attention to the syntactic structure. In each layer of encoding, a residual network structure is also introduced to pass the input of each layer to the next layer through residual connections, reducing the gradient vanishing problem and improving the ability to capture the details of the input text.

[0010] Step 4: Decoder decoding, continue to use the masked multi-head self-attention mechanism, so that the decoder only sees the previously generated words at each step. The interactive attention mechanism used at the same time will focus on the output of the encoder, and combined with the residual network, the decoder will gradually optimize the generated sentences. Syntactic information is further used as an auxiliary condition in the interactive attention to help the decoder generate output that conforms to syntactic rules.

[0011] Step 5: Beam Search tuning of syntactic information: Use Beam Search to generate multiple candidate sentences, then score them using syntactic rules, and finally select the most appropriate output.

[0012] 1. In the above step 1, the dataset is imported. Here, the NLPCC 2018+HSK dataset is selected. The dataset consists of two public Chinese text grammar correction datasets. The former is the dataset released by NLPCC (Natural Language Processing and Chinese Computing) in the Chinese grammar correction competition held in 2018, and the latter is the HSK (Chinese Proficiency Test) dynamic composition corpus, which collects a large number of composition texts of foreign students learning Chinese, including various types of grammatical errors. Through processing, the two datasets are combined into one, in the format of sentence pairs (source sentence-target sentence).

[0013] 2. In the above step 2, word embedding and syntactic information integration includes the following steps:

[0014] (1) Use the BERT word segmenter SentencePiece to segment and encode the sentence pairs, and obtain the BART default Chinese text word embedding e1, e2, e3...en .

[0015] (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n . Embed the subwords into i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i Assuming that the trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is:

[0016] H=α·E+(1-α)·P

[0017] Here, α is a hyperparameter that is usually adjusted in the range [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings.

[0018] (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency. Then a graph convolutional network (GCN) is used to perform convolution operations on the dependency graph. The initial representation H of the fusion part-of-speech tag is used as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep In the lth layer of GCN, the representation update formula for each word is:

[0019] H (l+1) =σ(AH (l) W (l) )

[0020] Among them, H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, and σ is the ReLU activation function. The representation obtained after L layers of GCN is H dep =H (L) The representation H after fusion of word embedding and part-of-speech tag and the representation H including dependency relationship dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as:

[0021] H final =β·H+(1-β)·H dep

[0022] 3. In the above step 3, the encoder encoding includes the following steps:

[0023] (1) Encoder self-attention processing. After obtaining the syntactically enhanced encoder input representation H final After that, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the sentence as a whole. First, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weights of each word with other words and apply them to the value vector to get the context-enhanced representation:

[0024]

[0025] where d k is the dimension of the key vector, used to scale the attention scores.

[0026] (2) Feedforward neural network layer and encoder output. After attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved:

[0027] H att =Attention(Q,K,V)+H final

[0028] The encoder’s self-attention output H att It is passed to the Feed Forward Network (FFN) layer and uses residual connections again to further retain the information after the self-attention mechanism:

[0029] Z=FFN(H att )+H att

[0030] FFN is a two-layer feedforward neural network. Ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

[0031] 4. In the above step 4, the decoder decoding includes the following steps:

[0032] (1) Decoder self-attention processing. Assume that we are currently generating the tth word, and the decoder input sequence (the embedding sequence of the currently generated word) is represented by Y <t express:

[0033] Y <t=[y1+pos1,y2+pos2,...,y t-1 +pos t-1 ]

[0034] At the decoder level l, a masked self-attention layer is applied, which generates an attention score for the currently generated word position. For the generation of the t-th word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as:

[0035]

[0036] The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position. This mask will block the position weights of future words, making the generation process sequentially dependent. The output will be processed through residual connections and unification to obtain a new representation:

[0037]

[0038] (2) Cross-Attention Processing of the Decoder. After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The cross-attention mechanism can help the decoder obtain the global context information about the entire sentence from the output of the encoder. The query matrix of the cross-attention layer comes from the output of the masked self-attention. The query, key, and value matrices are and The interactive attention calculation formula is:

[0039]

[0040] After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer:

[0041]

[0042] In order to further combine syntactic information, we introduce dependency word embedding H dep , and linearly fuse it with the interactive attention result. Assuming the fusion scale factor is γ, we have:

[0043]

[0044] (3) Feedforward neural network layer and decoder output. After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation:

[0045]

[0046] Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained:

[0047]

[0048] The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feed-forward network. (L) . It is then passed into the output layer to generate the probability distribution of each time step, thereby predicting the word at the current time step t:

[0049] P(y t |y <t )=softmax(Y t W out +b out )

[0050] Among them, Y t is the hidden layer representation generated by the decoder, W out is a mapping matrix that transforms Y t Mapped to the vocabulary space, b out is the bias term. The decoder generates words at each time step step by step until the termination condition or maximum length is reached.

[0051] 5. In step 5 above, the Beam Search tuning of syntactic information includes the following steps:

[0052] Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only depends on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. stntax To evaluate the grammatical correctness of each candidate sequence.

[0053] S syntax (y1,y2,...,y t )=f(P,H dep )

[0054] Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·S syntax (y1,y2,...,y t )

[0055] Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score.

[0056] Then, the selected B candidate paths are extended until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

[0057] The beneficial effects of the present invention are:

[0058] (1) A text correction method combining syntactic information and pre-trained language models is proposed, which makes Chinese text grammatical correction more accurate;

[0059] (2) The part-of-speech tags and dependencies are used as auxiliary information and fused with the default word embedding. The performance of the encoder is optimized through the syntactic-attention mechanism, which enables the model to understand the syntactic structure more accurately and improve the accuracy of grammatical error correction in Chinese text.

[0060] (3) The residual network structure is introduced in each layer to effectively alleviate the gradient vanishing problem and ensure the stable training of deep networks. At the same time, the residual connection is used to capture the detailed information of the input text, enhancing the ability to understand long texts and complex syntactic structures.

[0061] (4) The self-attention mechanism and the interactive attention mechanism are used in the decoder, combined with the syntactic information output by the encoder, to make the generated sentences more grammatical. This not only corrects grammatical errors but also generates more natural and fluent texts.

[0062] (5) With the help of Beam Search optimization, on the basis of generating multiple candidate sentences, the syntactic consistency and semantic rationality are comprehensively considered, and scoring and screening are carried out through syntactic rules to output the optimal candidate sentence.

[0063] This method significantly improves the grammatical standardization and semantic rationality of sentences generated by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a Chinese text grammar correction method that combines syntactic information and a pre-trained language model; DETAILED DESCRIPTION

[0065] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0066] The flowchart of the Chinese text grammar correction method combining syntactic information and pre-trained language model is as follows: Figure 1 As shown, the steps include:

[0067] Step 1: Import the dataset and convert the relevant dataset into the target format, that is, the sentence pair form of source sentence-target sentence.

[0068] Step 2: Word embedding integrates syntactic information. BERT word segmenter SentencePiece is used to perform word segmentation and encoding to obtain BART's default Chinese text word embedding. The natural language processing tool HanLP is used to generate syntactic information, including dependency syntax trees and part-of-speech tags. Part-of-speech tags and dependency relationships are converted into vector representations. The word vectors and part-of-speech tag vectors are passed through linear layers respectively, and then added together to obtain word embeddings that integrate syntactic information.

[0069] Step 3: Encoder encoding, using the syntactic-attention mechanism to give higher attention scores to word pairs with close syntactic dependencies so that the encoder pays more attention to the syntactic structure. In each layer of encoding, a residual network structure is also introduced to pass the input of each layer to the next layer through residual connections, reducing the gradient vanishing problem and improving the ability to capture the details of the input text.

[0070] Step 4: Decoder decoding, continue to use the masked multi-head self-attention mechanism, so that the decoder only sees the previously generated words at each step. The interactive attention mechanism used at the same time will focus on the output of the encoder, and combined with the residual network, the decoder will gradually optimize the generated sentences. Syntactic information is further used as an auxiliary condition in the interactive attention to help the decoder generate output that conforms to syntactic rules.

[0071] Step 5: Beam Search tuning of syntactic information: Use Beam Search to generate multiple candidate sentences, then score them using syntactic rules, and finally select the most appropriate output.

[0072] 1. In the above step 1, the dataset is imported. Here, the NLPCC 2018+HSK dataset is selected. The dataset consists of two public Chinese text grammar correction datasets. The former is the dataset released by NLPCC (Natural Language Processing and Chinese Computing) in the Chinese grammar correction competition held in 2018, and the latter is the HSK (Chinese Proficiency Test) dynamic composition corpus, which collects a large number of composition texts of foreign students learning Chinese, including various types of grammatical errors. Through processing, the two datasets are combined into one, in the format of sentence pairs (source sentence-target sentence).

[0073] 2. In the above step 2, word embedding and syntactic information integration includes the following steps:

[0074] (1) Use the BERT word segmenter SentencePiece to segment and encode the sentence pairs, and obtain the BART default Chinese text word embedding e1, e2, e3...e n .

[0075] (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n . Embed the subwords into i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i Assuming that the trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is:

[0076] H=α·E+(1-α)·P

[0077] Here, α is a hyperparameter that is usually adjusted in the range [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings.

[0078] (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency. Then a graph convolutional network (GCN) is used to perform convolution operations on the dependency graph. The initial representation H of the fusion part-of-speech tag is used as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep In the lth layer of GCN, the representation update formula for each word is:

[0079] H (l+1) =σ(AH (l) W (l) )

[0080] Among them, H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, and σ is the ReLU activation function. The representation obtained after L layers of GCN is H dep =H (L) The representation H after fusion of word embedding and part-of-speech tag and the representation H including dependency relationship dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as:

[0081] Hfinal =β·H+(1-β)·H dep

[0082] 3. In the above step 3, the encoder encoding includes the following steps:

[0083] (1) Encoder self-attention processing. After obtaining the syntactically enhanced encoder input representation H final After that, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the sentence as a whole. First, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weights of each word with other words and apply them to the value vector to get the context-enhanced representation:

[0084]

[0085] where d k is the dimension of the key vector, used to scale the attention scores.

[0086] (2) Feedforward neural network layer and encoder output. After attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved:

[0087] H att =Attention(Q,K,V)+H final

[0088] The encoder’s self-attention output H att It is passed to the Feed Forward Network (FFN) layer and uses residual connections again to further retain the information after the self-attention mechanism:

[0089] Z=FFN(H att )+H att

[0090] FFN is a two-layer feedforward neural network. Ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

[0091] 4. In the above step 4, the decoder decoding includes the following steps:

[0092] (1) Decoder self-attention processing. Assume that we are currently generating the tth word, and the decoder input sequence (the embedding sequence of the currently generated word) is represented by Y <t express:

[0093] Y <t =[y1+pos1,y2+pos2,...,y t-1 +pos t-1 ]

[0094] At the decoder level l, a masked self-attention layer is applied, which generates an attention score for the currently generated word position. For the generation of the t-th word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as:

[0095]

[0096] The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position. This mask will block the position weights of future words, making the generation process sequentially dependent. The output will be processed through residual connections and unification to obtain a new representation:

[0097]

[0098] (2) Cross-Attention Processing of the Decoder. After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The cross-attention mechanism can help the decoder obtain the global context information about the entire sentence from the output of the encoder. The query matrix of the cross-attention layer comes from the output of the masked self-attention. The query, key, and value matrices are and The interactive attention calculation formula is:

[0099]

[0100] After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer:

[0101]

[0102] In order to further combine syntactic information, we introduce dependency word embedding H dep , and linearly fuse it with the interactive attention result. Assuming the fusion scale factor is γ, we have:

[0103]

[0104] (3) Feedforward neural network layer and decoder output. After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation:

[0105]

[0106] Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained:

[0107]

[0108] The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feed-forward network. (L) . It is then passed into the output layer to generate the probability distribution of each time step, thereby predicting the word at the current time step t:

[0109] P(y t |y <t )=softmax(Y t W out +b out )

[0110] Among them, Y t is the hidden layer representation generated by the decoder, W out is a mapping matrix that transforms Y t Mapped to the vocabulary space, b out is the bias term. The decoder generates words at each time step step by step until the termination condition or maximum length is reached.

[0111] 5. In step 5 above, the Beam Search tuning of syntactic information includes the following steps:

[0112] Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only depends on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. stntax To evaluate the grammatical correctness of each candidate sequence.

[0113] S syntax (y1,y2,...,y t )=f(P,H dep )

[0114] Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·Ssyntax (y1,y2,...,y t )

[0115] Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score.

[0116] Then, the selected B candidate paths are extended until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

Claims

1. A Chinese text grammar correction method combining syntactic information and a pre-trained language model Chinese BART, characterized in that The following steps are involved: Step 1: Import the dataset and convert the relevant dataset into the target format, that is, the sentence pair form of source sentence-target sentence. Step 2: Word embedding integrates syntactic information. BERT word segmenter SentencePiece is used to perform word segmentation and encoding to obtain BART's default Chinese text word embedding. The natural language processing tool HanLP is used to generate syntactic information, including dependency syntax trees and part-of-speech tags. Part-of-speech tags and dependency relationships are converted into vector representations. The word vectors and part-of-speech tag vectors are passed through linear layers respectively, and then added together to obtain word embeddings that integrate syntactic information. Step 3: Encoder encoding, using the syntactic-attention mechanism to give higher attention scores to word pairs with close syntactic dependencies so that the encoder pays more attention to the syntactic structure. In each layer of encoding, a residual network structure is also introduced to pass the input of each layer to the next layer through residual connections, reducing the gradient vanishing problem and improving the ability to capture the details of the input text. Step 4: Decoder decoding, continue to use the masked multi-head self-attention mechanism, so that the decoder only sees the previously generated words at each generation step. The interactive attention mechanism used at the same time will focus on the output of the encoder, and combined with the residual network, the decoder will gradually optimize the generated sentences. Syntactic information is further used as an auxiliary condition in the interactive attention to help the decoder generate output that conforms to syntactic rules. Step 5: Beam Search tuning of syntactic information: Use Beam Search to generate multiple candidate sentences, then score them using syntactic rules, and finally select the most appropriate output.

2. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In the step 1, a data set is imported. Here, the NLPCC 2018+HSK data set is selected. The data set consists of two public Chinese text grammar correction data sets. The former is a data set released by NLPCC (Natural Language Processing and Chinese Computing) in the Chinese grammar correction competition held in 2018, and the latter is the HSK (Chinese Proficiency Test) dynamic composition corpus, which collects a large number of composition texts of foreign students learning Chinese, including various types of grammatical errors. Through processing, the two data sets are combined into one, in the format of sentence pairs (source sentence-target sentence).

3. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In the step 2, word embedding and syntactic information fusion includes the following steps: (1) Use the BERT word segmenter SentencePiece to segment and encode the sentence pairs, and obtain the BART default Chinese text word embedding e1, e2, e3...e n . (2) Generate part-of-speech tags using the natural language processing tool HanLP, and generate part-of-speech tag embeddings p1, p2, p3...p n . Embed the subwords into i and part-of-speech tag embedding p i Linearly combine to generate the final word embedding representation h i Assuming that the trainable fusion weight parameters α and (1-α) are used to control the weights of the two word embeddings, the embedding representation of the fusion of part-of-speech tags is: H=α·E+(1-α)·P Here, α is a hyperparameter that is usually adjusted in the range [0, 1] to balance the weights of word embeddings and part-of-speech tag embeddings. (3) Use the natural language processing tool HanLP to generate the syntactic dependency matrix A, which represents the dependency relationship between each word, where A ij =1 indicates that there is a syntactic dependency between word i and word j, A ij = 0 means that word i and word j have no direct dependency. Then a graph convolutional network (GCN) is used to perform convolution operations on the dependency graph. The initial representation H of the fusion part-of-speech tag is used as input, and the syntactically enhanced embedding representation H is obtained through GCN. dep In the lth layer of GCN, the representation update formula for each word is: H (l+1) =σ(AH (l) W (l) ) Among them, H (l) is the input representation of the lth layer (the initial layer H (0) =H), W (l) is the weight matrix of the lth layer, and σ is the ReLU activation function. The representation obtained after L layers of GCN is H dep =H (L) The representation H after fusion of word embedding and part-of-speech tag and the representation H including dependency relationship dep Further fusion, assuming the fusion weight is β, the final encoder input is expressed as: H final =β·H+(1-β)·H dep 。 4. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In step 3, the encoder encoding includes the following steps: (1) Encoder self-attention processing. After obtaining the syntactically enhanced encoder input representation H final After that, the BART encoder inputs the representation into the self-attention layer to capture the contextual information of the sentence as a whole. First, the representation of each word can generate a query vector Q = H through a trainable weight matrix final W Q , key vector K = H final W K Sum vector V = H final W V , then calculate the attention weights of each word with other words and apply them to the value vector to get the context-enhanced representation: where d k is the dimension of the key vector, used to scale the attention scores. (2) Feedforward neural network layer and encoder output. After attention calculation, the residual connection is applied to the self-attention result and the original input H final to ensure that the input information is preserved: H att =Attention(Q,K,V)+H final The encoder’s self-attention output H att It is passed to the FeedForwardNetwork (FFN) layer and the residual connection is used again to further retain the information after the self-attention mechanism: Z=FFN(H att ) + H att FFN is a two-layer feedforward neural network. Ultimately, the encoder’s output representation Z will integrate vocabulary, part of speech, dependency, and context information for the decoder to generate a corrected sentence.

5. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In step 4, the decoder decoding includes the following steps: (1) Decoder self-attention processing. Assume that we are currently generating the tth word, and the decoder input sequence (the embedding sequence of the currently generated word) is represented by Y <t express: Y <t =[y1+pos1,y2+pos2,...,y t-1 +post -1 ] At the decoder level l, a masked self-attention layer is applied, which generates an attention score for the currently generated word position. For the generation of the t-th word, the query, key, and value matrices of the self-attention mechanism are and The attention score is calculated as: The mask matrix M ensures that the t-th position can only "see" the words before the t-1-th position. This mask will block the position weights of future words, making the generation process sequentially dependent. The output will be processed through residual connections and unification to obtain a new representation: (2) Cross-Attention Processing of the Decoder. After the self-attention layer, the decoder obtains the context information of the encoder through cross-attention. The interactive attention mechanism can help the decoder obtain the global context information about the entire sentence from the encoder's output. The query matrix of the interactive attention layer comes from the output of the masked self-attention, and the query, key, and value matrices are and The interactive attention calculation formula is: After the interaction attention is calculated, a residual connection can be applied to the output of the interaction attention and the output of the self-attention layer: In order to further combine syntactic information, we introduce dependency word embedding H dep , and linearly fuse it with the interactive attention result. Assuming the fusion scale factor is γ, we have: (3) Feedforward neural network layer and decoder output. After the syntactic information is combined, the output will pass through a feedforward neural network layer and further processed into a high-level feature representation: Then, through residual connection and layer normalization, the final representation of the decoder layer l+1 is obtained: The final layer of the decoder represents Y = Y after all layers of masked self-attention, interactive attention and feed-forward network. (L) . It is then passed into the output layer to generate the probability distribution of each time step, thereby predicting the word at the current time step t: P(y t |y <t )=softmax(Y t W out +b out ) in, Y t is the hidden layer representation generated by the decoder, W out is a mapping matrix that transforms Y t Mapped to the vocabulary space, b out is the bias term. The decoder generates words at each time step step by step until the termination condition or maximum length is reached.

6. The Chinese text grammar error correction method combining syntactic information and the pre-trained language model Chinese BART according to claim 1 is characterized in that In step 5, the Beam Search tuning of the syntactic information includes the following steps: Generate candidate sequences starting from the initial word, and generate multiple candidate words y in each step t , according to the comprehensive score function Score(y1,y2,...,y t ) selects B optimal candidate paths. This function not only depends on the generation probability of the language model, but also adds a scoring function S based on syntactic rules. syntax To evaluate the grammatical correctness of each candidate sequence. S syntax (y1,y2,...,y t )=f(p,H dep ) Score(y1,y2,...,y t )=λ·P(y1,y2,...,y t )+(1-λ)·S syntax (y1,y2,...,y t ) Among them, λ is a hyperparameter used to balance the influence between generation probability and syntactic score. Then, the selected B candidate paths are extended until the generation is completed or the maximum length is reached. After the generation is completed, the candidate with the highest score is output as the final result.

Images