A Chinese grammar correction method based on Transformer and multi-task learning

Through the method based on Transformer and multi-task learning, a Chinese grammar error correction model is constructed, which solves the problems of slow Chinese grammar error correction speed and poor generalization ability in the existing technology, and achieves more efficient error correction results and better generalization ability to transform.

CN114881010BActive Publication Date: 2024-11-26SHANGHAI NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210452081.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-11-26
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

The existing Chinese grammar error correction technology has problems such as slow speed, high cost and poor generalization capabilities, and common methods are prone to error transmission, affecting the performance of the error correction system.

Method used

Using Chinese syntax correction methods based on Transformer and multi-task learning, collect corpus through crawler technology, preprocessing and editing operations to generate tags, build a multi-task learning model of Transformer and CRF layers, and combine it with pre-training models for fine-tuning training. Implement syntax error labeling and error correction.

Benefits of technology

The performance and generalization capabilities of the error correction model are improved, the error correction effect is significantly improved, error transmission is reduced, and calculation speed and model accuracy are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114881010B_ABST
    Figure CN114881010B_ABST
Patent Text Reader

Abstract

The present invention relates to a Chinese grammar error correction method based on Transformer and multi-task learning, the method comprising the following steps: step A: using crawler technology to collect written sentences and corresponding corrected sentences of non-native Chinese learners on the Lang‑8 website, cleaning and screening sentence pairs, and forming a corpus; step B: preprocessing the sentence pair data in the corpus; including removing stop words, converting traditional Chinese to simplified Chinese, and dividing training sets and verification sets; step C: generating labels required for training samples based on an edit operation set; step D: constructing a Chinese grammar error correction model based on Transformer and multi-task learning and using the training set for training; step E: verifying the trained Chinese grammar error correction model through a verification set, evaluating the error correction effect according to evaluation indicators, and using the model to input the sentence to be corrected for error correction. Compared with the prior art, the present invention has the advantages of high accuracy, strong generalization ability, and good performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of Chinese text error correction, and in particular to a Chinese grammar error correction method based on Transformer and multi-task learning. Background Art

[0002] In the Internet age, massive amounts of text data and text information are generated every day. People need to check and correct this data, otherwise it will interfere with subsequent important work. Conventional manual proofreading methods are slow and costly, and cannot adapt to the current era of information explosion. With the development of computer technology and natural language processing technology, academia and industry have begun to study automatic text proofreading to reduce the burden of manual proofreading. Early text proofreading used machine learning methods based on rules and statistics. As text data expanded, this method had the disadvantages of being inflexible and having poor generalization ability. With the development of deep learning, the mainstream has chosen more neural network models based on deep learning to implement text proofreading and conduct large-scale text data training.

[0003] Chinese grammar correction refers to proofreading grammatical errors in Chinese texts, including tasks such as error detection, error location, and error correction. There are two commonly used grammar correction solutions: one is to regard error correction as subtasks such as error detection, error labeling, and error correction, and integrate multiple sub-models. Error labeling usually uses a sequence labeling model, such as the BiLSTM+CRF classic model. This method is prone to error transmission, and the result of the previous sub-model affects the next sub-model, thereby affecting the performance of the entire error correction system; the other is to directly regard the error correction task as a "translation" task, "translating" the sentence to be corrected into the correct sentence. Usually, an end-to-end neural translation model (NMT) is selected to implement the error correction task, but the current error correction performance and speed cannot meet the requirements. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a Chinese grammar correction method based on Transformer and multi-task learning.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A Chinese grammar correction method based on Transformer and multi-task learning, the method comprising the following steps:

[0007] Step A: Use crawler technology to collect sentences written by non-native Chinese learners and corresponding corrected sentences on the Lang-8 website, clean and filter sentence pairs, and form a corpus;

[0008] Step B: Preprocess the sentences in the corpus, including removing stop words, converting traditional Chinese to simplified Chinese, and dividing the sentences into training and validation sets.

[0009] Step C: Generate labels required for training samples based on the edit operation set;

[0010] Step D: Build a Chinese grammar correction model based on Transformer and multi-task learning and train it using the training set;

[0011] Step E: The trained Chinese grammar correction model is verified through a verification set, the correction effect is evaluated according to the evaluation index, and the model is used to input the sentences to be corrected for correction.

[0012] In step A, the principles for cleaning and screening the statement pairs specifically include:

[0013] If the original sentence corresponds to more than one revised sentence, then one of the revised sentences and the original sentence are selected to form a sentence pair;

[0014] If the original sentence has no corresponding correction sentence, it means that the sentence is correct, and then copy it to form a sentence pair;

[0015] Select original sentences with a sentence length ranging from 20 to 90.

[0016] In the step B, the preprocessing includes removing stop words, converting traditional Chinese to simplified Chinese, and dividing the training set and the validation set. The removal of stop words is specifically as follows:

[0017] Remove non-Chinese characters including numbers, English characters, and random characters to improve the accuracy of the corpus data;

[0018] The traditional Chinese to simplified Chinese conversion is specifically:

[0019] Convert the traditional Chinese characters in the sentence to simplified Chinese characters.

[0020] The step C specifically comprises the following steps:

[0021] Step C1: Cut the sentence pairs into token sequences according to word granularity;

[0022] Step C2: Calculate the token-level edit distance between the sentence to be corrected and the corresponding correct sentence to obtain an edit operation set;

[0023] Step C3: According to the editing operation set, the grammatical errors are classified into four types: extra words, missing words, disorder and replacement, and the grammatical errors in the sentence sequence to be corrected are marked to obtain an error label sequence.

[0024] In the step C2, the edit distance is calculated using the toolkit MaxMatch (M2).

[0025] In the step D, the Chinese grammar correction model based on Transformer and multi-task learning includes a pre-training model, Transformer and CRF layers. The Encoder end of the Transformer is composed of an Embedding layer and N identical neural modules, each neural module is composed of a Multi-Head Attention and a Feed Forward sublayer, and each sublayer is followed by a residual connection. The Decoder end of the Transformer is composed of N identical neural modules, which has an additional encoding and decoding multi-head attention layer than the Encoder end.

[0026] The Chinese grammar correction model includes two subtasks: grammar error marking and grammar correction. The Encoder end and CRF constitute the grammar error marking task, and the Encoder end and Decoder end constitute the grammar correction task, and the two share the Encoder end hidden layer weight. The Encoder end Embedding layer input is the sequence of sentences to be corrected, and the Decoder end Embedding layer input is the correct sentence sequence. The CRF layer outputs the error labels existing in the sentence, and the Decoder end outputs the final result of the correction model, that is, the corrected sentence corresponding to the sentence to be corrected. The Embedding layer is used to map the input sentence sequence to a word vector, the Multi-Head Attention of the neural module is used to calculate attention, Feed Forward is used to perform linear transformation on the input data, the residual connection is used to prevent the model from overfitting and speed up the convergence speed, and the CRF layer is used to add constraints to ensure that the predicted label sequence is legal.

[0027] The pre-training model specifically adopts the open source Chinese pre-training model Chinese-RoBERTa-wwm-ext of Hugging Face.

[0028] In the step D, training the Chinese grammar error correction model based on Transformer and multi-task learning is specifically as follows:

[0029] The Transformer Encoder is initialized using the pre-trained model weights, and the other parts are randomly initialized. Hyperparameter fine-tuning training is performed. The hyperparameters include batch size, epoch, learning rate, and drop rate. During the training process, a batch of sentence pair data is input each time.

[0030] The step E is specifically as follows:

[0031] The trained Chinese grammar error correction model is verified using the validation set, and the model error correction effect is evaluated using evaluation indicators, which include precision, recall and comprehensive indicator F value.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. The present invention proposes a method for generating error labels based on an editing operation set, which increases the diversity of corpus data.

[0034] 2. The Transformer model based on the attention mechanism adopted in the present invention can better capture contextual semantic information, realize parallel processing, and speed up the calculation speed.

[0035] 3. The present invention proposes a Chinese grammar correction method based on multi-task learning, which further improves the performance and generalization ability of the correction model and enhances the correction effect compared with the single-task model.

[0036] 4. The present invention effectively accelerates the model convergence speed by combining fine-tuning training of the pre-trained model, thereby further improving the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a step structure flowchart of the present invention.

[0038] Figure 2 It is a flow chart of corpus preprocessing of the present invention.

[0039] Figure 3 This is a structural diagram of the Transformer model of the present invention.

[0040] Figure 4 This is a structural diagram of the Chinese grammar correction model based on Transformer and multi-task learning of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] like Figure 1 As shown, the present invention provides a Chinese grammar correction method based on Transformer and multi-task learning, comprising the following steps:

[0043] Step A: Use crawler technology to crawl the sentences written and reviewed by non-Chinese speaking users on the Lang-8 website, and clean and filter the sentence pairs according to the following principles to form a corpus:

[0044] ① If the original sentence corresponds to more than one revised sentence, select one of the revised sentences and the original sentence to form a sentence pair;

[0045] ② If the original sentence does not have a corresponding revised sentence, it means that the sentence is correct, and then copy it to form a sentence pair;

[0046] ③Select original sentences with a length ranging from 20 to 90 words.

[0047] After screening, the final corpus size is about 500,000.

[0048] Step B: Preprocess the sentences in the corpus by removing stop words, converting traditional Chinese to simplified Chinese, and dividing them into training sets and validation sets;

[0049] Further, the method comprises the following sub-steps:

[0050] ① Remove non-Chinese characters such as numbers, English, and random characters to improve the accuracy of corpus data;

[0051] ②Convert the traditional Chinese characters in the sentence into simplified Chinese characters;

[0052] ③ Divide the corpus into training set and validation set. Randomly select 5,000 sentence pairs from the corpus as the validation set, and the rest as the training set.

[0053] Step C: Generate labels required for training samples based on the edit operation set;

[0054] Further, the method comprises the following sub-steps:

[0055] ① Cut the sentence pairs into token sequences according to word granularity;

[0056] ② Calculate the token-level edit distance between the error to be corrected and the corresponding correct sentence to obtain the edit operation set;

[0057] ③ According to the editing operation set, the grammatical errors in the sentence sequence to be corrected are marked according to the four grammatical error classifications of multiple words (R), fewer words (M), disordered order (W), and substitution (S), and an error label sequence is generated.

[0058] Furthermore, the calculation of the edit distance in step C uses the public toolkit MaxMatch (M2).

[0059] Step D: Combine the pre-trained model to train the Chinese grammar correction model based on Transformer and multi-task learning;

[0060] Further,

[0061] The pre-trained model uses the open source Chinese pre-trained model provided by Hugging Face: Chinese-RoBERTa-wwm-ext. The Chinese grammar correction model structure based on Transformer and multi-task learning (such as Figure 4 As shown in Figure 2, the Transformer model (shown in Figure 2) consists of the Transformer encoder (Encoder), decoder (Decoder) and CRF part. Figure 3 The encoder (shown in the figure) consists of an Embedding layer and N identical neural modules. Each neural module consists of a Multi-Head Attention and a Feed Forward sublayer, and each sublayer is followed by a residual connection (Residual Connection); the decoder (Decoder) is also composed of N identical neural modules, with only an additional encoder-decoder multi-head attention layer (Scaled Dot-Product Attention) compared to the Encoder module.

[0062] like Figure 4 As shown in the figure, the multi-task learning model includes two subtasks: grammatical error labeling and grammatical error correction. The Encoder end and CRF constitute the grammatical error labeling task, and the Encoder end and Decoder end constitute the grammatical error correction task. The two share the hidden layer weight of the Encoder end. The input of the Embedding layer on the Encoder end is the sequence of sentences to be corrected, and the input of the Embedding layer on the Decoder end is the sequence of correct sentences. The CRF layer outputs the error labels in the sentence, and the output of the Decoder end is the final result of the error correction model, that is, the corrected sentence corresponding to the sentence to be corrected. The role of the Embedding layer is to map the input sequence into a word vector, the Multi-Head Attention of the neural module is used to calculate attention, Feed Forward is used to perform linear transformation on the input data, and the residual connection is to prevent the model from overfitting and speed up convergence. The role of the CRF layer is to add some constraints to ensure that the predicted label sequence is legal.

[0063] During the training process, the pre-trained model weights are used to initialize the Encoder side of the multi-task learning model, and the other parts are randomly initialized for fine-tuning training.

[0064] The hyperparameters of the model mainly include: batch size, epoch, learning rate and drop rate, etc. Adjust the appropriate hyperparameters to improve the performance of the model.

[0065] Step E: Verify the trained error correction model and evaluate the error correction effect based on the evaluation indicators.

[0066] Furthermore, the model evaluation indicators are: precision, recall and comprehensive evaluation indicator F value (F0.5). The method proposed in the present invention performs on the validation set: precision is 31.23, recall is 5.90, and F0.5 is 17.29. Compared with the single-task model (baseline) of the existing method, the performance on the validation set is: precision is 24.95, recall is 4.95, and F0.5 is 13.80, which has a significant improvement in precision and F value, further optimizing the model performance.

[0067] In summary, the Chinese grammar correction method based on Transformer and multi-task learning proposed in the present invention, on the basis of the Chinese grammar correction task training, adds the grammatical error labeling task, performs multi-task learning training, and combines the pre-trained model for fine-tuning training, which greatly improves the model accuracy and generalization ability, and further improves the error correction level.

[0068] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A Chinese grammar correction method based on Transformer and multi-task learning, characterized in that: The method comprises the following steps: Step A: Use crawler technology to collect sentences written by non-native Chinese learners and corresponding corrected sentences on the Lang-8 website, clean and filter sentence pairs, and form a corpus; Step B: Preprocess the sentences in the corpus, including removing stop words, converting traditional Chinese to simplified Chinese, and dividing the sentences into training and validation sets. Step C: Generate labels required for training samples based on the edit operation set; Step D: Build a Chinese grammar correction model based on Transformer and multi-task learning and train it using the training set; Step E: The trained Chinese grammar correction model is verified through a verification set, the correction effect is evaluated according to the evaluation index, and the model is used to input the sentence to be corrected for correction; The step C specifically comprises the following steps: Step C1: Cut the sentence pairs into token sequences according to word granularity; Step C2: Calculate the token-level edit distance between the sentence to be corrected and the corresponding correct sentence to obtain an edit operation set; Step C3: According to the editing operation set, the grammatical errors are classified into four types: extra words, missing words, disorder and replacement, and the grammatical errors in the sentence sequence to be corrected are marked to obtain an error label sequence.

2. According to claim 1, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: In step A, the principles for cleaning and screening the statement pairs specifically include: If the original sentence corresponds to more than one revised sentence, then one of the revised sentences and the original sentence are selected to form a sentence pair; If the original sentence has no corresponding correction sentence, it means that the sentence is correct, and then copy it to form a sentence pair; Select original sentences with a sentence length ranging from 20 to 90.

3. According to claim 1, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: In the step B, the preprocessing includes removing stop words, converting traditional Chinese to simplified Chinese, and dividing the training set and the validation set. The removal of stop words is specifically as follows: Remove non-Chinese characters including numbers, English characters, and random characters to improve the accuracy of the corpus data; The traditional Chinese to simplified Chinese conversion is specifically: Convert the traditional Chinese characters in the sentence to simplified Chinese characters.

4. According to claim 1, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: In the step C2, the edit distance is calculated using the toolkit MaxMatch (M2).

5. According to claim 1, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: In the step D, the Chinese grammar correction model based on Transformer and multi-task learning includes a pre-training model, Transformer and CRF layers. The Encoder end of the Transformer is composed of an Embedding layer and N identical neural modules, each neural module is composed of a Multi-Head Attention and a Feed Forward sublayer, and each sublayer is followed by a residual connection. The Decoder end of the Transformer is composed of N identical neural modules, which has an additional encoding and decoding multi-head attention layer than the Encoder end.

6. According to claim 5, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: The Chinese grammar correction model includes two subtasks: grammar error marking and grammar correction. The Encoder end and CRF constitute the grammar error marking task, and the Encoder end and Decoder end constitute the grammar correction task, and the two share the Encoder end hidden layer weight. The Encoder end Embedding layer input is the sequence of sentences to be corrected, and the Decoder end Embedding layer input is the correct sentence sequence. The CRF layer outputs the error labels existing in the sentence, and the Decoder end outputs the final result of the correction model, that is, the corrected sentence corresponding to the sentence to be corrected. The Embedding layer is used to map the input sentence sequence to a word vector, the Multi-Head Attention of the neural module is used to calculate attention, Feed Forward is used to perform linear transformation on the input data, the residual connection is used to prevent the model from overfitting and speed up the convergence speed, and the CRF layer is used to add constraints to ensure that the predicted label sequence is legal.

7. According to claim 5, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: The pre-training model specifically adopts the open source Chinese pre-training model Chinese-RoBERTa-wwm-ext of Hugging Face.

8. According to claim 7, a Chinese grammar correction method based on Transformer and multi-task learning is characterized in that: In the step D, training the Chinese grammar error correction model based on Transformer and multi-task learning is specifically as follows: The Transformer Encoder is initialized using the pre-trained model weights, and the other parts are randomly initialized. Hyperparameter fine-tuning training is performed. The hyperparameters include batch size, epoch, learning rate, and drop rate. During the training process, a batch of sentence pair data is input each time.

9. The Chinese grammar correction method based on Transformer and multi-task learning according to claim 1, characterized in that: The step E is specifically as follows: The trained Chinese grammar error correction model is verified using the validation set, and the model error correction effect is evaluated using evaluation indicators, which include precision, recall and comprehensive indicator F value.