A Chinese grammar correction method based on UniLM

Through the Chinese syntax error correction method based on UniLM, fine-tuning training is used to use crawler technology and pre-trained models to solve the problem of insufficient deep text grammar error correction performance in the existing technology, achieving more efficient error correction effects and lower errors judgment rate.

CN114881009BActive Publication Date: 2024-11-26SHANGHAI NORMAL UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210450638.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-11-26
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

The existing technology has insufficient performance in deep text grammar error correction, and traditional methods are difficult to meet the needs of efficient error correction, especially because of the problem that the error transmission of sub-model results affects the performance of the system.

Method used

Using the Chinese syntax correction method based on UniLM, the corpus is collected through crawler technology, preprocessed and constructed UniLM models, combined with pre-trained models such as Chinese-RoBERTa-wwm-ext for fine-tuning training, and using Multi-Head Attention and Feed Forward sublayer enhancement Context semantic information learning, simplifying sequence-to-sequence model structure to improve parallel computing efficiency.

Benefits of technology

It significantly improves the error correction effect, reduces the error judgment rate, improves the model's error correction performance and convergence speed, and shows higher accuracy, recall and F value.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114881009B_ABST
    Figure CN114881009B_ABST
Patent Text Reader

Abstract

The present invention relates to a Chinese grammar error correction method based on UniLM, comprising the following steps: step A: using crawler technology to collect written sentences and corresponding corrected sentences of non-native Chinese learners on the Lang‑8 website, and cleaning the sentences to generate a corpus; step B: pre-processing the corpus and dividing it into a training set and a verification set; step C: combining the pre-training model, fine-tuning the constructed Chinese grammar error correction model based on UniLM using the training set; step D: using the trained error correction model to verify using the verification set, and evaluating the error correction effect, and using the error correction model to input the sentence to be corrected for error correction. Compared with the prior art, the present invention has the advantages of high accuracy, low misjudgment rate, good performance, etc.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of Chinese text error correction, and in particular to a Chinese grammar error correction method based on UniLM. Background Art

[0002] In the Internet age, the world generates massive amounts of text data every moment, which contains many erroneous information. If it is not proofread, these erroneous data will have a great impact on subsequent work. Conventional manual proofreading can no longer keep up with the speed of text generation today. With the development of deep learning and natural language processing technology, academia and industry have launched research on text error correction. Text errors can be divided into shallow and deep levels. Spelling errors and punctuation errors belong to the former, while grammatical errors belong to the latter. Shallow errors can be corrected through rules and language models, while traditional machine learning-based correction methods are not satisfactory for deep errors. It can be seen that deep error correction is the core and difficulty of current text error correction technology.

[0003] At present, the main research direction of text grammatical error correction is based on deep learning, using neural network models to conduct large-scale grammatical error correction task training. There are two mainstream deep learning-based error correction schemes: one is to divide the error correction task into subtasks such as error judgment, error recognition, error labeling and error correction, and integrate multiple sub-models to achieve grammatical error correction, usually using the BiLSTM+CRF sequence labeling model. In this method, the quality of the results of a sub-model will affect the next sub-model, resulting in error transmission, thereby affecting the performance of the entire system; the other is to regard the error correction task as a "translation" task from an incorrect sentence to a correct sentence, select an end-to-end neural translation model, and directly implement grammatical error correction, but the current error correction performance and speed cannot meet the requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a Chinese grammar correction method based on UniLM in order to overcome the defects of the above-mentioned prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A Chinese grammar error correction method based on UniLM, the method comprising the following steps:

[0007] Step A: Use crawler technology to collect sentences written by non-native Chinese learners and corresponding corrected sentences on the Lang-8 website, and clean the sentences to generate a corpus;

[0008] Step B: Preprocess the corpus and divide it into training set and validation set;

[0009] Step C: Combined with the pre-trained model, the constructed Chinese grammar error correction model based on UniLM is fine-tuned using the training set;

[0010] Step D: Use the trained error correction model to verify with the validation set, evaluate the error correction effect, and use the error correction model to input the sentence to be corrected for correction.

[0011] In step A, the principles for cleaning and screening statements specifically include:

[0012] If each original sentence corresponds to more than one revised sentence, then one of the revised sentences is selected to form a sentence pair with the original sentence;

[0013] If the original sentence has no corresponding correction sentence, that is, the sentence is correct, then copy it to form a sentence pair;

[0014] The length of the original sentence and the revised sentence ranges from 10 to 80 words.

[0015] In the step B, the preprocessing includes removing stop words, converting traditional Chinese to simplified Chinese, and segmentation. The removal of stop words is specifically as follows:

[0016] Remove non-Chinese characters including numbers and English;

[0017] The traditional Chinese to simplified Chinese conversion is specifically:

[0018] Convert the traditional Chinese characters in the sentence to simplified Chinese characters;

[0019] The segmentation process is specifically as follows:

[0020] Sentences are segmented according to word granularity to generate model input sequences.

[0021] In the process of removing stop words, if the character is a number, it is replaced with a special character [num] representing a number. If the character is a non-Chinese language, it is replaced with a special character [eng] representing a non-Chinese language.

[0022] In the step C, the Chinese grammar correction model based on UniLM adopts a deep learning model, which is specifically composed of a pre-trained model, a UniLM part, and a Linear+Dropout layer. The UniLM part structure consists of an Embedding layer and N identical neural modules, and each module consists of a Multi-Head Attention and a Feed Forward sublayer, and each sublayer is followed by a residual connection.

[0023] The UniLM part is used to extract input semantic features, the Embedding layer is used to map the input sentence into a word vector form, the Multi-Head Attention sublayer of the neural module is used to calculate attention and integrate contextual semantic information, the Feed Forward sublayer is used to perform linear transformation on the input data, the residual connection is used to prevent the model from overfitting and speed up the convergence speed, the Linear layer is used to convert the input data features into output data features, and the Dropout layer is used to prevent the model from overfitting

[0024] The pre-training model uses the open source Chinese pre-training model Chinese-RoBERTa-wwm-ext of Hugging Face.

[0025] In the step C, training the Chinese grammar error correction model is specifically as follows:

[0026] The pre-trained model is used to initialize the Chinese grammar correction model, and the training set is used to fine-tune the hyperparameters. During the training process, a batch of sentence data is input each time.

[0027] The hyperparameters include batch size, epoch, learning rate, and drop rate.

[0028] In the step D, the trained Chinese grammar error correction model is verified, and the error correction effect is evaluated according to evaluation indicators, which include precision, recall and F value.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. The present invention simplifies the existing sequence-to-sequence model structure based on the UniLM model framework, and can better parallelize calculations based on the attention mechanism to improve error correction efficiency.

[0031] 2. The present invention adopts the UniLM model to enhance the learning ability of contextual semantic information and improve the error correction effect of the model.

[0032] 3. The present invention can accelerate the model convergence speed and further improve the performance of the model by combining fine-tuning training with the pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a step structure flowchart of the present invention.

[0034] Figure 2 It is a flow chart of corpus preprocessing of the present invention.

[0035] Figure 3 This is a partial model structure diagram of UniLM of the present invention.

[0036] Figure 4 This is a structural diagram of the error correction model based on UniLM of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] Example

[0039] like Figure 1 As shown, the present invention provides a Chinese grammar error correction method based on UniLM, comprising the following steps:

[0040] Step A: Use crawler technology to collect sentences written by non-native Chinese learners and corresponding corrected sentences on the Lang-8 website, and clean the sentences to generate a corpus.

[0041] Furthermore, the sentences are cleaned and screened according to the following principles to generate a corpus:

[0042] ① Since each original sentence corresponds to more than one revised sentence, select one of the revised sentences and the original sentence to form a sentence pair;

[0043] ② If the original sentence does not have a corresponding revised sentence, that is, the sentence is correct, then copy it and form a sentence pair;

[0044] ③Select original sentences and revised sentences with a length ranging from 10 to 80 words.

[0045] After screening, this example finally selected 526,502 sentence pairs from more than 1 million sentence pairs.

[0046] Step B: Preprocess the corpus, including removing stop words, converting traditional Chinese to simplified Chinese, and segmenting characters, and then divide it into training set and validation set;

[0047] Further, step B comprises the following steps:

[0048] Step B1, remove non-Chinese characters such as numbers and English. If the character is a number, replace it with the special character "[num]" representing a number; if the character is a non-Chinese language such as English or Japanese, replace it with the special character "[eng]" representing a non-Chinese language;

[0049] Step B2, converting the traditional Chinese characters in the sentence into simplified Chinese characters;

[0050] Step B3: Segment the sentence according to word granularity to generate a model input sequence;

[0051] Step B4: Divide the preprocessed corpus into a training set and a validation set, where the training set has 520,761 sentence pairs and the validation set has 5,741 sentence pairs.

[0052] Step C: Use the pre-trained model weights to initialize the Chinese grammar error correction model based on UniLM, adjust the model hyperparameters, and perform fine-tuning training;

[0053] Further,

[0054] The pre-trained model uses the open source Chinese pre-trained model provided by Hugging Face: Chinese-RoBERTa-wwm-ext, and the error correction model structure (such as Figure 4 As shown in Figure 2, the UniLM part consists of a UniLM part and a Linear+Dropout layer, where the UniLM part structure (as shown in Figure 2) consists of a UniLM part and a Linear+Dropout layer. Figure 3 As shown in the figure, it consists of an Embedding layer and N identical neural modules. Each module consists of a Multi-Head Attention sublayer and a Feed Forward sublayer, and each sublayer is followed by a residual connection.

[0055] The UniLM part is used to extract input semantic features, and its Embedding layer is used to map the input sentence into a word vector form. The Multi-Head Attention sublayer of the neural module is used to calculate attention and fuse contextual semantic information. The FeedForward sublayer is used to perform linear transformation on the input data. The residual connection is to prevent the model from overfitting and speed up convergence. The Linear layer is used to convert the input data features into output data features, and the Dropout layer is to prevent the model from overfitting.

[0056] The hyperparameters of the model mainly include: batch size, epoch, learning rate and drop rate, etc. Adjust the appropriate hyperparameters to improve the performance of the model.

[0057] Step D: Use the trained error correction model for verification and evaluate the error correction effect.

[0058] In this example, the model evaluation indicators include precision, recall and comprehensive evaluation indicator F value (F0.5). The method proposed in the present invention performs well on the validation set: precision is 29.94, recall is 5.80, and F0.5 is 16.34. Compared with the performance of the baseline model of the existing method on the validation set: precision is 22.25, recall is 3.98, and F0.5 is 11.61, there is a significant improvement, which reduces the false positive rate of error correction.

[0059] In summary, the Chinese grammar correction method based on UniLM proposed in the present invention uses UniLM as the overall framework and combines it with the pre-trained model to fine-tune the correction model, which greatly improves the level of error correction and reduces the misjudgment rate of error correction.

[0060] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A Chinese grammar correction method based on UniLM, characterized in that: The method comprises the following steps: Step A: Use crawler technology to collect sentences written by non-native Chinese learners and corresponding corrected sentences on the Lang-8 website, and clean the sentences to generate a corpus; Step B: Preprocess the corpus and divide it into training set and validation set; Step C: Combined with the pre-trained model, the constructed Chinese grammar error correction model based on UniLM is fine-tuned using the training set; Step D: Use the trained error correction model to verify with the validation set, evaluate the error correction effect, and use the error correction model to input the sentence to be corrected for error correction; In the step C, the Chinese grammar error correction model based on UniLM adopts a deep learning model, which is specifically composed of a pre-training model, a UniLM part, and a Linear+Dropout layer. The UniLM part structure consists of an Embedding layer and N identical neural modules, and each module consists of a Multi-Head Attention and a Feed Forward sublayer, and each sublayer is followed by a residual connection. The UniLM part is used to extract input semantic features, the Embedding layer is used to map the input sentence into a word vector form, the Multi-Head Attention sublayer of the neural module is used to calculate attention and fuse contextual semantic information, the Feed Forward sublayer is used to perform linear transformation on the input data, the residual connection is used to prevent the model from overfitting and speed up the convergence speed, the Linear layer is used to convert the input data features into output data features, and the Dropout layer is used to prevent the model from overfitting.

2. According to a UniLM-based Chinese grammar correction method according to claim 1, it is characterized in that: In step A, the principles for cleaning and screening statements specifically include: If each original sentence corresponds to more than one revised sentence, then one of the revised sentences is selected to form a sentence pair with the original sentence; If the original sentence has no corresponding correction sentence, that is, the sentence is correct, then copy it to form a sentence pair; The length of the original sentence and the revised sentence ranges from 10 to 80 words.

3. The Chinese grammar error correction method based on UniLM according to claim 1, characterized in that: In the step B, the preprocessing includes removing stop words, converting traditional Chinese to simplified Chinese, and segmentation. The removal of stop words is specifically as follows: Remove non-Chinese characters including numbers and English; The conversion from traditional Chinese to simplified Chinese is specifically as follows: Convert the traditional Chinese characters in the sentence to simplified Chinese characters; The segmentation process is specifically as follows: Sentences are segmented according to word granularity to generate model input sequences.

4. The Chinese grammar error correction method based on UniLM according to claim 3, characterized in that: In the process of removing stop words, if the character is a number, it is replaced with a special character [num] representing a number. If the character is a non-Chinese language, it is replaced with a special character [eng] representing a non-Chinese language.

5. The Chinese grammar error correction method based on UniLM according to claim 1, characterized in that: The pre-training model uses the open source Chinese pre-training model Chinese-RoBERTa-wwm-ext of Hugging Face.

6. The Chinese grammar error correction method based on UniLM according to claim 1, characterized in that: In the step C, training the Chinese grammar error correction model is specifically as follows: The pre-trained model is used to initialize the Chinese grammar correction model, and the training set is used to fine-tune the hyperparameters. During the training process, a batch of sentence data is input each time.

7. A Chinese grammar error correction method based on UniLM according to claim 6, characterized in that: The hyperparameters include batch size, epoch, learning rate, and drop rate.

8. The Chinese grammar error correction method based on UniLM according to claim 1, characterized in that: In the step D, the trained Chinese grammar error correction model is verified, and the error correction effect is evaluated according to evaluation indicators, which include precision, recall and F value.

Citation Information

Patent Citations

  • Chinese text error correction system, method, device and computer readable storage medium

    CN111639489A

  • Chinese text error correction method and device, electronic equipment and storage medium

    CN113642316A

  • Chinese grammar error correction method and device based on data enhancement

    CN113822054A

  • Method and apparatus for recognizing chinese grammatical errors, computer device and storage medium

    WO2021072851A1