A Chinese text grammatical error detection method based on ELECTRA-GCNN-CRF model

Abstract

The present invention is a method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model. The present invention relates to the technical field of natural language processing. The present invention regards text grammatical error detection as a sequence labeling task, and uses the ELECTRA pre-trained language model to replace the BERT model in the input layer to avoid the problem of data mismatch between the pre-training stage and the fine-tuning stage; obtains local information of the text through GCNN to reduce the impact of grammatical errors on contextual semantics; obtains a label sequence representing the range and type of grammatical errors contained in the text through CRF, and finally outputs the grammatical error detection result. The present invention has strong versatility, does not require manual extraction of lexical and syntactic features, saves manpower and time costs, has good effects, and meets the needs of automatic detection of Chinese grammatical errors.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and is a method for detecting grammatical errors in Chinese text based on an ELECTRA-GCNN-CRF model. Background Art

[0002] With the widespread use of the internet, the volume of Chinese electronic texts has exploded. Due to the varying quality of texts, grammatical errors have severely impacted reading efficiency. Faced with this massive volume of Chinese text, how to quickly and efficiently automatically detect grammatical errors has become a pressing issue. The goal of Chinese Grammatical Error Diagnosis (CGED) is to automatically detect grammatical errors in Chinese sentences, such as missing or redundant components and improper word order. For a given text, CGED's detection tasks generally include determining whether there are errors, the type of error, and the location of the error. Chinese grammatical error detection is highly valuable for assisting Chinese writing and article proofreading. In Chinese assisted writing, with the help of a Chinese grammatical error detection system, learners can quickly locate the points and types of errors, revise their articles, and improve their writing efficiency. Furthermore, Chinese grammatical error detection tools can save reviewers a significant amount of time and improve the efficiency of proofreading in the publishing industry.

[0003] Traditional research on grammatical error detection has primarily focused on English. Compared to English, Chinese lacks significant word boundaries, nor does it have identifying elements such as tense, singular, and plural. Its grammar is highly complex and contains rich semantic information. Currently, many researchers have used generative methods, drawing on English grammatical error correction methods, to directly correct errors, skipping the error detection step. Only a small number of studies have employed sequence labeling methods for Chinese grammatical error detection. However, generative methods often start from global semantics, ignoring the role that local information in Chinese text can play in grammatical error detection. Furthermore, generative methods have issues such as the large amount of data required, difficulty in training, and poor reliability, making them unsuitable for Chinese grammatical error detection tasks. Therefore, how to fully utilize local text information to improve the effectiveness of Chinese grammatical error detection when data is limited is a key challenge in this research. Summary of the Invention

[0004] In order to highlight text features and improve the accuracy of grammatical error detection, the present invention provides a Chinese text grammatical error detection method based on the ELECTRA-GCNN-CRF model. The present invention provides the following technical solutions:

[0005] A method for detecting grammatical errors in Chinese text based on an ELECTRA-GCNN-CRF model, the method comprising:

[0006] Step 1: Replace the BERT model with the ELECTRA pre-trained language model at the input layer, and take the discriminator as the final ELECTRA pre-trained language model;

[0007] Step 2: Build a GCNN network and add gating and residual mechanisms to the convolutional network to obtain local information of the text and reduce the impact of grammatical errors on contextual semantics;

[0008] Step 3: Obtain the dependency of text labels through CRF;

[0009] Step 4: Output the grammatical error detection result by outputting a triplet of the starting position, the ending position and the grammatical error type format.

[0010] Preferably, the step 1 is specifically:

[0011] The input layer is used to vectorize the text. The ELECTRA model is used instead of the BERT model to improve the model's grammatical error detection capability. The ELECTRA model consists of two parts: a generator and a discriminator. The generator is a small MLM model responsible for randomly replacing the input tokens. The discriminator determines whether the generator's output has been replaced. Finally, the discriminator is used as the final ELECTRA pre-trained language model.

[0012] Preferably, the step 2 is specifically as follows:

[0013] The word vectors corresponding to each word in each sentence are directly used as input and further processed by the residual gated convolutional neural network. The residual gated convolutional neural network consists of multiple GCNN units, each of which contains a convolution kernel with a gating mechanism and a residual unit.

[0014] The operations within the GCNN unit are expressed as follows:

[0015]

[0016]

[0017] Among them, H represents the hidden layer vector obtained from the input layer, Conv represents the convolution operation, σ represents the sigmoid activation function, Represents the Hadamard product of the vector, C represents the local feature vector after the gated convolution unit, LayerNorm represents layer normalization, Represents vector addition, and Z represents the text feature vector obtained by GCNN.

[0018] Preferably, the step 3 is specifically:

[0019] The CRF layer outputs labels based on the extracted text feature vectors. The CRF model calculates the probability score through the feature transfer function to determine the dependency relationship between the current position and the labels of adjacent positions.

[0020] The present invention has the following beneficial effects:

[0021] The present invention uses the ELECTRA pre-trained language model to avoid the problem of data mismatch between the pre-training stage and the fine-tuning stage of the traditional pre-trained language model; proposes a text feature extraction method that is different from the traditional method, and extracts the local semantics and position information of the text based on the global semantic information of the text through the residual gated convolutional neural network, effectively solving the problem of ignoring local information in the traditional Chinese text error detection method. It does not require manual extraction of lexical and syntactic features, saves manpower and time costs, improves the text error detection effect, and meets the needs of automatic detection of Chinese grammatical errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall structural diagram of the present invention;

[0023] Figure 2 Pre-trained language model training process for ELECTRA;

[0024] Figure 3 This is the GCNN network structure diagram;

[0025] Figure 4 The figure is the experimental result. DETAILED DESCRIPTION

[0026] The present invention is described in detail below with reference to specific embodiments. Specific embodiment one:

[0028] according to Figures 1 to 4 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention relates to a Chinese text grammatical error detection method based on the ELECTRA-GCNN-CRF model.

[0029] Step 1: Replace the BERT model with the ELECTRA pre-trained language model at the input layer, and take the discriminator as the final ELECTRA pre-trained language model;

[0030] The step 1 is specifically as follows:

[0031] The input layer vectorizes the text and uses the ELECTRA model instead of the BERT model to improve its grammatical error detection capabilities. The ELECTRA model consists of a generator and a discriminator. The generator is a small MLM model that randomly replaces input tokens. The discriminator determines whether the generator's output has been replaced, and the discriminator is used as the final ELECTRA pre-trained language model. This pre-training approach avoids the data mismatch between the pre-training and fine-tuning phases caused by the [MASK] tag, significantly improves training efficiency, and makes the ELECTRA model particularly sensitive to semantic changes in text, making it suitable for grammatical error detection tasks, with performance significantly higher than that of the BERT model.

[0032] Since the text contains grammatical errors, traditional contextual semantic extraction methods such as those based on LSTM networks will be affected by grammatical errors, which is not conducive to grammatical error detection tasks. Therefore, the present invention uses a residual gated convolutional network (GCNN) to extract the local semantics of the text and alleviate the impact of grammatical errors on text feature extraction. The GCNN structure is as follows: Figure 3 As shown in Figure 1, it consists of convolution units with kernel lengths of 5 and 3. The addition of GLU gating mechanism and residual mechanism in the convolution unit enables it to better capture text features and reduce the impact of invalid information.

[0033] Step 2: Build a GCNN network and add gating and residual mechanisms to the convolutional network to obtain local information of the text and reduce the impact of grammatical errors on contextual semantics;

[0034] The step 2 is specifically as follows:

[0035] The word vectors corresponding to each word in each sentence are directly used as input and further processed by the residual gated convolutional neural network. The residual gated convolutional neural network consists of multiple GCNN units, each of which contains a convolution kernel with a gating mechanism and a residual unit.

[0036] The operations within the GCNN unit are expressed as follows:

[0037]

[0038]

[0039] Among them, H represents the hidden layer vector obtained from the input layer, Conv represents the convolution operation, σ represents the sigmoid activation function, Represents the Hadamard product of the vector, C represents the local feature vector after the gated convolution unit, LayerNorm represents layer normalization, Represents vector addition, and Z represents the text feature vector obtained by GCNN.

[0040] Step 3: The CRF layer outputs labels based on the extracted text feature vectors. The CRF model calculates probability scores using a feature transfer function to determine the dependency between the labels at the current position and those at adjacent positions. This calculation helps resolve issues where the output sequence doesn't conform to labeling rules. For example, a "BS" label cannot be followed by a "BW" label, but can only be followed by an "IS" or "O" label.

[0041] Step 4: Output the grammatical error detection result in the format of a triple of the starting position, the ending position, and the grammatical error type, making the output more intuitive.

[0042] In order to verify the effectiveness of the present invention, experiments were conducted using the NLPTEA Chinese grammatical error detection dataset. This dataset is a manually annotated grammatical error dataset, and the corpus source is error samples generated by learners whose native language is not Chinese in Chinese writing. This dataset divides grammatical errors into four types: Selection error (denoted as "S", i.e., improper use of words), Redundant error (denoted as "R", i.e., redundant components), Missing error (denoted as "M", i.e., missing components) and Word ordering error (denoted as "W", i.e., improper word order). For each grammatical error, a triple in the format of "starting position, ending position, grammatical error type" is generated. If there is no grammatical error in the sentence, "correct" is output. The sentences in the dataset may have no grammatical errors, or may contain one or more grammatical errors. The data size is shown in Table 1.

[0043] Table 1 Dataset size

[0044] type training set Validation set Test set sentence 31178 7795 1457 character 1417950 351986 55313 S type error 30420 7517 1695 R Type Error 14932 3724 768 M type error 20530 5165 864 WTypeError 4823 1222 327

[0045] This example uses the F1 score to evaluate model performance. The hidden layer dimension is 768. The Transformer architecture of the ELECTRA and BERT pre-trained models is 12 layers. The multi-head attention mechanism has 12 heads, and the batch size is 240. The Adam optimizer is used with a dropout rate of 0.15 and a maximum number of iterations of 20. Early stopping is also used to mitigate overfitting. The learning rate for the CRF layer is 10-3, and the learning rate for other components is 10-5. The convolution kernel widths for the GCNN layer are 5 and 3, respectively.

[0046] This example sets up 4 groups of control experiments under the same environment to verify the effectiveness of this method:

[0047] (1) ELECTRA-softmax: only uses the ELECTRA pre-trained language model;

[0048] (2) ELECTRA-CRF: uses the ELECTRA pre-trained model and CRF layer;

[0049] (3) BERT-GCNN-CRF: The pre-trained language model of this method is replaced with the BERT-base pre-trained model, and the rest remains unchanged;

[0050] (4) ELECTRA-BiLSTM-CRF: The GCNN layer of the proposed model is replaced with a 2-layer BiLSTM neural network, while the rest of the model remains unchanged;

[0051] (5) ELECTRA-GCNN-CRF: A Chinese grammatical error detection method proposed by this invention.

[0052] The experimental results are as follows Figure 4 This method achieved the highest F1 score among all models, indicating that ELECTRA's adversarial training method is more sensitive to semantic changes and is more suitable for Chinese grammatical error detection tasks than BERT. The GCNN network can better extract text features than the BiLSTM network, mitigating the impact of grammatical errors on contextual semantics.

[0053] This paper approaches text grammatical error detection as a sequence labeling task. It replaces the BERT model with the ELECTRA pre-trained language model at the input layer to avoid data mismatches between the pre-training and fine-tuning phases. It also uses GCNN to capture local information about the text, mitigating the impact of grammatical errors on contextual semantics. It also uses CRF to generate a sequence of labels representing the range and type of grammatical errors contained in the text, ultimately outputting the grammatical error detection results. This paper is highly versatile and eliminates the need for manual extraction of lexical and syntactic features, saving both manpower and time. It delivers excellent results and meets the demand for automatic grammatical error detection in Chinese.

[0054] The above is only a preferred embodiment of a method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model. The scope of protection of a method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model is not limited to the above embodiment. All technical solutions under this concept fall within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, several improvements and variations without departing from the principles of the present invention should also be considered as the scope of protection of the present invention.

Claims

1. A method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model, characterized by: The method comprises: Step 1: Replace the BERT model with the ELECTRA pre-trained language model at the input layer, and take the discriminator as the final ELECTRA pre-trained language model; Step 2: Build a GCNN network and add gating and residual mechanisms to the convolutional network to obtain local information of the text and reduce the impact of grammatical errors on contextual semantics; The step 2 is specifically as follows: The word vectors corresponding to each word in each sentence are directly used as input and further processed by the residual gated convolutional neural network. The residual gated convolutional neural network consists of multiple GCNN units, each of which contains a convolution kernel with a gating mechanism and a residual unit. The operations within the GCNN unit are expressed as follows: in, represents the hidden layer vector obtained from the input layer, represents the convolution operation, represents the sigmoid activation function, represents the Hadamard product of vectors, represents the local feature vector after the gated convolution unit, Representation layer normalization, represents vector addition, Represents the text feature vector obtained by GCNN; Step 3: Obtain the dependency of text labels through CRF; Step 4: Output a triplet of the start position, end position, and syntax error type format, and use the triplet as the output syntax error detection result.

2. The method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model according to claim 1, wherein: The step 1 is specifically as follows: The input layer is used to vectorize the text. The ELECTRA model is used instead of the BERT model to improve the model's grammatical error detection capability. The ELECTRA model consists of two parts: a generator and a discriminator. The generator is a small MLM model responsible for randomly replacing the input tokens. The discriminator determines whether the generator's output has been replaced. Finally, the discriminator is used as the final ELECTRA pre-trained language model.

3. The method for detecting grammatical errors in Chinese text based on the ELECTRA-GCNN-CRF model according to claim 2, wherein: The step 3 is specifically as follows: The CRF layer outputs labels based on the extracted text feature vectors. The CRF model calculates the probability score through the feature transfer function to determine the dependency relationship between the current position and the labels of adjacent positions.

Images