A Chinese character correction method and system based on attribution network and BERT

Abstract

The present invention discloses a Chinese character correction method based on an attribution network and BERT. The method comprises the following steps: collecting a Chinese text dataset containing erroneous characters, and dividing the dataset into a training set and a test set for model training and testing, or using the complete dataset for training; training a BERT-based binary classification model used as an attribution network; using the trained binary classification model to perform feedforward calculations on sentences in the training set, with the output denoted as F(X); using the output F(X) to calculate the gradient of X to obtain error attribution information; setting a threshold filter to process the error attribution information obtained above; integrating the filtered error attribution information into a second BERT model for character correction training; and predicting the target character of the erroneous character. The present invention also discloses a system for implementing the above method. The method of the present invention can significantly improve the precision and recall rates in the field of Chinese text error correction, while having strong versatility and scalability.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and deep learning technology, and relates to a Chinese character correction method and system based on attribution network and BERT. Background Art

[0002] Correcting Chinese character spelling is a valuable research endeavor with widespread real-life applications. For example, automatic correction of search keywords on search platforms: users may enter incorrect characters that are similar in sound and form when searching. If the platform can correctly identify and correct the incorrect characters, it can eliminate the interference caused by spelling errors. Another example is providing spelling correction feedback to Chinese beginners: For Chinese beginners, Chinese character correction can help them reduce the chance of making typos. Furthermore, when generating text using technologies like speech recognition and optical character recognition, many typos are generated, requiring further correction.

[0003] Currently, the first type of Chinese character error correction tools is based on statistical language models. For example, Pycorrector uses methods such as word segmentation and common dictionary matching to detect errors. It then uses confusion set replacement and statistical language model perplexity calculation to recall incorrect characters. This method is simple and easy to use, but it relies heavily on the constructed dictionary and the quality of the confusion set, and lacks flexibility.

[0004] The second category is based on RNN models. Early approaches used RNNs to assess the perplexity of sentences or phrases to detect errors, but these methods struggled to accurately correct errors within sentences. To address this issue, researchers adopted a more flexible encoder-decoder model, incorporating an attention mechanism. A pointer network was used on the decoder side to replicate similar characters from the confusion set, achieving more accurate recall. These methods are end-to-end, and during decoding and generation, it's difficult to fully guarantee that the generated sentences retain complete and coherent semantics.

[0005] The third category is based on pre-trained language models. With the rise of pre-trained models, such as BERT, which explicitly model relationships between characters, they have become a powerful tool for solving the problem of spelling correction. Various attempts have been made based on BERT, such as using phonetic and glyph information to filter BERT's candidate sets, or using graph neural networks to integrate confusion set information to classify BERT-encoded characters. While these attempts have achieved some success, they still face challenges such as insufficient detection capabilities and the difficulty of constructing high-quality confusion sets. Summary of the Invention

[0006] To address the shortcomings of the prior art, the present invention provides a Chinese character correction method based on an attribution network and BERT. The method proposes using a binary classification model with true / false labels to construct a BERT-based attribution network, which is used to determine the probability of a sentence containing an incorrect character and extract information about the character errors in the sentence. This information is then incorporated into BERT's self-attention score calculation, allowing BERT to pay more attention to incorrect characters, thereby improving the model's ability to detect and correct incorrect characters. This significantly improves precision and recall without using confusion sets, making the model extremely flexible and transferable.

[0007] The Chinese character correction method based on attribution network and BERT proposed in the present invention includes the following steps:

[0008] Step 1: Collect a Chinese text dataset containing incorrect characters and divide it into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model's performance. You can also use the entire dataset for training without dividing it.

[0009] Step 2: Train a BERT-based binary classification model for the attribution network.

[0010] Step 3: Use the binary classification model obtained in step 2 to perform feedforward calculation on the sentences in the training set, and the output is recorded as F(X), X = x 1:n is the input sentence, n is the number of Chinese characters;

[0011] Step 4: Take the gradient of the output F(X) in step 3 with respect to X to obtain the error attribution information;

[0012] Step 5: Set a threshold filter to process the error attribution information obtained in step 4;

[0013] Step 6: Integrate the error attribution information filtered in step 5 into the second BERT model, and use the correct and incorrect sentence pairs of the Chinese dataset to perform character correction training;

[0014] Step 7: Inference stage: When a new sentence that needs Chinese character correction is obtained, the BERT model in step 2 is first used to calculate the error attribution information of the sentence. Then, the BERT model in step 6 is used to predict the target characters of the incorrect characters in the sentence in combination with the error attribution information.

[0015] In step 1, the Chinese text dataset is from the Chinese Spell Check (CSC) dataset Sighan13 and additional data containing incorrect characters automatically generated by a specific tool OpenCC; the Chinese dataset includes both correct and incorrect sentences.

[0016] In step 2, the binary classification model is used to predict whether there is an error in the sentence (1 indicates an error, 0 indicates no error), and the final output result is the probability that the sentence contains an error character; the attribution network refers to the trained binary classification model. For a given input X = x 1:n And the binary classification model output F(X), use the gradient to calculate the components in X that play a key role in the model output F(X), that is, which characters in X are responsible for the prediction result, n is the number of Chinese characters in X.

[0017] The formula for finding the gradient is in, It is a vector with the same dimension as x, i represents the ordinal number of the input component in the model, and n represents the number of Chinese characters in the input sentence.

[0018] In step 4, the gradient of the output result F(X) is calculated to obtain the attribution information vector of each character in X. The attribution information vector is L2 normalized and summed and normalized to obtain the attribution information scalar, that is, the probability of error for each character in X.

[0019] In step 5, the processing operation of the threshold filter is to reset the probabilities of the characters whose probabilities obtained in step 4 are less than 0.5 to 0.

[0020] In step 6, the incorporation operation refers to adding the filtered error attribution information to the self-attention score calculation of each layer of the second BERT model, strengthening the information of the erroneous characters themselves, so that the model focuses on the erroneous characters to achieve better detection and recall effects; the BERT self-attention calculation formula is:

[0021]

[0022] Among them, Q, K, and V are the three matrices of Query, Key, and Value transformed from the model input X, respectively. k is the number of columns of the Q, K matrix, QK T It is an n×n matrix, each row represents each character x in the sentence X i The correlation between (i≤n) and all characters including itself, Scores is also an n×n matrix, each row represents the error probability of each character in X, and the n row vectors of Scores are exactly the same.

[0023] In step 6, the loss function of the model training adopts the following cross entropy loss function:

[0024]

[0025] in, is the predicted character of the model, y i For the correct character, l cel is the cross entropy function.

[0026] In step 7, for each input character x i , the prediction formula of the target character is:

[0027]

[0028] in, represents the predicted probability distribution of all candidate characters, w represents the parameter matrix, e i Represents x i The representation after BERT encoding; finally, the character with the largest Softmax value is used as the predicted target character.

[0029] The method is as follows: First, a dataset is collected, and then an attribution network is designed. In layman's terms, given an input X and a model F(X), the attribution network finds which components of the model input X are critical to the model's prediction. This method employs a simple and effective attribution technique, directly calculating the gradient of the model's predicted output. The formula is: From a mathematical point of view, according to Taylor expansion:

[0030]

[0031] because is a vector of the same dimension as x, is its i-th component, and for Δx of the same size i , The larger the absolute value of , the greater the change of F(x+Δx) relative to F(x), that is: It measures the sensitivity of the model to the i-th component of the input, and can be used As a measure of the importance of the i-th component. Based on this, the present invention first uses a small amount of data to train / fine-tune a BERT-based binary classification model. This binary classification model can predict whether there are errors in a sentence. Then, the gradient of the predicted error probability is calculated to obtain the attribution information vector of each character, that is, the contribution of each character to the sentence error. These vectors are L2-normalized and summed, and then all L2 sums are normalized. The resulting score can be regarded as the probability of error for each character.

[0032] Then, a threshold filter is used to reset the probability of characters with an error probability less than 0.5 in the normalized attribution detection information to 0, thereby reducing the interference of correct characters on the detection of incorrect characters in the subsequent model and improving the model detection capability.

[0033] After extracting the attribution detection information, it is fused with the self-attention score of each layer of the BERT model to strengthen the information of the incorrect characters themselves, so that the model focuses on those incorrect characters to achieve better detection and recall effects. The specific formula is:

[0034]

[0035] Where Q, K, and V are the Query, Key, and Value matrices transformed by the model input X, respectively. k is the number of columns of the Q, K matrix, QK T and Scores are both n×n matrices. Each row of Scores represents the error probability of each character in sentence X. QK T Each line represents each character x in sentence X i The connection between other characters including itself, all rows of Scores are the same. It can be seen from the formula that, assuming that for a sentence X(x1,x2,…,x n ), QK T The meaning is that each character x in sentence X i The correlation between (i≤n) and other characters (including itself) or the proportion of information, that is, BERT can learn context information. T Adding attribution detection information to the ,can strengthen the mapping relationship between incorrect input characters and ,correct output characters, thus improving the prediction effect.

[0036] When BERT extracts the sentence representation E=(e1,e2,…,e n ), the fully connected layer is used to predict e i The corresponding target character, the prediction formula of the target character is:

[0037]

[0038] in, represents the predicted probability distribution of all candidate characters, w represents the parameter matrix, e i Represents x i The representation after BERT encoding; and take the first character of the BERT prediction candidate set, that is, the character with the largest Softmax value as the final correction result.

[0039] The principle of the present invention is that BERT itself has certain error detection and correction capabilities. After BERT encoding, it can generate a candidate set for each word in the sentence. However, due to the differences in the MASK characters of its mask model during pre-training and fine-tuning of downstream tasks, BERT’s detection and recall capabilities for incorrect Chinese characters are insufficient. For example, when using BERT to correct the sentence “I hope you are in good health”, it will output “I hope you are in good health” as it is, indicating that no error is detected; and when correcting “I introduce him to Taiwan’s favorite dishes”, it will generate “I introduce him to Taiwan’s new dishes”, indicating that it will be subject to the pre-training parameters and ignore the information of the word “favorite”. After the attribution network extracts more accurate detection information and integrates the detection information into BERT, the model can learn more mapping information between incorrect characters and correct characters (labels), allowing the incorrect characters to receive more “attention” from the model, thereby improving the detection and recall of incorrect characters.

[0040] By adopting the above technical solution, the present invention provides a Chinese character correction method based on attribution network and BERT, which abandons the confusion set or filter curve, can efficiently correct erroneous characters in Chinese sentences, and can greatly improve the detection and correction accuracy and recall rate.

[0041] The present invention also provides a system for implementing the above method, the system comprising a training module and an inference module;

[0042] The training module completely executes the process of training the BERT-based binary classification model and the Chinese character correction model on the training set;

[0043] The inference module can perform the operation of predicting target characters of incorrect characters on a test set or a user's own data set.

[0044] The beneficial effects of the present invention include: in the field of text error correction, two important indicators are precision and recall. Compared with existing related similar technologies, the present invention can significantly improve precision and recall.

[0045] 1) This method has stronger error detection capabilities than the BERT model. Because BERT is a large-scale pre-trained model based on masking tasks, some characters are replaced by mask characters during pre-training. These characters only contain contextual information, which can lead to insufficient detection of erroneous characters. By adding an attribution network, the model pays more attention to erroneous characters, which can improve both the precision and recall of error correction.

[0046] 2) This invention eliminates confusion sets, making the model simpler and easier to use. Confusion sets are complex to construct, time-consuming, and require constant maintenance. Furthermore, how to effectively utilize confusion sets is a complex issue, and improper use of confusion sets can introduce new interference into the model.

[0047] 3) The present invention has stronger versatility and scalability. By abandoning the confusion set and using only BERT, the model can be extended to error correction in other languages whose basic units are characters, such as Japanese and Korean, not just Chinese.

[0048] 4) The present invention can be used in many scenarios such as keyword search, optical character recognition, speech recognition, automatic correction of typos in Chinese language education, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is the model structure of the present invention.

[0050] Figure 2 This is a prediction flow chart of the present invention.

[0051] Figure 3 It is a training flow chart of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0053] Example

[0054] The first step is to collect a data set. The data set used in this paper is Sighan13, a standard data set for the CSC (Chinese Spelling Check) task. This is a small Chinese data set derived from Chinese test articles. Each data (sentence) contains several incorrect characters and has correct character labels. Considering that the amount of data in the training set is relatively small, and experiments have shown that in terms of character error correction based on the BERT model, the larger the training set, the better the effect, this paper adds 270,000 additional data. These data are automatically generated by a specific tool OpenCC based on the correct sentences. The sentence contains incorrect characters, and the format is the same as the Sighan13 data set.

[0055] The second step is to train a binary classification model for the attribution network. This classification model is also based on BERT, with a 12-layer BERT model selected for this paper. The training data includes both correct and incorrect sentences, labeled as true / false. After 16 epochs of training, the accuracy on the test set reached 98.6%.

[0056] In the third step, the model trained in the second step is used to make predictions. The input training data has dimensions (32, 128, 768), where the dimensions represent the batch size, sentence length, and hidden layer dimensions, respectively. The input data is first passed through the binary classification model trained in the second step to predict whether the input sentence is correct or not, that is, whether there are any errors in the sentence, denoted as F(X).

[0057] The fourth step is to calculate the gradient of F(X) to obtain attribution information. The dimension of the attribution information is also (32, 128, 768). Because the attribution information implicitly contains error detection information, the last dimension is normalized to obtain error probability information of dimension (32, 128, 1). Table 1 shows two examples of using the designed attribution network to detect incorrect characters. It can be seen that the attribution network can correctly locate the incorrect characters and has good discrimination between correct and incorrect characters.

[0058] For example, in Table 1 below, for the input "I'm very happy today," the attribution detection information is "0.17, 0.15, 0.11, 0.4, 0.67, 1.0," representing the probability of each character being an error. For the input "My classmates are reviewing the results," the attribution detection information is "0.12, 0.07, 0.04, 1.00, 0.43, 0.20."

[0059] Table 1

[0060]

[0061] In the fifth step, to improve the discrimination of error probabilities in the attribution detection information, highlighting the probabilities of incorrect characters and reducing those of correct characters, the present invention implements a simple filter that reduces probability values less than 0.5 to 0. For example, "0.17, 0.15, 0.11, 0.4, 0.67, 1.0" will become "0, 0, 0, 0, 0.67, 1.0" after passing through the filter. This is done to reduce the interference of correct characters on the detection of incorrect characters in subsequent models, thereby improving the model's detection capabilities.

[0062] The sixth step is to integrate the filtered attribution detection information into the second BERT model. The integration method is to add the detection information to the self-attention score of each layer of BERT. In formula [2], QK T The value represents each character x in the sentence i The larger the value of the connection with other characters (including itself), the more information the character carries. In Scores, the probability value of the wrong character is the largest, and the probability value of the correct character is the smallest, so QK T+Scores means that the error characters carry much more information of themselves, not just the context information. Other correct characters will also carry more information of the error characters. In this way, the model can learn more mapping information between error characters and correct labels, improving the detection and recall of error characters.

[0063] In the seventh step, the output after passing through the attribution network and BERT encoding also has a dimension of (32, 128, 768). Subsequently, a fully connected layer is used to predict the probability distribution of the target characters for each input character. The formula is as follows: The dimension is 768 (vocabulary size). When training the model, the loss of each predicted character in the output sentence is calculated using the cross-entropy loss function. The formula is: where is the predicted character of the model, y i is the correct character, and l cel is the cross-entropy function. Additionally, in the inference and testing stage after the model training is completed, for the candidate characters given by the model softmax(w×e i ), the character with the largest Softmax value is directly taken as the final corrected character. For example, in "I am very high today", among the predicted candidate characters of "行", the Softmax value of "兴" is the largest, so the corrected result is "I am very happy today"; in "Students have returned to school", among the predicted candidate characters of "反", the Softmax value of "返" is the largest, so the corrected result is "Students have returned to school".

[0064] The present invention has conducted experiments on the Sighan13 dataset and compared with the Pycorrector, the RNN-based LMC model, and the BERT model, which are respectively denoted as: Pycorrector, LMC, BERT. The model of the present invention is denoted as: BERT-G. At the same time, it is compared with the BERT model on the Sighan13 + 270,000 large dataset. The experimental results are shown in Table 2. It can be seen that after adding the attribution network, whether on a small-scale or large-scale dataset, the detection and correction effects have been significantly improved, indicating that the Chinese character error correction method based on the attribution network and BERT in this solution has achieved good results.

[0065] Table 2

[0066]

[0067] The protection scope of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, the changes and advantages that those skilled in the art can think of are included in the present invention, and the scope of protection is defined by the appended claims.

Claims

1. A Chinese character correction method based on attribution network and BERT, characterized in that: The method comprises the following steps: Step 1: Collect a Chinese text dataset containing incorrect characters and divide the dataset into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model effect; or use the complete dataset for model training; Step 2: Train a BERT-based binary classification model for the attribution network. Step 3: Use the binary classification model trained in step 2 to perform feedforward calculations on the sentences in the training set. The output is denoted as F(X), where X is the input sentence. Step 4: Use the output F(X) in step 3 to calculate the gradient of X to obtain the error attribution information; Step 5: Set a threshold filter to process the error attribution information obtained in step 4; Step 6: Integrate the error attribution information filtered in step 5 into the second BERT model, and use the correct and incorrect sentence pairs in the Chinese dataset to perform character correction training; Step 7: Inference stage: When a new sentence that needs Chinese character correction is obtained, the BERT model in step 2 is first used to calculate the error attribution information of the sentence. Then, the BERT model in step 6 is used to predict the target character of each incorrect character in the sentence in combination with the error attribution information.

2. The method according to claim 1, wherein In step 1, the Chinese text dataset is from the Chinese Spell Check dataset Sighan13 and additional data containing incorrect characters automatically generated by a specific tool OpenCC; the Chinese text dataset includes both correct and incorrect sentences.

3. The method according to claim 1, wherein In step 2, the binary classification model is used to predict whether there is an error in the sentence, and the final output result is the probability that the sentence contains an erroneous character; the attribution network refers to the trained binary classification model. For a given input X=x 1:n And the binary classification model output F(X), use the gradient to calculate the component in X that plays a key role in the model output result F(X), n is the number of Chinese characters in X.

4. The method according to claim 1, wherein The formula for finding the gradient is in, It is a vector with the same dimension as x, i represents the ordinal number of the input component in the model, and n represents the number of Chinese characters in the input sentence.

5. The method according to claim 1, wherein In step 4, the gradient of the output result F(X) is calculated to obtain the attribution information vector of each character in X. The attribution information vector is L2 normalized and summed and normalized to obtain the attribution information scalar, which is the probability of error for each character in X.

6. The method according to claim 1, wherein In step 5, the processing operation of the threshold filter is to reset the probabilities of the characters whose probabilities obtained in step 4 are less than 0.5 to 0.

7. The method according to claim 1, wherein In step 6, the incorporation operation refers to adding the filtered error attribution information to the self-attention score calculation of each layer of the second BERT model, strengthening the information of the erroneous characters themselves, so that the model focuses on the erroneous characters to achieve better detection and recall effects; the BERT self-attention calculation formula is: Among them, Q, K, and V are the three matrices of Query, Key, and Value transformed by the model input X, respectively. k is the number of columns of Q and K matrices, Scores is an n×n matrix, each row represents the error probability of each character in sentence X; for a sentence X(x1,x2,…,x n ), QK T It is an n×n matrix, each row represents each character x in the sentence X i The connection between other characters including itself, i≤n; the n row vectors of Scores are exactly the same.

8. The method according to claim 1, wherein In step 6, the model calculates the loss of each prediction in the output sentence using the following cross entropy loss function: in, is the predicted character of the model, y i For the correct character, l cel is the cross entropy function.

9. The method according to claim 1, wherein In step 7, for each input character x i , the prediction formula of the target character is: in, represents the predicted probability distribution of all candidate characters, w represents the parameter matrix, e i Represents x i Representation after BERT encoding; The character with the largest Softmax value is used as the predicted target character.

10. A system for implementing the method according to any one of claims 1 to 9, characterized in that: The system includes a training module and an inference module; the training module is used to completely execute the process of training a BERT-based binary classification model and a Chinese character correction model on a training set; The reasoning module is used to perform an operation of predicting target characters of erroneous characters on a test set or a user's own data set.

Images