A Chinese spelling error correction method

Abstract

This invention discloses a Chinese spelling error correction method. The method increases training difficulty by replacing common words and characters in the original dataset with easily confused words and characters, thereby increasing the accuracy of model predictions. Simultaneously, the error correction model provided by this invention is optimized by integrating glyph and pinyin information based on the Macbert model, resulting in a more accurate error detection score and thus a more accurate identification of the corrected character. When the error correction model cannot obtain a suitable corrected character for a specific position in a sentence, this invention introduces a binary classification model to predict each candidate character at that specific position to obtain the corrected character, thereby enabling more accurate and efficient error correction of Chinese spelling.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and specifically relates to a Chinese spelling error correction method. Background Technology

[0002] Chinese text correction (CTC) is a fundamental task in Natural Language Processing (NLP). It includes two tasks: Chinese spelling check (CSC) and grammatical error correction (GEC).

[0003] Chinese spelling correction detects and corrects spelling errors (misspellings, incorrect characters) in Chinese text. It typically uses aligned corpora, where incorrect and correct sentences in the training data are of equal length. Chinese grammar correction corrects spelling, punctuation, and grammatical errors in Chinese text. Unaligned corpora can be used. It can correct grammatical errors such as extra characters, missing characters, and out-of-order characters.

[0004] Currently, Chinese spelling correction is practiced in two main areas: industry and academia.

[0005] Taking Baidu's Chinese text correction as an example (currently not open source, only a process introduction and a paid API are provided), the core technology of Baidu's Chinese text correction is:

[0006] 1) Language knowledge: Application of lexical and syntactic methods, combined with language models. Traditional language models predict the next word based on the preceding words, while Baidu uses a restricted language model. The restricted vocabulary is generated based on the original words and can utilize some information about misspelled words, such as pinyin and character shape.

[0007] 2) Knowledge Computing: Based on Memory / IR knowledge assistance, including related knowledge and text understanding, it solves the problem of knowledge generalization in low-frequency domains.

[0008] Features of Baidu Chinese Error Correction:

[0009] 1) Corrects various types of errors, including spelling and grammar correction.

[0010] 2) Supports different input formats such as text and voice.

[0011] 3) Fast, flexible, and highly customizable.

[0012] However, Baidu's text correction uses many technologies at each step and requires a large amount of corpus data, which is difficult for an individual to implement.

[0013] The difference between academia and industry lies in their approaches. Industry typically employs a three-step strategy (Pipeline model), while academia focuses on model structure, primarily using Transformer-based models and their derivatives, such as BERT, Roberta, and Electra. They also incorporate attention mechanisms from different perspectives and handle different modalities of text input, such as speech and images, resulting in end-to-end models. Academia mainly focuses on semantic and grammatical errors, while industry pays more attention to factual errors. Academia also emphasizes developing different models; some papers use a pre-training + fine-tuning strategy, employing large-scale corpora (millions of samples) for pre-training, which remains very costly. Summary of the Invention

[0014] This invention provides a Chinese spelling error correction method that achieves a high level of error correction while using a relatively easy-to-obtain model and requiring a small corpus.

[0015] A Chinese spelling correction method includes:

[0016] Obtain the original dataset, replace the common Chinese characters in the original dataset with the corresponding easily confused characters according to the easily confused character replacement probability to obtain the first training sample subset, and replace one character of the common words in the original dataset with one character of the corresponding easily confused word according to the easily confused word replacement probability to obtain the second training sample subset. The first training sample subset and the second training sample subset are used to construct the training sample set. Each training sample includes the original sentence and the corresponding correct sentence.

[0017] A correction model is obtained, which includes a Macbert module, a glyph encoding and pinyin encoding module, and an output fusion module;

[0018] The original sentence is input into the Macbert module to obtain the error detection probability of each position in the original sentence, the first probability group of each character in the Chinese character table at the corresponding position, and the hidden state output corresponding to each position.

[0019] The original sentence is encoded using glyph encoding and pinyin encoding modules to obtain initial glyph encoding and initial pinyin encoding. The initial glyph encoding and initial pinyin encoding are then processed with a fully connected layer and an activation function to obtain glyph encoding and pinyin encoding respectively.

[0020] The output fusion module performs dot products on the glyph encoding and pinyin encoding with the hidden state output at the corresponding positions, assigns weights to the dot product results and adds them together, performs residual connection with the hidden state output at the corresponding positions, and passes the residual connection results through a fully connected layer to obtain the second probability group of each character in the corresponding Chinese character table at the corresponding position.

[0021] When the largest probability value in the first probability group meets the probability score threshold, the character corresponding to the largest probability value in the second probability group is taken as the final corrected character.

[0022] When the highest probability value in the first probability group does not meet the probability score threshold, the probability of the TOPK ranked probability value is selected from the first probability group corresponding to each position in the original sentence. The candidate character set for the corresponding position is constructed by the characters corresponding to the selected TOPK probability and the original characters at the corresponding positions.

[0023] A binary classification model is obtained by inputting the candidate ranking features corresponding to the candidate character set at each position in the original sentence into the binary classification model to obtain the binary classification results of multiple candidate characters at each position. When the binary classification results of each candidate character at a specific position in the original sentence are all 0, the final corrected character at the specific position is the character corresponding to the largest probability value in the corresponding second probability group.

[0024] When at least one of the predicted values ​​of each candidate character at a specific position in the original sentence is not 0, the candidate character with the highest binary classification probability and a binary classification result of 1 is selected as the final corrected character.

[0025] Furthermore, the original sentences of the training samples are input into the VGG19 network, and the initial glyph encoding is obtained through the penultimate fully connected layer of the VGG19 network.

[0026] Furthermore, the original sentence is encoded using a pinyin encoding module to obtain the initial pinyin code, including:

[0027] Obtain the pinyin and tone of each character in the original sentence in the training sample. Then, obtain the corresponding subscript vector by mapping the pinyin and tone of each character in the original sentence to the vector representation. The subscript vector is composed of numeric subscripts, and the dimension of the subscript vector of each character is the same.

[0028] The initial pinyin encoding of the original sentence is obtained by sequentially transforming the dimension of the subscript vector of each character through a GRU layer and a pooling layer.

[0029] Furthermore, the method for constructing the mapping relationship between the pinyin and the vector representation includes:

[0030] The initials and finals are represented by 26 English characters. The tones are represented by "1", "2", "3", and "4", with "0" used for filling and "5" indicating that they are not used. There are a total of 32 characters. Each character corresponds to a numerical index from 0 to 31, and each character corresponds to an index vector of a set dimension.

[0031] Furthermore, the candidate ranking features corresponding to candidate characters at specific positions in the training samples are input into the binary classification model. These candidate ranking features include the following:

[0032] (1) The pinyin encoding, glyph encoding, cosine similarity of word vectors, Euclidean distance, Chebyshev distance and Manhattan distance of each candidate character and the original character corresponding to the original sentence;

[0033] (2) The shortest edit distance between each candidate character and the original character corresponding to the original sentence, based on their pinyin and stroke order encoding;

[0034] (3) The first probability value for each candidate character;

[0035] (4) Replace each candidate character with the original character to obtain the original sentence of the training sample, and calculate the PPL perplexity of the original sentence of the training sample after replacement using a language model.

[0036] (5) For each candidate character, calculate the cosine similarity, Euclidean distance, Chebyshev distance, and Manhattan distance between the Chinese semantics, pinyin encoding, glyph encoding, and word vector encoding of up to three original characters before and after the position of the original character in the original sentence, respectively, and the Chinese semantics are obtained through the Encoder layer of the Macbert module, and the word vector encoding is obtained through the Embedding layer of the Macbert module.

[0037] (6) The word frequency of each candidate character and the words formed by at most three original characters before and after the position of the candidate character in the original sentence;

[0038] (7) Add a Linear layer to the 6th and 12th layers of the MacBert model and calculate the error detection probability at each position in the original sentence of the training sample.

[0039] (8) The entropy value of the TOPK probability corresponding to each position in the original sentence;

[0040] (9) The ranking score feature of each candidate character in the candidate character set; and

[0041] (10) Binary classification model label, wherein the binary classification model label indicates whether each candidate character is a correct character, the label of a correct character is 1, and the label of an incorrect character is 0.

[0042] Furthermore, the entropy H(x) of the probability ranking TOPK from the first probability group corresponding to each position in the original sentence is:

[0043]

[0044] Where x is the index of the word corresponding to the probability of TOPK at each position in the original sentence, p(x) is the first probability value corresponding to the x-th word, and K is the number of words corresponding to the probability of TOPK at each position in the original sentence.

[0045] Furthermore, the total loss of the error correction model is:

[0046] The MacBert model consists of multiple encoder layers and linear layers connected to some encoder layers. The linear layers predict the probability of a misspelled word at each position in the original sentence of the training sample. The error detection loss is calculated by the Focal loss function based on the error detection probability.

[0047] The prediction results of the original sentence are corrected by outputting the fully connected layer of the error correction model. The error correction loss is obtained by using the cross-entropy loss function based on the prediction results and the correct sentence.

[0048] The total loss of the error correction model is obtained by weighting the error detection loss and the error correction loss and then summing them.

[0049] Furthermore, the step of replacing commonly used characters with corresponding easily confused characters based on the replacement probability of easily confused characters, wherein:

[0050] The easily confused characters are obtained from the confusion set corresponding to the commonly used Chinese characters. The confusion set includes multiple characters that are similar to the commonly used Chinese characters in terms of pinyin and shape, selected from the data dictionary.

[0051] The confusion set includes phonetically similar characters and visually similar characters. The phonetically similar characters are further divided into homophones with the same tone, homophones with different tones, near-homophones with the same tone, or near-homophones with different tones.

[0052] The probability of replacing commonly used characters with similar-sounding and similar-looking characters is set as the replacement probability of easily confused characters.

[0053] Furthermore, the step of replacing one character in a common word with one character in a corresponding easily confused word based on the probability of easily confused word replacement, wherein the easily confused word set includes words with the same full spelling, words with the same abbreviation, and misspelled words corresponding to the common words;

[0054] The method for obtaining the full-pinyin identical words and abbreviated pinyin identical words is as follows: when the commonly used words are input through the pinyin input method, the full-pinyin identical words and abbreviated pinyin identical words of the commonly used words can be obtained;

[0055] The method for obtaining the misspelled words is as follows: misspelled words are formed by pressing the wrong adjacent key when inputting the commonly used words through the Pinyin input method;

[0056] The probability of replacing easily confused words is the probability of replacing commonly used words with the same full spelling, the same abbreviation, and the wrong word, respectively, with the probability of replacing the same full spelling word being the highest.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0058] This invention increases the training difficulty by replacing common words and characters in the original dataset with easily confused words and characters, thereby increasing the accuracy of model predictions. Simultaneously, this invention provides an error detection model that optimizes the Macbert model, fusing glyph and pinyin information to more accurately obtain the corrected character. When the error detection model is unsure of obtaining a suitable corrected character for a specific position in a sentence (i.e., the first probability value does not meet the predicted probability value), this invention introduces a binary classification model to predict each candidate character at that specific position to obtain the corrected character, thus enabling more accurate and efficient error correction of Chinese spelling. Attached Figure Description

[0059] Figure 1 A framework diagram of a Chinese spelling correction method provided for a specific embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of an error detection model provided for a specific embodiment of the present invention. Detailed Implementation

[0061] This invention provides a Chinese spelling error correction method, such as... Figure 1 As shown, it includes:

[0062] Obtain the original dataset, which includes 280,000 (270,000 + 10,000) sentences downloaded from the open source (hereinafter referred to as 280,000 Data), sentences obtained from articles in the People's Daily in 2014, and sentences from the NLPCC dataset with a length close to the average length of sentences in 280,000 Data.

[0063] The processing method for 280,000 Data provided by the specific embodiments of the present invention is as follows: Calculate the average sentence length (about 45 characters) of all samples in the 280,000 Data training set and the average number of misspelled characters per sample (1 to 2 characters). At the same time, count information such as the distribution of Chinese characters, the distribution of misspelled characters, and the number of consecutive misspelled characters (such as 2 consecutive misspelled characters, three misspelled characters, four misspelled characters) in this dataset. The statistics in this step are for reference when constructing training data in the next step.

[0064] The specific embodiments of the present invention provide for extracting all the articles in People's Daily in 2014. For the extracted articles, according to the information statistically processed from 280,000 Data, truncate them into sentences of appropriate length, and do not truncate within the sentences, retaining the complete information of the sentences, ending with a full stop, a question mark, or an exclamation mark.

[0065] The specific embodiments of the present invention obtain the statistical information of Chinese characters and words from relevant websites. Based on the obtained statistical information of Chinese characters and words, obtain common Chinese characters, common words, and construct a dictionary for each Chinese character. The dictionary can obtain information such as the corresponding character frequency, the word frequency and词性 of the words containing the character, the stroke order encoding (encoding the strokes of Chinese characters with numbers, which can represent the glyph to a certain extent, for example, 'one' is represented by 1, 'two' is represented by '11', 'three' is represented by '111'), the corresponding picture, the homophonic and near-homophonic characters of the character, and the similar-shaped characters of the character. The statistical information of Chinese characters and words includes but is not limited to the following information:

[0066] (1) Obtain a dataset of common Chinese characters, about 3,500 characters (the commonly used 3,500 Chinese characters already include almost all Chinese characters in the 280,000 dataset).

[0067] (2) Obtain the word frequency table of the modern Chinese corpus.

[0068] (3) Obtain the character frequency table of the modern Chinese corpus.

[0069] (4) Obtain the word frequency table of word segmentation categories in the modern Chinese corpus (the content is the词性 and word frequency information of words). (5) Obtain the confusion sets of Chinese character pinyin and glyphs, including traditional Chinese, simplified Chinese, and the corresponding conversion tables.

[0070] (6) Obtain the stroke order encoding of Chinese characters.

[0071] (7) Obtain the picture corresponding to a certain font of Chinese characters.

[0072] The easily confused characters in the confusion set provided in this specific embodiment of the invention can be divided into five categories: sound (representing pinyin), tone (representing tone, including 1, 2, 3, 4, and neutral tone), form (representing character shape), near-sound (indicating two Chinese characters with similar but not identical pronunciations), and homophones (indicating two Chinese characters with the same pinyin, excluding tone). The five categories are: homophone with the same tone, homophone with different tone, near-sound with the same tone, near-sound with different tone, and similar in form. The first four categories can also be similar in form, meaning some Chinese characters have the same pinyin and similar shapes.

[0073] Specific embodiments of the present invention provide a method for constructing a training sample set using easily confused characters, including:

[0074] The first training sample subset is obtained by replacing common Chinese characters in the original dataset with their corresponding easily confused characters based on the substitution probability of easily confused characters. The second training sample subset is obtained by replacing one character in a common word in the original dataset with one character in a corresponding easily confused word based on the substitution probability of easily confused words. The first and second training sample subsets are used to construct the training sample set. Each training sample includes the original sentence and the corresponding correct sentence.

[0075] In one specific implementation, only the top 2000 most frequently used Chinese characters are replaced. For these 2000 commonly used characters, a set of confused characters can be obtained for each character based on a data dictionary. This confused character set needs to be filtered (to remove uncommon characters and traditional characters). The confused character set includes phonetically similar characters and visually similar characters. Phonetically similar characters are divided into four categories: homophones with the same tone, homophones with different tones, near-homophones with the same tone, and near-homophones with different tones. Since the confused character set contains many easily confused characters, only common easily confused characters are selected, with manual selection performed when necessary. Different probabilities are set for these easily confused characters, with each character having a different replacement probability; more samples are constructed for characters with higher probabilities.

[0076] In one specific embodiment, the substitution probability of each obfuscated word is set as follows:

[0077] For a single character's confusion set, it includes the five categories mentioned earlier. First, homophones and characters with the same tone are the fewest and most similar, allowing for the construction of more training samples. The subsequent categories can be assigned different replacement probabilities. The current strategy is: first, choose between replacing with a character that is similar in pinyin or similar in shape. A 70% probability of selecting a character with similar pinyin and a 30% probability of selecting a character with similar shape can be set. The 7:3 ratio is chosen because more misspelled characters with similar pinyin appear. If a character with similar pinyin is selected, different probabilities can be set: homophones with the same tone: 60%; homophones with different tones: 20%; near-homophones with the same tone: 15%; near-homophones with different tones: 5%. Once the choice is determined (homophones with the same tone, homophones with different tones, near-homophones with the same tone, or near-homophones with different tones), a Chinese character can be randomly selected from that category for replacement.

[0078] For each dataset, if it contains one of the 2,000 common Chinese characters, and the character is not a misspelling in the original sentence, and the adjacent characters before and after are not misspellings either, then replace the character with a certain probability with one of the easily confused characters in the confusion set. After construction, the average number of misspelled characters in the original sentence of each sample in the dataset is maintained at about 1 to 2, and the number of consecutive misspelled characters in the training set is controlled. There are several hyperparameters in this step: 1. The number of common Chinese characters (here 2,000 is selected); 2. The number of samples constructed for each popular Chinese character (the number of samples constructed for each popular Chinese character is obtained by using the formula: the frequency of the Chinese character divided by 250. The frequency of the Chinese character is obtained from the Chinese character frequency table. The denominator is set to 250, and at the same time, the first 2,000 popular Chinese characters are used, and finally about 100,000 misspelled characters can be added to the original training set. Setting 2,000 and the denominator 250 depends on how many misspelled characters to add). By observing the error examples in the test set, it is found that the model has poor prediction effects on common easily confused characters such as "得的地", "那哪", "他她它", etc., and the situation of misspelling two or more consecutive characters. More error samples of these categories can be constructed to improve the error correction ability of the model for such errors.

[0079] The specific embodiment of the present invention provides a method for constructing a training sample set through easily confused words, including:

[0080] Construct data according to two common input methods, namely the 26-key pinyin and the 9-key pinyin. The general habits of pinyin input are full-pinyin input and simple-pinyin input. The situations where users make mistakes include, for example, choosing similar-shaped words due to misreading, and choosing homophonic and phonetically similar words due to clicking on the wrong pinyin. At the same time, due to the distribution rules of the 26-key and 9-key keyboards and the adjacent keys, users may click on the wrong key, resulting in the final selection of the wrong word. First, for the correct sentences in the training set, use a word segmentation tool (jieba word segmentation) to segment the words, select common words according to the Chinese word frequency table. For each word, construct a table of common words and corresponding confused words according to the full-pinyin, simple-pinyin, and words selected wrongly due to clicking on adjacent keys. At the same time, different replacement probabilities are assigned to each part, with the highest replacement probability for full-pinyin, and the other probabilities decreasing in turn. For the common words (with lengths of 2, 3, or 4) that appear in the training set, select one from the corresponding confused words of the word as a reference for the replacement word (not replacing the whole word, only replacing one character in the word), and use one of the characters in the selected replacement word to replace the corresponding character in the original word.

[0081] The specific embodiment of the present invention provides a confusion set of common words, which can be divided into three parts: the same full-pinyin, the same simple-pinyin, and those caused by clicking on adjacent keys due to the input method. The quantities of these three parts are increasing, while the similarity with the original word is decreasing. Therefore, a higher replacement probability is assigned to the words with the same full-pinyin, and for the other two categories, the replacement probabilities decrease in turn.

[0082] The specific embodiment of the present invention provides an error correction model, such as Figure 2 As shown, the error correction model includes a Macbert module, a glyph encoding and pinyin encoding module, and an output fusion module. The original sentences in the training samples are input into the Macbert module to obtain the error detection probability of each position in the original sentence, the first probability group of each character in the corresponding Chinese character table (vocab.txt, the character table corresponding to the Chinese Bert model, about 20,000 entries) in the corresponding position, the first probability group includes 21,128 probability values, each probability value is the probability of each character in the Chinese character table in the corresponding position, and the hidden state output corresponding to each position;

[0083] In this embodiment of the invention, the original sentence is encoded by glyph encoding and pinyin encoding respectively to obtain initial glyph encoding and initial pinyin encoding. The initial glyph encoding and initial pinyin encoding are then processed by fully connected components and activation functions to obtain glyph encoding and pinyin encoding respectively.

[0084] In one specific embodiment, the original sentences of the training samples are input into the VGG19 network, and the initial glyph encoding is obtained through the penultimate fully connected layer of the VGG19 network.

[0085] In one specific embodiment, the method for obtaining an initial pinyin code by utilizing the mapping relationship between pinyin and vector representation, and constructing the mapping relationship between pinyin and vector representation, includes:

[0086] Initials and finals are represented by 26 English characters. Tones (first, second, third, and fourth) are represented by "1", "2", "3", and "4", with "0" used for padding and "5" indicating unused tones, totaling 32 characters. Each character corresponds to a numerical index from 0 to 31. A character mapping table is established: [0: '0', 1: '1', 2: '2', 3: '3', 4: '4', 5: '5', 6: 'a', 7: 'b', ..., 31: 'z']. Word embeddings are used to represent the pinyin, with each character corresponding to a 128-dimensional vector. The overall word vector weight matrix is ​​32×128. For non-Chinese characters in a sentence, the pinyin encoding is set to all zeros.

[0087] In one specific embodiment, the initial pinyin encoding is obtained by utilizing the mapping relationship between pinyin and vector representation, including: obtaining the pinyin (including tone) of each Chinese character in the original sentence in the training sample; obtaining a subscript vector of a specific length composed of corresponding subscripts by mapping the pinyin and tone of the character through the vector representation; and performing dimensional transformation on the subscript vector through a GRU layer and a pooling layer in sequence to obtain the initial pinyin encoding.

[0088] In a specific embodiment, for the Chinese character "今", the pinyin() function in the pypinyin library is used to obtain all the pinyins of the character (including tones, which are represented by numbers and placed at the end of the pinyin. It may be a polyphonic character). The result is "jin1". If it is a polyphonic character, the first pinyin is selected. For each character in "jin1", the corresponding subscript of each character is obtained and a list is formed. The result is [15, 14, 19, 1, 0, 0, 0, 0]. The reason for the length of this list being 8 is that the lengths of the pinyins of each Chinese character are not necessarily the same. Therefore, the character list of each Chinese character is padded, and the length after padding is fixed at 8. Then, the word vector of the character is calculated through the Embedding layer, and the dimension is 8×128. Similarly, the pinyin word vectors of all characters in the sentence can be obtained. A single-layer bidirectional GRU layer is used to transform the dimension of the pinyin word vectors of the sentence. The output of the GRU is used as the pinyin word vector of each character in the sentence, and the initial pinyin encoding of the Chinese character "今" is obtained through the pooling layer.

[0089] Due to the high dimensions of Chinese character semantics, pinyin encoding, glyph encoding, and Embedding (768 dimensions), these features need to be dimensionally reduced before candidate ranking. The method used in the specific embodiment of the present invention is principal component analysis (PCA).

[0090] In the present invention, the glyph encoding and pinyin encoding are respectively dot-multiplied with the hidden state output at the corresponding position through the output fusion module, the dot-multiplication results are weighted and added, the added result is subjected to residual connection with the hidden state output at the corresponding position, and the result of the residual connection is passed through the fully connected layer to obtain the second probability group of each word in the corresponding Chinese character table at the corresponding position. The second probability group includes 21128 probability values; the glyph information and encoding information are fused into the hidden state output through the fusion module, making the predicted second probability group more accurate.

[0091] As Figure 2 shown, when the maximum probability value in the first probability group satisfies the probability score threshold in the embodiment of the present invention, the word corresponding to the maximum probability value in the second probability group is used as the final corrected word, and then the corrected sentence is obtained.

[0092] When the maximum probability value in the first probability group does not satisfy the probability score threshold in the embodiment of the present invention, the probabilities ranked TOPK are screened from the first probability group corresponding to each position in the original sentence, and the words corresponding to the TOPK probabilities are used to construct the candidate word set at the corresponding position with the original word at the corresponding position.

[0093] A specific embodiment of the present invention provides that when the largest probability value in the first probability group does not meet the probability score threshold, the probability of the TOPK ranking of probability values ​​is selected from the first probability group corresponding to each position in the original sentence, and the candidate character set for the corresponding position is constructed by the characters corresponding to the selected TOPK probabilities and the original characters at the corresponding positions.

[0094] In one specific embodiment, the value of K in TopK is calculated by analyzing the distribution of correct characters among the candidates. After analysis, Top5 can effectively contain correct results (over 96%). Adding the original characters to the obtained TopK characters results in a maximum of 6 candidate characters for that specific position.

[0095] In one specific embodiment, the total loss of the error correction model of the present invention is:

[0096] The MacBert model comprises multiple encoder layers and linear layers connected to some of the encoder layers. The linear layers predict the probability of a misspelled word at each position in the original sentence from the training samples. The error detection loss is calculated using the Focal loss function based on this probability. The fully connected layers of the error correction model output the predicted result of correcting the original sentence. The error correction loss is obtained by applying the cross-entropy loss function to the predicted result and the correct sentence. The error detection loss and the error correction loss are weighted and summed to obtain the total loss of the error correction model.

[0097] In one specific embodiment, a Linear layer is added to the sixth and twelfth layers of the MacBert model provided in this invention for error detection. The Focal Loss function is used because, in a long sentence (40-50 words), the number of misspelled words is relatively small (1 to 2), and the Focal Loss function can solve the problem of a severe imbalance between positive and negative samples. Positive sample: A single misspelled word is called a positive sample, and it is the sample that the model focuses on more. Negative sample: Correct words in the sentence are called negative samples. Since the number of misspelled words is small, the Focal Loss function is used to prevent the loss from negative samples from dominating the overall loss.

[0098] In the binary classification model provided by the specific embodiment of the present invention, the candidate sorting features corresponding to the candidate character sets at each position in the original sentence are respectively input into the binary classification model to obtain the binary classification results of multiple candidate characters corresponding to each position. When the binary classification results of each candidate character corresponding to a specific position in the original sentence are all 0, the final corrected character at the specific position is the character corresponding to the maximum probability value in the corresponding second probability group. When at least one of the prediction values of each candidate character corresponding to a specific position in the original sentence is not 0, the candidate character with a binary classification result of 1 and the largest binary classification prediction probability is used as the final corrected character. In a specific embodiment, the candidate sorting features corresponding to the candidate characters at a specific position in the training sample are input into the binary classification model. The candidate sorting features mainly include the following features:

[0099] (1) The cosine similarity, Euclidean distance, Chebyshev distance, Manhattan distance, etc. between each candidate character and the pinyin encoding, glyph encoding, word vector of the original character corresponding to the original sentence.

[0100] (2) The shortest edit distance between each candidate character and the pinyin (including tones) and stroke order encoding (using numbers to encode the strokes of Chinese characters, which can represent the glyph to a certain extent, for example, 'one' is represented by 1, 'two' is represented by '11', and 'three' is represented by '111') of the original character corresponding to the original sentence.

[0101] (3) The first probability value for each candidate character.

[0102] (4) The training sample original sentence obtained by replacing the original character with each candidate character is used to calculate the PPL perplexity (representing the rationality of the sentence, the smaller the value, the smoother the sentence) of the replaced training sample original sentence through a language model (used to predict the rationality or smoothness of a sentence).

[0103] (5) Calculate the cosine similarity, Euclidean distance, Chebyshev distance, and Manhattan distance between the Chinese character semantics, pinyin encoding, glyph encoding, and word vector encoding of at most three original characters before and after the position of the original character corresponding to each candidate character in the original sentence, respectively, and the Chinese character semantics, pinyin encoding, glyph encoding, and word vector encoding of the candidate character; wherein, the Chinese character semantics are obtained through the Encoder layer of the Macbert module, and the word vector encoding is obtained through the Embedding layer of the Macbert module.

[0104] (6) The word frequency of the words formed by at most three original characters before and after the position of the original character corresponding to each candidate character in the training sample original sentence after replacing the original character with each candidate character.

[0105] (7) Add a Linear layer to the 6th and 12th layers of the MacBert model and calculate the error detection probability at each position in the original sentence of the training sample.

[0106] (8) The entropy value of the TOPK probability corresponding to each position in the original sentence;

[0107] The entropy H(x) of the probability ranking TOPK from the first probability group corresponding to each position in the original sentence is:

[0108]

[0109] Where x is the index of the word corresponding to the probability of TOPK at each position in the original sentence, p(x) is the first probability value corresponding to the x-th word, and K is the number of words corresponding to the probability of TOPK at each position in the original sentence.

[0110] (9) The ranking score feature of each candidate character in the candidate character set.

[0111] And (10) binary classification model label, wherein the binary classification model label indicates whether each candidate character is a correct character, the label of a correct character is 1, and the label of an incorrect character is 0.

[0112] In one specific embodiment, the ranking score feature provided by this invention has a probability value for each of the six candidates. These six candidates are sorted in descending order based on their probabilities. Rank: After sorting, the first candidate has a rank value of 1, the second candidate has a rank value of 2, and so on. TopP: P represents probability. For each candidate's TopP value: After sorting, the sum of the probabilities of all candidate characters preceding the current character, including the probability of the current character. TopK value: After sorting, the sum of the rank values ​​of all candidate characters preceding the current character, including the probability of the current character. For example, the second candidate after sorting has a TopK = 1 + 2 = 3. 1 is the rank value of the first character, and 2 is the rank value of the second character.

[0113] In one specific embodiment, when the maximum predicted probability of the Macbert module for a certain position in the sample is less than a certain threshold, the Top K candidates for that position are obtained, and all features are constructed for each candidate character, resulting in a total of K sets of samples (all features of each candidate are training data of the binary classification model). Since at most only one sample has a label of 0, the ratio of positive to negative samples is unbalanced (positive samples are those with the correct candidate character and a label of 1; negative samples are those with the incorrect candidate character and a label of 0).

[0114] In one specific embodiment, the XGBOOST classifier is used. This model itself has many hyperparameters, such as the maximum tree depth, learning rate, number of weak classifiers, feature sampling rate, and sample balancing adjustment parameters. The optimal set of hyperparameters is selected using the GridSearchCV method (grid-based parameter search).

Claims

1. A method for correcting Chinese spelling errors, characterized in that, include: Obtain the original dataset, replace the common Chinese characters in the original dataset with the corresponding easily confused characters according to the easily confused character replacement probability to obtain the first training sample subset, and replace one character of the common words in the original dataset with one character of the corresponding easily confused word according to the easily confused word replacement probability to obtain the second training sample subset. The first training sample subset and the second training sample subset are used to construct the training sample set. Each training sample includes the original sentence and the corresponding correct sentence. A correction model is obtained, which includes a Macbert module, a glyph encoding and pinyin encoding module, and an output fusion module; The original sentence is input into the Macbert module to obtain the error detection probability of each position in the original sentence, the first probability group of each character in the Chinese character table at the corresponding position, and the hidden state output corresponding to each position. The original sentence is encoded using glyph encoding and pinyin encoding modules to obtain initial glyph encoding and initial pinyin encoding. The initial glyph encoding and initial pinyin encoding are then processed with a fully connected layer and an activation function to obtain glyph encoding and pinyin encoding respectively. The output fusion module performs dot products on the glyph encoding and pinyin encoding with the hidden state output at the corresponding positions, assigns weights to the dot product results and adds them together, performs residual connection with the hidden state output at the corresponding positions, and passes the residual connection results through a fully connected layer to obtain the second probability group of each character in the corresponding Chinese character table at the corresponding position. When the largest probability value in the first probability group meets the probability score threshold, the character corresponding to the largest probability value in the second probability group is taken as the final corrected character. When the highest probability value in the first probability group does not meet the probability score threshold, the probability of the TOPK ranked probability value is selected from the first probability group corresponding to each position in the original sentence. The candidate character set for the corresponding position is constructed by the characters corresponding to the selected TOPK probability and the original characters at the corresponding positions. A binary classification model is obtained by inputting the candidate ranking features corresponding to the candidate character set at each position in the original sentence into the binary classification model to obtain the binary classification results of multiple candidate characters at each position. When the binary classification results of each candidate character at a specific position in the original sentence are all 0, the final corrected character at the specific position is the character corresponding to the largest probability value in the corresponding second probability group. When the predicted value of each candidate character at a specific position in the original sentence is at least not 0, the candidate character with the highest binary classification probability and the result of 1 is taken as the final corrected character. The original sentences of the training samples are input into the VGG19 network, and the initial glyph encoding is obtained through the penultimate fully connected layer of the VGG19 network. The original sentence is encoded using a pinyin encoding module to obtain the initial pinyin code, including: Obtain the pinyin and tone of each character in the original sentence in the training sample. Then, obtain the corresponding subscript vector by mapping the pinyin and tone of each character in the original sentence to the vector representation. The subscript vector is composed of numeric subscripts, and the dimension of the subscript vector of each character is the same. The initial pinyin encoding of the original sentence is obtained by sequentially transforming the dimension of the subscript vector of each character through a GRU layer and a pooling layer.

2. The Chinese spelling correction method according to claim 1, characterized in that, The method for constructing the mapping relationship between pinyin and vector representation includes: The initials and finals are represented by 26 English characters. The tones are represented by "1", "2", "3", and "4", with "0" used for filling and "5" indicating that they are not used. There are a total of 32 characters. Each character corresponds to a numerical index from 0 to 31, and each character corresponds to an index vector of a set dimension.

3. The Chinese spelling correction method according to claim 1, characterized in that, The candidate ranking features corresponding to candidate characters at specific positions in the training samples are input into the binary classification model. The candidate ranking features include the following features: (1) The pinyin encoding, glyph encoding, cosine similarity of word vectors, Euclidean distance, Chebyshev distance and Manhattan distance of each candidate character and the original character corresponding to the original sentence; (2) The shortest edit distance between each candidate character and the original character corresponding to the original sentence, based on their pinyin and stroke order encoding; (3) The first probability value for each candidate character; (4) Replace each candidate character with the original character to obtain the original sentence of the training sample, and calculate the PPL perplexity of the original sentence of the training sample after replacement by the language model; (5) For each candidate character, calculate the cosine similarity, Euclidean distance, Chebyshev distance, and Manhattan distance between the Chinese semantics, pinyin encoding, glyph encoding, and word vector encoding of up to three original characters before and after the position of the original character in the original sentence, respectively, and the Chinese semantics, pinyin encoding, glyph encoding, and word vector encoding of the candidate character; wherein, the Chinese semantics are obtained through the Encoder layer of the Macbert module, and the word vector encoding is obtained through the Embedding layer of the Macbert module; (6) The word frequency of each candidate character and the words formed by at most three original characters before and after the position of the candidate character in the original sentence; (7) Add a Linear layer to the 6th and 12th layers of the Macbert module and calculate the error detection probability at each position in the original sentence of the training sample; (8) The entropy value of the TOPK probability corresponding to each position in the original sentence; (9) The ranking score feature of each candidate character in the candidate character set; and (10) Binary classification model label, wherein the binary classification model label indicates whether each candidate character is a correct character, the label of a correct character is 1, and the label of an incorrect character is 0.

4. The Chinese spelling correction method according to claim 3, characterized in that, The entropy value of the probability of the top K probabilities selected from the first probability group corresponding to each position in the original sentence. H ( x )for: in, x This is the index of the word corresponding to the probability of the TOPK at each position in the original sentence. This is the first probability value corresponding to the x-th character. K This represents the number of characters corresponding to the probability of the TOPK at each position in the original sentence.

5. The Chinese spelling correction method according to claim 1, characterized in that, The total loss of the error correction model is: The Macbert module includes multiple encoder layers and a Linear layer connected to some of the encoder layers. The Linear layer predicts the probability of a misspelled word at each position in the original sentence of the training sample. The error detection loss is calculated by the Focal loss function based on the error detection probability. The prediction results of the original sentence are corrected by outputting the fully connected layer of the error correction model. The error correction loss is obtained by using the cross-entropy loss function based on the prediction results and the correct sentence. The total loss of the error correction model is obtained by weighting the error detection loss and the error correction loss and then summing them.

6. The Chinese spelling correction method according to claim 1, characterized in that, The process of replacing common characters with corresponding easily confused characters based on the replacement probability of easily confused characters, wherein: The easily confused characters are obtained from the confusion set corresponding to the commonly used Chinese characters. The confusion set includes multiple characters that are similar to the commonly used Chinese characters in terms of pinyin and shape, selected from the data dictionary. The confusion set includes phonetically similar characters and visually similar characters. The phonetically similar characters are further divided into homophones with the same tone, homophones with different tones, near-homophones with the same tone, or near-homophones with different tones. The probability of replacing commonly used characters with similar-sounding and similar-looking characters is set as the replacement probability of easily confused characters.

7. The Chinese spelling correction method according to claim 1, characterized in that, The step of replacing one character in a common word with one character in a corresponding easily confused word based on the probability of easily confused word replacement, wherein the easily confused word set includes words with the same full spelling, words with the same abbreviation, and misspelled words corresponding to the common words; The method for obtaining the full-pinyin identical words and abbreviated pinyin identical words is as follows: when the commonly used words are input through the pinyin input method, the full-pinyin identical words and abbreviated pinyin identical words of the commonly used words can be obtained; The method for obtaining the misspelled words is as follows: misspelled words are formed by pressing the wrong adjacent key when inputting the commonly used words through the Pinyin input method; The probability of replacing easily confused words is the probability of replacing commonly used words with the same full spelling, the same abbreviation, and the wrong word, respectively, with the probability of replacing the same full spelling word being the highest.

Images