Chinese spelling error correction method and device based on multiple representations and multiple pre-training models

By combining multiple representations and pre-trained models with word boundary and Chinese character component feature extraction, and using error character recognition models and multi-model prediction, the problem of comprehensive correction of Chinese spelling errors is solved, and the accuracy of error correction is improved.

CN113627158BActive Publication Date: 2024-12-03NANJING UNIV OF SCI & TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110751495.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-02
Publication Date
2024-12-03
Estimated Expiration
2041-07-02

AI Technical Summary

Technical Problem

Existing technologies cannot achieve comprehensive correction of Chinese spelling errors, mainly because of the lack of word boundary information, misidentification of implicit information in Chinese character morphology, and homophone substitution. Existing methods only focus on one aspect and cannot achieve comprehensive correction.

Method used

We employ a method based on multiple representations and multiple pre-trained models. By combining word boundary fusion and Chinese character component feature extraction with a pre-trained error character recognition model and multi-model prediction, we use conditional random fields and Viterbi optimization algorithms for error correction. Finally, we select the best correction result through ternary grammar and confusion set scoring.

Benefits of technology

It effectively improves the accuracy of Chinese spelling correction, solves the shortcomings of existing technologies that cannot fully correct errors, and takes into account phonetic and visual similarities to enhance the error correction effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113627158B_ABST
    Figure CN113627158B_ABST
Patent Text Reader

Abstract

The present application proposes a Chinese spelling correction method and device based on multiple representations and multiple pre-trained models, the method comprising: fusing word boundaries and extracting radical features for each character in the Chinese to be corrected, obtaining the Chinese to be corrected with feature values; the feature values ​​include word boundary feature values ​​and radical feature values; inputting the Chinese to be corrected with feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected; replacing the characters to be corrected with a pre-set mark to obtain the middle Chinese to be corrected; inputting the middle Chinese to be corrected into a pre-trained multiple pre-trained model, selecting the target correct character from a pre-set confusion set to replace the characters to be corrected, and obtaining the corrected Chinese. The method provided by the present application can identify spelling errors from multiple angles, thereby improving the accuracy of spelling correction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of Chinese spelling correction, and in particular to a Chinese spelling correction method and device based on multiple representations and multiple pre-training models. Background Art

[0002] Spelling errors, including speech recognition errors, image-to-text errors, and writing errors, are widely present in our lives. According to a survey, about 83% of spelling errors are related to speech similarity, and about 48% of spelling errors are related to visual similarity. These spelling errors have a great impact on downstream tasks such as image recognition and named entity recognition.

[0003] Spelling correction is also a very challenging task. A good spelling correction method requires deep language understanding and the ability to connect with context. Currently, spelling errors are mainly caused by the following three reasons: Reason 1 is that the Chinese text lacks word boundary information, resulting in errors in word division; Reason 2 is that the implicit information in the morphology of Chinese is misrecognized; Reason 3 is that during the spelling process, words are replaced by homophones. In response to the above three main causes of errors, the staff also adopted corresponding spelling correction methods. Although the above spelling errors all have certain correction methods, the current methods only focus on one aspect of the function and cannot achieve comprehensive error correction in the Chinese spelling method. Summary of the invention

[0004] The present application provides a Chinese spelling error correction method and device based on multiple representations and multiple pre-trained models, which can be used to solve the technical problem that comprehensive error correction cannot be achieved in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a Chinese spelling correction method based on multiple representations and multiple pre-trained models, the method comprising:

[0006] Each character in the Chinese characters to be corrected is subjected to word boundary fusion and radical feature extraction processing to obtain the Chinese characters to be corrected with feature values; the feature values ​​include word boundary feature values ​​and radical feature values;

[0007] Inputting the Chinese characters to be corrected with the feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected;

[0008] Replacing the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected;

[0009] The intermediate Chinese characters to be corrected are input into a pre-trained multi-pre-trained model, and a target correct character is selected from a pre-set confusion set to replace the characters to be corrected, so as to obtain the corrected Chinese characters.

[0010] In combination with the first aspect, in an implementation manner of the first aspect, if the target correct word does not exist in the confusion set, the candidate sentence is scored using tri-grams and perplexity;

[0011] The candidate sentence with the highest score is used as the corrected Chinese.

[0012] In combination with the first aspect, in an implementable manner of the first aspect, before fusing word boundaries and extracting radical features for each character in the Chinese character to be corrected, the method further includes:

[0013] The Chinese to be corrected is subjected to data preprocessing; the data preprocessing includes converting the Chinese to be corrected from traditional Chinese to simplified Chinese, marking incorrect characters in the Chinese to be corrected, and marking the beginning and end of the sentence of the Chinese to be corrected.

[0014] In combination with the first aspect, in an implementation manner of the first aspect, the radical feature extraction process includes:

[0015] The radical of each Chinese character to be corrected is mapped to a linear space, and then the value of the linear space is compressed and encoded to obtain a radical vector, and the radical vector is added to the vector of the corresponding character.

[0016] In conjunction with the first aspect, in an implementation manner of the first aspect, the wrong character recognition model is determined by the following method:

[0017] The wrong character recognition model is a bidirectional long short-term memory network model;

[0018] Divide the sample data into multiple groups of training set data and test set data; the sample data is Chinese characters to be corrected with feature values ​​for training;

[0019] Input the training set data into the wrong character recognition model for training, and determine the parameters and weights of the wrong character recognition model until the expected recognition result is obtained after the test set data is input into the wrong character recognition model;

[0020] The method for constructing the bidirectional long short-term memory network model is as follows:

[0021]

[0022]

[0023]

[0024] In the formula, is the positive order hidden layer information of the previous word; is the reverse hidden layer information of the previous word; wi represents any i-th Chinese character to be corrected with a feature value;

[0025] The recognition result of the wrong character recognition model is predicted using a conditional random field algorithm and optimized using a Viterbi optimization algorithm.

[0026] In combination with the first aspect, in an implementable manner of the first aspect, a conditional random field algorithm is used to predict a recognition result of the wrong character recognition model, and a Viterbi optimization algorithm is used to optimize the prediction, including:

[0027] The labeled training samples are represented as (W t ,y t ), where W t represents the tth sentence in a batch, y t =(y1,…,y i ,…,y n ) represents the true label sequence of the tth sentence, y i represents the i-th item of the true label sequence, and n represents the length of the true label sequence;

[0028] Assume that the input of the conditional random field layer is Z t =(z1,z2,…,z i ,…z n ), where Z t represents the output of the feedforward neural network layer, z i represents the i-th item in the output sequence, and n represents Z t Length;

[0029] The posterior probability is is the predicted label sequence, represents the i-th item in the predicted label sequence, The formula is as follows:

[0030]

[0031] In the formula, Y(Z t ) is the sequence of all predicted labels, and, It means (y j-1 ,y j )’s weight; It means (y j-1 ,y j ) bias parameter; j represents any label in the sequence; the one that maximizes the posterior probability This is the final prediction result;

[0032] Specifically, it is implemented by minimizing the loss function, and the minimizing loss function is as follows:

[0033]

[0034] Where L is the minimization loss function;

[0035] When testing on the test set data, find the optimal sequence y best We need to maximize p(y t |Z t ), specifically:

[0036]

[0037] The above steps are optimized using the Viterbi optimization algorithm.

[0038] In combination with the first aspect, in an implementation manner of the first aspect, scoring the candidate sentences using the tri-gram and the perplexity includes:

[0039] Score tp =λ·Score Tri-gram -(1-λ)·Score ppl ,

[0040] Score Tri-Gram =p(w1)·p(w2|w1)·p(w3|w2,w1)...p(w n |w n-1 ,w n-2 ),

[0041]

[0042] In the formula, Score Tri-Gram is the trigram score; Score ppl is the perplexity score; Score tp represents the weighted sum of the scores of the ternary grammar and the perplexity, where λ∈[0,1] is the weight coefficient.

[0043] In the second aspect, the present application proposes a Chinese spelling correction device based on multiple representations and multiple pre-trained models, the device comprising:

[0044] An acquisition module is used to fuse word boundaries and extract radical features for each character in the Chinese characters to be corrected, so as to obtain the Chinese characters to be corrected with feature values; the feature values ​​include word boundary feature values ​​and radical feature values;

[0045] A recognition module, used for inputting the Chinese characters to be corrected with feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected;

[0046] A replacement module, used to replace the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected;

[0047] The processing module is used to input the intermediate Chinese characters to be corrected into the pre-trained multiple pre-trained models, select the target correct characters from the pre-set confusion set to replace the characters to be corrected, and obtain the corrected Chinese characters.

[0048] In conjunction with the second aspect, in an implementation manner of the second aspect, the processing module is further used to:

[0049] If the target correct word does not exist in the confusion set, scoring the candidate sentence using tri-gram and perplexity;

[0050] The candidate sentence with the highest score is used as the corrected Chinese.

[0051] In conjunction with the second aspect, in an implementation manner of the second aspect, the processing module is further used to:

[0052] Data preprocessing is performed on the Chinese to be corrected; the data preprocessing includes converting the Chinese to be corrected from traditional Chinese to simplified Chinese, marking the wrong characters in the Chinese to be corrected, and marking the beginning and end of the sentence of the Chinese to be corrected.

[0053] In conjunction with the second aspect, in an implementation manner of the second aspect, the acquisition module is specifically used to:

[0054] The radical of each Chinese character to be corrected is mapped to a linear space, and then the value of the linear space is compressed and encoded to obtain a radical vector, and the radical vector is added to the vector of the corresponding character.

[0055] In conjunction with the second aspect, in an implementation manner of the second aspect, the identification module is specifically used to:

[0056] The wrong character recognition model is a bidirectional long short-term memory network model;

[0057] Divide the sample data into multiple groups of training set data and test set data; the sample data is Chinese characters to be corrected with feature values ​​for training;

[0058] Input the training set data into the wrong character recognition model for training, and determine the parameters and weights of the wrong character recognition model until the expected recognition result is obtained after the test set data is input into the wrong character recognition model;

[0059] The method for constructing the bidirectional long short-term memory network model is as follows:

[0060]

[0061]

[0062]

[0063] In the formula, is the positive order hidden layer information of the previous word; is the reverse hidden layer information of the previous word; w i represents any i-th Chinese character to be corrected with a feature value;

[0064] The recognition result of the wrong character recognition model is predicted using a conditional random field algorithm and optimized using a Viterbi optimization algorithm.

[0065] In conjunction with the second aspect, in an implementation manner of the second aspect, the identification module is specifically used to:

[0066] The labeled training samples are represented as (W t ,y t ), where W t represents the tth sentence in a batch, y t =(y1,…,y i ,…,y n ) represents the true label sequence of the tth sentence, y i represents the i-th item of the true label sequence, and n represents the length of the true label sequence;

[0067] Assume that the input of the conditional random field layer is Z t =(z1,z2,…,z i ,…z n ), where Z t represents the output of the feedforward neural network layer, z i represents the i-th item in the output sequence, and n represents Z t Length;

[0068] The posterior probability is is the predicted label sequence, represents the i-th item in the predicted label sequence, The formula is as follows:

[0069]

[0070] In the formula, Y(Z t ) is the sequence of all predicted labels, and, It means (y j-1 ,y j )’s weight; It means (y j-1 ,y j ) bias parameter; j represents any label in the sequence; the one that maximizes the posterior probability This is the final prediction result;

[0071] Specifically, it is implemented by minimizing the loss function, and the minimizing loss function is as follows:

[0072]

[0073] Where L is the minimization loss function;

[0074] When testing on the test set data, find the optimal sequence y best We need to maximize p(y t |Z t ), specifically:

[0075]

[0076] The above steps are optimized using the Viterbi optimization algorithm.

[0077] In conjunction with the second aspect, in an implementation manner of the second aspect, the processing module is specifically used to:

[0078] Score tp =λ·Score Tri-gram -(1-λ)·Score ppl ,

[0079] Score Tri-Gram =p(w1)·p(w2|w1)·p(w3|w2,w1)...p(w n |w n-1 ,w n-2 ),

[0080]

[0081] In the formula, Score Tri-Gram is the trigram score; Score ppl is the perplexity score; Score tp represents the weighted sum of the scores of the ternary grammar and the perplexity, where λ∈[0,1] is the weight coefficient.

[0082] This application integrates deep semantics, word boundaries and character radicals in the encoding stage of feature extraction, solving the shortcomings of existing models that cannot correctly segment words and cannot learn visual information. On the other hand, a variety of pre-trained models are proposed, and the characters that cannot be predicted by the pre-trained models are re-sorted after scoring using perplexity and Tri-gram mechanism on the confusion set, and the best result is used for correction. Because the phonetic and visual similarities between the wrong characters and the candidate characters are taken into account, the accuracy of Chinese text correction is effectively improved compared with the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A flowchart corresponding to a Chinese spelling error correction method based on multiple representations and multiple pre-trained models applicable to an embodiment of the present application;

[0084] Figure 2 A word boundary schematic diagram provided for an embodiment of the present application;

[0085] Figure 3 A schematic diagram of radical information provided in an embodiment of the present application;

[0086] Figure 4 A schematic diagram of an error detection phase provided in an embodiment of the present application;

[0087] Figure 5 A flowchart of a calibration phase provided in an embodiment of the present application;

[0088] Figure 6 A schematic diagram of the structure of a Chinese spelling correction device based on multiple representations and multiple pre-trained models provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0090] For the convenience of describing the embodiments of the present application, the professional terms involved in the embodiments of the present application are first explained.

[0091] Chinese spelling correction is to detect and correct misspelled words in sentences.

[0092] Word boundary features, that is, the boundary features between phrases in Chinese sentences.

[0093] BiLSTM network, namely bidirectional long short-term memory network (LSTM), is a neural network suitable for sequence data. BiLSTM is composed of forward LSTM and backward LSTM.

[0094] Conditional Random Field (CRF), a discriminative probability model, is used to find the label sequence closest to the truth among many predicted label sequences.

[0095] The Viterbi optimization algorithm, a dynamic programming algorithm used to find the path sequence that is most likely to produce the observed time series, can be used to accelerate the search process of conditional random fields.

[0096] One-hot encoding uses an N-bit state register to encode N states, and each state has its own independent register bit.

[0097] Hidden layer, in addition to the input and output layers in the neural network model, there is also a basic layer called a hidden layer.

[0098] The Word2vec model is a related model used to generate word vectors. It can be used to map each word to a vector that represents the relationship between words. The vector is the hidden layer of the neural network.

[0099] The attention mechanism is a neural network model that focuses on special information. Its function is to enable the neural network to focus on its important input information and ignore minor input information.

[0100] BERT encoder, a pre-trained language representation model encoder based on a bidirectional attention mechanism, can encode words into hidden layer vectors of a neural network.

[0101] The BERT pre-trained model is a pre-trained model that can predict masked words (MASK) and generate multiple candidate words. During the pre-training stage, 15% of the words in each sentence are randomly masked and their context is used to predict the masked words.

[0102] The ROBERTA pre-trained model is a pre-trained model that can predict masked words and generate multiple candidate words. Based on BERT, it adopts a dynamic masking strategy in the pre-training stage. The masked words in each round are guaranteed to be different, and then their context is used to predict the masked words.

[0103] The ELECTRA pre-trained model is a pre-trained model that can predict masked words and generate multiple candidate words. It uses a discriminant module to determine whether each word in a sentence is replaced.

[0104] The ERNIE pre-trained model, a pre-trained model that can predict masked words and generate multiple candidate words, is based on BERT. It masks entities and phrases based on pre-training and uses their context to predict masked words.

[0105] The confusion set is a dictionary that contains most of the homophones and homographs in Chinese, similar characters with front and back nasal sounds, similar characters with flat and rolled tongues, and commonly used characters. It is used to generate candidate characters for masked characters and to determine whether the characters predicted by the pre-trained model are correct.

[0106] N-gram is a probabilistic algorithm based on the Markov model. It operates the content of the text according to a sliding window of phrase size N to form multiple phrase fragments, and calculates the posterior probability of the current word based on the previous N words.

[0107] Tri-gram, that is, N-gram with N=3.

[0108] Perplexity (ppl), which is an indicator for evaluating whether an entire sentence is smooth.

[0109] The error detection stage is to train an error detection model through neural network training. After marking the error positions, the characters at the error positions are masked, and then the sentence containing the masked characters is sent to the correction stage.

[0110] The correction stage is to predict the characters at the masked positions based on the combination of multiple pre-trained models and confusion sets.

[0111] Currently, there are mainly three reasons for Chinese spelling mistakes, as shown in Table 1, which are the three reasons for Chinese spelling mistakes.

[0112] Table 1: Examples of Chinese text error correction

[0113]

[0114] Examples of Chinese spelling mistakes are shown in Table 1. Compared with Western languages such as English, Chinese spelling check is more complex because Chinese texts lack word boundary markers. In the first example, "他既 / 温柔又幽默" was wrongly recognized as "他祭文 / 柔又幽默". This is due to the lack of word boundary information in the recognition process. As one of the oldest pictographic characters, Chinese also implies information in its form. In the second example, "耕耘" was wrongly converted to "耕云". This is because the model ignored the feature that both "耕" and "耘" contain the same radical "耒" during the conversion process. Chinese is also a context-sensitive language. In the third example, "素描" was wrongly recognized as "速描". Actually, both are acceptable, but the probability of using "素描" is a bit higher.

[0115] For the first type of problem, previous technologies mainly relied on Chinese word segmentation systems. However, since general word segmentation systems are trained on correct corpora, sentences containing errors may cause word segmentation errors. Although word vector representation can avoid word segmentation errors, the error correction effect is not good due to the lack of word segmentation information.

[0116] For the second type of problem, previous technologies mainly corrected errors based on the rules of radicals. However, the design of the rules requires additional human and material resources and is not scalable. Another technology, the neural network-based language model, can learn the logical relationships between words, but it is difficult to learn the visual features of characters.

[0117] For the third problem, previous technologies used statistical models to count the probability of each word appearing with the previous n words, such as the N-gram language model. However, the N-gram model fails to take into account long-distance dependencies. Another technology, the pre-trained model based on the attention mechanism, solves the problem of long-distance dependencies based on the correlation between words in the whole sentence, but the pre-trained model of the attention mechanism ignores the phonetic and visual similarities between the masked words and the candidate words.

[0118] The method provided in the embodiments of the present application can effectively solve the above problems.

[0119] Let's first combine Figure 1 The possible system architecture applicable to the embodiments of the present application is introduced.

[0120] Please refer to Figure 1 , which exemplarily shows a flow chart corresponding to a Chinese spelling error correction method based on multiple representations and multiple pre-trained models applicable to an embodiment of the present application. The method provided in the embodiment of the present application comprises the following steps:

[0121] Before executing step S101, the following data preprocessing steps are first performed:

[0122] Data preprocessing is performed on the Chinese text to be corrected. Data preprocessing includes converting the Chinese text to be corrected from traditional Chinese to simplified Chinese, marking incorrect characters in the Chinese text to be corrected, and marking the beginning and end of the sentence of the Chinese text to be corrected.

[0123] Specifically, a correct mark is added to the correct word. A start mark is added to the beginning of the Chinese sentence to be corrected, such as [CLS]. A start mark is added to the end of the Chinese sentence to be corrected, such as [SEP].

[0124] Use the BERT encoder to obtain the word vector of the sentence to be corrected. The specific implementation plan is to convert each word into a 768-dimensional pre-trained word vector, and use the unknown word vector. <pad>Mark fill.

[0125] Step S101 , fusing word boundaries and extracting radical features for each character in the Chinese characters to be corrected, to obtain the Chinese characters to be corrected with feature values.

[0126] The feature values ​​include word boundary feature values ​​and radical feature values.

[0127] like Figure 2 The figure shows a word boundary diagram provided by an embodiment of the present application. The method provided by the embodiment of the present application is to add word boundary features to each character and perform fusion word boundary processing.

[0128] In the embodiment of the present application, the radical feature extraction process includes:

[0129] The radical of each Chinese character to be corrected is mapped to a linear space, and then the value of the linear space is compressed and encoded to obtain a radical vector, and the radical vector is added to the vector of the corresponding character.

[0130] Specifically, the word segmentation module is used to segment the correct sentence, and then mapped to the linear space and converted into a one-hot encoding form for storage. The radical contains some deep information about Chinese. Here, the radical extraction module (Radical) is used to extract the radical information of Chinese, such as Figure 3 Shown is a schematic diagram of radical information provided in an embodiment of the present application.

[0131] The specific implementation plan is to map the radicals to a linear space, and finally use Word2vec to compress and encode the values ​​in the linear space to obtain the radical vectors.

[0132] Step S102: input the Chinese characters to be corrected with the feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected.

[0133] Specifically, the wrong character recognition model in the embodiment of the present application is a bidirectional long short-term memory network model.

[0134] The sample data is divided into multiple sets of training set data and test set data. The sample data is the Chinese text to be corrected with feature values ​​used for training.

[0135] The training set data is input into the wrong character recognition model for training, and the parameters and weights of the wrong character recognition model are determined until the test set data is input into the wrong character recognition model to obtain the expected recognition result.

[0136] Among them, the construction method of the bidirectional long short-term memory network model is as follows:

[0137]

[0138]

[0139]

[0140] In the formula, It is the positive order hidden layer information of the previous word. is the reverse hidden layer information of the previous word. i Represents the i-th character in any Chinese character to be corrected with a feature value.

[0141] The conditional random field algorithm is used to predict the recognition results of the wrong word recognition model, and the Viterbi optimization algorithm is used to optimize it.

[0142] Specifically, the labeled training samples are represented as (W t ,y t ). Among them, W t represents the tth sentence in a batch, y t =(y1,…,y i ,…,y n ) represents the true label sequence of the tth sentence, y i represents the i-th item of the true label sequence, and n represents the length of the true label sequence.

[0143] Assume that the input of the conditional random field layer is Z t =(z1,z2,…,z i ,…z n ), where Z t represents the output of the feedforward neural network layer, z i represents the i-th item in the output sequence, and n represents Z t Length.

[0144] The posterior probability is is the predicted label sequence, represents the i-th item in the predicted label sequence, The formula is as follows:

[0145]

[0146] In the formula, Y(Z t ) is the sequence of all predicted labels, and, It means (y j-1 ,y j )’s weight. It means (y j-1 ,y j ) is the bias parameter. j represents any label in the sequence. This is the final prediction result.

[0147] The role of the conditional random field is to find the predicted label sequence that is closest to the true label sequence among many predicted label sequence results, and the role of the Viterbi algorithm is to accelerate the search process of the conditional random field.

[0148] Specifically, the loss minimization function is used to achieve this, and the loss minimization function is as follows:

[0149]

[0150] Where L is the minimization loss function.

[0151] When testing on the test set data, find the optimal sequence y best We need to maximize p(y t |Z t ), specifically:

[0152]

[0153] The above steps are optimized using the Viterbi optimization algorithm.

[0154] Step S103, replacing the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected.

[0155] Specifically, the words suspected of being mislabeled are masked using the [MASK] tag.

[0156] Step S104, input the intermediate Chinese characters to be corrected into the pre-trained multiple pre-trained models, select the target correct characters from the pre-set confusion set to replace the characters to be corrected, and obtain the corrected Chinese characters.

[0157] In the embodiment of the present application, the multiple pre-trained models include a BERT pre-trained model, a ROBERTA pre-trained model, an ELECTRA pre-trained model and an ERNIE pre-trained model.

[0158] The pre-trained model does not require additional training steps. It only needs to select appropriate materials, such as 5 rounds of fine-tuning on the People's Daily 2014 edition and speech dataset to adapt to the dataset, and then directly predict. The data is input into each pre-trained model, and prediction is performed separately to generate a total set of candidate words.

[0159] If the candidate word set hits the confusion set, the candidate word is filled into the sentence and the sentence is output.

[0160] It should be noted that, in order to better adapt to the error correction of speech-to-text, most of the Chinese homophones, homographs, words with similar front and back nasal sounds, words with similar flat and rolled tongue sounds, and common words are included in the confusion set. For example, "千 (qian)" and "嵌 (qian)" are homophones, "千" and "迁" are homographs, "跟 (gen)" and "更 (geng)" have similar front and back nasal sounds, and "冲 (chong)" and "匆 (cong)" have similar flat and rolled tongue sounds.

[0161] In the process of implementing the embodiments of the present application, there is also a possibility that the candidate word set does not hit the confusion set, and then the following steps are continued to be executed:

[0162] Score the candidate sentence using trigram grammar and perplexity;

[0163] And take the candidate sentence with the highest score as the corrected Chinese.

[0164] Specifically, the following method is adopted:

[0165] Score tp = λ · Score Tri-gram - (1 - λ) · Score ppl ,

[0166] Score Tri-Gram = p(w1) · p(w2|w1) · p(w3|w2, w1)... p(w n |w n-1 , w n-2 ),

[0167]

[0168] In the formula, Score Tri-Gram is the trigram grammar score; Score ppl is the perplexity score; Score tp represents the weighted sum of the scores of trigram grammar and perplexity, and λ ∈ [0, 1] is the weight coefficient.

[0169] To further illustrate the method provided by the embodiments of the present application, the following is illustrated with an example.

[0170] If the Chinese text to be corrected is: "一分耕云一分收货阿。After data preprocessing, it becomes "[CLS]一分耕云一分收货阿[SEP]". In step S101, it is marked that "云", "货", and "阿" are incorrect. Using a multi-pre-trained model to predict "耕耘", "耕作", "耕种", "耕读", "耕农", the probability of "耕耘" is the highest. Similarly, the probability of "收获" is also the highest. However, when the multi-pre-trained model predicts the last "阿", the top 5 are all punctuation marks. This situation occurs because the character "阿" is at the end of the sentence, causing the model to mistakenly think that the sentence has ended. Use the target correct characters in the confusion set corresponding to "阿", such as "啊", "阿", "俄", "额" to replace the masked character. Finally, score and sort, and the probability of "啊" is the highest. Finally, the correct sentence "一分耕耘一分收获啊” is output.

[0171] The method provided by the embodiment of the present application includes two stages. The first stage is the error detection stage. As Figure 4 shown, it is a schematic diagram of the error detection stage provided by the embodiment of the present application, corresponding to steps S101 to S102 in the embodiment of the present application.

[0172] As Figure 5 shown, it is a flowchart of the correction stage provided by the embodiment of the present application, corresponding to steps S103 to S104 in the embodiment of the present application.

[0173] In the encoding stage of feature extraction in the embodiment of the present application, deep semantics, word boundaries, and character radical features are fused, solving the disadvantages of existing models being unable to correctly segment words and unable to learn visual information. On the other hand, multiple pre-trained models are proposed. The words that cannot be predicted by the pre-trained models are scored and reordered using perplexity and the Tri-gram mechanism on the confusion set, and the optimal result is used for correction. Due to considering the phonetic and visual similarity between the incorrect word and the candidate word, the accuracy of Chinese text correction is effectively improved compared with the prior art.

[0174] Figure 6 Exemplarily shows a schematic structural diagram of a Chinese spelling correction device based on multi-representation and multi-pre-trained models provided by the embodiment of the present application. As Figure 6 shown, the device has the function of implementing the above-mentioned Chinese spelling correction method based on multi-representation and multi-pre-trained models. The function can be implemented by hardware or by hardware executing corresponding software. The device may include: an acquisition module 601, an identification module 602, a replacement module 603, and a processing module 604.

[0175] The acquisition module 601 is used to perform processing of fusing word boundaries and extracting radical features on each character in the Chinese text to be corrected, and obtain the Chinese text to be corrected with feature values. The feature values include word boundary feature values and radical feature values.

[0176] The recognition module 602 is used to input the Chinese characters to be corrected with feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected.

[0177] The replacement module 603 is used to replace the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected.

[0178] The processing module 604 is used to input the intermediate Chinese characters to be corrected into the pre-trained multiple pre-trained models, select the target correct characters from the pre-set confusion set to replace the characters to be corrected, and obtain the corrected Chinese characters.

[0179] Optionally, the processing module 604 is further configured to:

[0180] If the target correct word does not exist in the confusion set, the candidate sentence is scored using tri-grams and perplexity.

[0181] And the candidate sentence with the highest score is used as the corrected Chinese.

[0182] Optionally, the processing module 604 is further configured to:

[0183] Data preprocessing is performed on the Chinese text to be corrected. Data preprocessing includes converting the Chinese text to be corrected from traditional Chinese to simplified Chinese, marking the wrong characters in the Chinese text to be corrected, and marking the beginning and end of the sentence in the Chinese text to be corrected.

[0184] Optionally, the acquisition module 601 is specifically used for:

[0185] The radical of each Chinese character to be corrected is mapped to a linear space, and then the value of the linear space is compressed and encoded to obtain a radical vector, and the radical vector is added to the vector of the corresponding character.

[0186] Optionally, the identification module 602 is specifically used for:

[0187] The misspelling recognition model is a bidirectional long short-term memory network model.

[0188] The sample data is divided into multiple sets of training set data and test set data. The sample data is the Chinese text to be corrected with feature values ​​used for training.

[0189] The training set data is input into the wrong character recognition model for training, and the parameters and weights of the wrong character recognition model are determined until the test set data is input into the wrong character recognition model to obtain the expected recognition result.

[0190] Among them, the construction method of the bidirectional long short-term memory network model is as follows:

[0191]

[0192]

[0193]

[0194] In the formula, It is the positive order hidden layer information of the previous word. is the reverse hidden layer information of the previous word. i Represents the i-th character in any Chinese character to be corrected with a feature value.

[0195] The conditional random field algorithm is used to predict the recognition results of the wrong word recognition model, and the Viterbi optimization algorithm is used to optimize it.

[0196] Optionally, the identification module 602 is specifically used for:

[0197] The labeled training samples are represented as (W t ,y t ). Among them, W t represents the tth sentence in a batch, y t =(y1,…,y i ,…,y n ) represents the true label sequence of the tth sentence, y i represents the i-th item of the true label sequence, and n represents the length of the true label sequence.

[0198] Assume that the input of the conditional random field layer is Z t =(z1,z2,…,z i ,…z n ), where Z t represents the output of the feedforward neural network layer, z i represents the i-th item in the output sequence, and n represents Z t Length.

[0199] The posterior probability is is the predicted label sequence, represents the i-th item in the predicted label sequence, The formula is as follows:

[0200]

[0201] In the formula, Y(Z t ) is the sequence of all predicted labels, and, It means (y j-1 ,y j )’s weight. It means (y j-1 ,y j ) is the bias parameter. j represents any label in the sequence. This is the final prediction result.

[0202] Specifically, the loss minimization function is used to achieve this, and the loss minimization function is as follows:

[0203]

[0204] Where L is the minimization loss function.

[0205] When testing on the test set data, find the optimal sequence y best We need to maximize p(y t |Z t ), specifically:

[0206]

[0207] The above steps are optimized using the Viterbi optimization algorithm.

[0208] Optionally, the processing module 604 is specifically configured to:

[0209] Score tp =λ·Score Tri-gram -(1-λ)·Score ppl ,

[0210] Score Tri-Gram =p(w1)·p(w2|w1)·p(w3|w2,w1)...p(w n |w n-1 ,w n-2 ),

[0211]

[0212] In the formula, Score Tri-Gram is the trigram score. ppl is the perplexity score. tp represents the weighted sum of the scores of the ternary grammar and the perplexity, where λ∈[0,1] is the weight coefficient.

[0213] The embodiment of the present application integrates deep semantics, word boundaries and character radical features in the encoding stage of feature extraction, which solves the shortcomings of existing models that cannot correctly segment words and cannot learn visual information. On the other hand, a variety of pre-trained models are proposed, and the characters that cannot be predicted by the pre-trained models are re-sorted after scoring using perplexity and Tri-gram mechanism on the confusion set, and the best result is used for correction. Because the phonetic and visual similarities between the wrong characters and the candidate characters are taken into account, the accuracy of Chinese text correction is effectively improved compared with the prior art.

[0214] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0215] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the service construction device and service loading device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0216] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.< / pad>

Claims

1. Chinese spelling correction method based on multiple representations and multiple pre-trained models, It is characterized in that The method comprises: Each character in the Chinese characters to be corrected is subjected to word boundary fusion and radical feature extraction processing to obtain the Chinese characters to be corrected with feature values; the feature values ​​include word boundary feature values ​​and radical feature values; Inputting the Chinese characters to be corrected with the feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected; Replacing the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected; Input the intermediate Chinese characters to be corrected into a pre-trained multi-pre-trained model, select a target correct character from a pre-set confusion set to replace the characters to be corrected, and obtain the corrected Chinese characters; Before fusing word boundaries and extracting radical features for each character in the Chinese characters to be corrected, the following steps are also included: Performing data preprocessing on the Chinese to be corrected; the data preprocessing includes converting the Chinese to be corrected from traditional Chinese to simplified Chinese, marking incorrect characters in the Chinese to be corrected, and marking the beginning and end of the sentence of the Chinese to be corrected; The radical feature extraction process comprises: Mapping the radical of each Chinese character to be corrected to a linear space, then compressing and encoding the value of the linear space to obtain a radical vector, and adding the radical vector to the vector of the corresponding character; The wrong character recognition model is determined by the following method: The wrong character recognition model is a bidirectional long short-term memory network model; Divide the sample data into multiple groups of training set data and test set data; the sample data is Chinese characters to be corrected with feature values ​​for training; Input the training set data into the wrong character recognition model for training, and determine the parameters and weights of the wrong character recognition model until the expected recognition result is obtained after the test set data is input into the wrong character recognition model; The method for constructing the bidirectional long short-term memory network model is as follows: In the formula, is the positive order hidden layer information of the previous word; is the reverse hidden layer information of the previous word; w i represents any i-th Chinese character to be corrected with a feature value; The recognition result of the wrong character recognition model is predicted using a conditional random field algorithm and optimized using a Viterbi optimization algorithm.

2. The method according to claim 1, It is characterized in that If the target correct word does not exist in the confusion set, scoring the candidate sentence using tri-gram and perplexity; The candidate sentence with the highest score is used as the corrected Chinese.

3. The method according to claim 1, It is characterized in that The recognition result of the wrong character recognition model is predicted using a conditional random field algorithm and optimized using a Viterbi optimization algorithm, including: The labeled training samples are represented as (W t ,y t ), where W t represents the tth sentence in a batch, y t =(y 1 ,…,y i ,…,y n ) represents the true label sequence of the tth sentence, y i represents the i-th item of the true label sequence, and n represents the length of the true label sequence; Assume that the input of the conditional random field layer is Z t =(z 1 ,z 2 ,…,z i ,…z n ), where Z t represents the output of the feedforward neural network layer, z i represents the i-th item in the output sequence, and n represents Z t Length; The posterior probability is is the predicted label sequence, represents the i-th item in the predicted label sequence, The formula is as follows: In the formula, Y(Z t ) is the sequence of all predicted labels, and, It means (y j-1 ,y j )’s weight; It means (y j-1 ,y j ) bias parameter; j represents any label in the sequence; the one that maximizes the posterior probability This is the final prediction result; Specifically, it is implemented by minimizing the loss function, and the minimizing loss function is as follows: Where L is the minimization loss function; When testing on the test set data, find the optimal sequence y best We need to maximize p(y t |Z t ), specifically: The above steps are optimized using the Viterbi optimization algorithm.

4. The method according to claim 1, It is characterized in that Use tri-grams and perplexity to score candidate sentences, including: Score tp =λ·Score Tri-gram -(1-λ)·Score ppl , Score Tri-Gram =p(w 1 )·p(w 2 |w 1 )·p(w 3 |w 2 ,w 1 )...p(w n |w n-1 ,w n-2 ), In the formula, Score Tri-Gram is the trigram score; Score ppl is the perplexity score; Score tp represents the weighted sum of the scores of the ternary grammar and the perplexity, where λ∈[0,1] is the weight coefficient.

5. Chinese spelling correction device based on multiple representations and multiple pre-trained models, It is characterized in that The device comprises: An acquisition module is used to fuse word boundaries and extract radical features for each character in the Chinese characters to be corrected, so as to obtain the Chinese characters to be corrected with feature values; the feature values ​​include word boundary feature values ​​and radical feature values; A recognition module, used for inputting the Chinese characters to be corrected with feature values ​​into a pre-trained wrong character recognition model to obtain the recognized characters to be corrected; A replacement module, used to replace the to-be-corrected character with a preset mark to obtain the middle Chinese character to be corrected; A processing module, used for inputting the intermediate Chinese characters to be corrected into a pre-trained multi-pre-trained model, selecting a target correct character from a pre-set confusion set to replace the characters to be corrected, and obtaining the corrected Chinese characters; The processing module is further used for: Performing data preprocessing on the Chinese to be corrected; the data preprocessing includes converting the Chinese to be corrected from traditional Chinese to simplified Chinese, marking incorrect characters in the Chinese to be corrected, and marking the beginning and end of the sentence of the Chinese to be corrected; The acquisition module is specifically used for: Mapping the radical of each Chinese character to be corrected to a linear space, then compressing and encoding the value of the linear space to obtain a radical vector, and adding the radical vector to the vector of the corresponding character; The wrong character recognition model is determined by the following method: The wrong character recognition model is a bidirectional long short-term memory network model; Divide the sample data into multiple groups of training set data and test set data; the sample data is Chinese characters to be corrected with feature values ​​for training; Inputting the training set data into the wrong character recognition model for training, and determining the parameters and weights of the wrong character recognition model, until the expected recognition result is obtained after the test set data is input into the wrong character recognition model; The method for constructing the bidirectional long short-term memory network model is as follows: In the formula, is the positive order hidden layer information of the previous word; is the reverse hidden layer information of the previous word; w i represents any i-th Chinese character to be corrected with a feature value; The recognition result of the wrong character recognition model is predicted using a conditional random field algorithm and optimized using a Viterbi optimization algorithm.

6. The device according to claim 5, It is characterized in that The processing module is further used for: If the target correct word does not exist in the confusion set, scoring the candidate sentence using tri-gram and perplexity; The candidate sentence with the highest score is used as the corrected Chinese.