Text automatic correction method and device, electronic equipment and storage medium

Abstract

The application relates to a text automatic error correction method and device, electronic equipment and a storage medium. The method combines word level and sentence level error correction methods to perform error correction, can perform more comprehensive and effective error correction on the text under the premise of ensuring error correction efficiency, and comprises the following steps: performing error detection on input text to determine error-prone words contained in the input text; generating a correct candidate word corresponding to each error-prone word; based on the correct candidate word corresponding to each error-prone word and the position of the error-prone word in the input text, a plurality of candidate error correction results are constructed and added to a candidate result set; based on a downstream error correction model, a plurality of candidate error correction results corresponding to the input text are obtained and added to the candidate result set; the candidate error correction results in the candidate result set are sorted, and a correction result corresponding to the input text is obtained based on a sorting result.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, electronic device and storage medium for automatic text error correction. Background Technology

[0002] With the rapid development of information technology, more and more users can access various information systems or websites via computers. However, when searching for content on websites, users inevitably enter incorrect search data or query content, resulting in the system or website displaying data that is not what the user wants. Currently, most mainstream technical solutions rely heavily on user search and click log data, using probabilistic methods to return more likely candidate options to the user based on this data. However, effective automatic search error correction solutions are relatively few and their correction results are unsatisfactory when user logs are unavailable and projects are in the cold start phase. Summary of the Invention

[0003] This application provides a text automatic error correction method, apparatus, electronic device, and storage medium, offering an effective text automatic error correction solution for the cold start phase of a project.

[0004] In a first aspect, one embodiment of this application provides a method for automatic text correction, including:

[0005] Error detection is performed on the input text to identify error-prone words contained in the input text;

[0006] Generate the correct candidate word for each error-prone word, and based on the correct candidate word for each error-prone word and the position of the error-prone word in the input text, construct multiple candidate error correction results and add them to the candidate result set;

[0007] Based on the downstream error correction model, several candidate error correction results corresponding to the input text are obtained and added to the candidate result set;

[0008] The candidate error correction results in the candidate result set are sorted, and the error correction result corresponding to the input text is obtained based on the sorting result.

[0009] Optionally, the error detection of the input text includes:

[0010] The input text is segmented to obtain several words;

[0011] For word segments containing pinyin, the pinyin in the word segments is converted into Chinese, and the converted word segments are identified as error-prone entries;

[0012] For word segments containing only Chinese characters, the word segments are converted into their corresponding pinyin, and then the corresponding pinyin is converted back into Chinese characters to obtain the converted word segments. If the converted word segments are inconsistent with the corresponding original word segments, and the text similarity between the converted word segments and the original word segments is greater than the similarity threshold, then the converted word segments are identified as error-prone candidate words. The original text is obtained, consisting of the original word segments corresponding to the error-prone candidate words and several word segments located before and after the original word segments. The original word segments in the original text are replaced with error-prone candidate words to obtain the concatenated text. The language scores of the original text and the concatenated text are calculated using a language model. If the language score of the concatenated text is greater than the language score of the original text, then the original word segments are identified as error-prone words.

[0013] Optionally, the step of converting the pinyin in word segmentation into Chinese includes:

[0014] Based on the pre-constructed initial probability distribution, state transition probability distribution, and state-observation probability distribution, and combined with the Viterbi algorithm, the optimal path to convert the pinyin in the word segmentation into Chinese is found. The pinyin in the word segmentation is replaced with the Chinese corresponding to the optimal path to obtain the converted word segmentation. The initial probability distribution records the probability that each character is at the beginning of the text, the state transition probability distribution records the probability that one character moves to another, and the state-observation probability distribution records the probability distribution that a character has a certain pronunciation.

[0015] Optionally, the error detection includes at least one of the following detection methods:

[0016] The system detects whether any words in the input text belong to the error-prone confusion set, and identifies these words as error-prone terms; or

[0017] The input text is segmented, and words not belonging to the preset dictionary are identified as error-prone terms; or

[0018] The input text is segmented into several bigram words according to the 2-gram model, and the language score of each bigram word is obtained using a language model. The input text is also segmented into several ternary words according to the 3-gram model, and the language score of each ternary word is obtained using a language model. For each character in the input text, the language score of the character at that position is determined based on the language scores of the bigram and ternary words containing the character at that position. Based on the language score of the character at each position and a preset score threshold, the error-prone positions in the input text are determined, and error-prone words are obtained based on the characters at the error-prone positions.

[0019] Optionally, generating the correct candidate term for each error-prone term includes:

[0020] For each character in the frequently misspelled terms, replace it with a corresponding homophone, similar-sounding character, similar-looking character, or easily confused character to obtain several correct candidate terms for the frequently misspelled terms; and / or

[0021] The prediction model performs a masking operation on the locations of error-prone words in the input text and generates prediction results corresponding to the locations of error-prone words. The prediction results are then used as the correct candidate words for the error-prone words.

[0022] Optionally, the downstream error correction model includes a Bi-LSTM-based detection network and an ALBERT-based error correction network;

[0023] The process of obtaining several candidate error correction results corresponding to the input text based on the downstream error correction model includes:

[0024] The vector representation corresponding to the input text is E = (e1, e2, e3, e4, ..., e n ) Input the detection network to obtain the error probability of each character in the input text, where e i This represents the vector corresponding to the i-th character in the input text;

[0025] The new vector for each character in the input text is obtained based on the following formula:

[0026] e' i =p i *e i,mask +(1-p i )*e i ,

[0027] Among them, e' i Let e ​​represent the new vector of the i-th word. i,mask This indicates that the prediction model is used to evaluate the vector representation E of e. i The result obtained after performing a masking operation is for e i The predicted vector at the location, p i This represents the error probability of the i-th character;

[0028] Let E'=(e'1,e'2,e'3,e'4,...,e' n Input the error correction network to obtain several candidate error correction results.

[0029] Optionally, sorting the candidate error correction results in the candidate result set includes:

[0030] Based on the local and global features of each candidate error correction result, the candidate error correction results in the candidate result set are sorted. The local features include at least one of the following: frequency features, word segmentation features, phonetic and morphological features, and PMI features of each corrected word in the candidate error correction results. The global features include at least one of the following: language model score of the candidate error correction result, and similarity between the candidate error correction result and the corresponding original text.

[0031] Secondly, embodiments of this application provide a text automatic error correction device, comprising:

[0032] An error detection module is used to perform error detection on the input text in order to identify the error-prone words contained in the input text;

[0033] The first candidate module is used to generate the correct candidate word corresponding to each error-prone word, and based on the correct candidate word corresponding to each error-prone word and the position of the error-prone word in the input text, construct multiple candidate error correction results and add them to the candidate result set;

[0034] The second candidate module is used to obtain several candidate error correction results corresponding to the input text based on the downstream error correction model, and add them to the candidate result set;

[0035] The sorting module is used to sort the candidate error correction results in the candidate result set, and obtain the error correction result corresponding to the input text based on the sorting result.

[0036] Thirdly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0037] Fourthly, one embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0038] Fifthly, one embodiment of this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative implementations of any of the above-described TCP transmission performance control methods.

[0039] The solution provided in this application embodiment, on the one hand, expands and recalls possible error positions at the character and word granularity level to generate error correction results for characters and words, and on the other hand, uses a downstream error correction model to directly provide possible error correction results at the sentence level. Under the premise of ensuring error correction efficiency, it can perform more comprehensive and effective error correction on text. It is simple, efficient, and highly scalable. After enriching the professional vocabulary of several fields, it can be quickly deployed to application scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram illustrating an application scenario of the automatic text correction method provided in the embodiments of this application;

[0042] Figure 2 A flowchart illustrating an automatic text correction method provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the structure of the downstream error correction model provided in the embodiments of this application;

[0044] Figure 4 This is a schematic diagram of the structure of the automatic text correction device provided in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0048] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0049] A language model (LM) aims to establish a distribution that describes the probability distribution of a given sequence of words in a language. In other words, a language model is a model that describes the probability distribution of words, reliably reflecting the probability distribution of words used in language recognition. Language models play a crucial role in natural language processing and are widely used in fields such as speech recognition and machine translation. For example, a language model can be used to obtain the most probable word sequence among multiple word sequences for speech recognition, or to predict the next most likely word given several words. Commonly used language models include N-Gram LM, Big-Gram LM, and Tri-Gram LM.

[0050] N-gram is an algorithm based on statistical language models, also known as a first-order Markov chain. Its basic idea is to process the text content byte-by-byte using a sliding window of size N, forming a sequence of byte segments of length N. Each byte segment is called a gram. The frequency of all grams is statistically analyzed, and then filtered according to a pre-defined threshold to form a list of key grams, which is the vector feature space of the text. Each gram in the list represents a feature vector dimension.

[0051] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0052] refer to Figure 1This is a schematic diagram illustrating an application scenario of the automatic text correction method provided in this application embodiment. The terminal device 101 and the server 102 can be connected via a wireless or wired network. The terminal device 101 includes, but is not limited to, desktop computers, mobile phones, mobile computers, tablets, media players, smart wearable devices, smart TVs, robots, and other electronic devices. The server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0053] In a search scenario, users can input text through the search box on terminal device 101. The backend can perform error detection on the user's input text in real time, correct erroneous text, and provide the corrected results to the user for selection. After the user selects the correct text, terminal device 101 sends the selected text to server 102. Server 102 then searches for relevant content from the network or database based on this text and sends it back to terminal device 101. In a document editor, terminal device 101 can perform error detection on the user's input text, correct erroneous text, and display possible correct results to the user. The user can then select one of these results to replace the original input text, enabling the document editor to automatically detect and correct text errors, providing users with greater convenience.

[0054] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible scenarios, and this application embodiment does not impose any limitations. Figure 1 The functions that each device in the application scenario shown can achieve will be described in the subsequent method embodiments, and will not be elaborated on here.

[0055] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0056] The following is combined with Figure 1 The application scenarios shown illustrate the technical solutions provided in the embodiments of this application. Any of the automatic text correction methods provided in the embodiments of this application can be applied to electronic devices with computing capabilities, such as terminal devices or servers; this application does not limit the scope of application.

[0057] refer to Figure 2 This application provides a method for automatic text correction, which specifically includes the following steps:

[0058] S201. Perform error detection on the input text to identify error-prone words contained in the input text.

[0059] In practice, before error detection, the input text can be preprocessed. For example, string formatting operations can be performed on the input text, including encoding standardization, full-width to half-width conversion, uppercase to lowercase conversion, and traditional to simplified Chinese conversion, to eliminate interference from special characters and special formats in subsequent processing. Then, error detection can be performed on the preprocessed input text.

[0060] In practice, if the input text is long, it can be broken down into short sentences according to its grammatical features, and then each short sentence can be checked for errors. If the input text is short, errors can be checked directly on the input text.

[0061] In practice, a set of error-prone terms can be built to store the detected error-prone terms and their positions in the input text, making it convenient to call these data later.

[0062] S202. Generate the correct candidate words corresponding to each error-prone word. Based on the correct candidate words corresponding to each error-prone word and the position of the error-prone word in the input text, construct multiple candidate error correction results and add them to the candidate result set.

[0063] S203. Based on the downstream error correction model, obtain several candidate error correction results corresponding to the input text and add them to the candidate result set.

[0064] The downstream error correction model is a new model for text error correction downstream tasks, trained on open-source data and pre-trained models. This model builds upon the idea of ​​constructing downstream tasks using pre-trained models, employing ALBERT-tiny to ensure text error correction capabilities while mitigating the problem of low prediction efficiency caused by excessively large pre-trained models. (Reference) Figure 3 The downstream error correction model mainly consists of two parts: a Bi-LSTM-based detection network and an ALBERT-based error correction network. The detection network is used to predict the error probability of each character in the text, and the error correction network is used to predict the correction probability of the erroneous character. The prediction results of the detection network will be used as the input of the error correction network.

[0065] The training data for the downstream error correction model mainly comes from existing sets of easily confused words, homophones, and homographs. It utilizes millions of news datasets obtained from public sources and web crawlers, and randomly replaces them with words, homophones, homographs, and random characters from the easily confused word sets at a certain ratio, supplemented by some manual annotations.

[0066] In practical implementation, existing vectorization methods (such as word2vec, BERT, XLNet, Electra, etc.) can be used to convert the input text into a vector representation E = (e1, e2, e3, e4, ..., e n ), where e i Let e ​​represent the vector corresponding to the i-th character in the input text. i Specifically, it's the sum of the word embedding, position embedding, and segment embedding, which is consistent with the ALBERT input. Then, the vector representation corresponding to the input text is E = (e1, e2, e3, e4, ..., e...). n The input detection network is used to obtain the error probability of each character in the input text, and a new vector for each character in the input text is obtained based on the following formula: e' i =p i *e i,mask +(1-p i )*e i , where e' i Let e ​​represent the new vector of the i-th word. i,mask This indicates that the prediction model is used to evaluate the vector representation E of e. i The result obtained after performing the MASK operation is for e i The predicted vector at the location, p i Let E' represent the error probability of the i-th character. Finally, let E' = (e'1, e'2, e'3, e'4, ..., e''). n Input the error correction network to obtain several candidate error correction results.

[0067] The input to the detection network is a vector representation E = (e1, e2, e3, e4, ..., e4) in units of words. n The output is a label sequence L = (l1, l2, l3, ... l) n ), where l i Represents the i-th character x i The tag, l i =1 indicates a word error, l i =0 indicates that there are no errors in the word. For each word, p i The character x i The probability of having a mislabeled item, pi The higher the value, the better the value of the character x. i The higher the probability of error, the better. The detection network is a bidirectional LSTM, where the output at each position is a concatenated vector of the forward and backward hidden states, i.e.:

[0068]

[0069]

[0070]

[0071] Then, the hidden states at each location are connected to a fully connected layer, and the output l is obtained. i The probability of an incorrect character

[0072]

[0073] Then, for the input vector representation E, e i After performing the mask operation, predict the resulting vector to obtain e. i The predicted vector at the location, i.e., e i,mask After that, with p i For weights of e i,mask and e i Weighted summation:

[0074] e' i =p i *e i,mask +(1-p i )*e i

[0075] If the probability of position i being incorrect is higher, then the new vector e' i Just like e i,mask The closer they are, the better; otherwise, they will be the same as the original input e. i They are quite close.

[0076] The error correction network is a multi-class sequence model based on ALBERT. The input is a new vector representation of the network's output, E' = (e'1, e'2, e'3, e'4, ..., e'). n The output target is the correct character sequence, which is the corrected text. The structure of the error correction network mainly refers to the structure of the pre-trained model ALBERT-tiny. In the last layer of ALBERT, a fully connected network is used to calculate the candidate word with the highest probability at each position. A residual network structure is used to connect the input vector and the output of the last layer of the detection network. The hidden state of the error correction network and the input of the detection network are linearly combined through the residual connections. This can prevent gradient vanishing during training. The last layer uses the softmax function.

[0077] A downstream error correction model constructed based on a pre-trained model consists of two modules: a detection network for error detection and an error correction network for error correction. It can effectively correct the input text while ensuring the error correction efficiency.

[0078] S204. Sort the candidate error correction results in the candidate result set, and obtain the error correction result corresponding to the input text based on the sorting result.

[0079] Specifically, during implementation, on the constructed data set, multiple-dimensional features can be used to train a sorting model for the candidate error correction results in the candidate result set. Based on this sorting model, the confidence of each candidate error correction result can be given according to the features of each candidate error correction result, and the candidate result set can be sorted in descending order of confidence to obtain the sorting result.

[0080] In a search scenario, based on the sorting result for the candidate result set, the top N candidate error correction results can be displayed to the user, and the user can select the appropriate text from them. In the case where the user does not select, the candidate error correction result with the highest confidence can be automatically selected as the error correction result.

[0081] The text automatic error correction method of the embodiments of the present application, on the one hand, expands and recalls the possible error positions at the character and word granularity levels to generate character and word error correction results, and on the other hand, uses the downstream error correction model to directly give possible error correction results at the sentence level. It can perform more comprehensive and effective error correction on the text while ensuring the error correction efficiency, is simple and efficient, and is an effective text automatic error correction solution during the cold start stage of the project.

[0082] Specifically, during implementation, one or more of the following error detection methods can be used to detect the error entries in the input text:

[0083] The first error detection method: Detect the input text for errors based on a pre-constructed error-prone and confusing set.

[0084] Among them, the error-prone and confusing set includes a large number of pre-collected words that are prone to errors or confusion. For example, the correct expression of "负贵" is "富贵". Error-prone and confusing words can be obtained from network channels and the error-prone and confusing set can be obtained through manual collation. When used, it is detected whether there are words belonging to the error-prone and confusing set in the input text, and the words belonging to the error-prone and confusing set are determined as error entries. Specifically, the input text can be segmented first to obtain several segments, and it is detected whether the obtained segments appear in the error-prone and confusing set, and the segments that appear in the error-prone and confusing set are determined as error entries.

[0085] The second error detection method: Detect the input text for errors based on a preset dictionary.

[0086] Among them, the preset dictionary is a custom dictionary, which can include common words and special words in a certain field. The included special words can be adjusted according to the field to which the method of this application is applied, and there is no limitation. Specifically, the input text can be segmented first to obtain several segmented words, and the segmented words that do not belong to the preset dictionary are determined as error-prone entries.

[0087] In specific implementation, the input text can be segmented by existing word segmenters (such as the Jieba word segmenter) and entity recognition models (such as the open-source LTP entity recognition tool).

[0088] The third error detection method: Use a language model to detect errors in the input text at the character level.

[0089] In specific implementation, the input text is segmented into several binary segmented words according to the 2-gram model, and the language scores of each binary segmented word are obtained using the language model; the input text is segmented into several ternary segmented words according to the 3-gram model, and the language scores of each ternary segmented word are obtained using the language model; for each character position in the input text, according to the language scores of the binary segmented word and the ternary segmented word containing the character at this position, the language score of the character at this position is determined; based on the language scores of each character position and the preset score threshold, the error-prone positions in the input text are determined, and error-prone entries are obtained based on the characters at the error-prone positions.

[0090] Next, take the input text "I love China" as an example to illustrate the specific implementation process of the third error detection method.

[0091] First, according to the 2-gram model, "I love China" can be split into 3 binary segmented words: "I love", "love China", and "China", and the language scores [a1, a2, a3] of these 3 binary segmented words are obtained using the language model. According to the 3-gram model, "I love China" can be split into 2 ternary segmented words: "I love China" and "love China", and the language scores [b1, b2] of these 2 ternary segmented words are obtained using the language model.

[0092] Then, using the language scores of binary word segmentation and ternary word segmentation, calculate the language scores of each character position. When calculating the language scores of each character in the input text, the average value of the language scores of the binary word segments containing the character can be obtained as the binary score of the character, and the ternary score of the character can be obtained based on the language scores of the ternary word segments containing the character. Then, calculate the average value of the binary score and the ternary score as the language score of the character at this position. Among them, when calculating the ternary score of a character at a certain position, if the character only appears in one ternary word segment, directly use the language score of this ternary word segment as the ternary score of the character; if the character appears in two ternary word segments, the weight of the ternary word segment where the character is in the middle position is 2 / 3, and the weight of the ternary word segment where the character is in the edge position is 1 / 3. Weighted average the language scores of the two ternary word segments according to the above weights to obtain the ternary score of the character; if the character appears in three ternary word segments, calculate the average value of the language scores of these three ternary word segments as the ternary score of the character.

[0093] For example, for the first character "我" in the input text, there is only one binary word segment "我爱" containing the character "我", so the binary score of the first character "我" is a1. There is only one ternary word segment "我爱中" containing the character "我", so the ternary score of the first character "我" is b1. The language score of the first character "我" is c1 = (a1 + b1) / 2; for the second character "爱" in the input text, the binary word segments containing the character "爱" are "我爱" and "爱中", so the binary score of the second character "爱" is (a1 + a2) / 2. There are two ternary word segments "我爱中" and "爱中国" containing the character "爱". "爱" is in the middle position of "我爱中" and in the edge position of "爱中国". So the weight of "我爱中" is 2 / 3, and the weight of "爱中国" is 1 / 3. Then the ternary score of the second character "爱" is (2b1 + b2) / 3. Therefore, the language score of the second character "爱" is c2 = [(a1 + a2) / 2 + (2b1 + b2) / 3] / 2; using the same method, the language score of "中" is c3 = [(a2 + a3) / 2 + (b1 + b2 + b2) / 3] / 2, and the language score of "国" is c4 = [a3 + b2] / 2.

[0094] In specific operation, when calculating the binary score of each character, the language scores of all binary segments can be completed according to the preset rules. That is, the language score of the first binary segment is added before the first position of the first score sequence, and the language score of the last binary segment is added after the last position of the first score sequence. For example, if the first score sequence is [a1,a2,a3], the completed first score sequence is [a1,a1,a2,a3,a3]. Then, the average of the completed first score sequence is calculated according to the sliding window of length 2, and the binary score of each character at each position in the input text is obtained in turn [(a1+a1) / 2,(a1+a2) / 2,(a2+a3) / 2,(a3+a3) / 2]. When calculating the ternary score for each character, a second score sequence composed of the language scores of all ternary words can be completed according to preset rules. Specifically, the language scores of the first two ternary words are added before the first position of the second score sequence, and the language scores of the last two ternary words are added after the last position. For example, if the second score sequence is [b1,b2], the completed second score sequence is [b1,b1,b1,b2,b2,b2]. Then, the completed second score sequence is averaged using a sliding window of length 3 to obtain the ternary score for each character at each position in the input text: [(b1+b1+b1) / 3,(b1+b1+b2) / 3,(b1+b2+b2) / 3,(b2+b2+b2) / 3]. Finally, the average of the binary and ternary scores for each character at each position is calculated as the language score for that character.

[0095] Finally, based on the language score of the character at each position and a preset score threshold, error-prone positions in the input text are determined. Specifically, the median of the sequence [c1,c2,c3,c4] is first calculated and denoted as med; then, the sequence [|c1-med|,|c2-med|,|c3-med|,|c4-md|] is obtained, where the symbol "||" represents the absolute value, and the median of [|c1-med|,|c2-med|,|c3-med|,|c4-md|] is calculated and denoted as med2; next, according to the formula: final_score=α*[|c1-med|,|c2-med|,|c3-med|,|c4-md|] / med2, the final score of the character at each position is obtained. If the final score of a certain position is greater than the preset score threshold, the character at that position is considered an error-prone character.

[0096] The fourth error detection method: using a language model to detect errors in the input text at the word level.

[0097] In practice, the input text is first segmented to obtain several words. Segmentation can be performed at the character level using 2-gram and 3-gram methods.

[0098] For word segments containing only Chinese characters, the word segments are converted into their corresponding pinyin, and then the corresponding pinyin is converted back into Chinese characters to obtain the converted word segments. If the converted word segments are inconsistent with the corresponding original word segments, and the text similarity between the converted word segments and the original word segments is greater than the similarity threshold, then the converted word segments are identified as error-prone candidate words. The original text is obtained, consisting of the original word segments corresponding to the error-prone candidate words and several word segments located before and after the original word segments. The original word segments in the original text are replaced with error-prone candidate words to obtain the concatenated text. The language scores of the original text and the concatenated text are calculated using a language model. If the language score of the concatenated text is greater than the language score of the original text, then the original word segments are identified as error-prone words.

[0099] The higher the language score of the text, the more reasonable the text is and the lower the probability of error. Therefore, when the language score of the concatenated text is greater than that of the original text, it indicates that the concatenated text is more reasonable than the original text, meaning that the original word segmentation that was replaced in the original text may contain errors. The similarity threshold can be determined according to the actual application scenario, and this application embodiment does not limit it; for example, it can be 0.5.

[0100] For example, the word segmentation results obtained after input text segmentation are: W1, W2, W3, W4, W5. Taking word segmentation W2 as an example, first convert word segmentation W2 into its corresponding pinyin, and then convert the pinyin into Chinese W2'. If W2 and W2' are the same, it indicates that word segmentation W2 is not a frequently misspelled word; if W2 and W2' are not the same, the text similarity between W2 and W2' is calculated. The higher the similarity, the higher the probability that the user will mistakenly input W2' as W2. Therefore, if the text similarity is greater than the similarity threshold, W2' is identified as a frequently misspelled candidate word; otherwise, W2' is determined not to be a frequently misspelled candidate word. Next, the original text is obtained, consisting of the original word segment W2 corresponding to the error-prone candidate word W2' and several words before and after W2. Here, one word segment W1 and one word W3 before and after W2 can be selected, and the original text formed by them is W1W2W3. The original word segment W2 in the original text W1W2W3 is replaced with the error-prone candidate word W2' to obtain the concatenated text W1W2'W3. Finally, the language scores of the original text W1W2W3 and the concatenated text W1W2'W3 are calculated using a language model. If the language score of the concatenated text is greater than the language score of the original text, then the original word segment W2 is identified as an error-prone word.

[0101] For word segments containing pinyin, the pinyin in the word segments is converted into Chinese, and the converted word segments are identified as error-prone entries.

[0102] In specific implementation, three probability distributions can be obtained based on open-source data: an initial probability distribution, a state transition probability distribution, and a state-observation value probability distribution. Among them, the initial probability distribution records the probability of each character being at the start position of the text; the state transition probability distribution is the probability distribution of various states transitioning to each other, that is, the probability distribution of one state transitioning to another state. In the embodiments of this application, the state transition probability distribution records the probability of one character transitioning to another character; the state-observation value probability distribution is the probability distribution from the state to the observation value. In the embodiments of this application, the state-observation value probability distribution records the probability distribution of a certain character being a certain pronunciation.

[0103] Based on the pre-constructed initial probability distribution, state transition probability distribution, and state-observation value probability distribution, combined with the Viterbi algorithm, find the optimal path to convert the pinyin in word segmentation into Chinese, and replace the pinyin in the word segmentation with the Chinese corresponding to the optimal path to obtain the converted word segmentation. The general process of the Viterbi algorithm is as follows: obtain several Chinese characters corresponding to the pinyin, starting from the first character, record the product of the initial probability distribution of the first character and the probability distribution of the current character transitioning to the next character, and finally multiply by the probability that the current character is a certain pinyin. Finally, sort all the path results and take the characters on the maximum value path to form the converted Chinese.

[0104] For example, "shebao payment" contains pinyin, and through these three probability distributions combined with the Viterbi algorithm, it can be converted into "social insurance payment" after the hidden Markov process.

[0105] The language model used in the embodiments of this application can be the character-level NgramLM model open-sourced by Baidu, and the size of this model is approximately 2.8G, or it can also be a language model trained using the KenLM statistical language model tool on open-source data.

[0106] Based on the above various error detection methods, the text automatic error correction method of this application can cover most error-prone input scenarios commonly encountered by users, such as homophones, confusing sounds, similar-looking characters, full pinyin, or mixed texts containing pinyin and Chinese. Especially in the text scenario containing pinyin, the common error correction method in the market is to build an index and give prompts to users when they input. However, the solution provided by this application can automatically convert the pinyin in the text into common Chinese, and in the error detection process, it can also convert the entry into pinyin and then back into the entry, thereby discovering potential error positions in the text and improving the error detection ability.

[0107] Based on any of the above implementation manners, the correct candidate entries corresponding to the error-prone entries can be generated through at least one of the following methods:

[0108] Method 1: For each character in an error-prone entry, replace it with corresponding homophones, phonetically similar characters, visually similar characters, and easily confused characters to obtain several correct candidate entries corresponding to the error-prone entry.

[0109] In specific implementation, homophone sets, phonetically similar character sets, visually similar character sets, and easily confused character sets can be pre-constructed. Among them, the homophone set records the homophones corresponding to each character. For example, "tong", "tong", and "tong" are homophones; the phonetically similar character set records phonetically similar characters. For example, "yin" and "ying" are phonetically similar characters; the visually similar character set records visually similar characters. For example, "rong" and "rong" are visually similar characters; the easily confused character set records other easily confused characters.

[0110] For an error-prone entry containing multiple characters, each character in it can be replaced with homophones, phonetically similar characters, visually similar characters, and easily confused characters. Through the combination of each replacement character, multiple correct candidate entries can be obtained.

[0111] Method 2: Perform a masking (MASK) operation on the position of the error-prone entry in the input text through a prediction model and generate a prediction result corresponding to the position of the error-prone entry, and use the prediction result as the correct candidate entry corresponding to the error-prone entry.

[0112] Among them, the prediction model can be BERT, ERNIE, ELECTRA, etc. Performing a MASK operation on the position of the error-prone entry in the input text means covering the error-prone entry in the input text, and then predicting the input text with the error-prone entry covered based on the prediction model, predicting the text of the covered part, that is, the prediction result. If there are multiple prediction results, the first n prediction results can be taken as the correct candidate entries corresponding to the error-prone entry.

[0113] Based on the above Method 1 and Method 2, the solution of this application has high scalability and can be quickly applied to application scenarios after enriching professional vocabulary in several fields. Method 2 uses the masking operation of the prediction model to predict the error correction result of the error-prone entry, starting from the semantic level of the entire input text, providing more reliable options for the error correction result, and also helping to improve the universality of the solution and expand the application scenarios of the solution.

[0114] Based on any of the above implementation manners, the candidate error correction results in the candidate result set can be sorted based on the local features and global features of each candidate error correction result. Specifically, the lightGBM sorting model that combines local features and global features can be selected to sort the candidate error correction results in the candidate result set. By comprehensively considering local features and global features, the accuracy of the sorting result can be improved.

[0115] Among them, the local features mainly statistically represent the features of the entry, specifically including at least one of the frequency features, word segmentation features, phonetic and visual features, and PMI features of each error-corrected entry in the candidate error correction result.

[0116] Frequency refers to how often a term appears in the training corpus; the higher the frequency, the more common the term. For each corrected term in the candidate error correction results, the frequency of the corrected term in the training corpus and the frequency of the corresponding incorrect term in the training corpus are statistically analyzed. Based on these two frequencies, the frequency change before and after the error correction is obtained, which is the frequency feature of the corrected term.

[0117] For each corrected term in the candidate error correction results, word segmentation is performed on the corrected term. Based on the word segmentation results, the number of phrases, the number of single characters, and the length of the text segment containing misspelled characters are counted. At the same time, the same processing is performed on the erroneous term corresponding to the corrected term to obtain the number of phrases, the number of single characters, and the length of the text segment containing misspelled characters contained in the erroneous term. Based on the above statistics of the data before and after error correction, the changes in word segmentation features before and after error correction are obtained, which are used as the word segmentation features corresponding to the corrected term.

[0118] Phonetic-semantic features refer to the Jaccard distance between the pinyin of the incorrect entry and the pinyin of the corrected entry, as well as the changes in the pinyin vowels.

[0119] PMI (Pointwise Mutual Information) features refer to the minimum and maximum PMI values ​​of the NGR (Not Given Memory) of erroneous terms and corrected terms.

[0120] The global features primarily consist of features of the entire input text, specifically including at least one: the language model score of the candidate error correction result, and the similarity between the candidate error correction result and the corresponding original text. The Baidu open-source NGRAM language model can be used to calculate the language model score of the candidate error correction result. A pre-trained model is used to obtain the language model vector features of both the input text and the candidate error correction result, and then the similarity between the two is calculated.

[0121] like Figure 4 As shown, based on the same inventive concept as the above-described automatic text correction method, this application also provides an automatic text correction device 40, comprising:

[0122] Error detection module 401 is used to perform error detection on input text to identify error-prone words contained in the input text;

[0123] The first candidate module 402 is used to generate the correct candidate word corresponding to each error-prone word, and construct multiple candidate error correction results based on the correct candidate word corresponding to each error-prone word and the position of the error-prone word in the input text and add them to the candidate result set;

[0124] The second candidate module 403 is used to obtain several candidate error correction results corresponding to the input text based on the downstream error correction model, and add them to the candidate result set;

[0125] The sorting module 404 is used to sort the candidate error correction results in the candidate result set and obtain the error correction result corresponding to the input text based on the sorting result.

[0126] Optionally, the error detection module 401 is specifically used for:

[0127] The input text is segmented to obtain several words;

[0128] For word segments containing pinyin, the pinyin in the word segments is converted into Chinese, and the converted word segments are identified as error-prone entries;

[0129] For word segments containing only Chinese characters, the word segments are converted into their corresponding pinyin, and then the corresponding pinyin is converted back into Chinese characters to obtain the converted word segments. If the converted word segments are inconsistent with the corresponding original word segments, and the text similarity between the converted word segments and the original word segments is greater than the similarity threshold, then the converted word segments are identified as error-prone candidate words. The original text is obtained, consisting of the original word segments corresponding to the error-prone candidate words and several word segments located before and after the original word segments. The original word segments in the original text are replaced with error-prone candidate words to obtain the concatenated text. The language scores of the original text and the concatenated text are calculated using a language model. If the language score of the concatenated text is greater than the language score of the original text, then the original word segments are identified as error-prone words.

[0130] Optionally, the error detection module 401 also includes a Pinyin-to-Chinese conversion unit, specifically used for: based on a pre-constructed initial probability distribution, state transition probability distribution, and state-observation probability distribution, combined with the Viterbi algorithm, finding the optimal path to convert the Pinyin in the word segmentation into Chinese, and replacing the Pinyin in the word segmentation with the Chinese corresponding to the optimal path to obtain the converted word segmentation. The initial probability distribution records the probability of each character being at the beginning of the text, the state transition probability distribution records the probability of one character transitioning to another, and the state-observation probability distribution records the probability distribution of a character having a certain pronunciation.

[0131] Optionally, the error detection in the error detection module 401 includes at least one of the following detection methods:

[0132] The system detects whether any words in the input text belong to the error-prone confusion set, and identifies these words as error-prone terms; or

[0133] The input text is segmented, and words not belonging to the preset dictionary are identified as error-prone terms; or

[0134] The input text is segmented into several bigram words according to the 2-gram model, and the language score of each bigram word is obtained using a language model. The input text is also segmented into several ternary words according to the 3-gram model, and the language score of each ternary word is obtained using a language model. For each character in the input text, the language score of the character at that position is determined based on the language scores of the bigram and ternary words containing the character at that position. Based on the language score of the character at each position and a preset score threshold, the error-prone positions in the input text are determined, and error-prone words are obtained based on the characters at the error-prone positions.

[0135] Optionally, the first candidate module 402 is specifically used for:

[0136] For each character in the frequently misspelled terms, replace it with a corresponding homophone, similar-sounding character, similar-looking character, or easily confused character to obtain several correct candidate terms for the frequently misspelled terms; and / or

[0137] The prediction model performs a masking operation on the locations of error-prone words in the input text and generates prediction results corresponding to the locations of error-prone words. The prediction results are then used as the correct candidate words for the error-prone words.

[0138] Optionally, the downstream error correction model includes a Bi-LSTM-based detection network and an ALBERT-based error correction network; the second candidate module 403 is specifically used for:

[0139] The vector representation corresponding to the input text is E = (e1, e2, e3, e4, ..., e n ) Input the detection network to obtain the error probability of each character in the input text, where e i This represents the vector corresponding to the i-th character in the input text;

[0140] The new vector for each character in the input text is obtained based on the following formula:

[0141] e' i =p i *e i,mask +(1-p i )*e i ,

[0142] Among them, e' i Let e ​​represent the new vector of the i-th word. i,mask This indicates that the prediction model is used to evaluate the vector representation E of e. i The result obtained after performing a masking operation for e i The predicted vector at the location, p i This represents the error probability of the i-th character;

[0143] Let E'=(e'1,e'2,e'3,e'4,...,e' n Input the error correction network to obtain several candidate error correction results.

[0144] Optionally, the sorting module 404 is specifically used to: sort the candidate error correction results in the candidate result set based on the local features and global features of each candidate error correction result, wherein the local features include at least one of the frequency features, word segmentation features, phonetic and morphological features and PMI features of each corrected word in the candidate error correction results, and the global features include at least one of the following: the language model score of the candidate error correction result, and the similarity between the candidate error correction result and the corresponding original text.

[0145] The text automatic error correction device proposed in this application adopts the same inventive concept as the above-described text automatic error correction method and can achieve the same beneficial effects, which will not be repeated here.

[0146] Based on the same inventive concept as the aforementioned automatic text correction method, this application also provides an electronic device. This electronic device can specifically be a control device or control system inside a robot, or it can be an external device communicating with intelligent devices, such as a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), server, etc. Figure 5 As shown, the electronic device 50 may include a processor 501 and a memory 502.

[0147] Processor 501 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0148] Memory 502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 502 in this embodiment can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0149] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to: mobile storage devices, random access memory (RAM), magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)) and other media capable of storing program code.

[0150] Alternatively, if the integrated units described above in this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes: mobile storage devices, random access memory (RAM), magnetic memory (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.) and other media capable of storing program code.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic text error correction, characterized in that, include: Error detection is performed on the input text to identify error-prone words contained in the input text; Generate the correct candidate word for each error-prone word, and based on the correct candidate word for each error-prone word and the position of the error-prone word in the input text, construct multiple candidate error correction results and add them to the candidate result set; Based on the downstream error correction model, several candidate error correction results corresponding to the input text are obtained and added to the candidate result set; The candidate error correction results in the candidate result set are sorted, and the error correction result corresponding to the input text is obtained based on the sorting result; The downstream error correction model includes a Bi-LSTM-based detection network and an ALBERT-based error correction network. The detection network is used to predict the error probability of each character in the text, and the error correction network is used to predict the corrected text. The process of obtaining several candidate error correction results corresponding to the input text based on the downstream error correction model includes: The vector representation of the input text The input is given to the detection network to obtain the error probability of each character in the input text, wherein, This represents the vector corresponding to the i-th character in the input text; vector representation In After performing the masking operation, the resulting vector is predicted using a prediction model to obtain... Predicted vector at location The prediction model and the error correction network are independent modules with different functions. The prediction model is used to generate the prediction vector of the masked position, and the error correction network is used to generate the final candidate error correction result based on the fused new vector. The new vector for each character in the input text is obtained based on the following formula: ， in, This represents the new vector of the i-th character. This indicates the vector representation through a prediction model. In The target obtained after performing the masking operation The predicted vector at the location, The new vector represents the error probability of the i-th character; The construction makes: when At higher levels, The closer to ;when The lower the value, the closer ei′ is to The output of the detection network is used as weights to directly control the construction of the input vector of the error correction network. Will Input the error correction network to obtain several candidate error correction results.

2. The method according to claim 1, characterized in that, The error detection of the input text includes: The input text is segmented to obtain several words; For word segments containing pinyin, the pinyin in the word segments is converted into Chinese, and the converted word segments are identified as error-prone entries; For word segments containing only Chinese characters, the word segments are converted into their corresponding pinyin, and then the corresponding pinyin is converted back into Chinese characters to obtain the converted word segments. If the converted word segments are inconsistent with the corresponding original word segments, and the text similarity between the converted word segments and the original word segments is greater than the similarity threshold, then the converted word segments are identified as error-prone candidate words. The original text is obtained, consisting of the original word segments corresponding to the error-prone candidate words and several word segments located before and after the original word segments. The original word segments in the original text are replaced with error-prone candidate words to obtain the concatenated text. The language scores of the original text and the concatenated text are calculated using a language model. If the language score of the concatenated text is greater than the language score of the original text, then the original word segments are identified as error-prone words.

3. The method according to claim 2, characterized in that, The process of converting pinyin in word segmentation into Chinese includes: Based on the pre-constructed initial probability distribution, state transition probability distribution, and state-observation probability distribution, and combined with the Viterbi algorithm, the optimal path to convert the pinyin in the word segmentation into Chinese is found. The pinyin in the word segmentation is replaced with the Chinese corresponding to the optimal path to obtain the converted word segmentation. The initial probability distribution records the probability that each character is at the beginning of the text, the state transition probability distribution records the probability that one character moves to another, and the state-observation probability distribution records the probability distribution that a character has a certain pronunciation.

4. The method according to claim 1, characterized in that, The error detection includes at least one of the following detection methods: The system detects whether any words in the input text belong to the error-prone confusion set, and identifies these words as error-prone terms; or The input text is segmented, and words not belonging to the preset dictionary are identified as error-prone terms; or The input text is segmented into several bigram words according to the 2-gram model, and the language score of each bigram word is obtained using a language model. The input text is also segmented into several ternary words according to the 3-gram model, and the language score of each ternary word is obtained using a language model. For each character in the input text, the language score of the character at that position is determined based on the language scores of the bigram and ternary words containing the character at that position. Based on the language score of the character at each position and a preset score threshold, the error-prone positions in the input text are determined, and error-prone words are obtained based on the characters at the error-prone positions.

5. The method according to any one of claims 1 to 4, characterized in that, The process of generating the correct candidate term for each error-prone term includes: For each character in the frequently misspelled terms, replace it with a corresponding homophone, similar-sounding character, similar-looking character, or easily confused character to obtain several correct candidate terms for the frequently misspelled terms; and / or The prediction model performs a masking operation on the locations of error-prone words in the input text and generates prediction results corresponding to the locations of error-prone words. The prediction results are then used as the correct candidate words for the error-prone words.

6. The method according to any one of claims 1 to 4, characterized in that, The step of sorting the candidate error correction results in the candidate result set includes: Based on the local and global features of each candidate error correction result, the candidate error correction results in the candidate result set are sorted. The local features include at least one of the following: frequency features, word segmentation features, phonetic and morphological features, and PMI features of each corrected word in the candidate error correction results. The global features include at least one of the following: language model score of the candidate error correction result, and similarity between the candidate error correction result and the corresponding original text.

7. A text automatic error correction device, characterized in that, include: An error detection module is used to perform error detection on the input text in order to identify the error-prone words contained in the input text; The first candidate module is used to generate the correct candidate word corresponding to each error-prone word, and based on the correct candidate word corresponding to each error-prone word and the position of the error-prone word in the input text, construct multiple candidate error correction results and add them to the candidate result set; The second candidate module is used to obtain several candidate error correction results corresponding to the input text based on the downstream error correction model, and add them to the candidate result set; The sorting module is used to sort the candidate error correction results in the candidate result set, and obtain the error correction result corresponding to the input text based on the sorting result; The downstream error correction model includes a Bi-LSTM-based detection network and an ALBERT-based error correction network. The detection network is used to predict the error probability of each character in the text, and the error correction network is used to predict the corrected text. The process of obtaining several candidate error correction results corresponding to the input text based on the downstream error correction model includes: The vector representation of the input text The input is given to the detection network to obtain the error probability of each character in the input text, wherein, This represents the vector corresponding to the i-th character in the input text; vector representation In After performing the masking operation, the resulting vector is predicted using a prediction model to obtain... Predicted vector at location The prediction model and the error correction network are independent modules with different functions. The prediction model is used to generate the prediction vector of the masked position, and the error correction network is used to generate the final candidate error correction result based on the fused new vector. The new vector for each character in the input text is obtained based on the following formula: ， in, This represents the new vector of the i-th character. This indicates the vector representation through a prediction model. In The target obtained after performing the masking operation The predicted vector at the location, The new vector represents the error probability of the i-th character; The construction makes: when At higher levels, The closer to ;when The lower the value, the closer ei′ is to The output of the detection network is used as weights to directly control the construction of the input vector of the error correction network. Will Input the error correction network to obtain several candidate error correction results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the computer program instructions implement the steps of the method according to any one of claims 1 to 6.

Images