Error correction method, device, equipment and storage medium based on language model

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses an error correction method based on a language model. The language model is used to perform the same word segmentation processing twice on text data to obtain a first word sequence and a second word sequence. The segmentation probability of keywords is calculated based on the two word sequences, and the text data is re-segmented based on the probability to obtain a third word sequence. The pinyin feature of the keywords is converted based on the third word sequence, and recall processing is performed to obtain candidate words. A keyword that meets the conditions is selected from the candidate words to perform error correction operations on the corresponding keywords in the text data. The present invention also provides an error correction device, equipment and storage medium based on the language model, thereby effectively improving the segmentation and recognition of keywords, and selecting the correct keywords for replacement based on the combination of recall and probability, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an error correction method, device, equipment and storage medium based on a language model. Background Art

[0002] The existing open source NLP (Natural Language Processing) error correction technology only relies on the text word order to correct errors, which is specifically implemented through a language model. In error correction, it is often necessary to first detect the location of the error, then recall the candidate words corresponding to the location of the error, and finally perform the error correction process.

[0003] At present, there are many methods for error detection, and the commonly used methods include sequence labeling, dictionary matching and traditional speech model card threshold methods. For the method using sequence labeling, a large amount of manually annotated corpus is required for supervised training. For the method using dictionary matching, it is very dependent on the quality of the dictionary, and it is difficult to ensure that words not in the dictionary are erroneous. It is difficult to determine the threshold using the traditional speech model. It can be seen that in the existing technology, the detection and correction of erroneous keywords are still relatively complex and the accuracy is not high. Summary of the invention

[0004] The main purpose of the present invention is to provide a language model-based error correction method, device, equipment and storage medium, aiming to solve the technical problems in the prior art that the text error correction detection process is complicated and the error correction accuracy is not high.

[0005] To solve the above-mentioned problem, in a first aspect of the present invention, a language model-based error correction method is provided, and the language model-based error correction method comprises: obtaining text data to be corrected, and performing word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence; calculating the probability that each keyword in the first word sequence and the second word sequence is segmented by the language model, and performing secondary word segmentation processing on the text data according to the probability to obtain a third word sequence; calculating the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary statistics by the keyword; judging whether the word frequency reaches a preset word frequency threshold; if not, converting the corresponding first keyword into a pinyin feature through the language model; recalling the candidate word corresponding to the first keyword according to the pinyin feature using a preset recall model, and selecting the correct word from the candidate words to correct the first keyword.

[0006] Optionally, in a feasible implementation manner of the first aspect of the present invention, the text data is segmented according to the segmentation model in the language model to obtain the first word sequence and the second word sequence, including: according to a forward maximum matching algorithm, based on a left-to-right direction, the field with the most consecutive characters in the text data is matched with a preset word segmentation table, if the field exists in the word segmentation table, a keyword is cut out, until the text data is segmented, and the first word sequence is obtained; according to a reverse maximum matching algorithm, based on a right-to-left direction, the field with the most consecutive characters in the text data is matched with a preset word segmentation table, if the field exists in the word segmentation table, a keyword is cut out, until the text data is segmented, and the second word sequence is obtained.

[0007] Optionally, in a feasible implementation manner of the first aspect of the present invention, the calculating the probability of each keyword in the first word sequence and the second word sequence appearing when segmented in the language model, and performing secondary segmentation processing on the text data according to the probability to obtain the third word sequence includes: determining the same second keyword and different third keywords in the first word sequence and the second word sequence; respectively calculating the probability of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented during the segmentation of the historical text data by the language model, and calculating the total probability of all keywords on the left and right sides; selecting the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and selecting the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence; forming a new word sequence based on the word sequence segments selected from the left and right sides of the second keyword, and obtaining the third word sequence.

[0008] Optionally, in a feasible implementation of the first aspect of the present invention, the preset dictionary contains at least keywords and homophonic keywords of the keywords, and the calculation of the word frequency of each keyword in the third word sequence includes: obtaining the index corresponding to each keyword in the third word sequence according to the correspondence between the keywords and the index; querying the corresponding keywords in the preset dictionary and the homophonic keywords corresponding to the keywords according to the index; calculating the ratio of the keywords to the sum of the keywords and the homophonic keywords, and obtaining the word frequency of each keyword in the third word sequence.

[0009] Optionally, in a feasible implementation of the first aspect of the present invention, the converting of the corresponding first keyword into pinyin features through the language model includes: performing pronunciation training on the first keyword according to the acoustic model in the language model to obtain the pronunciation syllables of the first keyword; converting the pronunciation syllables into corresponding pinyin characters according to the encoding model in the language model, and sorting the pinyin characters in order of pronunciation, and outputting the corresponding pinyin features.

[0010] Optionally, in a feasible implementation of the first aspect of the present invention, the method of recalling candidate words corresponding to the first keyword using a preset recall model based on the pinyin feature, and selecting a correct word from the candidate words to correct the first keyword includes: querying keywords with the same pinyin feature from a homophone dictionary based on the pinyin feature to form a homophone recall set; querying the corresponding inverted index based on each single word in the first keyword; querying the corresponding keyword from a preset dictionary based on the inverted index to form an inverted recall set; calculating the intersection of the homophone recall set and the inverted recall set to obtain the correction keyword for the first keyword, and replacing the first keyword.

[0011] Optionally, in a feasible implementation of the first aspect of the present invention, after calculating the intersection of the homophone recall set and the inverted recall set to obtain the correction keyword of the first keyword, before replacing the first keyword according to the correction keyword, it also includes: calculating the edit distance between the first keyword and each keyword in the intersection, wherein the edit distance refers to the minimum number of editing operations required to convert the first keyword into a keyword in the intersection, and the keyword with the shortest edit distance in the intersection is the keyword with the highest similarity to the first keyword; and selecting a keyword with the shortest edit distance as the correction keyword.

[0012] In addition, to solve the above-mentioned problem, in the second aspect of the present invention, a language model-based error correction device is provided, and the language model-based error correction device includes: a first segmentation module, used to obtain text data to be corrected, and according to the segmentation model in the language model, the text data is segmented to obtain a first word sequence and a second word sequence; a second segmentation module, used to calculate the probability that each keyword in the first word sequence and the second word sequence is segmented by the language model, and according to the probability, the text data is subjected to a secondary word segmentation process to obtain a third word sequence; a word frequency statistics module, used to calculate the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary statistics by the keyword; a judgment module, used to judge whether the word frequency reaches a preset word frequency threshold; a conversion module, used to convert the corresponding first keyword into a pinyin feature through the language model when the word frequency does not reach the preset word frequency threshold; an error correction module, used to recall the candidate word corresponding to the first keyword using a preset recall model according to the pinyin feature, and select the correct word from the candidate words to correct the first keyword.

[0013] Optionally, in a feasible implementation of the second aspect of the present invention, the first segmentation module includes a forward segmentation unit and a reverse segmentation unit; the forward segmentation unit is used to match the field with the most consecutive characters in the text data with a preset word segmentation table based on a left-to-right direction according to a forward maximum matching algorithm, and if the field exists in the word segmentation table, a keyword is cut out, until the text data is segmented, and the first word sequence is obtained; the reverse segmentation unit is used to match the field with the most consecutive characters in the text data with a preset word segmentation table based on a right-to-left direction according to a reverse maximum matching algorithm, and if the field exists in the word segmentation table, a keyword is cut out, until the text data is segmented, and the second word sequence is obtained.

[0014] Optionally, in a feasible implementation of the second aspect of the present invention, the second segmentation module includes a determination unit, a calculation unit, an extraction unit and a combination unit; the determination unit is used to determine the same second keyword and different third keywords in the first word sequence and the second word sequence; the calculation unit is used to respectively calculate the probability of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented out during the segmentation of the historical text data by the language model, and calculate the total probability of all keywords on the left and right sides; the extraction unit is used to select the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and select the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence; the combination unit is used to compose a new word sequence based on the word sequence segments selected from the left and right sides of the second keyword, and the second keyword, to obtain the third word sequence.

[0015] Optionally, in a feasible implementation of the second aspect of the present invention, the word frequency statistics module includes a query unit and a word frequency calculation unit; the query unit is used to obtain the index corresponding to each keyword in the third word sequence according to the correspondence between the keyword and the index; according to the index, the corresponding keyword in the preset dictionary and the homophonic keyword corresponding to the keyword are queried; the word frequency calculation unit is used to calculate the ratio of the keyword to the sum of the keyword and the homophonic keyword to obtain the word frequency of each keyword in the third word sequence.

[0016] Optionally, in a feasible implementation of the second aspect of the present invention, the conversion module includes a pronunciation training unit and an encoding unit; the pronunciation training unit is used to perform pronunciation training on the first keyword according to the acoustic model in the language model to obtain the pronunciation syllables of the first keyword; the encoding unit is used to convert the pronunciation syllables into corresponding pinyin characters according to the encoding model in the language model, sort the pinyin characters in order of pronunciation, and output corresponding pinyin features.

[0017] Optionally, in a feasible implementation of the second aspect of the present invention, the error correction module includes a homophone query unit, an index query unit and a recall unit; the homophone query unit is used to query keywords with the same pinyin features from a homophone dictionary according to the pinyin features to form a homophone recall set; the index query unit is used to query the corresponding inverted index according to each single word in the first keyword; according to the inverted index, the corresponding keyword is queried from a preset dictionary to form an inverted recall set; the recall unit is used to calculate the intersection of the homophone recall set and the inverted recall set, obtain the error correction keyword of the first keyword, and replace the first keyword.

[0018] Optionally, in a feasible implementation of the second aspect of the present invention, the error correction module also includes a distance calculation unit, which is used to calculate the edit distance between the first keyword and each keyword in the intersection, wherein the edit distance refers to the minimum number of editing operations required to convert the first keyword into a keyword in the intersection, and the keyword with the shortest edit distance in the intersection is the keyword with the highest similarity to the first keyword; and the keyword with the shortest edit distance is selected as the error correction keyword.

[0019] In addition, to solve the above-mentioned problem, a language model-based error correction device is provided in the third aspect of the present invention, and the error correction device includes: a memory, a processor, and a computer-readable program stored in the memory and executable on the processor, and when the computer-readable program is executed by the processor, the language model-based error correction method as described in any one of the above items is implemented.

[0020] In addition, to solve the above-mentioned problems, a computer-readable storage medium is provided in the fourth aspect of the present invention, on which a computer-readable program is stored. When the computer-readable program is executed by one or more processors, the language model-based error correction method as described in any of the above items is implemented.

[0021] The present invention provides a language model-based error correction method, device, equipment and storage medium. By calculating the word frequency of each word in the corpus, it is judged whether the word will be wrong based on the word frequency. The judgment principle is based on the characteristic that the correct expression is much greater than the wrong expression. Based on this characteristic, whether there is an erroneous word in the corpus to be corrected, the word is recalled and corrected, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of a terminal provided by the present invention;

[0023] Figure 2 A schematic diagram of a flow chart of a first embodiment of an error correction method provided by the present invention;

[0024] Figure 3 A schematic diagram of a flow chart of a second embodiment of the error correction method provided by the present invention;

[0025] Figure 4 A schematic diagram of the mobile frame segmentation keywords provided by the present invention;

[0026] Figure 5 Another schematic diagram of the mobile frame segmentation keyword provided by the present invention;

[0027] Figure 6A schematic diagram of the forward and reverse maximum matching algorithm word segmentation provided by the present invention;

[0028] Figure 7 A schematic diagram of functional modules of an embodiment of an error correction device provided by the present invention;

[0029] Figure 8 A schematic diagram of functional modules of another implementation of the error correction device provided by the present invention.

[0030] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0033] The embodiment of the present invention provides an error correction scheme based on a language model. The scheme mainly calculates the word frequency of each word in the corpus, and judges whether the word is wrong based on the word frequency. The judgment principle is based on the characteristic that the correct statement is much greater than the wrong statement. Based on this characteristic, whether there is an erroneous word in the corpus to be corrected, the word is recalled and corrected, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection.

[0034] Figure 2 The flowchart of the error correction method provided by the embodiment of the present invention is mainly used to realize that when inputting corpus, the error information in the input corpus can be quickly identified and corrected, and it can also be applied to the inspection and error correction processing of certain documents. The error correction method specifically includes the following steps:

[0035] 201, obtaining text data to be corrected, and performing word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence;

[0036] In this step, the text data may be text information currently input by the user, or text information converted from the voice input by the user, or even text information of some work templates related to the insurance business.

[0037] In this embodiment, the language model is a model obtained by training with a variety of different algorithms. Specifically, the language model can realize the function of word segmentation and can also realize the extraction and conversion of pinyin features.

[0038] In actual applications, when text data is segmented, the text data is specifically divided according to the segmentation rules of the segmentation model. For example, two bytes are divided at a time, and they can be separated by the space bar or by a slash. Furthermore, after the division is completed, it also includes judging whether the divided words can be formed into words and whether they belong to the conventional words in the business scenario. If not, the adjacent segmented words are reorganized and re-divided. Of course, it can also be divided in a 2-3-2 manner, etc.

[0039] In this embodiment, the segmentation model can also be a segmentation model trained by a bidirectional maximum matching algorithm. The bidirectional maximum matching algorithm is a collection of two algorithms, mainly including: a forward maximum matching algorithm and a reverse maximum matching algorithm. If the two algorithms obtain the same word segmentation results, it is considered that the segmentation is successful. Otherwise, there is ambiguity or an unregistered word problem.

[0040] 202, calculating the probability of each keyword in the first word sequence and the second word sequence being segmented by the language model, and performing secondary word segmentation processing on the text data according to the probability to obtain a third word sequence;

[0041] In this step, after obtaining two word sequences with different division results, the maximum probability of each keyword in the two word sequences is calculated according to the keyword division probability calculation formula, and the sum of the probabilities of the keywords in each word sequence is determined to determine whether the sum is greater than the probability of the correctly segmented words. If so, step S30 is executed to directly calculate the word frequency of the keywords in the word sequence. If not, the keywords of the text data are re-divided. In the process of re-dividing, the keywords whose individual keyword probabilities are greater than the probability of the correctly segmented words are retained, and other keywords that do not meet the requirements are reorganized and divided.

[0042] In this embodiment, during the secondary word segmentation process, the word segmentation process also includes matching the re-divided keywords with the word list. Only those that match can be counted as word segments. If they do not meet the requirements, the segmentation length is extended.

[0043] 203, calculating the word frequency of each keyword in the third word sequence;

[0044] In this embodiment, the word frequency is obtained by querying a preset dictionary with the keyword. For this preset dictionary, it is a keyword dictionary learned in advance from the corpus of observing user or robot conversations in different business scenarios, or the dictionary stores standard text data of different business types. By counting the number of times the keyword appears in the standard text data, and calculating the probability of occurrence based on this parameter and the total number of times of the standard text data, this probability is the word frequency.

[0045] 204. Determine whether the word frequency reaches a preset word frequency threshold;

[0046] In this embodiment, for different business scenarios, the incorrect occurrences of some words will be different. For example, in the insurance business scenario, "Ping An Fu" is the correct expression, while "Ping An Fu" is the incorrect expression. However, in general use, "Ping An Fu" is more common. Therefore, by determining whether the word frequency of "Ping An Fu" meets the requirements in the insurance scenario, if not, it is determined that it is an incorrect keyword, and error correction processing is required.

[0047] For example, after segmenting "right calf vein thrombosis insurance application" into "right calf / vein / thrombosis / insurance application"; because the character "拴" is incorrect, the frequencies of the 2-gram words: "venous thrombosis" and "thrombosis insurance application" are both less than 10; but its correct expression "venous embolism" is 121, which is more than ten times that of the error. If the frequencies of the previous and subsequent 2-grams are both very low or the frequencies of the candidates homophonic to the current 2-gram are very high, it can be considered that there is probably an error at this position.

[0048] 205. If not, convert the corresponding first keyword into a pinyin feature through the language model;

[0049] In this step, the conversion is specifically performed by querying the dictionary, or it can also be performed through an encoding model.

[0050] 206. According to the pinyin feature, recall the candidate words corresponding to the first keyword by using a preset recall model, and select the correct word from the candidate words to correct the first keyword.

[0051] In this step, the recall specifically includes homophone recall and inverted index recall. The homophone recall is to find homophonic words according to the specific pinyin sequence of the keyword. Further, the recall here is mainly to recall candidate words, and then select the keyword that meets the requirements of this text data from the candidate words.

[0052] Through the implementation of the above method, the text data is subjected to the same word segmentation processing twice through the language model to obtain the first word sequence and the second word sequence, the segmentation probability of the keywords is calculated based on the two word sequences, the text data is re-segmented based on the probability to obtain the third word sequence, the pinyin feature conversion of the keywords is performed on the basis of the third word sequence, and the recall processing is performed to obtain candidate words, and a keyword that meets the conditions is selected from the candidate words to perform error correction operations on the corresponding keywords in the text data, thereby effectively improving the segmentation and recognition of keywords, and at the same time, the correct keywords are selected for replacement based on the combination of recall and probability, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection.

[0053] Figure 3 The second implementation process of the error correction method provided in the embodiment of the present invention mainly expands candidate words according to the pinyin features of the suspected error object, and then selects a closest one from the candidate words to implement the error correction replacement processing, and the closest here means the highest probability, and the specific implementation steps are as follows:

[0054] 301, obtaining text data to be corrected;

[0055] This step also includes copying the text data to obtain two copies of the text data, and performing two word segmentation operations based on the text data copies.

[0056] 302, matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a left-to-right direction according to a forward maximum matching algorithm;

[0057] 303, if the field exists in the word segmentation table, a keyword is segmented out, until the text data segmentation is completed to obtain the first word sequence;

[0058] In this embodiment, in the process of word segmentation through the forward maximum matching algorithm, the first step is to determine the field length of the text box selection, select the field from the text data copy based on the field length, form a keyword sequence, and query from the word segmentation table based on the keyword sequence to determine whether the keyword exists. If not, adjust the field length of the text box selection, and re-segment the text data copy based on the adjusted field length until all matches are found.

[0059] 304, matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a right-to-left direction according to a reverse maximum matching algorithm;

[0060] 305, if the field exists in the word segmentation table, a keyword is segmented out, until the text data segmentation is completed to obtain the second word sequence;

[0061] In this embodiment, the process of reverse matching and segmenting the text data copy is the same as that of forward matching and segmenting, and will not be repeated here.

[0062] In practical applications, when the forward and reverse maximum matching algorithm is used to segment text data, when setting the text box, it can be implemented in the following ways: Figure 4 As shown, a moving box is set, starting from the first word of the text data, and the continuous characters selected in the moving box are matched with the preset vocabulary. If they match, the continuous characters selected in the moving box are cut out, and then the starting position of the moving box is adjusted to the next character of the continuous characters; if they do not match, the moving box is expanded by the length of one character, and the continuous characters selected in the adjusted moving box are re-matched with the vocabulary.

[0063] For example, first set the moving box to a size of two characters, and then based on the text box, for "I have a cold these days, can I cast a peace charm?", that is, start from the word "I" and select "I here", and use "I here" to query whether there is the same keyword in the word segmentation table. If not, lengthen the moving box and set it to three characters until it is found in the word segmentation table, that is, "I these days", and move the moving box to the position after the keyword segmented by the value (that is, after the word "day"), and restore the field length of the moving box to segment the remaining text data, such as Figure 5 shown.

[0064] 306, calculating the probability that each keyword in the first word sequence and the second word sequence is segmented by the language model, and performing secondary word segmentation processing on the text data according to the probability to obtain a third word sequence;

[0065] In this step, the process of forming the third word sequence can be specifically implemented in the following manner:

[0066] Determine the same second keyword and different third keyword in the first word sequence and the second word sequence;

[0067] Respectively calculating the probabilities of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented in the process of segmenting the historical text data by the language model, and calculating the total probability of all the keywords on the left and right sides;

[0068] Select the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and select the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence;

[0069] The third word sequence is obtained based on the word sequence segments selected from the left and right of the second keyword and the new word sequence composed of the second keyword.

[0070] In practical applications, such as Figure 6 As shown, for "our soldier comrades want to go down to eat", a bidirectional maximum algorithm is used. The division result of the forward maximum matching algorithm is: "our / soldier / comrades / want / to / go down to eat / eat", and the division result of the reverse maximum matching algorithm is: "our / soldier / comrades / want / to / go down to eat / eat". Based on these two division results, the same participles and different participles are separated, and the different participles at both ends of the same participle are determined. The total probability of the different participles at both ends of the same participle is calculated in each word sequence, that is, the total probability of the small participle sequence marked as ①②③④ in the figure, and then a small participle sequence with a higher probability is selected from the two ends, and the selected small participle sequence is spliced ​​to form a new word sequence, that is, the third word sequence. For example, ①>②,③<④, then ① and ④ are selected to be re-spliced ​​to form a new word sequence.

[0071] In this embodiment, after determining the keywords with a higher probability of word segmentation, the word frequency of each keyword in the third word sequence is calculated, and it is determined whether the keyword is wrong based on the word frequency. In practical applications, the calculation of the word frequency can be specifically counted by indexing, counting the number of times it appears in the dictionary, selecting the keyword by the word frequency, and then expanding the features of the keyword. Optionally, here, the feature of the keyword is expanded by using homophones.

[0072] 307, calculating the word frequency of each keyword in the third word sequence;

[0073] In this embodiment, when the preset dictionary contains at least a keyword and a homophone keyword of the keyword, this step is specifically implemented as follows:

[0074] According to the correspondence between the keywords and the indexes, obtaining the index corresponding to each keyword in the third word sequence;

[0075] Querying the preset dictionary for corresponding keywords and homophonic keywords corresponding to the keywords according to the index;

[0076] The ratio of the keyword to the sum of the keyword and homophonic keywords is calculated to obtain the word frequency of each keyword in the third word sequence.

[0077] In practical applications, the dictionary and index are specifically obtained by pre-learning historical dialogue corpus, segmenting the corpus, and counting the 2gram frequency dictionary of the corpus after segmentation, a homophone dictionary based on the pinyin level of 2gram; an inverted dictionary based on the 2gram character level; for example, after the word segmentation of "right calf venous thrombosis insurance", "right calf / vein / embolism / insurance" is obtained; the frequency of occurrence of one of the 2gram words "venous thrombosis" in the corpus is counted: 121; the pinyin of "venous thrombosis" is taken: jingmai shuan sai; other 2gram words with the same pinyin as "venous thrombosis" in the corpus are grouped together, such as "meridian thrombosis"; and an inverted index is established for each word of "venous thrombosis", such as one of the words suture->(venous thrombosis, fire hydrant, suppository,..).

[0078] 308, determining whether the word frequency reaches a preset word frequency threshold;

[0079] 309, if not, performing pronunciation training on the corresponding first keyword according to the acoustic model in the language model to obtain the pronunciation syllables of the first keyword;

[0080] 310, converting the pronounced syllables into corresponding pinyin characters according to the encoding model in the language model, sorting the pinyin characters in the order of pronunciation, and outputting corresponding pinyin features;

[0081] In this embodiment, the acoustic model is specifically learned based on the correspondence between Chinese keywords and pinyin. In actual applications, the correspondence is queried through the face model and Chinese keywords to extract the corresponding pinyin features, and the corresponding homophones are queried based on the pinyin features. Finally, the corresponding correct keywords are selected from the homophones for replacement and error correction.

[0082] 311, according to the pinyin feature, searching for keywords with the same pinyin feature from a preset homophone dictionary to form a homophone recall set;

[0083] 312, according to each word in the first keyword, query the corresponding preset inverted index;

[0084] 213, according to the inverted index, query the corresponding keywords from the preset inverted index dictionary to form an inverted recall set;

[0085] 314 , calculating the intersection of the homophone recall set and the inverted recall set, obtaining a correction keyword for the first keyword, and replacing the first keyword.

[0086] In this embodiment, when the corresponding homophonic keywords are queried, it can be specifically implemented through a recall algorithm. In practical applications, the corresponding candidates are recalled by using the above-mentioned homophone dictionary and inverted dictionary:

[0087] Homophone recall: Incorrect "shuan sai tou bao" -> jing mai shuan sai -> Correct "venous embolism";

[0088] Inverted index recall: Use the intersection of the inverted indexes of (jing, mai, sai) to recall the correct "venous embolism".

[0089] In this embodiment, the error detection and correction processing of the text or statement are performed in the above-mentioned manner. Based on the method of comparing word frequencies, the characteristic that the correct expression is greater than the incorrect expression is fully utilized, providing a strong theoretical basis for error detection. Avoiding traditional methods such as sequence labeling models, which are black-box algorithms that do not require relying on manually labeled data, only need to count a large number of corpora in the existing field. For example, the customer question corpora in life insurance can be used to count the word frequency information of word-level ngrams, which is convenient to be migrated to other fields. By this method, only some simple table lookup operations are required, thus greatly improving the efficiency.

[0090] To further improve the screening of correct keywords, this embodiment further includes before error correction and replacement:

[0091] Calculate the edit distance between the first keyword and each keyword in the intersection. Here, the edit distance refers to the minimum number of edit operations required to convert the first keyword into the keyword in the intersection. The keyword with the shortest edit distance in the intersection is the keyword with the highest similarity to the first keyword;

[0092] Select the keyword with the smallest edit distance as the error correction keyword.

[0093] For the above-mentioned edit operations, the edit operations can include: character replacement operation, character insertion operation, character deletion operation, etc. For example, converting "sheng ming bu neng cheng shou zhi qing" to "sheng ming bu neng cheng shou zhi qing" only requires three steps: replacing "sheng" with "sheng", replacing "ming" with "ming", and replacing "qing" with "qing". Therefore, "sheng ming bu neng cheng shou zhi qing" is used as the word with the highest similarity to "sheng ming bu neng cheng shou zhi qing".

[0094] To sum up, by calculating the word frequency of keywords and judging whether it is a wrong word based on the word frequency, if it is wrong, recall and correct it according to the pinyin, so as to reduce the difficulty of detection and improve the accuracy and efficiency of detection.

[0095] In order to solve the above problems, the embodiment of the present invention also provides an error correction device based on a language model, such as Figure 7 As shown, the error correction device comprises:

[0096] A first segmentation module 701 is used to obtain text data to be corrected, and perform word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence;

[0097] A second segmentation module 702 is used to calculate the probability of each keyword in the first word sequence and the second word sequence being segmented by the language model, and perform secondary word segmentation processing on the text data according to the probability to obtain a third word sequence;

[0098] A word frequency statistics module 703, used to calculate the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary for the keyword;

[0099] A determination module 704 is used to determine whether the word frequency reaches a preset word frequency threshold;

[0100] A conversion module 705, configured to convert the corresponding first keyword into a pinyin feature through the language model when the word frequency does not reach a preset word frequency threshold;

[0101] The error correction module 706 is used to recall the candidate words corresponding to the first keyword using a preset recall model according to the pinyin features, and select the correct word from the candidate words to correct the first keyword.

[0102] The above-mentioned device is used to implement error correction processing of phrases or fields in the text. The text data is subjected to the same word segmentation processing twice through the language model to obtain a first word sequence and a second word sequence. The segmentation probability of the keywords is calculated based on the two word sequences. The text data is re-segmented based on the probability to obtain a third word sequence. The pinyin features of the keywords are converted based on the third word sequence, and recall processing is performed to obtain candidate words. A keyword that meets the conditions is selected from the candidate words to perform error correction operations on the corresponding keywords in the text data, thereby effectively improving the segmentation and recognition of the keywords. At the same time, the correct keywords are selected for replacement based on the combination of recall and probability, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection.

[0103] like Figure 8 As shown, the embodiment of the present invention further provides another error correction device, which includes:

[0104] A first segmentation module 701 is used to obtain text data to be corrected, and perform word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence;

[0105] A second segmentation module 702 is used to calculate the probability of each keyword in the first word sequence and the second word sequence being segmented by the language model, and perform secondary word segmentation processing on the text data according to the probability to obtain a third word sequence;

[0106] A word frequency statistics module 703, used to calculate the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary for the keyword;

[0107] A determination module 704 is used to determine whether the word frequency reaches a preset word frequency threshold;

[0108] A conversion module 705, configured to convert the corresponding first keyword into a pinyin feature through the language model when the word frequency does not reach a preset word frequency threshold;

[0109] The error correction module 706 is used to recall the candidate words corresponding to the first keyword using a preset recall model according to the pinyin features, and select the correct word from the candidate words to correct the first keyword.

[0110] In another embodiment of the present invention, the first segmentation module 701 includes a forward segmentation unit 7011 and a reverse segmentation unit 7012;

[0111] The forward segmentation unit 7011 is used to match the field with the most consecutive characters in the text data with a preset word segmentation table based on the forward maximum matching algorithm from left to right, and if the field exists in the word segmentation table, a keyword is segmented out, until the text data segmentation is completed to obtain the first word sequence;

[0112] The reverse segmentation unit 7012 is used to match the field with the most consecutive characters in the text data with a preset word segmentation table based on the reverse maximum matching algorithm from right to left. If the field exists in the word segmentation table, a keyword is segmented out until the text data is segmented and the second word sequence is obtained.

[0113] Optionally, the second segmentation module 702 includes a determination unit 7021, a calculation unit 7022, an extraction unit 7023 and a combination unit 7024;

[0114] The determining unit 7021 is used to determine the same second keyword and different third keyword in the first word sequence and the second word sequence;

[0115] The calculation unit 7022 is used to calculate the probability of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented in the process of segmenting the historical text data by the language model, and calculate the total probability of all the keywords on the left and right sides;

[0116] The extraction unit 7023 is used to select the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and select the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence;

[0117] The combining unit 7024 is used to form a new word sequence based on the word sequence segments selected from the left and right of the second keyword and the second keyword to obtain the third word sequence.

[0118] In this embodiment, the word frequency statistics module 703 includes a query unit 7031 and a word frequency calculation unit 7032;

[0119] The query unit 7031 is used to obtain the index corresponding to each keyword in the third word sequence according to the corresponding relationship between the keyword and the index; query the corresponding keyword in the preset dictionary and the homophonic keyword corresponding to the keyword according to the index;

[0120] The word frequency calculation unit 7032 is used to calculate the ratio of the keyword to the sum of the keyword and the homophonic keywords to obtain the word frequency of each keyword in the third word sequence.

[0121] Optionally, in another feasible implementation, the conversion module 705 includes a pronunciation training unit 7051 and an encoding unit 7052;

[0122] The pronunciation training unit 7051 is used to perform pronunciation training on the first keyword according to the acoustic model in the language model to obtain the pronunciation syllables of the first keyword;

[0123] The encoding unit 7052 is used to convert the pronounced syllables into corresponding pinyin characters according to the encoding model in the language model, sort the pinyin characters in the order of pronunciation, and output corresponding pinyin features.

[0124] Optionally, in another feasible implementation of this embodiment, the error correction module 706 includes a homophone query unit 7061, an index query unit 7062 and a recall unit 7063;

[0125] The homophone query unit 7061 is used to query keywords with the same pinyin feature from a homophone dictionary according to the pinyin feature to form a homophone recall set;

[0126] The index query unit 7062 is used to query the corresponding inverted index according to each word in the first keyword; according to the inverted index, query the corresponding keyword from the preset dictionary to form an inverted recall set;

[0127] The recall unit 7063 is used to calculate the intersection of the homophone recall set and the inverted recall set, obtain the error correction keyword of the first keyword, and replace the first keyword.

[0128] In this embodiment, the error correction module 706 also includes a distance calculation unit 7064, which is used to calculate the edit distance between the first keyword and each keyword in the intersection, wherein the edit distance refers to the minimum number of editing operations required to convert the first keyword into a keyword in the intersection, and the keyword with the shortest edit distance in the intersection is the keyword with the highest similarity to the first keyword; and a keyword with the shortest edit distance is selected as the error correction keyword.

[0129] Since the execution functions of the device and the execution process corresponding to the functions are the same as those described in the above-mentioned error correction method embodiment of the present invention, this embodiment does not elaborate on the contents of the error correction device embodiment.

[0130] In the embodiment of the present invention, the error correction device may be implemented in the form of a server, that is, the device for implementing the error correction method is set as a function on the server in the input method system.

[0131] The present invention also provides an error correction device based on a language model, the error correction device comprising: a memory, a processor, and a computer-readable program stored in the memory and executable on the processor. The method implemented when the computer-readable program is executed by the processor can refer to the various embodiments of the error correction method based on the language model of the present invention, so it will not be described in detail.

[0132] In practical applications, the error correction device here can be an existing terminal structure, and a mobile terminal is commonly used. The error correction function is started by the scanning function of the mobile terminal, and the function can be realized by setting a computer readable program to implement the function of the above-mentioned error correction method, such as Figure 1 As shown, a schematic diagram of the structure of the operating environment of the terminal involved in the embodiment of the present invention.

[0133] like Figure 1As shown, the terminal includes: a processor 101, such as a CPU, a communication bus 102, a user interface 103, a network interface 104, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 103 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the network interface 104 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 105 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 105 may optionally also be a storage device independent of the aforementioned processor 101.

[0134] Those skilled in the art will understand that Figure 1 The hardware structure of the terminal shown in the figure does not constitute a limitation on the error correction device and apparatus in the present invention, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0135] like Figure 1 As shown, the memory 105 as a computer-readable storage medium may include an operating system, a network communication program module, a user interface program module, and a computer-readable program / instruction for implementing the mail forwarding method. Among them, the operating system is a computer-readable program / instruction for scheduling the communication between the modules in the terminal and executing the error correction method in the above embodiment.

[0136] exist Figure 1 In the hardware structure of the terminal shown, the network interface 104 is mainly used to access the network; the user interface 603 is mainly used to monitor and obtain email data to be sent, wherein the email data can be online emails or offline letters, and the processor 101 can be used to call the computer-readable program stored in the memory 105 and execute the operations of the various embodiments of the following error correction method.

[0137] The present invention also provides a computer-readable storage medium.

[0138] In this embodiment, a computer-readable program is stored on the computer-readable storage medium. The method implemented when the computer-readable program is executed by one or more processors can refer to the various embodiments of the error correction method based on the language model of the present invention, so it will not be described in detail.

[0139] The method and device provided in the embodiment of the present invention calculate the word frequency of each word in the corpus, and judge whether the word is wrong based on the word frequency. The judgment principle is based on the characteristic that the correct expression is much greater than the wrong expression. Based on this characteristic, whether there is an erroneous word in the corpus to be corrected, the word is recalled and corrected, thereby reducing the difficulty of detection and improving the accuracy and efficiency of detection.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0141] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

Claims

1. A language model-based error correction method, characterized in that: The error correction method comprises the following steps: Acquire text data to be corrected, and perform word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence; Calculating the probability of each keyword in the first word sequence and the second word sequence being segmented by the language model, and performing secondary word segmentation processing on the text data according to the probability to obtain a third word sequence; Calculating the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary for the keyword; Determining whether the word frequency reaches a preset word frequency threshold; If not, converting the corresponding first keyword into a pinyin feature through the language model; According to the pinyin feature, recall candidate words corresponding to the first keyword using a preset recall model, and select a correct word from the candidate words to correct the first keyword; The word segmentation processing of the text data according to the segmentation model in the language model to obtain the first word sequence and the second word sequence includes: matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a forward maximum matching algorithm from left to right, and if the field exists in the word segmentation table, a keyword is segmented out until the text data segmentation is completed to obtain the first word sequence; matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a reverse maximum matching algorithm from right to left, and if the field exists in the word segmentation table, a keyword is segmented out until the text data segmentation is completed to obtain the second word sequence; The method of calculating the probability of each keyword in the first word sequence and the second word sequence appearing when segmenting the words in the language model, and performing secondary segmentation processing on the text data according to the probability to obtain the third word sequence includes: determining the same second keyword and different third keywords in the first word sequence and the second word sequence; respectively calculating the probability of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented during the segmentation of the historical text data by the language model, and calculating the total probability of all keywords on the left and right sides; selecting the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and selecting the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence; forming a new word sequence based on the word sequence segments selected from the left and right sides of the second keyword and the second keyword to obtain the third word sequence.

2. The error correction method based on language model according to claim 1, characterized in that: The preset dictionary contains at least keywords and homophones of the keywords, and the calculating of the word frequency of each keyword in the third word sequence includes: According to the correspondence between the keywords and the indexes, obtaining the index corresponding to each keyword in the third word sequence; Querying the preset dictionary for corresponding keywords and homophonic keywords corresponding to the keywords according to the index; The ratio of the keyword to the sum of the keyword and homophonic keywords is calculated to obtain the word frequency of each keyword in the third word sequence.

3. The error correction method based on language model according to claim 2, characterized in that: The converting the corresponding first keyword into a pinyin feature through the language model includes: Performing pronunciation training on the first keyword according to the acoustic model in the language model to obtain pronunciation syllables of the first keyword; According to the encoding model in the language model, the pronounced syllables are converted into corresponding pinyin characters, and the pinyin characters are sorted in the order of pronunciation, and the corresponding pinyin features are output.

4. The error correction method based on language model according to claim 3, characterized in that: The step of recalling candidate words corresponding to the first keyword using a preset recall model according to the pinyin feature, and selecting a correct word from the candidate words to correct the first keyword includes: According to the pinyin feature, searching for keywords with the same pinyin feature from a homophone dictionary to form a homophone recall set; According to each word in the first keyword, query the corresponding inverted index; According to the inverted index, corresponding keywords are searched from a preset dictionary to form an inverted recall set; The intersection of the homophone recall set and the inverted recall set is calculated to obtain an error correction keyword for the first keyword, and the first keyword is replaced.

5. The error correction method based on language model according to claim 4, characterized in that: After calculating the intersection of the homophone recall set and the inverted recall set to obtain the error correction keyword of the first keyword, and before replacing the first keyword according to the error correction keyword, the method further includes: Calculate the edit distance between the first keyword and each keyword in the intersection, wherein the edit distance refers to the minimum number of edit operations required to convert the first keyword into a keyword in the intersection, and the keyword with the shortest edit distance in the intersection is the keyword with the highest similarity to the first keyword; A keyword with the smallest edit distance is selected as the error correction keyword.

6. An error correction device based on a language model, characterized in that: The error correction device based on the language model comprises: A first segmentation module is used to obtain text data to be corrected, and perform word segmentation processing on the text data according to the segmentation model in the language model to obtain a first word sequence and a second word sequence; A second segmentation module is used to calculate the probability that each keyword in the first word sequence and the second word sequence is segmented by the language model, and perform secondary word segmentation processing on the text data according to the probability to obtain a third word sequence; A word frequency statistics module, used to calculate the word frequency of each keyword in the third word sequence, wherein the word frequency is obtained by querying a preset dictionary for the keyword; A judgment module, used to judge whether the word frequency reaches a preset word frequency threshold; A conversion module, configured to convert the corresponding first keyword into a pinyin feature through the language model when the word frequency does not reach a preset word frequency threshold; An error correction module, configured to recall candidate words corresponding to the first keyword using a preset recall model according to the pinyin feature, and select a correct word from the candidate words to correct the first keyword; The word segmentation processing of the text data according to the segmentation model in the language model to obtain the first word sequence and the second word sequence includes: matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a forward maximum matching algorithm from left to right, and if the field exists in the word segmentation table, a keyword is segmented out until the text data segmentation is completed to obtain the first word sequence; matching the field with the most consecutive characters in the text data with a preset word segmentation table based on a reverse maximum matching algorithm from right to left, and if the field exists in the word segmentation table, a keyword is segmented out until the text data segmentation is completed to obtain the second word sequence; The method of calculating the probability of each keyword in the first word sequence and the second word sequence appearing when segmenting the words in the language model, and performing secondary segmentation processing on the text data according to the probability to obtain the third word sequence includes: determining the same second keyword and different third keywords in the first word sequence and the second word sequence; respectively calculating the probability of the third keywords located on the left and right sides of the second keyword in the first word sequence and the second word sequence being segmented during the segmentation of the historical text data by the language model, and calculating the total probability of all keywords on the left and right sides; selecting the word sequence segment with a large total probability from the left side of the second keyword in the first word sequence and the second word sequence, and selecting the word sequence segment with a large total probability from the right side of the second keyword in the first word sequence and the second word sequence; forming a new word sequence based on the word sequence segments selected from the left and right sides of the second keyword and the second keyword to obtain the third word sequence.

7. An error correction device based on a language model, characterized in that: The language model-based error correction device includes: a memory, a processor, and a computer-readable program stored in the memory and executable on the processor. When the computer-readable program is executed by the processor, the language model-based error correction method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable program is executed by one or more processors, the language model-based error correction method according to any one of claims 1 to 5 is implemented.

Images