Text error correction method, device, electronic device and storage medium
By obtaining error text and using the fusion method of missing word recall list and MacBert model, the automation and efficiency of text error correction are achieved, and the problems of high cost and low efficiency of manual error correction in the prior art are solved.
Patent Information
- Application Number
- CN202111602590.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-12-24
AI Technical Summary
Text error correction of missing words and/or words in the prior art requires manual inspection and correction, resulting in high cost, low accuracy and efficiency.
By obtaining the error text, using the preset word-missing recall list and MacBert model, the candidate correction text is determined, and finally the target correction text is obtained by integrating the two correction methods to achieve automated error correction.
Without manual proofreading, the automation and efficiency of text error correction are achieved, and the accuracy of error correction is improved.
Smart Images

Figure CN114528824B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular, relates to a text error correction method, device, electronic device and storage medium. Background Art
[0002] At present, there are often missing words and errors in the text content of social tools, news releases or other media. Usually, these missing words and errors in the text need to be discovered through manual inspection and verification, and then corrected manually. However, this method has high labor costs and low accuracy and efficiency. Summary of the invention
[0003] In view of this, embodiments of the present application provide a text correction method, device, electronic device and storage medium to solve the problem in the prior art of how to automatically and accurately correct texts that are missing characters and / or words.
[0004] A first aspect of an embodiment of the present application provides a text error correction method, comprising:
[0005] Get the error text;
[0006] Determining a first target number of first candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset missing-word recall list; wherein the missing-word recall list contains a preset number of characters that can be constructed into words with characters in the erroneous text;
[0007] Determining a second target number of second candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model;
[0008] According to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts, a target corrected text corresponding to the erroneous text is determined.
[0009] A second aspect of an embodiment of the present application provides a text error correction device, comprising:
[0010] Get unit, used to get error text;
[0011] A first correction unit is used to determine a first target number of first candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset missing character recall list; wherein the missing character recall list contains a preset number of characters that can be constructed into words with characters in the erroneous text;
[0012] A second correction unit, configured to determine a second target number of second candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model;
[0013] The target corrected text determining unit is used to determine a second target number of second candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model.
[0014] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the steps of the text error correction method.
[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the electronic device implements the steps of the text error correction method.
[0016] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the text error correction method as described in the first aspect.
[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: in the embodiments of the present application, after obtaining the error text, the first target number of first candidate correction texts corresponding to the error text are determined according to the error text and the preset missing word recall list, and the second target number of second candidate correction texts corresponding to the error text are determined according to the error text and the preset MacBert model. Afterwards, the target correction text corresponding to the error text is determined according to the first candidate correction text and the second candidate correction text. Through this method, the error correction of the error text can be automatically and efficiently realized without relying on manual proofreading; and, since the target correction text is obtained based on the first candidate correction text and the second candidate correction text, and the first candidate correction text is determined based on the preset missing word recall list, and the second candidate correction text is obtained based on the preset MacBert model, the target correction text is a correction text obtained by integrating two different correction methods, the missing word recall list and the MacBert model, so that the accuracy of text correction can be improved by combining the two correction methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1It is a schematic diagram of an implementation flow of a text error correction method provided in an embodiment of the present application;
[0020] Figure 2 is an example diagram of a missing word recall list provided in an embodiment of the present application;
[0021] Figure 3 is a schematic diagram of a text error correction device provided in an embodiment of the present application;
[0022] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0024] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.
[0025] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0026] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0028] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0029] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0030] At present, there are often various text errors in the text content of social tools, news releases or other carriers, including homophonic word errors, confusing word errors, word order reversal errors, missing characters and / or word errors, similar characters errors, grammatical errors, etc. Among them, missing characters and / or word errors usually need to be discovered by manual inspection and then corrected manually. For example, through manual inspection, it is found that the sentence "I eat a hamburger at McDonald's" is an erroneous sentence with missing characters or words. Through inspection and correction, the corresponding corrected text can be obtained: "I eat a hamburger at McDonald's." However, the manual inspection and correction method has high labor costs, and low accuracy and efficiency.
[0031] In order to solve the above-mentioned technical problems, embodiments of the present application provide a text error correction method, device, electronic device and storage medium, including: obtaining an error text; determining a first target number of first candidate correction texts corresponding to the error text according to the error text and a preset missing character recall list; wherein the missing character recall list contains a preset number of characters that can be constructed into words with characters in the error text; determining a second target number of second candidate correction texts corresponding to the error text according to the error text and a preset MacBert model; and determining a target correction text corresponding to the error text according to the first target number of first candidate correction texts and the second target number of second candidate correction texts.
[0032] Through this method, erroneous text can be corrected automatically and efficiently without relying on manual proofreading; and since the target corrected text is obtained based on the first candidate corrected text and the second candidate corrected text, and the first candidate corrected text is determined based on a preset missing character recall list, and the second candidate corrected text is obtained based on a preset MacBert model, the target corrected text is a corrected text obtained by integrating two different correction methods, the missing character recall list and the MacBert model, thereby being able to improve the accuracy of text correction by combining the two correction methods.
[0033] Embodiment 1:
[0034] Figure 1 A flowchart of a text error correction method provided by an embodiment of the present application is shown, and the text error correction method is applied to an electronic device, as described in detail as follows:
[0035] In S101, an error text is acquired.
[0036] In an embodiment of the present application, the error text may be a Chinese sentence with missing characters and / or missing words. In one embodiment, the text content in social software messages, news manuscripts, and office software manuscripts can be detected by a preset text detection model to find sentences with missing characters and / or missing words as error text.
[0037] Exemplarily, the text detection model may include a word segmentation module and a term detection module; for each sentence in the aforementioned text content, the sentence is input into the word segmentation module for processing to obtain the word segmentation text corresponding to the sentence; the word segmentation text is detected by the term detection module, and if it is found that a certain character in the word segmentation text cannot form a word with other adjacent characters and the character is not a character that can express meaning alone (that is, the word formed by the character and its adjacent characters does not exist in the preset word dictionary, and the character does not exist in the preset single word dictionary), the sentence is determined to be an erroneous text.
[0038] In S102, a first target number of first candidate corrected texts corresponding to the erroneous text is determined based on the erroneous text and a preset missing-character recall list; wherein the missing-character recall list contains a preset number of characters that can be constructed into words with characters in the erroneous text.
[0039] In an embodiment of the present application, the preset missing character recall list includes a linked list corresponding to each Chinese character in a preset Chinese character data set, and the linked list corresponding to each Chinese character includes a preset number of Chinese characters that can be constructed into a word with the Chinese character.
[0040] After obtaining the erroneous text, for each character in the erroneous text (for the sake of distinction, it is called the original character), search the missing character recall list for a character that can be constructed into a word with the original character (for the sake of distinction, it is called the completion character), and insert the completion character into a position adjacent to the original character in the erroneous text, to obtain the first target number of candidate corrected texts as the first candidate corrected text.
[0041] In S103, a second target number of second candidate corrected texts corresponding to the erroneous text is determined according to the erroneous text and a preset MacBert model.
[0042] In the embodiment of the present application, the preset MacBert model is a Chinese natural language pre-training model that has been trained in advance. Specifically, MacBert is based on the Bert model and replaces the mask in the Bert model with similar words to reduce the gap between the pre-training and fine-tuning stages. Among them, the Bert model is a conversion model expressed by a bidirectional encoder (BidirectionalEncoder Representations from Transformer), which adopts a new masked language model (masked language model, MLM) to generate deep bidirectional language representations.
[0043] After obtaining the erroneous text, the erroneous text can be input into the pre-trained MacBert model for processing, and a second target number of second candidate corrected texts can be obtained by inserting pre-set similarity words into the interval positions of each character and / or word in the erroneous text.
[0044] In the embodiment of the present application, the execution order of the above-mentioned step S102 and step S103 can be arbitrarily reversed, or executed simultaneously. For example, by enabling multithreading, step S102 and step S103 can be executed in parallel by different threads, so that while determining the first target number of first candidate corrected texts based on the missing character recall list, the second target number of second candidate corrected texts can be determined based on the MacBert model.
[0045] In S104, a target corrected text corresponding to the erroneous text is determined according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts.
[0046] After the first candidate corrected text and the second candidate corrected text are determined, a target corrected text corresponding to the erroneous text is determined through a preset fusion algorithm according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts.
[0047] In one embodiment, the sentence scores of each first candidate corrected text and each second candidate corrected text can be obtained respectively by a preset sentence scoring algorithm, and the candidate corrected text (the first candidate corrected text or the second candidate corrected text) with the highest score is determined as the target corrected text. Alternatively, the first preset number of candidate corrected texts with the highest scores are selected from each first candidate corrected text and each second candidate corrected text, and then the correction information (i.e., information different from the original erroneous text) existing in the preset number of candidate corrected texts is added to the position corresponding to the erroneous text to obtain the target corrected text. Exemplarily, the aforementioned sentence scoring algorithm can be a deep learning model trained based on a certain number of positive sample sentences (i.e., sentences with completely correct texts) and negative sample sentences (i.e., sentences with text errors) carrying score labels.
[0048] Through the method of the embodiment of the present application, it is possible to automatically and efficiently correct erroneous texts without relying on manual proofreading; and since the target corrected text is obtained based on the first candidate corrected text and the second candidate corrected text, and the first candidate corrected text is determined based on a preset missing character recall list, and the second candidate corrected text is obtained based on a preset MacBert model, the target corrected text is a corrected text obtained by integrating two different correction methods, the missing character recall list and the MacBert model, thereby being able to improve the accuracy of text correction by combining the two correction methods.
[0049] Optionally, the missing word recall list includes a first linked list and a second linked list, wherein the characters in the first linked list are used to construct words starting with the characters in the erroneous text; and the characters in the second linked list are used to construct words ending with the characters in the erroneous text.
[0050] In the embodiment of the present application, for each character in the above-mentioned Chinese character data set, there is a corresponding first linked list and a second linked list, and the missing character recall list in the embodiment of the present application specifically includes the first linked list and the second linked list corresponding to each character. Exemplarily, the characters in the error text are called original characters, and the characters stored in the missing character recall list that can form words with the original characters are called complementary characters. For each original character, the complementary characters stored in the corresponding first linked list are used to construct a word starting with the original character, and the complementary characters stored in the corresponding second linked list are used to construct a word ending with the original character. Specifically, for the preset number of complementary characters corresponding to each original character, half are complementary characters in the first linked list, and the other half are complementary characters in the second linked list.
[0051] Exemplarily, for the original character "生" in the error text, among the 200 completed characters included in its corresponding missing character recall list, it may include 100 words starting with the character "生" stored in the first linked list, and 100 words ending with the character "生" stored in the second linked list. Exemplarily, a partial schematic diagram of the missing character recall list corresponding to the character "生" is as shown in Figure 2 the following figure.
[0052] In the embodiments of the present application, since the missing character recall list specifically includes a first linked list for constructing words starting with the character in the error text, and a second linked list for constructing words ending with the character in the error text, the words constructed based on this missing character recall list and the generated corrected text are more comprehensive and complete, thereby improving the accuracy of text error correction.
[0053] Optionally, the determining of the first target number of first candidate corrected texts corresponding to the error text according to the error text and the preset missing character recall list includes:
[0054] Determining the corresponding preset number of preliminary corrected texts for each character in the error text according to the missing character recall list to obtain a third target number of the preliminary corrected texts;
[0055] Based on the n-gram model, respectively obtaining the first target perplexity corresponding to each of the preliminary corrected texts, and using the preliminary corrected texts with the first target perplexity less than the original perplexity corresponding to the error text as the first candidate corrected texts to obtain the first target number of the first candidate corrected texts.
[0056] In the embodiments of the present application, the perplexity is an index used to describe the quality of a sentence and can be determined according to the probability distribution value of a sentence. For example, for a sentence S with K word segmentations of W 1 ~W k (i.e., S = W 1 , W 2 ,..., W k ), its probability calculation formula is:
[0057] P(S) = P(W 1 , W 2 ,..., W k ) = p(W 1 )P(W 2 |W 1 )...P(W k |W 1 , W 2 ,..., W k-1 )
[0058] The calculation formula of the confusion PP(S) corresponding to the sentence is:
[0059]
[0060] From the above calculation formula, we can know that the better the sentence quality, that is, the greater the sentence probability P(S), the smaller the corresponding perplexity PP(S), that is, the less confused the language model is about the sentence.
[0061] The calculation of the above-mentioned perplexity depends on the word segmentation model of the text, that is, the sentence needs to be segmented before the probability of the sentence can be calculated, and then the perplexity of the sentence can be calculated. In the embodiment of the present application, the n-gram model (also called the n-gram model) is specifically used as the word segmentation model, and the calculation of sentence probability and perplexity is implemented based on the n-gram model. The n-gram model is an algorithm based on a statistical language model, which performs a sliding window operation of size n on the content of the text according to bytes, forming a sequence of byte segments of length n, and each byte segment is called a gram. Wherein n is an arbitrary positive integer. Exemplarily, when n=5, that is, when the n-gram model is specifically a 5-gram model, it can effectively balance the accuracy and efficiency of the model, and efficiently and accurately determine the perplexity of the sentence.
[0062] In one embodiment, the n-gram model can be trained before obtaining the erroneous text. Specifically, considering that different corpora provide a variety of expression styles such as rich semantics, sentence patterns and conjunctions, the original corpus data set can be obtained by crawling public news data, public account text data and other public text data sets through web crawler tools. Then, the original corpus data set is cleaned and spliced to obtain a Chinese text data set (the data volume of the Chinese text data set can reach 14G). Afterwards, the n-gram model is trained according to the Chinese text data set (the n-gram model can be trained based on the kenlm tool) to obtain the trained n-gram model, so that the sentence confusion calculation can be accurately implemented according to the trained n-gram model.
[0063] In an embodiment of the present application, after obtaining the error text, according to the above-mentioned missing word recall list, a corresponding preset number of correction texts with corresponding complementary words added are determined for each word in the error text, and a preset number of preliminary correction texts corresponding to each word are obtained. Each corresponding preset number of preliminary correction texts are summarized to finally obtain a third target number of preliminary correction texts. Among them, assuming that the above-mentioned preset number is Q, and the sentence length of the error text is M, then the third target number is equal to Q*M. For example, for each word in the error text, there are corresponding Q=200 complementary words in the preset missing word recall list, and the sentence length is 5, then the number of preliminary correction texts currently determined is 200*5=1000.
[0064] In an embodiment of the present application, after obtaining the erroneous text, the perplexity of the erroneous text can be obtained based on the above-mentioned n-gram model, and the perplexity of the erroneous text obtained is called the original perplexity. After determining the third preset number of preliminary corrected texts, for each preliminary corrected text, the perplexity of the preliminary corrected text is obtained based on the above-mentioned n-gram model, and the perplexity corresponding to the preliminary corrected text is called the first target perplexity; then the first target perplexity is compared with the original perplexity corresponding to the erroneous text, and the preliminary corrected text whose first target perplexity is less than the original perplexity is retained as the first candidate corrected text, and finally the first target number of first candidate corrected texts is obtained. The first target number is less than or equal to the above-mentioned third preset number.
[0065] In an embodiment of the present application, after a third preset number of preliminary corrected texts are preliminarily determined based on the missing-word recall list, the perplexity is further calculated based on n-gram to retain the preliminary corrected texts whose corresponding first target perplexity is less than the original perplexity of the erroneous text as the final first candidate corrected text. Therefore, a more accurate first candidate corrected text can be screened out based on the perplexity of the sentence, thereby improving the accuracy of text correction.
[0066] Optionally, determining a second target number of second candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model includes:
[0067] Adding a mask at each position of the erroneous text in sequence by using the MacBert model to obtain a fourth target number of masked texts;
[0068] Based on the n-gram model, the second target perplexity corresponding to each mask text is calculated respectively, and the mask text whose second target perplexity is less than the original perplexity corresponding to the erroneous text is taken as the second candidate corrected text, to obtain a second target number of the second candidate corrected texts.
[0069] In the embodiment of the present application, after obtaining the error text, the above-mentioned MacBert model is used to sequentially add a mask to each position in the error text (including the interval between every two characters and / or words, the position before the first character, and the position after the last character). Specifically, the error text is input into the masked language model in the MacBert model for prediction, and the mask text with the highest confidence corresponding to each position is determined to obtain the fourth target number of mask texts. Wherein, assuming that M is the sentence length of the error text, the fourth target number is equal to M+1.
[0070] After obtaining the fourth target number of masked texts, for each masked text, the perplexity of the masked text is calculated based on the above n-gram model, and the perplexity corresponding to the masked text is called the second target perplexity. After that, the mask whose corresponding second target perplexity is less than the original perplexity corresponding to the wrong text is determined as the second candidate corrected text, and finally the second target number of second candidate corrected texts is obtained, where the second target number is less than or equal to M+1.
[0071] In an embodiment of the present application, after first determining the fourth target number of mask texts based on the MacBert model, the second target perplexity corresponding to each of the mask texts is obtained based on the n-gram model, and the mask text whose second target perplexity is less than the original perplexity corresponding to the erroneous text is used as the second candidate corrected text. Therefore, a more accurate second candidate corrected text can be screened out based on the perplexity of the sentence, thereby improving the accuracy of text correction.
[0072] Optionally, determining the target corrected text corresponding to the erroneous text according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts includes:
[0073] Determine, according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts, a candidate corrected text with a lower perplexity corresponding to each position in the erroneous text, to obtain a third candidate corrected text;
[0074] According to the third candidate corrected text, a target corrected text corresponding to the erroneous text is determined.
[0075] In an embodiment of the present application, after adding a complement or mask to each position of the erroneous text respectively by two different methods based on the missing word recall list and based on the MacBert model, for the correction of the same position of the erroneous text, there may be two different candidate correction texts, namely the first candidate correction text and the second candidate correction text. At this time, for the same position, the perplexity of the first candidate correction text and the second candidate correction text can be obtained respectively based on the above-mentioned n-gram model, and the candidate correction text with lower perplexity is determined as the third candidate correction text at the position of the erroneous text. Assuming that the sentence length of the erroneous text is M, the number of third candidate correction texts determined by this method is M+1.
[0076] After determining the third candidate corrected text, further select the best one (for example, the one with the lowest confusion) from the third candidate corrected text as the target corrected text, or merge at least two correction information of the third candidate corrected text (that is, the additional completion word information at certain positions relative to the erroneous text) to obtain the final target corrected text.
[0077] In an embodiment of the present application, after determining the first candidate correction text and the second candidate correction text corresponding to the erroneous text by two different methods, the different candidate correction texts corresponding to the same position of the erroneous text can be selected based on the perplexity to obtain the third candidate correction text, so that the quality of the target correction text obtained based on the third candidate correction text is better, thereby improving the accuracy of text correction.
[0078] Optionally, determining a target corrected text corresponding to the erroneous text according to the third candidate corrected text includes:
[0079] The third candidate corrected text with the lowest confusion among the third candidate corrected texts is used as the benchmark corrected text, and the third candidate corrected text other than the benchmark corrected text is used as the revised text. The benchmark corrected text is corrected based on the correction information of the revised text to obtain the target corrected text corresponding to the erroneous text.
[0080] In an embodiment of the present application, after determining M+1 third candidate correction texts, a third candidate correction text with the lowest perplexity is selected from the third candidate correction texts as a benchmark correction text. Afterwards, the benchmark correction text is used as a benchmark, and the third candidate correction text other than the benchmark correction text is used as a correction text. The benchmark correction text is corrected according to the correction information of the correction text to obtain the target correction text corresponding to the erroneous text. Among them, the correction information of the correction text refers to the additional complementary characters in the correction text relative to the erroneous text and the position information corresponding to the complementary characters. In one embodiment, the correction process of the correction text to the benchmark correction text is as follows:
[0081] A1: Calculate the perplexity of the current benchmark corrected text to obtain the current benchmark perplexity;
[0082] A2: Based on the current reference corrected text, a complementary word corresponding to the corrected text and position information corresponding to the complementary word are obtained from each corrected text in a preset order, and the complementary word is added to the corresponding position of the reference corrected text according to the position information of the complementary word to obtain a pending corrected text;
[0083] A3: Calculate the perplexity of the pending correction text to obtain the pending perplexity;
[0084] A4: If the pending perplexity is less than the reference perplexity, the pending corrected text is used as the new current reference corrected text and the process returns to step A1; otherwise, the process returns directly to step A2 until all corrected texts have been obtained and step A2 has been executed.
[0085] In the embodiment of the present application, since the third candidate correction text with the lowest perplexity can be determined from the third candidate correction texts as the benchmark correction text, and based on the benchmark correction text, the benchmark correction text is corrected by correction text other than the benchmark correction text, it is possible to fuse multiple third candidate correction texts while ensuring the lowest perplexity, accurately obtain the target correction text, and improve the accuracy of text error correction.
[0086] Optionally, before obtaining the error text, the method further includes:
[0087] Get the Chinese text dataset;
[0088] Selecting a first preset ratio of characters and / or words from the Chinese text data set to perform a masking operation to obtain a target data set;
[0089] The MacBert model to be trained is trained according to the target data set to obtain the trained MacBert model.
[0090] In the embodiment of the present application, before obtaining the error text, the original MacBert model to be trained is first trained. The training data source of the MacBert model can be consistent with the data source for training the n-gram model mentioned above. That is, the Chinese text dataset can be obtained by crawling news data, public account text data and other public text datasets, and performing data cleaning.
[0091] After obtaining the Chinese text data set, a first preset ratio (e.g., 15%) of characters and / or words can be selected as data to be masked. For the first preset ratio of data to be masked, 80% of the characters and / or words can be obtained and replaced with a masking mark [MASK], and 20% of the characters and / or words remain unchanged to obtain a target data set.
[0092] The target data set is then input into the MacBert model to be trained for training. During the training process, the MacBert model predicts masked words based on the bidirectional context information of the target data set, and fine-tunes the model to finally obtain a trained MacBert model.
[0093] In the embodiment of the present application, before obtaining the erroneous text, a Chinese text data set can be obtained and after obtaining the target data set through processing, the MacBert model is trained, so that after obtaining the erroneous text, the second candidate correction text corresponding to the erroneous text can be efficiently and accurately determined based on the trained MacBert model, thereby improving the accuracy of text error correction.
[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0095] Embodiment 2:
[0096] Figure 3 A schematic diagram of the structure of a text error correction device provided in an embodiment of the present application is shown. For the sake of convenience, only the part related to the embodiment of the present application is shown:
[0097] The text error correction device comprises: an acquisition unit 31, a first correction unit 32, a second correction unit 33, and a target correction text determination unit 34. Among them:
[0098] The acquiring unit 31 is used to acquire the error text.
[0099] The first correction unit 32 is used to determine a first target number of first candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset missing-character recall list; wherein the missing-character recall list contains a preset number of characters that can be constructed into words with the characters in the erroneous text.
[0100] The second correction unit 33 is used to determine a second target number of second candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model.
[0101] The target corrected text determining unit 34 is configured to determine a second target number of second candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model.
[0102] Optionally, the missing word recall list includes a first linked list and a second linked list, wherein the characters in the first linked list are used to construct words starting with the characters in the erroneous text; and the characters in the second linked list are used to construct words ending with the characters in the erroneous text.
[0103] Optionally, the first correction unit 32 is specifically used to determine the preset number of preliminary corrected texts corresponding to each word in the erroneous text according to the missing-word recall list, so as to obtain the third target number of the preliminary corrected texts; based on the n-gram model, respectively obtain the first target perplexity corresponding to each of the preliminary corrected texts, and take the preliminary corrected text whose first target perplexity is less than the original perplexity corresponding to the erroneous text as the first candidate corrected text, so as to obtain the first target number of the first candidate corrected texts.
[0104] Optionally, the second correction unit 33 is specifically used to add a mask to each position of the erroneous text in turn through the MacBert model to obtain a fourth target number of masked texts; based on the n-gram model, respectively calculate the second target perplexity corresponding to each masked text, and use the masked text whose second target perplexity is less than the original perplexity corresponding to the erroneous text as the second candidate corrected text, to obtain a second target number of second candidate corrected texts.
[0105] Optionally, the target corrected text determination unit 34 includes:
[0106] A third candidate corrected text determination module is used to determine a candidate corrected text with a lower perplexity corresponding to each position in the erroneous text according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts, to obtain a third candidate corrected text;
[0107] The target corrected text determination module is used to determine the target corrected text corresponding to the erroneous text according to the third candidate corrected text.
[0108] Optionally, the target corrected text determination unit is specifically used to take the third candidate corrected text with the lowest perplexity among the third candidate corrected texts as the benchmark corrected text, and take the third candidate corrected text other than the benchmark corrected text as the corrected text, and correct the benchmark corrected text based on the correction information of the corrected text to obtain the target corrected text corresponding to the erroneous text.
[0109] Optionally, the text error correction device further includes:
[0110] The training unit is used to obtain a Chinese text data set; select a first preset ratio of characters and / or words from the Chinese text data set for masking operation to obtain a target data set; train a MacBert model to be trained according to the target data set to obtain the trained MacBert model.
[0111] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0112] Embodiment three:
[0113] Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40, such as a text error correction program. When the processor 40 executes the computer program 42, the steps in the above-mentioned various text error correction method embodiments are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 3 The functions of the units 31 to 34 are shown.
[0114] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 42 in the electronic device 4.
[0115] The electronic device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0116] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0117] The memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 may also be used to temporarily store data that has been output or is to be output.
[0118] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0119] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0121] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0124] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0125] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A text error correction method, characterized in that: include: Get the error text; Determining a first target number of first candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset missing-character recall list; wherein the missing word recall list contains a preset number of words that can be constructed into words with the words in the erroneous text; The preset missing character recall list includes a linked list corresponding to each Chinese character in the preset Chinese character data set, and the linked list corresponding to each Chinese character includes a preset number of Chinese characters that can be constructed into a word with the Chinese character; According to the erroneous text and a preset MacBert model, determining a second target number of second candidate corrected texts corresponding to the erroneous text; wherein the MacBert model replaces the mask in the Bert model with similar words on the basis of the Bert model to reduce the gap between the pre-training and fine-tuning stages; Determine the target corrected text corresponding to the erroneous text according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts, including: determine the candidate corrected text with lower perplexity corresponding to each position in the erroneous text according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts to obtain a third candidate corrected text; determine the target corrected text corresponding to the erroneous text according to the third candidate corrected text.
2. The text error correction method according to claim 1, characterized in that: The missing word recall list includes a first linked list and a second linked list, wherein the characters in the first linked list are used to construct words starting with the characters in the erroneous text; and the characters in the second linked list are used to construct words ending with the characters in the erroneous text.
3. The text error correction method according to claim 1, characterized in that: The step of determining a first target number of first candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset missing character recall list includes: According to the missing-character recall list, determining the preset number of preliminary corrected texts corresponding to each character in the erroneous text, to obtain a third target number of preliminary corrected texts; Based on the n-gram model, the first target perplexity corresponding to each of the preliminary corrected texts is calculated respectively, and the preliminary corrected text whose first target perplexity is less than the original perplexity corresponding to the erroneous text is taken as the first candidate corrected text, to obtain the first target number of the first candidate corrected texts.
4. The text error correction method according to claim 1, characterized in that: The step of determining a second target number of second candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model includes: Adding a mask at each position of the erroneous text in sequence by using the MacBert model to obtain a fourth target number of masked texts; Based on the n-gram model, the second target perplexity corresponding to each mask text is calculated respectively, and the mask text whose second target perplexity is less than the original perplexity corresponding to the erroneous text is taken as the second candidate corrected text, to obtain a second target number of the second candidate corrected texts.
5. The text error correction method according to claim 1, characterized in that: The step of determining a target corrected text corresponding to the erroneous text according to the third candidate corrected text includes: The third candidate corrected text with the lowest confusion among the third candidate corrected texts is used as the benchmark corrected text, and the third candidate corrected text other than the benchmark corrected text is used as the revised text. The benchmark corrected text is corrected based on the correction information of the revised text to obtain the target corrected text corresponding to the erroneous text.
6. The text error correction method according to claim 1, characterized in that: Before getting the error text, also include: Get the Chinese text dataset; Selecting a first preset ratio of characters and / or words from the Chinese text data set to perform a masking operation to obtain a target data set; The MacBert model to be trained is trained according to the target data set to obtain the trained MacBert model.
7. A text error correction device, characterized in that: include: Get unit, used to get error text; A first correction unit is used to determine a first target number of first candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset missing character recall list; wherein the missing character recall list contains a preset number of characters that can be constructed into words with characters in the erroneous text; The preset missing character recall list includes a linked list corresponding to each Chinese character in the preset Chinese character data set, and the linked list corresponding to each Chinese character includes a preset number of Chinese characters that can be constructed into a word with the Chinese character; A second correction unit is used to determine a second target number of second candidate correction texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model; wherein the MacBert model replaces the mask in the Bert model with similar words on the basis of the Bert model to reduce the gap between the pre-training and fine-tuning stages; a target corrected text determining unit, configured to determine a second target number of second candidate corrected texts corresponding to the erroneous text according to the erroneous text and a preset MacBert model; The target corrected text determination unit comprises: A third candidate corrected text determination module is used to determine a candidate corrected text with a lower perplexity corresponding to each position in the erroneous text according to the first target number of first candidate corrected texts and the second target number of second candidate corrected texts, to obtain a third candidate corrected text; The target corrected text determination module is used to determine the target corrected text corresponding to the erroneous text according to the third candidate corrected text.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the electronic device implements the steps of the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Chinese text error correction method and system
CN112149406A
Dialogue model training method and device, dialogue generation method and device, equipment and medium
CN112417127A
