A method and system for correcting multiple identical wrong words in Chinese text

Abstract

The present invention discloses a method and system for correcting multiple identical wrong words in a Chinese text, which is applied to a Chinese text with multiple identical wrong words. The method comprises: obtaining a sentence based on the candidate set of replacement words; finding the identical wrong words based on the sentence, replacing the identical wrong words with preset marks, and generating a preset sentence; initially setting the score value and preset sequence value of a class ppl to be 0; judging whether the preset sequence value is not less than the preset sentence length, and if so, outputting the total score value of the current class ppl; otherwise, entering a loop judgment, accumulating the score value of the class ppl and adding 1 to the preset sequence value if the preset condition is met, and directly adding 1 to the preset sequence value if the condition is not met, repeating the loop judgment until the accumulated preset sequence value is not less than the preset sentence length, and outputting the total score value of the current class ppl. The identical wrong words are covered to avoid errors in the evaluation of the position error.

Description

Technical Field

[0001] The present application relates to the technical field of Chinese text error correction, and more specifically, to a method and system for correcting multiple identical incorrect words in a Chinese text. Background Art

[0002] Chinese text error correction is a very challenging research field. The complexity of Chinese grammar also makes it more difficult for computers to process text errors. With the development of deep learning, especially the emergence of large-scale corpus pre-training models, the network structure is becoming more and more complex, and the models with more and more powerful functions can be applied to the field of Chinese text error correction, which also brings about the development and improvement of the field of Chinese text error correction.

[0003] There are currently two commonly used methods for solving Chinese text error correction using deep learning models: one is an end-to-end model, which takes a Chinese sentence X (x1, x2, ... xn) as input and obtains another Chinese sentence Y (y1, y2, ... yn) as output, where X is the sentence to be judged and Y is the correct sentence after modification. The model construction process of this method is relatively simple and relies on the learning ability of the deep learning model; the other method is to use a two-stage method, which first identifies the location and type of the error, and then constructs a candidate set based on the error type, using the fluency of the sentence as an evaluation indicator to determine the optimal result. The two-stage method introduces prior knowledge, and the candidate set construction uses a user dictionary with similar pronunciation and form.

[0004] The calculation and construction of the two-stage method in the two methods are more complex, and the functional effect is also better. Deep learning models are used in the error recognition and error correction stages respectively. When evaluating the fluency of a sentence, the calculation indicator used is the sentence perplexity (PPL for short). There are many ways to calculate it. The calculation method involved in the present invention is to use the BERT model, and use the mask token method used in its pre-training task MLM (mask language mode) to calculate the score of each single word to accumulate a value similar to PPL to evaluate the fluency of the sentence (the lower the PPL value, the lower the perplexity, and the smoother the sentence). The advantage of this method is that it utilizes the learning ability of the large-scale corpus pre-training model, can draw inferences from one example, has strong generalization ability, and can learn semantic information that does not appear in the corpus.

[0005] However, some problems will be introduced. For example, the problem solved by the present invention is that when multiple identical wrong words appear in the same sentence, the score calculated by Bert for the wrong word is very low, resulting in an erroneous result. For example, in the sentence to be judged, "Life is square root, study is square root, work is square root", the candidate set of "square root" in the first position is modified to ["square root", "better"]. Since the Bert model calculates the same wrong word, the class PPL score of "square root" will be lower than that of "better", which leads to the conclusion that "square root" is better than "better".

[0006] When multiple identical wrong words appear in a Chinese text, when BERT is used for class PPL calculation, the score value of the current error is lower due to wrong words in other positions, so it is considered that its class PPL value is lower, and the fluency of the sentence is better, resulting in the same wrong words in the text, which cannot be corrected, thus causing omissions.

[0007] The input methods used in today's society generally have a memory function. The priority of previously input words is often higher than other words in a short period of time, so there is a greater possibility that the same wrong word will be made repeatedly. If text error correction cannot correct such errors better, it will easily cause "misunderstandings" when relevant social departments release information, causing their credibility to decline and the public to distrust them.

[0008] Therefore, how to avoid the class PPL score of the wrong word being lower than the correct word during error correction is a technical problem that needs to be solved. Summary of the invention

[0009] The present invention provides a method for correcting multiple identical wrong words in a Chinese text, so as to solve the technical problem in the prior art that the class PPL score of the wrong word is lower than that of the correct word during error correction. The method is applied to a Chinese text containing multiple identical wrong words, and the method comprises:

[0010] Perform sequence marking of errors on a text to be judged containing several identical wrong words to obtain multiple identical suspected wrong words, obtain a candidate set of replacement words based on the first suspected wrong word, and obtain a sentence based on the candidate set of replacement words;

[0011] Find the same wrong words based on the sentence, replace the same wrong words with preset marks, and generate a preset sentence;

[0012] The score value and preset order value of the ppl in the initial setting category are both 0;

[0013] Determine whether the preset sequence value is not less than the preset sentence length, and if so, output the total score value of the current class ppl;

[0014] If not, enter the loop judgment, if the preset condition is met, accumulate the score value of the class ppl and add 1 to the preset sequence value, if not, directly add 1 to the preset sequence value, repeat the loop judgment until the accumulated preset sequence value is not less than the preset sentence length, and output the total score value of the current class ppl;

[0015] Among them, the candidate set of alternative words includes correct words and incorrect words, the preset order value is 0 corresponding to the first Chinese character in the preset sentence, the preset order value gradually increases corresponding to the position in the preset sentence gradually moving backward, and the ppl-like score value represents the sentence confusion.

[0016] In some embodiments of the present application, the same wrong word is replaced with a preset mark to generate a preset sentence, specifically:

[0017] Based on the sentence, the same wrong words are found, and the rest of the same wrong words except the current position are replaced with preset marks to generate a preset sentence, wherein the preset sentence includes original text and preset marks.

[0018] In some embodiments of the present application, if the preset conditions are met, the score values ​​of the class ppl are accumulated and the preset sequence value is increased by 1, specifically:

[0019] Determine whether the current character is a Chinese character, the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value;

[0020] If so, the preset tag is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

[0021] In some embodiments of the present application, judging whether the preset sequence value is not less than the preset sentence length is specifically:

[0022] It is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

[0023] In some embodiments of the present application, the system is applied to a Chinese text containing multiple identical incorrect words, and includes:

[0024] A replacement module is used to perform sequence marking of errors on a text to be judged containing several identical wrong words, obtain multiple identical suspected wrong words, obtain a candidate set of replacement words based on the first suspected wrong word, obtain a sentence based on the candidate set of replacement words, find the identical wrong words based on the sentence, replace the identical wrong words with preset marks, and generate a preset sentence;

[0025] The initial module is used to initially set the score value and preset order value of the class ppl to 0;

[0026] A judgment module, used to judge whether the preset sequence value is not less than the preset sentence length, and if so, output the total score value of the current class ppl;

[0027] A loop module is used to enter a loop judgment if the preset condition is met, accumulate the score value of the class ppl and add 1 to the preset sequence value, and directly add 1 to the preset sequence value if the preset condition is not met, repeat the loop judgment until the accumulated preset sequence value is not less than the preset sentence length, and output the total score value of the current class ppl;

[0028] Among them, the candidate set of alternative words includes correct words and incorrect words, the preset order value is 0 corresponding to the first Chinese character in the preset sentence, the preset order value gradually increases corresponding to the position in the preset sentence gradually moving backward, and the ppl-like score value represents the sentence confusion.

[0029] In some embodiments of the present application, the replacement module is specifically used for:

[0030] Based on the sentence, the same wrong words are found, and the rest of the same wrong words except the current position are replaced with preset marks to generate a preset sentence, wherein the preset sentence includes original text and preset marks.

[0031] In some embodiments of the present application, the circulation module is specifically used for:

[0032] Determine whether the current character is a Chinese character, the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value;

[0033] If so, the preset tag is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

[0034] In some embodiments of the present application, the judgment module is specifically used to:

[0035] It is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

[0036] By applying the above technical solution, the text to be judged containing several identical wrong words is marked with sequence identification, and multiple identical suspected wrong words are obtained. Based on the first suspected wrong word, a candidate set of alternative words is obtained, and a sentence is obtained based on the candidate set of alternative words; the identical wrong words are found based on the sentence, and the identical wrong words are replaced with preset marks to generate a preset sentence; the score value and preset sequence value of the class ppl are initially set to 0; it is judged whether the preset sequence value is not less than the preset sentence length, and if so, the total score value of the current class ppl is output; if not, a loop judgment is entered, and if the preset condition is met, the score value of the class ppl is accumulated and the preset sequence value is added by 1, and if it is not met, the preset sequence value is directly added by 1, and the loop judgment is repeated until the accumulated preset sequence value is not less than the preset sentence length, and the total score value of the current class ppl is output. There are identical wrong words in the sentence, and these words are covered to avoid errors caused by the evaluation of this position error. It can accurately correct texts containing multiple identical wrong words, improve the accuracy of error correction, and avoid omissions and false alarms caused by the same words. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flowchart of a method for correcting multiple identical wrong words in a Chinese text proposed by an embodiment of the present invention is shown;

[0039] Figure 2 A structural diagram of a system for correcting multiple identical wrong words in a Chinese text proposed by an embodiment of the present invention is shown;

[0040] Figure 3 The flowchart of the error identification stage in the prior art is shown;

[0041] Figure 4 The flowchart of the error correction stage in the prior art is shown;

[0042] Figure 5 A flow chart of calculating class ppl in the prior art is shown;

[0043] Figure 6 A flow chart of a method according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] There are currently two stages in text error correction: identifying the location and type of the error, and correcting the error. In error identification and type, sequence annotation is used to mark the label of each word (BIO mode, B-type represents the word that starts with a certain error type, I-type represents the word that is not the beginning of a certain error type, and O represents the current word is correct). Then, for each error in each sequence annotation, according to its type, its related candidate set is constructed, and then the fluency of the sentence is judged for its candidate set, so as to obtain the best answer for replacement; after each error is processed in the same way, the final sentence is obtained. The process of error identification and error correction is as follows Figure 3 and Figure 4 shown.

[0046] When BERT is used to calculate the class PPL, the MaskLanguageMode task of BERT is used to mask the word at the marked position, and then obtain the class PPL score for that position. For example, in "Life is a square root, learning is a square root, and work is a square root.", when the sequence annotation obtains the label [0,0,0,B-yinxiangsi,I-yinxiangsi,0,0,0,0,B-yinxiangsi,I-yinxiangsi,0,0,0,0,B-yinxiangsi,I-yinxiangsi,0], that is, the label is three wrong "square roots" with similar suspected sounds. Then, for the "square root" in the first position, assuming that the candidate set we obtain is ["square root", "better"], that is, we judge which sentence has a lower class PPL score between "Life is a square root, learning is a square root, and work is a square root." and "Life is better, learning is a square root, and work is a square root.", because after the first "square root" is masked separately, the score obtained will be lower than "better", which will cause misjudgment, resulting in missed reports in error correction. The reason for this type of missed report is that when BERT calculates the class PPL of multiple identical words, the class PPL value of the same word is very low.

[0047] The prior art lacks a method for correcting multiple wrong words in a Chinese sentence, thereby identifying the wrong words as correct, resulting in underreporting of Chinese text. The flow chart of the prior art calculation class ppl is shown in Figure 5.

[0048] The present application embodiment provides a method for correcting multiple identical wrong words in a Chinese text, which is applied to a Chinese text containing multiple identical wrong words, such as Figure 1 As shown, the method comprises the following steps:

[0049] Step S101, finding the same wrong words based on the sentence, replacing the same wrong words with preset marks, and generating a preset sentence.

[0050] In this embodiment, according to a sentence, multiple identical wrong words in the sentence are found, and the identical wrong words are replaced with preset marks to obtain a preset sentence.

[0051] To ensure accuracy, in some embodiments of the present application, identical incorrect words are found based on the sentence, and preset marks are used to replace the remaining identical incorrect words except the current position to generate a preset sentence, which includes the original text and the preset mark.

[0052] In this embodiment, the same wrong words are found according to the sentence, and the wrong words include but are not limited to characters and words. As long as they are the same errors in Chinese characters, they all fall within the protection scope of this application. The same wrong words except the current position are replaced with preset marks, and the current position is the position in the sentence corresponding to the current sequence value. The generated preset sentence includes original text and preset marks, and the original text is the Chinese characters without errors in the sentence.

[0053] In order to further improve the accuracy, in some embodiments of the present application, before finding the same wrong words based on the sentence, the method further includes marking the sequence of errors for the text to be judged containing several same wrong words to obtain multiple

[0054] For the same suspected wrong word, a candidate set of alternative words is obtained based on the first suspected wrong word, and a sentence is obtained based on the candidate set of alternative words.

[0055] In this embodiment, a sequence is used to annotate the label of each word in the text to be judged, thereby obtaining multiple identical suspected wrong words, obtaining a candidate set of alternative words based on the first suspected wrong word, and obtaining a sentence based on the candidate set of alternative words, wherein the candidate set of alternative words includes correct words and wrong words.

[0056] Step S102, initially setting the score value and preset order value of class ppl to 0.

[0057] In this embodiment, the ppl-like score value and the preset sequence value are initially assigned 0, the preset sequence value of 0 corresponds to the first Chinese character in the preset sentence, and the preset sequence value gradually increases and the position in the preset sentence gradually moves backward. For example, when the preset sequence value is 2, it corresponds to the position of the third character in the preset sentence (the order of the characters in the preset sentence starts from left to right), wherein the ppl-like score value represents the sentence perplexity, the lower the ppl-like score value, the better the sentence fluency, and the higher the ppl-like score value, the opposite is true.

[0058] Step S103, determining whether the preset sequence value is not less than the preset sentence length, and if so, outputting the total score value of the current class ppl.

[0059] In this embodiment, it is determined whether the preset sequence value is not less than the preset sentence length. If so, the total score of the current class ppl is output. When the total score of the current class ppl is negative, the cumulative value of the class ppl score of each word calculated in the loop determination is performed.

[0060] In order to ensure reliability, in some embodiments of the present application, it is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

[0061] In this embodiment, it is determined whether the preset sequence value is greater than or equal to the number of Chinese characters in the preset sentence. For example, if the preset sequence value is 3 and there are 5 Chinese characters in the preset sentence, then the preset sequence value is less than the number of Chinese characters in the preset sentence.

[0062] Step S104: if the preset sequence value is less than the preset sentence length, a loop determination is performed. If the preset condition is met, the score value of the class ppl is accumulated and the preset sequence value is increased by 1; if the condition is not met, the preset sequence value is directly increased by 1, and the loop determination is repeated until the accumulated preset sequence value is not less than the preset sentence length, and the total score value of the current class ppl is output.

[0063] In this embodiment, when the preset sequence value is less than the preset sentence length, a loop determination is entered. If the preset condition is met, the score values ​​of the previous class ppl (including the current position) are accumulated and the preset sequence value is increased by 1. If it is not met, the preset sequence value is directly increased by 1, and the above loop determination is repeated until the accumulated preset sequence value is not less than the preset sentence length, and the total score value of the current class ppl is output.

[0064] In order to further improve reliability, in some embodiments of the present application, it is determined whether the current word is a Chinese character, and the current word is the Chinese character position in the preset sentence corresponding to the preset sequence value; if so, the preset mark is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

[0065] In this embodiment, the preset condition is to determine whether the current character is a Chinese character, and the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value. If the current character is a Chinese character, the current character is replaced with a preset tag, and the BERT model is entered to calculate the class PPL score of the current character. The class PPL scores are accumulated, and finally the preset sequence value is added by 1, and the above judgment is repeated until the preset sequence value is not less than the preset sentence length.

[0066] By applying the above technical solution, the text to be judged containing several identical wrong words is marked with sequence identification, and multiple identical suspected wrong words are obtained. Based on the first suspected wrong word, a candidate set of alternative words is obtained, and a sentence is obtained based on the candidate set of alternative words; the identical wrong words are found based on the sentence, and the identical wrong words are replaced with preset marks to generate a preset sentence; the score value and preset sequence value of the class ppl are initially set to 0; it is judged whether the preset sequence value is not less than the preset sentence length, and if so, the total score value of the current class ppl is output; if not, a loop judgment is entered, and if the preset condition is met, the score value of the class ppl is accumulated and the preset sequence value is added by 1, and if it is not met, the preset sequence value is directly added by 1, and the loop judgment is repeated until the accumulated preset sequence value is not less than the preset sentence length, and the total score value of the current class ppl is output. There are identical wrong words in the sentence, and these words are covered to avoid errors caused by the evaluation of this position error. It can accurately correct texts containing multiple identical wrong words, improve the accuracy of error correction, and avoid omissions and false alarms caused by the same words.

[0067] In order to further explain the technical idea of ​​the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.

[0068] The rest of the same wrong words except the current position are marked with a mask, so that the influence of the same wrong words is removed, making the evaluation more accurate and fair.

[0069] For the text to be judged "Life is a square root, learning is a square root, and work is a square root.", after the sequence of errors is marked, three wrong words "square root" with similar suspected pronunciations are obtained. For the first "square root", assuming that the candidate set ["square root", "better"] is obtained, two sentences are obtained ["Life is better, learning is a square root, and work is a square root.", "Life is a square root, learning is a square root, and work is a square root."], and then ppl-like calculations are performed.

[0070] Assume that the calculation statement is "to live better, the square root of study is obtained, and the square root of work is obtained." The suspected error is "square root".

[0071] like Figure 6 As shown, the calculation process of the class ppl score value is as follows:

[0072] 1. Find the same suspected errors and mask the same suspected error words (the same misspelled words), excluding the current misspelled word. Then the new sentence becomes "Live better, study [mask][mask], work [mask][mask]."

[0073] 2. Initialize the score of class ppl to 0 and the value of subscript i (preset marker) to 0.

[0074] 3. Determine whether the current subscript is greater than or equal to the length of the sentence; if so, output the score of class ppl.

[0075] If not, proceed to the next step (step 4).

[0076] 4. Determine whether the character at the current position is a Chinese character; if it is not a Chinese character, increment subscript i by 1 and then return to step 3; if it is a Chinese character, proceed to the next step (step 5).

[0077] 5. Replace the Chinese character at the current position with mask. Assuming i equals 0, the sentence becomes "[mask] Live better, study [mask][mask], work [mask][mask]." After mapping this sentence using the bert dictionary and passing it through the bert model, obtain the score for the first position (i = 0). Find the score for the character "生" at this position and add it to the sum of the scores of class ppl. Then proceed to the next step (step 6).

[0078] 6. Increment subscript i by 1 and then return to step 3.

[0079] In this calculation, the ppl value of "Live better, study sqrt, work sqrt." is lower than that of "Live sqrt, study sqrt, work sqrt.", thus determining that the first position "better" is more excellent than "sqrt", which solves the problem that the class ppl score of the correct word is higher than that of the wrong word. Continuing like this will result in the output of the modified original sentence being "Live better, study better, work better."

[0080] In this embodiment, several alternative solutions are also proposed. Since mask is used for marking, the mask label will also have a small impact on it; in addition, the process of the present invention mentions only targeting misspelled words, and there may be deviations if not all are marked in sequence annotation; also, when there are multiple different words in the candidate set in the original sentence, it will also have an impact; for the above situations, the following three alternative solutions are given:

[0081] 1) Regarding the influence of mask marking, since the mask used in the pre-training task is marked, the calculation of the mask score of the current position will also be affected accordingly. For such cases, it is recommended to use other unknown identifiers, and mark them with "unsed" in the dictionary. Then the original "live better, learn square root, work square root." will be modified to "live better, learn [unsed][unsed], work [unsed][unsed]." This can reduce the influence of the mask, but the pre-training model needs to be retrained, and then the same words need to be replaced with unsed.

[0082] 2) In order to prevent the impact of incomplete sequence annotation, the mask position range is not limited to the suspected errors, but the entire sentence, so this situation can be effectively avoided.

[0083] 3) For the following sentence: It is better to say to everyone: "Life is the square root, study is the square root, and work is the square root." The candidate set contains ["square root", "better"]. We can construct all possible candidate set sentences and get the following 16 sentences:

[0084] (1) It is better to say to everyone: "Life is the square root, study is the square root, and work is the square root."

[0085] (2) It is better to say to everyone: "Live with the square root, study with the square root, and work better."

[0086] (3) It is better to say to everyone: "Live with the square root, study better, and work with the square root."

[0087] (4) It is better to say to everyone: "Live better, study better, and work better."

[0088] (5) It is better to say to everyone: "Live better, study better, work better."

[0089] (6) It is better to say to everyone: "Live better, study better, and work better."

[0090] (7) It is better to say to everyone: "Live better, study better, and work better."

[0091] (8) Better to say to everyone: "Live better, study better, work better."

[0092] (9) The square root says to everyone: “The square root of life, the square root of study, and the square root of work.”

[0093] (10) Root says to everyone: "Live with the root of a square, study with the root of a square, and work better."

[0094] (11) Root says to everyone: "Live with the root of a square, study better, and work with the root of a square."

[0095] (12) Root says to everyone: "Live with the root of the root, study better, and work better."

[0096] (13) The square root says to everyone: "To live better, study with the square root, and work with the square root."

[0097] (14) Root says to everyone: "Live better, study better, work better."

[0098] (15) The square root says to everyone: "Live better, study better, work better."

[0099] (16) Root says to everyone: “Live better, study better, work better.”

[0100] In this case, no mask is used, and then the sentence with the lowest class ppl score is found as the final output.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present invention.

[0102] The present application also proposes a system for correcting multiple identical wrong words in a Chinese text, such as Figure 2 As shown, the system is applied to a Chinese text with multiple identical wrong words, and includes:

[0103] A replacement module 202 is used to find the same wrong words based on the sentence, replace the same wrong words with preset marks, and generate a preset sentence;

[0104] An initial module 203, used to initially set the score value and the preset order value of the class ppl to be 0;

[0105] A judging module 204, configured to judge whether the preset sequence value is not less than the preset sentence length, and if so, output the total score value of the current class ppl;

[0106] Loop module 205, for entering into loop determination if otherwise, accumulating the score value of class ppl and adding 1 to the preset sequence value if the preset condition is met, and directly adding 1 to the preset sequence value if the preset condition is not met, repeating the loop determination until the accumulated preset sequence value is not less than the preset sentence length, and outputting the total score value of the current class ppl;

[0107] Among them, the candidate set of alternative words includes correct words and incorrect words, the preset order value is 0 corresponding to the first Chinese character in the preset sentence, the preset order value gradually increases corresponding to the position in the preset sentence gradually moving backward, and the ppl-like score value represents the sentence confusion.

[0108] In the specific application scenario of the present application, the system further includes an error identification module 201, which is used to:

[0109] A text to be judged containing several identical wrong words is sequence-marked to obtain multiple identical suspected wrong words, a candidate set of replacement words is obtained based on the first suspected wrong word, and a sentence is obtained based on the candidate set of replacement words.

[0110] In the specific application scenario of the present application, the replacement module 202 is specifically used for:

[0111] Based on the sentence, the same wrong words are found, and the rest of the same wrong words except the current position are replaced with preset marks to generate a preset sentence, wherein the preset sentence includes original text and preset marks.

[0112] In the specific application scenario of the present application, the circulation module 205 is specifically used for:

[0113] Determine whether the current character is a Chinese character, the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value;

[0114] If so, the preset tag is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

[0115] In the specific application scenario of the present application, the judgment module 204 is specifically used to:

[0116] It is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

[0117] Those skilled in the art will appreciate that the modules in the system in the implementation scenario can be distributed in the system of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more systems different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for correcting multiple identical wrong words in a Chinese text, characterized in that: Applied to a Chinese text containing multiple identical incorrect words, the method comprises: Perform sequence marking of errors on a text to be judged containing several identical wrong words to obtain multiple identical suspected wrong words, obtain a candidate set of replacement words based on the first suspected wrong word, and obtain a sentence based on the candidate set of replacement words; Find the same wrong words based on the sentence, replace the same wrong words with preset marks, and generate a preset sentence; The score value and preset order value of the ppl in the initial setting category are both 0; Determine whether the preset sequence value is not less than the preset sentence length, and if so, output the total score value of the current class ppl; If not, enter the loop judgment, if the preset condition is met, accumulate the score value of the class ppl and add 1 to the preset sequence value, if not, directly add 1 to the preset sequence value, repeat the loop judgment until the accumulated preset sequence value is not less than the preset sentence length, and output the total score value of the current class ppl; Among them, the candidate set of alternative words includes correct words and incorrect words, the preset order value is 0 corresponding to the first Chinese character in the preset sentence, the preset order value gradually increases corresponding to the position in the preset sentence gradually moving backward, and the ppl-like score value represents the sentence confusion.

2. The method according to claim 1, characterized in that Find the same wrong words based on the sentence, replace the same wrong words with preset tags, and generate a preset sentence, specifically: Based on the sentence, the same wrong words are found, and the rest of the same wrong words except the current position are replaced with preset marks to generate a preset sentence, wherein the preset sentence includes original text and preset marks.

3. The method according to claim 1, characterized in that If the preset conditions are met, the scores of the class ppl are accumulated and the preset sequence value is increased by 1, specifically: Determine whether the current character is a Chinese character, the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value; If so, the preset tag is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

4. The method according to claim 1, characterized in that Determine whether the preset sequence value is not less than the preset sentence length, specifically: It is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

5. A system for correcting multiple identical wrong words in a Chinese text, characterized in that: Applied to Chinese texts containing multiple identical incorrect words, the system comprises: A replacement module is used to perform sequence marking of errors on a text to be judged containing several identical wrong words, obtain multiple identical suspected wrong words, obtain a candidate set of replacement words based on the first suspected wrong word, obtain a sentence based on the candidate set of replacement words, find the identical wrong words based on the sentence, replace the identical wrong words with preset marks, and generate a preset sentence; The initial module is used to initially set the score value and preset order value of the class ppl to 0; A judgment module, used to judge whether the preset sequence value is not less than the preset sentence length, and if so, output the total score value of the current class ppl; A loop module is used to enter a loop judgment if the preset condition is met, accumulate the score value of the class ppl and add 1 to the preset sequence value, and directly add 1 to the preset sequence value if the preset condition is not met, repeat the loop judgment until the accumulated preset sequence value is not less than the preset sentence length, and output the total score value of the current class ppl; Among them, the candidate set of alternative words includes correct words and incorrect words, the preset order value is 0 corresponding to the first Chinese character in the preset sentence, the preset order value gradually increases corresponding to the position in the preset sentence gradually moving backward, and the ppl-like score value represents the sentence confusion.

6. The system according to claim 5, characterized in that The replacement module is specifically used for: Based on the sentence, the same wrong words are found, and the rest of the same wrong words except the current position are replaced with preset marks to generate a preset sentence, wherein the preset sentence includes original text and preset marks.

7. The system according to claim 5, characterized in that The circulation module is specifically used for: Determine whether the current character is a Chinese character, the current character is the Chinese character position in the preset sentence corresponding to the preset sequence value; If so, the preset tag is used to replace the current word, enter the BERT model, calculate the class PPL score of the current word, accumulate the class PPL score and add 1 to the sequence value.

8. The system according to claim 5, characterized in that The judgment module is specifically used for: It is determined whether the preset sequence value is not less than the number of Chinese characters in the preset sentence.

Images