Method, apparatus and electronic device for obtaining text error correction training corpus

Abstract

The present disclosure discloses a method for obtaining text error correction training corpus, which relates to the field of data processing technology, especially to the fields of big data, natural language processing, artificial intelligence, etc. The specific implementation scheme is: obtaining a reference error correction model and initial training corpus; inputting the text to be corrected into the reference error correction model, obtaining multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text; determining the characters to be rewritten and the rewriting pattern in the text to be corrected based on the predicted text, the first probability and the annotated text; rewriting the characters to be rewritten based on the rewriting pattern, and obtaining the text error correction training corpus corresponding to the target field. Thus, by using the reference error correction model to predict the field text, determining the weak points of the reference error correction model in the prediction of the field, and generating training corpus for the weak points, the quality of the generated training corpus is improved, and conditions are provided for obtaining the reliability and accuracy of the text error correction model for a specific field.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the fields of big data, natural language processing, artificial intelligence, and more specifically to a method, device, and electronic device for acquiring text error correction training corpus. Background Art

[0002] In the process of training a text error correction model, the quality of the training corpus directly affects the reliability of the text error correction model. Therefore, there is an urgent need for a reliable method to generate text error correction training corpus to improve the reliability and accuracy of the text error correction model. Summary of the Invention

[0003] The present disclosure provides a method and device for acquiring text error correction training corpus.

[0004] According to one aspect of the present disclosure, a method for acquiring text error correction training corpus is provided, comprising:

[0005] Obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the target domain text to be corrected and the corresponding annotated text;

[0006] Inputting the text to be corrected into the reference error correction model to obtain a plurality of predicted texts and a first probability corresponding to the predicted character at each position in each predicted text;

[0007] Determining the characters to be rewritten and the rewriting mode in the text to be corrected based on the plurality of predicted texts, the first probability corresponding to the predicted characters at each position, and the annotated text;

[0008] Based on the rewriting pattern, the characters to be rewritten are rewritten to obtain the updated text error correction training corpus corresponding to the target domain.

[0009] According to another aspect of the present disclosure, a device for acquiring text error correction training corpus is provided, comprising:

[0010] An acquisition module is used to obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the target domain text to be corrected and the corresponding annotated text;

[0011] A prediction module, configured to input a text to be corrected into a reference correction model to obtain a plurality of predicted texts and a first probability corresponding to a predicted character at each position in each predicted text;

[0012] a determination module, configured to determine a character to be rewritten and a rewriting mode in the text to be corrected based on the plurality of predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text;

[0013] The rewriting module is used to rewrite the characters to be rewritten based on the rewriting mode to obtain the updated text error correction training corpus corresponding to the target field.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiment.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the steps of the method described in the above embodiment when executed by a processor.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 A flowchart of a method for obtaining text error correction training corpus provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of another method for obtaining text error correction training corpus provided in an embodiment of the present disclosure;

[0024] Figure 3 A flowchart of another method for obtaining text error correction training corpus provided in an embodiment of the present disclosure;

[0025] Figure 4 A flowchart of another method for obtaining text error correction training corpus provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a flow chart of another apparatus for acquiring text error correction training corpus provided by an embodiment of the present disclosure;

[0027] Figure 6 It is a block diagram of an electronic device used to implement the method for acquiring text error correction training corpus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Big data, or massive data, refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps businesses make more proactive business decisions within a reasonable time using current mainstream software tools.

[0030] NLP (Natural Language Processing) is an important field in computer science and artificial intelligence. The content of NLP research includes but is not limited to the following branches: text classification, information extraction, automatic summarization, intelligent question answering, topic recommendation, machine translation, keyword recognition, knowledge base construction, deep text representation, named entity recognition, text generation, text analysis (lexical, syntactic, grammatical, etc.), speech recognition and synthesis, etc.

[0031] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies encompass computer vision, speech recognition, natural language processing, as well as deep learning, big data processing, and knowledge graphs.

[0032] In the present disclosure, by detecting weak points in the training text and replacing the weak points with confusing characters, a domain correction text set that is more consistent with the real errors in the domain is generated, and the domain correction text set is used to train the text correction model, thereby improving the reliability of the text correction model.

[0033] The following describes in detail the method, device, electronic device, and storage medium for acquiring text error correction training corpus according to the embodiments of the present disclosure with reference to the accompanying drawings.

[0034] Figure 1Flow chart of a method for obtaining a text error correction training corpus provided by an embodiment of the present disclosure.

[0035] As Figure 1 shown, the method includes:

[0036] Step 101, obtain a reference error correction model and an initial training corpus. Among them, the reference error correction model is trained and generated based on a general domain corpus, and the initial training corpus includes error-prone texts in the target domain and corresponding annotated texts.

[0037] Generally, the situations of confused characters in different fields may be different. For example, in the field of novels, there may be a large number of situations where "跋" is confused with "拔", while in news releases, the situation where "跋" is confused with "拔" is less. Therefore, in the present disclosure, by generating a text error correction training corpus in a specific domain, the reliability of the text error correction model trained based on this text error correction training corpus for text error correction in this domain is relatively high.

[0038] In the present disclosure, some characters in the target domain text can be replaced with the confused characters corresponding to each character in the confused character set of this domain to generate error-prone texts in this domain. For example, the "气" in "今天天气不错" can be replaced with the homophonic character "起" to generate the error-prone text "今天天起不错".

[0039] Among them, the domain confused character set can include a pinyin confusion set and an error-prone character confusion set. The pinyin confusion set can be obtained by counting the pinyins corresponding to each character in the text of this domain and combining each pinyin with the characters corresponding to the same pinyin or similar pinyins that appear in this domain. For example, the confused character set corresponding to the pinyin "da" can be: da: 大、打、答、达. The error-prone character confusion set can be obtained by counting each character in the text of this domain and the characters that each character is confused with, and combining each character with the corresponding easily confused characters. For example, the confused character set corresponding to "己" can be: 己: 已、巳.

[0040] In the present disclosure, the initial text error correction model can be trained based on a general domain corpus to obtain a reference error correction model. Among them, the general domain corpus includes error correction text pairs in each domain and corresponding annotated texts. The error correction texts in the general domain corpus can be generated based on a general confused character set, and the general confused character set includes confused characters in each domain. The text error correction model can be a knowledge-enhanced semantic representation model (ERNIE), etc., and the present disclosure does not limit this.

[0041] Step 102, input the error-prone text into the reference error correction model to obtain multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text.

[0042] In the present disclosure, the reference error correction model can output multiple predicted texts and a first probability corresponding to the predicted character at each position in each predicted text based on the probability of the predicted character at each position in the predicted text. For example, the probability corresponding to each predicted character at each position can be compared with a threshold. When the probability of a predicted character is greater than the threshold, the probability corresponding to the character set at that position can be output. If the probability of a predicted character is less than the threshold, the predicted character and the corresponding probability may not be output.

[0043] It is understandable that, since there may be multiple predicted characters at a certain position with corresponding probabilities greater than a threshold, there may be multiple predicted texts outputted.

[0044] Step 103 : determining the characters to be rewritten and the rewriting mode in the text to be corrected based on the multiple predicted texts, the first probability corresponding to the predicted characters at each position, and the annotated text.

[0045] Among them, the rewriting mode may include a pinyin-based rewriting mode, a glyph-based rewriting mode, etc. The pinyin-based rewriting mode may replace the character with a character that is different from the pinyin of the character; the glyph-based rewriting mode may rewrite the character with a character that is different from the glyph of the character, etc., which is not limited in this disclosure.

[0046] In the present disclosure, the probability of the predicted character at each position in each predicted text and the corresponding character in the annotated text can be used to determine whether each position is an error-prone point, i.e., a weak point predicted by a reference error correction model. This allows the training corpus to be updated based on the weak points predicted by the reference error correction model, resulting in more targeted training corpus. Furthermore, the text error correction model generated based on this training corpus has higher reliability and accuracy.

[0047] Step 104 : rewrite the characters to be rewritten based on the rewriting pattern to obtain updated text error correction training corpus corresponding to the target domain.

[0048] In the present disclosure, after determining the rewriting pattern of a character to be rewritten, the character can be rewritten using the rewriting pattern, i.e., the character to be rewritten is rewritten, thereby generating adversarial error correction samples to obtain updated text error correction training corpus corresponding to the target domain. Thus, character replacement is performed based on the weak points of error correction, thereby improving the quality of the generated error correction samples, and further training a text error correction model based on the error correction samples can improve the reliability of the text error correction model.

[0049] Optionally, since the rewritten adversarial error correction text may be a fluent sentence, for example, rewriting "end of the year" into "mid-year", this rewritten adversarial error correction text may affect the reliability of the trained text error correction model. Therefore, the rewritten training corpus can be filtered. For example, unsupervised training can be performed using the text in this field to obtain a fluency evaluation model. After that, a group of rewritten texts corresponding to the text to be corrected and the annotated text corresponding to the text to be corrected can be input into the fluency evaluation model, and the fluency of each rewritten text and annotated text can be determined based on the output of the evaluation model. If the fluency of the annotated text is less than the fluency of any rewritten text, the group of rewritten texts corresponding to the text to be corrected can be removed from the training corpus.

[0050] Optionally, after obtaining the updated text correction training corpus corresponding to the target domain, the reference correction model can be trained using the text correction training corpus to obtain the text correction model corresponding to the domain, thereby improving the reliability of the text correction model for domain text correction.

[0051] In the present disclosure, after obtaining a reference error correction model generated based on general domain corpus training and an initial training corpus including the text to be corrected in the target domain and the corresponding annotated text, the text to be corrected can be input into the reference error correction model to obtain multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text. Afterwards, based on the multiple predicted texts, the first probability corresponding to the predicted character at each position and the annotated text, the characters to be rewritten and the rewriting pattern in the text to be corrected are determined. Then, based on the rewriting pattern, the characters to be rewritten are rewritten to obtain the updated text error correction training corpus corresponding to the target domain. Thus, by using the reference error correction model to predict the domain text, the weak points of the reference error correction model in the prediction of the domain are determined, and then training corpus is generated for the weak points, thereby improving the quality of the generated training corpus and providing conditions for obtaining the reliability and accuracy of the text error correction model for a specific domain.

[0052] Figure 2 A flowchart of a method for obtaining text error correction training corpus provided in an embodiment of the present disclosure.

[0053] like Figure 2 As shown, the method includes:

[0054] Step 201: Obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the text to be corrected in the target domain and the corresponding annotated text.

[0055] In step 202 , the text to be corrected is input into a reference error correction model to obtain a plurality of predicted texts and a first probability corresponding to a predicted character at each position in each predicted text.

[0056] In the present disclosure, the specific implementation process of step 201-step 202 can be found in the detailed description of any embodiment of the present disclosure and will not be repeated here.

[0057] Step 203 : determining a first target probability corresponding to the annotated character at each position in the annotated text based on the matching degree between the predicted text and the annotated text and the first probability corresponding to the predicted character at each position.

[0058] In the present disclosure, each predicted text can be matched with the annotated text to determine whether each character in the predicted text is consistent with each character in the annotated text. When a character at a certain position in a predicted text is consistent with the annotated text, the first probability corresponding to the character in the predicted text can be determined as the first target probability.

[0059] It can be understood that the first target probability is the probability corresponding to the marked character, and the marked character is the correct character, so the first target probability is the probability that the prediction result of the reference error correction model is the correct character.

[0060] Step 204 : Determine a first error probability corresponding to each position based on the difference between the maximum first probability corresponding to each position and the first target probability.

[0061] In the present disclosure, in each predicted text, when the predicted character at each position is different from the marked character, it means that the predicted character is wrong. The maximum first probability corresponding to the wrongly predicted character at the position can be subtracted from the first target probability corresponding to the correctly predicted character, and the difference can be determined as the first error probability corresponding to the position, and the weak points of the reference error correction model can be determined based on the first error probability.

[0062] Step 205 : Determine the characters to be rewritten and the rewriting mode in the text to be corrected according to the first error probability corresponding to each position in the text to be corrected.

[0063] In this disclosure, the first error probability corresponding to each position in the text to be corrected can be compared with a preset threshold. When the first error probability corresponding to a position is greater than the preset threshold, it indicates that the probability of the correct prediction at that position is low, and thus the character corresponding to that position can be determined to be a weak point in the reference error correction model. Therefore, the character corresponding to that position can be determined as the character to be rewritten, and the rewriting mode can be determined to be the glyph-based rewriting mode.

[0064] Therefore, according to the first error probability corresponding to each position in the text to be corrected, the weak points predicted by the reference correction model are determined, and the characters to be rewritten are determined based on the weak points, so that the obtained training corpus is more targeted. Furthermore, the reliability and accuracy of the text correction model generated based on the training corpus are higher.

[0065] Step 206 : rewrite the characters to be rewritten based on the rewriting mode to obtain updated text error correction training corpus corresponding to the target domain.

[0066] In the present disclosure, the specific implementation process of step 206 can be found in the detailed description of any embodiment of the present disclosure and will not be repeated here.

[0067] In the present disclosure, after obtaining a reference error correction model generated based on general domain corpus training and an initial training corpus including a target domain text to be corrected and a corresponding annotated text, the text to be corrected can be input into the reference error correction model to obtain multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text. Afterwards, the first target probability corresponding to the annotated character at each position in the annotated text can be determined based on the matching degree between the predicted text and the annotated text and the first probability corresponding to the predicted character at each position, and the first error probability corresponding to each position can be determined based on the difference between the maximum first probability corresponding to each position and the first target probability. Thus, the character to be rewritten and the rewriting pattern in the text to be corrected can be determined based on the first error probability corresponding to each position in the text to be corrected. Then, based on the rewriting pattern, the correction text is rewritten to obtain an updated training corpus. By using the reference error correction model to predict the domain text, the weak points of the reference error correction model in the prediction of the domain are determined, and then training corpus is generated for the weak points, thereby improving the quality of the generated training corpus and providing conditions for obtaining the reliability and accuracy of the error correction model for the text in a specific domain.

[0068] In the present disclosure, the text error correction model may include a variety of weak points, and different types of weak points may correspond to different prediction errors. For example, a glyph weak point is a point where the pinyin prediction is correct but the glyph prediction is incorrect, and a pinyin weak point is a point where the glyph prediction is correct but the pinyin prediction is incorrect. Therefore, in the present disclosure, in order to make the acquired training corpus more generalizable, the pinyin corresponding to the text may be included in the training corpus, so that the various types of weak points of the reference error correction model can be determined based on the prediction results, and then different rewritings can be performed based on the type of predicted weak points, so that the rewritten training corpus can be more targeted for the weak point training of the text error correction model, and the reliability of the text error correction model can be further improved. The following is combined with Figure 3 , and explain the above situation in detail.

[0069] Figure 3A flowchart of a method for obtaining text error correction training corpus provided in an embodiment of the present disclosure.

[0070] like Figure 3 As shown, the method includes:

[0071] Step 301, obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the text to be corrected in the target domain, the annotated text corresponding to the text to be corrected, the first pinyin sequence corresponding to the text to be corrected, and the second pinyin sequence corresponding to the annotated text.

[0072] For a specific explanation of the reference error correction model and the text to be corrected, please refer to the detailed description of any embodiment of the present disclosure, which will not be repeated here.

[0073] In the present disclosure, the text to be corrected can be input into a pinyin generator to generate a first pinyin sequence corresponding to each corrected text. The marked text is the text with the corrected error, and the second pinyin sequence corresponding to the corresponding marked text is the pinyin sequence with the corrected error. The second pinyin sequence can also be understood as the marked pinyin sequence.

[0074] In step 302, the text to be corrected and the first pinyin sequence are input into a reference correction model to obtain multiple predicted texts, a first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences, and a second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence.

[0075] In the present disclosure, the pinyin vector and character vector corresponding to each character in the error correction text can be fused. For example, the pinyin vector and character vector corresponding to the character are spliced to generate a fusion vector corresponding to each character, and then the fusion vector corresponding to the text to be corrected can be input into the reference error correction model, and then the reference error correction model can output multiple predicted texts, a first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences, and a second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence based on the probability of the predicted characters at each position in the predicted text and the probability of the predicted pinyin.

[0076] Alternatively, the probability of each predicted character at each position may be multiplied by the probability of each predicted pinyin to determine the joint probability of each predicted character and each predicted pinyin. When a certain joint probability is greater than a threshold, the predicted character, the probability of the predicted character, the predicted pinyin and the probability of the predicted pinyin corresponding to the joint probability may be output at that position. If the joint probability is less than the threshold, the predicted character and the corresponding predicted pinyin may not be output.

[0077] In the present disclosure, the pinyin subsequence corresponding to each character can be input into a neural network model for vector mapping to obtain the pinyin vector corresponding to each character. Alternatively, the vectors corresponding to each pinyin letter in the pinyin subsequence corresponding to each character can be fused to obtain the pinyin vector corresponding to each character, which is not limited in the present disclosure.

[0078] It is understandable that, since there may be multiple situations where the joint probability is greater than the threshold at a certain position, there may be multiple error correction texts output.

[0079] Therefore, by inputting the text to be corrected and the first pinyin sequence into the reference correction model, it is possible to simultaneously predict the text and the pinyin sequence corresponding to the text, providing conditions for determining various types of weaknesses in the reference model.

[0080] Step 303 : Determine a first target probability corresponding to the annotated character at each position in the annotated text based on the matching degree between the predicted text and the annotated text and the first probability corresponding to the predicted character at each position.

[0081] Step 304: Determine a first error probability corresponding to each position based on the difference between the maximum first probability corresponding to each position and the first target probability.

[0082] In the present disclosure, the specific implementation process of step 303-step 304 can be found in the detailed description of any embodiment of the present disclosure, and will not be repeated here.

[0083] Step 305 , determining a second target probability corresponding to the second pinyin at each position in the second pinyin sequence based on the matching degree between the predicted pinyin sequence and the second pinyin sequence, and the second probability corresponding to the predicted pinyin at each position.

[0084] In the present disclosure, each predicted pinyin sequence can be matched with the second pinyin sequence to determine whether each pinyin in the predicted pinyin sequence is consistent with each pinyin in the second pinyin sequence. When the predicted pinyin at a certain position in a predicted pinyin sequence is consistent with the pinyin at that position in the second pinyin sequence, the second probability corresponding to the predicted pinyin in the predicted pinyin sequence can be determined as the second target probability.

[0085] It can be understood that the second target probability is the probability corresponding to the marked pinyin, and the marked pinyin is the correct pinyin, so the second target probability is the probability that the reference error correction model predicts the correct pinyin.

[0086] Step 306: Determine the second error probability corresponding to each position according to the difference between the maximum second probability corresponding to each position and the second target probability.

[0087] In the present disclosure, in each predicted pinyin sequence, if the predicted pinyin at each position is different from the marked pinyin, it means that the predicted pinyin is wrong. The maximum second probability corresponding to the wrongly predicted pinyin at the position can be subtracted from the second target probability corresponding to the correctly predicted pinyin, and the difference can be determined as the second error probability corresponding to the position, and the weak points of the reference error correction model can be determined based on the second error probability.

[0088] Step 307: Determine the position to be rewritten in the text to be corrected and / or the first pinyin sequence, and the rewriting mode according to the first error probability and the second error probability corresponding to each position in the text to be corrected.

[0089] The specific explanation of the rewriting mode can be found in the detailed description of any embodiment of the present disclosure, and will not be repeated here.

[0090] In the present disclosure, the first error probability corresponding to each position in the text to be corrected can be compared with a preset threshold. When the first error probability corresponding to a position is greater than the preset threshold, it indicates that the probability of the correct prediction of the position is low, and thus the character corresponding to the position can be determined to be a weak point in the glyph of the reference correction model. Therefore, the character corresponding to the position can be determined as the character to be rewritten.

[0091] Optionally, the second error probability corresponding to each position in the text to be corrected can be compared with a preset threshold. When the second error probability corresponding to a position is greater than the preset threshold, it indicates that the probability of the correct prediction of the position is low, and thus the character corresponding to the position can be determined to be a pinyin weak point of the reference correction model. Therefore, the character corresponding to the position can be determined as the character to be rewritten.

[0092] In addition, different types of weak points have different corresponding prediction errors. For example, a glyph weak point is a point where the pinyin prediction is correct but the glyph prediction is incorrect, and a pinyin weak point is a point where the glyph prediction is correct but the pinyin prediction is incorrect. Therefore, glyph weak points can be rewritten with characters with the same pinyin but different glyphs, that is, based on the glyph rewriting model. Pinyin weak points can be rewritten with characters with different pinyins, that is, based on the pinyin rewriting model. This can make the rewritten training corpus more targeted for the weak point training of the text error correction model, and further improve the reliability of the text error correction model.

[0093] Optionally, the pinyin corresponding to the character to be modified in the first pinyin sequence may be determined as the pinyin to be rewritten, and then the pinyin of the confusing character corresponding to the character to be modified may be used to rewrite the pinyin to be rewritten.

[0094] Step 308: rewrite the characters to be rewritten based on the rewriting mode to obtain updated text error correction training corpus corresponding to the target domain.

[0095] In the present disclosure, the specific implementation process of step 308 can be found in the detailed description of any embodiment of the present disclosure and will not be repeated here.

[0096] In the present disclosure, after obtaining a reference error correction model and an initial training corpus, the text to be corrected and the first pinyin sequence can be input into the reference error correction model to obtain multiple predicted texts, a first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences and a second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence. Afterwards, based on the matching degree between the predicted text and the annotated text, and the first probability corresponding to the predicted character at each position, the matching degree between the predicted pinyin sequence and the second pinyin sequence, and the second probability corresponding to the predicted pinyin at each position, the position to be rewritten in the text to be corrected and / or the first pinyin sequence, and the rewriting mode can be determined. Then, based on the rewriting mode, the character to be rewritten is rewritten to obtain the updated text correction training corpus corresponding to the target field. Therefore, by using the reference error correction model to predict the initial training corpus of the target domain, the weaknesses of the reference error correction model when correcting the text in the target domain can be determined, and then character replacement is performed based on the weaknesses, so that more targeted training corpus can be obtained, and then the text error correction model can be trained based on the training corpus, which can improve the reliability and accuracy of error correction of the text in the target domain.

[0097] Figure 4 A flowchart of a method for obtaining text error correction training corpus provided in an embodiment of the present disclosure.

[0098] like Figure 4 As shown, the method includes:

[0099] Step 401: Obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the text to be corrected in the target domain and the corresponding annotated text.

[0100] Step 402: Input the text to be corrected into a reference error correction model to obtain a plurality of predicted texts and a first probability corresponding to a predicted character at each position in each predicted text.

[0101] Step 403 : Determine the characters to be rewritten and the rewriting mode in the text to be corrected based on the multiple predicted texts, the first probability corresponding to the predicted characters at each position, and the annotated text.

[0102] In the present disclosure, the specific implementation process of steps 401 to 403 can be found in the detailed description of any embodiment of the present disclosure, and will not be repeated here.

[0103] Step 404 : Obtain the obfuscated character set corresponding to the character to be rewritten and the distribution probability of each obfuscated character in the obfuscated character set in the target domain.

[0104] In the present disclosure, considering that the confusion probabilities of the respective confusion characters corresponding to each character may be different in a certain field, or the confusion probability of each character may be different. For example, the probability that "的" is confused with "得" is different from the probability that "的" is confused with "地", or the confusion probability of "的" is different from the confusion probability of "好". In order to make the generated error correction text as close as possible to the real text in the field, the distribution probabilities of the respective confusion characters in the target field and the distribution probabilities of the confusion of each character can be statistically calculated, so as to generate an error correction text that better conforms to the error situation of the actual text in the field according to the distribution probabilities corresponding to the respective confusion characters and the distribution probabilities of the confusion of each character.

[0105] Step 405: Determine the target character corresponding to the character to be rewritten according to the distribution probability of each confusion character in the target field.

[0106] In the present disclosure, when rewriting a certain character to be rewritten, multiple target characters with the same distribution probability can be generated according to the distribution probability of each confusion character corresponding to the character to be rewritten in the target field. For example, assume that the distribution probability corresponding to the confusion character "得" is 0.7, and the distribution probability corresponding to the confusion character "地" is 0.3. If 10 error correction sentences need to be generated, then the target characters include 7 "得"s and 3 "地"s.

[0107] Step 406: Use the target character to replace the character to be rewritten to generate an updated text error correction training corpus.

[0108] In the present disclosure, each character to be rewritten in the error correction text can be rewritten multiple times with one character from multiple target characters to generate multiple updated text error correction training corpora.

[0109] After obtaining a reference error correction model generated based on general domain corpus training and an initial training corpus including the text to be corrected in the target domain and the corresponding annotated text, the text to be corrected can be input into the reference error correction model to obtain multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text, and based on the multiple predicted texts, the first probability corresponding to the predicted character at each position and the annotated text, the characters to be rewritten and the rewriting pattern in the text to be corrected are determined. After that, the confusion character set corresponding to the character to be rewritten and the distribution probability of each confused character in the confusion character set in the target domain can be obtained, and based on the distribution probability of each confused character in the target domain, the target character corresponding to the character to be rewritten can be determined. Then, the target character can be used to replace the character to be rewritten to generate an updated text error correction training corpus. Therefore, by using the reference error correction model to predict the domain text, the weak points of the reference error correction model in the prediction of the domain are determined, and then the characters are rewritten according to the distribution probability of each confusing character in the domain to generate training corpus that is more in line with the actual text error situation in the domain, thereby improving the quality of the generated training corpus and providing conditions for obtaining the reliability and accuracy of the text error correction model in a specific domain.

[0110] In order to implement the above embodiment, the embodiment of the present disclosure further proposes a device for acquiring text error correction training corpus. Figure 5 A schematic diagram of the structure of a device for acquiring text error correction training corpus provided in an embodiment of the present disclosure.

[0111] like Figure 5 As shown, the apparatus 500 for acquiring text error correction training corpus includes: an acquisition module 510 , a prediction module 520 , a determination module 530 , and a rewriting module 540 .

[0112] An acquisition module 510 is configured to acquire a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the target domain text to be corrected and the corresponding annotated text;

[0113] Prediction module 520, configured to input the text to be corrected into a reference correction model to obtain a plurality of predicted texts and a first probability corresponding to a predicted character at each position in each predicted text;

[0114] A determination module 530 is configured to determine a character to be rewritten and a rewriting mode in the text to be corrected based on the plurality of predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text;

[0115] The rewriting module 540 is configured to rewrite the characters to be rewritten based on the rewriting pattern to obtain updated text error correction training corpus corresponding to the target domain.

[0116] In a possible implementation of the embodiment of the present disclosure, the determination module 530 is configured to:

[0117] Determining a first target probability corresponding to the annotated character at each position in the annotated text based on the matching degree between the predicted text and the annotated text and the first probability corresponding to the predicted character at each position;

[0118] Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability;

[0119] According to the first error probability corresponding to each position in the text to be corrected, the characters to be rewritten and the rewriting mode in the text to be corrected are determined.

[0120] In a possible implementation of the embodiment of the present disclosure, the initial training corpus also includes a first pinyin sequence corresponding to the text to be corrected and a second pinyin sequence corresponding to the annotated text. The prediction module 520 is used to:

[0121] The text to be corrected and the first pinyin sequence are input into the reference correction model to obtain multiple predicted texts, the first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences and the second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence.

[0122] In a possible implementation of the embodiment of the present disclosure, the determination module 530 is configured to:

[0123] Determining a first target probability corresponding to the annotated character at each position in the annotated text based on the matching degree between the predicted text and the annotated text and the first probability corresponding to the predicted character at each position;

[0124] Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability;

[0125] Determining a second target probability corresponding to the second pinyin at each position in the second pinyin sequence based on the matching degree between the predicted pinyin sequence and the second pinyin sequence and the second probability corresponding to the predicted pinyin at each position;

[0126] Determining a second error probability corresponding to each position according to a difference between the maximum second probability corresponding to each position and the second target probability;

[0127] According to the first error probability and the second error probability corresponding to each position in the text to be corrected, the position to be rewritten in the text to be corrected and / or the first pinyin sequence, and the rewriting mode are determined.

[0128] In a possible implementation of the embodiment of the present disclosure, the prediction module 520 is configured to:

[0129] Determine a pinyin subsequence corresponding to each character in the first pinyin sequence;

[0130] Aggregate each pinyin subsequence to determine the pinyin vector corresponding to each character;

[0131] Fuse the character vector corresponding to each character in the text to be corrected with the pinyin vector to obtain a fused vector corresponding to each character;

[0132] The fusion vector corresponding to the text to be corrected is input into the reference error correction model.

[0133] In a possible implementation of the embodiment of the present disclosure, the rewriting module 540 is configured to:

[0134] Obtaining the obfuscated character set corresponding to the character to be rewritten and the distribution probability of each obfuscated character in the obfuscated character set in the target field;

[0135] Determine the target character corresponding to the character to be rewritten according to the distribution probability of each obfuscated character in the target field;

[0136] The target characters are used to replace the characters to be rewritten to generate updated text error correction training corpus.

[0137] It should be noted that the explanation of the aforementioned embodiment of the method for obtaining text error correction training corpus is also applicable to the device of this embodiment, so it will not be repeated here.

[0138] In the present disclosure, after obtaining a reference error correction model generated based on general domain corpus training and an initial training corpus including the text to be corrected in the target domain and the corresponding annotated text, the text to be corrected can be input into the reference error correction model to obtain multiple predicted texts and the first probability corresponding to the predicted character at each position in each predicted text. Afterwards, based on the multiple predicted texts, the first probability corresponding to the predicted character at each position and the annotated text, the characters to be rewritten and the rewriting pattern in the text to be corrected are determined. Then, based on the rewriting pattern, the characters to be rewritten are rewritten to obtain the updated text error correction training corpus corresponding to the target domain. Thus, by using the reference error correction model to predict the domain text, the weak points of the reference error correction model in the prediction of the domain are determined, and then training corpus is generated for the weak points, thereby improving the quality of the generated training corpus and providing conditions for obtaining the reliability and accuracy of the text error correction model for a specific domain.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 602 or a computer program loaded from a storage unit 608 into a RAM (Random Access Memory) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0142] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0143] The computing unit 601 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for obtaining text error correction training corpus. For example, in some embodiments, the method for obtaining text error correction training corpus can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for obtaining text error correction training corpus described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for acquiring text error correction training corpus in any other appropriate manner (for example, by means of firmware).

[0144] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0149] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0150] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product. When an instruction processor in the computer program product is executed, the method for acquiring text error correction training corpus proposed in the above embodiment of the present disclosure is executed.

[0151] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0152] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for obtaining text error correction training corpus, comprising: Obtain a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the target domain text to be corrected and the corresponding annotated text; Inputting the text to be corrected into the reference error correction model to obtain a plurality of predicted texts, and using the reference error correction model to predict the probability of a character at each position in the predicted text, outputting a plurality of predicted texts and a first probability corresponding to the predicted character at each position in each predicted text; Determining a character to be rewritten and a rewriting mode in the text to be corrected based on the multiple predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text; Based on the rewriting pattern, the character to be rewritten is rewritten to obtain an updated text error correction training corpus corresponding to the target domain; wherein: a confusion character set corresponding to the character to be rewritten and a distribution probability of each confused character in the confusion character set in the target domain are obtained; a target character corresponding to the character to be rewritten is determined according to the distribution probability of each confused character in the target domain; and the target character is used to replace the character to be rewritten to generate the updated text error correction training corpus.

2. The method according to claim 1, wherein The determining, based on the plurality of predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text, the character to be rewritten and the rewriting mode in the text to be corrected includes: Determining a first target probability corresponding to the annotated character at each position in the annotated text based on a matching degree between the predicted text and the annotated text and a first probability corresponding to the predicted character at each position; Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability; According to the first error probability corresponding to each position in the text to be corrected, the characters to be rewritten and the rewriting mode in the text to be corrected are determined.

3. The method according to claim 1, wherein The initial training corpus also includes a first pinyin sequence corresponding to the text to be corrected and a second pinyin sequence corresponding to the annotated text. The text to be corrected is input into the reference correction model to obtain a plurality of predicted texts. The reference correction model is used to predict the probability of a character at each position in the predicted text to output a plurality of predicted texts and a first probability corresponding to the predicted character at each position in each predicted text, including: The text to be corrected and the first pinyin sequence are input into the reference error correction model to obtain multiple predicted texts, a first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences and a second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence.

4. The method according to claim 3, wherein: The determining, based on the plurality of predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text, the character to be rewritten and the rewriting mode in the text to be corrected includes: Determining a first target probability corresponding to the annotated character at each position in the annotated text based on a matching degree between the predicted text and the annotated text and a first probability corresponding to the predicted character at each position; Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability; Determining a second target probability corresponding to the second pinyin at each position in the second pinyin sequence based on the matching degree between the predicted pinyin sequence and the second pinyin sequence and the second probability corresponding to the predicted pinyin at each position; Determining a second error probability corresponding to each position according to a difference between the maximum second probability corresponding to each position and the second target probability; According to the first error probability and the second error probability corresponding to each position in the text to be corrected, the position to be rewritten in the text to be corrected and / or the first pinyin sequence, and the rewriting mode are determined.

5. The method according to claim 3, wherein: The step of inputting the text to be corrected and the first pinyin sequence into the reference error correction model includes: Determining a pinyin subsequence corresponding to each character in the first pinyin sequence; Aggregating each of the pinyin subsequences to determine a pinyin vector corresponding to each character; Fusing the character vector corresponding to each character in the text to be corrected with the pinyin vector to obtain a fusion vector corresponding to each character; The fusion vector corresponding to the text to be corrected is input into the reference error correction model.

6. A device for acquiring text error correction training corpus, comprising: An acquisition module is used to acquire a reference error correction model and initial training corpus, wherein the reference error correction model is generated based on general domain corpus training, and the initial training corpus includes the target domain text to be corrected and the corresponding annotated text; a prediction module configured to input the text to be corrected into the reference error correction model to obtain a plurality of predicted texts, and output a plurality of predicted texts and a first probability corresponding to the predicted character at each position in each predicted text based on the probability of predicting a character at each position in the predicted text using the reference error correction model; a determination module, configured to determine a character to be rewritten and a rewriting mode in the text to be corrected based on the plurality of predicted texts, the first probability corresponding to the predicted character at each position, and the annotated text; A rewriting module is used to rewrite the character to be rewritten based on the rewriting mode to obtain an updated text error correction training corpus corresponding to the target domain; wherein: a confusion character set corresponding to the character to be rewritten and a distribution probability of each confusion character in the confusion character set in the target domain are obtained; a target character corresponding to the character to be rewritten is determined based on the distribution probability of each confusion character in the target domain; and the target character is used to replace the character to be rewritten to generate the updated text error correction training corpus.

7. The device according to claim 6, wherein The determining module is configured to: Determining a first target probability corresponding to the annotated character at each position in the annotated text based on a matching degree between the predicted text and the annotated text and a first probability corresponding to the predicted character at each position; Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability; According to the first error probability corresponding to each position in the text to be corrected, the characters to be rewritten and the rewriting mode in the text to be corrected are determined.

8. The device according to claim 6, wherein The initial training corpus also includes a first pinyin sequence corresponding to the text to be corrected and a second pinyin sequence corresponding to the annotated text. The prediction module is used to: The text to be corrected and the first pinyin sequence are input into the reference error correction model to obtain multiple predicted texts, a first probability corresponding to the predicted character at each position in each predicted text, multiple predicted pinyin sequences and a second probability corresponding to the predicted pinyin at each position in each predicted pinyin sequence.

9. The device according to claim 8, wherein The determining module is configured to: Determining a first target probability corresponding to the annotated character at each position in the annotated text based on a matching degree between the predicted text and the annotated text and a first probability corresponding to the predicted character at each position; Determine a first error probability corresponding to each position according to a difference between the maximum first probability corresponding to each position and the first target probability; Determining a second target probability corresponding to the second pinyin at each position in the second pinyin sequence based on the matching degree between the predicted pinyin sequence and the second pinyin sequence and the second probability corresponding to the predicted pinyin at each position; Determining a second error probability corresponding to each position according to a difference between the maximum second probability corresponding to each position and the second target probability; According to the first error probability and the second error probability corresponding to each position in the text to be corrected, the position to be rewritten in the text to be corrected and / or the first pinyin sequence, and the rewriting mode are determined.

10. The device according to claim 8, wherein The prediction module is used to: Determining a pinyin subsequence corresponding to each character in the first pinyin sequence; Aggregating each of the pinyin subsequences to determine a pinyin vector corresponding to each character; Fusing the character vector corresponding to each character in the text to be corrected with the pinyin vector to obtain a fusion vector corresponding to each character; The fusion vector corresponding to the text to be corrected is input into the reference error correction model.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images