A text error correction method, system, electronic device and storage medium

Abstract

The application discloses a text error correction method and system, electronic equipment and a storage medium, wherein the method comprises: obtaining a text to be corrected; performing error detection on the text to be corrected to obtain different types of text errors existing in the text to be corrected, and setting corresponding first labels for the different types of text errors; wherein the first labels comprise different types of error labels to correspondingly mark different types of text errors; correcting the different types of text errors based on the different types of first labels to obtain a corrected text; wherein in the correction stage, a word group at a correction position is scored based on a preset word table, and a candidate target text corresponding to the highest word group score is determined as the corrected text. That is, the application classifies text errors and sets corresponding labels, and then performs text error correction, that is, determines the corrected text by scoring the word group at the correction position based on the preset word table, which can reduce the probability of text ambiguity and improve the accuracy of text error correction.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a text correction method, system, electronic device, and storage medium. Background Technology

[0002] With the increasing maturity and popularity of computer technology, data analysis and processing, especially text correction for natural language, often rely on high-quality dictionaries for text correction.

[0003] In the research and practice of current technologies, the current method of using high-quality dictionaries for error correction is prone to causing ambiguity in the text and affecting the accuracy of text correction. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a text correction method, system, electronic device, and storage medium that can reduce the probability of text ambiguity and improve the accuracy of text correction.

[0005] To address the aforementioned technical problems, this application provides a text correction method, comprising: acquiring a text to be corrected; performing error detection on the text to be corrected to obtain different types of text errors present in the text, and assigning corresponding first tags to the different types of text errors; wherein the first tags include error tags of different types to correspondingly mark the different types of text errors; correcting the different types of text errors based on the first tags of different types to obtain corrected text; wherein, in the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text.

[0006] In some embodiments, the step of performing error detection on the text to be corrected to obtain different types of text errors present in the text to be corrected, and setting corresponding first labels for the different types of text errors, includes: inputting the text to be corrected into a preset error detection model to obtain different types of text errors present in the text to be corrected; wherein, the different types of text errors include: a first text error, a second text error, and a third text error; setting corresponding first labels for the different types of text errors, and setting second labels for the normal text; wherein, the first labels include: a first error label for replacement, a second error label for insertion, and a third error label for deletion, and the second label includes a normal label; setting the first error label for the phrase corresponding to the first text error, setting the second error label for the phrase corresponding to the second text error, setting the third error label for the phrase corresponding to the third text error, and setting the normal label for the phrase corresponding to the normal text; and constructing a label sequence for the text to be corrected based on the first error label, the second error label, the third error label, and the normal label.

[0007] In some embodiments, after constructing the tag sequence of the text to be corrected, the method further includes: editing the text to be corrected based on the tag sequence to obtain edited text; wherein, the phrases corresponding to the normal tags are restored to the original text, and the phrases corresponding to the first error tag, the second error tag, and the third error tag are edited into third tags; the restored original text and the third tags are used to determine the candidate target text.

[0008] In some embodiments, the step of scoring the phrases at the error correction points based on a preset vocabulary, and determining the corrected text by using the candidate target text corresponding to the highest score, includes: scoring the phrases corresponding to the third tag in the candidate target text based on the preset vocabulary to obtain the phrase with the highest score in the preset vocabulary for each third tag; wherein, the preset vocabulary is a vocabulary used by the error correction model to perform contextual semantic judgment; and replacing the corresponding phrase in the candidate target text with the phrase with the highest score to obtain the corrected text.

[0009] In some embodiments, correcting different types of text errors based on different types of the first tag to obtain corrected text includes: replacing the first error text corresponding to the first error tag; inserting the second error text corresponding to the second error tag; deleting the third error text corresponding to the third error tag; and determining the corrected text based on the text replacement, text insertion, and text deletion performed on the text error.

[0010] In some embodiments, the first label is generated based on a preset minimum edit distance; wherein the minimum edit distance is the minimum number of edits required to convert the text to be corrected into corrected text through an edit operation.

[0011] In some embodiments, after determining the corrected text, the method further includes: matching and scoring the corrected text based on a preset confusion set; wherein the preset confusion set includes a glyph confusion set and a phonetic confusion set; increasing the adjustment weight for phrases that match the preset confusion set matching score; adjusting the phrase score with the adjustment weight and a preset penalty coefficient, and then obtaining the candidate target text corresponding to the highest adjusted phrase score to determine the final corrected text.

[0012] In some embodiments, the step of matching and scoring the corrected text based on a preset confusion set includes: matching and scoring the phrases that were corrected in the corrected text based on the preset confusion set; wherein, the matching score for phonetic text errors is obtained based on the phonetic confusion set, and the matching score for character shape text errors is obtained based on the character shape confusion set; the step of adding adjustment weights to phrases that meet the matching score of the preset confusion set includes: adding adjustment weights to phrases that meet the matching score of the preset confusion set for both phonetic and character shape text errors, so as to improve the matching score of the corresponding text errors for both phonetic and character shape.

[0013] To address the aforementioned technical problems, another technical solution of this application is to provide a text correction system, comprising: an acquisition module, an error detection module, and a correction module; wherein, the acquisition module is used to acquire the text to be corrected; the error detection module is used to perform error detection on the text to be corrected to obtain different types of text errors existing in the text to be corrected, and to assign corresponding first tags to the different types of text errors; wherein, the first tags include error tags of different types to correspondingly mark the different types of text errors; the correction module corrects the different types of text errors based on the first tags of different types to obtain the corrected text; wherein, in the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, the electronic device including a memory and a processor coupled to the memory, the memory storing at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the text correction method as described above.

[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium having at least one program, which, when loaded and executed by a processor, is used to implement the text correction method as described above.

[0016] Unlike current technologies, the text correction method provided in this application includes: acquiring the text to be corrected; performing error detection on the text to be corrected to obtain different types of text errors present in the text, and assigning corresponding first tags to different types of text errors; wherein the first tags include error tags of different types to mark different types of text errors; correcting different types of text errors based on the different types of first tags to obtain corrected text; wherein, in the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text. That is, this application classifies text errors, constructs corresponding tags for the classified text errors, and then performs correction based on the tags. Specifically, it scores the phrases at the correction points using a preset vocabulary, and determines the corrected text based on the candidate target text with the highest score. Determining the corrected text based on the score effectively reduces the probability of text ambiguity and improves the accuracy of text correction. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the text correction method of this application;

[0018] Figure 2 This is a flowchart illustrating an embodiment of step S2 in this application;

[0019] Figure 3 This is a flowchart illustrating an embodiment of the process after constructing the tag sequence of the text to be corrected in this application;

[0020] Figure 4 This is a flowchart illustrating an embodiment of step S13 in this application;

[0021] Figure 5 This is a flowchart illustrating an embodiment of step S14 in this application;

[0022] Figure 6 This is a flowchart illustrating the second embodiment of the text correction method of this application;

[0023] Figure 7 This is a schematic diagram of the structure of an embodiment of the text correction system of this application;

[0024] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0025] Figure 9This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] Current text correction methods, especially those for natural language, rely on high-quality dictionaries and cannot resolve ambiguities or classify text errors, resulting in a large number of errors and impacting the accuracy of text correction.

[0029] Therefore, a text correction method is proposed. By classifying text errors and constructing corresponding labels for the classified text errors, the method then performs correction based on the labels. Specifically, the phrases at the point of error are scored, and the text with the highest score is determined as the final corrected text. Determining the corrected text based on the score can effectively reduce the probability of text ambiguity and improve the accuracy of text correction.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the text correction method in this application. It should be noted that if substantially the same result is achieved, the method of this application is not necessarily identical. Figure 1 The sequence of processes shown is limited.

[0031] like Figure 1 As shown, this text correction method includes the following steps:

[0032] S11. Obtain the text to be corrected.

[0033] The text to be corrected refers to the original text that has not undergone error correction. The text to be corrected may contain various text errors, or it may not contain any text errors. Text errors include, but are not limited to, missing words, redundant words, word errors, word order errors, etc.

[0034] S12. Perform error detection on the text to be corrected to obtain different types of text errors in the text to be corrected, and set corresponding first labels for different types of text errors; wherein, the first label contains error labels of different types to mark the different types of text errors.

[0035] Error detection involves using text error detection methods to detect errors in the text to be corrected and then classifying them according to their type.

[0036] Specifically, after obtaining the text to be corrected, the text correction detection method is used to detect errors in the text. For the detected text errors, they are classified according to their text error types, and corresponding error labels are set for each text error type.

[0037] For example, text error types include text errors to be replaced, text errors to be inserted, and text errors to be deleted. After error detection, all text errors to be replaced, text errors to be inserted, and text errors to be deleted in the text to be corrected are obtained. For all text errors to be replaced, a corresponding error label is set. For text errors to be inserted, a corresponding error label is set. For text errors to be deleted, a corresponding error label is set.

[0038] S13. Correct different types of text errors based on different types of first tags to obtain corrected text; wherein, in the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text.

[0039] The first tag contains different types of error tags; that is, different types of first tags correspond to different types of error tags. For example, the first tag of the text to be replaced corresponds to the error tag to be replaced, the first tag of the text to be inserted corresponds to the error tag to be inserted, and the first tag of the text to be deleted corresponds to the error tag to be deleted. The preset vocabulary can refer to a commonly used vocabulary, such as a vocabulary containing over 5000 Chinese characters and other characters, totaling over 20000 characters. The preset vocabulary is mainly used to score semantic coherence and to score the phrases at the error points in the corrected text.

[0040] Specifically, after acquiring different types of text errors in the text to be corrected and assigning corresponding first tags to the text errors, the text error correction method corrects and edits the text errors at the first tags according to the type of the first tags to obtain the corrected text. Specifically, for different text errors in the text to be corrected, corresponding corrections are performed based on their error tags. For corrected sections, a preset vocabulary is used to score the corrected section using word groups, and the word group corresponding to the highest word group score is obtained. Based on this word group, a corresponding candidate target text is obtained, and this candidate target text is then identified as the corrected text.

[0041] For example, for text errors corresponding to the first tag to be replaced, corresponding text replacement is performed; for text errors corresponding to the first tag to be inserted, corresponding text insertion is performed; and for text errors corresponding to the first tag to be deleted, corresponding text deletion is performed.

[0042] In this embodiment, the present application classifies the text error types of the text to be corrected, sets corresponding labels based on the classification results, and then performs text correction based on the labels. Furthermore, it scores the phrases at the correction points based on a preset vocabulary, and determines the candidate target text corresponding to the highest phrase score as the corrected text. This can effectively reduce the probability of text ambiguity and improve the accuracy of text correction.

[0043] In some embodiments, a language representation model, such as BERT, can be used for text correction of the text to be corrected. BERT training consists of two steps: pre-training and fine-tuning. In the pre-training phase, the model is trained on different pre-training tasks using unlabeled data, thus the BERT model contains a large amount of common-sense knowledge. One pre-training task is Masked Language Modeling, a cloze-style language modeling task that requires the model to select words to fill in based on context. Specifically, words in the text are randomly replaced with "[MASK]", and then the BERT model is used to reconstruct the text and predict the replaced words. In fine-tuning, the model is first initialized based on the parameters obtained from pre-training, and then all parameters are fine-tuned using labeled data from downstream specific tasks. This application mainly uses the BERT model for fine-tuning, enabling it to learn domain-specific knowledge, such as relevant information in telecommunications fraud cases.

[0044] See Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of step S2. To more accurately identify text errors in the text to be corrected, it is necessary to classify the text error types, such as... Figure 2 As shown, it includes the following steps:

[0045] S121. Input the to-be-corrected text into a preset error detection model, to obtain different types of text errors existing in the to-be-corrected text;

[0046] Wherein, the different types of text errors include: a first text error, a second text error, and a third text error.

[0047] Wherein, the first text error may be a replaceable text error, the second text error may be an insertable text error, and the third text error may be a deletable text error.

[0048] Specifically, after obtaining the to-be-corrected text, the to-be-corrected text is input into the preset error detection model, and the preset error detection model performs error detection on the to-be-corrected text, so as to obtain and locate different types of text errors existing in the to-be-corrected text.

[0049] For example, the to-be-corrected text "potential potential bidder must check the confirmation information in time" is processed into the input form of the BERT model. Specifically, after being divided at the character granularity, "[CLS]" and "[SEP]" are added at the head and tail to mark the start and end of the sentence, so as to obtain the input sequence "[CLS], potential, potential, potential, potential, bidder, person, must, timely, check, view, confirm, information, [SEP]". The text sequence is input into the text error detection model, and the annotation corresponding to each character is obtained, that is, "E, E, E, D, D, E, E, E, R, E, E, E, E, I, E, E, E, E".

[0050] In some embodiments, the preset error detection model may be a sequence labeling model. After the to-be-corrected text is input into the preset error detection model, a corresponding label is output at the position of each word phrase, for locating the text error existing in the to-be-corrected text, and the corresponding editing operation. It is a sequence labeling model constructed based on the BERT model. Generally speaking, errors occur at positions with incoherent semantics, so the BERT model can identify them efficiently. The model includes two parts: a BERT word vector encoding layer and an error detection layer. The BERT word vector encoding layer can generate dynamic word vector representations according to different input texts, so as to capture the context semantics; the error detection layer is a multi-class classifier composed of a multi-layer fully connected neural network, which maps the dynamic hidden layer word vector representations to the label space to perform category discrimination, and the labels are the four most basic sequence operations: "keep", "replace", "insert" and "delete".

[0051] S122. Set corresponding first labels for different types of text errors, and set a second label for normal text;

[0052] The first label includes: a first error label for replacement, a second error label for insertion, and a third error label for deletion. The second label includes a normal label. The first error label is set for the phrase corresponding to the first text error, the second error label is set for the phrase corresponding to the second text error, the third error label is set for the phrase corresponding to the third text error, and the normal label is set for the phrase corresponding to the normal text.

[0053] Specifically, after inputting the text to be corrected into the preset error detection model, for each type of text error found in the text, a corresponding first label is set according to the type of text error. That is, for replaceable text errors, a first error label for replacement is set; for insertable text errors, a second error label for insertability is set; for deleteable text errors, a third error label for deletionability is set; and for normal text without text errors, a second label is set.

[0054] For example, the first error label is R (replace, indicating that the word at the corresponding position should be replaced), the second error label is I (insert, indicating that several words need to be inserted before the position), the third error label is D (delete, indicating that the word at the corresponding position should be deleted), and the second label is E (equal, indicating that the word at the corresponding position should not be changed). If the text to be corrected is "Potential bidders must check the confirmation information in a timely manner," and the corrected text is "Potential bidders must check the email confirmation information in a timely manner," then for text errors in the text to be corrected that can be replaced, the first error label is R; for text errors in the text to be corrected that can be inserted, the second error label is I; for text errors in the text to be corrected that can be deleted, the third error label is D; and for normal text, the second label is E. The constructed text annotation sequence is as follows:

[0055] E E E D D E E E R E E E E I E E E E

[0056] Where CLS is the start position of the text and SEP is the end position of the text.

[0057] S123. Construct a label sequence for the text to be corrected based on the first error label, the second error label, the third error label, and the normal label.

[0058] After obtaining the first error label, the second error label, the third error label, and the normal label, the text error can be accurately located. To facilitate subsequent operations, the text to be corrected is used to construct a label sequence. Subsequent operations can be performed quickly based on the label sequence, thereby improving the accuracy of error correction.

[0059] Specifically, the corresponding first error label, second error label, third error label and normal label in the text to be error-corrected are spliced and combined to construct a label sequence of the text to be error-corrected. For example, if the text to be error-corrected is "潜在潜在投标人须及时查看确认信息", the constructed label sequence is "EEDDEEEREEEEIEEE".

[0060] In this embodiment, by setting different labels for different text errors and further constructing the label sequence of the text to be error-corrected, the error correction efficiency can be effectively improved, and the error correction accuracy can be improved at the same time.

[0061] In some embodiments, the first label is generated based on a preset minimum edit distance; wherein the minimum edit distance refers to the number of edits required to convert one string to another, including the three operations of inserting a character, deleting a character, and replacing a character.

[0062] For example, the minimum edit distance is the minimum number of edits required for the text to be error-corrected to be converted into the corrected text through editing operations, wherein the minimum edit distance.

[0063] Wherein, during the training of the error detection model, the data labels are obtained according to the minimum edit distance. By calculating the minimum edit distance and the corresponding edit sequence between the text to be error-corrected and the corrected text, the detection label can be constructed according to the edit sequence, so as to solve all possible text errors in a simple form.

[0064] For example, taking the text to be error-corrected "潜在潜在投标人须及时查看确认信息" and the correct corrected text "潜在投标人需及时查看邮件确认信息" as examples, the minimum edit distance between these two texts is 5, and the corresponding edit operation sequence is: Step 1: Delete "潜" to transform into "潜在在投标人须及时查看确认信息"; Step 2: Delete "在" to transform into "潜在投标人须及时查看确认信息"; Step 3: Replace "须" with "需" to transform into "潜在投标人需及时查看确认信息"; Step 4: Insert "邮" to transform into "潜在投标人需及时查看邮确认信息"; Step 5: Insert "件" to transform into "潜在投标人需及时查看邮件确认信息", therefore, the corresponding label sequence of the text to be error-corrected can be constructed.

[0065] Wherein, the loss of the error detection model is calculated by cross entropy, that is:

[0066]

[0067] Wherein, L is the sequence length, y i is the data annotation at the position, is the probability corresponding to the category at the position output by the error detection model, which is obtained after normalization by the function:

[0068]

[0069] Among them, z ij K is the score for category j at position i in the error detection model output, and K is the number of categories.

[0070] The output of the error detection model is used as the input to build the error correction model, specifically by editing the input text based on the output editing operation.

[0071] See Figure 3 , Figure 3 This is a flowchart illustrating one embodiment of the process after constructing the tag sequence of the text to be corrected in this application. To directly perform a second edit based on the text to be corrected and improve the accuracy of error correction, this embodiment directly edits the text to be corrected, such as... Figure 3 As shown, it includes:

[0072] S124. Edit the text to be corrected based on the label sequence to obtain the edited text;

[0073] Among them, the phrases corresponding to the normal tags are restored to the original text, and the phrases corresponding to the first, second, and third error tags are edited into the third tag;

[0074] Editing involves editing the text to be corrected, specifically editing, replacing, supplementing, or deleting erroneous phrases in the text.

[0075] Specifically, after obtaining the tag sequence corresponding to the text to be corrected, the text to be corrected is edited accordingly based on the tag sequence. The phrases corresponding to the third error tag are deleted, the phrases corresponding to the first and second error tags are edited to the third tag, and the phrases corresponding to the second tag are left unchanged. Thus, an edited text consisting of the third tag and normal text can be obtained.

[0076] S125. Determine candidate target texts using the restored original text and the third tag.

[0077] Among them, the text corresponding to the normal label is the original text, and the third label is the label on the text to be corrected. Based on the original text and the third label of the part that needs to be corrected, a candidate target text can be determined.

[0078] For example: Assuming the third tag is MASK, taking the text to be corrected as "Potential bidders must check and confirm the information in a timely manner" and the corrected text as "Potential bidders need to check and confirm the email information in a timely manner", the candidate target text would be "Potential bidders [MASK] check and confirm the [MASK][MASK] information in a timely manner".

[0079] In this embodiment, the text to be corrected is edited using a tag sequence to obtain the edited text. The original text and the third tag are then concatenated and combined to determine the candidate target text. This method can accurately identify which parts of the text to be corrected contain text errors and require further editing. Subsequent editing can be performed directly based on the candidate target text, thereby improving the accuracy of error correction.

[0080] See Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of step S13 in this application. To further improve error correction accuracy, each label is configured in detail, such as... Figure 4 As shown, it includes:

[0081] S131. Replace the text of the first error corresponding to the first error label.

[0082] The first error label is the error label that can be replaced with text. Therefore, the first error text corresponding to the first error label can be replaced with text.

[0083] S132. Insert text into the second error text corresponding to the second error label.

[0084] The second error label is the error label for which text can be inserted. Therefore, text can be inserted into the second error text corresponding to the second error label.

[0085] S133. Delete the text corresponding to the third error label.

[0086] Among them, the third error label is the error label that can be deleted. Therefore, the third error text corresponding to the third error label can be deleted.

[0087] S134. Based on text replacement, text insertion, and text deletion performed on text errors, determine the corrected text.

[0088] Specifically, text replacement, text insertion, and text deletion are performed on the text to be corrected based on the first error label, the second error label, and the third error label, respectively.

[0089] Specifically, after replacing text based on the first error label, inserting text based on the second error label, and deleting text based on the third error label, the obtained text can be identified as the corrected text.

[0090] In one embodiment, an error correction module can be used for error correction. The error correction model has the same model structure as the BERT pre-training stage model, including a BERT word vector encoding layer and an MLM prediction correction layer. It can fully import pre-trained weights for training, thereby effectively utilizing pre-trained knowledge and providing a large amount of effective prior knowledge for generating candidate words. Specifically, the calculation process involves inputting the edited text to be corrected into the model, and then predicting the output corresponding to each "[MASK]". This output is a |V|-dimensional vector, where each value represents the score of the selected word in the vocabulary V. It is worth noting that since the "insertion" operation specifies a fixed number of "[MASK]" marks, when the predicted text is insufficient to fill all the positions of the "[MASK]" marks, it is filled with "[EMPTY]" marks. The error detection model also uses cross-entropy to calculate the model loss.

[0091] The difference between the model inference process and the training process is that the text error detection model is used to predict the corresponding operation sequence. The text is then edited based on the inferred operation sequence and input into the text correction model to predict the character at each "[MASK]" position, thus obtaining the final result.

[0092] In this embodiment, based on the different error labels corresponding to different types of errors in the text to be corrected, text correction can be performed effectively and accurately, thus significantly improving the accuracy of error correction.

[0093] See Figure 5 , Figure 5 This is a flowchart illustrating an embodiment following step S134 in this application. After obtaining the third tag, to improve the accuracy of error correction, it is necessary to score the corrected text using phrases, and determine the corrected text based on the phrase with the highest score, such as... Figure 5 As shown, it includes:

[0094] S135. Based on the preset vocabulary list, score the word groups corresponding to the third tag in the candidate target text to obtain the word group with the highest preset vocabulary score for each third tag.

[0095] The pre-defined vocabulary can refer to a commonly used vocabulary, such as one containing over 5,000 Chinese characters and other characters, totaling over 20,000 characters. The pre-defined vocabulary is primarily used to score semantic coherence, specifically for scoring the phrases at the error correction points in the corrected text. For example, the pre-defined vocabulary might be used by the error correction model to perform contextual semantic judgment. The phrase corresponding to the third label is the phrase at the error correction point. The score refers to the judgment of the contextual semantics between the phrase in the vocabulary and the phrase corresponding to the third label; the more fluent the contextual semantics, the higher the score.

[0096] Specifically, after obtaining the candidate target text, which contains a third tag, the word groups corresponding to the third tag are scored using a preset vocabulary and contextual semantics, and the scoring results are sorted to obtain the word group with the highest score.

[0097] S136. Replace the corresponding phrase in the candidate target text with the phrase with the highest score to obtain the corrected text.

[0098] In this process, the corrected text needs to be replaced with selected phrases to obtain more accurate correction results.

[0099] Specifically, after obtaining the highest-scoring phrase, the phrases corresponding to each third tag are replaced according to the highest-scoring phrases in each place, thus obtaining the corrected text.

[0100] See Figure 6 , Figure 6 This is a flowchart illustrating the second embodiment of the text correction method in this application. To further improve the accuracy of error correction, this embodiment adjusts the word group scoring by increasing the proportion of homophones, near-homophones, and similar-looking characters, thereby improving the accuracy of error correction. Figure 6 As shown, the text correction method includes:

[0101] S21. Obtain the text to be corrected.

[0102] S22. Perform error detection on the text to be corrected to obtain different types of text errors in the text to be corrected, and set corresponding first labels for different types of text errors; wherein, the first label contains error labels of different types to mark the different types of text errors.

[0103] S23. Correct different types of text errors based on different types of first tags to obtain corrected text; wherein, in the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text.

[0104] S24. Match and score the corrected text based on the preset confusion set; wherein, the preset confusion set includes a character shape confusion set and a character sound confusion set;

[0105] Among them, the character form confusion set refers to the set of word groups that are easily confused with the word groups that need to be corrected in terms of character form, and the character pronunciation confusion set refers to the set of word groups that are easily confused with the word groups that need to be corrected in terms of pronunciation.

[0106] Specifically, after obtaining the corrected text, the corrections in the corrected text are matched and scored based on the context semantics using a preset obfuscation set to obtain the matching score results.

[0107] S25. Adding an adjustment weight to a phrase that meets a preset confusion set matching score.

[0108] The adjustment weight is used to adjust the previously obtained phrase score for the qualifying phrase, so that the qualifying phrase obtains a relatively appropriate phrase score.

[0109] Specifically, after the matching scores are acquired, sorting is performed according to the matching score results, and the sorted results are compared with the preset confusion set matching score, so that the corresponding phrases meeting the preset confusion set matching score can be obtained, and the corresponding adjustment weight is added according to the actual situation, so that the phrases meeting the preset confusion set matching score requirement can obtain an appropriate phrase score.

[0110] S26. Adjusting the phrase score with the adjustment weight and a preset penalty coefficient, and further acquiring the phrase corresponding to the adjusted highest phrase score to determine the final error-corrected text.

[0111] Herein, in order to adjust the phrase score more accurately, it is also necessary to add a preset penalty coefficient, and adjust the adjustment weight by adjusting the preset penalty coefficient.

[0112] Specifically, after the adjustment weight is acquired, in order to reduce errors caused by glyph differences and pronunciation differences, a preset penalty coefficient is set, the adjustment weight is adjusted through the preset penalty coefficient, the phrase corresponding to the highest adjusted phrase score is further acquired, and the final error-corrected text is determined.

[0113] For example: during the correction process, the character at the "[MASK]" position can only be judged according to the context semantics, that is, a contextually coherent result can be generated to solve semantic errors. However, the output phrase is not necessarily the optimal one. For example, filling characters such as "must", "need", "please", "should" at the "[MASK]" position in "potential bidders [MASK] check in time" is reasonable. Considering that text typos are generally caused by similar pronunciation and glyphs, it is obviously more reasonable to output results with similar pronunciation and glyphs from the candidate results. For example, in the above results, "须" (must) is consistent with the original character "需" (need) in pronunciation, and "须" (must) is the correct result.

[0114] Therefore, a pronunciation confusion set and a glyph confusion set are defined for each character, to add the above pronunciation and glyph priors in the correction process. Specifically: 1) Convert the characters into Pinyin, and add homophones (such as "须" and "需") and near-homophones (such as "真" and "增") of each character to the pronunciation confusion set; 2) Disassemble the character into stroke orders, evaluate the similarity between two characters using the minimum edit distance, and add similar glyph characters (such as "需" and "雷") to the glyph confusion set; 3) Merge the pronunciation confusion set and the glyph confusion set.

[0115] Specify a penalty coefficient, 0 < k < 1, and perform correction on the phrase scoring vector output by the error correction model to increase the proportion of each character in the confusion set. The specific calculation is as follows:

[0116]

[0117] where s i ′ is the adjusted weight, k is the penalty coefficient, and s is the |V|-dimensional scoring vector.

[0118] Taking the character "xu" (需) as an example, after the character is replaced with "[MASK]", a dimensional scoring vector is obtained. The confusion set "须,墟,胥,訏,虚,吁,..." is converted into the vocabulary V to obtain a corresponding index set, such as "1152, 2315, 1254, 5231, 3215, ...", where each index value represents the position of the character in the vocabulary, for example, the position of "须" in the vocabulary is "1152". The weight is adjusted by adjusting the penalty coefficient, for example, k=0.75, for the score s of "须" 1152 , the above formula is used to increase its weight. After processing the scores corresponding to all characters in the confusion set, the one with the highest score is selected from the |V|-dimensional scoring vector as the output result.

[0119] In this embodiment, in order to further improve the error correction accuracy, the confusion set is used to increase the proportion of homophones, near-homophones, and similar-shaped characters, and a preset penalty coefficient and adjusted weight are set to adjust the phrase score, so that each phrase has a more accurate phrase score, and then the final error-corrected text is determined according to the highest phrase score, which improves the error correction accuracy.

[0120] In one embodiment, performing matching scoring on the error-corrected text based on the preset confusion set includes: performing matching scoring on the phrases that have been error-corrected in the error-corrected text based on the preset confusion set;

[0121] wherein the matching score of the phonetic type text error is obtained based on the phonetic confusion set, and the matching score of the glyph type text error is obtained based on the glyph confusion set.

[0122] In one embodiment, increasing the adjusted weight for phrases that meet the preset confusion set matching score includes:

[0123] For phrases corresponding to text errors of the phonetic type and glyph type that meet the preset confusion set matching score, the adjusted weight is increased to improve the matching score of the text errors corresponding to the phonetic type and glyph type.

[0124] Please refer to Figure 7 , Figure 7This is a schematic diagram of an embodiment of the text correction system in this application. As shown in the figure, the text correction system 300 includes: an acquisition module 310, an error detection module 320, and a correction module 330. The acquisition module 310 acquires the text to be corrected; the error detection module 320 performs error detection on the text to be corrected to identify different types of text errors present in the text, and assigns corresponding first tags to different types of text errors; wherein the first tags include error tags of different types to mark different types of text errors; the correction module 330 corrects different types of text errors based on the different types of first tags to acquire the corrected text; wherein, during the correction stage, the phrases at the correction points are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text.

[0125] Please see Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device can perform the steps in the above-described text correction method. The electronic device 400 includes: a memory 410 and a processor 420 coupled to the memory. The memory 410 stores at least one computer program. When the at least one computer program is loaded and executed by the processor, it is used to implement the above-described text correction method.

[0126] Please see Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 500 stores at least one program 510, which, when loaded and executed by a processor, is used to implement the above-described text correction method.

[0127] The above scheme improves the accuracy of error correction by classifying the text error types in the text to be corrected and setting corresponding labels, and then correcting errors according to the labels. On the other hand, it further improves the accuracy of error correction by scoring the phrases at the error points according to a preset vocabulary and determining the candidate target text with the highest phrase score as the corrected text.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A text error correction method, characterized in that, The text correction method includes: Get the text to be corrected; Error detection is performed on the text to be corrected to obtain different types of text errors present in the text to be corrected, and a corresponding first label is set for each type of text error; wherein, the first label contains error labels of different types to mark the different types of text errors accordingly; Based on the first tag of different types, the text errors of different types are corrected to obtain the corrected text; wherein, in the correction stage, the phrases at the correction point are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text. After confirming the corrected text, the process also includes: The corrected text is matched and scored based on a preset confusion set; wherein, the preset confusion set includes a character shape confusion set and a character sound confusion set; For phrases that match the preset confusion set score, the weight is adjusted. The word group score is adjusted using the adjusted weight and preset penalty coefficient, and then the candidate target text corresponding to the highest adjusted word group score is obtained to determine the final correction text.

2. The text correction method according to claim 1, characterized in that, The step of performing error detection on the text to be corrected, to obtain different types of text errors present in the text to be corrected, and to assign corresponding first tags to the different types of text errors, includes: The text to be corrected is input into a preset error detection model to obtain different types of text errors present in the text; wherein, the different types of text errors include: a first text error, a second text error, and a third text error; A first label is set for different types of text errors, and a second label is set for normal text; wherein, the first label includes: a first error label for replacement, a second error label for insertion, and a third error label for deletion, and the second label includes a normal label; the first error label is set for the phrase corresponding to the first text error, the second error label is set for the phrase corresponding to the second text error, the third error label is set for the phrase corresponding to the third text error, and the normal label is set for the phrase corresponding to the normal text; The tag sequence of the text to be corrected is constructed based on the first error tag, the second error tag, the third error tag, and the normal tag.

3. The text correction method according to claim 2, characterized in that, After constructing the tag sequence of the text to be corrected, the following steps are also included: The text to be corrected is edited based on the tag sequence to obtain the edited text; wherein, the phrases corresponding to the normal tags are restored to the original text, and the phrases corresponding to the first error tag, the second error tag, and the third error tag are edited into the third tag; The candidate target text is determined by the restored original text and the third tag.

4. The text correction method according to claim 3, characterized in that, The step of scoring the phrases at the error correction points based on a preset vocabulary, and determining the corrected text using the candidate target text corresponding to the highest score, includes: The word groups corresponding to the third tag in the candidate target text are scored based on a preset vocabulary to obtain the word group with the highest score in the preset vocabulary for each third tag; wherein, the preset vocabulary is the vocabulary used by the error correction model to make contextual semantic judgments. Replace the corresponding phrase in the candidate target text with the phrase that received the highest score to obtain the corrected text.

5. The text correction method according to claim 3, characterized in that, The step of correcting different types of text errors based on the first tag of different types to obtain corrected text includes: Replace the text of the first error label with the first error text. Insert text into the second error text corresponding to the second error label; Delete the text corresponding to the third error label; Based on the text replacement, text insertion, and text deletion performed on the text errors, the corrected text is determined.

6. The text correction method according to claim 2, characterized in that, The first tag is generated based on a preset minimum edit distance; wherein the minimum edit distance is the minimum number of edits required to convert the text to be corrected into corrected text through editing operations.

7. The text correction method according to claim 1, characterized in that, The matching and scoring of the corrected text based on a preset obfuscation set includes: The corrected text is matched and scored based on a preset confusion set; wherein, the matching score of the pronunciation text error is obtained based on the pronunciation confusion set, and the matching score of the character shape text error is obtained based on the character shape confusion set. The method of adjusting the weight of phrases that match the preset confusion set score includes: For word phrases with text errors in the phonetic and glyph categories that meet the preset confusion set matching score, the adjustment weight is increased to improve the matching score of the text errors in the phonetic and glyph categories.

8. A text correction system, characterized in that, include: The acquisition module is used to acquire the text to be corrected. An error detection module is used to perform error detection on the text to be corrected, to obtain different types of text errors in the text to be corrected, and to set corresponding first tags for different types of text errors; wherein, the first tags include error tags of different types to mark different types of text errors. The error correction module corrects different types of text errors based on the first tag of different types to obtain the corrected text; wherein, in the error correction stage, the phrases at the error correction point are scored based on a preset vocabulary, and the candidate target text corresponding to the highest phrase score is determined as the corrected text. The error correction module is also used to match and score the corrected text based on a preset confusion set; wherein, the preset confusion set includes a character shape confusion set and a character sound confusion set; For phrases that match the preset confusion set score, the weight is adjusted. The word group score is adjusted using the adjusted weight and preset penalty coefficient, and then the candidate target text corresponding to the highest adjusted word group score is obtained to determine the final correction text.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the text correction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium has at least one program that, when loaded and executed by a processor, is used to implement the text correction method as described in any one of claims 1-7.

Images