Text processing method and device, electronic equipment and readable storage medium

Abstract

This application provides a text processing method, apparatus, electronic device, and readable storage medium, relating to the field of computer technology. In this application, the embodiment can receive target text and perform text rule detection and model detection on the target text. Furthermore, based on the results of rule detection and model detection, the embodiment can perform error correction processing on the target text to determine the corrected text corresponding to the target text. Since this application performs both text rule detection and model detection on the target text during the error correction process, it can correct the target text from multiple dimensions, achieving comprehensive error correction.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a text processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, when users input text through electronic devices such as computers, the electronic devices can often correct errors in the text input, that is, check, mark, or correct errors in the text.

[0003] In related technologies, text correction is achieved through models. However, the models process text in a relatively singular dimension, meaning they cannot comprehensively detect the various types of errors that appear in the text. Summary of the Invention

[0004] In view of this, embodiments of this application provide a text processing method, apparatus, electronic device, and readable storage medium to correct errors in target text from multiple dimensions, thereby achieving comprehensive error correction of the target text.

[0005] Firstly, a text processing method is provided, the method comprising: Obtain the target text; Perform text rule detection on the target text and determine the rule detection result; The target text is input into the text detection model to determine the model detection result; and Based on the detection results of the rules and the detection results of the model, the target text is corrected to determine the corrected text corresponding to the target text; The rule detection results include duplicate field detection results; The step of performing text rule detection on the target text and determining the rule detection result includes: The target text is subjected to duplicate field detection according to the pre-set duplicate field detection rules in order to determine the duplicate field detection results; The rule-based detection results include common word detection results; The step of performing text rule detection on the target text and determining the rule detection result includes: The target text is subjected to common word detection based on a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results. The step of performing error correction processing on the target text based on the rule detection results and the model detection results, and determining the error-corrected text corresponding to the target text, includes: Based on the rule detection results and the model detection results, at least one error correction type label is determined; Based on the error correction rules corresponding to each of the error correction type labels, an error correction candidate set is determined for each of the error correction type labels, wherein the error correction candidate set includes candidate characters or candidate words for correcting the target text; and The target text is corrected according to each of the error correction candidate sets to determine the error correction text corresponding to the target text. The target text is input into a pre-trained Chinese natural language training model. The Chinese natural language training model performs model detection on the target text based on word granularity, and the model detection result is determined.

[0006] In some embodiments, the step of performing error correction processing on the target text according to each of the error correction candidate sets, and determining the error-corrected text corresponding to the target text, includes: Based on a pre-set language model and each of the error correction candidate sets, simulated error correction is performed on the target text to determine the scores corresponding to candidate characters or candidate words in each of the error correction candidate sets.

[0007] Based on the score, the target character or target word corresponding to each of the error correction candidate sets is determined.

[0008] Based on the target characters or target words corresponding to each of the error correction candidate sets, the target text is corrected to determine the error correction text corresponding to the target text.

[0009] In some embodiments, the error correction type label includes one or more of the following: pronunciation similarity label, character shape similarity label, position inversion label, extra character label, missing character label, and particle error label.

[0010] In some embodiments, the method further includes: The rule detection results and the model detection results are verified based on a pre-set result correction dictionary.

[0011] In response to the rule detection result and / or the model detection result matching a character or word in the result correction dictionary, the rule detection result and / or the model detection result are updated.

[0012] Secondly, a text processing apparatus is provided, the apparatus comprising: The target text acquisition module is configured to acquire the target text. The rule detection module is configured to perform text rule detection on the target text and determine the rule detection result; The model detection module is configured to input the target text into a text detection model and determine the model detection result; and The error correction module is configured to perform error correction processing on the target text based on the detection results of the rules and the detection results of the model, and determine the error-corrected text corresponding to the target text. The rule detection results include duplicate field detection results; The rule detection module is also configured to: The target text is subjected to duplicate field detection according to the pre-set duplicate field detection rules in order to determine the duplicate field detection results; The rule-based detection results include common word detection results; The rule detection module is also configured to: The target text is subjected to common word detection based on a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results. The error correction module is further configured to include: Based on the rule detection results and the model detection results, at least one error correction type label is determined; Based on the error correction rules corresponding to each of the error correction type labels, an error correction candidate set is determined for each of the error correction type labels, wherein the error correction candidate set includes candidate characters or candidate words for correcting the target text; and The target text is corrected according to each of the error correction candidate sets to determine the error correction text corresponding to the target text. The text detection model is configured to input the target text into a pre-trained Chinese natural language training model, and to perform model detection on the target text based on word granularity through the Chinese natural language training model to determine the model detection result.

[0013] In some embodiments, the error correction module is specifically configured to perform: Based on a pre-set language model and each of the error correction candidate sets, simulated error correction is performed on the target text to determine the scores corresponding to candidate characters or candidate words in each of the error correction candidate sets.

[0014] Based on the score, the target character or target word corresponding to each of the error correction candidate sets is determined.

[0015] Based on the target characters or target words corresponding to each of the error correction candidate sets, the target text is corrected to determine the error correction text corresponding to the target text.

[0016] In some embodiments, the error correction type label includes one or more of the following: pronunciation similarity label, character shape similarity label, position inversion label, extra character label, missing character label, and particle error label.

[0017] In some embodiments, the apparatus further includes: The verification module is configured to verify the rule detection results and the model detection results by executing a pre-set result correction dictionary.

[0018] The update module is configured to update the rule detection results and / or the model detection results in response to the rule detection results and / or the model detection results hitting a character or word in the result correction dictionary.

[0019] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0021] In this embodiment, the target text can be received and subjected to text rule detection and model detection. Furthermore, based on the results of rule detection and model detection, the target text can be corrected to determine the corrected text. Since this embodiment performs both text rule detection and model detection on the target text during the correction process, it can correct the target text from multiple dimensions, achieving comprehensive correction. Attached Figure Description

[0022] The above and other objects, features and advantages of the present application will become clearer from the following description of embodiments of the present application with reference to the accompanying drawings, in which: Figure 1 This is a flowchart illustrating the text processing method according to an embodiment of this application; Figure 2 This is a flowchart of the text processing method according to an embodiment of this application; Figure 3 This is a flowchart of another text processing method according to an embodiment of this application; Figure 4 This is a flowchart of another text processing method according to an embodiment of this application; Figure 5 This is a flowchart of another text processing method according to an embodiment of this application; Figure 6 This is a flowchart of another text processing method according to an embodiment of this application; Figure 7This is a flowchart of another text processing method according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the text processing device according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0024] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0025] Unless the context explicitly requires it, words such as "including" or "contains" in the instruction manual should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0026] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0027] Currently, with the development of computer technology, most users choose to use electronic devices such as computers to input text. When users input text through electronic devices, the electronic devices can often correct errors in the text, that is, check, mark or correct errors in the text.

[0028] In related technologies, text correction is achieved through models. For example, the Bidirectional Encoder Representations from Transformers (BERT) model can be used for text correction. However, these models have a relatively limited scope in processing text, meaning they cannot comprehensively detect the various types of errors present in the text. Taking the BERT model as an example, when using it for text correction, related technologies often only address pronunciation or glyph issues, failing to handle other types of text errors simultaneously. This can lead to problems such as accumulated text error, missed detections, and false positives. Therefore, achieving comprehensive text correction is a pressing issue that needs to be addressed.

[0029] To address the aforementioned issues, this application provides a text processing method applicable to electronic devices. The electronic device can be a terminal or a server. The terminal can be a smartphone, tablet, or personal computer (PC), etc. The server can be a single server, a server cluster configured in a distributed manner, or a cloud server.

[0030] like Figure 1 As shown, user 11 can input target text 12 through external input devices such as a mouse or keyboard, or through the input unit of electronic device 13 itself (such as a keyboard or touch screen). Correspondingly, electronic device 13 can receive target text 12 and, based on the aforementioned text processing method, perform text rule detection and model detection on target text 12. Furthermore, electronic device 13 can perform error correction processing on target text 12 based on the results of rule detection and model detection, thereby determining the corresponding error-corrected text 14. Since this embodiment performs both text rule detection and model detection on target text 12 during the error correction process, it can correct target text 12 from multiple dimensions, achieving comprehensive error correction of target text 12.

[0031] Specifically, such as Figure 2 As shown, the above text processing method may include the following steps: In step S100, the target text is obtained.

[0032] The target text may include multiple characters, words, and sentences.

[0033] In one optional implementation, after obtaining the target text, the present application embodiment can preprocess the target text to improve the efficiency of text correction.

[0034] The preprocessing may include noise filtering, text segmentation, space detection, and symbol detection. Specifically, the target text may contain meaningless text noise such as garbled characters. In this case, the embodiments of this application can identify and delete the aforementioned text noise, thereby preserving the valid information in the target text.

[0035] This application embodiment can also perform sentence-level segmentation of the target text based on punctuation marks in the target text, in order to annotate, record, and return the set of sentences corresponding to the target text. This application embodiment can also perform word recognition on the target text, thereby performing word-level segmentation of the target text, in order to annotate, record, and return the set of words corresponding to the target text.

[0036] This application embodiment can also perform space detection or symbol detection on the target text, thereby recording or deleting spaces or symbols in the target text.

[0037] Therefore, by preprocessing the target text, the embodiments of this application can remove noise from the target text, segment the target text, and detect spaces or symbols in the target text, thereby improving the efficiency of subsequent error correction for the target text.

[0038] In step S200, text rule detection is performed on the target text, and the rule detection result is determined.

[0039] Text rules can be used to characterize the writing rules corresponding to the text. For example, errors in text rules can include word order reversal, extra or missing characters, and incorrect use of auxiliary words. This application embodiment can detect the above-mentioned text rules to determine the text rule problems appearing in the target text, thereby determining the rule detection result. The rule detection result can include the field where the text rule error occurs, the position of that field in the target text, and the error type marker corresponding to that field.

[0040] It should be noted that there is no fixed execution order between step S200 and step S300 in the embodiments of this application. That is, after step S100 is executed, step S200 can be executed first and then step S300 can be executed, or step S300 can be executed first and then step S200 can be executed, or step S200 and step S300 can be executed simultaneously.

[0041] In one optional implementation, the rule detection results may include duplicate field detection results. These duplicate field detection results may include fields that appear repeatedly in the target text, the position of the repeated field in the target text, and the error type flag corresponding to the repeated field.

[0042] Specifically, step S200 above may include the following steps: In step S210, duplicate field detection is performed on the target text according to the pre-set duplicate field detection rules to determine the duplicate field detection results.

[0043] In this embodiment of the application, if the user makes a mistake when inputting text, the target text may contain consecutive identical fields. In this case, this embodiment of the application can mark the repeated fields according to the pre-set duplicate field detection rules, thereby determining the duplicate field detection result.

[0044] Specifically, embodiments of this application can perform duplicate field detection on target text based on the number of characters in the duplicate fields. For example, such as Figure 3 As shown, the process of performing duplicate field detection on target text may include the following steps: In step S31, the duplicate field portions in the target text are identified.

[0045] The repeated field part is used to represent the repeated part in consecutive identical fields. For example, in the field "slowly", the repeated field part is "slow", and in the field "hello hello hello", the repeated field part is "hello".

[0046] In step S32, it is determined whether the number of characters in the repeated field is greater than 1. If the number of characters in the repeated field is greater than 1, then step S33 is executed. If the number of characters in the repeated field is less than or equal to 1, then step S31 is executed.

[0047] In practical applications, since there are words composed of multiple single characters (such as slowly, gradually, or often), classifying these words as duplicate fields would lead to a large number of false positives. Therefore, this embodiment of the application can exclude words composed of single characters in the duplicate field detection rules, that is, filter out the part of the duplicate field with a number of characters less than or equal to 1, thereby improving the accuracy of duplicate field detection.

[0048] In step S33, the duplicate field detection result is determined.

[0049] The duplicate field detection results can include the field that appears repeatedly in the target text, the position of the repeated field in the target text, and the error type flag corresponding to the repeated field.

[0050] By performing duplicate field detection on the target text, the embodiments of this application can increase the detection range of the target text, thereby achieving comprehensive error correction of the target text.

[0051] In one optional implementation, the rule detection results may include common word detection results. These common word detection results may include uncommon words appearing in the target text, the corresponding position of the uncommon word in the target text, and the error type marker corresponding to the uncommon word.

[0052] Specifically, step S200 above may include the following steps: In step S220, common words are detected in the target text according to a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results.

[0053] In this embodiment, the target text can be traversed and each word in the target text can be filtered according to a pre-set common word list, and words not included in the common word list can be identified as uncommon words.

[0054] For example, in embodiments of this application, a common word list of non-entity words can be pre-set for common word detection. Entity words can be used to represent words with entity information (e.g., nouns), and correspondingly, non-entity words can be used to represent words without entity information (e.g., verbs).

[0055] It should be noted that, due to the large number and rapid update speed of entity words such as nouns, common word detection for entity words often results in false positives and incorrect detections. For example, if the common word list is not updated in a timely manner for new terms arising from new things, incorrect detections will occur. Since non-entity words do not have entity information, their number is relatively stable and their update frequency is low. Therefore, in this application embodiment, a common word list for non-entity words can be constructed for common word detection.

[0056] Furthermore, in this embodiment of the application, when performing common word detection based on a common word list of non-entity words, the target text can first be segmented to identify each non-entity word in the target text. Then, this embodiment of the application can filter each non-entity word in the target text according to the common word list of non-entity words to identify uncommon words among them and determine the common word detection results. The common word detection results may include uncommon words appearing in the target text, the corresponding position of the uncommon word in the target text, and the error type marker corresponding to the uncommon word.

[0057] By performing common word detection on the target text, the embodiments of this application can increase the detection range of the target text, thereby achieving comprehensive error correction of the target text.

[0058] In step S300, the target text is input into the text detection model, and the model detection result is determined.

[0059] In particular, if model detection is performed on Chinese target text, the embodiments of this application can perform model detection on the target text based on the pre-trained Chinese natural language training model Mac BERT (MLM as correction BERT).

[0060] The MacBERT model is pre-trained using a full-word masking method. Therefore, compared to the BERT model, the MacBERT model performs text detection at the word level in practical applications. Furthermore, since most words in Chinese text are composed of multiple characters, the MacBERT model's word-level detection capability allows it to detect not only typos but also extra or missing characters, word order errors, and particle mistakes at the word level.

[0061] In step S400, the target text is corrected based on the rule detection results and the model detection results to determine the corrected text corresponding to the target text.

[0062] In this embodiment, the location of errors in the target text can be located based on rule-based detection results and model detection results. Furthermore, this embodiment can directly correct the target text based on the language model to determine the corresponding correction text. Alternatively, this embodiment can first determine one or more candidate fields from the various locations where errors occur in the target text, and then select the appropriate field from these candidate fields to correct the target text, thereby determining the corresponding correction text.

[0063] Therefore, embodiments of this application can receive target text and perform text rule detection and model detection on the target text. Furthermore, embodiments of this application can perform error correction processing on the target text based on the results of rule detection and model detection, thereby determining the corrected text corresponding to the target text. Since embodiments of this application perform both text rule detection and model detection on the target text during the error correction process, they can correct the target text from multiple dimensions, achieving comprehensive error correction of the target text.

[0064] In other words, since both text rule detection and model detection have certain limitations, this application embodiment can achieve complementarity between text rule detection and model detection by simultaneously performing text rule detection and model detection on the target text. This allows for error correction of the target text from multiple dimensions, achieving comprehensive error correction of the target text.

[0065] In one alternative implementation, such as Figure 4 As shown, step S400 above may include the following steps: In step S410, at least one error correction type label is determined based on the rule detection results and the model detection results.

[0066] Since the rule detection results and model detection results can be used to mark and locate erroneous fields in the target text, this application embodiment can determine the error correction type label corresponding to each result after determining the rule detection results and model detection results, so as to achieve the classification of each result.

[0067] Specifically, regarding the rule detection results, since rule detection is based on relatively fixed writing rules to detect target text, this application embodiment can pre-set different error correction type labels corresponding to rule detection. For example, the above-mentioned duplicate field detection result can correspond to the error correction type label of "multiple characters", and the above-mentioned common word detection result can correspond to the error correction type label of "common word error".

[0068] Regarding the model detection results, embodiments of this application can add error correction type labels to the training set during the training process of the model used for model detection, and train the model based on this training set. This allows the model used for model detection, after training, to output text errors along with the corresponding error correction type labels, ensuring that the model detection results include the error correction type labels.

[0069] In step S420, the error correction candidate set corresponding to each error correction type label is determined according to the error correction rules corresponding to each error correction type label.

[0070] The error correction candidate set includes candidate characters or candidate words for correcting the target text. A single error correction candidate set may include multiple candidate characters or multiple candidate words, and a single error correction candidate set may include both candidate characters and candidate words.

[0071] Furthermore, in the embodiments of this application, different error correction type labels generally correspond to different error types. Therefore, the embodiments of this application can set different error correction rules for different error correction type labels (i.e. different error types) to achieve accurate error correction of the target text.

[0072] In one alternative implementation, the error correction type label may include one or more of the following: pronunciation similarity label, character shape similarity label, position inversion label, extra character label, missing character label, and particle error label.

[0073] Among them, pronunciation similarity labels and character shape similarity labels can be error correction type labels in the model detection results, while position inversion labels, extra character labels, missing character labels, and auxiliary word error labels can be error correction type labels corresponding to the rule detection results.

[0074] Specifically, in the training process of the model used for model detection, this application adds pronunciation similarity labels and character shape similarity labels to the training set, and trains the model based on this training set. This allows the model used for model detection to output text errors along with corresponding error correction type labels after training, ensuring that the model detection results include error correction type labels.

[0075] Meanwhile, embodiments of this application can set different text rule detection processes for position inversion tags, extra-character tags, missing-character tags, and particle error tags to determine the rule detection results including position inversion tags, extra-character tags, missing-character tags, or particle error tags. For example, embodiments of this application can identify repeated fields in the target text and determine extra-character tags through the above-mentioned duplicate field detection. Embodiments of this application can also identify the use of incorrect particles in the target text and determine particle error tags through the above-mentioned common word detection. Embodiments of this application can also identify cases of reversed character order, reversed word order, or missing characters in the target text through grammar detection and determine position inversion tags or missing-character tags.

[0076] Furthermore, in this application embodiment, different error correction rules can be set for each of the above-mentioned error correction type tags.

[0077] For pronunciation similarity tags, this application embodiment can pre-construct and maintain a pronunciation similarity confusion dataset based on similar pronunciations such as initials, finals, and tones within Chinese Pinyin. Once this application embodiment determines a pronunciation similarity tag, it can determine candidate characters or words in the pronunciation similar to the field in the pronunciation similarity confusion dataset based on the fields corresponding to the pronunciation similarity tags, thereby determining the error correction candidate set corresponding to the pronunciation similarity tag. Alternatively, the aforementioned pronunciation similarity confusion dataset can also be a pronunciation similarity confusion dataset pre-constructed and maintained based on the pronunciation of other languages ​​(e.g., English phonetics).

[0078] For glyph-similar labels, the embodiments of the present application may pre-construct and maintain a glyph-similar confusion dataset according to Chinese glyphs. After the embodiments of the present application determine the glyph-similar label, candidate characters or candidate words that are glyph-similar to the target field in the glyph-similar confusion dataset can be determined according to the glyph-similar confusion dataset and the field corresponding to the glyph-similar label, so as to determine the error correction candidate set corresponding to the glyph-similar label. In addition, the above glyph-similar confusion dataset may also be a glyph-similar confusion dataset pre-constructed and maintained by the embodiments of the present application according to glyphs of other languages (for example, letter composition of English words).

[0079] For position-reversed labels, after determining the position-reversed label, the embodiments of the present application may perform full permutation on all characters in the field corresponding to the position-reversed label, and use each permutation result after full permutation as the error correction candidate set corresponding to the position-reversed label.

[0080] For extra-character labels, after determining the extra-character label, the embodiments of the present application may determine the field corresponding to the extra-character label, and randomly delete one or more characters from the field, so as to determine candidate characters or candidate words. After determining a plurality of candidate characters or candidate words, the embodiments of the present application may use each candidate character or candidate word, and the field corresponding to the extra-character label (that is, the original field without any deletion) as the error correction candidate set corresponding to the extra-character label.

[0081] For missing-character labels, after determining the missing-character label, the embodiments of the present application may determine the field corresponding to the missing-character label, and perform predictive word filling on the field through a preset language model (e.g., BERT model), so as to determine a plurality of candidate characters or candidate words. After determining a plurality of candidate characters or candidate words, the embodiments of the present application may use each candidate character or candidate word, and the field corresponding to the missing-character label (that is, the original field without adding any characters or words) as the error correction candidate set corresponding to the missing-character label.

[0082] For auxiliary word error labels, the embodiments of the present application may use a preset auxiliary word set (for example, the preset auxiliary word set may include auxiliary words such as "de" (the structural auxiliary word used before an attribute), "de" (the structural auxiliary word used before an adverbial), and "de" (the structural auxiliary word used after a complement)) as the error correction candidate set corresponding to the auxiliary word error label.

[0083] According to the embodiments of the present application, since different error correction type labels generally correspond to different error types, the embodiments of the present application can set different error correction rules for different error correction type labels (i.e., different error types), so as to determine the error correction candidate sets respectively corresponding to each error correction type label, thereby achieving accurate error correction for the target text.

[0084] In step S430, error correction processing is performed on the target text according to each error correction candidate set, and the error-corrected text corresponding to the target text is determined.

[0085] In this embodiment of the application, a target character or word can be selected from each candidate character or word in the error correction candidate set, and the target character or word can be used to replace the corresponding field in the target text, thereby realizing the error correction processing of the target text and determining the error correction text corresponding to the target text.

[0086] In one alternative implementation, such as Figure 5 As shown, step S430 may include the following steps: In step S431, the target text is simulated for error correction based on the pre-set language model and each error correction candidate set, so as to determine the score corresponding to the candidate character or candidate word in each error correction candidate set.

[0087] In this embodiment, if the target text is Chinese text, each candidate character or word from the error correction candidate set can be used to replace the corresponding field in the target text. Then, based on a pre-set Chinese language model (N-Gram), the replaced target text is simulated to determine the score output by the N-Gram model. Since the N-Gram model calculates scores based on word frequency statistics and cannot understand semantic information, this embodiment can use the score output by the N-Gram model as a basis for coarse sorting, thereby achieving coarse sorting of the sentences corresponding to candidate characters or words in each error correction candidate set.

[0088] Furthermore, embodiments of this application can calculate the semantic perplexity (PPL) of each statement in the above coarse sorting, thereby determining the final score corresponding to the candidate character or candidate word in each error correction candidate set based on the PPL.

[0089] In step S432, the target character or target word corresponding to each error correction candidate set is determined according to the score.

[0090] Specifically, in the embodiments of this application, the candidate character or candidate word with the highest score can be used as the target character or target word.

[0091] In step S433, the target text is corrected according to the target character or target word corresponding to each error correction candidate set, so as to determine the error correction text corresponding to the target text.

[0092] Therefore, embodiments of this application can receive target text and perform text rule detection and model detection on the target text. Furthermore, embodiments of this application can determine various error correction type labels based on the results of rule detection and model detection, and then perform error correction processing on the target text according to the error correction rules corresponding to each error correction type label, thereby determining the corrected text corresponding to the target text. Since embodiments of this application perform both text rule detection and model detection on the target text during the error correction process, embodiments of this application can correct the target text from multiple dimensions, achieving comprehensive error correction of the target text.

[0093] In an optional implementation, the embodiments of this application can further correct the rule detection results and model detection results, specifically, as follows: Figure 6 As shown, the process may include the following steps: In step S61, the rule detection results and model detection results are verified according to the pre-set result correction dictionary.

[0094] The result correction dictionary may include fixed expressions, new words, hot words, confusing phrases, date formats, and pre-defined writing rules.

[0095] In step S62, in response to the word or phrase in the rule detection result and / or model detection result hit result correction dictionary, the rule detection result and / or model detection result are updated.

[0096] Specifically, in this application embodiment, the characters or words in the hit result correction dictionary can be removed from the rule detection results and / or model detection results to update the rule detection results and / or model detection results.

[0097] In the process of correcting target text in this application embodiment, if new words, hot words, confusing phrases, or other words that are easily identified as errors appear in the target text, this application embodiment may identify such words as erroneous words. In this case, this application embodiment can verify the rule detection results and model detection results based on a pre-set result correction dictionary to avoid the above words being identified as erroneous words.

[0098] In addition, since the result correction dictionary can also include date formats and predefined writing rules, when the target text contains misdetected dates and text with specific writing styles (such as popular online terms), the embodiments of this application can correct the above situation through the result correction dictionary.

[0099] Therefore, after the target text is comprehensively corrected from multiple dimensions in the embodiments of this application, the detection results can be corrected by the result correction dictionary, thereby further improving the accuracy of text correction.

[0100] In an optional implementation, this application embodiment can further perform post-processing on the error-corrected text. Specifically, this application embodiment can set up a text correction database based on frequently occurring false detections in history, and verify and correct the error-corrected text according to the text correction database, such as revoking incorrectly modified fields in the error-corrected text or marking incorrectly modified fields in the error-corrected text, thereby further improving the accuracy of text error correction.

[0101] In conjunction with the above-described embodiments, this application embodiment can receive target text and preprocess it. Further, this application embodiment can perform text rule detection and model detection on the target text, and correct the rule detection results and / or model detection results according to a result correction dictionary. Then, based on the corrected rule detection results and model detection results, the target text is corrected to determine the corrected text corresponding to the target text. Furthermore, this application embodiment can also perform post-processing on the corrected text to further improve the accuracy of text correction.

[0102] For example, such as Figure 7 As shown, the above process may include the following steps: In step S71, the target text is obtained.

[0103] In step S72, the target text is preprocessed.

[0104] The preprocessing process may include noise filtering, text segmentation, space detection, and symbol detection.

[0105] In step S73, duplicate field detection is performed on the target text, and the duplicate field detection result is determined.

[0106] The duplicate field detection results can include the field that appears repeatedly in the target text, the position of the repeated field in the target text, and the error type flag corresponding to the repeated field.

[0107] In step S74, common word detection is performed on the target text to determine the common word detection results.

[0108] The common word detection results can include uncommon words appearing in the target text, the corresponding position of the uncommon word in the target text, and the error type marker corresponding to the uncommon word.

[0109] In step S75, model detection is performed on the target text to determine the model detection result.

[0110] It should be noted that there is no fixed execution order for the above steps S73, S74 and S75. That is to say, in the embodiments of this application, steps S73, S74 and S75 can be executed in a certain order, or steps S73, S74 and S75 can be executed simultaneously.

[0111] in addition, Figure 7 This is merely one example of an embodiment of this application. In practical applications, embodiments of this application may also include other types of detection (such as syntax detection).

[0112] In step S76, the detection results are corrected.

[0113] In this embodiment of the application, the results of duplicate field detection, common word detection and model detection can be corrected based on the result correction dictionary.

[0114] In step S77, the error correction candidate set is recalled.

[0115] Among them, the recall and correction candidate set is determined according to the correction rules corresponding to each correction type label.

[0116] In step S78, the error correction candidate set is sorted to determine the error correction text.

[0117] In this embodiment, the target text can be simulated for error correction based on a language model and an error correction candidate set to determine the score corresponding to the candidate character or word in each error correction candidate set. Then, based on the score, the target character or word corresponding to each error correction candidate set is determined, and the target text is corrected based on the target character or word corresponding to each error correction candidate set to determine the corrected text corresponding to the target text.

[0118] In step S79, the error correction text is post-processed.

[0119] In this embodiment of the application, a text correction database can be set up based on historically frequent false detections, and the text to be corrected can be verified and corrected based on the text correction database.

[0120] Through the embodiments of this application, since the target text is corrected by performing both text rule detection and model detection, the target text can be corrected from multiple dimensions, thus achieving comprehensive correction of the target text.

[0121] Based on the same technical concept, embodiments of this application also provide a text processing device, such as... Figure 8As shown, the device includes: a target text acquisition module 81, a rule detection module 82, a model detection module 83, and an error correction module 84.

[0122] The target text acquisition module 81 is configured to acquire the target text.

[0123] The rule detection module 82 is configured to perform text rule detection on the target text and determine the rule detection result.

[0124] The model detection module 83 is configured to input the target text into a text detection model and determine the model detection result.

[0125] The error correction module 84 is configured to perform error correction processing on the target text based on the detection results of the rules and the detection results of the model, and determine the error correction text corresponding to the target text.

[0126] In some embodiments, the rule detection results include duplicate field detection results.

[0127] The rule detection module 82 is specifically configured to execute: The target text is subjected to duplicate field detection according to the pre-set duplicate field detection rules in order to determine the duplicate field detection results.

[0128] In some embodiments, the rule detection results include common word detection results.

[0129] The rule detection module 82 is specifically configured to execute: The target text is subjected to common word detection based on a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results.

[0130] In some embodiments, the error correction module 84 is specifically configured to perform: Based on the rule detection results and the model detection results, at least one error correction type label is determined.

[0131] Based on the error correction rules corresponding to each of the error correction type labels, an error correction candidate set corresponding to each of the error correction type labels is determined. The error correction candidate set includes candidate characters or candidate words for correcting the target text.

[0132] The target text is corrected according to each of the error correction candidate sets to determine the corrected text corresponding to the target text.

[0133] In some embodiments, the error correction module 84 is specifically configured to perform: Based on a pre-set language model and each of the error correction candidate sets, simulated error correction is performed on the target text to determine the scores corresponding to candidate characters or candidate words in each of the error correction candidate sets.

[0134] Based on the score, the target character or target word corresponding to each of the error correction candidate sets is determined.

[0135] Based on the target characters or target words corresponding to each of the error correction candidate sets, the target text is corrected to determine the error correction text corresponding to the target text.

[0136] In some embodiments, the error correction type label includes one or more of the following: pronunciation similarity label, character shape similarity label, position inversion label, extra character label, missing character label, and particle error label.

[0137] In some embodiments, the apparatus further includes: The verification module is configured to verify the rule detection results and the model detection results by executing a pre-set result correction dictionary.

[0138] The update module is configured to update the rule detection results and / or the model detection results in response to the rule detection results and / or the model detection results hitting a character or word in the result correction dictionary.

[0139] In this embodiment, the target text can be received and subjected to text rule detection and model detection. Furthermore, based on the results of rule detection and model detection, the target text can be corrected to determine the corrected text. Since this embodiment performs both text rule detection and model detection on the target text during the correction process, it can correct the target text from multiple dimensions, achieving comprehensive correction.

[0140] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of this application. For example... Figure 9 As shown, Figure 9The illustrated electronic device is a general-purpose device, comprising a general-purpose computer hardware architecture, including at least a processor 91 and a memory 92. The processor 91 and memory 92 are connected via a bus 93. The memory 92 is adapted to store instructions or programs executable by the processor 91. The processor 91 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 91 executes the instructions stored in the memory 92, thereby performing the method flow described in the embodiments of this application as above to process data and control other devices. The bus 93 connects the aforementioned components together, and also connects these components to a display controller 94, a display device, and an input / output (I / O) device 95. The input / output (I / O) device 95 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 95 is connected to the system via an input / output (I / O) controller 96.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0143] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0144] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0145] Another embodiment of this application relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0146] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0147] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A text processing method, characterized in that, The method includes: Obtain the target text; Perform text rule detection on the target text and determine the rule detection result; The target text is input into the text detection model to determine the model detection result; and Based on the detection results of the rules and the detection results of the model, the target text is corrected to determine the corrected text corresponding to the target text; The rule detection results include duplicate field detection results; The step of performing text rule detection on the target text and determining the rule detection result includes: The target text is subjected to duplicate field detection according to the pre-set duplicate field detection rules in order to determine the duplicate field detection results; The rule-based detection results include common word detection results; The step of performing text rule detection on the target text and determining the rule detection result includes: The target text is subjected to common word detection based on a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results. The step of performing error correction processing on the target text based on the rule detection results and the model detection results, and determining the error-corrected text corresponding to the target text, includes: Based on the rule detection results and the model detection results, at least one error correction type label is determined; Based on the error correction rules corresponding to each of the error correction type labels, an error correction candidate set is determined for each of the error correction type labels, wherein the error correction candidate set includes candidate characters or candidate words for correcting the target text; and The target text is corrected according to each of the error correction candidate sets to determine the error correction text corresponding to the target text. The target text is input into a pre-trained Chinese natural language training model. The Chinese natural language training model performs model detection on the target text based on word granularity, and the model detection result is determined.

2. The method according to claim 1, characterized in that, The step of performing error correction processing on the target text according to each of the error correction candidate sets and determining the error-corrected text corresponding to the target text includes: Based on a pre-set language model and each of the error correction candidate sets, simulated error correction is performed on the target text to determine the scores corresponding to candidate characters or candidate words in each of the error correction candidate sets; Based on the score, the target character or target word corresponding to each of the error correction candidate sets is determined; and Based on the target characters or target words corresponding to each of the error correction candidate sets, the target text is corrected to determine the error correction text corresponding to the target text.

3. The method according to claim 1 or 2, characterized in that, The error correction type labels include one or more of the following: similar pronunciation labels, similar character shape labels, reversed position labels, extra character labels, missing character labels, and particle error labels.

4. The method according to claim 1, characterized in that, The method further includes: The rule-based detection results and the model-based detection results are verified based on a pre-set result correction dictionary; and In response to the rule detection result and / or the model detection result matching a character or word in the result correction dictionary, the rule detection result and / or the model detection result are updated.

5. A text processing device, characterized in that, The device includes: The target text acquisition module is configured to acquire the target text. The rule detection module is configured to perform text rule detection on the target text and determine the rule detection result; The model detection module is configured to input the target text into a text detection model and determine the model detection result; and The error correction module is configured to perform error correction processing on the target text based on the detection results of the rules and the detection results of the model, and determine the error-corrected text corresponding to the target text. The rule detection results include duplicate field detection results; The rule detection module is also configured to: The target text is subjected to duplicate field detection according to the pre-set duplicate field detection rules in order to determine the duplicate field detection results; The rule-based detection results include common word detection results; The rule detection module is also configured to: The target text is subjected to common word detection based on a pre-set common word list, and uncommon words in the target text are marked to determine the common word detection results. The error correction module is further configured to include: Based on the rule detection results and the model detection results, at least one error correction type label is determined; Based on the error correction rules corresponding to each of the error correction type labels, an error correction candidate set is determined for each of the error correction type labels, wherein the error correction candidate set includes candidate characters or candidate words for correcting the target text; and The target text is corrected according to each of the error correction candidate sets to determine the error correction text corresponding to the target text. The text detection model is configured to input the target text into a pre-trained Chinese natural language training model, and to perform model detection on the target text at the word granularity through the Chinese natural language training model to determine the model detection result.

6. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-4.

Images