Chinese text correction method and device, electronic equipment and readable storage medium

Abstract

The application relates to the technical field of natural language processing, and provides a Chinese text error correction method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-corrected Chinese text, and performing error position detection on the to-be-corrected Chinese text to obtain a detection result; performing error type classification on the detection result; after determining the error type corresponding to the detection result, the error type corresponding to the detection result can be corrected to obtain a corrected detection result, so that the accuracy of the detection result is improved; furthermore, for the error type corresponding to the corrected detection result, a first priority error correction model corresponding to the error type is found in a pre-stored error correction model, and the corrected detection result is corrected based on the first priority error correction model to obtain an error correction result of the to-be-corrected Chinese text. In this way, the error correction can be performed in a targeted manner according to the error type of the to-be-corrected Chinese text, the calculation amount is reduced, and the processing rate and error correction accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a Chinese text error correction method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Chinese text correction is an important technology used to check for grammatical or semantic errors in Chinese text and to automatically correct any errors. It is in great demand in both human input and machine recognition fields. Therefore, Chinese text correction has received widespread attention from various industries as a practical technology, and there is an urgent need for more mature correction technologies to be applied in various industries and in daily life and production.

[0003] In existing technologies, an end-to-end neural network-based translation model can be used to transform the input erroneous text into the output correct text content. Alternatively, a confusion set can be formed by collecting misspelled words or phrases, and then the possible results can be used to replace the suspicious misspelled words. Statistical language models can then be used for error correction.

[0004] However, the above methods involve large amounts of computation, slow processing speed, low accuracy, and lack specificity. Summary of the Invention

[0005] This application provides a Chinese text error correction method, apparatus, electronic device, and readable storage medium, which can perform targeted error correction. This method can reduce the amount of computation and improve the processing speed and accuracy.

[0006] Firstly, this application provides a Chinese text error correction method, the method comprising:

[0007] Obtain the Chinese text to be corrected, and perform error location detection on the Chinese text to be corrected to obtain the detection result;

[0008] The detection results are classified into error types and then corrected to obtain corrected detection results;

[0009] For the error type corresponding to the corrected detection result, the first priority error correction model corresponding to the error type is found based on the pre-stored error correction model to correct the corrected detection result, and the error correction result of the Chinese text to be corrected is obtained.

[0010] Optionally, error location detection is performed on the Chinese text to be corrected to obtain detection results, including:

[0011] The first error detection module is used to detect the Chinese text to be corrected, and the detection result is obtained; the first error detection module includes a first error detection model and / or a pre-trained second error detection model.

[0012] Optionally, the detection results are classified into error types and corrected to obtain corrected detection results, including:

[0013] Based on the error locations in the detection results, the second detection module classifies the error types of the detection results and corrects the error types to obtain corrected detection results.

[0014] Optionally, based on a pre-stored error correction model, the first priority error correction model corresponding to the error type is found to correct the corrected detection result, thereby obtaining the error correction result of the Chinese text to be corrected, including:

[0015] Based on the pre-stored error correction models, the first priority error correction model corresponding to the error type is found, and the first priority error correction model is used to correct the Chinese text to be corrected to obtain the first candidate set. The candidate word with the highest probability in the first candidate set is selected to replace the error position in the detection result to obtain the first text.

[0016] Calculate the first text perplexity of the first text and determine whether the first text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, obtain the correction result; otherwise, find the secondary priority correction model corresponding to the error type based on the pre-stored correction model, and use the secondary priority correction model to correct the Chinese text to be corrected to obtain a second candidate set, and select the candidate word with the highest probability in the second candidate set to replace the error position in the detection result to obtain the second text;

[0017] Calculate the second text perplexity of the second text and determine whether the second text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, obtain the correction result; otherwise, determine that the error position in the detection result does not need to be corrected.

[0018] Optionally, the method further includes:

[0019] Obtain a training dataset, wherein each training data in the training dataset includes: unlabeled statements and labeled statements;

[0020] The second error detection model is trained based on the training dataset.

[0021] Accordingly, the first error detection module is used to detect the Chinese text to be corrected, and the detection results are obtained, including:

[0022] The Chinese text to be corrected is input into the first error detection model to obtain the first detection result;

[0023] The Chinese text to be corrected is input into the second error detection model trained on the training dataset to obtain the second detection result;

[0024] The first detection result and the second detection result are combined to obtain the detection result.

[0025] Optionally, the error types include: character errors, word errors, and semantic errors; a character error is an error containing a single character in the text, a word error is an error containing two, three, or four characters in the text, and a semantic error is an error containing more than four characters in the text; based on the error location in the detection result, the second detection module classifies the error type of the detection result and corrects the error type, including:

[0026] Based on the number of characters at the error location, the second detection module classifies the error type of the detection result to obtain the classification result;

[0027] The detection results corresponding to the classification results are processed by word segmentation using a classification correction method to correct the error positions and error types in the detection results.

[0028] Optionally, the first priority error correction model corresponding to this error type can be found based on pre-stored error correction models, including:

[0029] If the error type of the detection result is a word error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained masking-based language model.

[0030] If the error type of the detection result is a word error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is the confusion set-based model;

[0031] If the error type of the detection result is a semantic error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained translation-based model.

[0032] Accordingly, based on the pre-stored error correction model, the next-priority error correction model corresponding to this error type is found, including:

[0033] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is the confusion set-based model.

[0034] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a pre-trained masking-based language model.

[0035] If the error type of the detection result is a semantic error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a masking language model.

[0036] The translation-based model and the masking-based language model are deep learning models based on neural networks.

[0037] Optionally, the first priority error correction model is used to correct the Chinese text to be corrected, resulting in a first candidate set. The candidate word with the highest probability in the first candidate set is selected to replace the error position in the detection result, resulting in a first text, including:

[0038] The Chinese text to be corrected is input into the first priority error correction model to obtain a first candidate set; the first candidate set includes multiple candidate words, each candidate word is used to replace the error position in the detection result;

[0039] The first candidate set is classified according to whether the part of speech is the same as the error position in the detection result, resulting in a first category with the same part of speech and a second category with different part of speech;

[0040] Calculate the probabilities of multiple candidate words in the first category and multiple candidate words in the second category; the probabilities are used to represent the rationality of the Chinese text to be corrected after replacement with candidate words;

[0041] The candidate words with the highest probability in the first category and the candidate words with the highest probability in the second category are selected to replace the erroneous positions in the detection results, respectively, to obtain the first text.

[0042] Optionally, calculating the first text perplexity of the first text includes:

[0043] Calculate the third text perplexity of the text corresponding to the candidate word with the highest probability in the first category after replacing the wrong position, and calculate the fourth text perplexity of the text corresponding to the candidate word with the highest probability in the second category after replacing the wrong position;

[0044] Determine whether the third text perplexity is greater than the fourth text perplexity. If so, determine that the fourth text perplexity is the first text perplexity; otherwise, determine that the third text perplexity is the first text perplexity.

[0045] Optionally, the method further includes:

[0046] After correcting the Chinese text, the correction results are obtained, and the staff judges whether the correction results are reasonable.

[0047] If reasonable, the error correction results will be visualized for users to view;

[0048] If the error is not valid, an instruction message will be sent to instruct the user or staff to correct the Chinese text to be corrected.

[0049] Secondly, this application provides a Chinese text correction device, the device comprising:

[0050] The detection module is used to acquire the Chinese text to be corrected, and to detect the error location in the Chinese text to be corrected, and to obtain the detection result;

[0051] The classification correction module is used to classify the detection results by error type and correct the error type to obtain the corrected detection results;

[0052] The error correction module is used to correct the error type of the corrected detection result by finding the first priority error correction model corresponding to the error type based on the pre-stored error correction model, and obtaining the error correction result of the Chinese text to be corrected.

[0053] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0054] The memory stores computer-executed instructions;

[0055] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0056] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the Chinese text correction method as described in any one of the first aspects.

[0057] Fifthly, this application provides a computer program product, including program code, which, when a computer runs the computer program, performs the method described in any one of the first aspects.

[0058] In summary, this application provides a Chinese text error correction method, apparatus, electronic device, and readable storage medium. The method acquires the Chinese text to be corrected and performs error location detection to obtain detection results. Further, it classifies the detection results by error type and corrects the error type to obtain corrected detection results. Further still, for the error type corresponding to the corrected detection results, it finds the first-priority error correction model corresponding to that error type based on pre-stored error correction models and corrects the corrected detection results to obtain the corrected Chinese text. Here, the first-priority error correction model refers to the error correction model with the highest priority level. This allows for targeted error correction based on the error type of the Chinese text to be corrected, reducing computational load and improving processing speed and accuracy. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0061] Figure 2 A flowchart illustrating a Chinese text correction method provided in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram illustrating a process for obtaining correction results by correcting Chinese text to be corrected, as provided in an embodiment of this application.

[0063] Figure 4 This is a flowchart illustrating an embodiment of the present application for detecting and correcting errors in Chinese text.

[0064] Figure 5 A schematic diagram of the structure of a Chinese text correction device provided in this application embodiment;

[0065] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0066] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the present invention in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0068] The term "multiple" in this application refers to two or more. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship between the preceding and following related objects. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0069] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first device" and "second device" are merely used to distinguish different devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0070] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0071] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0072] It is understood that, in the embodiments of this application, the order of the following processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0073] The embodiments of this application will now be described in conjunction with the accompanying drawings. Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. The Chinese text error correction method provided in this application can be applied to, for example... Figure 1 In the application scenario shown, the scenario includes a first terminal device 101, a second terminal device 102, a server 103, a third terminal device 104, and a staff member 105. The first terminal device 101 can send voice messages to the server 103, and the second terminal device 102 can send image messages to the server 103. Correspondingly, the server 103 can receive the voice messages sent by the first terminal device 101 and the image messages sent by the second terminal device 102. Furthermore, the server 103 can convert the voice messages sent by the first terminal device 101 into text using Automated Speech Recognition (ASR) technology, and convert the image messages sent by the second terminal device 102 into text using Optical Character Recognition (OCR) technology. Furthermore, the error detection module in the server 103 can perform error detection on the above text, and the error correction module can correct the detected errors to finally obtain the error correction result.

[0074] Optionally, the server 103 can also send the error correction result to the third terminal device 104 for staff 105 to check and determine whether the error correction result is reasonable. If reasonable, the error correction result is sent to the first terminal device 101 and the second terminal device 102 for visualization display for users to view. If unreasonable, the staff 105 can correct the error correction result and further send the corrected result to the first terminal device 101 and the second terminal device 102 for visualization display for users to view.

[0075] It is understood that the server 103 can process the messages sent by the first terminal device 101 and the second terminal device 102 in the order of their sending time, or they can be processed simultaneously. This application embodiment does not specifically limit this. The server 103's technology for converting information such as voice, text, and images into text is not specifically limited in this application embodiment. Optionally, the first terminal device 101 and the second terminal device 102 can also use technologies such as OCR and ASR to convert information such as voice, text, and images into text. Furthermore, the converted text is sent to the server 103 for error correction processing. This application embodiment does not specifically limit the executing entity that uses a certain technology to convert information such as voice, text, and images into text.

[0076] The first terminal device 101 and the second terminal device 102 described above can be either wireless terminals or wired terminals. This application embodiment does not specifically limit the number of terminal devices. A wireless terminal can be a device that provides voice and / or other service data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a Radio Access Network (RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, or computer-embedded mobile device, which exchanges voice and / or data with the RAN. For example, a wireless terminal can also be a Personal Communication Service (PCS) phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), or other devices. A wireless terminal can also be referred to as a system, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote terminal, access terminal, user terminal, user agent, user device, or user equipment, without limitation. Optionally, the aforementioned terminal devices can also be smartphones, tablets, or other similar devices.

[0077] It should be noted that the third terminal device 104 may be a large screen (or smart screen), mobile phone, tablet computer, smartwatch, smart bracelet, smart glasses or other terminal devices with a display screen, etc., and this application embodiment does not specifically limit it.

[0078] In existing technologies, when it is necessary to correct errors in Chinese text, an end-to-end neural network-based translation model can be used to output correct text content from the input erroneous text. Alternatively, a confusion set can be formed by collecting misspelled characters or words, and then the possible results can be searched to replace the suspicious misspelled characters. Statistical language models can then be used for error correction.

[0079] However, the above methods involve large amounts of computation, slow processing speed, low accuracy, and lack specificity.

[0080] Therefore, this application provides a Chinese text error correction method that employs a method of first detecting errors and then correcting them. Specifically, the method involves acquiring the Chinese text to be corrected, performing error detection on it, obtaining the error detection results, further classifying the errors, and correcting the error types, for example, primarily classifying them as character errors, word errors, and semantic errors. After determining the type corresponding to the error detection result, the method uses its corresponding first-priority error correction model to correct the Chinese text, obtaining the correction result. In this way, error correction can be targeted at the error type of the Chinese text to be corrected, reducing computational load and improving processing speed and accuracy.

[0081] For example, Figure 2 This is a flowchart illustrating a Chinese text correction method provided in an embodiment of this application, as shown below. Figure 2 As shown, the method in this application embodiment includes:

[0082] S201. Obtain the Chinese text to be corrected, and perform error location detection on the Chinese text to be corrected to obtain the detection result.

[0083] In this embodiment of the application, the method of obtaining the Chinese text to be corrected can be to convert data such as video, audio and images into Chinese text to be corrected through technologies such as OCR and ASR, or to directly extract data from the text and convert it into Chinese text to be corrected, and then further detect the Chinese text to be corrected.

[0084] OCR refers to the process by which electronic devices (such as scanners or digital cameras) examine printed characters on paper, determine their shapes by detecting dark and light patterns, and then translate the shapes into computer text using character recognition methods. In other words, for printed characters, it uses optical methods to convert the text in a paper document into a black-and-white dot matrix image file, and then uses recognition software to convert the text in the image into text format for further editing by word processing software. ASR is a technology that converts human speech into text.

[0085] In this embodiment, the error location can refer to the location in the text where an error exists. It can be a misspelled word, an incorrect phrase, or an incorrect sentence. This application does not make any specific limitation on this.

[0086] For example, in Figure 1In the application scenario, taking the voice message sent by the first terminal device 101 as an example, the server 103 can receive the voice message sent by the first terminal device 101. Furthermore, it can use technologies such as OCR and ASR to convert the voice information into Chinese text to be corrected. Furthermore, the server 103 can obtain the Chinese text to be corrected and perform error location detection on the Chinese text to be corrected. For example, it can use a sequence labeling model to detect the Chinese text to be corrected and obtain the detection result.

[0087] S202. Classify the detection results by error type and correct the error type to obtain the corrected detection results.

[0088] In this embodiment of the application, the error types can be divided into character errors, word errors, and semantic errors. Among them, a character error is an error in which the text contains a single character, a word error is an error in which the text contains two, three, or four characters, and a semantic error is an error in which the text contains more than four characters. It should be noted that a semantic error indicates that the statement has a lot of errors, which may include errors in order, extra characters, missing characters, etc.

[0089] It is understood that the embodiments of this application do not specifically limit the method for error type correction. As long as a certain method can achieve the purpose of error type correction, it is acceptable. For example, a misclassification correction method can be used to perform word segmentation on the detection results to correct the error position and classification result.

[0090] For example, in Figure 1 In the application scenario, server 103 performs error location detection on the text converted from the voice message sent by the first terminal device 101. After obtaining the error detection result, server 103 can determine which error type the error location in the detection result belongs to and classify it. Furthermore, it can use misclassification correction methods to correct the classification result. For example, in the initial classification, a certain character in the error detection result is classified as a character error, but in fact, merging a certain character in the error detection result with adjacent characters and classifying it as a word error has a better error correction effect. Therefore, the previous error detection result can be corrected to the "word error type". Further, the detection result after the error type is corrected can be obtained.

[0091] S202. For the error type corresponding to the corrected detection result, find the first priority error correction model corresponding to the error type based on the pre-stored error correction model to correct the corrected detection result, and obtain the error correction result of the Chinese text to be corrected.

[0092] In this embodiment of the application, the first priority error correction model can refer to the error correction model with the highest priority that needs to be executed first. Different error types correspond to different first priority error correction models. This first priority error correction model can be a pre-stored error correction model, which can be stored in a database and can be directly called when needed. Specifically, as shown in Table 1:

[0093] Table 1. First-priority error correction models for different error types.

[0094] Error Type First-priority error correction model Word error Based on masking language model Word error Based on the confusion set model semantic error Based on translation model

[0095] Among them, the first priority error correction model for character error type is based on masking language model; the first priority error correction model for word error type is based on confusion set model; and the first priority error correction model for semantic error type is based on translation model.

[0096] It is understandable that a masking-based language model can refer to a model that masks words at incorrect positions and then inputs them into the model to obtain the output result. A confusion-based model can refer to a model that calls candidate words from a confusion set to replace words at incorrect positions. The confusion set is a pre-set collection of words with the same or different parts of speech but similar appearance. A translation-based model can refer to a model that represents entity vectors in a low-dimensional dense vector space and then performs calculations and inferences. That is, it can be a model that maps entities and relations in a knowledge base to a low-dimensional vector space to calculate implicit relations.

[0097] It should be noted that the translation-based model and the masking-based language model are deep learning models based on neural networks. Neural networks are an important machine learning technique. Their network structure includes an input layer, hidden layers, and an output layer. The number of hidden layers can be set. The training process of neural networks mainly uses the principle of backpropagation to perform network gradient descent optimization to find the best model parameters, which can be used for text recognition.

[0098] For example, in Figure 1 In the application scenario, taking the voice message sent by the first terminal device 101 as an example, if the server 103 detects that the error type of the text converted from the voice message is a word error, it can find the first priority error correction model corresponding to the word error type in the pre-stored error correction models, which is the masking language model. Furthermore, the server 103 can use the masking language model to perform error correction processing on the text converted from the voice message sent by the first terminal device 101 to obtain the error correction result. The pre-stored error correction models are shown in Table 1. They can be stored in advance in the database of the server 103, or they can be entered by the user when using them. This application embodiment does not make specific limitations on this.

[0099] Therefore, the Chinese text correction method provided in this application embodiment can perform targeted correction based on the error type corresponding to the Chinese text to be corrected, which can reduce the amount of computation, improve the processing speed and accuracy, and achieve good correction results.

[0100] Optionally, error location detection is performed on the Chinese text to be corrected to obtain detection results, including:

[0101] The first error detection module is used to detect the Chinese text to be corrected, and the detection result is obtained; the first error detection module includes a first error detection model and / or a pre-trained second error detection model.

[0102] Preferably, the first error detection model is a statistical language model, and the second error detection model is a deep learning model based on sequence labeling.

[0103] In this embodiment, the statistical language model refers to the fundamental model of Natural Language Processing (NLP), a mathematical model that addresses the context-dependent characteristics of natural language from a probabilistic and statistical perspective. It is used to determine the probability of a sentence appearing in text. For example, all possible sentences in a text follow a probability distribution, with the sum of the probabilities of each sentence being 1. The statistical language model can predict the probability of each sentence appearing in the text. For common sentences in the text, a relatively high probability can be calculated, while for ungrammatical sentences, the calculated probability approaches zero.

[0104] Preferably, the statistical language model of the present application is an n-gram model. Specifically, the algorithm of the n-gram model is to perform a sliding window operation of size N on the content in the text by bytes, forming a sequence of byte segments of length N. Each byte segment is called a gram. The occurrence frequencies of all grams are counted and filtered according to a preset threshold to form a list of key grams, which is the vector feature space of this text. Each gram in the list is a dimension of the feature vector. It can be understood that the occurrence of the Nth word is only related to the previous N - 1 words and has nothing to do with any other words. The probability of the whole sentence is the product of the occurrence probabilities of each word. Therefore, these probabilities can be obtained by directly counting the number of times N words appear simultaneously in the corpus. For example, for a text "想我吃中国食物" (I want to eat Chinese food), when performing a sliding window operation with a size of 2 bytes, the text is divided into: "想我" (want me), "我吃" (I eat), "吃中" (eat Chinese), "中国" (China), "国食" (national food), "食物" (food), a total of 6 grams. Further, the occurrence frequencies of all grams are counted. Suppose the probabilities obtained from the corpus are: P("我想") = 0.25, P("想我") = 0.05, P("想吃") = 0.21, P("吃我") = 0.02, P("我吃") = 0.12, P("中吃") = 0.03, P("吃中") = 0.01, P("中国") = 0.26, P("国中") = 0.04, P("国食") = 0.08, P("食物") = 0.2, etc. By calculation, P("想我吃中国食物") = 0.05 * 0.12 * 0.01 * 0.26 * 0.08 * 0.2 = 0.0000002496, and P("我想吃中国食物") = 0.25 * 0.21 * 0.01 * 0.26 * 0.08 * 0.2 = 0.000002184. The probability of P("我想吃中国食物") is greater than P("想我吃中国食物"), so the text "我想吃中国食物" is more reasonable, while the text "想我吃中国食物" has a semantic error. Among them, the probability of the text "想我吃中国食物" can also be compared with the probabilities of other texts such as "我想吃食物中国", "我吃想中国食物", etc. However, usually, the text with the largest probability in the calculated probabilities is compared with the text to be compared to determine whether there is a semantic error.

[0105] It can be understood that the numerical values of the occurrence frequencies of the above-mentioned various grams are only for illustrative purposes, and the specific values should be based on the actual situation.

[0106] It should be noted that the present application does not specifically limit the specific value of the size of the sliding window N, and Bi-Gram, Tri-Gram or a combination of both can be selected.

[0107] Sequence labeling can be used to solve a range of character classification problems, such as word segmentation, part-of-speech tagging, named entity recognition, and relation extraction. A deep learning model refers to an architecture based on a deep learning algorithm that can be used to iteratively train training data. A deep learning model for sequence labeling specifically refers to a deep learning model that takes an observed sequence as input and outputs a labeled sequence or a state sequence. This model is trained on a large training set and can provide a labeled sequence as a prediction for the observed sequence.

[0108] It should be noted that the first error detection model focuses more on low-level text errors, while the second error detection model uses a sequence labeling model based on pre-trained techniques to detect text with semantic errors. The second error detection model has better error detection performance than the first model, but its processing efficiency is lower. Therefore, if the error set is formed by combining the error positions obtained from the first and second error detection models, and for errors with repeated positions but different other factors in the detection results, such as the byte length of the error position, this application uses the error position obtained from the second error detection model as the standard.

[0109] For example, in Figure 1 In the application scenario, taking the voice message sent by the first terminal device 101 as an example, the server 103 can receive the voice message sent by the first terminal device 101. Furthermore, it uses technologies such as OCR and ASR to convert the voice information into Chinese text to be corrected. Further, the server 103 obtains the Chinese text to be corrected and uses an n-gram model and a pre-trained sequence labeling model to detect the Chinese text to be corrected, and obtains the detection result.

[0110] Therefore, this application introduces different error detection models to better detect erroneous text in the original text. The first error detection model focuses more on low-level text errors, while the second error detection model uses a sequence labeling model based on pre-training technology to detect semantically erroneous text, thereby improving the accuracy of error location detection.

[0111] Optionally, the detection results are classified into error types and corrected to obtain corrected detection results, including:

[0112] Based on the error locations in the detection results, the second detection module classifies the error types of the detection results and corrects the error types to obtain corrected detection results.

[0113] In this embodiment, the second detection module includes a module for classifying error types in the detection results and a module for correcting error types, which can effectively improve the subsequent error correction effect.

[0114] For example, in Figure 1 In the application scenario, after the server 103 detects an error in the text converted from the voice message sent by the first terminal device 101 using the first error detection module and outputs the error detection result, the server 103 can determine which type of error the error in the detection result belongs to and classify it. Furthermore, it can use a classification correction method to determine whether the classification result is correct. If it is incorrect, the classification result can be corrected. For example, the error position may be classified as a character error, but classifying the error position as a word error has a better error correction effect. Therefore, the previous error type classification result "character error category" can be corrected to "word error type". Furthermore, the detection result after correcting the error type can be obtained.

[0115] It is understood that the classification correction method may refer to the misclassification correction method mentioned in the above embodiments, or it may be other correction methods. This application embodiment does not specifically limit it in this regard.

[0116] Therefore, using the second detection module to classify and correct the error types in the detection results can greatly improve the subsequent error correction effect and reduce error correction mistakes.

[0117] Optional, Figure 3 This application provides a schematic diagram of a process for obtaining correction results by correcting Chinese text to be corrected, as illustrated in the embodiments of this application. Figure 3 As shown, a further refinement of step S203 involves finding the first priority error correction model corresponding to the error type based on the pre-stored error correction models to correct the corrected detection result, thereby obtaining the error correction result of the Chinese text to be corrected. The execution steps include:

[0118] S301. Based on the pre-stored error correction models, find the first priority error correction model corresponding to the error type, and use the first priority error correction model to correct the Chinese text to be corrected to obtain a first candidate set. Then, select the candidate word with the highest probability in the first candidate set to replace the error position in the detection result to obtain the first text.

[0119] In this embodiment of the application, the first candidate set may include multiple candidate words, each candidate word being used to replace the erroneous position in the detection result. The first candidate set contains candidate words with the same part of speech as the erroneous position text and words with different parts of speech. The same part of speech means that the words have the same attributes, which can be divided into verbs, nouns, adjectives, prepositions, conjunctions, etc. For example, if the erroneous position is "blow", then the first candidate set may contain words with the same part of speech such as "shake", "float", and "piao", or words with different parts of speech such as "hang", "cook", and "hammer".

[0120] Optionally, the candidate word with the highest probability refers to the maximum probability of a candidate word appearing in the Chinese text to be corrected. The probability is used to represent the rationality of the Chinese text to be corrected after the candidate word is replaced. The probability of multiple candidate words appearing in the Chinese text to be corrected can be estimated by the Markov hypothesis algorithm. Furthermore, the candidate word corresponding to the maximum value among multiple probabilities is the candidate word with the highest probability.

[0121] It should be noted that the embodiments of this application can also use other algorithms to calculate the probability of each candidate word in the first candidate set appearing in the Chinese text to be corrected. The embodiments of this application do not specifically limit the algorithm for calculating the probability of candidate words appearing in the Chinese text to be corrected, and it can be the Markov hypothesis algorithm, the GloVe algorithm, etc.

[0122] For example, in Figure 1 In the application scenario, taking the voice message sent by the first terminal device 101 as an example, if the server 103 detects that the error type of the detection result of the text converted from the voice message is a word error, it can find the first priority error correction model corresponding to the error type based on the pre-stored error correction model, which is the masking language model. The server 103 then uses the masking language model to correct the text converted from the voice message, and obtains a candidate set. In this candidate set, there are candidate words with the same and different parts of speech as the text at the error position. Furthermore, the server uses the Markov hypothesis algorithm to calculate all probabilities of candidate words with the same and different parts of speech, and selects the candidate word with the highest probability among the candidate words with the same and different parts of speech to replace the error position, thus obtaining the first text after replacement.

[0123] S302. Calculate the first text perplexity of the first text and determine whether the first text perplexity is less than the text perplexity of the Chinese text to be corrected. If so, obtain the correction result. Otherwise, find the secondary priority correction model corresponding to the error type based on the pre-stored correction model, and use the secondary priority correction model to correct the Chinese text to be corrected to obtain a second candidate set. Then, select the candidate word with the highest probability in the second candidate set to replace the error position in the detection result to obtain the second text.

[0124] In this embodiment, the secondary priority error correction model refers to an error correction model with a priority lower than the first priority, which can only be executed after the first priority model has been executed. Different error types correspond to different secondary priority error correction models. This secondary priority error correction model can be a pre-stored error correction model, which can be stored in a database and directly called when needed. Specific details are shown in Table 2.

[0125] Table 2. Sub-priority error correction models for different error types.

[0126] Error Type Second-priority error correction model Word error Based on the confusion set model Word error Based on masking language model semantic error Based on masking language model

[0127] Among them, the secondary priority error correction model corresponding to character error type is based on confusion set model; the secondary priority error correction model corresponding to word error type is based on masking language model; and the secondary priority error correction model corresponding to semantic error type is based on masking language model.

[0128] Text perplexity can be used to measure how well a probability distribution or probability model predicts a sample. Generally, the lower the text perplexity, the more reasonable the sentence is. The definition of the second candidate set is similar to that of the first candidate set. For details, please refer to the description of the first candidate set, which will not be repeated here.

[0129] Specifically, text perplexity can be calculated using the following formula:

[0130]

[0131] Where, PPL(S)=P(w1w2…w N ) -1 / N PPL(S) represents the text perplexity level, S represents the Chinese text to be corrected, and S = w1w2…w N w i This represents the i-th word (or character) in the Chinese text to be corrected.

[0132] For example, taking word error as an example to determine the error type of the detection result, for the Chinese text S to be corrected... o :

[0133] S o =[C1,C2,C3,C4,C5,…,C n ]

[0134] Among them, C i This represents the i-th Chinese character (i = 1, 2, ..., n). After the error detection process, S o The text at positions (C3, C4) contains erroneous words. After passing through the first priority error correction model, the Chinese text to be corrected is processed, resulting in the first candidate set. This is equivalent to obtaining the text candidate set C after using the confusion-based error correction module. cand :

[0135]

[0136] in, Let represent the i-th candidate set at the error position (C3, C4). Further, select the candidate words with the highest probability of having the same part of speech as the error position (C3, C4) and those with different parts of speech. Assume the candidate word with the highest probability of having the same part of speech is... The candidate words with the highest probability and different parts of speech are Furthermore, the perplexity of the replaced Chinese text to be corrected is calculated separately:

[0137]

[0138]

[0139] Among them, PP(S) m ) represents the text perplexity after replacing the error location in the Chinese text to be corrected with alternative words of the same part of speech, PP(S) n ) represents the text perplexity after replacing the error position in the Chinese text to be corrected with alternative words of different parts of speech. The text perplexity of the Chinese text to be corrected is: PP(S) o )=F([C1,C2,C3,C4,C5,…,C n ]), F represents the method for calculating text perplexity, optional, PP(S m ), PP(S n ) and PP(S o The calculation formula for PPL(S) can be found in the calculation process of PPL(S) above, and will not be repeated here.

[0140] It should be noted that the method for calculating the probability of the candidate word with the highest probability is similar to that in S301, and can be found in step S301. It will not be repeated here. This application does not specify which method to use to calculate the text perplexity. The above is just an example.

[0141] For example, the server can use the above formula to calculate the text perplexity PP(S) after replacing the error location in the Chinese text to be corrected with candidate words of the same part of speech. m The perplexity PP (Personalities Per Second) of the text after replacing the error location in the Chinese text to be corrected with alternative words of different parts of speech. n Furthermore, by judging PP(S) m ) and PP(S n The result of the first priority error correction model is determined by whether the minimum value in PP(S) is less than the text perplexity of the Chinese text to be corrected; if PP(S) is less than the minimum value in PP(S), the result of the first priority error correction model is determined by whether the minimum value in PP(S) is less than the text perplexity of the Chinese text to be corrected. o )≥min(PP(S m ),PP(S n If the result is min(PP(S), it means the corrected text achieved a lower perplexity than the original Chinese text. m ),PP(S n The candidate word with lower perplexity is selected as the final error correction result. If PP(S) o )<min(PP(S m ),PP(S nIf the result obtained by the first priority error correction model is not satisfactory, then it is necessary to find the second priority error correction model corresponding to the error type based on the pre-stored error correction model, and use the second priority error correction model to correct the Chinese text to be corrected, and obtain a candidate set. Then, select the candidate word with the highest probability in the candidate set to replace the error position in the detection result, and obtain the required second text.

[0142] It should be noted that if the error type is a word error or a semantic error, the process of calculating the text perplexity is similar to that described above, and will not be repeated here.

[0143] S303. Calculate the second text perplexity of the second text and determine whether the second text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, obtain the correction result; otherwise, determine that the error position in the detection result does not need to be corrected.

[0144] For example, the second text perplexity is calculated using a calculation process similar to step S302, which will not be repeated here. The second text includes candidate words with the same part of speech to replace the Chinese text to be corrected and candidate words with different parts of speech to replace the Chinese text to be corrected. Further, the lower text perplexity calculated in the second text is compared with the text perplexity calculated in the Chinese text to be corrected. If the lower text perplexity calculated in the second text is less than the text perplexity of the Chinese text to be corrected, then the candidate word corresponding to the lower perplexity is selected as the correct result for correction. If the lower text perplexity calculated in the second text is greater than the text perplexity of the Chinese text to be corrected, then it indicates that the error location detection is incorrect and no correction is required.

[0145] Therefore, by using the first-priority error correction model and the second-priority model to correct the Chinese text to be corrected, replacing the erroneous position in the Chinese text to be corrected with the candidate word with the highest probability in the candidate set, and selecting the text with the lowest text perplexity as the final error correction result, the accuracy of error correction can be improved.

[0146] For example, Figure 4 This application provides a flowchart illustrating the process of detecting and correcting errors in Chinese text, as shown in the embodiments. Figure 4As shown, in step S401, the server can obtain the text to be corrected (i.e., the Chinese text to be corrected). This text to be corrected is converted from external data, voice, images, text, etc. using technologies such as OCR and ASR. Further, step S402 is executed; in step S402, the error detection module in the server performs error detection on the text to be corrected and can detect errors. After the detection is completed, step S403 can be executed; in step S403, the error correction module corrects the detected errors and obtains the correction result. After the correction is completed, step S404 can be executed; in step S404, the correction result from step S403 is obtained and displayed.

[0147] Optionally, the method further includes:

[0148] Obtain a training dataset, wherein each training data in the training dataset includes: unlabeled statements and labeled statements;

[0149] The second error detection model is trained based on the training dataset.

[0150] Accordingly, the first error detection module is used to detect the Chinese text to be corrected, and the detection results are obtained, including:

[0151] The Chinese text to be corrected is input into the first error detection model to obtain the first detection result;

[0152] The Chinese text to be corrected is input into the second error detection model trained on the training dataset to obtain the second detection result;

[0153] The first detection result and the second detection result are combined to obtain the detection result.

[0154] In this embodiment of the application, a training dataset provided by an individual can be obtained. This training dataset includes multiple training data sets, each of which may include unlabeled statements and labeled statements. The unlabeled statements may refer to unsupervised data, while the labeled statements may refer to supervised data. For example, labeled statements may refer to certain statements and their corresponding error locations. The number of unlabeled statements is greater than that of labeled statements, and the labeled statements can also be used to verify the effectiveness of model training.

[0155] It should be noted that the training dataset for training the second error detection model only needs to be done once. After that, the trained second error detection model can be used directly to detect the Chinese text to be corrected.

[0156] For example, the server can train the second error detection model in advance, that is, obtain a training dataset, which may include multiple training data, each of which may include unlabeled statements and labeled statements; further, the second error detection model is trained based on the training dataset.

[0157] For example, the server can obtain a series of statements and the corresponding error locations in these statements, and then train the second error detection model based on this data.

[0158] Correspondingly, after the server trains the second error detection model, it can input the Chinese text to be corrected, which is converted from external data, speech, images, and text using technologies such as OCR and ASR, into the trained second error detection model. In this way, the error positions of the Chinese text to be corrected can be detected, and a second detection result can be obtained. Before this, the server can also input the Chinese text to be corrected, which is converted from external data, speech, images, and text using technologies such as OCR and ASR, into the first error detection model to obtain a first detection result containing the error positions. Furthermore, the first and second detection results can be merged to obtain the desired detection result.

[0159] Therefore, by using the first error detection model to detect the Chinese text to be corrected, and simultaneously using the pre-trained second error detection model to detect the error locations in the Chinese text to be corrected, a second detection result can be obtained. This can improve the accuracy and precision of the detection. Moreover, the first error detection model has a fast detection rate, and using different error detection models can better detect the erroneous text in the Chinese text to be corrected.

[0160] Optionally, the error types include: character errors, word errors, and semantic errors; a character error is an error containing a single character in the text, a word error is an error containing two, three, or four characters in the text, and a semantic error is an error containing more than four characters in the text; based on the error location in the detection result, the second detection module classifies the error type of the detection result and corrects the error type, including:

[0161] Based on the number of characters at the error location, the second detection module classifies the error type of the detection result to obtain the classification result;

[0162] The detection results corresponding to the classification results are processed by word segmentation using a classification correction method to correct the error positions and error types in the detection results.

[0163] In the embodiments of the present application, the initial classification method may make misclassifications. The misclassifications are mainly due to the lack of comprehensiveness in error position detection. Therefore, the misclassifications mainly focus on misclassifying word error types as character errors. Thus, the classification correction method can correct such misclassifications. The classification correction method mainly uses the method of word segmentation to process the original error text (error position). For the error position of character errors, if the character at this error position is divided into words (including two, three or four characters), it should be corrected to a word error.

[0164] Exemplarily, taking the Chinese text to be corrected “上海市恩行于近期推出了多项措施” as an example, in which the error detection result classifies “恩” as a character error category. However, the above classification may make a misclassification. For example, classifying “恩行” as a word error has a better error correction effect. Therefore, the present application uses the classification correction method to correct the error position and classification result. After the classification correction method, the original text is segmented into: “上海市”, “恩行”, “于”, “近期”, “推出”, “了”, “多项”, “措施”. Therefore, this error type can be corrected to a word error, and the error position is corrected to “恩行”.

[0165] It should be noted that the second detection module is used to classify the error types of the detection results and use the classification correction method to correct the above classification results.

[0166] Therefore, the present application introduces different error classifications to divide text errors into character errors, word errors and semantic errors. To prevent misclassification, the classification correction method is also used to correct the situation of misclassifying word error types as character errors, which can effectively improve the error correction effect and accuracy.

[0167] Optionally, for the detection results of the determined error types, find the first-priority error correction model corresponding to the error type in the pre-stored error correction model, including:

[0168] If the error type of the detection result is a character error, find its corresponding first-priority error correction model in the pre-stored error correction model as the pre-trained masked language model;

[0169] If the error type of the detection result is a word error, find its corresponding first-priority error correction model in the pre-stored error correction model as the confusion set model;

[0170] If the error type of the detection result is a semantic error, find its corresponding first-priority error correction model in the pre-stored error correction model as the pre-trained translation model;

[0171] Accordingly, based on the pre-stored error correction model, the next-priority error correction model corresponding to this error type is found, including:

[0172] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is the confusion set-based model.

[0173] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a pre-trained masking-based language model.

[0174] If the error type of the detection result is a semantic error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a masking language model.

[0175] The translation-based model and the masking-based language model are deep learning models based on neural networks.

[0176] For example, in Figure 1 In the application scenario, server 103 can use the first error detection module to detect the Chinese text to be corrected, obtain the detection result, and further determine the error type of the detection result. If it is determined that the error type of the detection result of the Chinese text to be corrected converted from the voice message sent by the first terminal device 101 is a word error, then the corresponding first priority error correction model is found in the pre-stored error correction model, which is a pre-trained masking language model, and the second priority error correction model is a confusion set model. If it is determined that the error type of the detection result of the Chinese text to be corrected converted from the image message sent by the second terminal device 102 is a word error, then the server 103 can use the first error detection module to detect the Chinese text to be corrected, obtain the detection result, and further determine the error type of the detection result. If the error type is a word error, the first priority error correction model is found in the pre-stored error correction models. The first priority error correction model is a confusion set-based model, and the second priority error correction model is a pre-trained masking language model. In this scenario, there may also be video messages sent by other terminal devices (not shown in the figure). If the server 103 determines that the error type of the detection result of the Chinese text to be corrected converted from the video message sent by other terminal devices is a semantic error, the first priority error correction model is found in the pre-stored error correction models. The first priority error correction model is a pre-trained translation-based model, and the second priority error correction model is a masking language model.

[0177] It is understood that the message type sent by each terminal device is not specifically limited in this application embodiment. Multiple types of messages such as video, voice and pictures can be sent to a single terminal device, or only one type of message can be sent to each terminal device.

[0178] Therefore, different classifications can adopt different error correction models. For each error type, there are corresponding first-priority error correction models and second-priority error correction models, which can quickly and effectively obtain error correction results and make the error correction more reasonable.

[0179] Optionally, the first priority error correction model is used to correct the Chinese text to be corrected, resulting in a first candidate set. The candidate word with the highest probability in the first candidate set is selected to replace the erroneous position in the detection result, resulting in a first text, including:

[0180] The Chinese text to be corrected is input into the first priority error correction model to obtain a first candidate set; the first candidate set includes multiple candidate words, each candidate word is used to replace the error position in the detection result;

[0181] The first candidate set is classified according to whether the part of speech is the same as the error position in the detection result, resulting in a first category with the same part of speech and a second category with different part of speech;

[0182] Calculate the probabilities of multiple candidate words in the first category and multiple candidate words in the second category; the probabilities are used to represent the rationality of the Chinese text to be corrected after replacement with candidate words;

[0183] The candidate words with the highest probability in the first category and the candidate words with the highest probability in the second category are selected to replace the erroneous positions in the detection results, respectively, to obtain the first text.

[0184] For example, in Figure 1 In the application scenario, taking the image message sent by the second terminal device 102 as an example, if the server 103 detects that the error type of the detection result of the text converted from the image message is a word error, the text converted from the image message can be further input into the confusion set-based model to obtain the required candidate set. The first candidate set includes multiple candidate words with the same and different parts of speech as the text at the error position. Each candidate word can be used to replace the error position in the detection result. Further, the candidate set is classified according to whether it has the same part of speech as the error position in the detection result, into categories with the same part of speech as the error position and categories with different parts of speech. Then, the probability of multiple candidate words in the above two categories is calculated using the Markov assumption algorithm, and the candidate word with the highest probability in the above two categories is selected to replace the error position in the detection result to obtain the required first text. The first text includes two Chinese texts to be corrected after replacement with candidate words.

[0185] Therefore, in this embodiment of the application, the probability of all candidate words in the first candidate set obtained by the first priority error correction model is calculated, and the two candidate words with the same part of speech and different part of speech with the highest probability are selected to replace the error position of the Chinese text to be corrected, so as to obtain the required first text and improve the accuracy of error correction.

[0186] It should be noted that the process of using the secondary priority error correction model to correct the Chinese text to be corrected, obtaining the second candidate set, and selecting the candidate word with the highest probability in the second candidate set to replace the error position in the detection result to obtain the second text is similar to the process of using the first priority error correction model to correct the Chinese text to be corrected, obtaining the first candidate set, and selecting the candidate word with the highest probability in the first candidate set to replace the error position in the detection result to obtain the first text in the above embodiment. Therefore, it will not be repeated here.

[0187] Optionally, calculating the first text perplexity of the first text includes:

[0188] Calculate the third text perplexity of the text corresponding to the candidate word with the highest probability in the first category after replacing the wrong position, and calculate the fourth text perplexity of the text corresponding to the candidate word with the highest probability in the second category after replacing the wrong position;

[0189] Determine whether the third text perplexity is greater than the fourth text perplexity. If so, determine that the fourth text perplexity is the first text perplexity; otherwise, determine that the third text perplexity is the first text perplexity.

[0190] For example, in Figure 1 In the application scenario, taking the image message sent by the second terminal device 102 as an example, if the server 103 determines that the error type of the detection result of the text converted from the image message is a word error, and selects the candidate word with the highest probability in the first candidate set obtained after error correction through the first priority error correction model to replace the Chinese text to be corrected, it is necessary to further determine the text perplexity PP(S) after replacing the error position in the Chinese text to be corrected with the candidate word with the same part of speech. m The perplexity PP (Personalities Per Second) of the text after replacing the error location in the Chinese text to be corrected with alternative words of different parts of speech. n Furthermore, by judging PP(S) m ) and PP(S n The minimum value in () is used to determine the required first text perplexity.

[0191] It should be noted that PP(S) m ) and PP(S n The calculation method for ) can be found in the description of step 302, and will not be repeated here.

[0192] Therefore, in this application, by calculating the text perplexity of the text after replacing the erroneous position of the Chinese text to be corrected with two candidate words of the same part of speech and two candidate words of different parts of speech with the highest probability, and selecting the one with the smallest text perplexity as the first text perplexity, the amount of computation can be reduced, and the error correction can be more reasonable and more accurate.

[0193] It should be noted that the process of calculating the perplexity of the second text is similar to the process of calculating the perplexity of the first text in the above embodiments, and will not be repeated here.

[0194] Optionally, the method further includes:

[0195] After correcting the Chinese text, the correction results are obtained, and the staff judges whether the correction results are reasonable.

[0196] If reasonable, the error correction results will be visualized for users to view;

[0197] If the error is not valid, an instruction message will be sent to instruct the user or staff to correct the Chinese text to be corrected.

[0198] In this embodiment, the instruction information is used to indicate that there is a problem in the Chinese text to be corrected and that it needs to be corrected. The method of sending the instruction information is not specifically limited in this embodiment. It can be to display a message prompt box on the screen of the staff's terminal device, which displays "There is a problem at a certain position in a certain Chinese text to be corrected" to prompt the staff to correct it in time, and then send the corrected text to the user's terminal device for display. Alternatively, the server can send a message prompt to the user's terminal device, which reads "There is a problem at a certain position in a certain Chinese text to be corrected" to prompt the user to check in time.

[0199] For example, in Figure 1 In the application scenario, taking the voice message sent by the first terminal device 101 as an example, after the server 103 corrects the Chinese text converted from the voice message, it can obtain the correction result. Further, the staff judges whether the correction result is reasonable. If it is reasonable, the correction result is sent to the first terminal device 101 for visualization display for the user to view. If it is unreasonable, the server sends an instruction message to the third terminal device 104 of the staff 105 to instruct the staff 105 to correct the Chinese text to be corrected. After the staff 105 corrects the Chinese text to be corrected, it can send the corrected text to the first terminal device 101 for display.

[0200] It is understandable that the staff member who judges whether the error correction result is reasonable can be staff member 105 or other staff members. The other staff member only judges the error correction result and does not make corrections.

[0201] Therefore, this application can further verify whether there are any problems with the error correction results, so that users or staff can understand the situation at any time and improve accuracy.

[0202] In the foregoing embodiments, the Chinese text error correction method provided in this application has been described. To implement the functions of the methods provided in the embodiments of this application, the electronic device serving as the execution subject may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0203] For example, Figure 5 A schematic diagram of the structure of a Chinese text correction device provided in this application embodiment is shown below. Figure 5 As shown, the device includes: a detection module 510, a classification correction module 520, and an error correction module 530; wherein, the detection module 510 is used to acquire the Chinese text to be corrected, and to detect the error position of the Chinese text to be corrected to obtain the detection result;

[0204] The classification correction module 520 is used to classify the detection results by error type and correct the error type to obtain the corrected detection results;

[0205] The error correction module 530 is used to correct the error type of the corrected detection result by finding the first priority error correction model corresponding to the error type based on the pre-stored error correction model, and obtain the error correction result of the Chinese text to be corrected.

[0206] Optional, the detection module 510 is specifically used for:

[0207] The first error detection module is used to detect the Chinese text to be corrected, and the detection result is obtained; the first error detection module includes a first error detection model and / or a pre-trained second error detection model.

[0208] Optional, the classification correction module 520 is specifically used for:

[0209] Based on the error locations in the detection results, the second detection module classifies the error types of the detection results and corrects the error types to obtain corrected detection results.

[0210] Optionally, the error correction module 530 includes a searching unit, a selecting unit, a first processing unit, and a second processing unit;

[0211] Specifically, the searching unit is used to find the first priority error correction model corresponding to the error type based on the pre-stored error correction model, and the selection unit is used to use the first priority error correction model to correct the Chinese text to be corrected, obtain a first candidate set, and select the candidate word with the highest probability in the first candidate set to replace the error position in the detection result to obtain the first text;

[0212] The first processing unit is used to calculate the first text perplexity of the first text and determine whether the first text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, the correction result is obtained; otherwise, the unit finds the secondary priority correction model corresponding to the error type based on the pre-stored correction model, and uses the secondary priority correction model to correct the Chinese text to be corrected to obtain a second candidate set, and selects the candidate word with the highest probability in the second candidate set to replace the error position in the detection result to obtain the second text.

[0213] The second processing unit is used to calculate the second text perplexity of the second text and determine whether the second text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, the correction result is obtained; otherwise, it is determined that the error position in the detection result does not need to be corrected.

[0214] Optionally, the device further includes a training module, the training module being used for:

[0215] Obtain a training dataset, wherein each training data in the training dataset includes: unlabeled statements and labeled statements;

[0216] The second error detection model is trained based on the training dataset.

[0217] Accordingly, the detection module 510 includes an input unit, a detection unit, and a merging unit;

[0218] Specifically, the input unit is used to input the Chinese text to be corrected into the first error detection model to obtain the first detection result;

[0219] The detection unit is used to input the Chinese text to be corrected into the second error detection model trained by the training dataset to obtain the second detection result.

[0220] The merging unit is used to merge the first detection result and the second detection result to obtain a detection result.

[0221] Optionally, the character error is an error containing a single character in the text, the word error is an error containing two, three, or four characters in the text, and the semantic error is an error containing more than four characters in the text; the classification correction module 520 includes a classification unit and a processing unit;

[0222] Specifically, the classification unit is used to classify the error type of the detection result based on the number of characters at the error position using the second detection module, and obtain the classification result;

[0223] The processing unit is used to perform word segmentation on the detection results corresponding to the classification results using a classification correction method, and to correct the error positions and error types in the detection results.

[0224] Optionally, the searching unit is specifically used for:

[0225] If the error type of the detection result is a word error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained masking-based language model.

[0226] If the error type of the detection result is a word error, then the first priority error correction model corresponding to it in the pre-stored error correction model is the confusion set-based model.

[0227] If the error type of the detection result is a semantic error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained translation-based model.

[0228] Accordingly, the searching unit is further configured to:

[0229] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is the confusion set-based model.

[0230] If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a pre-trained masking-based language model.

[0231] If the error type of the detection result is a semantic error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a masking language model.

[0232] The translation-based model and the masking-based language model are deep learning models based on neural networks.

[0233] Optionally, the selection unit is specifically used for:

[0234] The Chinese text to be corrected is input into the first priority error correction model to obtain a first candidate set; the first candidate set includes multiple candidate words, each candidate word is used to replace the error position in the detection result;

[0235] The first candidate set is classified according to whether the part of speech is the same as the error position in the detection result, resulting in a first category with the same part of speech and a second category with different part of speech;

[0236] Calculate the probabilities of multiple candidate words in the first category and multiple candidate words in the second category; the probabilities are used to represent the rationality of the Chinese text to be corrected after replacement with candidate words;

[0237] The candidate words with the highest probability in the first category and the candidate words with the highest probability in the second category are selected to replace the erroneous positions in the detection results, respectively, to obtain the first text.

[0238] Optionally, the first processing unit is specifically used for:

[0239] Calculate the third text perplexity of the text corresponding to the candidate word with the highest probability in the first category after replacing the wrong position, and calculate the fourth text perplexity of the text corresponding to the candidate word with the highest probability in the second category after replacing the wrong position;

[0240] Determine whether the third text perplexity is greater than the fourth text perplexity. If so, determine that the fourth text perplexity is the first text perplexity; otherwise, determine that the third text perplexity is the first text perplexity.

[0241] Optionally, the device further includes a verification module, the verification module being used for:

[0242] After correcting the Chinese text, the correction results are obtained, and the staff judges whether the correction results are reasonable.

[0243] If reasonable, the error correction results will be visualized for users to view;

[0244] If the error is not valid, an instruction message will be sent to instruct the user or staff to correct the Chinese text to be corrected.

[0245] The specific implementation principle and effects of the Chinese text correction device provided in this application embodiment can be found in the relevant descriptions and effects of the above embodiments, and will not be elaborated further here.

[0246] For example, this application also provides a schematic diagram of the structure of an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6As shown, the electronic device may include: a processor 602 and a memory 601 communicatively connected to the processor; the memory 601 stores a computer program; the processor 602 executes the computer program stored in the memory 601, causing the processor 602 to perform the method described in any of the above embodiments.

[0247] The memory 601 and the processor 602 can be connected via a bus 603.

[0248] This application also provides a computer-readable storage medium storing computer program execution instructions, which, when executed by a processor, are used to implement the Chinese text error correction method as described in any of the foregoing embodiments of this application.

[0249] This application also provides a chip for executing instructions, which is used to perform Chinese text correction methods executed by an electronic device as described in any of the foregoing embodiments of this application.

[0250] This application also provides a computer program product, including program code, which, when a computer runs the computer program, executes the Chinese text correction method executed by an electronic device in any of the foregoing embodiments of this application.

[0251] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0252] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0253] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0254] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0255] It should be understood that the aforementioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0256] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0257] The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0258] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0259] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0260] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for correcting errors in Chinese text, characterized in that, include: Get the Chinese text to be corrected; Error locations are detected in the Chinese text to be corrected using a first error detection model and a second error detection model, respectively, to obtain detection results. The first error detection model is a statistical language model, and the second error detection model is a deep learning model based on sequence labeling. For repeated error locations in the detection results, the error locations detected by the second error detection model shall be used. The second detection module is used to classify the error locations in the detection results into error types and to determine whether the classification results are correct. If they are incorrect, the error types are corrected to obtain the corrected detection results. For the error type corresponding to the corrected detection result, the first priority error correction model corresponding to the error type is found based on the pre-stored error correction model, and the first priority error correction model is used to correct the Chinese text to be corrected to obtain a first candidate set; the first candidate set includes multiple candidate words with the same part of speech as the error position in the corrected detection result and multiple candidate words with different part of speech than the error position in the corrected detection result. The candidate word with the highest probability among multiple candidate words with the same part of speech and the candidate word with the highest probability among multiple candidate words with different parts of speech are selected to replace the erroneous positions in the corrected detection result to obtain the first text; the first text includes two Chinese texts to be corrected after being replaced by the candidate word with the highest probability. Calculate the first text perplexity of the first text; the first text perplexity is determined based on the one with the smallest text perplexity among the two candidate words to be corrected after being replaced with the two candidate words with the highest probabilities; Determine whether the perplexity of the first text is less than the perplexity of the Chinese text to be corrected; if so, obtain the correction result; otherwise, find the secondary priority correction model corresponding to the error type based on the pre-stored correction model to correct the corrected detection result and obtain the correction result of the Chinese text to be corrected; wherein, the secondary priority correction model corresponding to the error type found is of a different type than the first priority correction model corresponding to the error type found. Different error types correspond to different first-priority error correction models and different second-priority error correction models.

2. The method according to claim 1, characterized in that, The step of finding the secondary priority error correction model corresponding to the error type based on the pre-stored error correction model to correct the corrected detection result, in order to obtain the error correction result of the Chinese text to be corrected, includes: Based on the pre-stored error correction model, the secondary priority error correction model corresponding to the error type is found, and the secondary priority error correction model is used to correct the Chinese text to be corrected to obtain a second candidate set. The candidate word with the highest probability in the second candidate set is selected to replace the error position in the detection result to obtain the second text. Calculate the second text perplexity of the second text and determine whether the second text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, obtain the correction result; otherwise, determine that the error position in the detection result does not need to be corrected.

3. The method according to claim 1, characterized in that, Also includes: Obtain a training dataset, wherein each training data in the training dataset includes: unlabeled statements and labeled statements; The second error detection model is trained based on the training dataset. Accordingly, the error location is detected in the Chinese text to be corrected according to the first error detection module and the second error detection model, and the detection results are obtained, including: The Chinese text to be corrected is input into the first error detection model to obtain the first detection result; The Chinese text to be corrected is input into the second error detection model trained on the training dataset to obtain the second detection result; The first detection result and the second detection result are combined to obtain the detection result.

4. The method according to claim 1, characterized in that, The error types include: character errors, word errors, and semantic errors; a character error is an error in which the text contains a single character, a word error is an error in which the text contains two, three, or four characters, and a semantic error is an error in which the text contains more than four characters. The second detection module categorizes the error locations in the detection results into error types and determines whether the categorization result is correct. If incorrect, error type correction is performed, including: Based on the number of characters at the error location, the second detection module classifies the error type of the detection result to obtain the classification result; If the classification result is incorrect, the detection result corresponding to the classification result is processed by word segmentation using the classification correction method to correct the error position and error type in the detection result.

5. The method according to claim 2, characterized in that, Finding the first priority error correction model corresponding to the error type based on the pre-stored error correction model includes: If the error type of the detection result is a word error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained masking-based language model. If the error type of the detection result is a word error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is the confusion set-based model; If the error type of the detection result is a semantic error, then the first priority error correction model corresponding to it is found in the pre-stored error correction model, which is a pre-trained translation-based model. Accordingly, based on the pre-stored error correction model, the next-priority error correction model corresponding to this error type is found, including: If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is the confusion set-based model. If the error type of the detection result is a word error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a pre-trained masking-based language model. If the error type of the detection result is a semantic error, then the corresponding secondary error correction model is found in the pre-stored error correction model, which is a masking language model. The translation-based model and the masking-based language model are deep learning models based on neural networks.

6. The method according to claim 2, characterized in that, The first priority error correction model is used to correct the Chinese text to be corrected, resulting in a first candidate set, including: The Chinese text to be corrected is input into the first priority error correction model to obtain a first candidate set; the first candidate set includes multiple candidate words, each candidate word is used to replace the error position in the detection result; The method further includes: classifying the first candidate set according to whether the part of speech of the candidate words is the same as that of the erroneous position in the detection result, to obtain a first category with the same part of speech and a second category with different part of speech; the step of selecting the candidate word with the highest probability among the multiple candidate words with the same part of speech and the candidate word with the highest probability among the multiple candidate words with different part of speech to replace the erroneous position in the corrected detection result respectively includes: Calculate the probabilities of multiple candidate words in the first category and multiple candidate words in the second category; the probabilities are used to represent the rationality of the Chinese text to be corrected after replacement with candidate words; The candidate words with the highest probability in the first category and the candidate words with the highest probability in the second category are selected to replace the erroneous positions in the detection results, respectively, to obtain the first text.

7. The method according to claim 6, characterized in that, The calculation of the first text perplexity includes: Calculate the third text perplexity of the text corresponding to the candidate word with the highest probability in the first category after replacing the wrong position, and calculate the fourth text perplexity of the text corresponding to the candidate word with the highest probability in the second category after replacing the wrong position; Determine whether the third text perplexity is greater than the fourth text perplexity. If so, determine that the fourth text perplexity is the first text perplexity; otherwise, determine that the third text perplexity is the first text perplexity.

8. The method according to any one of claims 1-7, characterized in that, Also includes: After correcting the Chinese text, the correction results are obtained, and the staff judges whether the correction results are reasonable. If reasonable, the error correction results will be visualized for users to view; If the error is not valid, an instruction message will be sent to instruct the user or staff to correct the Chinese text to be corrected.

9. A Chinese text correction device, characterized in that, The device includes: The detection module is used to acquire the Chinese text to be corrected; and to detect the error locations of the Chinese text to be corrected using a first error detection model and a second error detection model, respectively, to obtain detection results; the first error detection model is a statistical language model, and the second error detection model is a deep learning model based on sequence labeling; wherein, for repeated error locations in the detection results, the error locations detected by the second error detection model shall be used as the standard. The classification correction module is used to classify the error locations in the detection results using the second detection module, and determine whether the classification result is correct. If it is incorrect, the error type is corrected to obtain the corrected detection result. An error correction module is used to find the first priority error correction model corresponding to the error type of the corrected detection result based on a pre-stored error correction model, and use the first priority error correction model to correct the Chinese text to be corrected, obtaining a first candidate set; the first candidate set includes multiple candidate words with the same part of speech as the error position in the corrected detection result and multiple candidate words with different part of speech than the error position in the corrected detection result; the candidate word with the highest probability among the multiple candidate words with the same part of speech and the candidate word with the highest probability among the multiple candidate words with different part of speech are selected to replace the error position in the corrected detection result, respectively, to obtain a first text; the first text includes two... The first text to be corrected is replaced with the candidate word with the highest probability; the first text perplexity is calculated; the first text perplexity is determined based on the one with the lowest text perplexity among the two Chinese texts to be corrected after replacing the candidate word with the candidate word with the highest probability; it is determined whether the first text perplexity is less than the text perplexity of the Chinese text to be corrected; if so, the correction result is obtained; otherwise, a secondary priority correction model corresponding to the error type is found based on the pre-stored correction model to correct the corrected detection result, and the correction result of the Chinese text to be corrected is obtained; wherein, the secondary priority correction model corresponding to the error type found is of a different type than the first priority correction model corresponding to the error type found. Different error types correspond to different first-priority error correction models and different second-priority error correction models.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the Chinese text error correction method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes program code that, when the computer runs the computer program, performs the method as described in any one of claims 1-8.

Images