Text correction method, apparatus, medium, and device

Abstract

The application discloses a text error correction method and device, a medium and equipment, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a text to be corrected, predicting the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected; determining one or more homophonic characters of the text prediction value according to a homophonic character list, wherein the homophonic character list is obtained by processing a text corpus in advance; and correcting the target text in the text to be corrected according to the text prediction value and the one or more homophonic characters of the text prediction value. The application establishes a homophonic character list based on word frequency analysis and difference quantization, improves the reliability in the error detection and correction process, and combines model prediction with the homophonic character list to correct the text, thereby improving the accuracy of text correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text correction method and device, medium and equipment. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which makes machines have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, which involves a wide range of fields, such as natural language processing, machine learning, deep learning and several other directions. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] In the way of correcting text, based on the language prediction model, the text at a specific position is predicted according to the context. If the predicted word is different from the original word at the position, it is directly considered that the original text has an error, and then the predicted word is used to replace the text for correction. However, the prediction performance of the model is not satisfactory, and directly replacing the initial text with the predicted text has a low accuracy of correction, which is easy to correct the correct text into an error. SUMMARY

[0004] In order to improve the accuracy of text correction, the present application provides a text correction method, device, medium and equipment. The technical solution is as follows:

[0005] In a first aspect, the present application provides a text correction method, which comprises:

[0006] obtaining a text to be corrected, predicting the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected;

[0007] determining one or more similar characters of the text prediction value according to a similar character list, the similar character list being obtained by processing a text corpus in advance;

[0008] correcting the target text in the text to be corrected according to the text prediction value and the one or more similar characters of the text prediction value.

[0009] In a second aspect, the present application provides a text correction device, which comprises:

[0010] a prediction module, configured to obtain a text to be corrected, predict the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected;

[0011] The shape similar character determination module is configured to determine one or more shape similar characters of the text prediction value according to a shape similar character list, which is obtained by processing a text corpus in advance.

[0012] The error correction module is configured to correct the target text in the text to be corrected according to the text prediction value and the one or more shape similar characters of the text prediction value.

[0013] In a third aspect, the present application provides a computer readable storage medium, wherein at least one instruction or at least one program is stored in the computer readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the text error correction method according to the first aspect.

[0014] In a fourth aspect, the present application provides a computer device, wherein the computer device comprises a processor and a memory, and at least one instruction or at least one program is stored in the memory, and the at least one instruction or at least one program is loaded and executed by the processor to implement the text error correction method according to the first aspect.

[0015] The text error correction method, device, equipment and storage medium provided by the present application have the following technical effects:

[0016] (1) The scheme provided by the present application considers the factor of shape similar characters, establishes a shape similar character list based on word frequency analysis and difference quantification, and improves the reliability in the error detection and correction process;

[0017] (2) The scheme provided by the present application predicts the text by a language model, and corrects the text in combination with the shape similar character list, thereby improving the accuracy of text error correction.

[0018] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0020] Figure 1 is an implementation environment schematic diagram of a text error correction method provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart schematic diagram of a text error correction method provided by an embodiment of the present application;

[0022] Figure 3 is a flowchart of a process for obtaining a text prediction value of a target text in the text to be corrected according to an embodiment of the present application;

[0023] Figure 4 is a flowchart of a process for obtaining a list of similar characters according to an embodiment of the present application;

[0024] Figure 5 is a flowchart of a process for detecting and correcting errors in a short text according to an embodiment of the present application;

[0025] Figure 6 is a flowchart of a process for detecting and correcting errors in a short text according to an embodiment of the present application;

[0026] Figure 7 is a flowchart of a process for detecting and correcting errors in a short text according to an embodiment of the present application;

[0027] Figure 8 is a flowchart of another text correction method according to an embodiment of the present application;

[0028] Figure 9 is a schematic diagram of a text correction device according to an embodiment of the present application;

[0029] Figure 10 is a hardware structure schematic diagram of a device for implementing a text correction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] Artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies.

[0031] The scheme provided by the embodiments of the present application relates to technologies such as deep learning (DL) of artificial intelligence, nature language processing (NLP) and the like.

[0032] Deep learning (DL) is a major research direction in the field of machine learning (ML), and it is introduced into machine learning to make it closer to the original goal-artificial intelligence. Deep learning is to learn the internal rules and representation levels of sample data, and the information obtained in the learning process is very helpful for the interpretation of data such as text, images and sound. The ultimate goal of deep learning is to enable machines to have analysis and learning ability like people, and to be able to recognize text, images and sound data. Deep learning is a complex machine learning algorithm, and the effect achieved in voice and image recognition far exceeds that of previous related technologies. Deep learning has achieved a lot of results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, voice, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as vision and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0033] Nature language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Nature language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, that is, the language used in people's daily life, so it has a close relationship with the study of linguistics. Nature language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and the like.

[0034] The scheme provided by the embodiments of the present application can be deployed in the cloud, and it also relates to cloud technologies and the like.

[0035] Cloud technology: refers to the series of resources such as hardware, software and network in the wide area network or local area network are unified, realize the data calculation, storage, processing and sharing of a kind of hosting technology, also can be understood as the network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model application, can constitute resource pool, use as needed, flexible and convenient. The background service of technology network system needs a lot of computing and storage resources, such as video website, picture website and more portal website, with the high development and application of internet industry, every item may have its own identification mark in the future, which needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system backup, so cloud technology needs to be supported by cloud computing. Cloud computing is a computing mode, which distributes computing tasks on a large number of computing resources to form a resource pool, so that various application systems can obtain computing power, storage space and information service according to needs. The network providing resources is called "cloud". The resources in the "cloud" can be infinitely expanded in the eyes of the user, and can be obtained at any time, used on demand, expanded at any time, and paid according to use. As a basic ability provider of cloud computing, a cloud computing resource pool platform, referred to as cloud platform, is established, generally referred to as infrastructure as a service (IaaS), and a variety of types of virtual resources are deployed in the resource pool for external customers to select and use. The cloud computing resource pool mainly includes: computing device (which can be virtual machine, including operating system), storage device and network device.

[0036] In order to improve the accuracy of text correction, the embodiment of the application provides a text correction method, device, medium and equipment. The technical solutions in the embodiments of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the application. The examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0038] In order to facilitate the understanding of the technical solutions described in the embodiments of the present application and the technical effects generated thereby, the related professional terms involved in the embodiments of the present application are explained:

[0039] OCR: Optical Character Recognition, optical character recognition, refers to the process of checking the characters printed on paper by an electronic device (such as a scanner or digital camera), determining their shape by detecting light and dark patterns, and then translating the shape into computer text by character recognition method; that is, for printed characters, use optical method to convert the characters in paper documents into black and white dot matrix image files, and convert the characters in the image into text format through recognition software, for further editing and processing by word processing software. How to debug or use auxiliary information to improve the recognition accuracy is the most important issue of OCR.

[0040] Edit distance: refers to the quantitative measure of the difference between two strings (such as English words), which measures how many times a string needs to be processed to become another string. Edit distance can be used in natural language processing, for example, spelling check can determine which (or which) is the most likely word according to the edit distance of a misspelled word and other correct words.

[0041] BERT model: Bidirectional Encoder Representations from Transformer, the goal of BERT model is to train large-scale unlabeled corpus to obtain text representation containing rich semantic information.

[0042] Please refer to Figure 1 , which is an implementation environment diagram of a text correction method provided by the embodiments of the present application, as Figure 1 shown, the implementation environment can at least include a client 01 and a server 02.

[0043] Specifically, the client 01 can include a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, a monitoring device, a voice interaction device, and the like, and can also include a software running in the device, such as a webpage provided by a service provider to a user, and an application provided by the service provider to the user. Specifically, the client 01 can be used to obtain text input or uploaded by the user, and display the corrected text by the server 02.

[0044] Specifically, the server 02 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server 02 can include a network communication unit, a processor, a memory, and the like. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. Specifically, the server 02 can be used for text error correction, and the server 02 can connect to a database through a network to obtain a corpus in a text library to generate a list of homophonic words, and train a language model and use it for text prediction.

[0045] The embodiments of the present application can also be implemented in combination with cloud technology. Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or a local area network to realize data calculation, storage, processing and sharing. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model application. Cloud technology needs to be supported by cloud computing. Cloud computing is a computing mode that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services according to needs. The network providing resources is called "cloud". Specifically, the server 02 and the database are located in the cloud, and the server 02 can be a physical machine or a virtual machine.

[0046] A text correction method provided by the present application is introduced below. Figure 2is a flowchart of a text error correction method provided by an embodiment of the present application. The present application provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment). Please refer to Figure 2 The text error correction method provided by an embodiment of the present application can include the following steps:

[0047] S201: Obtain a text to be corrected, and predict the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected.

[0048] In an embodiment of the present application, the text to be corrected can be a directly input text that needs to be checked and corrected, or a text obtained through optical character recognition (OCR) or speech recognition.

[0049] In an embodiment of the present application, as shown in Figure 3 The obtaining of the text to be corrected and the prediction of the text to be corrected based on the language model to obtain the text prediction value of the target text in the text to be corrected can include the following steps:

[0050] S301: Train a machine learning model based on the text corpus to obtain a language model.

[0051] It can be understood that the distributed representation (Embedding) of the multi-granularity natural language unit by the deep learning technology has achieved great success in the field of natural language processing. A deep neural network model is constructed as the machine learning model based on the deep learning technology, and the model is trained through long text corpus in the text library to obtain the language model. In addition, multiple language models can be trained and tested according to factors such as the language of the corpus, the application field or scene, the language style, etc., to predict the text to be corrected in a targeted manner.

[0052] In a feasible implementation, considering that the cost required for completely retraining a model is increasing as the network deepens and the data set expands, the weight of an open-source pre-trained language model can be fine-tuned as a language model for subsequent application. For example, based on the BERT (Bidirectional Encoder Representations from Transformer) model, the model is fine-tuned using a specific field long text corpus, and the obtained model is used as the speech model in this application. It can be understood that the BERT model adopts a Masked Language Model (MLM) and a Next Sentence Prediction during pre-training. The success of the BERT model also comes from the size of the model and the amount of training data. Therefore, the BERT model has an advantage in big data. The main input of the BERT model is the original word vector of each word in the text, which can be randomly initialized or pre-trained using Word2Vector algorithm as an initial value; the output is the vector representation of each word in the text after fusing the semantic information of the whole text.

[0053] S303: determining a target text in the text to be corrected and context before or after the target text.

[0054] In the embodiments of the present application, the language model used can be predicted according to the context of the target text. For example, the language model obtained by fine-tuning the BERT model can predict the text at a specific position through the context before and after the text, that is, a bidirectional prediction method. In some other language models that can be used, a unidirectional prediction method can be used, such as predicting the text at a specific position only according to the context before or only according to the context after.

[0055] Specifically, the target text at each position in the text to be corrected is traversed, and the context before or after the target text is selected according to a preset step length. For example, in a Chinese long sentence, for each Chinese character in the long sentence, the two Chinese characters before or after each Chinese character are taken as the input of a language model, and the step length is 2. For the text at the beginning or end of the long sentence, the context before or after can be supplemented by adding a default value.

[0056] S305: based on the language model, predicting according to the context before or after the target text to obtain a text prediction value of the target text in the text to be corrected.

[0057] Specifically, while obtaining the text prediction value, a confidence of the text prediction value is also given, the confidence represents a confidence degree of the text prediction value, and the confidence can be used as a judgment basis in the error correction process. For example, if the confidence of the text prediction value is lower than a preset threshold, the text prediction value is not used for error detection or error correction, and it can be directly considered that the target text is not wrong, or other methods are used for error detection and correction of the target text. For example, the confidence can also be used for judgment according to the result of whether the target text and the corresponding text prediction value are the same or different, which is not described herein again.

[0058] S203: determining one or more near-form words of the text prediction value according to a near-form word list, the near-form word list being obtained by processing a text corpus in advance.

[0059] In the embodiments of the present application, while using the language model, the near-form word factor is also considered, and the reliability and accuracy of error detection and correction are improved by generating the near-form word list. The near-form word list includes one or more near-form word pairs.

[0060] In the embodiments of the present application, before determining one or more near-form words of the text prediction value according to the near-form word list, the method can further include:

[0061] The near-form word list is obtained by frequency analysis and difference quantification of the text corpus.

[0062] In the embodiments of the present application, the number of occurrences of the corpus in the text corpus can be counted, the corpus is divided into high-frequency corpus and low-frequency corpus according to a preset frequency threshold, and the high-frequency corpus is regarded as accurate and reliable standard corpus. Then, by measuring the difference degree between the high-frequency corpus and the low-frequency corpus, a corpus pair of the high-frequency corpus and the low-frequency corpus is constructed, and the difference degree is also the similarity of the text, which can be represented by the literal distance of the text. For example, the literal distance between two texts can be determined by calculating the edit distance (Levenshtein distance), Jaro distance, SimHash (a kind of local sensitive hash), etc. In a corpus pair, the text causing the difference can constitute a near-form word pair, and a near-form word list composed of one or more near-form word pairs can be obtained by circulating the high-frequency corpus.

[0063] Specifically, as shown in Figure 4 The near-form word list can be obtained by frequency analysis and difference quantification of the text corpus, which can include the following steps:

[0064] S401: obtaining short text corpus in a text corpus.

[0065] In the embodiments of the present application, corpus is the language material of the text library, and is the basic unit of the text library. The corpus can be a word, a Chinese character, a vocabulary, an idiom, a phrase, a short sentence, a long sentence, a poem, an article, etc. According to the number of the smallest unit of text in the corpus, the corpus can be divided into short text and long text. For example, the corpus with no more than 5 Chinese characters is regarded as short text corpus, and the corpus with no less than 10 Chinese characters is regarded as long text corpus. For example, “from this it can be seen” and “I don't know” are short text corpus, and “at the XX press conference, the spokesperson again responded to the question of XXX” is long text corpus.

[0066] Exemplarily, the text with a length of 3 to 5 Chinese characters is selected from the Chinese text library as the short text corpus used in the embodiments of the present application. In addition, the long text can also be divided into short texts based on separators or preset length moving boxes, etc. The long text contains more text and more semantic content, and the frequency difference is not obvious compared with the short text. Therefore, the short text corpus can be preferably used as the original data. In addition, the short text corpus can also be screened according to the language style, application field, etc.

[0067] S403: Counting and sorting the word frequency of the short text corpus to obtain the first vocabulary set and the second vocabulary set.

[0068] In a feasible implementation, the short text corpus is counted, the number of occurrences of each vocabulary in the short text corpus, i.e. the word frequency, is analyzed, and all short texts in the short text corpus are sorted based on the word frequency as an index to obtain a sorted list of short texts from high frequency to low frequency.

[0069] In a feasible implementation, after obtaining the sorted list of short texts from high frequency to low frequency, the short texts are divided according to a preset word frequency threshold. The short texts can be divided into at least two categories. The short texts with a word frequency not lower than the preset word frequency threshold are classified into the first vocabulary set, and the short texts with a word frequency lower than the preset word frequency threshold are classified into the second vocabulary set. For example, the sum of the word frequencies of all short texts is calculated, and the average value is calculated. The double of the average value is taken as the word frequency threshold. The short texts with a word frequency higher than the word frequency threshold are high-frequency words, and the short texts with a word frequency lower than the word frequency threshold are low-frequency words. In some necessary application scenarios, a small amount of manual review can be performed to determine the final high-frequency vocabulary and low-frequency vocabulary to obtain the first vocabulary set and the second vocabulary set.

[0070] Further, when counting and sorting the word frequency of the short text corpus, the short text corpus is first classified according to the text length, language, application field, etc. to obtain the first vocabulary set and the second vocabulary set under each subcategory.

[0071] S405: For a first target vocabulary in the first vocabulary set, a second target vocabulary in the second vocabulary set is obtained, where a difference degree of the first target vocabulary and the second target vocabulary satisfies a preset condition.

[0072] In the embodiments of the present application, for a high-frequency vocabulary, an error-prone vocabulary of the high-frequency vocabulary is determined in the low-frequency vocabulary set, and a probability of error occurrence is determined according to the difference degree of the analyzed vocabulary.

[0073] In a feasible implementation, obtaining the second target vocabulary in the second vocabulary set, where the difference degree of the first target vocabulary and the second target vocabulary satisfies the preset condition, can include the following steps:

[0074] S4051: Calculate an edit distance between each second vocabulary in the second vocabulary set and the first target vocabulary.

[0075] It can be understood that the edit distance refers to the minimum number of editing operations required to convert one string into another. The permitted editing operations include replacing one character with another character, inserting a character, and deleting a character. For example, for a high-frequency vocabulary ABC in the first vocabulary set, A, B, and C can refer to a Chinese character, and a low-frequency vocabulary ABE in the second vocabulary set, changing ABE to ABC only needs one processing, i.e., E is changed to B, and the edit distance between ABC and ABE is 1. In English text, in addition to changing a letter, one letter can also be added or deleted, which is not described here.

[0076] S4053: Determine the second target vocabulary as the second vocabulary whose edit distance satisfies the preset condition.

[0077] In a feasible implementation, the preset condition is a distance condition, the edit distance corresponding to each second vocabulary is compared with a preset edit distance threshold, the each second vocabulary is screened according to the comparison result, and the second target vocabulary is determined. For example, for Chinese text, the preset edit distance threshold is 2, and the second vocabulary whose edit distance is not more than 2 is taken as the second target vocabulary of the first target vocabulary, that is, the first target vocabulary is changed to the second target vocabulary only by changing one or two Chinese characters.

[0078] In another possible implementation, the preset condition can include a quantity condition in addition to the distance condition. The second vocabulary is screened in the first round according to the distance condition, and is screened or supplemented in the second round according to the quantity condition. For example, there are 10 second vocabularies with an edit distance of no more than 2, including 4 second vocabularies with an edit distance of 1 and 6 second vocabularies with an edit distance of 2. If the preset quantity threshold is 3, 3 second vocabularies with an edit distance of 1 are selected preferentially, and if the preset quantity threshold is 5, one second vocabulary with an edit distance of 2 is selected from the 6 second vocabularies as a second target vocabulary in addition to the 4 second vocabularies with an edit distance of 1, which can be selected randomly or based on a vocabulary frequency or the like, and the present application does not make a specific limitation in this regard. In addition, if there are less than 3 second vocabularies with an edit distance of no more than 2, the preset edit distance threshold can be enlarged by a certain percentage to determine the second target vocabulary again.

[0079] In another possible implementation, the preset condition can include other screening conditions in addition to the distance condition, such as a language type condition, a text type condition or an application field condition. The vocabularies in the second vocabulary set are screened according to the other screening conditions to obtain a key second vocabulary set, and then the second target vocabulary is determined in the key second vocabulary set. For example, the second vocabularies are screened in the first round according to whether the application field is a living language, and then the edit distance of each second vocabulary is compared with the preset edit distance threshold in the second vocabularies screened in the first round, and each second vocabulary is screened according to the comparison result to determine the second target vocabulary.

[0080] S407: determining a difference text according to the first target vocabulary and the second target vocabulary, and generating at least one set of homophone pairs according to the difference text.

[0081] In the embodiments of the present application, for the first target vocabulary and the second target vocabulary with a difference degree satisfying a preset condition, the text causing the difference can be regarded as a set of homophone pairs, and the set of homophone pairs includes a standard text and a confused text. For example, for the first target vocabulary ABC and the second target vocabulary ABE, where A, B, C and E are Chinese characters, the difference text causing the difference can be determined as C and E, and C and E constitute a set of homophone pairs, where C is the standard text and E is the error text of C or the confused text of C. In English texts, the minimum granularity of the difference text can be a letter, and the finally generated set of homophone pairs can be a standard word and a confused word.

[0082] S409: traversing all vocabularies in the first vocabulary set to determine the homophone pairs of all vocabularies in the first vocabulary set, and obtaining a homophone list.

[0083] In the embodiments of the present application, a series of similar character pairs can be obtained by circulating the high-frequency word list, i.e., the first vocabulary set, and finally a similar character list is formed. In a feasible implementation, the series of similar character pairs are de-duplicated and integrated, or before de-duplication, the multiple similar character pairs of the same standard text are set with corresponding weights for subsequent error detection and correction.

[0084] In some feasible implementations, different error detection and correction methods can be used for long texts and short texts in the text to be corrected. For short texts with too few words, the model predicts less context information, and the prediction effect is not good, so the error vocabulary candidate set obtained based on the similar character list can be used to directly correct the short text. Specifically, as shown in Figure 5 the following steps can be included:

[0085] S402: generating an error vocabulary candidate set according to the similar character list.

[0086] In the embodiments of the present application, the vocabularies in the error vocabulary candidate set can be regarded as confused vocabularies of each high-frequency vocabulary in the first vocabulary set. The confused vocabularies are obtained by combining the similar characters of each text in the high-frequency vocabulary, which are used as the basis for short text correction.

[0087] Specifically, the step of generating an error vocabulary candidate set according to the similar character list can include the following steps:

[0088] S4021: for a first target vocabulary in the first vocabulary set, obtaining all similar character pairs of each text in the first target vocabulary from the similar character list.

[0089] It can be understood that for a given high-frequency vocabulary (i.e., the first vocabulary), each text in the high-frequency vocabulary can have its own similar character. For example, for a given high-frequency vocabulary BCI, the similar character of B is E, the similar character of C is G, and the similar character of I is L and J.

[0090] S4023: combining the all similar character pairs to obtain confused vocabularies of the first target vocabulary.

[0091] In the embodiment of the present application, when there are similar characters at each position in the first target word, the combination of similar characters can be regarded as a confused word of the first target word. It should be noted that in the process of combining similar characters, the similar characters of each text are combined according to the position sequence of each text in the first target word. As described above, for a given high-frequency word BCI, the similar characters of each text are combined by cross-multiplication, and all possible error words can be obtained, including: BCL, BCJ, BGI, ECI, EGI, BGL, BGJ, ECL, ECJ, EGL and EGJ, and the 11 words are the confused words of the given high-frequency word BCI.

[0092] S4025: traversing all words in the first word set to determine the confused words of all words in the first word set, generating an error word candidate set, and the word pair in the error word candidate set includes a standard word and a corresponding confused word.

[0093] In an embodiment of the present application, in the error word candidate set, the word pair can be a standard word and at least one confused word of the standard word, or a confused word and its corresponding standard word. In a feasible implementation, a confused word can also correspond to multiple standard words, that is, high-frequency words, and at this time, the weight of each high-frequency word can be set by calculating the probability of each high-frequency word becoming the confused word by error, etc.

[0094] S404: obtaining a short text in the text to be corrected, and performing error detection and correction on the short text according to the error word candidate set.

[0095] In the embodiment of the present application, considering that the model needs the context before or after when predicting, and the amount of context before or after the short text to be corrected is not enough, the effect of model prediction is not good, therefore, other preferred error correction methods can be adopted for the short text of the text to be corrected, that is, whether the short text is a confused word in the error candidate word set is confirmed, and if so, the short text is modified to the standard word corresponding to the confused word.

[0096] In a feasible implementation, specifically, as shown in Figure 6 The step of obtaining a short text in the text to be corrected and performing error detection and correction on the short text according to the error word candidate set can include the following steps:

[0097] S4041: obtaining a short text in the text to be corrected.

[0098] In the embodiment of the present application, the short text can be determined from the text to be corrected by a delimiter such as a punctuation mark.

[0099] S4043: For the target short text with a word frequency lower than the preset word frequency threshold in the short text, it is judged whether the short text is a confused word in the error word candidate set.

[0100] Specifically, first, it can be judged whether the short text is a low-frequency word according to the short text from high frequency to low frequency sorting list and the word frequency threshold, if it is a high-frequency word, the error detection step is not executed, if it is a low-frequency word, it can be judged whether the short text is a confused word in the error word candidate set through character matching or the like.

[0101] S4045: If the target short text is a confused word in the error word candidate set, the target short text is corrected to the standard word corresponding to the confused word according to the error word candidate set.

[0102] In an embodiment of the present application, in the error word candidate set, the word pair can be a standard word and at least one confused word of the standard word, or can be a confused word and the standard word corresponding thereto. According to the corresponding relationship in the error word candidate set, the short text is corrected to the corresponding standard word.

[0103] S4047: If the target short text is not a confused word in the error word candidate set, the target short text is not corrected.

[0104] Taking the example of the short text in step S4023, for the short texts EGL and EGJ, the error word candidate set can be found, and the corresponding standard word is a high-frequency word BCI, then the first short text EGL and EGJ is corrected to BCI, that is, the intelligent correction of the short text can be completed.

[0105] S205: Correcting the target text in the text to be corrected according to the text prediction value and one or more near-homophones of the text prediction value.

[0106] In an embodiment of the present application, it is judged whether the target text is a near-homophone of the text prediction value, and on the basis of the model prediction, the near-homophone is combined to improve the accuracy of the correction of the target text.

[0107] In a feasible implementation manner, specifically, as shown in Figure 7 The step of correcting the target text in the text to be corrected according to the text prediction value and one or more near-homophones of the text prediction value can include the following steps:

[0108] S501: Judging whether the target text is a near-homophone of the text prediction value according to one or more near-homophones of the text prediction value.

[0109] In the embodiments of the present application, in addition to the text prediction value of the reference language model, the factor of similar characters is also considered, and the combination of the two can improve the accuracy and reliability of text correction.

[0110] S503: If the target text is a similar character of the text prediction value, the target text is corrected to the text prediction value.

[0111] In the embodiments of the present application, if the target text is a similar character of the text prediction value, that is, in the case that the target text is different from the text prediction value, the target text is checked for error by considering whether the target text is a similar character of the text prediction value. If the target text is a similar character of the text prediction value, the credibility of the text prediction value is increased to a certain extent, and the accuracy of replacing the target text with the text prediction value is improved. Further, whether the text needs to be corrected can be determined according to the confidence of the text prediction value. If the target text is a similar character of the text prediction value and the confidence of the text prediction value is higher than a preset confidence threshold, it can be considered that the target text has an error, and the target text needs to be replaced with the text prediction value. If the confidence of the text prediction value is lower than the preset confidence threshold, even if the target text is a similar character of the text prediction value, the text is not corrected.

[0112] S505: If the target text is not a similar character of the text prediction value, the target text is not corrected.

[0113] In the embodiments of the present application, if the target text is not a similar character of the text prediction value, it can be considered that the target text has no error and does not need to be modified. It can be understood that the target text can be the text prediction value, in which case the target text is definitely not a similar character of the text prediction value, and thus the text does not need to be corrected. The target text can also be different from the text prediction value, in which case the target text is also not a similar character of the text prediction value, and thus the text does not need to be corrected, which can reduce the case that the correct target text is replaced with the incorrect text prediction value in the case that the prediction effect of the model is poor.

[0114] In a feasible implementation, according to the previous BC (B and C can refer to a text character) in the text, the language model predicts that the text at the next position should be I, and the confidence of I is 0.8, which is greater than a preset threshold 0.7. The similar characters of I include L and J. If the target text at the position in the text to be corrected is L, it can be considered that L is an incorrect text, and the high-confidence predicted text I is used to replace L to complete the correction. If the target text at the position in the text to be corrected is K, which is neither the text prediction value I nor the similar characters L and J of I, even if the confidence of I is greater than the preset threshold, the text is not modified.

[0115] As Figure 8As shown, the text error correction method provided by the embodiment of the present application can also be used to adopt different error correction manners for short texts and long texts in the text to be corrected. A list of homophonic characters between high-frequency words and low-frequency words is established through word frequency analysis of a text corpus, the list of homophonic characters is combined to obtain a candidate set of error words, and then intelligent error correction is performed on the short text; for the long text, whether there is an error is judged based on the prediction of the text by the language model in combination with the list of homophonic characters, and then error correction is performed on the long text, and the accuracy of text error correction can be further improved by the method provided by the present application to more accurately identify the user's intention and serve subsequent application services.

[0116] The embodiment of the present application also provides a text error correction device 900, as shown in the accompanying drawings, Figure 9 The device can include:

[0117] A prediction module 910 is configured to obtain text to be corrected, and predict the text to be corrected based on a language model to obtain a text prediction value of a target position in the text to be corrected.

[0118] A homophonic character determination module 920 is configured to determine one or more homophonic characters of the text prediction value according to a list of homophonic characters, the list of homophonic characters being obtained by processing a text corpus in advance.

[0119] An error correction module 930 is configured to correct the target text in the text to be corrected according to the text prediction value and the one or more homophonic characters of the text prediction value.

[0120] In an embodiment of the present application, the device can further include:

[0121] A list of homophonic characters generation module is configured to obtain the list of homophonic characters by performing word frequency analysis and difference quantification on a text corpus.

[0122] In a feasible implementation, the list of homophonic characters generation module can include:

[0123] A corpus acquisition unit is configured to acquire short text corpus in a text corpus;

[0124] A word frequency analysis unit is configured to perform word frequency statistics and sorting on the short text corpus to obtain a first word set and a second word set;

[0125] A word selection unit is configured to, for a first target word in the first word set, acquire a second target word from the second word set, the difference degree of the second target word and the first target word satisfying a preset condition;

[0126] The similar character group pair unit is configured to determine a difference text according to the first target vocabulary and the second target vocabulary, and generate at least one similar character pair according to the difference text.

[0127] The similar character list unit is configured to traverse all vocabularies in the first vocabulary set, determine similar character pairs of all vocabularies in the first vocabulary set, and obtain a similar character list.

[0128] In an embodiment of the present application, the vocabulary selection unit can further include:

[0129] The calculation sub-unit is configured to calculate an edit distance between each second vocabulary in the second vocabulary set and the first target vocabulary.

[0130] The screening sub-unit is configured to determine a second vocabulary with an edit distance satisfying a preset condition as a second target vocabulary.

[0131] In an embodiment of the present application, the prediction module 910 can include:

[0132] The model training unit is configured to train a machine learning model to obtain a language model through the text corpus.

[0133] The text determination unit is configured to determine a target text in the text to be corrected and context before or after the target text.

[0134] The model prediction unit is configured to predict, based on the language model, according to the context before or after the target text, to obtain a text prediction value of the target text in the text to be corrected.

[0135] In an embodiment of the present application, the text correction apparatus 900 can further include:

[0136] The error vocabulary module is configured to generate an error vocabulary candidate set according to the similar character list.

[0137] The short text correction module is configured to obtain a short text in the text to be corrected, and perform error detection and correction on the short text according to the error vocabulary candidate set.

[0138] It should be noted that the apparatus provided in the above embodiments, in realizing its functions, only takes the above-mentioned division of each functional module as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0139] The embodiment of the present application provides a computer device, which comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the text correction method provided in the above method embodiment.

[0140] Figure 10 A hardware structure schematic diagram of a device for implementing the text correction method provided in the embodiment of the present application is shown, and the device can participate in constituting or containing the apparatus or system provided in the embodiment of the present application. As shown in the figure, Figure 10 The device 10 can comprise one or more processors 1002 (the processor 1002 can comprise but is not limited to a microprocessor MCU or a programmable logic device FPGA and the like processing device), a memory 1004 for storing data, and a transmission device 1006 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 10 is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the device 10 can also comprise more or less components than those shown in Figure 10 , or have a different configuration from Figure 10 .

[0141] It should be noted that the one or more processors 1002 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the device 10 (or mobile device). As referred to in the embodiment of the present application, the data processing circuit controls as a processor (for example, the selection of the variable resistance terminal path connected with the interface).

[0142] The memory 1004 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the method in the embodiments of the present application, and the processor 1002 can execute various functional applications and data processing, i.e., implement the above-described text correction method, by running the software programs and modules stored in the memory 1004. The memory 1004 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 1004 can further include a memory remotely arranged with respect to the processor 1002, which can be connected to the device 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0143] The transmission device 1006 is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 1006 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0144] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the device 10 (or mobile device).

[0145] The embodiments of the present application also provide a computer readable storage medium, which can be arranged in a server to save at least one instruction or at least one program related to a text correction method in the method embodiments, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the text correction method provided by the above-mentioned method embodiments.

[0146] Optionally, in the present embodiment, the above-mentioned storage medium can be located in at least one of a plurality of network servers of a computer network. Optionally, in the present embodiment, the above-mentioned storage medium can include, but is not limited to, a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing program codes.

[0147] The embodiment of the present application further provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the text error correction method provided in the various optional embodiments.

[0148] From the above, it can be seen that the embodiments of the text error correction method, device, medium and equipment provided by the present application,

[0149] (1) The scheme provided by the present application considers the factor of similar forms of words, establishes a similar form word list based on word frequency analysis and difference quantization, and improves the reliability in the error detection and correction process.

[0150] (2) The scheme provided by the present application predicts the text through a language model, and corrects the text in combination with the similar form word list, thereby improving the accuracy of text error correction.

[0151] It should be noted that the above-mentioned embodiments of the present application are in the order of description only, and do not represent the advantages and disadvantages of the embodiments. The above describes the specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or can be advantageous.

[0152] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0153] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0154] The above is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of text correction, characterized by, The method comprises: acquiring a text to be corrected, predicting the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected; determining one or more homophonic characters of the text prediction value according to a homophonic character list, the homophonic character list being obtained by previously performing word frequency analysis and difference quantization processing on a text corpus; correcting the target text in the text to be corrected according to the text prediction value and the one or more homophonic characters of the text prediction value; wherein the word frequency analysis and difference quantization processing on the text corpus comprises: acquiring short text corpus in a text corpus; performing word frequency statistics and sorting on the short text corpus to obtain a first vocabulary set and a second vocabulary set; for a first target vocabulary in the first vocabulary set, acquiring a second target vocabulary from the second vocabulary set, the difference degree of which with the first target vocabulary satisfying a preset condition; determining a difference text according to the first target vocabulary and the second target vocabulary, and generating at least one group of homophonic character pairs according to the difference text; traversing all vocabularies in the first vocabulary set to determine the homophonic character pairs of all vocabularies in the first vocabulary set, and obtaining the homophonic character list.

2. The method of claim 1, wherein, The method further comprises: calculating the edit distance of each second vocabulary in the second vocabulary set from the first target vocabulary; determining the second vocabulary whose edit distance satisfies the preset condition as the second target vocabulary.

3. The method of claim 1, wherein, The method further comprises: training a machine learning model to obtain a language model through the text corpus; determining a target text in the text to be corrected and the context of the target text; predicting the context of the target text based on the language model to obtain a text prediction value of the target text in the text to be corrected.

4. The method of claim 1, wherein, The method further comprises: determining whether the target text is a homophonic character of the text prediction value according to the one or more homophonic characters of the text prediction value; if the target text is a homophonic character of the text prediction value, correcting the target text to the text prediction value.

5. The method of claim 1, wherein, The method further comprises: for a first target vocabulary in the first vocabulary set, acquiring all homophonic character pairs of each text in the first target vocabulary from the homophonic character list; combining the all homophonic character pairs to obtain a confused vocabulary of the first target vocabulary; traversing all vocabularies in the first vocabulary set to determine the confused vocabulary of all vocabularies in the first vocabulary set, generating a candidate set of error vocabularies, and the vocabulary pair in the candidate set of error vocabularies comprising a standard vocabulary and a corresponding confused vocabulary; The short text in the text to be corrected is obtained, and the short text is detected and corrected according to the candidate set of error vocabularies.

6. The method of claim 5, wherein, The obtaining of the short text in the text to be corrected and the detection and correction of the short text according to the candidate set of error vocabularies comprises: Obtaining a short text in the text to be corrected; For a target short text with a word frequency lower than a preset word frequency threshold in the short text, if the target short text is a confused vocabulary in the candidate set of error vocabularies, the target short text is corrected to a standard vocabulary corresponding to the confused vocabulary according to the candidate set of error vocabularies.

7. A text correction apparatus characterized by comprising: The device comprises: A prediction module is configured to obtain a text to be corrected, and predict the text to be corrected based on a language model to obtain a text prediction value of a target text in the text to be corrected; A similar character determination module is configured to determine one or more similar characters of the text prediction value according to a similar character list, the similar character list being obtained by previously performing word frequency analysis and difference quantization processing on a text corpus; A correction module is configured to correct the target text in the text to be corrected according to the text prediction value and the one or more similar characters of the text prediction value; The device further comprises: A similar character list generation module is configured to obtain the similar character list by performing word frequency analysis and difference quantization on a text corpus; The similar character list generation module can comprise: A corpus acquisition unit is configured to obtain short text corpus in a text corpus; A word frequency analysis unit is configured to perform word frequency statistics and sorting on the short text corpus to obtain a first vocabulary set and a second vocabulary set; A vocabulary selection unit is configured to, for a first target vocabulary in the first vocabulary set, obtain a second target vocabulary from the second vocabulary set, the difference degree of the second target vocabulary and the first target vocabulary satisfying a preset condition; A similar character pair unit is configured to determine a difference text according to the first target vocabulary and the second target vocabulary, and generate at least one similar character pair according to the difference text; A similar character list unit is configured to traverse all vocabularies in the first vocabulary set, determine similar character pairs of all vocabularies in the first vocabulary set, and obtain the similar character list.

8. The apparatus of claim 7, wherein, The vocabulary selection unit comprises: A calculation subunit is configured to calculate the edit distance between each second vocabulary in the second vocabulary set and the first target vocabulary; A screening subunit is configured to determine a second target vocabulary with an edit distance satisfying a preset condition.

9. The apparatus of claim 7, wherein, The prediction module comprises: A model training unit is configured to train a machine learning model to obtain a language model through the text corpus; A text determination unit is configured to determine a target text in the text to be corrected and the context of the target text; A model prediction unit is configured to predict the context of the target text based on the language model to obtain a text prediction value of the target text in the text to be corrected.

10. The apparatus of claim 7, wherein, The error correction module is configured to determine whether the target text is a homophone of the text prediction value according to one or more homophones of the text prediction value; and if the target text is a homophone of the text prediction value, correct the target text to the text prediction value.

11. The apparatus of claim 7, wherein, The device further comprises: an error vocabulary module configured to generate an error vocabulary candidate set according to the homophone list, including: obtaining all homophone pairs of each text in a first target vocabulary in the first vocabulary set from the homophone list; combining the all homophone pairs to obtain a confused vocabulary of the first target vocabulary; traversing all vocabularies in the first vocabulary set to determine confused vocabularies of all vocabularies in the first vocabulary set, and generating the error vocabulary candidate set, wherein each pair of vocabularies in the error vocabulary candidate set includes a standard vocabulary and a corresponding confused vocabulary; a short text error correction module configured to obtain a short text in the text to be corrected, and perform error detection and correction on the short text according to the error vocabulary candidate set.

12. The apparatus of claim 11, wherein, The short text error correction module is configured to obtain a short text in the text to be corrected; for a target short text with a word frequency lower than a preset word frequency threshold in the short text, if the target short text is a confused vocabulary in the error vocabulary candidate set, correct the target short text to a standard vocabulary corresponding to the confused vocabulary according to the error vocabulary candidate set.

13. A computer readable storage medium, characterized in that, The computing readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the text error correction method according to any one of claims 1 to 6.

14. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the text error correction method according to any one of claims 1 to 6.

Images