Text error correction method, device, electronic device and storage medium
Through the methods of word segmentation, dictionary screening and language model scoring, the problems of the accuracy and efficiency of existing text error correction methods are solved, and higher accuracy and efficient text error correction effects are achieved.
Patent Information
- Application Number
- CN202010914967.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-09-03
AI Technical Summary
The existing text error correction methods have low accuracy and low efficiency, especially the method based on the easy-to-miscellaneous dictionary, and the deep learning methods require a large number of wrong labeling corpus for training, which is inefficient.
By obtaining the text to be corrected for word segmentation, using the preset dictionary to determine suspected wrong words, determining candidate phrases based on suspected wrong words, and using the pre-trained language model to score candidate error correction sentences to determine corrective words.
It improves the accuracy and efficiency of text error correction, uses dictionary to quickly locate wrong words, expands the scope of replacement for error correction, and incorporates semantic information to improve the accuracy of error correction.
Smart Images

Figure CN112016304B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text error correction method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the development of computers, many texts are no longer written with pens, but are input and printed by computers. However, many errors often occur in the process of text writing, such as incorrect use of words (short board is written as short class), grammatical errors (the words "di" and "de" are mixed up), and incorrect word use (support decision-making is written as support decision-making).
[0003] There are two main types of text error correction methods. One type is divided into two stages: error recognition and error correction. Error recognition refers to locating the position of the error in the sentence, and error correction refers to finding all possible alternative words for the located error and finding the most appropriate word to replace and correct it. The second type uses deep learning methods to complete the error recognition and error correction steps at the same time, and outputs the corrected sentence results. However, of the above two types of methods, the first type is mainly based on a dictionary of easily confused words, and has a low accuracy rate; the second type requires the collection of a large amount of error-annotated corpus for training, which is inefficient. Summary of the invention
[0004] The present invention provides a text error correction method, device, electronic device and computer-readable storage medium, the main purpose of which is to provide a text error correction method with higher accuracy and higher efficiency.
[0005] To achieve the above object, the present invention provides a text error correction method, comprising:
[0006] Obtaining a text to be corrected, and performing word segmentation on the text to be corrected to obtain a word set;
[0007] Using a preset dictionary to determine suspected wrong words in the word set;
[0008] Determining a candidate phrase set that can be used to replace the suspected wrong word according to the suspected wrong word;
[0009] Using each candidate phrase in the candidate phrase set to replace the suspected wrong word in the text to be corrected, to obtain a candidate correction sentence corresponding to each candidate phrase;
[0010] The candidate error correction sentences are scored using a pre-trained language model, and the suspected wrong words are corrected according to the scores.
[0011] Optionally, the segmenting the text to be corrected to obtain a word set includes:
[0012] Dividing the text to be corrected by sentences to obtain a set of sentences to be corrected;
[0013] Each sentence in the set of sentences to be corrected is segmented to obtain a word set.
[0014] Optionally, the word segmentation of each sentence in the set of sentences to be corrected includes:
[0015] Segment the sentence using a prefix segmentation dictionary to obtain an original word set in one or more segmentation modes;
[0016] According to the original word set, a directed acyclic graph is constructed by connecting the words end to end;
[0017] A dynamic programming algorithm is used to calculate the maximum probability path in the directed acyclic graph, and a word segmentation result is obtained by selecting a partitioning method corresponding to the maximum probability path.
[0018] Optionally, the using a preset dictionary to determine the suspected wrong words in the word set includes:
[0019] Selecting a word from the word set;
[0020] Check whether the word exists in a preset common word dictionary and scene-specific word dictionary;
[0021] When the word does not exist in the common word dictionary and the scene professional word dictionary, the word is classified as a suspected wrong word.
[0022] Optionally, the determining, according to the suspected wrong word, a candidate phrase set that can be used to replace the suspected wrong word includes:
[0023] Splitting the suspected wrong word by characters to obtain one or more suspected wrong characters;
[0024] Finding homophones and homographs corresponding to the suspected wrong characters in a preset homophone table and homograph table to obtain a candidate character set;
[0025] According to the suspected wrong word, the characters in the candidate character set are combined to form words to obtain a candidate word set.
[0026] Optionally, the using a pre-trained language model to score the candidate error correction sentences, and determining to correct the suspected wrong words according to the scores, includes:
[0027] Calculating the probability value of each of the candidate error correction sentences;
[0028] Calculate the score of each candidate error correction sentence using a preset scoring algorithm according to the probability value;
[0029] The candidate phrase corresponding to the candidate error correction sentence with the highest score is selected to correct the suspected wrong word.
[0030] Optionally, the scoring algorithm includes:
[0031]
[0032] Where P is the score; N is the total number of candidate error correction sentences; q(x i ) is the probability value of the i-th candidate error correction sentence output by the language model.
[0033] In order to solve the above problem, the present invention also provides a text error correction device, the device comprising:
[0034] A word segmentation module is used to obtain the text to be corrected and perform word segmentation on the text to be corrected to obtain a word set;
[0035] A suspected wrong word determination module is used to determine suspected wrong words in the word set using a preset dictionary;
[0036] A candidate phrase set acquisition module is used to determine a candidate phrase set that can be used to replace the suspected wrong word according to the suspected wrong word;
[0037] A module for obtaining candidate error correction sentences is used to replace the suspected wrong words in the text to be corrected with each candidate phrase in the candidate phrase set to obtain a candidate error correction sentence corresponding to each candidate phrase;
[0038] The wrong word correction module is used to score the candidate error correction sentences using a pre-trained language model, and determine to correct the suspected wrong word according to the score.
[0039] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0040] a memory storing at least one computer program instruction; and
[0041] A processor executes computer program instructions stored in the memory to implement any one of the above-mentioned text error correction methods.
[0042] In order to solve the above problems, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement any one of the above text error correction methods.
[0043] The embodiment of the present invention obtains a text to be corrected, and performs word segmentation on the text to be corrected to obtain a word set, and divides the text into words, so as to facilitate subsequent detection and correction of the words; a preset dictionary is used to determine suspected wrong words in the word set, and the dictionary can be used to quickly locate possible wrong words, thereby improving detection efficiency; a candidate phrase set that can be used to replace the suspected wrong words is determined according to the suspected wrong words, and all possible candidate word combinations are found, so as to expand the replaceable range of the suspected wrong words and improve the accuracy of error correction; each candidate phrase in the candidate phrase set is used to replace the suspected wrong words in the text to be corrected, and a candidate error correction sentence corresponding to each candidate phrase is obtained, and the candidate phrase is placed in the original sentence, so as to facilitate subsequent judgment of the rationality of the match between the candidate words and the original sentence; the candidate error correction sentence is scored using a pre-trained language model, and the suspected wrong words are corrected according to the score, and the language model can be used to integrate semantic information to score the rationality of the candidate error correction sentence, so as to improve the accuracy of error correction and improve efficiency. Therefore, the text error correction method, device and computer-readable storage medium proposed in the present invention can improve the accuracy and efficiency of text error correction. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A flowchart of a text error correction method provided by an embodiment of the present invention;
[0045] Figure 2 A schematic diagram of a flow chart of a word segmentation method provided by an embodiment of the present invention;
[0046] Figure 3 A schematic diagram of a flow chart of a method for determining suspected wrong words provided by an embodiment of the present invention;
[0047] Figure 4 A schematic diagram of a flow chart of a method for determining candidate phrases provided by an embodiment of the present invention;
[0048] Figure 5 A schematic diagram of a flow chart of a method for determining candidate error correction sentences provided by an embodiment of the present invention;
[0049] Figure 6 A flow chart of a sentence scoring method provided by an embodiment of the present invention;
[0050] Figure 7 A schematic diagram of a module of a text error correction device provided by an embodiment of the present invention;
[0051] Figure 8 A schematic diagram of the internal structure of an electronic device for implementing a text error correction method provided by an embodiment of the present invention;
[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0053] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0054] The execution subject of the text error correction method provided in the embodiment of the present application includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the text error correction method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.
[0055] Reference Figure 1 FIG. 1 is a flow chart of a text error correction method provided by an embodiment of the present invention. In this embodiment, the text error correction method includes:
[0056] S1. Obtain a text to be corrected, and segment the text to be corrected to obtain a word set.
[0057] Preferably, the text to be corrected is a computer document type, mainly used to record and store text information. In the embodiment of the present invention, the text to be corrected can be various documents written by the user, or documents such as notices.
[0058] In detail, the obtaining of the text to be corrected includes: receiving the text to be corrected uploaded by the user; obtaining the text to be corrected from a preset database; downloading the text to be corrected from a web page; and / or obtaining the text to be corrected from a node of the blockchain.
[0059] For details, see Figure 2 As shown, the text to be corrected is segmented to obtain a word set, including:
[0060] S10, dividing the text to be corrected by sentences to obtain a set of sentences to be corrected;
[0061] S11, segmenting each sentence in the set of sentences to be corrected to obtain a word set.
[0062] Furthermore, the embodiment of the present invention uses a word segmentation technology based on a prefix word segmentation dictionary to segment each sentence in the set of sentences to be corrected. The specific implementation steps of the word segmentation include:
[0063] Segment the sentence using a prefix segmentation dictionary to obtain an original word set in one or more segmentation modes;
[0064] According to the original word set, a directed acyclic graph is constructed by connecting the words end to end;
[0065] A dynamic programming algorithm is used to calculate the maximum probability path in the directed acyclic graph, and a word segmentation result is obtained by selecting a partitioning method corresponding to the maximum probability path.
[0066] S2. Determine suspected wrong words in the word set by using a preset dictionary.
[0067] Preferably, the preset dictionary includes a commonly used dictionary and a scenario-specific word dictionary, wherein the commonly used word dictionary is a word set containing multiple modern high-frequency commonly used words, and the scenario-specific word dictionary is a word set containing multiple specific scenario-specific words. For example, when the specific scenario is to handle insurance, the specific words include insurer, policyholder, beneficiary, etc.
[0068] In an optional embodiment of the present invention, before using a preset dictionary to determine the suspected wrong words in the word set, it also includes constructing a common word dictionary. Wherein, the constructing of the common word dictionary includes:
[0069] Acquire common basic vocabulary;
[0070] Using the basic vocabulary as keywords to search in web pages, obtaining network data text;
[0071] Analyze the network data text and extract high-frequency words in the network data text;
[0072] The basic vocabulary and high-frequency words are collected to obtain a common word dictionary.
[0073] For details, see Figure 3 As shown, the use of a preset dictionary to determine suspected wrong words in the word set includes:
[0074] S20, selecting a word in the word set;
[0075] S21, searching whether the word exists in a common word dictionary and a scene-specific word dictionary;
[0076] When the word does not exist in the common word dictionary and the scene professional word dictionary, executing S22, classifying the word as a suspected wrong word;
[0077] When the word has been classified as a suspected wrong word, or the word exists in the common word dictionary and the scene professional word dictionary, return to S21 until every word in the word set is selected.
[0078] S3. Determine a candidate phrase set that can be used to replace the suspected wrong word according to the suspected wrong word.
[0079] For details, see Figure 4 As shown, the step of determining a candidate phrase set that can be used to replace the suspected wrong word according to the suspected wrong word includes:
[0080] S30. Split the suspected misspelled word character by character to obtain one or more suspected misspelled characters.
[0081] S31. Find the homophonic and homographic characters corresponding to the suspected misspelled characters in the preset homophonic character table and homographic character table to obtain a candidate character set.
[0082] S32. Combine the characters in the candidate character set according to the suspected misspelled word to form candidate word groups, and obtain a candidate word group set.
[0083] For example, "shí jiàn" is a suspected misspelled word, which can be split into "shí" and "jiàn". The homophonic and homographic characters corresponding to "shí" are shì, shí, shén, shí, shí, guò, tǎo, fù, etc., and the homophonic and homographic characters corresponding to "jiàn" are jiān, jiàn, jiàn, jiàn, jiàn, wǔ, etc. Combine them to form words such as shí jiān, shí jiàn, shí jiàn, etc. Gather these words to obtain the candidate word group set corresponding to the suspected misspelled word "shí jiàn".
[0084] Preferably, in the embodiment of the present invention, the possible misspelled words are quickly found using a preset dictionary first, and then all possible candidate word combinations are found by combining the homophonic character table and homographic character table, which expands the range of candidate words, is beneficial to subsequent correction, and improves the accuracy of correction.
[0085] S4. Replace the suspected misspelled word in the text to be corrected with each candidate word group in the candidate word group set to obtain candidate corrected sentences corresponding to each candidate word group.
[0086] Specifically, in the embodiment of the present invention, one suspected misspelled word corresponds to one candidate word group set. Refer to Figure 5 As shown, the step of replacing the suspected misspelled word in the text to be corrected with each candidate word group in the candidate word group set to obtain candidate corrected sentences corresponding to each candidate word group includes:
[0087] S40. Determine the corresponding suspected misspelled word according to the candidate word group set;
[0088] S41. Determine the text to be corrected with the suspected misspelled word according to the suspected misspelled word;
[0089] S42. Replace the corresponding suspected misspelled word in the text to be corrected with the candidate word group in the candidate word group set to obtain a candidate corrected sentence corresponding to the candidate word group.
[0090] S5. Score the candidate corrected sentences using a pre-trained language model, and determine the correction of the suspected misspelled word according to the score.
[0091] Preferably, the pre-trained language model is a language model based on a neural network, and the language model can calculate the probability value of a sentence. The probability of the whole sentence is the product of the probability of each word appearing. The larger the probability value, the more reasonable the sentence. In order to ensure the security of the language model, the language model can be stored in a node of a blockchain.
[0092] For details, see Figure 6 As shown, the S5 includes in detail:
[0093] S50, using the language model to calculate the probability value of each of the candidate error correction sentences;
[0094] S51, calculating the score of each candidate error correction sentence using a preset scoring algorithm according to the probability value;
[0095] S52: Select the candidate phrase corresponding to the candidate error correction sentence with the highest score to correct the suspected wrong word.
[0096] Furthermore, the scoring algorithm in the embodiment of the present invention includes:
[0097]
[0098] Where P is the score; N is the total number of candidate error correction sentences; q(x i ) is the probability value of the i-th candidate error correction sentence output by the language model.
[0099] Preferably, the embodiment of the present invention can find a candidate phrase corresponding to a suspected wrong word by using the above-mentioned scoring algorithm, and use the candidate phrase to replace the corresponding suspected wrong word in the text to be corrected to obtain a corrected text.
[0100] Preferably, the embodiment of the present invention utilizes a pre-trained language model to determine the rationality of the candidate phrase in a sentence. The language model can incorporate semantic information to determine the rationality of a sentence, thereby improving the accuracy of text error correction.
[0101] The embodiment of the present invention obtains a text to be corrected, and performs word segmentation on the text to be corrected to obtain a word set, and divides the text into words, so as to facilitate subsequent detection and correction of the words; a preset dictionary is used to determine suspected wrong words in the word set, and the dictionary can be used to quickly locate possible wrong words, thereby improving detection efficiency; a candidate phrase set that can be used to replace the suspected wrong words is determined according to the suspected wrong words, and all possible candidate word combinations are found, so as to expand the replaceable range of the suspected wrong words and improve the accuracy of error correction; each candidate phrase in the candidate phrase set is used to replace the suspected wrong words in the text to be corrected, and a candidate error correction sentence corresponding to each candidate phrase is obtained, and the candidate phrase is placed in the original sentence, so as to facilitate subsequent judgment of the rationality of the match between the candidate words and the original sentence; the candidate error correction sentence is scored using a pre-trained language model, and the suspected wrong words are corrected according to the score, and the language model can be used to integrate semantic information to score the rationality of the candidate error correction sentence, so as to improve the accuracy of error correction and improve efficiency. Therefore, the text error correction method, device and computer-readable storage medium proposed in the present invention can improve the accuracy and efficiency of text error correction.
[0102] like Figure 7 The figure shows a functional module diagram of the text error correction device of the present invention.
[0103] The text error correction device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the text error correction device can include a word segmentation module 101, a suspected wrong word determination module 102, a candidate phrase set acquisition module 103, a candidate error correction sentence acquisition module 104 and a wrong word correction module 105. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.
[0104] In this embodiment, the functions of each module / unit are as follows:
[0105] The word segmentation module 101 is used to obtain the text to be corrected, and perform word segmentation on the text to be corrected to obtain a word set.
[0106] Preferably, the text to be corrected is a computer document type, mainly used to record and store text information. In the embodiment of the present invention, the text to be corrected can be various documents written by the user, or documents such as notices.
[0107] In detail, the obtaining of the text to be corrected includes: receiving the text to be corrected uploaded by the user; obtaining the text to be corrected from a preset database; downloading the text to be corrected from a web page; and / or obtaining the text to be corrected from a node of the blockchain.
[0108] In detail, the word segmentation module 101 is specifically used for:
[0109] Dividing the text to be corrected by sentences to obtain a set of sentences to be corrected;
[0110] Each sentence in the set of sentences to be corrected is segmented to obtain a word set.
[0111] Furthermore, the embodiment of the present invention uses a word segmentation technology based on a prefix word segmentation dictionary to segment each sentence in the set of sentences to be corrected. The specific implementation steps of the word segmentation include:
[0112] Segment the sentence using a prefix segmentation dictionary to obtain an original word set in one or more segmentation modes;
[0113] According to the original word set, a directed acyclic graph is constructed by connecting the words end to end;
[0114] A dynamic programming algorithm is used to calculate the maximum probability path in the directed acyclic graph to obtain the word segmentation result of the text sentence.
[0115] The module 102 for determining suspected wrong words is used to determine suspected wrong words in the word set by using a preset dictionary.
[0116] Preferably, the preset dictionary includes a commonly used dictionary and a scenario-specific word dictionary, wherein the commonly used word dictionary is a word set containing multiple modern high-frequency commonly used words, and the scenario-specific word dictionary is a word set containing multiple specific scenario-specific words. For example, when the specific scenario is to handle insurance, the specific words include insurer, policyholder, beneficiary, etc.
[0117] In an optional embodiment of the present invention, before using a preset dictionary to determine the suspected wrong words in the word set, it also includes constructing a common word dictionary. Wherein, the constructing of the common word dictionary includes:
[0118] Acquire common basic vocabulary;
[0119] Using the basic vocabulary as keywords to search in web pages, obtaining network data text;
[0120] Analyze the network data text and extract high-frequency words in the network data text;
[0121] The basic vocabulary and high-frequency words are collected to obtain a common word dictionary.
[0122] In detail, when using the preset dictionary to determine the suspected wrong words in the word set, the suspected wrong word determination module 102 specifically performs the following operations:
[0123] Selecting a word from the word set;
[0124] Check whether the word exists in the common word dictionary and the scenario-specific word dictionary;
[0125] When the word does not exist in the common word dictionary and the scenario-specific word dictionary, classify the word as a suspected misspelled word;
[0126] Until each word in the word set is selected.
[0127] The candidate phrase set acquisition module 103 is used to determine a candidate phrase set that can be used to replace the suspected misspelled word according to the suspected misspelled word.
[0128] Specifically, the candidate phrase set acquisition module 103 is specifically used for:
[0129] Split the suspected misspelled word character by character to obtain one or more suspected misspelled characters;
[0130] Find the homophonic and homographic characters corresponding to the suspected misspelled characters in the preset homophone table and homographic character table to obtain a candidate character set;
[0131] Combine the characters in the candidate character set according to the suspected misspelled word to form candidate phrases, and obtain a candidate phrase set.
[0132] For example, "shijian" is a suspected misspelled word, which can be split into "shi" and "jian". The homophonic and homographic characters corresponding to "shi" are found to be shi, shi, shi, shi, shi, guo, tao, fu, etc., and the homophonic and homographic characters corresponding to "jian" are jian, jian, jian, jian, jian, wu, etc. Combine them to form phrases such as time, stone sword, practice, etc. Gather these words to obtain the candidate phrase set corresponding to the suspected misspelled word "shijian".
[0133] Preferably, in the embodiment of the present invention, the preset dictionary is first used to quickly find the possible misspelled words, and then the homophone table and homographic character table are combined to find all possible candidate word combinations, expanding the range of candidate words, which is beneficial to subsequent correction and improves the accuracy of correction.
[0134] The candidate error correction sentence acquisition module 104 is used to replace the suspected misspelled word in the text to be error-corrected with each candidate phrase in the candidate phrase set to obtain a candidate error correction sentence corresponding to each candidate phrase.
[0135] Specifically, in the embodiment of the present invention, one suspected misspelled word corresponds to one candidate phrase set. The candidate error correction sentence acquisition module 104 is specifically used for:
[0136] Determine the corresponding suspected misspelled word according to the candidate phrase set;
[0137] Determine the text to be error-corrected with the suspected misspelled word according to the suspected misspelled word;
[0138] In the sentence to be corrected, a candidate phrase in the candidate phrase set is used to replace a corresponding suspected wrong word, so as to obtain a candidate corrected sentence corresponding to the candidate phrase.
[0139] The wrong word correction module 105 is used to score the candidate error correction sentences using a pre-trained language model, and determine to correct the suspected wrong word according to the score.
[0140] Preferably, the pre-trained language model is a language model based on a neural network, and the language model can calculate the probability value of a sentence. The probability of the whole sentence is the product of the probability of each word appearing. The larger the probability value, the more reasonable the sentence. In order to ensure the security of the language model, the language model can be stored in a node of a blockchain.
[0141] In detail, the wrong word correction module 105 is specifically used for:
[0142] Calculating the probability value of each of the candidate error correction sentences using the language model;
[0143] Calculate the score of each candidate error correction sentence using a preset scoring algorithm according to the probability value;
[0144] The candidate phrase corresponding to the candidate error correction sentence with the highest score is selected to correct the suspected wrong word.
[0145] Furthermore, the scoring algorithm in the embodiment of the present invention includes:
[0146]
[0147] Where P is the score; N is the total number of candidate error correction sentences; q(x i ) is the probability value of the i-th candidate error correction sentence output by the language model.
[0148] Preferably, the embodiment of the present invention can find a candidate phrase corresponding to a suspected wrong word by using the above-mentioned scoring algorithm, and use the candidate phrase to replace the corresponding suspected wrong word in the text to be corrected to obtain a corrected text.
[0149] Preferably, the embodiment of the present invention utilizes a pre-trained language model to determine the rationality of the candidate phrase in a sentence. The language model can incorporate semantic information to determine the rationality of a sentence, thereby improving the accuracy of text error correction.
[0150] like Figure 8 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the text error correction method of the present invention.
[0151] The electronic device 1 may include a processor 10 , a memory 11 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a text error correction program 12 .
[0152] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the text error correction program 12, etc., but also be used to temporarily store data that has been output or is to be output.
[0153] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing text error correction programs, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.
[0154] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0155] Figure 8Only an electronic device with components is shown, and those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0156] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
[0157] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.
[0158] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.
[0159] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0160] The text error correction program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:
[0161] Obtaining a text to be corrected, and performing word segmentation on the text to be corrected to obtain a word set;
[0162] Using a preset dictionary to determine suspected wrong words in the word set;
[0163] Determining a candidate phrase set that can be used to replace the suspected wrong word according to the suspected wrong word;
[0164] Using each candidate phrase in the candidate phrase set to replace the suspected wrong word in the text to be corrected, to obtain a candidate correction sentence corresponding to each candidate phrase;
[0165] The candidate error correction sentences are scored using a pre-trained language model, and the suspected wrong words are corrected according to the scores.
[0166] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0168] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0169] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0170] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0171] Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any associated diagram mark in the claims should not be regarded as limiting the claims involved.
[0172] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.
[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A text error correction method, characterized in that: The method comprises: Obtaining a text to be corrected, and performing word segmentation on the text to be corrected to obtain a word set; Determining suspected wrong words in the word set by using a preset dictionary includes: selecting a word in the word set, searching whether the word exists in a preset common word dictionary and a scene professional word dictionary, and when the word does not exist in the common word dictionary and the scene professional word dictionary, classifying the word as a suspected wrong word, wherein the common word dictionary is a word set containing common words, and the scene professional word dictionary is a word set containing scene-specific words; Splitting the suspected wrong word by characters to obtain one or more suspected wrong characters, finding homophones and homographs corresponding to the suspected wrong characters in a preset homophone table and homograph table to obtain a candidate character set, and combining the characters in the candidate character set into words according to the suspected wrong word to obtain a candidate word set; Determine, in the text to be corrected, a sentence to be corrected that has a suspected wrong word according to the candidate phrase set, and replace the suspected wrong word in the sentence to be corrected with each candidate phrase in the candidate phrase set to obtain a candidate corrected sentence corresponding to each candidate phrase; The candidate error correction sentences are scored using a pre-trained language model, and the suspected wrong words are corrected according to the scores.
2. The text error correction method according to claim 1, characterized in that: The word segmentation of the text to be corrected to obtain a word set includes: Dividing the text to be corrected by sentences to obtain a set of sentences to be corrected; Each sentence in the set of sentences to be corrected is segmented to obtain a word set.
3. The text error correction method according to claim 2, characterized in that: The word segmentation of each sentence in the set of sentences to be corrected includes: Segment the sentence using a prefix segmentation dictionary to obtain an original word set in one or more segmentation modes; According to the original word set, a directed acyclic graph is constructed by connecting the words end to end; A dynamic programming algorithm is used to calculate the maximum probability path in the directed acyclic graph, and a word segmentation result is obtained by selecting a partitioning method corresponding to the maximum probability path.
4. The text error correction method according to any one of claims 1 to 3, characterized in that: The step of scoring the candidate error correction sentences using the pre-trained language model and correcting the suspected wrong words according to the scores includes: Calculating the probability value of each of the candidate error correction sentences; Calculate the score of each candidate error correction sentence using a preset scoring algorithm according to the probability value; The candidate phrase corresponding to the candidate error correction sentence with the highest score is selected to correct the suspected wrong word.
5. The text error correction method according to claim 4, characterized in that: The scoring algorithm includes: Where P is the score; N is the total number of candidate error correction sentences; q(x i ) is the probability value of the i-th candidate error correction sentence output by the language model.
6. A text error correction device, characterized in that: The device comprises: A word segmentation module is used to obtain the text to be corrected and perform word segmentation on the text to be corrected to obtain a word set; A suspected wrong word determination module is used to determine suspected wrong words in the word set using a preset dictionary, including: selecting a word in the word set, searching whether the word exists in a preset common word dictionary and a scene professional word dictionary, and when the word does not exist in the common word dictionary and the scene professional word dictionary, classifying the word as a suspected wrong word, the common word dictionary is a word set containing common words, and the scene professional word dictionary is a word set containing scene-specific words; A candidate phrase set acquisition module is used to split the suspected wrong word by characters to obtain one or more suspected wrong characters, find homophones and homographs corresponding to the suspected wrong characters in a preset homophone table and homograph table to obtain a candidate character set, and combine the characters in the candidate character set into words according to the suspected wrong word to obtain a candidate phrase set; A module for obtaining candidate error correction sentences is used to determine the sentences to be corrected that have suspected wrong words in the text to be corrected according to the candidate phrase set, and to replace the suspected wrong words in the sentences to be corrected with each candidate phrase in the candidate phrase set to obtain candidate error correction sentences corresponding to each candidate phrase; The wrong word correction module is used to score the candidate error correction sentences using a pre-trained language model, and determine to correct the suspected wrong word according to the score.
7. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one computer program instruction; and A processor is configured to execute computer program instructions stored in the memory to perform the text error correction method according to any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text error correction method according to any one of claims 1 to 5 is implemented.
