A lightweight text error correction method and device
Through the lightweight text error correction method, the confusion model and early retreat inference strategy are used to solve the problems of text error correction accuracy and calculation complexity in the existing technology, and efficient text error correction effect is achieved.
Patent Information
- Application Number
- CN202310878929.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-18
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2043-07-18
AI Technical Summary
The existing text error correction scheme has defects in accuracy and computational complexity. The end-to-end method is difficult to explain and has low accuracy. The large model used in pipeline method has high computational complexity, resulting in increased memory usage and time-consuming. It cannot be used especially when the hardware equipment is poor.
Using a lightweight text error correction method, the candidate set is obtained by obtaining the error token in the text to be corrected, and early retreat inference is performed using the trained confusion model, and the first masking strategy and the RoBERTa-Base model are used to accelerate the confusion. Compute to reduce memory usage.
While ensuring the accuracy of calculation, the memory usage and calculation time of the confusion model are reduced, the error correction efficiency is improved, and it is suitable for various hardware devices.
Smart Images
Figure CN116882391B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of text data processing, and in particular to a lightweight text error correction method and a lightweight text error correction device. Background Art
[0002] Language is complex. Every language has a set of complex grammar and syntax rules. These grammar and syntax rules are complex and changeable. For example, some words or phrases have multiple pronunciations, multiple meanings, and multiple uses, which places high demands on language users. Once language users do not master the language well or are careless, it is easy to make mistakes such as inappropriate word use and misattribution. Although these errors seem insignificant, in certain vertical fields, such as the writing and publication of government documents, even small language errors can have very bad effects. With the rapid development of the Internet and various media platforms in recent years, the dissemination of various normative texts has increased, and the importance of text error correction technology has also increased significantly.
[0003] Automatic checking and automatic error correction is an important technology in the field of machine learning. Its purpose is to improve language correctness while reducing the cost of manual verification. With the development of machine learning technology, the error correction problem has received more and more attention from academia and industry. There are two mainstream methods: one solution is to classify language errors, build confusion sets of various errors based on features such as similarity of sound and form, and then sort the confusion sets for error correction; the other idea is to draw on the idea of statistical machine translation and equate language error correction to the process of machine translation, that is, translating incorrect text into correct text, and a series of optimization methods have emerged.
[0004] Current text correction solutions can generally be divided into two categories: end-to-end and pipeline. End-to-end text correction usually uses the Seq2Seq model that has been successfully applied in the field of machine translation, that is, "translating" incorrect sentences into correct sentences through machine translation.
[0005] The pipeline text error correction is usually divided into two modules: error detection and error correction. Taking the pycorrector Chinese text error correction tool as an example, the error detection and error correction steps are as follows:
[0006] Error detection module:
[0007] First, use the word segmentation tool to segment words. Words in the general dictionary are not considered as wrong words. Chinese words / characters that are not in the general dictionary are added to the list of suspected wrong words.
[0008] The sentence is divided into 2-gram and 3-gram sequences based on characters, and then the language model trained by kenlm is used to calculate the n-gram score. In most cases, the entire sentence will not be wrong. Under normal circumstances, the score will be within a specific range, but the score of the wrong position will be far away from this score. The corresponding positions with relatively far scores are extracted and added to the list of suspected wrong words.
[0009] Error correction module:
[0010] Use the confusion word dictionary to obtain candidate words for each error position.
[0011] The language model trained by kenlm calculates the perplexity scores of the original sentence and the candidate words after the candidate words are replaced. The smaller the perplexity score, the greater the possibility of the sentence. If the perplexity score of the candidate sentence is less than the original sentence by more than a certain distance, the post-selection sentence is used to correct the original sentence.
[0012] The masked language model calculates the perplexity of a sentence by traversing the sentence tokens (word segmentation), masking the token at a position, and then calculating the probability of the token in the context. The perplexity is the average of all token probabilities, and the formula is as follows:
[0013]
[0014] Where T is the sentence length, is the sentence after masking at the corresponding position, m t is the weight, and PPL is the calculated perplexity.
[0015] The correction results of end-to-end text error correction solutions are difficult to explain, relatively uncontrollable, and have low accuracy.
[0016] The kenlm language model used by the pycorrector error correction tool is relatively small, and the n-gram score is too dependent on the commonness of words, so the accuracy is not high. If a large transformer-type model is used for error detection and correction, although the accuracy of error correction is improved, the calculation of perplexity requires traversal calculation over the length of the sentence, and the calculation complexity is too high, which will increase memory usage and time consumption, and may even not be used on some poor hardware devices.
[0017] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art. Summary of the invention
[0018] The object of the present invention is to provide a lightweight text error correction method to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.
[0019] One aspect of the present invention provides a lightweight text error correction method, the lightweight text error correction method comprising:
[0020] Get the text to be corrected;
[0021] Obtain the error token in the text to be corrected according to the text to be corrected;
[0022] Acquire a candidate set, wherein the candidate set includes at least one replaceable token for replacing the erroneous token;
[0023] Obtaining a trained perplexity model, wherein the perplexity model is trained by a first masking strategy, wherein the first masking strategy adopts a masking strategy other than a random dynamic masking strategy;
[0024] Each replaceable token in the candidate set replaces the corresponding erroneous token in the text to be corrected to form a text to be calculated for perplexity;
[0025] Inputting the perplexity text to be calculated into the trained perplexity model, thereby obtaining the perplexity calculation result through the perplexity model early-retreat reasoning strategy;
[0026] The text to be corrected is modified according to the perplexity calculation result to obtain the correct text.
[0027] Optionally, the trained perplexity model is trained in the following manner:
[0028] Get the perplexity model;
[0029] Labeled data: Use the text errors and corresponding correction results in actual business scenarios as training data for the perplexity model, and supplement with a large amount of unlabeled data as the training set for the perplexity model;
[0030] Use the training data to train the model on masks and calculate the mask task loss function for the output of each encoder layer.
[0031] Optionally, in the mask training of the model using the training data, the training is performed using the first mask strategy.
[0032] Optionally, the first mask strategy includes:
[0033] Calculate the geometric mean of the token probabilities in a window around the replaceable token in each perplexity text to be calculated, wherein the window size is determined by a window size determination strategy.
[0034] Optionally, the window size determination strategy includes:
[0035] Get the relationship between the distance of the nearest punctuation mark before and after the replaceable token and the window threshold size;
[0036] The window position information is obtained according to the relationship between the distance between the nearest punctuation marks before and after the replaceable token and the window threshold size.
[0037] Optionally, the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size;
[0038] When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes:
[0039] Use the distance between the two punctuation marks as the window, and the token between the two punctuation marks as the window content.
[0040] Optionally, the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold value includes that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold value;
[0041] When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes:
[0042] Use a syntactic analysis tool to perform syntactic analysis on the content between two punctuation marks to obtain the syntactic analysis tree of the replaceable token and the candidate token. Select the smallest position with a syntactic dependency relationship with the replaceable token and the candidate token as the starting position of the window, and the last position with a syntactic dependency relationship as the final position of the window.
[0043] Optionally, the perplexity model is a RoBERTa-Base perplexity model, and the RoBERTa-Base perplexity model includes 12 perplexity calculators connected in series, wherein the last perplexity calculator is called a final perplexity calculator, and the other perplexity calculators are called early-leaving perplexity calculators.
[0044] Optionally, the perplexity model early exit reasoning strategy includes:
[0045] Mask the characters in the mask window of a sentence one by one. Assume that the window size is M, which is equivalent to the input of the model being a batch of sentences with batch=M. The perplexity calculation formula is:
[0046] in,
[0047] Set the model early exit threshold N, t, where N is an integer; the perplexity calculated by the model through the first encoder is ppl1, and the perplexity calculated through the second encoder layer is ppl2, and so on. When the model calculates ppl N times in a row, that is, ppl to ppl(i+N) satisfies:
[0048]
[0049] When , the model retreats early from the output position of the current encoder and uses ppl(i+N) as the result of perplexity calculation.
[0050] The present application also provides a lightweight text error correction device, the lightweight text error correction device comprising:
[0051] A module for acquiring text to be corrected, wherein the module is used to acquire text to be corrected;
[0052] An error token acquisition module, wherein the error token acquisition module is used to acquire an error token in the text to be corrected according to the text to be corrected;
[0053] A candidate set acquisition module, the candidate set acquisition module is used to acquire a candidate set, the candidate set includes at least one replaceable token for replacing the erroneous token;
[0054] A perplexity model acquisition module, wherein the perplexity model acquisition module is used to acquire a trained perplexity model, wherein the perplexity model is trained by a first masking strategy, wherein the first masking strategy adopts a masking strategy other than a random dynamic masking strategy;
[0055] A module for generating a text with a perplexity to be calculated, wherein the module is used to replace a corresponding erroneous token in the text to be corrected with each replaceable token in the candidate set to form a text with a perplexity to be calculated;
[0056] A perplexity calculation module, wherein the perplexity calculation module is used to input the perplexity text to be calculated into the trained perplexity model, thereby obtaining the perplexity calculation result through the perplexity model early-retreat reasoning strategy;
[0057] A modification module is used to modify the text to be corrected according to the perplexity calculation result, so as to obtain the correct text.
[0058] Beneficial effects:
[0059] The lightweight text correction method of the present application uses appropriate strategies to achieve early exit in the perplexity model reasoning process, and can still ensure the accuracy of model calculation in the case of early exit, modify the sentence perplexity traversal method to accelerate the reasoning of the perplexity model, and reduce the memory usage of the perplexity model. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is a flowchart of a lightweight text error correction method according to an embodiment of the present application.
[0061] Figure 2 An electronic device used to implement Figure 1 The lightweight text error correction method shown.
[0062] Figure 3 Schematic diagram of the framework of the RoBERTa model. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.
[0064] It should be noted that, in the description of the present invention, the terms “first” and “second” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.
[0065] Figure 1 It is a flowchart of a lightweight text error correction method according to an embodiment of the present application.
[0066] like Figure 1 The lightweight text error correction method shown includes:
[0067] Step 1: Get the text to be corrected;
[0068] Step 2: Obtain the error token in the text to be corrected according to the text to be corrected;
[0069] Step 3: Obtain a candidate set, where the candidate set includes at least one replaceable token for replacing the erroneous token;
[0070] Step 4: Obtain a trained perplexity model, where the perplexity model is trained using a first masking strategy, wherein the first masking strategy uses a masking strategy other than a random dynamic masking strategy;
[0071] Step 5: Replace the corresponding erroneous token in the text to be corrected with each replaceable token in the candidate set to form the text to be calculated for perplexity;
[0072] Step 6: input the perplexity text to be calculated into the trained perplexity model, thereby obtaining the perplexity calculation result through the perplexity model early retreat reasoning strategy;
[0073] Step 7: Modify the text to be corrected according to the perplexity calculation result to obtain the correct text.
[0074] The lightweight text correction method of the present application uses appropriate strategies to achieve early exit in the perplexity model reasoning process, and can still ensure the accuracy of model calculation in the case of early exit, modify the sentence perplexity traversal method to accelerate the reasoning of the perplexity model, and reduce the memory usage of the perplexity model.
[0075] In this embodiment, the trained perplexity model is trained in the following manner:
[0076] Get the perplexity model;
[0077] Labeled data, using text errors and corresponding correction results that occur in actual business scenarios as training data for the perplexity model, and supplementing a large amount of unlabeled data as a training set for the perplexity model; in this embodiment, the data is official document data.
[0078] Use the training data to train the model on masks and calculate the mask task loss function for the output of each encoder layer.
[0079] In this embodiment, in the mask training of the model using the training data, the training is performed using the first mask strategy.
[0080] In this embodiment, the first mask strategy includes:
[0081] The geometric mean of the token probabilities in a window around the replaceable token in each candidate sentence is calculated, wherein the window size is determined by a window size determination strategy.
[0082] In this embodiment, the window size determination strategy includes:
[0083] Get the relationship between the distance of the nearest punctuation mark before and after the replaceable token and the window threshold size;
[0084] The window position information is obtained according to the relationship between the distance between the nearest punctuation marks before and after the replaceable token and the window threshold size.
[0085] In this embodiment, the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size;
[0086] When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes:
[0087] Use the distance between the two punctuation marks as the window, and the token between the two punctuation marks as the window content.
[0088] In this embodiment, the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold value includes that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold value;
[0089] When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes:
[0090] Use a syntactic analysis tool to perform syntactic analysis on the content between two punctuation marks to obtain the syntactic analysis tree of the replaceable token and the candidate token. Select the smallest position with a syntactic dependency relationship with the replaceable token and the candidate token as the starting position of the window, and the last position with a syntactic dependency relationship as the final position of the window.
[0091] In this embodiment, the perplexity model is a RoBERTa-Base perplexity model, which includes 12 perplexity calculators connected in series, wherein the last perplexity calculator is called a final perplexity calculator, and the other perplexity calculators are called early-leaving perplexity calculators.
[0092] In this embodiment, the perplexity model early exit reasoning strategy includes:
[0093] Mask the characters in the mask window of a sentence one by one. Assume that the window size is M, which is equivalent to the input of the model being a batch of sentences with batch=M. The perplexity calculation formula is:
[0094] in,
[0095] Set the model early exit threshold N, t, where N is an integer; the perplexity calculated by the model through the first encoder is ppl1, and the perplexity calculated through the second encoder layer is ppl2, and so on. When the model calculates ppl N times in a row, that is, ppl to ppl(i+N) satisfies:
[0096]
[0097] When , the model retreats early from the output position of the current encoder and uses ppl(i+N) as the result of perplexity calculation.
[0098] The present application is further described in detail below by way of examples. It should be understood that the examples do not constitute any limitation to the present application.
[0099] In this embodiment, the present application also performs error detection through an error detection module before error correction, wherein the error detection module determines the error position and error type in the text, and then recalls candidate words from the candidate set dictionary according to the error words and error types to form multiple candidate sentences. The error detection module here can use n-gram or ner model, which are conventional methods and will not be repeated.
[0100] In this embodiment, the perplexity model can be trained using an unlabeled data set, and random masks are used as model training data on the unlabeled data. In order to enable the model to learn data in specific task scenarios, we collected more than 5 million pieces of data and constructed more than 200,000 labeled data based on the positive and negative example data of text error correction in the business to supplement the original unlabeled data set.
[0101] In this embodiment, the RoBERTa model uses random dynamic masks in pre-training, and the mask positions and results are dynamically random. In order to enable the labeled data contained in our own data set to receive proper training, we improve on the original random dynamic mask strategy of RoBERTa and use a larger probability to keep the labeled mask of the labeled data unchanged. Other unlabeled token positions are still trained according to the original mask strategy.
[0102] See also Figure 2In this embodiment, taking the perplexity model of RoBERTa-Base as an example, the model has 12 encoder blocks connected in series, and the downstream structure for calculating perplexity is connected at the last layer. The lightweight solution of this article is to connect a perplexity calculator after each encoder of the model. The last perplexity calculator can be called the final perplexity calculator, and the previous perplexity calculator can be called the early-leaving perplexity calculator. In the process of calculating perplexity by model reasoning, the model evaluates whether the model is early-leaving based on the perplexity calculation results from the front to the back. Therefore, the number of encoders used in model reasoning is less than or equal to 12, thereby achieving the purpose of speeding up model reasoning.
[0103] The training steps of the perplexity model are as follows:
[0104] Labeled data: the text errors and corresponding correction results that appear in official documents in actual business scenarios are used as training data for the perplexity model, and a large amount of unlabeled official document data is supplemented as the training set of the perplexity model.
[0105] Use the training data to perform mask training on the model and calculate the mask task loss function of the output of each encoder layer, namely Lossmlm. The overall loss function of the model is the sum of the loss functions of each encoder, namely Loss = ∑ i loss i , lossi is the loss of the i-th encoder layer.
[0106] Confusion model masking strategy:
[0107] The perplexity score of a sentence is the geometric mean of the probability of each token in the sentence. In order to reduce the amount of calculation of the perplexity model and to ensure the effectiveness of the perplexity calculation, this application only calculates the geometric mean of the token probabilities in a window near the replaceable token in each candidate sentence. The window size determination strategy is as follows:
[0108] If the distance between the nearest punctuation marks before and after the replaceable token is within the window threshold, the distance between the two punctuation marks is used as the window, and the token between the two punctuation marks is used as the window content;
[0109] If the distance between the nearest punctuation marks before and after the corrected token is greater than the window threshold, the syntax analysis tool is used to perform syntax analysis on the content between the two punctuation marks to obtain the syntax analysis tree of the replaceable token and the candidate token. The smallest position with a syntactic dependency relationship with the replaceable token and the candidate token is selected as the starting position of the window, and the last position with a syntactic dependency relationship is selected as the final position of the window.
[0110] If the perplexity score of a candidate sentence is less than a threshold of the original sentence, the candidate sentence is used to correct the original sentence.
[0111] The perplexity model early exit reasoning strategy:
[0112] Mask the characters in the mask window in a sentence one by one. Assuming the window size is M, the input of the model is equivalent to a batch of sentences with batch=M. The perplexity calculation formula is:
[0113] in,
[0114] i is the starting position of the mask window in the sentence, M is the window size, and m is the weight (the default is 1). is the sentence after masking the corresponding position, is the probability that the current mask position is xi calculated by the model.
[0115] During the inference process, we first set the model's early exit threshold N, t, where N is an integer. The perplexity calculated by the model through the first encoder is ppl1, and the perplexity calculated through the second encoder layer is ppl2, and so on. When the model calculates ppl N times in a row, that is, ppl to ppl(i+N) satisfies:
[0116]
[0117] , the model retreats early from the output position of the current encoder and uses ppl(i+N) as the result of perplexity calculation.
[0118] The advantages of this application are:
[0119] Make a mask within the window to calculate the perplexity instead of traversing the entire sentence to speed up the calculation of perplexity.
[0120] The perplexity model infers the early exit strategy and uses the results of N consecutive ppl calculations to determine whether to exit early.
[0121] Fewer model structures are actually used in perplexity model reasoning, which speeds up the reasoning of the perplexity model. The window calculation method of perplexity further speeds up the calculation of perplexity.
[0122] For example, the specific process of the lightweight text error correction method of the present application is as follows:
[0123] For example, the sentence: Just as high-speed rail is used by ordinary people to make fire (typo, should be life), large aircraft will gradually fly into the ordinary world. The error correction process is:
[0124] First, through an error detection module, which is generally an ngram or ner model. The model determines that "starting a fire" and "spatula" are tokens with similar pronunciation and form errors, and thus obtains the text to be corrected.
[0125] According to the list of words with similar pronunciation and form, obtain the candidate sets for "starting a fire" and "spatula". For example, the candidate set for "starting a fire" is (life, holy fire, surplus goods, divine fire, surpass, etc.), and the candidate set for "spatula" is (domestic products, orphaned and disabled, etc.).
[0126] 3. Replace each token (replaceable token) in the candidate set into the example sentence for perplexity calculation. Among the candidate set of "starting a fire", the sentence "(Just as high-speed trains have entered people's lives), spatulas and large airplanes will also gradually fly into ordinary people's homes." has the lowest perplexity score (the () part in the sentence is the interval window for calculating ppl), and satisfies:
[0127]
[0128] threshold 音形 is the threshold for correction of words with similar pronunciation and form, then make the correction of "starting a fire" -> "life".
[0129] Similarly, in the candidate set of "spatula", the perplexity score of "domestic products" is the lowest, and satisfies:
[0130]
[0131] Then make the correction of "spatula" -> "domestic products".
[0132] This application also provides a lightweight text error correction device. The lightweight text error correction device includes a text to be corrected acquisition module, an error token acquisition module, a candidate set acquisition module, a perplexity model acquisition module, a text for calculating perplexity generation module, a perplexity calculation module, and a modification module. Among them,
[0133] The text to be corrected acquisition module is used to acquire the text to be corrected;
[0134] The error token acquisition module is used to acquire the error tokens in the text to be corrected according to the text to be corrected;
[0135] The candidate set acquisition module is used to acquire a candidate set, and the candidate set includes at least one replaceable token for replacing the error token;
[0136] The perplexity model acquisition module is used to acquire a trained perplexity model, and the perplexity model is trained by a first masking strategy, where the first masking strategy uses a masking strategy other than the random dynamic masking strategy;
[0137] The to-be-calculated perplexity text generation module is used to replace the corresponding erroneous token in the to-be-corrected text with each replaceable token in the candidate set to form the to-be-calculated perplexity text;
[0138] The perplexity calculation module is used to input the perplexity text to be calculated into the trained perplexity model, so as to obtain the perplexity calculation result through the perplexity model early retreat reasoning strategy;
[0139] The modification module is used to modify the text to be corrected according to the perplexity calculation result, so as to obtain the correct text.
[0140] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the lightweight text error correction method as described above when executing the computer program.
[0141] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the lightweight text error correction method as described above.
[0142] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing the lightweight text error correction method provided according to an embodiment of the present application.
[0143] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 501 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.
[0144] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 Describes a lightweight text correction method.
[0145] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors 503 configured to run the executable program code stored in the memory 504 to execute the lightweight text error correction method in the above embodiment.
[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0147] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0148] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0150] In addition, it is clear that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware. The words first, second, etc. are used to identify the name, not any specific order.
[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes marked in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or the overall flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0152] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0153] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0154] In this embodiment, if the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0155] It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0156] Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made thereto based on the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.
Claims
1. A lightweight text error correction method, characterized in that: The lightweight text error correction method includes: Get the text to be corrected; Obtain the error token in the text to be corrected according to the text to be corrected; Acquire a candidate set, wherein the candidate set includes at least one replaceable token for replacing the erroneous token; Obtaining a trained perplexity model, wherein the perplexity model is trained by a first masking strategy, wherein the first masking strategy adopts a masking strategy other than a random dynamic masking strategy; Each replaceable token in the candidate set replaces the corresponding erroneous token in the text to be corrected to form a text to be calculated for perplexity; Inputting the perplexity text to be calculated into the trained perplexity model, thereby obtaining the perplexity calculation result through the perplexity model early-retreat reasoning strategy; Modify the text to be corrected according to the perplexity calculation result to obtain the correct text; The trained perplexity model is trained as follows: Get the perplexity model; Labeled data: text errors and corresponding correction results in actual business scenarios are used as training data for the perplexity model, and a large amount of unlabeled data is supplemented as the training set of the perplexity model; Use the training data to perform mask training on the model and calculate the mask task loss function for the output of each encoder layer; In the mask training of the model using the training data, the training is performed using the first mask strategy; The first mask strategy includes: Calculate the geometric mean of token probabilities in a window around a replaceable token in each perplexity text to be calculated, wherein the window size is determined by a window size determination strategy; The window size determination strategy includes: Get the relationship between the distance of the nearest punctuation mark before and after the replaceable token and the window threshold size; Obtaining window position information according to the relationship between the distance between the nearest punctuation marks before and after the replaceable token and the window threshold size; The relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size; When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes: Use the distance between the two punctuation marks as the window, and the token between the two punctuation marks as the window content; The relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold value includes that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold value; When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes: Use a syntactic analysis tool to perform syntactic analysis on the content between two punctuation marks to obtain the syntactic analysis tree of the replaceable token and the candidate token. Select the smallest position with a syntactic dependency relationship with the replaceable token and the candidate token as the starting position of the window, and the last position with a syntactic dependency relationship as the final position of the window.
2. The lightweight text error correction method according to claim 1, characterized in that: The perplexity model is a RoBERTa-Base perplexity model, which includes 12 serially connected perplexity calculators, wherein the last perplexity calculator is called a final perplexity calculator, and the other perplexity calculators are called early-leaving perplexity calculators.
3. The lightweight text error correction method according to claim 2, characterized in that: The perplexity model early exit reasoning strategy includes: Mask the characters in the mask window of a sentence one by one. Assume that the window size is M, which is equivalent to the input of the model being a batch of sentences with batch=M. The perplexity calculation formula is: in, i is the starting position of the mask window in the sentence, M is the window size, and m is the weight. is the sentence after masking the corresponding position, is the probability that the current mask position is xi calculated by the model; set the model early exit threshold N, t, where N is an integer; the perplexity calculated by the model through the first encoder is ppl1, and the perplexity calculated through the second encoder layer is ppl2, and so on. When the model calculates ppl N times in a row, that is, ppl to ppl(i+N) satisfies: When , the model retreats early from the output position of the current encoder and uses ppl(i+N) as the result of perplexity calculation.
4. A lightweight text error correction device, characterized in that: The lightweight text error correction device comprises: A module for acquiring text to be corrected, wherein the module is used to acquire text to be corrected; An error token acquisition module, wherein the error token acquisition module is used to acquire an error token in the text to be corrected according to the text to be corrected; A candidate set acquisition module, the candidate set acquisition module is used to acquire a candidate set, the candidate set includes at least one replaceable token for replacing the erroneous token; A perplexity model acquisition module, wherein the perplexity model acquisition module is used to acquire a trained perplexity model, wherein the perplexity model is trained by a first masking strategy, wherein the first masking strategy adopts a masking strategy other than a random dynamic masking strategy; A module for generating a text with a perplexity to be calculated, wherein the module is used to replace a corresponding erroneous token in the text to be corrected with each replaceable token in the candidate set to form a text with a perplexity to be calculated; A perplexity calculation module, wherein the perplexity calculation module is used to input the perplexity text to be calculated into the trained perplexity model, thereby obtaining the perplexity calculation result through the perplexity model early-retreat reasoning strategy; A modification module, wherein the modification module is used to modify the text to be corrected according to the perplexity calculation result, so as to obtain a correct text; The trained perplexity model is trained as follows: Get the perplexity model; Labeled data: text errors and corresponding correction results in actual business scenarios are used as training data for the perplexity model, and a large amount of unlabeled data is supplemented as the training set of the perplexity model; Use the training data to perform mask training on the model and calculate the mask task loss function for the output of each encoder layer; In the mask training of the model using the training data, the training is performed using the first mask strategy; The first mask strategy includes: Calculate the geometric mean of token probabilities in a window around a replaceable token in each perplexity text to be calculated, wherein the window size is determined by a window size determination strategy; The window size determination strategy includes: Get the relationship between the distance of the nearest punctuation mark before and after the replaceable token and the window threshold size; Obtaining window position information according to the relationship between the distance between the nearest punctuation marks before and after the replaceable token and the window threshold size; The relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size; When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the replaceable token is within the window threshold size, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes: Use the distance between the two punctuation marks as the window, and the token between the two punctuation marks as the window content; The relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold value includes that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold value; When the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size is that the distance between the nearest punctuation mark before and after the corrected token is greater than the window threshold, the acquiring the window position information according to the relationship between the distance between the nearest punctuation mark before and after the replaceable token and the window threshold size includes: Use a syntactic analysis tool to perform syntactic analysis on the content between two punctuation marks to obtain the syntactic analysis tree of the replaceable token and the candidate token. Select the smallest position with a syntactic dependency relationship with the replaceable token and the candidate token as the starting position of the window, and the last position with a syntactic dependency relationship as the final position of the window.