Text processing method, training method, storage medium and electronic device
The generation of error text through n-member statistical methods and error-making processing technology solves the problem of low efficiency in the acquisition of training data in the existing technology, and efficient text rewriting tool training is realized. The generated error text is diverse and effective, improving training efficiency.
Patent Information
- Application Number
- PCT/CN2023/128085
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2023-10-31
- Publication Date
- 2024-11-28
AI Technical Summary
In the prior art, the acquisition of text rewriting tools based on manual annotation and manual review is less efficient and requires a large amount of manual resources.
Provide a text processing method, generates error text through n-member statistical methods, and uses techniques such as word segmentation processing, word-level error formation and word-level error formation to generate error text based on the target text, which is used to train text rewriting tools to reduce manual labor Mark the requirements.
It improves the efficiency of text rewriting tool training, and the generated error text is diverse and effective, which can better reflect errors in existing text, avoid invalid data structure, and improve data mining effect.
Smart Images

Figure CN2023128085_28112024_PF_FP_ABST
Abstract
Description
Text processing method, training method, storage medium and electronic device Technical Field
[0001] The present application belongs to the field of artificial intelligence technology and relates to a processing method, in particular to a text processing method, a training method, a storage medium and an electronic device. Background Art
[0002] With the development of the Internet, people like to write online more and more. Spelling errors, grammatical errors and other problems often occur during the writing process. To address these problems, text rewriting tools based on artificial intelligence (AI) have been developed. Text rewriting tools can help people express themselves better, make documents more fluent and easy to read, and reduce spelling, grammatical and punctuation errors. By using rewriting tools, people can focus more on the content itself when writing without worrying about grammatical and spelling errors, which is conducive to improving writing efficiency. Text rewriting tools based on artificial intelligence need to be trained before use, and a large amount of training data is required during training. The existing technology usually adopts manual annotation and manual review to obtain training data, which is less efficient.
[0003] Summary of the Invention
[0004] The present application provides a text processing method, apparatus, storage medium, and electronic device for generating erroneous text based on target text, wherein the erroneous text can be used to train a text rewriting tool.
[0005] In a first aspect, an embodiment of the present application provides a text processing method, which includes: obtaining a target text; performing word segmentation processing on the target text and obtaining words to be misconstructed; performing misconstruction processing on each of the words to be misconstructed to obtain an incorrect word corresponding to each of the words to be misconstructed; determining a valid incorrect word based on an n-gram model of the incorrect word, where n is a positive integer; and generating an incorrect text based on the valid incorrect word and the target text.
[0006] In an implementation of the first aspect, the words to be misconstructed include a first word to be misconstructed and / or a second word to be misconstructed, and performing misconstruction processing on each of the words to be misconstructed to obtain an incorrect word corresponding to each of the words to be misconstructed includes: performing character-level misconstruction on the first word to be misconstructed to obtain an incorrect word corresponding to the first word to be misconstructed; and / or performing word-level misconstruction on the second word to be misconstructed to obtain an incorrect word corresponding to the second word to be misconstructed.
[0007] In an implementation of the first aspect, performing character-level misconstruction on the first word to be misconstructed includes: performing any one or more of the following steps to perform character-level misconstruction on the first word to be misconstructed; replacing the target characters in the first word to be misconstructed with first characters, the first characters including characters that sound similar, characters that look similar, characters that start with the same head, characters that end with the same head, and / or random common characters of the target characters; deleting the target characters in the first word to be misconstructed; inserting second characters into the first word to be misconstructed, the second characters including characters that start with the same head, characters that end with the same head, random common characters, and / or random characters in the target text.
[0008] In an implementation of the first aspect, performing word-level misconstruction on the second word to be misconstructed includes: performing any one or more of the following steps to perform word-level misconstruction on the second word to be misconstructed; replacing the target word in the second word to be misconstructed with a first word, the first word including a phonetic word of the target word, a keyboard recommended word and / or a model recommended word; deleting the target word in the second word to be misconstructed; inserting a second word into the second word to be misconstructed, the second word including an adverb and / or a random word in the target text.
[0009] In an implementation of the first aspect, the words to be misconstructed include a third word to be misconstructed and a fourth word to be misconstructed, and performing misconstruction processing on the words to be misconstructed to obtain the wrong words corresponding to the words to be misconstructed includes: configuring the wrong word corresponding to the third word to be misconstructed as the fourth word to be misconstructed, and configuring the wrong word corresponding to the fourth word to be misconstructed as the third word to be misconstructed.
[0010] In an implementation of the first aspect, n is a positive integer greater than 1, and determining a valid error word based on the n-gram model of the error word includes: obtaining the n-gram of the error word; if the n-gram of the error word is a high-frequency word and the probability of the n-gram is greater than a probability threshold, determining that the error word is an invalid error word, otherwise determining that the error word is a valid error word.
[0011] In an implementation of the first aspect, generating the error text includes: translating the target text from a first language into a second language to obtain a target text in the second language; and translating the target text in the second language into the first language to generate the error text.
[0012] In a second aspect, an embodiment of the present application provides a training method, which includes: obtaining original corpus; processing the text in the original corpus using the text processing method described in any one of the first aspects of the present application to obtain erroneous text; and training a text rewriting tool using the text in the original corpus and the erroneous text.
[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text processing method described in any one of the first aspects of the present application and / or the training method described in the second aspect of the present application.
[0014] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory storing a computer program; a processor communicatively connected to the memory, and for executing the text processing method described in any one of the first aspect of the present application and / or the training method described in the second aspect of the present application when calling the computer program.
[0015] The text processing method provided in the embodiment of the present application can use the n-ary statistical method to judge the validity of construction errors, which is helpful to avoid constructing invalid data and thus achieve better mining effects.
[0016] In some embodiments, the text processing method can generate a large amount of erroneous text based on an existing normal corpus, which can be used to train the text rewriting tool. Compared with traditional training schemes, this method does not require a large amount of manual annotation, which helps improve training efficiency.
[0017] In some embodiments, the text processing method can use a variety of methods, such as replacement, addition, deletion, and reversal, to generate erroneous text. The error data constructed in this way is diverse and can better reflect the various errors that appear in the existing text. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] FIG1A is a schematic structural diagram of an electronic device according to an embodiment of the present application.
[0019] FIG1B is a flowchart of a text processing method according to an embodiment of the present application.
[0020] FIG2A shows a flow chart of obtaining erroneous words in an embodiment of the present application.
[0021] FIG. 2B shows a flow chart of performing word-level misspelling on a first word to be misspelled in an embodiment of the present application.
[0022] FIG2C shows a flow chart of word-level misspelling of the second word to be misspelled in an embodiment of the present application.
[0023] FIG3 shows a flow chart of determining valid error words in an embodiment of the present application.
[0024] FIG4 shows a flow chart of generating error text in an embodiment of the present application.
[0025] FIG5 is a flow chart showing a training method in an embodiment of the present application.
[0026] Component Reference Numerals 1 Electronic device 11 Text input device 12 General purpose processor 121 Central processing unit 122 Neural network processor 13 Display 14 Memory S11 to S15 Steps DETAILED DESCRIPTION
[0027] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.
[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.
[0029] Artificial intelligence-based text rewriting tools need to be trained before use, which requires a large amount of training data. Existing technologies typically use manual annotation and manual review to obtain training data, which is inefficient.
[0030] At least to address the above-mentioned issues, embodiments of the present application provide a text processing method. The text processing method can use the n-gram statistical method to determine the validity of the construction error, and can avoid constructing invalid data, thereby achieving better mining results. In addition, in some embodiments, the text processing method can generate a large amount of erroneous text based on an existing normal corpus, and the erroneous text can be used to train a text rewriting tool. Compared with traditional training schemes, this method does not require a large amount of manual annotation, which is conducive to improving training efficiency.
[0031] The text processing method provided in the embodiments of the present application can be applied to electronic devices. The electronic devices described in the embodiments of the present application can include mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other terminal devices. They can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
[0032] For example, the electronic device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future-evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future-evolved Non-terrestrial Network (NTN).
[0033] As an example and not a limitation, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as gloves and watches equipped with near-field communication modules. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. It is attached to the user and performs payment, authentication and other operations through a pre-bound electronic card. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are full-featured, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart watches and smart bracelets with display screens.
[0034] Figure 1A is a schematic diagram of the structure of an electronic device 1 according to an embodiment of the present application. The text processing method provided in this embodiment of the present application can be applied to the electronic device 1 shown in Figure 1A, but is not limited thereto. As shown in Figure 1A, the electronic device 1 includes a text input device 11, at least one general-purpose processor 12, a display 13, and a memory 14.
[0035] The text input device 11 may be any type of device capable of inputting text. The electronic device 1 supports users inputting text and other data in various ways to meet different user needs and application scenarios. For example, the text input device 11 may include one or more of a keyboard, a mouse, a touch screen, a voice recognition device, a stylus input device, and the like.
[0036] The general-purpose processor 12 can be any type of device capable of processing electronic instructions. In the embodiment of the present application, the electronic device 1 may include one or more general-purpose processors 12, such as one or both of a central processing unit (CPU) 121 and a neural network processor (NPU) 122. In addition, it may also include one or more of a graphics processing unit (GPU), a microprocessor, a microcontroller, a main processor, a controller, and an application-specific integrated circuit (ASIC). The general-purpose processor 12 is configured to execute various types of digital storage instructions, such as software or firmware programs stored in the memory 14, which enables the electronic device 1 to provide a variety of services. For example, the processor 11 can execute programs or process data to execute at least a portion of the text processing method and / or training method provided in the embodiment of the present application.
[0037] The main function of the central processing unit 121 is to parse computer instructions and process data in computer software to achieve overall control of the electronic device 1 and control and allocate all hardware resources of the electronic device 1 (such as storage resources, communication resources, I / O interfaces, etc.).
[0038] Neural network processor is a general term for new processors based on neural network algorithms and acceleration. It is specially designed for artificial intelligence to accelerate neural network operations and solve the problem of low efficiency of traditional chips in neural network operations.
[0039] It should be noted that the name of the neural network processor does not constitute a limitation of this application. For example, in other application scenarios, the neural network processor can also be transformed and replaced by other processors with similar functions, such as a tensor processing unit (TPU), a deep learning processing unit (DPU), and so on.
[0040] The display 13 may specifically include a display screen (display panel). In some implementations, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. The display device may also be a touch panel (touch screen, touch screen), which may include a display screen and a touch-sensitive surface. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the central processor 121 to determine the type of touch event. The central processor 121 then provides a corresponding visual output on the display device based on the type of touch event.
[0041] The memory 14 may include volatile memory, such as random access memory (RAM) or cache. The memory 14 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The memory 14 may be used to store data such as images captured by the image acquisition device 11. The memory 14 may also be used to store program instructions for the processor to call and execute the text processing method and / or training method described in this application.
[0042] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings in the embodiments of the present application.
[0043] FIG1B shows a flowchart of a text processing method according to an embodiment of the present application. As shown in FIG1B , the text processing method according to an embodiment of the present application includes the following steps S11 to S15 .
[0044] S11, obtaining a target text. The target text may be, for example, one or more complete sentences, but the embodiment of the present application is not limited thereto.
[0045] S12: Segment the target text and obtain the words to be misspelled. Multiple words may be obtained after segmenting the target text, and the words to be misspelled are part or all of the multiple words. For example, in some embodiments, a certain proportion of the multiple words may be selected as the words to be misspelled.
[0046] In some possible implementations, the target text includes characters outside of a first language, which may be, for example, Chinese. Segmenting the target text and obtaining the word to be misspelled includes segmenting the target text to obtain a plurality of words, deleting characters outside of the first language from the plurality of words, and selecting the word to be misspelled from the remaining words.
[0047] S13, performing misconstruction processing on each word to be misconstructed to obtain an incorrect word corresponding to each word to be misconstructed.
[0048] S14: Determine a valid incorrect word based on an n-gram model of the incorrect word, where n is a positive integer. In some possible implementations, the valid incorrect word may be determined based on a bigram model or a trigram model of the incorrect word.
[0049] S15: Generate an error text based on the valid error words and the target text. Specifically, the error text can be generated by replacing the corresponding words in the target text with the valid error words. For example, if the target text is "Today is sunny", the foreign word to be misconstructed is "today", and the corresponding valid error word is "lingtian", then the error text generated based on the valid error words and the target text is "lingtian is sunny".
[0050] The text processing method provided in the embodiment of the present application can use the n-ary statistical method to determine the validity of construction errors, thereby avoiding the construction of invalid data and achieving better mining effects.
[0051] In one embodiment of the present application, the method for obtaining the target text includes: selecting correct text from a corpus as the original corpus; filtering the sentences in the original corpus for fluency to obtain fluent sentences, which can be used as the target text.
[0052] In one embodiment of the present application, the method for obtaining the target text includes: selecting correct text from a corpus as the original corpus; using a sentence fluency model to score the sentences in the original corpus, and filtering the sentences in the original corpus according to the scoring results to obtain correct sentences, which can be used as the target text.
[0053] In one embodiment of the present application, the words to be error-constructed include the first word to be error-constructed and the second word to be error-constructed. Among them, the first word to be error-constructed refers to the word for character-level error construction, and the second word to be error-constructed refers to the word for word-level error construction. The first word to be error-constructed and the second word to be error-constructed may be the same, partially the same, or completely different. Figure 2A shows a flowchart of error construction processing for the words to be error-constructed in an embodiment of the present application. As shown in Figure 2A, the error construction process in an embodiment of the present application includes the following steps S21 and S22.
[0054] S21, perform character-level error construction on the first word to be error-constructed to obtain the error word corresponding to the first word to be error-constructed. Among them, character-level error construction refers to constructing corresponding errors for one or more characters in the first word to be error-constructed to obtain the corresponding error word.
[0055] S:22, perform word-level error construction on the second word to be error-constructed to obtain the error word corresponding to the second word to be error-constructed. Among them, word-level error construction refers to constructing corresponding errors for one or more words in the second word to be error-constructed to obtain the corresponding error word.
[0056] Figure 2B shows a flowchart of character-level error construction for the first word to be error-constructed in an embodiment of the present application. As shown in Figure 2B, the character-level error construction for the first word to be error-constructed in an embodiment of the present application includes the following steps S211 to S213.
[0057] S211, replace the target character in the first word to be error-constructed with the first character. The first character includes homophonic characters, similar-shaped characters, same-start characters, same-end characters, and / or random common characters of the target character. The definition of same-start characters is, for example: if the first characters of two words are the same, then the second characters of the two words are each other's same-start characters. For example, for the two words "character" and "document", "character" and "document" are each other's same-start characters. The definition of same-end characters is, for example, if the second characters of two words are the same, then the first characters of the two words are each other's same-end characters. For example, for the two words "character" and "Chinese character", "wen" and "han" are each other's same-end characters.
[0058] In some possible implementation manners, the target character may be one or more characters randomly selected from the first word to be error-constructed, or one or more characters specified in the first word to be error-constructed.
[0059] S212, delete the target character in the first word to be error-constructed. For example, one or more characters can be randomly selected from the first word to be error-constructed for deletion.
[0060] S213, insert the second character into the first word to be error-constructed. The second character includes same-start characters, same-end characters, random common characters, and / or random characters in the target text.
[0061] In some possible implementation manners, homophonic words of a target word can be obtained according to a homophonic dictionary. The homophonic dictionary can be generated by tools provided by FASPell, for example, and includes multiple words with the same or similar pinyin as the target word. For example, if the target word is "中", homophonic words such as "众重肿终衷种" can be obtained through the homophonic dictionary.
[0062] In some possible implementation manners, shape-similar words of a target word can be obtained according to a shape-similar dictionary. The shape-similar dictionary can be generated by tools provided by FASPell, for example, and includes multiple words with a shape similar to the target word. For example, if the target word is "中", shape-similar words such as "由甲申口巾" can be obtained through the shape-similar dictionary.
[0063] In some possible implementation manners, head-identical words of a target word can be obtained according to a head-identical dictionary. The method for obtaining the head-identical dictionary can be: performing a bigram model statistics on single characters of a large amount of corpus to obtain a high-frequency dictionary, and then generating the head-identical dictionary by using the high-frequency dictionary.
[0064] In some possible implementation manners, tail-identical words of a target word can be obtained according to a tail-identical dictionary. The method for obtaining the tail-identical dictionary can be: performing a bigram model statistics on single characters of a large amount of corpus to obtain a high-frequency dictionary, and then generating the tail-identical dictionary by using the high-frequency dictionary.
[0065] In some possible implementation manners, random common Chinese characters can be obtained by using a common Chinese character table, which can be the BERT table, for example.
[0066] It should be noted that performing character-level error construction on the first error-constructible word by executing steps S211 to S213 in the embodiments of the present application is only a feasible manner of the present application, but the present application is not limited thereto. In some other embodiments, any one or more of steps S211 to S213 may also be executed to perform character-level error construction on the first error-constructible word. For example, only step S211 may be executed to perform error construction on the first error-constructible word, or steps S211 and S213 may be executed to perform error construction on the first error-constructible word.
[0067] FIG. 2C shows a flowchart of performing word-level error construction on the second error-constructible word in the embodiments of the present application. As shown in FIG. 2C, performing word-level error construction on the second error-constructible word in the embodiments of the present application includes the following steps S221 to S223.
[0068] S221, replacing a target word in the second error-constructible word with a first word, where the first word includes a homophonic word of the target word, a keyboard recommended word, and / or a model recommended word.
[0069] In some possible implementations, the target word may be one or more words randomly selected from the second words to be misspelled, or may be one or more words specified in the second words to be misspelled.
[0070] S222: Delete the target word in the second word to be misspelled. For example, a word other than a noun may be randomly selected from the second word to be misspelled and deleted.
[0071] S223: inserting a second word into the second word to be misspelled, where the second word includes any adverb in the adverb list and / or a random word in the target text.
[0072] In some possible implementations, a keyboard interface may be called and an input method may be used to obtain a number of words in a high-frequency word list as a confusion set of high-frequency words. In step S221, a high-frequency word may be selected from the confusion set to replace the target word.
[0073] In some possible implementations, a prediction model can be used to obtain model-recommended words. For example, a position in the second word can be randomly selected as a mask, a Chinese-roformer-base model can be used for prediction, and a word can be randomly selected from the predicted list as the incorrect word for the target word. In some other implementations, models such as Bart and Robert can also be used for prediction, but this application is not limited to this.
[0074] It should be noted that, in the embodiment of the present application, executing steps S221 to S223 to perform word-level misconstruction on the second word to be misconstructed is only one feasible method of the present application, but the present application is not limited thereto. In some other embodiments, any one or more steps of steps S221 to S223 may also be executed to perform word-level misconstruction on the second word to be misconstructed. For example, only step S222 may be executed to perform word-level misconstruction on the second word to be misconstructed, or steps S221 and S222 may be executed to perform word-level misconstruction on the second word to be misconstructed.
[0075] It should be understood that, in the above embodiment, the word to be misformed includes both the first word to be misformed and the second word to be misformed, which is only one feasible embodiment of the present application, but the present application is not limited thereto. In some other embodiments, the word to be misformed may include only the first word to be misformed or the second word to be misformed.
[0076] In an embodiment of the present application, the words to be misconstructed include the third word to be misconstructed and the fourth word to be misconstructed. The misconstruction process for the words to be misconstructed to obtain the misspelled words corresponding to the words to be misconstructed includes: configuring the misspelled word corresponding to the third word to be misconstructed as the fourth word to be misconstructed, and configuring the misspelled word corresponding to the fourth word to be misconstructed as the third word to be misconstructed. For example, if the target text is "Today is sunny", the third word to be misconstructed and the fourth word to be misconstructed are "Today" and "sunny" respectively, then the misspelled word corresponding to the third word to be misconstructed is "sunny", and the misspelled word corresponding to the fourth word to be misconstructed is "Today". In this way, any two words to be misconstructed in the target text can be replaced in position, so as to construct an error of reversed word order type.
[0077] In some possible implementation manners, the third word to be misconstructed and the fourth word to be misconstructed are respectively two words before and after the first verb in the target text. By replacing the positions of the two words before and after the first verb, an error of inverted type can be constructed.
[0078] FIG. 3 shows a flowchart of determining valid misspelled words according to the n-gram model of misspelled words in an embodiment of the present application. In the embodiment of the present application, n is a positive integer greater than 1. As shown in FIG. 3, the process of determining valid misspelled words in the embodiment of the present application includes the following steps S31 and S32.
[0079] S31, obtain the n-gram model of the misspelled words, where the n-gram model of the misspelled words refers to a set of n-tuples composed of the words constructed by the misconstruction process and their surrounding words.
[0080] In some possible implementation manners, for any misspelled word, the method for obtaining its n-gram model is: in the target text, use the misspelled word to replace its corresponding word to be misconstructed to obtain an intermediate text and a misconstruction position; obtain an n-tuple composed of n words around the misconstruction position, and the set of this n-tuple is the n-gram model of the misspelled word. Among them, for the misspelled word generated by the insertion method, the misconstruction position is the position of the inserted text. For the misspelled word generated by the replacement method, the misconstruction position is the position of the replaced text. For the misspelled word generated by the deletion method, the misconstruction position is the deletion position. Taking the target text "Today the weather is nice" as an example, if "weather" is the word to be misconstructed and the misspelled word is "weathar", then the generated intermediate text is "Today weathar is nice", the misconstruction position is the position of "ar", and the two-word tuple composed of the two words around this position is "weathar" or "ar is", and the two-word tuple of the misspelled word is also "weathar" or "ar is".
[0081] S32, if the n-gram of the wrong word is a high-frequency word and the probability of the n-gram is greater than the probability threshold, the wrong word is determined to be an invalid wrong word, otherwise the wrong word is determined to be a valid wrong word. Among them, the probability threshold can be pre-configured according to demand. For example, if the target text is "according to the document record", the word to be misconstructed is "according to", and the wrong word constructed by insertion is "according to", then the intermediate text is "according to the document record", the misconstruction position is the position of "root", and the bigram composed of the two words around the position, that is, the bigram of the wrong word, is "according to". Since "according to" is a high-frequency word and the binary probability of "according to" is greater than the probability threshold, the wrong word is judged to be an invalid wrong word in step S32.
[0082] In some possible implementations, an n-gram dictionary can be generated by performing n-gram model statistics on the data in the corpus. Based on the n-gram dictionary, it can be determined whether the n-gram of the incorrect word is a high-frequency word and whether the n-gram probability of the n-gram is greater than the probability threshold.
[0083] Figure 4 shows a flow chart of generating error text in an embodiment of the present application. As shown in Figure 4, generating error text in the embodiment of the present application includes steps S41 and S42.
[0084] S41: translating a target text from a first language into a second language to obtain a target text in the second language.
[0085] S42, translating the target text in the second language into the first language to generate an error text.
[0086] For example, if the target text is in Chinese, the target text may be translated into English in step S41 , and the English target text may be translated into Chinese in step S42 to obtain an error text.
[0087] The present application also provides a training method that can be used to train a text rewriting tool. FIG5 shows a flow chart of the training method in the present application. As shown in FIG5, the training method provided in the present application includes the following steps S51 to S53.
[0088] S51, obtain the original corpus.
[0089] S52: Process the text in the original corpus using the text processing method provided in any embodiment of the present application to obtain error text.
[0090] S53, using the text in the original corpus and the erroneous text to train the text rewriting tool.
[0091] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the text processing method and / or the training method provided in any embodiment of the present application are implemented.
[0092] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments can be performed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive (SSD)).
[0093] In summary, the text processing method provided in the embodiments of the present application increases and refines the types of misconstruction errors, making the generated erroneous text more diverse and capable of covering most possible errors. In addition, the text processing method can use n-gram statistics to judge the validity of construction errors, which can achieve better mining effects. In some embodiments, the text processing method uses the Roformer model for mask prediction, which can directly process words of any length. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.
[0094] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.
Claims
1. A text processing method, characterized in that: The text processing method comprises: Get the target text; Performing word segmentation processing on the target text and obtaining the words to be misspelled; Performing misconstruction processing on each of the words to be misconstructed to obtain an incorrect word corresponding to each of the words to be misconstructed; Determine a valid error word according to an n-gram model of the error word, wherein n is a positive integer; An error text is generated according to the valid error words and the target text.
2. The text processing method according to claim 1, characterized in that: The words to be misformed include a first word to be misformed and / or a second word to be misformed, and performing misformation processing on each of the words to be misformed to obtain an incorrect word corresponding to each of the words to be misformed includes: Performing word-level misconstruction on the first word to be misconstructed to obtain an incorrect word corresponding to the first word to be misconstructed; and / or Perform word-level misconstruction on the second word to be misconstructed to obtain an incorrect word corresponding to the second word to be misconstructed.
3. The text processing method according to claim 2, characterized in that: Performing character-level misconstruction on the first word to be misconstructed includes: performing any one or more of the following steps to perform character-level misconstruction on the first word to be misconstructed; Replacing the target character in the first to-be-misformed word with a first character, wherein the first character includes a character that sounds similar to the target character, a character that looks similar to the target character, a character that begins with the same letter, a character that ends with the same letter, and / or a random common character; Deleting the target character in the first word to be misformed; A second character is inserted into the first word to be misformed, wherein the second character includes a character having the same beginning or ending as the target character, a random common character, and / or a random character in the target text.
4. The text processing method according to claim 2, characterized in that: Performing word-level misconstruction on the second word to be misconstructed includes: performing any one or more of the following steps to perform word-level misconstruction on the second word to be misconstructed; replacing the target word in the second to-be-misspelled word with a first word, wherein the first word includes a phonetic word of the target word, a keyboard recommended word, and / or a model recommended word; Deleting the target word in the second word to be misformed; A second word is inserted into the second word to be misformed, wherein the second word includes an adverb and / or a random word in the target text.
5. The text processing method according to claim 1, characterized in that: The words to be misconstructed include a third word to be misconstructed and a fourth word to be misconstructed, and performing misconstruction processing on the words to be misconstructed to obtain incorrect words corresponding to the words to be misconstructed includes: The incorrect word corresponding to the third word to be misformed is configured as the fourth word to be misformed, and the incorrect word corresponding to the fourth word to be misformed is configured as the third word to be misformed.
6. The text processing method according to claim 1, characterized in that: n is a positive integer greater than 1, and determining a valid error word according to the n-gram model of the error word includes: Obtaining n-grams of the erroneous words; If the n-gram of the erroneous word is a high-frequency word and the probability of the n-gram is greater than a probability threshold, the erroneous word is determined to be an invalid erroneous word, otherwise the erroneous word is determined to be a valid erroneous word.
7. The text processing method according to claim 1, characterized in that: Generating said error text includes: translating the target text from a first language into a second language to obtain a target text in the second language; The target text in the second language is translated into the first language to generate the error text.
8. A training method, characterized in that: The training method comprises: Get the original corpus; Using the text processing method described in any one of claims 1 to 7 to process the text in the original corpus to obtain erroneous text; The text rewriting tool is trained using the text in the original corpus and the erroneous text.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text processing method described in any one of claims 1 to 7 and / or the training method described in claim 8 are implemented.
10. An electronic device, characterized in that: The electronic device comprises: A memory storing a computer program; A processor is communicatively connected to the memory, and executes the text processing method described in any one of claims 1 to 7 and / or the training method described in claim 8 when calling the computer program.
Citation Information
Patent Citations
Speech text error correction method, system and equipment based on vertical field and medium
CN110210029A
Chinese marked error corpus generation method, calculation device and storage medium
CN111488466A
Text error correction method and device, medium and equipment
CN113392635A
Artificial intelligence-based text error correction method and device, equipment, and storage medium
CN113962215A
Text processing method, training method, storage medium and electronic equipment
CN116306620A
