Training method and device of pre-trained language model

Abstract

The application discloses a pre-training language model training method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: obtaining an input text, adding noise to the input text to obtain a noisy input text; traversing each slot of the output text from front to back: obtaining a historical decoding text formed by words of each slot before the current slot, adding noise to the historical decoding text to obtain a noisy historical decoding text, inputting the noisy input text and the noisy historical decoding text into a pre-training language model to obtain a probability distribution of each word in the noisy input text as a word of the current slot, iteratively training the pre-training language model according to the probability distribution to obtain the word of the current slot. The embodiment can make the decoding process not only train the language model, but also pay attention to the input text when noise is encountered, thereby improving the robustness of the pre-training language model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a training method and apparatus for a pre-trained language model. Background Technology

[0002] When decoding noisy text, decoders typically decode each word sequentially from left to right. However, in some scenarios, the decoder can infer the decoding result of the next word based on the decoding results of the preceding words. For example, if the input text is "popcorn is delicious," once "popcorn" is decoded, the decoder can directly "guess" that the next word in the slot is "flower" using a language model. This decoding method focuses too much on the output text and too little on the input text, resulting in low robustness of the pre-trained language model. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a training method and apparatus for a pre-trained language model, which can solve the above-mentioned problems existing in the prior art and improve the robustness of the pre-trained language model.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for training a pre-trained language model is provided, comprising:

[0005] Obtain the input text, and add noise to the input text to obtain the noisy input text;

[0006] Traverse each slot of the output text from front to back: obtain the historical decoded text formed by the words of each slot before the current slot, add noise to the historical decoded text to obtain the noisy historical decoded text, input the noisy input text and the noisy historical decoded text into a pre-trained language model to obtain the probability distribution of each word in the noisy input text for the word in the current slot, and iteratively train the pre-trained language model according to the probability distribution to obtain the word in the current slot;

[0007] The number of slots in the output text is equal to the number of words in the input text.

[0008] Optionally, noise is added to the input text, including at least one of the following: scrambling the input text, adding words to the input text, or replacing words with synonyms, similar-looking words, or parts of the input text.

[0009] Optionally, noise is added to the historical decoded text, including deleting some words from the historical decoded text.

[0010] Optionally, deleting some words in the historical decoded text includes deleting any one word in the historical decoded text.

[0011] Optionally, the pre-trained language model is iteratively trained according to the probability distribution, including: taking the word corresponding to the current slot in the input text as the target word, taking the maximum probability of the target word in the probability distribution as the objective function, and iteratively optimizing the model parameters in the pre-trained language model.

[0012] Optionally, the pre-trained language model is implemented based on the RNN framework or the Transformer framework.

[0013] Optionally, the pre-trained language model is implemented based on the Transformer framework; the pre-trained language model determines the probability distribution of each word in the noisy input text by using a mask matrix.

[0014] According to a second aspect of the present invention, a training apparatus for a pre-trained language model is provided, comprising:

[0015] The acquisition module acquires the input text and adds noise to the input text to obtain the noise input text.

[0016] The iterative module traverses each slot of the output text from front to back: it obtains the historical decoded text formed by the words of each slot before the current slot, adds noise to the historical decoded text to obtain the noisy historical decoded text, inputs the noisy input text and the noisy historical decoded text into the pre-trained language model, obtains the probability distribution of each word in the noisy input text for the word in the current slot, and iteratively trains the pre-trained language model according to the probability distribution to obtain the word in the current slot;

[0017] The number of slots in the output text is equal to the number of words in the input text.

[0018] Optionally, the acquisition module adds noise to the input text, including at least one of the following: scrambling the input text, adding words to the input text, or replacing words with synonyms, similar-looking words, or parts of the input text.

[0019] Optionally, the acquisition module adds noise to the historical decoded text, including deleting some words in the historical decoded text.

[0020] Optionally, the acquisition module performs partial word deletion processing on the historical decoded text, including: deleting any one word in the historical decoded text.

[0021] Optionally, the iterative module iteratively trains the pre-trained language model according to the probability distribution, including: taking the word corresponding to the current slot in the input text as the target word, taking the maximum probability of the target word in the probability distribution as the objective function, and iteratively optimizing the model parameters in the pre-trained language model.

[0022] Optionally, the pre-trained language model is implemented based on the RNN framework or the Transformer framework.

[0023] Optionally, the pre-trained language model is implemented based on the Transformer framework; the pre-trained language model determines the probability distribution of each word in the noisy input text by using a mask matrix.

[0024] According to a third aspect of the present invention, an electronic device for text recognition is provided, comprising:

[0025] One or more processors;

[0026] Storage device for storing one or more programs.

[0027] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present invention.

[0028] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0029] One embodiment of the above invention has the following advantages or beneficial effects: by adding noise to the historical decoded text corresponding to the current slot to obtain the noisy historical decoded text, and decoding the word in the current slot according to the noisy input text and the noisy historical decoded text, the decoding process not only trains the language model, but also needs to pay attention to the input text when encountering noise, thereby improving the robustness of the pre-trained language model.

[0030] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0031] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0032] Figure 1 This is a schematic diagram of the main process of training a pre-trained language model according to an embodiment of the present invention;

[0033] Figure 2This is a schematic diagram of the main modules of the training device for the pre-trained language model according to an embodiment of the present invention;

[0034] Figure 3 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0035] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0036] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] According to one aspect of the present invention, a method for training a pre-trained language model is provided. The pre-trained language model is used to decode noisy text. The present invention aims to train the pre-trained language model using input text as samples, wherein the input text is the input of the pre-trained language model, and the output text of the pre-trained language model is the same as the input text. For each input text contained in the sample set, the pre-trained language model can be trained separately according to the method of the present invention.

[0038] Figure 1 This is a schematic diagram of the main flow of the training method for the pre-trained language model according to an embodiment of the present invention, as shown below. Figure 1 As shown, the training method for the pre-trained language model according to an embodiment of the present invention includes steps S101-S106.

[0039] In step S101, the input text is obtained, and noise is added to the input text to obtain noisy input text. The noisy input text after adding noise in this step is used as the decoding object of the pre-trained language model. The words in each slot of the output text are decoded through steps S102-S106. By training the pre-trained language model, the output text decoded by the pre-trained language model is made to be the same as the input text.

[0040] The method of adding noise to the input text can be selectively set according to the actual situation. For example, the input text can be scrambled, for example, adding noise to the input text "popcorn is delicious" to scramble it, resulting in the noisy input text "popcorn is delicious, rice is delicious"; words can be added to the input text, for example, adding words to the input text "popcorn is delicious" to obtain the noisy input text "popcorn is delicious, rice is delicious, don't eat rice is delicious"; synonyms, similar-looking words, or parts of the words in the input text can be used for replacement, for example, converting the input text "popcorn is delicious" into the noisy input text "pocket flower is delicious, rice is delicious, don't eat rice is delicious".

[0041] Each slot in the output text corresponds to a word, and the number of slots in the output text is equal to the number of words in the input text. Initially, all slots in the output text are empty. In step S102, the historical decoded text formed by the words in all slots before the i-th slot in the output text is obtained; i is a positive integer, and its initial value is 1. For example, taking the input text "popcorn is delicious" as an example, when decoding the word in the 3rd slot of the output text, the previously decoded historical decoded text "popcorn" is obtained; when decoding the word in the 4th slot of the output text, the previously decoded historical decoded text "popcorn" is obtained; when decoding the word in the 5th slot of the output text, the previously decoded historical decoded text "popcorn is very" is obtained; when decoding the word in the 6th slot of the output text, the previously decoded historical decoded text "popcorn is very good" is obtained, and so on.

[0042] In step S103, noise is added to the historical decoded text to obtain noisy historical decoded text. The method of adding noise to the historical decoded text can be selectively set according to the actual situation. For example, taking the input text "popcorn is delicious" as an example, when decoding the word in the 4th slot of the output text, the previously decoded historical decoded text "popcorn" is scrambled to obtain the noisy historical decoded text "popcorn". In an optional embodiment, adding noise to the historical decoded text includes deleting some words in the historical decoded text. In actual application, only one word in the historical decoded text can be deleted, or two or more words can be deleted; any one or more words in the historical decoded text can be deleted, or one or more words at preset positions in the historical decoded text can be deleted. For example, taking the input text "popcorn is delicious" as an example, when decoding the word in the 6th slot of the output text, noise is added to the previously decoded historical decoded text "popcorn is delicious" to obtain the noisy historical decoded text "popcorn is delicious" or "popcorn is delicious".

[0043] In step S104, the noisy input text and the noisy historical decoded text are input into the pre-trained language model to obtain the probability distribution that the word in the i-th slot is each word in the noisy input text. That is, the probability that all the words in the noisy input text that have not been filled into the output text slot should be filled into the current slot. Exemplarily, taking the noisy input text "zhong bao hua hen mi hao bu chi fan hao" as an example, when decoding the word in the second slot of the output text, the probabilities that the words "zhong", "bao", "hua", "hen", "mi", "hao", "bu", "chi", "fan", "hao" should be filled into the second slot are respectively determined. To minimize the computational amount as much as possible, when determining the probability distribution that the word in the i-th slot is each word in the noisy input text, only the words that have not been filled into the output text slot may be analyzed. Exemplarily, taking the noisy input text "zhong bao hua hen mi hao bu chi fan hao" as an example, when decoding the word in the second slot of the output text, since the first word "bao" in the noisy input text has been written into the first slot of the output text, at this time, the probabilities that the words "zhong", "hua", "hen", "mi", "hao", "bu", "chi", "fan", "hao" should be filled into the second slot can be determined, and the probability that "bao" should be filled into the second slot is not determined.

[0044] In this step, when decoding the word in the current slot, decoding is performed according to the historical decoded text with added noise, thereby increasing the decoding difficulty and making the decoder pay more attention to the input text when eliminating noise, so as to improve the robustness of the pre-trained language model obtained by training.

[0045] In step S105, iterative training is performed on the pre-trained language model according to the probability distribution. The model parameters of the pre-trained language model are continuously adjusted through iterative training so that the probability that the word corresponding to the current slot in the input text should be filled into the current slot is greater than the set threshold as the objective function. Exemplarily, taking the input text "popcorn is very delicious" as an example, when decoding the word in the second slot of the output text, the probability that the word to be filled into the second slot in the output text is "mi" is greater than the set threshold as the objective function. In an optional embodiment, performing iterative training on the pre-trained language model according to the probability distribution includes: using the word corresponding to the current slot in the input text as the target word, and using the probability corresponding to the target word in the probability distribution being the largest as the objective function, and iteratively optimizing the model parameters in the pre-trained language model.

[0046] Optionally, the pre-trained language model is implemented based on the RNN (Recurrent Neural Network) framework or the Transformer (a NLP model) framework.

[0047] Optionally, the pre-trained language model is implemented based on the Transformer framework; the pre-trained language model determines the probability distribution of each word in the input text that the word in the current slot is noise through a mask matrix. The method of this embodiment of the invention, implemented based on the Transformer framework, is computationally fast and efficient.

[0048] After the iteration in step S105 is completed, proceed to step S106. In step S106, determine whether the current slot number is equal to N, where N represents the number of words in the input text. If yes, it indicates that the decoding process based on the current input text has ended, and the output text obtained is the same as the input text, and the process ends; if no, it indicates that the decoding process based on the current input text has not ended, and some slots in the output text are still undecoded. In this case, increment i by 1 and jump to step S102. Each execution of steps S102-S106 can decode one word in the output text. By repeatedly executing steps S102-S106, each word in the output text can be obtained sequentially.

[0049] The process of this embodiment of the invention is described in detail below with reference to specific embodiments. Noise is added during decoding, and some decoded words are randomly deleted. When decoding the current word, decoding is performed based on the historical decoded text with added noise, thereby increasing the decoding difficulty and making the decoder pay more attention to the input text when removing noise. The training method of the pre-trained language model in this embodiment includes the following steps:

[0050] Step 1) Given a monolingual dataset, for each sentence x, randomly shuffle it, and denote the new sequence as x. m .

[0051] Step 2) Based on the Transformer encoder-decoder framework, the encoder first applies x... m Encode the hidden state sequence to complete the encoding.

[0052] Step 3) The decoder recovers the target text x one by one from left to right, and the training objective function is:

[0053] L t =-logP(x t |x m , x′ ＜t-1 )

[0054] Where x′ ＜t-1 This involves adding noise to the historical decoded text, specifically by deleting certain words, such as the word "x". i Then x′ ＜t-1 = (x1, x2, ..., x i-1 x i+1 , ..., x t-1 ).

[0055] The pre-trained language model proposed in this embodiment can be directly implemented using an RNN framework. Since the Transformer-based pre-trained language model performs parallel computation during decoding, the specific implementation process is as follows:

[0056] The Transformer takes a one-time input (x1, x2, ..., x3) on the decoding side. n This method uses a mask matrix to decode from left to right. In other words, words on the right can see words on the left during decoding, but words on the left cannot see words on the right. For example, x3 can see x1 and x2, but cannot see x4 and the text after it. Let M represent the mask matrix. ij For x i For x j The mask. When i ≤ j, M ij =0, that is, x i Cannot see x j When i > j, M ij =1, that is, x i We can see x j The deletion of certain decoded words proposed in this embodiment can be achieved by modifying the mask matrix M, for example, during the decoding of x. k When, delete x l This can make M kl =0, thus making x =0 k Cannot see x l That is, the task of x has been completed. l The deletion operation.

[0057] The system itself makes the pre-trained language model more robust by randomly deleting certain words that have already been decoded in the decoder during the training of the pre-trained language model.

[0058] According to a second aspect of the present invention, an apparatus for implementing the above-described method is provided.

[0059] Figure 2 This is a schematic diagram of the main modules of the training device for the pre-trained language model according to an embodiment of the present invention. Figure 2 As shown, the training device 200 for the pre-trained language model includes:

[0060] Module 201 acquires the input text and adds noise to it to obtain the noisy input text.

[0061] Iteration module 202 traverses each slot of the output text from front to back: obtains the historical decoded text formed by the words of each slot before the current slot, adds noise to the historical decoded text to obtain the noisy historical decoded text, inputs the noisy input text and the noisy historical decoded text into the pre-trained language model, obtains the probability distribution of each word in the noisy input text as the word in the current slot, and iteratively trains the pre-trained language model according to the probability distribution to obtain the word in the current slot;

[0062] The number of slots in the output text is equal to the number of words in the input text.

[0063] Optionally, the acquisition module adds noise to the input text, including at least one of the following: scrambling the input text, adding words to the input text, or replacing words with synonyms, similar-looking words, or parts of the input text.

[0064] Optionally, the acquisition module adds noise to the historical decoded text, including deleting some words in the historical decoded text.

[0065] Optionally, the acquisition module performs partial word deletion processing on the historical decoded text, including: deleting any one word in the historical decoded text.

[0066] Optionally, the iterative module iteratively trains the pre-trained language model according to the probability distribution, including: taking the word corresponding to the current slot in the input text as the target word, taking the maximum probability of the target word in the probability distribution as the objective function, and iteratively optimizing the model parameters in the pre-trained language model.

[0067] Optionally, the pre-trained language model is implemented based on the RNN framework or the Transformer framework.

[0068] Optionally, the pre-trained language model is implemented based on the Transformer framework; the pre-trained language model determines the probability distribution of each word in the noisy input text by using a mask matrix.

[0069] According to a third aspect of the present invention, an electronic device for text recognition is provided, comprising:

[0070] One or more processors;

[0071] Storage device for storing one or more programs.

[0072] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present invention.

[0073] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, wherein the program, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0074] Figure 3 An exemplary system architecture 300 is shown, which can be used to train a pre-trained language model according to embodiments of the present invention, or to train a pre-trained language model.

[0075] like Figure 3 As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305. Network 304 serves as the medium for providing communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0076] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, and 303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0077] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0078] Server 305 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 301, 302, and 303 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information and product information - for example only) to the terminal device.

[0079] It should be noted that the training method of the pre-trained language model provided in the embodiments of the present invention is generally executed by the server 305, and correspondingly, the training device of the pre-trained language model is generally set in the server 305.

[0080] It should be understood that Figure 3 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0081] The following is for reference. Figure 4It shows a schematic diagram of the structure of a computer system 400 suitable for implementing a terminal device of the present invention. Figure 4 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0082] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0083] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0084] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this invention.

[0085] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module and an iteration module. The names of these modules do not necessarily limit the module itself; for example, the acquisition module can also be described as "a module for iteratively training a pre-trained language model according to a probability distribution."

[0088] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0089] The process involves: acquiring the input text, adding noise to it to obtain the noisy input text; traversing each slot of the output text from front to back; acquiring the historical decoded text formed by the words of the slots preceding the current slot, adding noise to the historical decoded text to obtain the noisy historical decoded text; inputting the noisy input text and the noisy historical decoded text into a pre-trained language model to obtain the probability distribution of each word in the noisy input text for the word in the current slot; iteratively training the pre-trained language model based on the probability distribution to obtain the word in the current slot; and ensuring that the number of slots in the output text is equal to the number of words in the input text.

[0090] According to the technical solution of the present invention, noise is added to the historical decoded text corresponding to the current slot to obtain noisy historical decoded text, and the word in the current slot is decoded according to the noisy input text and the noisy historical decoded text. This makes the decoding process not only train the language model, but also pay attention to the input text when encountering noise, thereby improving the robustness of the pre-trained language model.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A training method for a pre-trained language model, characterized in that, include: Obtain the input text, and add noise to the input text to obtain the noisy input text; Traverse each slot of the output text from front to back: obtain the historical decoded text formed by the words of each slot before the current slot, add noise to the historical decoded text to obtain the noisy historical decoded text, input the noisy input text and the noisy historical decoded text into a pre-trained language model to obtain the probability distribution of each word in the noisy input text for the word in the current slot, and iteratively train the pre-trained language model according to the probability distribution to obtain the word in the current slot; The number of slots in the output text is equal to the number of words in the input text; adding noise to the input text includes: scrambling the input text; deleting some words in the historical decoded text includes: deleting any one word in the historical decoded text.

2. The method as described in claim 1, characterized in that, Adding noise to the input text further includes at least one of the following: adding words to the input text, replacing them with synonyms, similar-looking words, or parts of the words in the input text.

3. The method as described in claim 1, characterized in that, Deleting certain words from the historical decoded text includes deleting any single word from the historical decoded text.

4. The method as described in claim 1, characterized in that, Iterative training of the pre-trained language model based on the probability distribution includes: The model parameters in the pre-trained language model are iteratively optimized by taking the word corresponding to the current slot in the input text as the target word and taking the probability of the target word being maximized in the probability distribution as the objective function.

5. The method as described in claim 1, characterized in that, The pre-trained language model is implemented based on the RNN framework or the Transformer framework.

6. The method as described in claim 1, characterized in that, The pre-trained language model is implemented based on the Transformer framework; the pre-trained language model determines the probability distribution of each word in the noisy input text by using a mask matrix.

7. A training device for a pre-trained language model, characterized in that, include: The acquisition module acquires the input text and adds noise to the input text to obtain the noise input text. The iterative module traverses each slot of the output text from front to back: it obtains the historical decoded text formed by the words of each slot before the current slot, adds noise to the historical decoded text to obtain the noisy historical decoded text, inputs the noisy input text and the noisy historical decoded text into the pre-trained language model, obtains the probability distribution of each word in the noisy input text for the word in the current slot, and iteratively trains the pre-trained language model according to the probability distribution to obtain the word in the current slot; The number of slots in the output text is equal to the number of words in the input text; adding noise to the input text includes: scrambling the input text; deleting some words in the historical decoded text includes: deleting any one word in the historical decoded text.

8. An electronic device for text recognition, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images