Training method and related device of language model based on RNN

Abstract

The present invention provides a training method and related device for a language model based on RNN, the method comprising dividing the elements of a long sentence training data into at least two parts according to time, calculating an Nce loss value once for each part; calculating a gradient using BPTT according to the Nce loss value; and finally updating the parameters of the RNN-based language model according to the calculated gradient. The Nce loss value is calculated once for each part of the long sentence training data, and the parameters of the RNN-based language model are updated multiple times, thereby improving the training speed of the language model. A long sentence training data is divided into at least two parts, and a gradient optimization model is calculated based on the split parts. Since the split parts have fewer time points in the time series than the unsplit long sentence training data, the probability of gradient disappearance is reduced, which can improve the training effect of the language model, that is, improve the prediction accuracy of the language model.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a training method and related device for a language model based on RNN. Background Art

[0002] The language model based on RNN (Recurrent Neural Network) is actually a prediction model. The prediction space is the dictionary space, and the result is the posterior probability of each event in the prediction space. At present, considering the problem of computational efficiency, the Nce (Noise-ContrastiveEstimation) function is usually used when training the language model based on RNN. When training the model, all positive samples are divided into n batches, each batch (i.e., batch) includes several data, and for each batch, a group of negative samples is prepared; the negative samples are extracted from the preset vocabulary. Using each batch of positive samples and the corresponding negative samples, the Nce loss is calculated; after calculating the Nce loss, the BPTT (Backpropagation Through Time) algorithm is used to update the parameters of the model.

[0003] The inventors found that RNN uses time-propagation, that is, the gradient is propagated backward step by step along the time series, which can easily cause the gradient to vanish. The gradient vanishing will cause the gradient during model training to be unable to affect the parameter adjustment, resulting in insufficient convergence and causing the prediction results of the trained language model to be biased. Summary of the invention

[0004] In view of this, the present invention proposes a training method and related device for a language model based on RNN, in order to increase the training speed of the language model and improve the training effect of the language model.

[0005] In order to achieve the above objectives, the proposed solution is as follows:

[0006] In a first aspect, a training method for a language model based on an RNN is provided, comprising:

[0007] Divide the elements of a long sentence training data into N parts according to time, N ≥ 2;

[0008] For the earliest part of the long sentence training data for which the Nce loss value has not been calculated, a group of negative samples corresponding thereto are combined to calculate the Nce loss value;

[0009] According to the calculated Nce loss value, the gradient is calculated using BPTT;

[0010] Updating parameters of the RNN-based language model according to the gradient;

[0011] Determine whether the Nce loss value has been calculated for all parts of the long sentence training data. If not, proceed to execute the step of calculating the Nce loss value for the earliest time in the long sentence training data for which the Nce loss value has not been calculated, combined with a group of negative samples corresponding thereto. If so, end.

[0012] Preferably, before each execution of the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto, the step further includes:

[0013] A number of elements are randomly selected from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not been calculated.

[0014] Preferably, after the step of dividing the elements of a long sentence training data into N parts according to time, and before the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto, the method further includes:

[0015] Randomly extract elements from the preset element library to generate M groups of negative samples, 2≤M≤N;

[0016] The N parts of the long sentence training data are divided into M groups, and each group in the M groups divided from the N parts corresponds one-to-one to each group of negative samples in the M groups of negative samples.

[0017] Preferably, the RNN-based language model is specifically:

[0018] LSTM-based language model.

[0019] In a second aspect, a training device for a language model based on an RNN is provided, comprising:

[0020] An element division unit is used to divide the elements of a long sentence training data into N parts according to time, where N ≥ 2;

[0021] A loss value calculation unit, configured to calculate the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto;

[0022] A gradient calculation unit is used to calculate the gradient using BPTT according to the calculated Nce loss value;

[0023] A model optimization unit, configured to update parameters of a RNN-based language model according to the gradient;

[0024] The judging unit is used to judge whether the Nce loss value has been calculated for all the parts of the long sentence training data. If not, the process proceeds to the loss value calculation unit. If yes, the process ends.

[0025] Preferably, the RNN-based language model training device further includes:

[0026] The first negative sample generating unit is used to randomly extract a number of elements from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not been calculated before executing the loss value calculating unit each time.

[0027] Preferably, the RNN-based language model training device further includes:

[0028] The second negative sample generating unit is used to randomly extract elements from a preset element library to generate M groups of negative samples, 2≤M≤N; and divide the N parts of the long sentence training data into M groups, and each group in the M groups divided by the N parts corresponds to each group of negative samples in the M groups of negative samples.

[0029] Preferably, the RNN-based language model is specifically:

[0030] LSTM-based language model.

[0031] In a third aspect, a readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the various steps of any one of the RNN-based language model training methods in the first aspect are implemented.

[0032] In a fourth aspect, a training device for a language model based on an RNN is provided, comprising: a memory and a processor;

[0033] The memory is used to store programs;

[0034] The processor is used to execute the program to implement the various steps of any one of the RNN-based language model training methods in the first aspect.

[0035] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0036] The above technical solution provides a training method and related device for a language model based on RNN, the method comprising: dividing the elements of a long sentence training data into at least two parts according to time, calculating the Nce loss value once for each part; calculating the gradient using BPTT according to the Nce loss value; and finally updating the parameters of the RNN-based language model according to the calculated gradient. The elements of a long sentence training data are divided into at least two parts according to time, calculating the Nce loss value once for each part, and updating the parameters of the RNN-based language model multiple times. Compared with updating the parameters of the RNN-based language model once for a long sentence training data, the training speed of the language model is improved. A long sentence training data is divided into at least two parts, and a gradient optimization model is calculated based on each split part. Since the time points in the time series of each split part are reduced compared to the unsplit long sentence training data, the probability of gradient disappearance is reduced, which can improve the training effect of the language model, that is, improve the prediction accuracy of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 Schematic diagram of existing schemes for training RNN-based language models;

[0039] Figure 2 A schematic diagram of a training method for a language model based on an RNN provided in an embodiment of the present invention;

[0040] Figure 3 A flowchart of a method for training a language model based on an RNN provided in an embodiment of the present invention;

[0041] Figure 4 A flowchart of another RNN-based language model training method provided by an embodiment of the present invention;

[0042] Figure 5 A flowchart of another RNN-based language model training method provided in an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of a training device for a language model based on an RNN provided by an embodiment of the present invention;

[0044] Figure 7A schematic diagram of a RNN-based language model training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1 , which is an existing solution for training a language model based on RNN. X1, X2, ..., Xn are all elements included in a long sentence training data. When X1 is input into the RNN-based language model to calculate Y1, X2 is input into the RNN-based language model to calculate Y2, ..., and Xn is input into the RNN-based language model to calculate Yn, the same set of negative samples are used. This is because it is necessary to uniformly calculate Nce loss (i.e., Nce loss value) after Y1, Y2, ..., Yn are all calculated, and then a vector (i.e., gradient) is calculated through the Nce loss value to update the parameters of the calculation function in the RNN-based language model so that it can better fit the current data distribution. However, the method of calculating the gradient through Nce loss is BPTT, which is time-related. One of the characteristics is that the more elements there are in each long sentence training data, the greater the probability of gradient disappearance.

[0047] In order to solve the above problems, the core idea of ​​the present invention is to divide the elements of the long sentence training data into at least two parts according to time, calculate the Nce loss once for each part, and then update the language model once. The elements of a long sentence training data are divided into at least two parts according to time, and the parameters of the RNN-based language model are updated multiple times. Compared with updating the parameters of the RNN-based language model once for a long sentence training data, the training speed of the language model is improved. A long sentence training data is divided into at least two parts, and the gradient optimization model is calculated based on each split part. Since the time points in the time series of each split part are reduced compared to the unsplit long sentence training data, the probability of gradient disappearance is reduced, which can improve the training effect of the language model, that is, improve the prediction accuracy of the language model.

[0048] See also Figure 2, which is a schematic diagram of a training method for a language model based on RNN provided in an embodiment of the present invention. In this embodiment, two elements are divided into a part; the earliest part in time series is combined with a group of negative samples corresponding thereto, the Nce loss value is calculated, and the parameters of the language model are updated; then the remaining earliest part in time series is combined with a group of negative samples corresponding thereto, the Nce loss value is calculated, and the parameters of the language model are updated, until the Nce loss value is calculated for each part and the parameters of the language model are updated. In a specific embodiment, the language model based on RNN is specifically a language model based on LSTM. The elements can be characters or words, which are not limited in the present invention and belong to the protection scope of the present invention.

[0049] See also Figure 3 , is a flowchart of a method for training a language model based on an RNN provided by an embodiment of the present invention, the method may include the following steps:

[0050] S31: Divide the elements of a long sentence training data into N parts according to time, where N≥2.

[0051] The data input to the recurrent neural network is a sequence of time series. Each element of a long sentence training data represents a time point in the time series, and an output is calculated at each time point. The calculation of this output depends on a set of negative samples. For example, the long sentence training data is "How is the weather today?" Today, weather, and how are they each an element, corresponding to a time point in the time series. Every time a word is input to the recurrent neural network, the recurrent neural network outputs the next most likely word so far.

[0052] S32: For the earliest part of the long sentence training data for which the Nce loss value has not been calculated, the Nce loss value is calculated in combination with a set of negative samples corresponding thereto.

[0053] The part where the Nce loss value is not calculated refers to the part where the output has not been calculated using the elements of the part, and the Nce loss value is calculated using all the outputs calculated using the elements of the part. The earliest part refers to the time point represented by the elements in the part, which is earlier than the time points represented by the elements of other parts.

[0054] Existing technical solutions such as Figure 1 As shown, after obtaining the output at each time point (i.e., Y1, Y2, ..., Yn-1, and Yn), the Nce loss value is calculated; while the present invention calculates the Nce loss value after obtaining the output at each time point of a certain part of the long sentence training data. For example, after obtaining Figure 2After the outputs Y1 and Y2 in , the Nce loss value is calculated, and a subsequent model parameter update is performed. Specifically, for a part of the long sentence training data, combined with a set of negative samples corresponding to it, the process of calculating the Nce loss value is as follows: after determining a part of the positive samples and multiple negative samples, according to the Nce binary classification criterion, combined with the sampling distribution probability and the cost value of the corresponding sample output by the RNN, the probability of the positive sample being classified as positive and the probability of the negative sample being classified as negative can be calculated respectively. Combining these probability values, the approximate expected loss function value, that is, the NCE loss value, can be calculated.

[0055] S33: According to the calculated Nce loss value, the gradient is calculated using BPTT.

[0056] S34: Update the parameters of the RNN-based language model according to the calculated gradient.

[0057] The Nce loss value in the prior art and the method of calculating the gradient using BPTT can be adopted in this embodiment; the method of updating the parameters of the RNN-based language model based on the gradient in the prior art can also be adopted in this embodiment, and the present invention does not limit this.

[0058] S35: Determine whether the Nce loss value has been calculated for all parts of the long sentence training data. If not, proceed to step S32. If yes, end.

[0059] The RNN-based language model training method provided in the above embodiment divides the elements of a long sentence training data into at least two parts according to time, and updates the parameters of the RNN-based language model multiple times, which improves the training speed of the language model compared to updating the parameters of the RNN-based language model once for a long sentence training data. A long sentence training data is divided into at least two parts, and a gradient optimization model is calculated based on each of the split parts. Since the time points in the time series of each split part are reduced compared to the unsplit long sentence training data, the probability of gradient disappearance is reduced, which can improve the training effect of the language model, that is, improve the prediction accuracy of the language model.

[0060] Furthermore, for Figure 1 In the scheme shown, the inventors also found that the batch of positive samples includes many pieces of data, and when the data in the batch uses the same set of negative samples, it is easily biased by the random distribution of the group of negative samples, which leads to deviations in the prediction results of the trained language model. In response to this technical problem, the present invention uses multiple groups of negative samples for a batch, and different groups of negative samples have different element distributions, which are more random and reduce the prediction deviation of the trained language model.

[0061] See also Figure 4 , which is another RNN-based language model training method provided in this embodiment. Figure 3 The method shown in the figure further includes, before each execution of the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a set of negative samples corresponding thereto, the following steps:

[0062] Step S42: randomly extracting a number of elements from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not yet been calculated.

[0063] An element library is pre-set, and the elements included in the element library are not correct elements. Multiple elements are obtained from the element library to form a group of negative samples. Each element in a group of negative samples is a negative sample. The number of elements in a group of negative samples is related to the actual application requirements, and the present invention does not limit this.

[0064] Steps S41, S43, S44, S45, and S46 are respectively consistent with S31, S32, S33, S34, and S35, and will not be repeated in this embodiment.

[0065] See also Figure 5 , which is another RNN-based language model training method provided in this embodiment. Figure 3 The method shown, after the step of dividing the elements of a long sentence training data into N parts according to time, and before the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a set of negative samples corresponding thereto, further includes:

[0066] Step S52: randomly extracting elements from a preset element library to generate M groups of negative samples, 2≤M≤N.

[0067] Execute step S52 to generate M groups of negative samples. The specific value of M can be set according to actual applications, and the present invention does not limit this. As long as 2≤M≤N, it falls within the protection scope of the present invention.

[0068] Step S53: Divide the N parts of the long sentence training data into M groups, and each group in the M groups divided from the N parts corresponds one-to-one to each group of negative samples in the M groups of negative samples.

[0069] When M=N, each of the N parts of the long sentence training data is a group, so each part corresponds to a group of negative samples.

[0070] Steps S51, S54, S55, S56, and S57 are respectively consistent with S31, S32, S33, S34, and S35, and will not be repeated in this embodiment.

[0071] For the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0072] See also Figure 6 , a training device for a language model based on RNN provided in an embodiment of the present invention, includes an element division unit 61, a loss value calculation unit 62, a gradient calculation unit 63, a model optimization unit 64 and a judgment unit 65.

[0073] The element division unit 61 is used to divide the elements of a long sentence training data into N parts according to time, where N≥2.

[0074] The loss value calculation unit 62 is used to calculate the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, combined with a group of negative samples corresponding thereto.

[0075] The gradient calculation unit 63 is used to calculate the gradient using BPTT according to the calculated Nce loss value.

[0076] The model optimization unit 64 is used to update the parameters of the RNN-based language model according to the gradient.

[0077] The judging unit 65 is used to judge whether the Nce loss value has been calculated for all the parts of the long sentence training data. If not, the process proceeds to the loss value calculating unit 62. If yes, the process ends.

[0078] In some specific embodiments, the RNN-based language model is an LSTM-based language model.

[0079] In some specific embodiments, the RNN-based language model training device also includes: a first negative sample generation unit, which is used to randomly extract a number of elements from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not been calculated before each execution of the loss value calculation unit.

[0080] In some specific embodiments, the training device of the RNN-based language model also includes: a second negative sample generation unit, used to randomly extract elements from a preset element library to generate M groups of negative samples, 2≤M≤N; and divide the N parts of the long sentence training data into M groups, and each group in the M groups divided by the N parts corresponds one-to-one to each group of negative samples in the M groups of negative samples.

[0081] This embodiment also provides a training device for a language model based on RNN, such as a PC terminal, a cloud platform, a server, and a server cluster. The server can be one or more of a rack server, a blade server, a tower server, and a cabinet server. Figure 7 , which is a RNN-based language model training device provided in this embodiment. The hardware structure of the RNN-based language model training device may include: at least one processor 71, at least one communication interface 72, at least one memory 73 and at least one communication bus 74; and the processor 71, the communication interface 72, and the memory 73 communicate with each other through the communication bus 74.

[0082] In some embodiments, the processor 71 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0083] The communication interface 72 may include a standard wired interface or a wireless interface, and is generally used to establish a communication connection between a training device based on an RNN language model and other electronic devices or systems.

[0084] The memory 73 includes at least one type of readable storage medium. The readable storage medium may be NVM (non-volatile memory) such as flash memory, hard disk, multimedia card, card-type memory, etc. The readable storage medium may also be a high-speed RAM (random access memory) memory. In some embodiments, the readable storage medium may be an internal storage unit of a training device based on a language model of RNN, such as a hard disk of the training device based on the language model of RNN. In other embodiments, the readable storage medium may also be an external storage device of a training device based on a language model of RNN, such as a plug-in hard disk, SMC (Smart Media Card, smart memory card), SD (Secure Digital, secure digital) card, flash card (FlashCard), etc. equipped on the training device based on the language model of RNN.

[0085] The memory 73 stores a computer program, and the processor 71 can call the computer program stored in the memory 73, wherein the computer program is used to:

[0086] Divide the elements of a long sentence training data into N parts according to time, N ≥ 2;

[0087] For the earliest part of the long sentence training data for which the Nce loss value has not been calculated, a group of negative samples corresponding thereto are combined to calculate the Nce loss value;

[0088] According to the calculated Nce loss value, the gradient is calculated using BPTT;

[0089] Updating parameters of the RNN-based language model according to the gradient;

[0090] Determine whether the Nce loss value has been calculated for all parts of the long sentence training data. If not, proceed to execute the step of calculating the Nce loss value for the earliest time in the long sentence training data for which the Nce loss value has not been calculated, combined with a group of negative samples corresponding thereto. If so, end.

[0091] The detailed and extended functions of the program can refer to the above description.

[0092] Figure 7 Only a training device for an RNN-based language model having components 71 to 74 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0093] Optionally, the RNN-based language model training device may further include a user interface, which may include an input unit (such as a keyboard), a voice input device (such as a device with a voice recognition function including a microphone) and / or a voice output device (such as a speaker, earphones, etc.). Optionally, the user interface may also include a standard wired interface and / or a wireless interface.

[0094] Optionally, the RNN-based language model training device may further include a display, which may also be referred to as a display screen or a display unit. In some embodiments, it may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) display.

[0095] Optionally, the training device of the RNN-based language model also includes a touch sensor. The area provided by the touch sensor for the user to perform touch operation is called a touch control area. In addition, the touch sensor can be a resistive touch sensor, a capacitive touch sensor, etc. Moreover, the touch sensor includes not only a contact touch sensor, but also a proximity touch sensor, etc. In addition, the touch sensor can be a single sensor or a plurality of sensors arranged in an array, for example. The user can input identity information by touching the touch control area.

[0096] In addition, the area of ​​the display of the training device based on the RNN language model can be the same as or different from the area of ​​the touch sensor. Optionally, the display and the touch sensor are stacked to form a touch display screen. The device detects touch operations triggered by the user based on the touch display screen.

[0097] The RNN-based language model training device may also include an RF (Radio Frequency) circuit, a sensor, an audio circuit, etc., which will not be repeated here.

[0098] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0099] Divide the elements of a long sentence training data into N parts according to time, N ≥ 2;

[0100] For the earliest part of the long sentence training data for which the Nce loss value has not been calculated, a group of negative samples corresponding thereto are combined to calculate the Nce loss value;

[0101] According to the calculated Nce loss value, the gradient is calculated using BPTT;

[0102] Updating parameters of the RNN-based language model according to the gradient;

[0103] Determine whether the Nce loss value has been calculated for all parts of the long sentence training data. If not, proceed to execute the step of calculating the Nce loss value for the earliest time in the long sentence training data for which the Nce loss value has not been calculated, combined with a group of negative samples corresponding thereto. If so, end.

[0104] The detailed and extended functions of the program can refer to the above description.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0106] In this article, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0107] Each embodiment in this specification focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other, and the features recorded in each embodiment in this specification can be replaced or combined with each other.

[0108] The above description of the embodiments disclosed in the present invention enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a language model based on RNN, characterized in that: include: Divide the elements of a long sentence training data into N parts according to time, N ≥ 2; For the earliest part of the long sentence training data for which the Nce loss value has not been calculated, a group of negative samples corresponding thereto are combined to calculate the Nce loss value; According to the calculated Nce loss value, the gradient is calculated using BPTT; Updating parameters of the RNN-based language model according to the gradient; Determine whether the Nce loss value has been calculated for all parts of the long sentence training data. If not, proceed to the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, combined with a group of negative samples corresponding to it. If yes, end.

2. The RNN-based language model training method according to claim 1, characterized in that: Before each execution of the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto, the step further includes: A number of elements are randomly selected from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not been calculated.

3. The RNN-based language model training method according to claim 1, characterized in that: After the step of dividing the elements of a long sentence training data into N parts according to time, and before the step of calculating the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto, the step further includes: Randomly extract elements from the preset element library to generate M groups of negative samples, 2≤M≤N; The N parts of the long sentence training data are divided into M groups, and each group in the M groups divided from the N parts corresponds one-to-one to each group of negative samples in the M groups of negative samples.

4. The RNN-based language model training method according to any one of claims 1 to 3, characterized in that: The RNN-based language model is specifically: LSTM-based language model.

5. A training device for a language model based on RNN, characterized in that: include: An element division unit is used to divide the elements of a long sentence training data into N parts according to time, where N ≥ 2; A loss value calculation unit, configured to calculate the Nce loss value for the earliest part of the long sentence training data for which the Nce loss value has not been calculated, in combination with a group of negative samples corresponding thereto; A gradient calculation unit is used to calculate the gradient using BPTT according to the calculated Nce loss value; A model optimization unit, configured to update parameters of a RNN-based language model according to the gradient; The judging unit is used to judge whether the Nce loss value has been calculated for all the parts of the long sentence training data. If not, the process proceeds to the loss value calculation unit. If yes, the process ends.

6. The RNN-based language model training device according to claim 5, characterized in that: Also includes: The first negative sample generating unit is used to randomly extract a number of elements from a preset element library as a group of negative samples corresponding to the earliest part of the long sentence training data for which the Nce loss value has not been calculated before executing the loss value calculating unit each time.

7. The RNN-based language model training device according to claim 5, characterized in that: Also includes: A second negative sample generating unit is used to randomly extract elements from a preset element library to generate M groups of negative samples, 2≤M≤N; And the N parts of the long sentence training data are divided into M groups, and each group in the M groups divided by the N parts corresponds one-to-one to each group of negative samples in the M groups of negative samples.

8. The RNN-based language model training device according to claim 5, characterized in that: The RNN-based language model is specifically: LSTM-based language model.

9. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, each step of the RNN-based language model training method as described in any one of claims 1 to 4 is implemented.

10. A training device for a language model based on RNN, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the RNN-based language model training method as described in any one of claims 1 to 4.

Images