A model training method, text generation method, device, medium and equipment

Abstract

The present invention discloses a model training method, a text generation method, a device, a medium and an equipment. The model training method includes: obtaining a sample text, inputting the sample text into a text generation model to be trained, obtaining the correlation coefficient between the words in the sample text output by the encoder, and the self-attention parameter and prediction result output by the decoder; determining the importance index of each word based on the correlation coefficient between the words in the sample text; generating a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text; adjusting the model parameters of the text generation model to be trained based on the loss function until the training process meets the training stop condition, and obtaining the target text generation model. The ability of the text generation model to extract important words from the input text is improved, and the quality of the output text is further improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of deep learning technology, and in particular to a model training method, a text generation method, an apparatus, a medium and a device. Background Art

[0002] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. For automatic abstracting systems, with the rapid growth of information, the significant increase in the length and quantity of chapter-level texts has brought new challenges to the utilization of network information. In order to better utilize effective information, various information extraction technologies have emerged. By automatically extracting effective information from the text, the utilization rate of information can be greatly improved. The purpose of article abstracts is to use computers to extract key information from a large amount of text. This technology is even more important for texts that are large in length and complex in semantics on the Internet.

[0003] The automatic generation of generative summaries is an important open problem in natural language processing. In the process of realizing the present invention, the inventors found that there are at least the following technical problems in the prior art: the existing methods face problems such as information redundancy, difficulty in obtaining significant information, and information duplication to varying degrees. Summary of the invention

[0004] Embodiments of the present invention provide a model training method, a text generation method, an apparatus, a medium and a device to improve the quality of generated text.

[0005] In a first aspect, an embodiment of the present invention provides a model training method, wherein the text generation model includes an encoder and a decoder, and the method includes:

[0006] Obtaining a sample text, inputting the sample text into a text generation model to be trained, obtaining a correlation coefficient between words in the sample text output by the encoder, and a self-attention parameter and a prediction result output by the decoder;

[0007] Determining the importance index of each word based on the correlation coefficient between the words in the sample text;

[0008] Generate a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text;

[0009] Based on the loss function, the model parameters of the text generation model to be trained are adjusted until the training process meets the training stop condition, thereby obtaining a target text generation model.

[0010] In a second aspect, an embodiment of the present invention further provides a text generation method, characterized in that it includes:

[0011] Obtaining an initial text, inputting the initial text into the target text generation model, and obtaining the predicted probability of each word cyclically output by the target text generation model, wherein the target text generation model is trained by the model training method provided in an embodiment of the present invention;

[0012] The target word corresponding to any cycle is determined according to the predicted probability of each word output by any cycle, and a target text is formed based on the target words corresponding to each cycle, wherein the target text is a summary text corresponding to the initial text.

[0013] In a third aspect, an embodiment of the present invention further provides a model training device, wherein the text generation model includes an encoder and a decoder, and the method includes:

[0014] A training data acquisition module, used to acquire sample text, input the sample text into the text generation model to be trained, and acquire the correlation coefficient between the words in the sample text output by the encoder, as well as the self-attention parameter and prediction result output by the decoder;

[0015] An importance index determination module, used to determine the importance index of each word based on the correlation coefficient between the words in the sample text;

[0016] A loss function determination module, used to generate a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text;

[0017] The model parameter adjustment module is used to adjust the model parameters of the text generation model to be trained based on the loss function until the training process meets the training stop condition to obtain the target text generation model.

[0018] In a fourth aspect, an embodiment of the present invention further provides a text generation device, comprising:

[0019] A word prediction module, used to obtain an initial text, input the initial text into the target text generation model, and obtain the prediction probability of each word cyclically output by the target text generation model, wherein the target text generation model is trained by the model training method provided in an embodiment of the present invention;

[0020] The prediction text generation module is used to determine the target word corresponding to any cycle according to the prediction probability of each word output by any cycle, and form a target text based on the target word corresponding to each cycle, wherein the target text is the summary text corresponding to the initial text.

[0021] In a fifth aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model training method and / or text generation method provided in any embodiment of the present invention is implemented.

[0022] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a model training method and / or a text generation method as provided in any embodiment of the present invention.

[0023] The technical solution provided by the embodiment of the present invention determines the importance index of each word in the sample text by a random walk method based on the correlation coefficient between each word in the sample text output by the text generation module, and generates a loss function based on the importance index of each word and the self-attention parameter. The loss function carries the importance information of each word in the standard text. Based on the loss function, the parameters of the text generation model are adjusted to enable the text generation model to learn the attention weight distribution in the sample text, thereby improving the ability of the text generation model to extract important words from the input text and further improving the quality of the output text. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a model training method provided in Embodiment 1 of the present invention;

[0025] Figure 2 It is a structural diagram of a text generation model provided by an embodiment of the present invention;

[0026] Figure 3 It is a flowchart of a text generation method provided in Embodiment 2 of the present invention;

[0027] Figure 4 is a structural schematic diagram of a model training device provided in Embodiment 3 of the present invention;

[0028] Figure 5 is a structural schematic diagram of a text generation device provided by Embodiment 4 of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a model training method provided in the first embodiment of the present invention. This embodiment is applicable to the case of training a text generation model. The method can be executed by the model training device provided in the embodiment of the present invention. The device can be formed by software and / or hardware and can be integrated in an electronic device such as a computer or server. Specifically, it includes the following steps:

[0033] S110, obtaining a sample text, inputting the sample text into a text generation model to be trained, obtaining the correlation coefficient between words in the sample text output by the encoder, and the self-attention parameter and prediction result output by the decoder.

[0034] S120: Determine the importance index of each word based on the correlation coefficient between the words in the sample text.

[0035] S130, generating a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text.

[0036] S140. Adjust model parameters of the text generation model to be trained based on the loss function.

[0037] S150, determining whether the current iteration satisfies the training stop condition, if so, obtaining the target text generation model, if not, returning to step S110 to perform iterative training.

[0038] In this embodiment, the text generation model includes an encoder and a decoder, for example, see Figure 2 , Figure 2 It is a structural diagram of a text generation model provided by an embodiment of the present invention. Figure 2 The encoder is connected to the decoder, the encoder is used to encode the input text to obtain the encoding features, and input the encoding features to the decoder, the decoder decodes the encoding features and outputs the target text corresponding to the input text. In some embodiments, the output text of the text generation model can be a summary text of the input text.

[0039] Optionally, the encoder is a first recurrent network module based on a self-attention mechanism, and the decoder is a second recurrent network module based on a self-attention mechanism. Figure 2It can be seen that the encoder outputs the correlation coefficient and self-attention parameter between each word in the input text, wherein, during the training process, the input text is a sample text, and each word in the sample text can be obtained by segmenting the sample text. Each word in the sample text is input into the encoder in turn, and the encoder is the first recurrent network module, which processes each input word.

[0040] Specifically, the first recurrent network module performs a recurrent process on each input word based on the self-attention mechanism. For any word x, the word x is converted into a word vector X, and the word vector X is vector-transformed based on the preset weight matrix in the self-attention mechanism to obtain a first vector Q, a second vector K, and a third vector V. Exemplarily, they can be calculated based on the following formula:

[0041]

[0042] Among them, W Q is the first weight matrix, W K is the second weight matrix, W V is a third weight matrix, wherein the first weight matrix, the second weight matrix and the third weight matrix are weight matrices to be trained.

[0043] Accordingly, the correlation coefficient between the words in the sample text is calculated based on the first vector Q and the second vector K. Exemplarily, it is calculated based on the following formula:

[0044] in, It can be 8 or other values ​​to ensure the stability of the gradient during training. The correlation coefficient between word i and word j can be marked as D i,j .

[0045] Furthermore, the self-attention parameter α output by the decoder t is the encoder-decoder attention parameter at time t in the input text, which is obtained based on the decoder's output word vector Q', the encoder's input word vector K and the vector V corresponding to time t, where the decoder's output word vector Q' is the output word and the decoder's W Q The calculated value is the input word vector K of the encoder, which is the sum of the input word and the encoder’s W. K Calculated.

[0046] Exemplarily, it can be calculated based on the following formula:

[0047]

[0048] On the basis of the above embodiment, the importance index of each word is determined by the correlation coefficient between the words in the sample text output by the encoder, and the importance of each word is evaluated by the importance index. The larger the importance index, the higher the importance of the corresponding word, which facilitates increasing the attention weight of the high-importance words in the training process, so that the text generation model can learn the high-importance words.

[0049] Optionally, the importance index of each word in the sample text is determined based on the correlation coefficient between the words, including: determining a walk parameter between the words based on the correlation coefficient between the words; and performing a random walk based on the walk parameter between the words to determine the importance index of each word in the sample text.

[0050] In some embodiments, the walk parameter between each word can be obtained by regularizing the correlation coefficient between each word. For any two words, the walk parameter between the first word and the second word is determined based on the correlation coefficient between the first word and the second word, and the ratio of the sum of the correlation coefficients of the first word and each word in the sample text.

[0051] For example, the correlation coefficient between the first word i and the second word j is D i,j The sample text may include M words, that is, M-1≤j≥0. For M words, determine the sum of the correlation coefficients of the M words with the first word i, and the correlation coefficient D between the first word i and the second word j i,j Divide by the sum of the correlation coefficients of the above M words with the first word i, and obtain the walk parameter between the first word i and the second word j. Exemplarily, the walk parameter between the first word i and the second word j can be obtained based on the following formula:

[0052] T i,j =D i,j / ∑ j D i,j , where T i,j is the walking parameter between the first word i and the second word j, M-1≤j≥0, and M is the number of words in the sample text, M is a positive integer. Based on the above method, the walking parameter between each word i and any word j in the sample text is determined, where word i is different from word j.

[0053] Based on the walk parameters between the words, a random walk is performed to determine the importance index of each word in the sample text. Specifically, the initial importance index of each word is set, and the initial importance index can be preset, for example, it can be 1 or 0.5, etc., and there is no limitation on this. The initial importance indexes of different words can be the same or different. Exemplarily, the initial importance index can be a random number in the range of 0-1.

[0054] During the random walk, the importance index of any word is determined based on the walk parameters between the word and each word in the sample text, as well as the importance index of each word. Specifically, for any word i, the sum of the products of the walk parameters between each word j in the sample text and word i and the current importance index of word j is determined. Among them, for any word j, if the word j has not undergone random walk, the current importance index of word j is the initial importance index, and if the word j has undergone random walk, the current importance index of word j is the importance index determined by random walk.

[0055] Exemplarily, the importance index of any word i can be determined based on the following formula:

[0056] score i =∑ j T j,i score j , where score i is the importance index of word i determined by random walk, score i is the current importance index of word j, T j,i is the walking parameter between word j and word i.

[0057] Based on the above method, the importance index of each word based on random walk is determined for each word in the sample text, and a random walk is completed.

[0058] On the basis of the above embodiment, the random walk based on the walk parameters between the words may be performed at least once, and the number of random walks may be preset, such as 1, 2 or 3 times. The number of random walks is positively correlated with the model training time. The more random walks, the longer the training time of the text generation model. For example, see Table 1, which is a comparison of training result parameters corresponding to different random walk times.

[0059] Table 1

[0060] Model ROUGE-1 ROUGE-2 ROUGE-L Increased training time Original Model 38.53％ 19.93％ 35.86％ - Model of the present invention (number of random walks = 1) 38.83％ 20.19％ 36.19％ 1.6％ Model of the present invention (number of random walks = 2) 38.65％ 20.11％ 35.01％ 1.8％ Model of the present invention (number of random walks = 3) 38.63％ 19.83％ 35.86％ 2.0％

[0061] Among them, ROUGE-1, ROUGE-2, and ROUGE-L are evaluation indicators of the text output by the text generation model, and the higher the value of the evaluation indicator, the higher the quality of the text generated by the text generation model. According to Table 1, the random walk coefficient can preferably be 1.

[0062] On the basis of the above embodiment, a loss function is determined based on the importance index of each word and the self-attention parameter, the text generation model is trained by carrying the loss function that carries the importance index of each word and the self-attention parameter, and the attention weight distribution of the text generation model is adjusted so that the text generation model learns the importance distribution of each word in the input text, thereby improving the summary generation quality of the text generation model.

[0063] Optionally, a loss function is generated based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text, including: generating a first loss function based on the importance index of each word and the self-attention parameter; generating a second loss function based on the prediction probability of each word in the prediction result and the standard words in the standard text; and obtaining a target loss function based on the first loss function and the second loss function.

[0064] The first loss function may be a divergence function of the importance index of each word and the self-attention parameter, and may be, for example, Among them, N is the word length in the predicted text output by the text generation model, that is, the number of words in the predicted text, α t is the encoder-decoder attention parameter at time t in the input text, score is the importance index of the input text, and is the mean importance index of each word in the input text, where t is a positive integer.

[0065] The standard text includes N standard words, and the prediction results include multiple candidate words corresponding to each standard word position and the prediction probability of the candidate words. For example, the standard text is "Huawei mobile phone", which includes two words "Huawei" and "mobile phone". The prediction results include multiple candidate words corresponding to the first word position (y t ) words and the predicted probabilities of candidate words, and the predicted probabilities of the candidate words corresponding to the second word position and the predicted probabilities of the candidate words. Exemplarily, the candidate words for the first word position in the prediction result are "Huawei" and "Samsung", and the predicted probability of "Huawei" is 0.7 and the predicted probability of "Samsung" is 0.3. According to the standard words at each word position, the predicted probabilities corresponding to the standard words at each word position in the prediction result are determined, and the second loss function is determined based on the predicted probabilities corresponding to the standard words at each word position. Exemplarily, the predicted probability of the standard word at the first word position is 0.7.

[0066] Exemplarily, the second loss function can be Among them, y t is the tth standard word in the standard text, P(y t ) is the standard word y at the tth word position in the prediction result t The predicted probability of .

[0067] In some embodiments, the first loss function and the second loss function are weighted to obtain a target loss function. For example, the target loss function is: Among them, λ is the weight parameter.

[0068] The above loss function is reversely input into the text generation model, and the text generation model is reversely parameterized, wherein the adjusted parameters may include but are not limited to the first weight matrix, the second weight matrix, and the third weight matrix in the decoder, and the first weight matrix, the second weight matrix, and the third weight matrix in the encoder.

[0069] The training process is iterated, and after each iteration, it is determined whether the training condition is met. If so, the iterative training is stopped, and the target text generation model is determined. If not, the training returns to step S110 to iteratively train the trained text generation model based on the new sample text. The training condition may be that the preset number of iterations reaches the preset number or the training accuracy of the text generation model meets the preset accuracy requirement.

[0070] The technical solution of this embodiment is to determine the importance index of each word in the sample text by random walk based on the correlation coefficient between each word in the sample text output by the text generation module, and generate a loss function based on the importance index of each word and the self-attention parameter. The loss function carries the importance information of each word in the standard text. The parameters of the text generation model are adjusted based on the loss function so that the text generation model learns the attention weight distribution in the sample text, improves the ability of the text generation model to extract important words from the input text, and further improves the quality of the output text. In the above method, there is no need to adjust the structure of the text generation model, and on the basis of improving the output quality of the text generation model, the research and development cost of the text generation model is avoided.

[0071] Embodiment 2

[0072] Figure 3 This is a flowchart of a text generation method provided in the second embodiment of the present invention. This embodiment is applicable to the training of generating target text based on a text generation model. The method can be executed by the text generation device provided in the embodiment of the present invention. The device can be formed by software and / or hardware and can be integrated into an electronic device such as a computer or server. Specifically, it includes the following steps:

[0073] S210, obtaining an initial text, inputting the initial text into the target text generation model, and obtaining the predicted probability of each word cyclically output by the target text generation model.

[0074] S220 , determining a target word corresponding to any cycle according to the prediction probability of each word output by any cycle, and forming a target text based on the target words corresponding to each cycle.

[0075] In this embodiment, the text generation model trained by the above embodiment is used to process the initial text to obtain a summary text of the initial text. The initial text may be, but is not limited to, a full news article, a full paper article, etc.

[0076] The target text generation model includes an encoder and a decoder, the encoder is a first recurrent network module based on the self-attention mechanism, and the decoder is a second recurrent network module based on the self-attention mechanism. The decoder outputs multiple candidate words and the predicted probabilities of the candidate words at the corresponding word position in each cycle, and determines the candidate word with the highest predicted probability as the target word at the word position, and forms the target text based on the target words corresponding to each cycle. Exemplarily, the decoder outputs the candidate words at the first word position and the predicted probabilities of each candidate word in the first cycle, for example, the candidate words are "Huawei" and "Samsung", the predicted probability of "Huawei" is 0.7, and the predicted probability of "Samsung" is 0.3, and the target word at the first word position is determined to be "Huawei". Similarly, the target of the second word position can be determined to be "mobile phone", the target of the third word position is "price", and the target of the fourth word position is "reasonable". Accordingly, according to the order of each word position, each target word is combined to obtain the target text that the price of Huawei mobile phones is reasonable. Optionally, the word length of the target text can be pre-set, and the number of cycles of the decoder can be determined according to the word length. When the number of cycles is met, the cycle ends. Optionally, an end marker may be preset, and if the candidate word with the maximum predicted probability output by the current word loop is the end marker, the loop ends.

[0077] The technical solution provided in this embodiment extracts the summary text of the initial text through the text generation model, extracts the important information in the initial text, and facilitates the user to quickly know the key content in the initial text, speeding up the information acquisition. In this embodiment, the text generation model introduces the importance information of the words in the input text during the training process, so that the model attention weight of the text generation model conforms to the importance information distribution of the input text, and accordingly, ensures the quality of the generated summary text.

[0078] Embodiment 3

[0079] Figure 4 : is a schematic diagram of the structure of a model training device provided in Embodiment 3 of the present invention, the device comprising:

[0080] A training data acquisition module 310 is used to acquire sample text, input the sample text into the text generation model to be trained, and acquire the correlation coefficient between the words in the sample text output by the encoder, and the self-attention parameter and prediction result output by the decoder;

[0081] An importance index determination module 320, configured to determine the importance index of each word based on the correlation coefficient between the words in the sample text;

[0082] A loss function determination module 330, configured to generate a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text;

[0083] The model parameter adjustment module 340 is used to adjust the model parameters of the text generation model to be trained based on the loss function until the training process meets the training stop condition to obtain the target text generation model.

[0084] Based on the above embodiment, the importance index determination module 320 is used to:

[0085] Determine the wandering parameters between the words based on the correlation coefficients between the words;

[0086] A random walk is performed based on the walk parameters between the words to determine the importance index of each word in the sample text.

[0087] Based on the above embodiment, the walk parameter between the first word and the second word is determined based on the correlation coefficient between the first word and the second word, and the ratio of the sum of the correlation coefficients between the first word and each word in the sample text.

[0088] On the basis of the above embodiment, the importance index of any word is determined based on the wandering parameters of the any word and each word in the sample text, and the importance index of each word.

[0089] Based on the above embodiment, the loss function determination module 330 is used to:

[0090] Generate a first loss function based on the importance index of each word and the self-attention parameter;

[0091] generating a second loss function based on the prediction probability of each word in the prediction result and the standard words in the standard text;

[0092] A target loss function is obtained based on the first loss function and the second loss function.

[0093] Based on the above embodiment, the encoder is a first recurrent network module based on a self-attention mechanism, and the decoder is a second recurrent network module based on a self-attention mechanism.

[0094] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Embodiment 4

[0096] Figure 5 : is a structural diagram of a text generation device provided by Embodiment 4 of the present invention, the device comprising:

[0097] The word prediction module 410 is used to obtain an initial text, input the initial text into the target text generation model, and obtain the prediction probability of each word cyclically output by the target text generation model, wherein the target text generation model is trained by the model training method provided in the above embodiment;

[0098] The prediction text generation module 420 is used to determine the target word corresponding to any loop according to the prediction probability of each word output by any loop, and form a target text based on the target word corresponding to each loop, wherein the target text is the summary text corresponding to the initial text.

[0099] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] Embodiment 5

[0101] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 6 A block diagram of an electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The device 12 is typically an electronic device that undertakes the image classification function.

[0102] like Figure 6 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, a storage device 28, and a bus 18 connecting various system components (including the storage device 28 and the processor 16).

[0103] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0104] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0105] The storage device 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The storage device 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0106] A program 36 having a set (at least one) of program modules 26 may be stored, for example, in a storage device 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include an implementation of a network environment. The program modules 26 generally perform the functions and / or methods of the embodiments described herein.

[0107] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, cameras, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0108] The processor 16 executes various functional applications and data processing by running the programs stored in the storage device 28, such as implementing the model training method and / or text generation method provided in the above-mentioned embodiments of the present invention.

[0109] Embodiment 6

[0110] Embodiment 6 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method and / or text generation method provided in the embodiment of the present invention.

[0111] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the method operation described above, but can also execute the model training method and / or text generation method provided by any embodiment of the present invention.

[0112] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0113] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable source code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0114] Source code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0115] Computer source code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The source code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0116] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A model training method, characterized in that: The text generation model includes an encoder and a decoder, and the method includes: Obtaining a sample text, inputting the sample text into a text generation model to be trained, obtaining a correlation coefficient between words in the sample text output by the encoder, and a self-attention parameter and a prediction result output by the decoder; Determining the importance index of each word based on the correlation coefficient between the words in the sample text; Generate a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text; Adjusting model parameters of the text generation model to be trained based on the loss function until the training process meets the training stop condition to obtain a target text generation model; The step of determining the importance index of each word based on the correlation coefficient between the words in the sample text includes: Determine the wandering parameters between the words based on the correlation coefficients between the words; Performing random walks based on the walk parameters between the words to determine the importance index of each word in the sample text; The walk parameter between the first word and the second word is determined based on the correlation coefficient between the first word and the second word, and the ratio of the sum of the correlation coefficients between the first word and each word in the sample text.

2. The method according to claim 1, characterized in that The importance index of any word is determined based on the wandering parameters of the any word and each word in the sample text, and the importance index of each word.

3. The method according to claim 1, characterized in that: The loss function is generated based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text, including: Generate a first loss function based on the importance index of each word and the self-attention parameter; generating a second loss function based on the prediction probability of each word in the prediction result and the standard words in the standard text; A target loss function is obtained based on the first loss function and the second loss function.

4. The method according to claim 1, characterized in that: The encoder is a first recurrent network module based on a self-attention mechanism, and the decoder is a second recurrent network module based on a self-attention mechanism.

5. A text generation method, characterized in that: include: Obtaining an initial text, inputting the initial text into the target text generation model, and obtaining the predicted probability of each word cyclically output by the target text generation model, wherein the target text generation model is trained by the model training method according to any one of claims 1 to 4; The target word corresponding to any cycle is determined according to the predicted probability of each word output by any cycle, and a target text is formed based on the target words corresponding to each cycle, wherein the target text is a summary text corresponding to the initial text.

6. A model training device, characterized in that: The text generation model includes an encoder and a decoder, and the device includes: A training data acquisition module, used to acquire sample text, input the sample text into the text generation model to be trained, and acquire the correlation coefficient between the words in the sample text output by the encoder, as well as the self-attention parameter and prediction result output by the decoder; An importance index determination module, used to determine the importance index of each word based on the correlation coefficient between the words in the sample text; A loss function determination module, used to generate a loss function based on the importance index of each word, the self-attention parameter, the prediction result and the standard text corresponding to the sample text; A model parameter adjustment module, used to adjust the model parameters of the text generation model to be trained based on the loss function until the training process meets the training stop condition to obtain a target text generation model; Wherein, the importance index determination module is specifically used for: Determine the wandering parameters between the words based on the correlation coefficients between the words; Performing random walks based on the walk parameters between the words to determine the importance index of each word in the sample text; The walk parameter between the first word and the second word is determined based on the correlation coefficient between the first word and the second word, and the ratio of the sum of the correlation coefficients between the first word and each word in the sample text.

7. A text generation device, characterized in that: include: A word prediction module, used to obtain an initial text, input the initial text into the target text generation model, and obtain the prediction probability of each word cyclically output by the target text generation model, wherein the target text generation model is trained according to any one of the model training methods described in claims 1-4; The prediction text generation module is used to determine the target word corresponding to any cycle according to the prediction probability of each word output by any cycle, and form a target text based on the target word corresponding to each cycle, wherein the target text is the summary text corresponding to the initial text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the model training method described in any one of claims 1 to 4 or the text generation method described in claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the model training method described in any one of claims 1 to 4 or the text generation method described in claim 5.

Images