Model training method, text abstract generation method, device, equipment and medium
By combining encoders and decoders, the decoding probability of predicted words is calculated and the model is trained when it matches the summary information. This solves the problem that traditional copying mechanisms do not consider the importance of words and improves the prediction accuracy and reliability of generative text summarization models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
- Filing Date
- 2021-10-18
- Publication Date
- 2026-05-22
AI Technical Summary
Traditional copying mechanisms do not consider copying history and the importance of each word in the input text during decoding, resulting in low accuracy and reliability of prediction results from generative text summarization models.
The training text is encoded by an encoder, the decoding hidden state is determined by a decoder, the decoding probability of the predicted words is calculated based on the encoding and decoding states, and the model is trained when the predicted words are contained in the summary information labeled in the training text to maximize the decoding probability.
It improves the prediction performance of generative text summarization models and enhances the accuracy and reliability of summarization generation.
Smart Images
Figure CN115934929B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, a text summarization generation method, an apparatus, a device, and a medium. Background Technology
[0002] Text summarization refers to the technology of using computer equipment to analyze text, summarize its content, and automatically generate summaries to express the main content of the text in a concise form, thereby improving readers' reading speed and quality. Currently, deep learning-based summarization models can be used to extract summaries from text. Training these models is crucial for improving their predictive performance. Summary of the Invention
[0003] This application aims to at least partially address one of the technical problems in the related art.
[0004] This application proposes a model training method, a text summarization generation method, an apparatus, a device, and a medium to train a summarization generation model and improve its prediction performance.
[0005] The first aspect of this application proposes a model training method, including:
[0006] Obtain training text, wherein the training text is annotated with summary information;
[0007] The training text is encoded using an encoder in a summary generation model to obtain the encoded hidden state;
[0008] The second decoding hidden state of the current output is determined by the decoder in the summary generation model based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word.
[0009] The decoding probability of the second predicted word is determined based on the encoded hidden layer state and the second decoded hidden layer state.
[0010] In response to the second predicted word being contained in the summary information, the summary generation model is trained to maximize the decoding probability.
[0011] A second aspect of this application provides a text summarization method, comprising:
[0012] Get the input text;
[0013] A trained summary generation model is used to predict the input text to obtain summary information corresponding to the input text; wherein the summary generation model is trained using the model training method described in the first aspect embodiment of this application.
[0014] A third aspect of this application provides a model training apparatus, comprising:
[0015] The acquisition module is used to acquire training text, wherein the training text is annotated with summary information;
[0016] The encoding module is used to encode the training text using the encoder in the summary generation model to obtain the encoded hidden layer state;
[0017] The first determining module is used to determine the second decoding hidden state of the current output based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word, using the decoder in the summary generation model.
[0018] The second determining module is used to determine the decoding probability of the second predicted word based on the encoded hidden layer state and the second decoded hidden layer state.
[0019] A training module is used to train the summary generation model in response to the second predicted vocabulary being contained in the summary information, so as to maximize the decoding probability.
[0020] A fourth aspect of this application provides a text summarization apparatus, comprising:
[0021] The acquisition module is used to acquire the input text;
[0022] The prediction module is used to predict the input text using a trained summary generation model to obtain summary information corresponding to the input text; wherein the summary generation model is trained using the model training device as described in the third aspect embodiment of this application.
[0023] A fifth aspect of this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a model training method as proposed in the first aspect of this application, or a text summarization generation method as proposed in the second aspect of this application.
[0024] A sixth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method proposed in the first aspect of this application, or the text summarization generation method proposed in the second aspect of this application.
[0025] A seventh aspect of this application provides a computer program product that, when executed by a processor, performs a model training method as proposed in a first aspect of this application, or a text summarization method as proposed in a second aspect of this application.
[0026] One embodiment of this application described above has at least the following advantages or beneficial effects:
[0027] The training text is encoded using an encoder in the summarization model to obtain the encoded hidden state. Based on the most recently outputted first decoded hidden state and its corresponding first predicted word, or based on a pre-defined initial first decoded hidden state and its corresponding first predicted word, the decoder in the summarization model determines the current output's second decoded hidden state. The decoding probability of the second predicted word is determined based on the encoded and second decoded hidden states. In response to the second predicted word being included in the summary information labeled in the training text, the summarization model is trained to maximize the decoding probability. Therefore, training the summarization model based on the decoding probability of each predicted word and whether each predicted word is included in the summary information labeled in the training text can improve the model's prediction performance.
[0028] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0030] Figure 1 This is a flowchart illustrating the model training method provided in Embodiment 1 of this application;
[0031] Figure 2 This is a schematic flowchart of the model training method provided in Embodiment 2 of this application;
[0032] Figure 3 This is a flowchart illustrating the model training method provided in Embodiment 3 of this application;
[0033] Figure 4 This is a flowchart illustrating the text digest generation method provided in Embodiment 4 of this application;
[0034] Figure 5 This is a schematic diagram of the model training device provided in Embodiment 5 of this application;
[0035] Figure 6 This is a schematic diagram of the text summarization device provided in Embodiment Six of this application;
[0036] Figure 7 A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Detailed Implementation
[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0038] Currently, text summarization techniques can be used to extract summaries from text. Specifically, for long input text, a decoder can be used to decode it, generating an encoded hidden sequence. The decoder then generates a summary of the long input text based on a replication mechanism.
[0039] In recent years, the Transformer-based encoder-decoder framework has become the standard model for generative text summarization tasks, and the copying mechanism is also a standard module for these tasks. The copying mechanism refers to directly copying certain words from the input text and placing those copied words into the output summary.
[0040] However, traditional copying mechanisms do not consider the copying history and the importance of each word in the input text during decoding. The copying decisions between each decoding moment are independent. Obviously, this copying method does not conform to the human habit of writing summaries. Therefore, it may lead to low accuracy and reliability of the model's prediction results.
[0041] Therefore, in order to address the above problems, this application proposes a model training method, apparatus, device, and storage medium.
[0042] The model training method, text summarization method, apparatus, device, and medium of embodiments of this application are described below with reference to the accompanying drawings.
[0043] Figure 1 This is a flowchart illustrating the model training method provided in Embodiment 1 of this application.
[0044] This application example illustrates the model training method configured in a model training device, which can be applied to any electronic device to enable the electronic device to perform model training functions.
[0045] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.
[0046] like Figure 1 As shown, the model training method may include the following steps:
[0047] Step 101: Obtain the training text, which is annotated with summary information.
[0048] In one possible implementation of this application embodiment, the training text can be obtained from an existing dataset, test set, or training set.
[0049] In another possible implementation of this application embodiment, training text can be collected online. For example, some news articles and other texts can be collected online using web crawler technology as training text.
[0050] In another possible implementation of this application, training text can be collected offline, for example, by taking an image containing text information by a user holding an image acquisition device, and then recognizing the text information in the image based on OCR (Optical Character Recognition) technology to obtain training text.
[0051] In another possible implementation of this application's embodiments, the training text can also be obtained through manual input by the user. Input methods include, but are not limited to, touch input (such as swiping, clicking, etc.), keyboard input, and voice input.
[0052] Optionally, to improve the model's prediction performance, the summary information in the training text can be manually annotated.
[0053] Step 102: The training text is encoded using the encoder in the summary generation model to obtain the encoded hidden state.
[0054] In the embodiments of this application, the structure of the summary generation model is not limited. For example, the summary generation model can be a model with Transformer as the basic structure, or it can be a model with other structures, such as a model with a variant structure of Transformer.
[0055] In the embodiments of this application, the summary generation model may include an encoder and a decoder. The encoder in the summary generation model can be used to encode the training text to obtain the encoded hidden layer state.
[0056] One possible approach is to segment the training text to obtain individual words, and then use an encoder to encode each word to obtain the corresponding hidden state of the encoding layer.
[0057] As an example, let the training text be labeled X, and the i-th word segmented in the training text be x. i Then encoder f enc Regarding x i Encoding is performed, and the generated hidden state can be:
[0058] h i =f enc (x i (1)
[0059] Among them, h i The word x represents the participle. i The corresponding encoded hidden state.
[0060] Step 103: Determine the second decoding hidden state of the current output using the decoder in the summary generation model, based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word.
[0061] In the embodiments of this application, when the current output is not the first output, the first predicted word refers to the predicted word of the most recent (or previous) output, while when the current output is the first output, the first predicted word is the word set initially.
[0062] In this embodiment of the application, when the current output is not the first output, the second decoding hidden state of the current output can be determined by the decoder in the summary generation model based on the first decoding hidden state of the most recent output and the corresponding first predicted vocabulary.
[0063] As an example, the first decoded hidden state of the most recent output is labeled as s. t-1 The most recent output's first predicted word is y. t-1 The second decoded hidden state output this time is s t Then decoder f dec The generated second decoding hidden state s t It can be:
[0064] s t =f dec (st-1 ,y t-1 (2)
[0065] In this embodiment of the application, when this output is the first output, the second decoding hidden state of this output can be determined by the decoder in the summary generation model based on the initial first decoding hidden state and the corresponding first predicted vocabulary.
[0066] As an example, if the initial first decoded hidden state is set to s0 and the initial first predicted word is set to y0, then the decoder f dec The generated second decoding hidden state s t It can be:
[0067] s t =f dec (s0,y0); (3)
[0068] Step 104: Determine the decoding probability of the second predicted word based on the encoding hidden state and the second decoding hidden state.
[0069] In this embodiment of the application, the second predicted word refers to the predicted word output in this instance. For example, if the first predicted word is "Tsinghua University", the second predicted word could be "university".
[0070] In this embodiment of the application, the decoding probability of the second predicted word can be determined based on the encoding hidden layer state and the second decoding hidden layer state. The smaller the decoding probability, the lower the importance of the second predicted word. Conversely, the larger the decoding probability, the more important the second predicted word is and the more likely it is to be a summary word.
[0071] Step 105: In response to the second predicted vocabulary being contained in the summary information, the summary generation model is trained to maximize the decoding probability.
[0072] In the embodiments of this application, when the second predicted word is included in the summary information, that is, the second predicted word is a summary word, the summary generation model can be trained to maximize the decoding probability.
[0073] The model training method of this application embodiment encodes the training text using an encoder in a summarization generation model to obtain an encoded hidden state; based on the most recently output first decoded hidden state and the corresponding first predicted word, or based on a set initial first decoded hidden state and the corresponding first predicted word, a decoder in the summarization generation model determines the currently output second decoded hidden state; based on the encoded hidden state and the second decoded hidden state, the decoding probability of the second predicted word is determined; in response to the second predicted word being included in the summary information labeled in the training text, the summarization generation model is trained to maximize the decoding probability. Therefore, by training the summarization generation model based on the decoding probability of each predicted word and whether each predicted word is included in the summary information labeled in the training text, the prediction performance of the summarization generation model can be improved.
[0074] To clearly illustrate how the decoding probability of the second predicted word is determined in this application, this embodiment provides another model training method.
[0075] Figure 2 This is a schematic flowchart of the model training method provided in Embodiment 2 of this application.
[0076] like Figure 2 As shown, the model training method may include the following steps:
[0077] Step 201: Obtain the training text, which is annotated with summary information.
[0078] Step 202: The training text is encoded using the encoder in the summary generation model to obtain the encoded hidden state.
[0079] Step 203: Determine the second decoding hidden state of the current output using the decoder in the summary generation model, based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word.
[0080] The execution process of steps 201 to 203 can be referred to the execution process of steps 101 to 103 in the above embodiment, and will not be repeated here.
[0081] Step 204: Determine the generation probability of the second predicted word based on the second decoding hidden layer state.
[0082] Optionally, the second predicted word is labeled as w, and the generation probability of the second predicted word w is labeled as P. vocab (w), then we can obtain:
[0083] P vocab (w) = softmax(W) b st (4)
[0084] Among them, W b These are the model parameters.
[0085] Step 205: Obtain the initial copy probability of the second predicted word based on the encoding hidden state of each word segment in the training text.
[0086] In this embodiment, the encoding hidden state h of each word segmentation word in the training text can be used as a basis. i Obtain the initial replication probability of the second predicted word w.
[0087] As one possible implementation, the encoding hidden state h corresponding to each segmented word can be used. i Second decoding hidden state s t The attention weight matrix between the encoder and decoder is determined, and the initial copy probability of the second predicted word is determined based on the attention weight matrix.
[0088] As an example, the encoding hidden state h corresponding to each segmented word can be used as a basis. i Second decoding hidden state s t Determine the QKV values in the attention mechanism between the encoder and decoder, where:
[0089] Q t K i V i =s t W Q h i W k h i W v (5)
[0090] Among them, W Q W k W v All of these are model parameters.
[0091] Optionally, the attention weight matrix between the tag encoder and decoder is α. t Then we can get:
[0092] α t =softmax(e t (6)
[0093] Among them, e t The elements in can be obtained using the following formula:
[0094]
[0095] Where, d kThis refers to the dimension of the hidden layer vector.
[0096] Optionally, the initial replication probability of the second predicted word w is . Then we can get:
[0097]
[0098] Where, α t,i This refers to the attention weight matrix α t The attention weight of the second predicted word w to the segmented word i.
[0099] Step 206: Determine the target replication probability corresponding to the second predicted word based on the initial replication probability and the intermediate replication probability; wherein, the intermediate replication probability is determined based on the target replication probability corresponding to the most recently output first predicted word.
[0100] In this embodiment of the application, the target replication probability of the second predicted word can be determined based on the initial replication probability and the intermediate replication probability of the second predicted word.
[0101] As an example, the probability of intermediate replication is marked as The target replication probability corresponding to the second predicted word is Then we can get:
[0102]
[0103] Where, p t It can be obtained through the following formula:
[0104] p t =σ(W g ∑ i (α t,i *V i (10)
[0105] Among them, W g Let σ be the model parameter matrix, and σ be the activation function (i.e., the sigmoid function).
[0106] Step 207: Determine the decoding probability corresponding to the second predicted word based on the generation probability and target replication probability corresponding to the second predicted word.
[0107] In this embodiment of the application, the generation probability P corresponding to the second predicted word can be used as a basis. vocab and target replication probability Determine the decoding probability corresponding to the second predicted word.
[0108] As an example, if we label the decoding probability corresponding to the second predicted word w as P(w), then we can obtain:
[0109]
[0110] Where, p gen It can be obtained through the following formula:
[0111] p gen =σ(W λ ∑ i (α t,i *V i (12)
[0112] Among them, W λ This is the model parameter matrix.
[0113] Step 208: In response to the second predicted vocabulary being contained in the summary information, the summary generation model is trained to maximize the decoding probability.
[0114] The execution process of step 208 can be referred to the execution process of step 105 in the above embodiments, and will not be repeated here.
[0115] The model training method of this application embodiment determines the generation probability of the second predicted word based on the second decoding hidden layer state; obtains the initial copy probability of the second predicted word based on the encoding hidden layer state of each word segment in the training text; determines the target copy probability corresponding to the second predicted word based on the initial copy probability and the intermediate copy probability; wherein the intermediate copy probability is determined based on the target copy probability corresponding to the most recently output first predicted word; and determines the decoding probability corresponding to the second predicted word based on the generation probability and the target copy probability corresponding to the second predicted word. Therefore, by determining the target copy probability corresponding to the currently output second predicted word based on the target copy probability corresponding to the most recently (i.e., the previous) output first predicted word, the copy decision between each decoding time point can be correlated, thereby optimizing the copy mechanism and improving the training effect of the model.
[0116] To clearly illustrate how the intermediate replication probability is determined in this application, this application provides an alternative model training method.
[0117] Figure 3 This is a flowchart illustrating the model training method provided in Embodiment 3 of this application.
[0118] like Figure 3 As shown, the model training method may include the following steps:
[0119] Step 301: Obtain the training text, which is annotated with summary information.
[0120] Step 302: The encoder in the summary generation model is used to encode the training text to obtain the encoded hidden state.
[0121] Step 303: Determine the second decoding hidden state of the current output using the decoder in the summary generation model, based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word.
[0122] Step 304: Determine the generation probability of the second predicted word based on the second decoding hidden layer state.
[0123] Step 305: Obtain the initial copy probability of the second predicted word based on the encoding hidden state of each word segment in the training text.
[0124] The execution process of steps 301 to 305 can refer to the execution process of any of the above embodiments, and will not be repeated here.
[0125] Step 306: Obtain the target replication probability corresponding to the first predicted word in the most recent output.
[0126] In this embodiment of the application, when the current output is not the first output, the target replication probability corresponding to the first predicted word of the most recent output can be obtained, while when the current output is the first output, the target replication probability corresponding to the first predicted word set initially can be obtained.
[0127] Optionally, the target replication probability corresponding to the first predicted word is labeled as
[0128] Step 307: Determine the random walk probability between the second predicted word and each word segmentation word.
[0129] It should be noted that random walks can help summary generation models find important words in text, that is, important words that are closely connected with other words in the text. Statistically, random walks are more likely to be "visited" and are more likely to be output in the summary, which is consistent with human cognitive common sense.
[0130] Therefore, in this embodiment of the application, in order to improve the training results of the model, the random walk probability between the second predicted word and each segmented word can be determined.
[0131] As one possible implementation, for each segmented word in the training text, the random walk probability between the second predicted word and the segmented word can be determined based on the encoding hidden state corresponding to the segmented word and the encoding hidden state corresponding to the second predicted word.
[0132] As an example, suppose the second predicted word w is the i-th segmented word x in the training text. i Then the second predicted word x iand the j-th segmented word x in the training text j The probability of a random walk between them can be:
[0133] walk(x j ,x i ) = softmax(e j,i (13)
[0134] Among them, e j,i It can be determined based on the encoder's attention weights, for example, e j,i It can be obtained through the following formula:
[0135]
[0136] Among them, Q t1 K i1 =h j W Q1 h i W k1 W Q1 W k1 These are the model parameters.
[0137] In other words, in this application, since the encoder layer forms a fully connected graph, this fully connected graph can be used as an adjacency matrix, which, after regularization, can be used as the random walk probability.
[0138] Step 308: Determine the intermediate replication probability based on the target replication probability corresponding to the first predicted word and the random walk probability mentioned above.
[0139] In this embodiment, the target replication probability corresponding to the most recently output first predicted word can be used. And the random walk probability between the second predicted word and each segmented word, to determine the intermediate replication probability.
[0140] As an example, the intermediate replication probability can be determined using the following formula.
[0141] Where X refers to the training text, and the first summation symbol on the right of "=" means that there may be multiple second predicted words in the training text. For example, if the second predicted word is "university", there may be multiple word segments of "university" in the training text. Therefore, for each second predicted word in the training text, the value corresponding to the second summation symbol on the right of "=" in formula (15) can be calculated, and then the values corresponding to all second predicted words can be added together to obtain the intermediate copy probability.
[0142] Step 309: Determine the target replication probability corresponding to the second predicted word based on the initial replication probability and the intermediate replication probability.
[0143] Step 310: Determine the decoding probability corresponding to the second predicted word based on the generation probability and the target replication probability corresponding to the second predicted word.
[0144] Step 311: In response to the second predicted vocabulary being contained in the summary information, the summary generation model is trained to maximize the decoding probability.
[0145] The execution process of steps 309 to 311 can refer to the execution process of any of the above embodiments, and will not be repeated here.
[0146] The model training method of this application, since the random walk probability represents the correlation between the second predicted word and each word segmentation word in the training text, determines the target copy probability corresponding to the second predicted word output this time based on the random walk probability and the target copy probability corresponding to the most recently output first predicted word. This not only enables the correlation of copy decisions between each decoding time, but also enables the determination of the target copy probability corresponding to the second predicted word output this time based on the correlation or importance between each word. This can further optimize the copy mechanism and thus improve the training effect of the model.
[0147] The above describes the training method for the summary generation model. This application also proposes an application method for the summary generation model.
[0148] Figure 4 This is a flowchart illustrating the text digest generation method provided in Embodiment 4 of this application.
[0149] like Figure 4 As shown, the text summarization method may include the following steps:
[0150] Step 401: Obtain the input text.
[0151] In one possible implementation of this application embodiment, the input text can be obtained from an existing dataset or test set.
[0152] In another possible implementation of this application embodiment, the input text can be collected online. For example, web crawler technology can be used to collect some news, articles and other texts online as input text.
[0153] In another possible implementation of this application, training text can be collected offline, for example, by taking an image containing text information by a user holding an image acquisition device, and then recognizing the text information in the image based on OCR technology to obtain the input text.
[0154] In another possible implementation of this application embodiment, the input text can also be obtained by the user manually inputting it.
[0155] Step 402: The trained summary generation model is used to predict the input text to obtain the summary information corresponding to the input text; wherein, the summary generation model is trained using the model training method proposed in any of the above embodiments.
[0156] In this embodiment of the application, a summary generation model trained using the model training method proposed in any of the above embodiments can be obtained. The trained summary generation model is then used to predict the input text in order to obtain the summary information corresponding to the input text.
[0157] The text summarization method of this application employs a trained summarization model to predict the input text and obtain the corresponding summary information. The summarization model is trained using the model training method proposed in any of the above embodiments. Therefore, the deep learning-based summarization model improves the accuracy and reliability of summarization extraction.
[0158] With the above Figures 1 to 3 Corresponding to the model training method provided in the embodiments, this application also provides a model training apparatus. Since the model training apparatus provided in the embodiments of this application is different from the one described above... Figures 1 to 3 The model training method provided in the embodiments corresponds to the model training device provided in the embodiments of this application, and will not be described in detail in the embodiments of this application.
[0159] Figure 5 This is a schematic diagram of the model training device provided in Embodiment 5 of this application.
[0160] like Figure 5 As shown, the model training device 500 may include: an acquisition module 510, an encoding module 520, a first determination module 530, a second determination module 540, and a training module 550.
[0161] The acquisition module 510 is used to acquire training text, which is annotated with summary information.
[0162] The encoding module 520 is used to encode the training text using the encoder in the summary generation model to obtain the encoded hidden state.
[0163] The first determining module 530 is used to determine the second decoding hidden state of the current output based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word, using the decoder in the summary generation model.
[0164] The second determining module 540 is used to determine the decoding probability of the second predicted word based on the encoding hidden layer state and the second decoding hidden layer state.
[0165] Training module 550 is used to train the summary generation model in response to the second predicted vocabulary being contained in the summary information, so as to maximize the decoding probability.
[0166] In one possible implementation of this application embodiment, the second determining module 540 may include:
[0167] The first determining unit is used to determine the generation probability of the second predicted word based on the second decoding hidden layer state.
[0168] The first acquisition unit is used to obtain the initial copy probability of the second predicted word based on the encoding hidden state of each word segment in the training text.
[0169] The second determining unit is used to determine the target replication probability corresponding to the second predicted word based on the initial replication probability and the intermediate replication probability; wherein the intermediate replication probability is determined based on the target replication probability corresponding to the most recently output first predicted word.
[0170] The third determining unit is used to determine the decoding probability corresponding to the second predicted word based on the generation probability and the target replication probability corresponding to the second predicted word.
[0171] In one possible implementation of this application embodiment, the first acquisition unit is specifically used to: determine the attention weight matrix between the encoder and decoder based on the encoding hidden state and the second decoding hidden state corresponding to each segmented word; and determine the initial copy probability of the second predicted word based on the attention weight matrix.
[0172] In one possible implementation of this application embodiment, the second determining module 540 may further include:
[0173] The second acquisition unit is used to acquire the target replication probability corresponding to the first predicted word in the most recent output.
[0174] The fourth determining unit is used to determine the random walk probability between the second predicted word and each word segmentation word.
[0175] The fifth determining unit is used to determine the intermediate replication probability based on the target replication probability and random walk probability corresponding to the first predicted word.
[0176] In one possible implementation of this application embodiment, the fourth determining unit is specifically used to: for each segmented word, determine the random walk probability between the second predicted word and the segmented word based on the encoding hidden state corresponding to the segmented word and the encoding hidden state corresponding to the second predicted word.
[0177] In one possible implementation of this application embodiment, the encoding module 520 is specifically used to: perform word segmentation on the training text to obtain each word segmentation vocabulary; and use an encoder to encode each word segmentation vocabulary to obtain the encoding hidden state corresponding to each word segmentation vocabulary.
[0178] The model training apparatus of this application embodiment encodes the training text using an encoder in a summarization generation model to obtain an encoded hidden state; based on the most recently output first decoded hidden state and the corresponding first predicted word, or based on a set initial first decoded hidden state and the corresponding first predicted word, a decoder in the summarization generation model determines the currently output second decoded hidden state; based on the encoded hidden state and the second decoded hidden state, the decoding probability of the second predicted word is determined; in response to the second predicted word being included in the summary information annotated in the training text, the summarization generation model is trained to maximize the decoding probability. Therefore, by training the summarization generation model based on the decoding probability of each predicted word and whether each predicted word is included in the summary information annotated in the training text, the prediction performance of the summarization generation model can be improved.
[0179] With the above Figure 4 Corresponding to the text summarization method provided in the embodiments, this application also provides a text summarization apparatus. Since the text summarization apparatus provided in the embodiments of this application is similar to the one described above... Figure 4 The text summarization method provided in the embodiments corresponds to the text summarization method provided in the embodiments of this application. Therefore, the implementation of the text summarization method is also applicable to the text summarization apparatus provided in the embodiments of this application, and will not be described in detail in the embodiments of this application.
[0180] Figure 6 This is a schematic diagram of the text summarization device provided in Embodiment Six of this application.
[0181] like Figure 6 As shown, the text summarization generation device 600 may include an acquisition module 610 and a prediction module 620.
[0182] The acquisition module 610 is used to acquire the input text.
[0183] The prediction module 620 is used to predict the input text using a trained summary generation model to obtain the summary information corresponding to the input text; wherein the summary generation model is trained using the model training device proposed in the aforementioned embodiments.
[0184] The text summarization generation apparatus of this application employs a trained summarization generation model to predict the input text and obtain the corresponding summary information. The summarization generation model is trained using the model training method proposed in any of the above embodiments. Therefore, the deep learning-based summarization generation model can improve the accuracy and reliability of summarization extraction.
[0185] To implement the above embodiments, this application also proposes a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the model training method proposed in any of the foregoing embodiments of this application, or the text summarization generation method proposed in the foregoing embodiments of this application.
[0186] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the model training method proposed in any of the foregoing embodiments of this application, or implements the text summarization generation method proposed in the foregoing embodiments of this application.
[0187] To implement the above embodiments, this application also proposes a computer program product, which, when the instructions in the computer program product are executed by a processor, executes the model training method proposed in any of the foregoing embodiments of this application, or executes the text summarization generation method proposed in the foregoing embodiments of this application.
[0188] Figure 7 A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Figure 7 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0189] like Figure 7 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).
[0190] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.
[0191] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0192] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.
[0193] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.
[0194] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0195] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.
[0196] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0197] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0198] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0199] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0200] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0201] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0203] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A model training method, characterized in that, The method includes the following steps: Obtain training text, wherein the training text is annotated with summary information; The training text is encoded using an encoder in a summary generation model to obtain the encoded hidden state; The second decoding hidden state of the current output is determined by the decoder in the summary generation model based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word. Based on the second decoding hidden layer state, determine the generation probability of the second predicted word; based on the encoding hidden layer state of each word segmentation word in the training text, obtain the initial copy probability of the second predicted word; Based on the initial replication probability and the intermediate replication probability, the target replication probability corresponding to the second predicted word is determined; wherein, the intermediate replication probability is determined based on the target replication probability corresponding to the most recently output first predicted word; The decoding probability of the second predicted word is determined based on the generation probability and the target replication probability of the second predicted word. In response to the second predicted word being contained in the summary information, the summary generation model is trained to maximize the decoding probability.
2. The method according to claim 1, characterized in that, The step of obtaining the initial copy probability of the second predicted word based on the encoding hidden state of each segmented word in the training text includes: Based on the encoding hidden state and the second decoding hidden state corresponding to each of the segmented words, the attention weight matrix between the encoder and the decoder is determined; The initial copy probability of the second predicted word is determined based on the attention weight matrix.
3. The method according to claim 1, characterized in that, Before determining the target replication probability corresponding to the first predicted word based on the initial replication probability and the intermediate replication probability, the method further includes: Obtain the target replication probability corresponding to the first predicted word in the most recent output; Determine the random walk probability between the second predicted word and each of the segmented words; The intermediate replication probability is determined based on the target replication probability corresponding to the first predicted word and the random walk probability.
4. The method according to claim 3, characterized in that, Determining the random walk probability between the second predicted word and each of the segmented words includes: For each segmented word, the random walk probability between the second predicted word and the segmented word is determined based on the encoding hidden state corresponding to the segmented word and the encoding hidden state corresponding to the second predicted word.
5. The method according to any one of claims 1-4, characterized in that, The step of encoding the training text using the encoder in the summary generation model to obtain the encoded hidden state includes: The training text is segmented into words to obtain the segmented vocabulary; The encoder is used to encode each of the segmented words to obtain the encoding hidden state corresponding to each of the segmented words.
6. A text summarization method, characterized in that, The method includes: Get the input text; A trained summary generation model is used to predict the input text to obtain summary information corresponding to the input text; wherein the summary generation model is trained using the model training method described in any one of claims 1-5.
7. A model training device, characterized in that, include: The acquisition module is used to acquire training text, wherein the training text is annotated with summary information; The encoding module is used to encode the training text using the encoder in the summary generation model to obtain the encoded hidden layer state; The first determining module is used to determine the second decoding hidden state of the current output based on the most recently output first decoding hidden state and the corresponding first predicted word, or based on the set initial first decoding hidden state and the corresponding first predicted word, using the decoder in the summary generation model. The second determining module is used to determine the decoding probability of the second predicted word based on the encoded hidden layer state and the second decoded hidden layer state. The training module is configured to train the summary generation model in response to the second predicted vocabulary being contained in the summary information, so as to maximize the decoding probability; The second determining module includes: The first determining unit is used to determine the generation probability of the second predicted word based on the second decoded hidden layer state; The first acquisition unit is used to acquire the initial copy probability of the second predicted word based on the encoding hidden layer state of each word segmentation word in the training text. The second determining unit is used to determine the target replication probability corresponding to the second predicted word based on the initial replication probability and the intermediate replication probability; wherein the intermediate replication probability is determined based on the target replication probability corresponding to the most recently output first predicted word; The third determining unit is used to determine the decoding probability corresponding to the second predicted word based on the generation probability and the target replication probability corresponding to the second predicted word.
8. The apparatus according to claim 7, characterized in that, The first acquisition unit is specifically used for: Based on the encoding hidden state and the second decoding hidden state corresponding to each of the segmented words, the attention weight matrix between the encoder and the decoder is determined; The initial copy probability of the second predicted word is determined based on the attention weight matrix.
9. The apparatus according to claim 7, characterized in that, The second determining module further includes: The second acquisition unit is used to acquire the target replication probability corresponding to the most recently output first predicted word; The fourth determining unit is used to determine the random walk probability between the second predicted word and each of the segmented words; The fifth determining unit is used to determine the intermediate replication probability based on the target replication probability corresponding to the first predicted word and the random walk probability.
10. The apparatus according to claim 9, characterized in that, The fourth determining unit is specifically used for: For each segmented word, the random walk probability between the second predicted word and the segmented word is determined based on the encoding hidden state corresponding to the segmented word and the encoding hidden state corresponding to the second predicted word.
11. The apparatus according to any one of claims 7-10, characterized in that, The encoding module is specifically used for: The training text is segmented into words to obtain the segmented vocabulary; The encoder is used to encode each of the segmented words to obtain the encoding hidden state corresponding to each of the segmented words.
12. A text summarization generation apparatus, characterized in that, include: The acquisition module is used to acquire the input text; The prediction module is used to predict the input text using a trained summary generation model to obtain summary information corresponding to the input text; wherein the summary generation model is trained using the model training device as described in any one of claims 7-11.
13. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the model training method as described in any one of claims 1-5, or the text summarization generation method as described in claim 6.
14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the model training method as described in any one of claims 1-5, or the text summarization generation method as described in claim 6.
15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor, the model training method as described in any one of claims 1-5 is executed, or the text summarization generation method as described in claim 6 is executed.