An abstract generation method and device, and a storage medium

Abstract

The application discloses a summary generation method and device and a storage medium, and belongs to the technical field of natural language processing. The summary generation method comprises the following steps: S1, a decoder gradually obtains the output probability of the related words of a corresponding role in an output word table according to a constructed word table in a training database; S2, the k words with the maximum output probability are extracted and spliced to a decoded word sequence in a model to serve as a candidate summary; and S3, the model sorts a plurality of candidate summaries according to the size of the output probability, and ensures that the size of a beam is k; after the model prediction is completed, the candidate sequence with the maximum output probability is taken as a summary. The application further comprises a summary generation device and a storage medium. The storage medium has a computer program stored thereon, and the computer program is executed by a processor to realize the steps of the summary generation method. The generation method helps the model to generate a better role-oriented summary, and the generated summary content is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and specifically to a method, apparatus, and storage medium for generating summaries. Background Technology

[0002] Text summarization is the task of summarizing the content of a long text into a short text. This technique allows people to quickly grasp the key information in a text. In recent years, with the continuous advancement of communication methods, more and more text information appears in the form of dialogues. Dialogue texts are characterized by long turns, semantic incoherence, and colloquial expression. Compared to directly reading the original dialogue, reading a summary of the dialogue can significantly improve reading efficiency; therefore, dialogue summarization technology has gradually gained attention.

[0003] Compared to regular text, a key characteristic of dialogues is that they consist of multiple speakers, each playing a specific role and holding their own viewpoint. Therefore, in addition to creating a general summary of the dialogue's content, we can also extract role-specific summaries—role-oriented dialogue summaries. In customer service, role-based summaries have significant practical value. User-oriented summaries primarily include user questions and difficulties encountered, reflecting which issues are frequently raised, facilitating statistical analysis by relevant platforms. Customer service-oriented summaries mainly include the customer service problem-solving process, helping the platform automatically evaluate service quality.

[0004] Most existing role-oriented dialogue summarization methods treat different roles separately and summarize the main content of each role. How to use the information of other roles to generate role-oriented dialogue summaries is a problem that existing technologies need to solve. Summary of the Invention

[0005] Because there is interaction between different characters in a dialogue, the content of one character's speech may provide necessary assistance to another character's summary. Therefore, this invention proposes a summary generation method, apparatus, and storage medium that includes different characters, extracting key information of other characters from both dialogue statements and summary aspects, to help the model generate more accurate summary content tailored to that character.

[0006] The present invention provides a method for generating an abstract, comprising the following steps:

[0007] S1. The decoder gradually obtains the output probability of the corresponding character in the output vocabulary based on the vocabulary constructed in the training database.

[0008] S2. Extract the k words with the highest output probabilities and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1;

[0009] S3. The model sorts multiple candidate summaries according to their output probabilities and ensures that the bundle size is k. After the model prediction is completed, the candidate sequence with the highest output probability is taken as the summary.

[0010] Furthermore, prior to step S1, the process includes establishing and optimizing the summary generation model, specifically comprising the following steps:

[0011] T1. Use different encoders to encode the dialogue content according to different roles;

[0012] T2. The decoder obtains the statement representation of the corresponding role. When decoding, the decoder uses different attention modules to focus on the statement representations of other roles.

[0013] T3. Calculate the KL divergence across the attention distribution for the statement representations of the same character that different decoders pay attention to, and obtain the loss function across attention interactions.

[0014] T4. In the self-attention module of the decoder, the hidden layer representation of each decoder pays attention to the hidden layer representation of other decoders, forming a role self-attention interaction.

[0015] T5. The decoder predicts the output probability of the word at each position, and obtains the loss function of the summary based on the output probability and the maximum likelihood estimation.

[0016] T6. Combine the loss function across attention interactions and the loss function of the summary to train and optimize the model using gradient descent.

[0017] Furthermore, in step T1, the step of encoding the dialogue content according to different roles using different encoders includes:

[0018] Step T11: Concatenate the speaker role information and dialogue content of the dialogue statements in turn order, and obtain the word embedding representation through the word embedding layer;

[0019] Step T12: Encode the word embedding representation using an encoder to obtain the representation of each word in the dialogue.

[0020] Further, in step T2, the decoder obtains the statement representation of the corresponding role. During decoding, the decoder uses different attention modules to focus on the statement representations of other roles, including:

[0021] Step T21: Based on the speaker role of each sentence in the dialogue, represent the dialogue according to different roles;

[0022] Step T22: The decoder focuses attention on the statements of all characters to obtain the distribution representation of attention and the context representation of the encoder.

[0023] Further, in step T4, the process of each decoder's hidden layer representation paying attention to the hidden layer representations of other decoders to form a role self-attention interaction in the decoder's self-attention module includes:

[0024] Step T41: Each decoder obtains the hidden state of the decoder at the current moment based on the decoding state of the previous moment and the current input information;

[0025] Step T42: Focus the current hidden state of the decoder on the hidden states of all previous times of other decoders to obtain the contextual representation of the role summary.

[0026] Further, in step T5, the decoder predicts the output probability of the word at each position, including:

[0027] The decoder predicts the output probability of words at each position in the summary based on its own hidden state information, the encoder context representation, and the context representation of the role summary.

[0028] Further, in step T6, the step of training and optimizing the model using gradient descent by combining the loss function across attention interactions and the loss function of the summary includes:

[0029] T61. Weighted fusion of the loss function across attention interactions and the loss function of summarization;

[0030] T62. Use the gradient descent algorithm to train and optimize the model until the loss no longer decreases on the validation set.

[0031] Furthermore, the present invention also proposes an apparatus for generating an abstract, comprising:

[0032] The decoder is used to gradually obtain the output probability of the corresponding character in the output vocabulary based on the vocabulary built in the training database.

[0033] The data processing unit is used to extract the k words with the highest output probabilities and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1;

[0034] The summary generation unit sorts multiple candidate summaries according to their output probabilities, ensuring that the bundle size is k; after prediction, it generates a summary from the word with the highest output probability.

[0035] Furthermore, the present invention also proposes a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described summary generation method.

[0036] Compared with existing technologies, the advantages of this invention include: existing role-oriented dialogue summarization methods treat different roles separately, without considering the relationships between different roles in the dialogue. Based on this, we propose a role-oriented dialogue summarization method based on role interaction, which extracts necessary information about other roles from both the dialogue text and the summary content, helping the model generate better summaries tailored to that role, resulting in a significant improvement in the quality of the generated summaries. Attached Figure Description

[0037] Figure 1 This is a flowchart of a method for generating an abstract according to a specific embodiment of the present invention.

[0038] Figure 2 This is a flowchart of establishing and optimizing the abstract generation model in a specific embodiment of the present invention.

[0039] Figure 3 This is a structural block diagram of the abstract generation device in a specific embodiment of the present invention.

[0040] Figure 4 This is an example of a dialogue between a user and customer service representative in Embodiment 1 of the present invention.

[0041] Figure 5 This is the result of generating a summary of the dialogue between the user and customer service in Embodiment 1 of the present invention. Detailed Implementation

[0042] Combination Figure 1 This specific embodiment provides a method for generating a summary, including the following steps:

[0043] S1. The decoder gradually obtains the output probability of the relevant words of the corresponding role in the output vocabulary based on the constructed vocabulary in the training database; the roles include user roles and customer service roles.

[0044] S2. Extract the k words with the highest output probabilities and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1;

[0045] S3. The model sorts multiple candidate summaries according to their output probabilities and ensures that the bundle size is k. After the model prediction is completed, the candidate sequence with the highest output probability is taken as the summary.

[0046] Furthermore, in step S1, the roles include user roles and customer service roles.

[0047] Furthermore, combined Figure 2Before step S1, the process also includes establishing and optimizing the summary generation model, specifically including the following steps:

[0048] T1. Different encoders are used to encode the dialogue content according to different roles; specifically, this includes:

[0049] Step T11: Concatenate the speaker role information and dialogue content of the dialogue statements in turn order, and obtain the word embedding representation through the word embedding layer;

[0050] Step T12: Encode the word embedding representation using an encoder to obtain the representation of each word in the dialogue;

[0051] T2. The decoder obtains the statement representation of the corresponding role, and during decoding, the decoder uses different attention modules to focus on the statement representations of other roles; specifically, this includes:

[0052] Step T21: Based on the speaker role of each sentence in the dialogue, represent the dialogue according to different roles;

[0053] Step T22: The user decoder focuses on the statements of all roles separately to obtain the distribution representation of attention and the encoder context representation;

[0054] T3. Calculate the KL divergence across the attention distribution for the statement representations of the same character that different decoders pay attention to, and obtain the loss function across attention interactions.

[0055] T4. In the self-attention module of the decoder, the hidden layer representation of each decoder pays attention to the hidden layer representation of other decoders, forming a role self-attention interaction; specifically, it includes:

[0056] Step T41: Each decoder obtains the hidden state of the decoder at the current moment based on the decoding state of the previous moment and the current input information;

[0057] Step T42: Focus the current hidden state of the decoder on the hidden states of all previous times of other decoders to obtain the contextual representation of the role summary;

[0058] T5. The decoder predicts the output probability of each word at each position, and obtains the loss function of the summary based on the output probability and the maximum likelihood estimation. The decoder predicts the output probability of each word at each position by: the decoder predicts the output probability of each word at each position in the summary based on its own hidden state information, the encoder context representation and the context representation of the role summary.

[0059] T6. Combining the loss function across attention interactions and the loss function of the summary, the model is trained and optimized using gradient descent; specifically, this includes:

[0060] T61. Weighted fusion of the loss function across attention interactions and the loss function of summarization;

[0061] T62. Use the gradient descent algorithm to train and optimize the model until the loss no longer decreases on the validation set.

[0062] Combination Figure 3 This specific embodiment also includes a summary generation apparatus, comprising:

[0063] The decoder is used to gradually obtain the output probability of the corresponding character in the output vocabulary based on the vocabulary built in the training database.

[0064] The data processing unit is used to extract the k words with the highest output probabilities and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1;

[0065] The summary generation unit sorts multiple candidate summaries according to their output probabilities, ensuring that the bundle size is k; after prediction, it generates a summary from the word with the highest output probability.

[0066] This specific embodiment also proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described summary generation method.

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] Example 1

[0069] This embodiment proposes a summary generation method, where the dialogue roles are user and customer service representative, denoted as user (u) and customer service representative (a); the method includes the following steps:

[0070] Step 1: Encode the dialogue content using an encoder; specifically, this includes:

[0071] Step 11: Concatenate the speaker role information and content of the dialogue statements in turn order, and obtain the word embedding representation through the word embedding layer; for example, such as Figure 2 As shown, the input for this example is: "User: My previously linked phone number is no longer in use, and I've forgotten my password. Customer Service: Would you mind if I transferred you to a specialist?..."

[0072] Step 12: Encode the word embedding representations using a neural network encoder to obtain the representation H of each word in the dialogue.enc H enc It is an n×d dimensional vector, where n represents the length of the input sequence and d represents the size of the hidden layer dimension;

[0073] Step 2: Obtain the statement representation for each speaker in the dialogue, and use different attention modules to focus on the content of each speaker during decoding; specifically, this includes:

[0074] Step 21: Based on the speaker role of each sentence in the dialogue, divide the dialogue representation into user sentence representations using user masks and customer service masks. Customer service statements In this example, sentences 1, 3, 5, and 7 correspond to the following representation: The corresponding representations for sentences 2, 4, 6, and 8 are as follows:

[0075] Step 22: The user decoder performs cross-attention attention on both user statements and customer service statements to obtain cross-attention distribution representations. and encoder context representation

[0076] Step 23: The customer service decoder focuses attention on both the user's statement and the customer service statement, obtaining a distribution representation that spans attention. and encoder context representation

[0077] Step 3: Calculate the KL divergence across attention distributions for content about the same character across different decoders, using it as the loss function across attention interactions; specifically, this includes:

[0078] Step 31: Calculate the KL divergence of the two decoders across the attention distribution for user role content, and obtain the user loss function across attention interactions, as shown in the formula:

[0079]

[0080] The Avg function averages the attention distribution across all decoder steps;

[0081] Step 32: Calculate the KL divergence of the two decoders across the attention distribution for customer service role content, and obtain the customer service loss function across attention interactions, the formula of which is:

[0082]

[0083] The Avg function averages the attention distribution across all decoder steps.

[0084] Step 4: In the self-attention module of the decoder, the hidden layer representation of each decoder pays attention to the hidden layer representations of other decoders, forming a role self-attention interaction; specifically, it includes:

[0085] Step 41: The user decoder uses the decoding layer to determine the decoding state from the previous moment. and current input information Obtain the hidden state of the user decoder at the current moment.

[0086] Step 42: The customer service decoder uses the decoding layer to determine the decoding state from the previous moment. and current input information Obtain the hidden state of the customer service decoder at the current moment.

[0087] Step 43: Focus on the hidden state of the user decoder at the current moment, and then focus on the hidden states of the customer service decoder at all previous moments to obtain the contextual representation of the user role summary. The formula is:

[0088]

[0089] The Attn function is the attention mechanism;

[0090] Step 44: Focus the current hidden state of the customer service decoder on all the previous hidden states of the user decoder to obtain the contextual representation of the customer service summary. The formula is:

[0091]

[0092] The Attn function is an attention mechanism; Mapped to And obtain a weight for each representation, and then sum them up to get

[0093] Step 5: The decoder predicts the output probability of the word at each position and uses maximum likelihood estimation to obtain the loss function for the summary; further, this includes:

[0094] Step 51: The user decoder predicts the probability of the output word at the current time step in the user summary based on its own hidden state information, the encoder context representation, and the context representation of the user role summary. The formula is:

[0095]

[0096] The function F contains a multi-layer perceptron layer and a softmax layer;

[0097] Step 52: The customer service decoder predicts the probability of the output word at the current time step in the customer service summary based on its own hidden state information, the encoder context representation, and the context representation of the customer service role summary. The formula is:

[0098]

[0099] The function F contains a multi-layer perceptron layer and a softmax layer;

[0100] Step 53: Calculate the loss function for user summary and customer service summary using maximum likelihood loss:

[0101]

[0102] Where 'a' is a hyperparameter that adjusts the weights of user and customer service summaries; and For terms in the standard answers for user summaries and customer service summaries;

[0103] Step 6: Combine the loss function across attention interactions and the loss function for summarizing to train and optimize the model using gradient descent; further, this includes:

[0104] Step 61: Weight and fuse the loss function across attention interactions and the summarization loss function, using the following formula:

[0105] L = L nll +β(L art-user +L att-agent )

[0106] Where β is a hyperparameter that adjusts the weights of the two loss functions;

[0107] Step 62: Train and optimize the model using the gradient descent algorithm until the loss no longer decreases on the validation set.

[0108] Step 7: The user decoder predicts the output probability of the relevant words in the output vocabulary for each position based on the vocabulary constructed from the training database in the model. The k words with the highest output probabilities are extracted and concatenated to the decoded word sequence in the model as candidate summaries, where k is greater than 1, for example, k could be 4 or 5, etc. The model sorts the multiple candidate summaries according to their output probabilities, ensuring that the bundle size is k. When the model predicts the end character... <eos>The time indicates the end of the sequence. After the model prediction is completed, the candidate sequence with the highest output probability is taken as the user summary.

[0109] Step 8: The customer service decoder predicts the output probability of relevant words in the output vocabulary for each position based on the vocabulary built from the training database in the model. The k words with the highest output probabilities are extracted and concatenated to the decoded word sequence in the model as candidate summaries, where k is greater than 1. The model sorts multiple candidate summaries according to their output probabilities, ensuring that the bundle size is k. When the model predicts the end character... <eos>The time indicates the end of the sequence. After the model prediction is completed, the candidate sequence with the highest output probability is taken as the customer service summary.

[0110] Experimental results

[0111] We applied our proposed role-based interaction method to two different summarization models, PGN and BERTAbs, and validated it on two different dialogue summarization datasets (CSDS, MC). We compared the results using multiple evaluation metrics, and the results are shown in Tables 1 and 2.

[0112] Table 1. Performance of different summarization methods on the CSDS dataset.

[0113]

[0114] Table 2 shows the performance of different summarization methods on the MC dataset.

[0115]

[0116] In the table, PGN-single and BERTAbs-single summarize each of the two roles individually. PGN-multi and BERTAbs-multi use a multi-task framework to jointly summarize the two roles. PGN-both and BERTAbs-both use our proposed two role interaction methods for joint summary generation. ROUGE-2, ROUGE-L, and MoverScore are three commonly used automatic summary evaluation metrics. Results marked with an asterisk indicate that the result significantly outperforms the corresponding two baseline methods. By comparing the results of user summaries and customer service summaries, we can see that our proposed generation method significantly improves upon individual summarization and multi-task training methods on both the summarization model and the dataset. This indicates that our proposed role interaction method can better utilize the key information provided by other roles for role summary generation.

[0117] When the user and customer service have a conversation like Figure 4 As shown, the user summary and customer service summary obtained by the summary generation method proposed in this invention are as follows: Figure 5 As shown.

[0118] Example 2

[0119] The difference between this embodiment and Embodiment 1 is that the number of participants in the dialogue exceeds two. Specifically, in a multi-person dialogue scenario, this includes, for example, m roles S1, ..., S2. m For one of the dialogue characters S i We use a separate decoder dec i Decoding is performed; first, during the encoding process, we obtain the character S. i Representation of corresponding statements In cross-attention interactions, each decoder pays attention to The decoder dec j for The attention distribution is denoted as Its encoder context is represented as Then we calculate the cross-attention interaction loss for all decoders for each character. Let character S... i For example, its cross-attention interaction loss is:

[0120]

[0121] In the self-attention module of the decoder, each decoder pays attention to the hidden layer representations of all other decoders, so that the decoder dec... i For example, the context representation of the role summary obtained at time t is as follows:

[0122]

[0123] The probability of each decoder predicting the output character of the summary answer at a given position is:

[0124]

[0125] The summary loss is calculated using the maximum likelihood loss:

[0126]

[0127] Where w k Let k be the weight of the summary corresponding to role k when calculating the loss.

[0128] The final total loss function is:

[0129] All other steps and methods are the same as in Example 1.

[0130] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.< / eos> < / eos>

Claims

1. A method for generating a summary, characterized in that, Includes the following steps: The establishment and optimization of the summary generation model includes the following steps: T1. Use different encoders to encode the dialogue content according to different roles; T2. The decoder obtains the statement representation of the corresponding role. When decoding, the decoder uses different attention modules to focus on the statement representations of other roles. T3. Calculate the KL divergence across the attention distribution for the statement representations of the same character that different decoders pay attention to, and obtain the loss function across attention interactions. T4. In the self-attention module of the decoder, the hidden layer representation of each decoder pays attention to the hidden layer representation of other decoders, forming a role self-attention interaction. T5. The decoder predicts the output probability of the word at each position, and obtains the loss function of the summary based on the output probability and the maximum likelihood estimation. T6. Combine the loss function across attention interactions and the loss function of the summary to train and optimize the model using gradient descent; After establishing and optimizing the summary generation model, perform the following steps: S1. The decoder gradually obtains the output probability of the corresponding character in the output vocabulary based on the vocabulary constructed in the training database. S2. Extract the k words with the highest output probabilities and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1; S3. The model sorts multiple candidate summaries according to their output probabilities and ensures that the bundle size is k. After the model prediction is completed, the candidate sequence with the highest output probability is taken as the summary. In step T1, the step of encoding the dialogue content according to different roles using different encoders includes: Step T11: Concatenate the speaker role information and dialogue content of the dialogue statements in turn order, and obtain the word embedding representation through the word embedding layer; Step T12: Encode the word embedding representation using an encoder to obtain the representation of each word in the dialogue; In step T2, the decoder obtains the statement representation of the corresponding role. During decoding, the decoder uses different attention modules to focus on the statement representations of other roles, including: Step T21: Based on the speaker role of each sentence in the dialogue, represent the dialogue according to different roles; Step T22: The decoder focuses attention on the statements of all characters separately, obtaining the distribution representation of attention and the context representation of the encoder; In step T4, the self-attention module of the decoder, in which the hidden layer representation of each decoder pays attention to the hidden layer representation of other decoders to form a role self-attention interaction, includes: Step T41: Each decoder obtains the hidden state of the decoder at the current moment based on the decoding state of the previous moment and the current input information; Step T42: Focus the current hidden state of the decoder on the hidden states of all previous times of other decoders to obtain the contextual representation of the role summary; In step T5, the decoder predicts the output probability of the word at each position, including: The decoder predicts the output probability of each word in the summary based on its own hidden state information, the encoder context representation, and the context representation of the role summary; In step T6, the step of training and optimizing the model using gradient descent by combining the loss function across attention interactions and the loss function of the summary includes: T61. Weighted fusion of the loss function across attention interactions and the loss function of summarization; T62. Use the gradient descent algorithm to train and optimize the model until the loss no longer decreases on the validation set.

2. The method for generating an abstract according to claim 1, characterized in that, In step S1, the roles include user roles and customer service roles.

3. An apparatus for generating a summary, characterized in that, include: The decoder is used in the model building and optimization stage to concatenate the speaker role information and dialogue content of the dialogue statements in turn order, pass them through the word embedding layer to obtain word embedding representations, and use different encoders to encode the word embedding representations according to different roles to obtain the representation of each word in the dialogue. as well as Based on the speaker role of each sentence in the dialogue, the dialogue is represented according to different roles, and attention is paid to the sentences of all roles separately to obtain the attention distribution representation and the encoder context representation. as well as Calculate the KL divergence across the attention distribution for the statement representations of the same character that different decoders focus on, in order to obtain the loss function across attention interactions; as well as In its own self-attention module, the hidden state at the current moment is obtained based on the decoding state at the previous moment and the current input information. This hidden state is then used to pay attention to the hidden states of all previous moments of other decoders, so as to form a character self-attention interaction and obtain a contextual representation of the character summary. as well as Based on its own hidden state information, the encoder context representation, and the role summary context representation, the output probability of each word at each position in the summary is predicted, and the loss function of the summary is obtained by combining maximum likelihood estimation; and The loss function across attention interactions and the loss function of the summary are weighted and fused, and the model is trained and optimized using the gradient descent algorithm until the loss no longer decreases on the validation set. as well as During the model application phase, the output probabilities of the corresponding characters in the output vocabulary are gradually obtained by constructing a vocabulary based on the training database. The data processing unit is used to extract the k words with the highest probability output by the decoder during the model application stage and concatenate them to the decoded word sequence in the model as candidate summaries, where k is greater than 1. The summary generation unit is used to sort the multiple candidate summaries obtained by the data processing unit according to the output probability during the model application phase, and ensure that the bundle size is k; after the model prediction is completed, the candidate sequence with the highest output probability is taken as the summary.

4. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the digest generation method as described in any one of claims 1 to 2.

Images