A method for distinguishing dialogue summaries from interlocutors based on contrastive learning

Abstract

This invention provides a method for distinguishing dialogue summaries from speakers based on contrastive learning. The method is characterized by first constructing an enhanced sequence-to-sequence neural network model based on BART, where the encoder-decoder can encode and decode dialogues. Then, based on the cross-entropy loss between the training dialogue summaries and the target summaries generated during training, and the contrastive learning loss in which the neural network model is assisted in encoding the training dialogue using three contrastive learning tasks, the neural network model is trained until a speaker-distinguishing model is generated that can distinguish dialogues based on speakers to generate summaries. Because three contrastive learning tasks are used to assist the encoder in encoding dialogues, the model of this invention can better understand dialogue data, resulting in higher accuracy in summarizing and avoiding fact-consistency errors.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and natural language processing, and specifically relates to a method for distinguishing dialogue summaries and interlocutors based on contrastive learning. Background Technology

[0002] With the development of pre-training techniques, pre-trained language models (such as BART) have achieved outstanding performance in many natural language processing generation tasks. For dialogue summarization tasks, multi-turn dialogues are concatenated into smooth text input for pre-trained models. However, due to differences from the pre-training objectives (such as BART's denoising generation), the performance of pre-trained models in processing dialogue data is not ideal. According to manual statistics, in the dialogue summarization dataset SAMsum, for multi-person dialogues, 50% of the summaries generated by BART contain fact consistency errors, of which 68.4% are directly related to the inability to identify the speakers.

[0003] Currently, there are two main technical approaches in the field of dialogue summarization: (1) hierarchical networks, where the entire dialogue is divided into multiple granularities such as words, turns, and whole sentences, and encoded separately. This method can capture the structural information of the dialogue, but it cannot utilize the parameters of pre-trained models; (2) pre-trained models, which concatenate the dialogue and treat it as flat text, and use other methods to assist in the generation of summaries. However, these methods are difficult to capture the structural information of the dialogue, making it difficult to identify the speakers in the dialogue. There has been no previous work on enhancing the pre-trained models to identify speakers. Summary of the Invention

[0004] To address the above problems, this invention provides a method for distinguishing between dialogue summaries and interlocutors. The technical solution adopted in this invention is as follows:

[0005] This invention provides a method for distinguishing between dialogue summaries and speakers based on contrastive learning, characterized by the following steps: Step S1, constructing a sequence-to-sequence neural network model based on BART; Step S2, inputting the training dialogue into the neural network model to generate a training dialogue summary, and calculating the cross-entropy loss between the training dialogue summary and the target summary; Step S3, using three contrastive learning tasks to assist the neural network model in encoding the training dialogue, and calculating the contrastive learning loss for the three contrastive learning tasks; Step S4, training the neural network model based on the overall training loss until a trained neural network model is generated, which is then used as the speaker discrimination model; Step S5, inputting the dialogue to be tested into the speaker discrimination model to obtain a dialogue summary corresponding to the dialogue to be tested.

[0006] The present invention provides a method for distinguishing dialogue summaries and interlocutors based on contrastive learning, which may also have the following technical features, wherein the three contrastive learning tasks are word-level contrastive learning, turn-level contrastive learning, and interlocutor-level contrastive learning.

[0007] The present invention provides a method for distinguishing dialogue summaries and interlocutors based on contrastive learning, which may also have the following technical features: Word-level contrastive learning involves: when encoding training dialogues, the neural network model brings the word representations of the same interlocutor closer together and distances the word representations of different interlocutors from the training dialogues. Round-level contrastive learning involves: obtaining one round of dialogue from one interlocutor based on the training dialogues, averaging all word representations contained in that round to obtain the representation of that round, and the neural network model bringing the representations of the same interlocutor's round closer together and distances the representations of different interlocutors from the round. Interlocutor-level contrastive learning involves: randomly sampling the training dialogues to obtain one interlocutor and all rounds of dialogue D from that interlocutor; randomly sampling one round of dialogue S from all rounds of dialogue D; using DS as global samples and S as positive samples; randomly sampling another interlocutor's dialogue from the training dialogues as negative samples; and using the neural network model to bring the global samples and positive samples closer together and distance the global samples and negative samples from the representations.

[0008] The present invention provides a method for distinguishing dialogue summaries from speakers based on contrastive learning, which also has the following technical feature: the overall training loss is the sum of cross-entropy loss and contrastive learning loss.

[0009] L = L gen +λL ctr

[0010] In the formula, L represents the overall training loss, L gen For cross-entropy loss, L ctr To compare the learning loss, λ is a hyperparameter that adjusts the two loss functions.

[0011] Invention Function and Effect

[0012] According to the method for distinguishing dialogue summaries and interlocutors based on contrastive learning of the present invention, firstly, an enhanced sequence-to-sequence neural network model is constructed based on BART. Three contrastive learning tasks are employed to assist the model in encoding dialogues based on interlocutors. The neural network model is then trained using the losses from these three contrastive learning tasks and cross-entropy loss. Since the contrastive learning loss function enables the model's encoder to generate different codes based on different interlocutors, a interlocutor discrimination model capable of distinguishing dialogues based on interlocutors is generated. Because this interlocutor discrimination model can distinguish interlocutors and thus generate summaries, compared to previous dialogue discrimination models, the method of distinguishing dialogue summaries and interlocutors based on contrastive learning of the present invention can better understand dialogue data, achieving higher accuracy in summarizing and avoiding factual consistency errors. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method for distinguishing dialogue summaries and interlocutors based on contrastive learning in this embodiment of the invention;

[0014] Figure 2 This is a schematic diagram of the coding aided by three contrastive learning tasks in an embodiment of the present invention;

[0015] Figure 3 This is a comparison chart of experimental results between the interlocutor differentiation model in this embodiment of the invention and existing differentiation models. Detailed Implementation

[0016] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes in detail a method for distinguishing dialogue summaries and interlocutors based on contrastive learning, in conjunction with embodiments and accompanying drawings.

[0017] <Example>

[0018] Figure 1 This is a flowchart of a method for distinguishing dialogue summaries and interlocutors based on contrastive learning in an embodiment of the present invention.

[0019] like Figure 1 As shown, the method for distinguishing dialogue summaries from interlocutors based on contrastive learning includes the following steps:

[0020] Step S1: Construct a sequence-to-sequence neural network model based on BART.

[0021] In this embodiment, the neural network model can be an arbitrary sequence-to-sequence pre-trained model, and the pre-trained model can also be PEGASUS.

[0022] Step S2: Input the training dialogue into the neural network model to generate a training dialogue summary, and calculate the cross-entropy loss between the training dialogue summary and the target summary.

[0023] In this embodiment, a training dialogue summary s is generated based on the neural network model, and the cross-entropy loss L between the summary s and the target summary d of the training dialogue is calculated. gen This cross-entropy loss is a commonly used cross-entropy loss function in existing technologies.

[0024] Step S3: Use three contrastive learning tasks to assist the neural network model in encoding the training dialogue and calculate the contrastive learning loss for the three contrastive learning tasks.

[0025] In this embodiment, the three contrastive learning tasks are word-level contrastive learning, turn-level contrastive learning, and interlocutor-level contrastive learning.

[0026] Figure 2 These are schematic diagrams illustrating three types of contrastive learning task-assisted coding in embodiments of the present invention.

[0027] like Figure 2 As shown, word-level contrastive learning (Token-level SCL in the figure) is as follows: when encoding the training dialogue, the auxiliary neural network model brings the word representations of the same speaker in the training dialogue closer together and pulls the word representations of different speakers in the training dialogue further apart.

[0028] Round-level contrastive learning (Token-level SCL in the diagram) is as follows: Based on the training dialogue, one round of dialogue of one of the speakers is obtained, and the average of all word representations contained in the round of dialogue is taken to obtain the representation of the round of dialogue. Round-level contrastive learning assists the neural network model to bring the representations of the rounds of dialogue of the same speaker closer together and to separate the representations of the rounds of dialogue of different speakers.

[0029] The interlocutor-level contrastive learning (Global-level SCL in the figure) is as follows: Randomly sample the training dialogue to obtain a one-person dialogue and all rounds of dialogue D of that one-person dialogue. Randomly sample a round of dialogue S based on all rounds of dialogue D. Use DS as global samples and S as positive samples. Randomly sample another one-person dialogue based on the training dialogue as negative samples. This helps the neural network model to narrow the gap between the representation of global samples and positive samples, and widen the gap between the representation of global samples and negative samples.

[0030] In this embodiment, to enable the model's encoder to incorporate more interlocutor information into the encoding, sampling was performed at both the word and turn levels during the dialogue, and the source interlocutor s of sample i was used as the basis for the sampling. i The samples are divided into positive and negative sample pairs.

[0031] The formulas for calculating the contrastive learning loss function for the three contrastive learning tasks mentioned above are as follows:

[0032]

[0033]

[0034] L ctr =L + +L -

[0035] In the formula, σ is the sigmoid function, o i o j These are the encoder's encodings of samples i and j, respectively, L + L represents the contrastive learning loss calculated on positive sample pairs. - L represents the contrastive learning loss calculated on negative sample pairs. ctr To compare learning loss.

[0036] Step S4: Train the neural network model based on the overall training loss until a trained neural network model is generated, and use it as the speaker differentiation model.

[0037] The overall training loss is the sum of the cross-entropy loss and the contrastive learning loss.

[0038] L = L gen +λL ctr

[0039] In the formula, L is the overall training loss, and λ is the hyperparameter that adjusts the two loss functions.

[0040] Step S5: Input the dialogue to be tested into the speaker discrimination model to obtain the dialogue summary corresponding to the dialogue to be tested.

[0041] Figure 3 This is a comparison chart of experimental results between the interlocutor differentiation model in this embodiment of the invention and existing differentiation models.

[0042] In this embodiment, to verify the effectiveness of the method for distinguishing dialogue summaries and speakers based on contrastive learning, a speaker discrimination model and existing discrimination models are tested on the dialogue summary dataset SAMsum and the AMI dataset, respectively. Figure 3 As shown, the interlocutor discrimination model of the present invention significantly outperforms other existing discrimination models. Among them, compared with the baseline model (BART(Ours) in the figure), the discrimination model (BART+ALL SCL tasks in the figure) which uses three contrastive learning methods to assist in the discrimination model achieves the best performance to date.

[0043] Functions and effects of the embodiments

[0044] According to the method for distinguishing dialogue summaries and speakers based on contrastive learning provided in this embodiment, an enhanced sequence-to-sequence neural network model is constructed based on BART. Three contrastive learning tasks are employed to assist the model in encoding dialogues based on speakers. The neural network model is then trained using the losses from these three contrastive learning tasks and cross-entropy loss, generating a speaker discrimination model capable of distinguishing dialogues based on speakers. Because this speaker discrimination model can distinguish speakers and thus generate summaries, compared to previous dialogue discrimination models, the method for distinguishing dialogue summaries and speakers based on contrastive learning in this embodiment can better understand dialogue data, resulting in higher accuracy in summarizing and avoiding factual consistency errors.

[0045] In this embodiment, since three contrastive learning tasks—word-level contrast, round-level contrast, and speaker-level contrast—are employed, and a contrastive learning loss function is used to train the neural network model, the model's encoder can generate different codes based on different speakers. This is beneficial for the model's decoder to generate corresponding summaries based on the speakers, especially for summaries that conform to the "someone does something" pattern.

[0046] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A method for distinguishing a dialogue summary from a dialogue participant based on contrastive learning, characterized in that, The method comprises the following steps: Step S1, constructing a sequence-to-sequence neural network model based on BART; Step S2, inputting a training dialogue into the neural network model to generate a training dialogue summary, and calculating a cross-entropy loss between the training dialogue summary and a target summary; Step S3, using three contrastive learning tasks to assist the neural network model in dialogue encoding of the training dialogue, and calculating a contrastive learning loss of the three contrastive learning tasks; Step S4, training the neural network model based on the overall training loss until a trained neural network model is generated, which is used as a dialogue distinguisher; Step S5, inputting a to-be-tested dialogue into the dialogue distinguisher to obtain a dialogue summary corresponding to the to-be-tested dialogue, wherein the three contrastive learning tasks are word-level contrastive learning, round-level contrastive learning, and dialogue-level contrastive learning, wherein the word-level contrastive learning is: when the neural network model encodes the training dialogue, it pulls the word representations of the same dialogue in the training dialogue closer and pulls the word representations of different dialogues in the training dialogue farther apart, the round-level contrastive learning is: based on the training dialogue, a round dialogue of one dialogue is obtained, the representation of the round dialogue is obtained by averaging all word representations contained in the round dialogue, the neural network model pulls the representations of the dialogue rounds of the same dialogue closer and pulls the representations of the dialogue rounds of different dialogues farther apart, the dialogue-level contrastive learning is: randomly sampling the training dialogue to obtain a dialogue and all round dialogues D of the dialogue, based on the all-round dialogue D, a round dialogue S is randomly sampled, D-S is used as a global sample, S is used as a positive sample, and the dialogue of another dialogue is randomly sampled based on the training dialogue and used as a negative sample, the neural network model is used to pull the representations of the global sample and the positive sample closer and pull the representations of the global sample and the negative sample farther apart.

2. The method for distinguishing dialogue summaries and dialogues based on contrastive learning according to claim 1, wherein: wherein the overall training loss is the sum of the cross-entropy loss and the contrastive learning loss: ， wherein is the total training loss, is the cross-entropy loss, is the contrastive learning loss, and λ is a hyperparameter that adjusts the two loss functions.

Images