Open-domain dialogue generation method and device based on lifelong distillation

Abstract

This invention provides a method and device for generating open-domain dialogues based on lifelong distillation. The method comprises steps 1 to 3. This method proposes sequentially modeling dialogue data at different levels, from shallow to deep, to ensure accurate modeling of features at each dialogue level, effectively improving model accuracy. A model trained on dialogues at previous levels is used as a teacher, and the teacher's knowledge is distilled to the student when modeling dialogues at new levels. This allows the student to comprehensively model various dialogue features, thus maintaining dialogue diversity.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of open-domain dialogue generation, and in particular to a method and device for generating open-domain dialogue based on lifelong distillation. Background Art

[0002] With the rapid development of information technology and the internet, the increasing popularity of social media, the surge in the number of smartphones, and the increasing number of human-computer dialogue scenarios, dialogue systems are playing an increasingly important role. Open-domain dialogue systems are an important branch of dialogue systems, capable of generating everyday conversations across a wide range of domains. Current open-domain dialogue generation methods tend to generate monotonous responses. Some methods attempt to alleviate this problem by increasing the diversity of generated responses, but the responses generated by these models still differ significantly from real responses. Furthermore, blindly improving the diversity of generated responses can lead the model to generate words irrelevant to the current conversation, making responses lengthy and cryptic. The accuracy of responses is difficult to control, resulting in meaningless responses, which contradicts the original purpose of dialogue systems. Therefore, developing an open-domain dialogue generation method and device based on lifelong distillation that can effectively overcome the shortcomings of the aforementioned related technologies has become a pressing technical issue in the industry. Summary of the Invention

[0003] In response to the above-mentioned problems existing in the prior art, embodiments of the present invention provide a method and device for generating open-domain dialogues based on lifelong distillation.

[0004] In a first aspect, an embodiment of the present invention provides an open-domain dialogue generation method based on lifelong distillation, comprising: step 1: preprocessing the original data to obtain a validation set, a test set, and a training set containing multiple dialogues with no more than three historical utterances; step 2: constructing an open-domain dialogue generation model based on lifelong distillation, randomly inputting training samples and verification samples into the open-domain dialogue generation model based on lifelong distillation in batches, starting an iterative model training process, obtaining evaluation indicators on the training set and the verification set, stopping when the indicator on the verification set no longer increases or the iteration reaches a predetermined number of times, and saving the optimal model on the verification set. The constructing of the open-domain dialogue generation model based on lifelong distillation comprises: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; step 3: loading the open-domain dialogue generation model based on lifelong distillation with the best evaluation indicator on the verification set, testing it with a batch of test samples, and outputting and saving the results of the dialogue generation.

[0005] Based on the content of the above method embodiment, in the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the original data in step 1 contains multi-round dialogue data, and each dialogue in the training set is processed into multiple dialogues with no more than 3 dialogue history utterances.

[0006] Based on the content of the above method embodiment, the open-domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the construction of the basic encoding and decoding model structure in step 2 is specifically: using a Transformer-based sequence-to-sequence generation framework to construct an embedding layer, encoder, decoder and output layer.

[0007] Based on the content of the above method embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the teacher-student architecture based on lifelong distillation is constructed in step 2 specifically as follows: when constructing the model, the old model is introduced as a teacher to help distill the dialogue knowledge of the previous level, and the model Mm-1 trained at the previous moment is used as the teacher of the model Mm trained at the current moment m. At the next moment m+1, the current model Mm serves as the teacher to guide the training of the model Mm+1, and so on for sequential distillation.

[0008] Based on the content of the above method embodiment, the open-domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the objective function constructed in step 2 is specifically: in order to preserve part of the knowledge learned in the previous training process during the current training process, so that the model can learn the characteristics of different rounds of dialogues, improve the accuracy and diversity of the model, in addition to the loss between the hard label and the predicted distribution of each sample, the model trained in the previous process is introduced and used as the teacher model, and the output soft label guides the current model to make predictions.

[0009] Based on the content of the above method embodiment, the open-domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, after constructing the open-domain dialogue generation model based on lifelong distillation in step 2, the training and verification samples are randomly input into the model in batches, and the Adam optimizer is used to enable the model to gradually learn the optimal parameter values, calculate the evaluation index on the verification set, stop training when the index on the verification set no longer increases or the model training iteration reaches a predetermined number of times, and save the relationship classification model with the best performance on the verification set.

[0010] Based on the content of the above method embodiment, the open-domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the use of test sample batches for testing in step 3 is specifically as follows: loading the open-domain dialogue generation model based on lifelong distillation trained in step 2, and then inputting the sample batches in the test set into the model. At this time, the parameters of the model are fixed and unchanged. After the model calculation, the dialogue response is generated.

[0011] In a second aspect, an embodiment of the present invention provides an open-domain dialogue generation device based on lifelong distillation, comprising: a first main module, configured to implement step 1: preprocessing the raw data to obtain a validation set, a test set, and a training set containing multiple dialogues with no more than three historical utterances; a second main module, configured to implement step 2: constructing an open-domain dialogue generation model based on lifelong distillation, randomly inputting training samples and validation samples into the open-domain dialogue generation model based on lifelong distillation in batches, starting an iterative model training process, obtaining evaluation indicators on the training set and validation set, stopping when the indicator on the validation set no longer increases or the iteration reaches a predetermined number of times, and saving the optimal model on the validation set. The constructing of the open-domain dialogue generation model based on lifelong distillation comprises: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; a third main module, configured to implement step 3: loading the open-domain dialogue generation model based on lifelong distillation with the best evaluation indicator on the validation set, testing it with a batch of test samples, and outputting and saving the dialogue generation results.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0013] at least one processor; and

[0014] at least one memory in communication with the processor, wherein:

[0015] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the open-domain dialogue generation method based on lifelong distillation provided by any one of the various implementations of the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the open-domain dialogue generation method based on lifelong distillation provided by any one of the various implementation methods of the first aspect.

[0017] The open-domain dialogue generation method and device based on lifelong distillation provided by the embodiments of the present invention propose to model dialogue data at different levels in sequence from shallow to deep, ensuring that the features of each dialogue level can be accurately modeled, which can effectively improve the accuracy of the model. The model trained with dialogues at the old level is used as a teacher, and the teacher's knowledge is distilled to students when modeling dialogues at the new level. Students can comprehensively model various dialogue features, which is conducive to maintaining the diversity of dialogues. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flow chart of a method for generating open-domain dialogues based on lifelong distillation according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of an open-domain dialogue generation device based on lifelong distillation provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention;

[0022] Figure 4 A flow chart of another open-domain dialogue generation method based on lifelong distillation provided by an embodiment of the present invention;

[0023] Figure 5 A framework diagram of the open-domain dialogue generation model based on lifelong distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0025] In order to solve the problem that monotonous dialogue responses are easily generated in open-domain dialogues, the present invention proposes an open-domain dialogue generation method based on lifelong distillation. This method introduces the idea of ​​lifelong distillation, and models dialogue data at different levels in sequence from shallow to deep, ensuring that each level of dialogue features can be accurately modeled. This method uses the model trained on the old level dialogue as a teacher, and distills the teacher's knowledge to the student when modeling the new level dialogue, so that it can comprehensively model various dialogue features and maintain the diversity of the dialogue. Based on this idea, an embodiment of the present invention provides an open-domain dialogue generation method based on lifelong distillation, see Figure 1 The method includes: step 1: preprocessing the original data to obtain a validation set, a test set, and a training set containing multiple dialogues with no more than 3 historical utterances; step 2: constructing an open-domain dialogue generation model based on lifelong distillation, randomly inputting training samples and verification samples into the open-domain dialogue generation model based on lifelong distillation in batches, starting an iterative model training process, obtaining evaluation indicators on the training set and the validation set, stopping when the indicator on the validation set no longer increases or the iteration reaches a predetermined number of times, and saving the optimal model on the validation set. The construction of the open-domain dialogue generation model based on lifelong distillation includes: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; step 3: loading the open-domain dialogue generation model based on lifelong distillation with the best evaluation indicator on the validation set, testing it with a batch of test samples, and outputting and saving the dialogue generation results.

[0026] Based on the content of the above method embodiment, as an optional embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the original data in step 1 contains multi-round dialogue data, and each dialogue in the training set is processed into multiple dialogues with no more than 3 dialogue history utterances.

[0027] Based on the content of the above method embodiment, as an optional embodiment, the open-domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the construction of the basic encoding and decoding model structure in step 2 is specifically: using a Transformer-based sequence-to-sequence generation framework to construct an embedding layer, encoder, decoder and output layer.

[0028] Based on the content of the above method embodiment, as an optional embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the teacher-student architecture based on lifelong distillation is constructed in step 2 specifically as follows: when constructing the model, the old model is introduced as a teacher to help distill the dialogue knowledge of the previous level, and the model Mm-1 trained at the previous moment is used as the teacher of the model Mm trained at the current moment m. At the next moment m+1, the current model Mm is used as the teacher to guide the training of the model Mm+1, and so on for sequential distillation.

[0029] Based on the content of the above method embodiment, as an optional embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the objective function constructed in step 2 is specifically: in order to preserve part of the knowledge learned in the previous training process during the current training process, so that the model can learn the characteristics of different rounds of dialogues, improve the accuracy and diversity of the model, in each sample, in addition to the loss between the hard label and the predicted distribution, the model trained in the previous process is introduced and used as the teacher model, and the output soft label guides the current model to make predictions.

[0030] Based on the content of the above method embodiment, as an optional embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, after constructing the open domain dialogue generation model based on lifelong distillation in step 2, the training and verification samples are randomly input into the model in batches, and the Adam optimizer is used to enable the model to gradually learn the optimal parameter values, calculate the evaluation index on the verification set, stop training when the index on the verification set no longer increases or the model training iteration reaches a predetermined number of times, and save the relationship classification model with the best performance on the verification set.

[0031] Based on the content of the above method embodiment, as an optional embodiment, the open domain dialogue generation method based on lifelong distillation provided in the embodiment of the present invention, the use of test sample batches for testing in step 3 is specifically as follows: loading the open domain dialogue generation model based on lifelong distillation trained in step 2, and then inputting the sample batches in the test set into the model. At this time, the parameters of the model are fixed and unchanged. After the model calculation, the dialogue response is generated.

[0032] The open-domain dialogue generation method based on lifelong distillation provided by the embodiment of the present invention proposes to model dialogue data at different levels in sequence from shallow to deep, ensuring that the features of each dialogue level can be accurately modeled, which can effectively improve the accuracy of the model. The model trained with dialogues at the old level is used as a teacher, and the teacher's knowledge is distilled to the student when modeling dialogues at the new level. The student can comprehensively model various dialogue features, which is conducive to maintaining the diversity of the dialogue.

[0033] Another embodiment provides an open-domain dialogue generation method based on lifelong distillation, comprising: Step 1: Preprocessing the dialogue data in the original dataset to obtain multiple dialogue segments with no more than three dialogue history utterances. In this embodiment, the DailyDialog and ESConv datasets, which are widely used in open-domain dialogue generation tasks, are selected. Since their introduction, these two datasets have gradually gained widespread use in the field of open-domain dialogue generation. DailyDialog contains high-quality multi-round dialogue data collected from daily conversations, including dialogue text, dialogue action labels, and emotion category labels. The present invention only uses the plain text of dialogues in this dataset. The ESConv dataset is collected and annotated to assist in the research of emotional support technology in dialogue systems. Therefore, in addition to the plain text of dialogues, it also includes dialogue category labels related to emotional support. The present invention also only uses the plain text of dialogues in this dataset.

[0034] This method processes each conversation in the training sets of the two datasets mentioned above into multiple conversations with no more than three utterances in the conversation history. The conversation types include multi-round and single-round conversations, which are more consistent with complex real-world conversation scenarios. The processed DailyDialog dataset contains 68.5K data in the training set, 6.1K data in the validation set, and 5.8K data in the test set, for a total of 77.7K data. The processed ESConv dataset contains 28.6K data in the training set, 8.0K data in the validation set, and 7.3K data in the test set, for a total of 43.9K data.

[0035] Step 2: Reference Figure 4 , randomly input the training samples and validation samples into the open-domain dialogue generation model based on lifelong distillation in batches, start the iterative model training process, obtain the evaluation indicators on the training set and validation set, stop when the indicator on the validation set no longer increases or the iteration reaches a certain number of times, and save the optimal model on the validation set.

[0036] In the embodiment, in order to comprehensively model multiple levels of dialogue features and ensure the performance of the model on different levels of dialogue, the method proposed in the present invention will sequentially model dialogue data from shallow to deep levels, and when modeling a new model, the old model will be introduced as a teacher to help distill the dialogue knowledge of the previous level. Figure 5 , here we explain the implementation process of the entire model.

[0037] 1) Building a Basic Encoder-Decoder Model Structure: This paper uses a Transformer-based sequence-to-sequence generation framework. Compared to recurrent neural network-based encoder-decoder frameworks, this framework better captures important information from the conversation history, eliminates long-range dependencies, and can universally model both single-turn and multi-turn conversations. The basic encoder-decoder model structure consists of four modules: an embedding layer, an encoder, a decoder, and an output layer.

[0038] The embedding layer is used to segment the input conversation text and pass each segmentation through a N-th v Dictionary Mapped to the corresponding vector, where D is the vector dimension of the embedding layer. This paper adopts the word segmentation method of the pre-trained language model BERT. If the current conversation history has multiple utterances, they will be separated by '[SEP]', as shown below:

[0039]

[0040] in, The word vector of the jth word in the i-th speech is . In addition, in order to use the relative and absolute positions in the text sequence, the present invention inputs the position information into the encoder. Consistent with the position encoding in Transformer, the present invention uses sine and cosine formulas to encode the position The present invention adds the word vector and the position vector, and finally inputs the entire sequence T into the encoder for encoding. The specific entire sequence T is as follows:

[0041]

[0042]

[0043] The encoder consists of N e The network is composed of a layer of encoding layers, each of which is divided into two sub-layers: a multi-head attention layer and a fully connected layer. Each sub-layer uses a residual connection method. When inputting the next sub-layer, the input and output of this sub-layer are added together and input together to retain the information of the previous layer, thereby preventing the gradient from disappearing or exploding due to the deepening of the deep network. In addition, a layer normalization operation is added after each sub-layer to keep its distribution stable. Therefore, the output of each sub-layer is expressed as LayerNorm(x+sublayer(x)), where sublayer() is the calculation function of the sub-layer. The encoder receives the text sequence of the conversation history and outputs its encoded sequence. Therefore, the output of the encoder is:

[0044] o E =N e ×EncoderLayer(T)

[0045] The decoder consists of Nd layers of decoding layers, each of which is divided into three sublayers: a masked multi-head attention layer, a multi-head attention layer, and a fully connected layer. The masked multi-head attention layer is similar to the multi-head attention head structure, but the difference is that the input is masked to ensure that only the previous response at the current moment is input. Similar to the encoding part, residual connections are used in each sublayer, followed by layer normalization. The decoding part inputs the encoded part and the response at the previous time t-1, and outputs the current moment's response segmentation:

[0046]

[0047] The output layer uses the response word y at the previous moment t-1 position t-1 The last layer of decoding layer is output o t , to predict the distribution probability p of the response segmentation at the current moment t , as shown below:

[0048]

[0049] Among them, linear() is a fully connected mapping.

[0050] 2) Constructing a teacher-student architecture based on lifelong distillation: In order to enable the model to comprehensively model multiple levels of dialogue features and ensure the performance of the model on different levels of dialogue, the present invention sequentially models dialogue data from shallow to deep levels, and introduces the old model as a teacher to help distill the dialogue knowledge of the previous level when modeling a new model. m For example (m>1), the current model M m is the student model, and the teacher model M m-1 is the m-1th model. Among them, the teacher model M m- 1 is the student model during the m-1th training. The model architectures of the teacher model and the student model in the present invention are both the basic encoding and decoding model structures described above.

[0051] Teacher Model M m-1 The training data is the number of dialogue history utterances 1≤N k ≤m-1 dialogue. The present invention does not use the teacher model M m-1 The training data is limited to the number of conversation history utterances N k =m-1 dialogue data, there are two reasons: on the one hand, the dialogue data of one level is limited, and only the number of dialogue history utterances is N. k=m-1 dialogue training may lead to insufficient model training due to too few dialogue samples at the current level; on the other hand, from a data perspective, it introduces knowledge of shallow dialogues to the current teacher model to prevent the model from overfitting the characteristics of the current level dialogue and ignoring the knowledge of the previous level dialogues.

[0052] After the m-1th training, the optimal model M during the training process m-1 Will be saved, and during the mth training, the saved teacher model will get the predicted probability distribution of the response word t The present invention refers to this probability distribution as a training model M m Soft label.

[0053] Student Model M m With Teacher Model M m-1 Similarly, the training data consists of the number of conversation history utterances 1≤N k The student model is composed of ≤m conversation data. When modeling the current level of conversation features, it can introduce the conversation knowledge of the previous level from the data perspective and learn the common conversation features of multiple levels.

[0054] In terms of model, when training model M m When the model M saved after training at the previous moment m-1 It is used as a teacher model to guide the training of the current model. m-1 It contains information about shallow dialogues, and transfers this knowledge to the student model M through distillation without damaging the current dialogue level modeling process. m Model M m It can not only learn the features of the current level of dialogue, but also introduce the dialogue knowledge of the previous level through distillation. Therefore, it integrates dialogue features of multiple levels in the modeling process.

[0055] Similarly, the student model M m The predicted probability distribution p of the generated response word t will be obtained t m .

[0056] 3) Constructing the objective function: The dialogue generation task basically uses the MLE method to calculate the loss between the true 0-1 label and the model's predicted distribution. The specific MLE method is as follows:

[0057]

[0058] Among them, q is a unique hot encoding, also known as a hard label, where only the dimension value corresponding to the real word is 1, and the rest of the values ​​are 0. iis the word in the $i$-th dimension of the dictionary $V$. During lifelong distillation, the present invention hopes that in the current training process, it can preserve some knowledge learned in the previous training processes, enabling the model to learn the features of different rounds of conversations and improving the accuracy and diversity of the model. Therefore, in addition to the loss between the hard label and the predicted distribution for each example, the present invention introduces the model trained in the previous process, treats it as a teacher model, and allows the soft labels output by it to guide the current model for prediction.

[0059] Except for the first round of training with only hard label supervision, in subsequent model training, the model from the previous round of training will be used as the teacher model for distillation. Assume that currently it is the training of the $m$-th ($1 < m \leq M$) model:

[0060]

[0061]

[0062]

[0063] where is the soft label of the current response token $t$ output by the $(m - 1)$-th model, is the current model $M$ m the distribution probability of the current response token $t$ output by. The training loss of the $m$-th ($m > 1$) model is the sum of the prediction losses of the hard label and the soft label, specifically as follows:

[0064]

[0065] After constructing the objective function, the training and validation samples are randomly batched and input into the model, and the Adam optimizer is used to gradually make the model learn the optimal parameter values. Calculate the evaluation metrics on the validation set. When the metrics on the validation set no longer improve or the model has been trained for a certain number of iterations, stop the training and save the dialogue generation model with the best performance on the validation set.

[0066] Step 3: Load the open-domain dialogue generation model trained in Step 2, and then batch the samples in the test set and input them into the dialogue generation model. At this time, the parameters of the model remain fixed. After model calculation, generate dialogue responses.

[0067] In the embodiment, load the model with the best performance on the validation set, input the test set $D_{test}$, and obtain the dialogue response results for each dialogue sample in the test set. The final experimental results show that the open-domain dialogue generation method based on lifelong distillation proposed by the present invention achieves the best results on both the DailyDialog and ESConv datasets.

[0068] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, the embodiment of the present invention provides an open domain dialogue generation device based on lifelong distillation, which is used to execute the open domain dialogue generation method based on lifelong distillation in the above method embodiment. Figure 2 The device includes: a first main module, used to implement step 1: preprocessing the original data to obtain a verification set, a test set, and a training set containing multiple dialogues with no more than three historical utterances; a second main module, used to implement step 2: constructing an open-domain dialogue generation model based on lifelong distillation, randomly inputting training samples and verification samples into the open-domain dialogue generation model based on lifelong distillation in batches, starting an iterative model training process, obtaining evaluation indicators on the training set and the verification set, stopping when the indicator on the verification set no longer increases or the iteration reaches a predetermined number of times, and saving the optimal model on the verification set. The construction of the open-domain dialogue generation model based on lifelong distillation includes: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; a third main module, used to implement step 3: loading the open-domain dialogue generation model based on lifelong distillation with the best evaluation indicator on the verification set, testing it with a batch of test samples, and outputting and saving the dialogue generation results.

[0069] The open domain dialogue generation device based on lifelong distillation provided by the embodiment of the present invention adopts Figure 2 In the several modules, it is proposed to model different levels of dialogue data sequentially from shallow to deep, ensuring that the features of each dialogue level can be accurately modeled, which can effectively improve the accuracy of the model. The model trained with the old level dialogue is used as a teacher, and the teacher's knowledge is distilled to the students when modeling the new level dialogue. The students can comprehensively model various dialogue features, which is conducive to maintaining the diversity of the dialogue.

[0070] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above-mentioned method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above-mentioned device embodiment provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, and ensure the practicality of the technical solutions, they can improve the device in the above-mentioned device embodiment to obtain the corresponding device class embodiment, thereby obtaining the corresponding device class embodiment for implementing the methods in other method class embodiments. For example:

[0071] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a first sub-module, which is used to realize that the original data in step 1 contains multi-round dialogue data, and processes each dialogue in the training set into multiple dialogues with a number of dialogue history utterances not exceeding 3.

[0072] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open-domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a second sub-module, which is used to implement the construction of the basic encoding and decoding model structure in step 2, specifically: using a Transformer-based sequence-to-sequence generation framework to construct an embedding layer, encoder, decoder and output layer.

[0073] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a third sub-module, which is used to implement the construction of the teacher-student architecture based on lifelong distillation in step 2, specifically: when constructing the model, the old model is introduced as a teacher to help distill the dialogue knowledge of the previous level, and the model Mm-1 trained at the previous moment is used as the teacher of the model Mm trained at the current moment m. At the next moment m+1, the current model Mm is used as a teacher to guide the training of the model Mm+1, and so on for sequential distillation.

[0074] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a fourth sub-module, which is used to implement the construction of the objective function in step 2: in order to preserve part of the knowledge learned in the previous training process during the current training process, so that the model can learn the characteristics of dialogues in different rounds and improve the accuracy and diversity of the model, in addition to the loss between the hard label and the predicted distribution in each sample, the model trained in the previous process is introduced and used as a teacher model, and the output soft label guides the current model to make predictions.

[0075] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open-domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a fifth sub-module, which is used to implement the construction of the open-domain dialogue generation model based on lifelong distillation in step 2, randomly input the training and verification samples into the model in batches, and use the Adam optimizer to enable the model to gradually learn the optimal parameter values, calculate the evaluation index on the verification set, stop training when the index on the verification set no longer increases or the model training iteration reaches a predetermined number of times, and save the relationship classification model with the best performance on the verification set.

[0076] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the open-domain dialogue generation device based on lifelong distillation provided in the embodiment of the present invention further includes: a sixth sub-module, which is used to implement the test using a batch of test samples in step 3, specifically: loading the open-domain dialogue generation model based on lifelong distillation trained in step 2, and then inputting the sample batches in the test set into the model. At this time, the parameters of the model are fixed and unchanged, and after the model calculation, a dialogue response is generated.

[0077] The method of the embodiment of the present invention is implemented by electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device includes: at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor can call logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the aforementioned method embodiments.

[0078] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0080] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0081] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] It should be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprise..." do not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating open-domain dialogues based on lifelong distillation, characterized in that: include: Step 1: Preprocess the raw data to obtain a validation set, a test set, and a training set containing no more than three historical conversations. Step 2: Construct an open-domain dialogue generation model based on lifelong distillation, randomly input training samples and verification samples into the open-domain dialogue generation model based on lifelong distillation in batches, start the iterative model training process, obtain evaluation indicators on the training set and verification set, stop when the indicator on the verification set no longer increases or the iteration reaches a predetermined number of times, and save the optimal model on the verification set. The construction of the open-domain dialogue generation model based on lifelong distillation includes: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; wherein, the construction of the basic encoding and decoding model structure includes: adopting a sequence-to-sequence generation framework based on Transformer to construct an embedding layer, an encoder, a decoder, and an output layer; The construction of a teacher-student architecture based on lifelong distillation includes: introducing the old model as a teacher to help distill the previous level of conversation knowledge when building the model, using the model Mm-1 trained at the previous moment as the teacher of the model Mm trained at the current moment m, and at the next moment m+1, the current model Mm serves as the teacher to guide the training of the model Mm+1, and so on for sequential distillation; The objective function is constructed by preserving some of the knowledge learned in previous training processes during the current training process, enabling the model to learn the characteristics of different rounds of dialogue, and improving the accuracy and diversity of the model. In addition to the loss between the hard label and the predicted distribution for each example, the model trained in the previous process is introduced as a teacher model, and the soft labels output by the model are used to guide the current model in making predictions. Step 3: Load the open-domain dialogue generation model based on lifelong distillation that has the best evaluation metrics on the validation set, test it in batches using test samples, and output and save the dialogue generation results.

2. The method for generating open-domain dialogue based on lifelong distillation according to claim 1, characterized in that The original data in step 1 contains multi-round dialogue data, and each dialogue in the training set is processed into multiple dialogues with no more than three historical utterances.

3. The method for generating open-domain dialogue based on lifelong distillation according to claim 1, characterized in that After building the open-domain dialogue generation model based on lifelong distillation in step 2, randomly input training and validation samples into the model in batches, and use the Adam optimizer to gradually learn the optimal parameter values. Calculate the evaluation indicators on the validation set. Stop training when the indicators on the validation set no longer increase or the model training iterations reach the predetermined number. Save the best performing relationship classification model on the validation set.

4. The method for generating open-domain dialogue based on lifelong distillation according to claim 3, characterized in that The specific steps of step 3 using a batch of test samples for testing are as follows: load the open-domain dialogue generation model based on lifelong distillation trained in step 2, and then input the sample batches in the test set into the model. At this time, the model parameters are fixed and unchanged. After the model calculation, the dialogue response is generated.

5. A device for generating open-domain dialogue based on lifelong distillation, wherein the device is used to implement the method for generating open-domain dialogue based on lifelong distillation according to claim 1, characterized in that: include: The first main module is used to implement step 1: preprocess the raw data to obtain a validation set, a test set, and a training set containing no more than three historical utterances. The second main module is used to implement step 2: constructing an open-domain dialogue generation model based on lifelong distillation, randomly inputting training samples and verification samples into the open-domain dialogue generation model based on lifelong distillation in batches, starting an iterative model training process, obtaining evaluation indicators on the training set and verification set, stopping when the indicator on the verification set no longer increases or after a predetermined number of iterations, and saving the optimal model on the verification set. The construction of the open-domain dialogue generation model based on lifelong distillation includes: constructing a basic encoding and decoding model structure, constructing a teacher-student architecture based on lifelong distillation, and constructing an objective function; The third main module is used to implement step 3: load the open-domain dialogue generation model based on lifelong distillation with the best evaluation indicators on the validation set, test it with batches of test samples, and output and save the dialogue generation results.

6. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 4.

Images