Dialogue model training method, device, equipment and medium

Abstract

The present disclosure relates to a method, apparatus, device and medium for training a dialogue model. The system response of the previous round and the user statement of the current round are encoded by an encoder to obtain a continuous first hidden state vector, and the first hidden state vector is passed through a first policy network or a second policy network to obtain a first probability distribution of a hidden action vector. According to the first probability distribution of the hidden action vector, the first value network is optimized to obtain an optimized first value network; the first policy network and the second value network are optimized according to the optimized first value network to obtain optimized first policy network and second value network; the second policy network is optimized according to the optimized first policy network to obtain an optimized second policy network. Thus, the training process of the dialogue model is realized. The probability distribution of the system response given by the trained dialogue model is made consistent with the data distribution in the fixed data set, thereby avoiding the occurrence of extrapolation errors.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a method, apparatus, device, and medium for training a dialogue model. Background Art

[0002] With the continuous development of technology, the functions of terminal devices are becoming increasingly diverse. For example, terminal devices can conduct human-computer dialogue through dialogue systems. Specifically, after the user enters a question into the terminal device, the terminal device will provide the user with a system response through the dialogue system.

[0003] Typically, sample data such as Renren logs or unlabeled Renren logs can be used to optimize the dialogue strategy used by the dialogue system.

[0004] However, the inventors of the present application have discovered that directly adopting the unlabeled person-to-person log optimization conversation strategy may lead to extrapolation errors. Summary of the Invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, equipment and medium for training a dialogue model to avoid the occurrence of extrapolation errors.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for training a dialogue model, wherein the dialogue model includes at least: an encoder, a decoder, a first policy network, a second policy network, a first value network, and a second value network; the method includes:

[0007] Obtaining current round conversation data, where the current round conversation data includes the system response from the previous round and the user's statement from the current round;

[0008] Encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector;

[0009] Passing the first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector;

[0010] According to the first probability distribution of the latent action vector, the first value network is optimized to obtain an optimized first value network; according to the optimized first value network, the first strategy network is optimized to obtain an optimized first strategy network; according to the optimized first value network, the second value network is optimized to obtain an optimized second value network; according to the optimized first strategy network, the second strategy network is optimized to obtain an optimized second strategy network.

[0011] In a second aspect, an embodiment of the present disclosure provides a training device for a dialogue model, wherein the dialogue model includes at least: an encoder, a decoder, a first policy network, a second policy network, a first value network, and a second value network; the device includes:

[0012] An acquisition module is used to obtain the current round of dialogue data, which includes the system response of the previous round and the user statement of the current round;

[0013] An encoding module, configured to encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector;

[0014] a processing module, configured to pass the continuous first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector;

[0015] The optimization module is used to optimize the first value network according to the first probability distribution of the latent action vector to obtain an optimized first value network; optimize the first strategy network according to the optimized first value network to obtain an optimized first strategy network; optimize the second value network according to the optimized first value network to obtain an optimized second value network; optimize the second strategy network according to the optimized first strategy network to obtain an optimized second strategy network.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0017] Memory;

[0018] processor; and

[0019] computer programs;

[0020] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0022] The conversation model training method, apparatus, device, and medium provided by the disclosed embodiments encode the system responses from the previous round and the user statements from the current round using an encoder to obtain a continuous first latent state vector. Furthermore, the first latent state vector is passed through a first policy network or a second policy network to obtain a first probability distribution of latent action vectors. Based on the first probability distribution of latent action vectors, the first value network is optimized to obtain an optimized first value network; the first policy network is optimized based on the optimized first value network to obtain an optimized first policy network; the second value network is optimized based on the optimized first value network to obtain an optimized second value network; and the second policy network is optimized based on the optimized first policy network to obtain an optimized second policy network. This completes the conversation model training process. The probability distribution of system responses generated by the trained conversation model is consistent with the data distribution in a fixed dataset, thereby avoiding extrapolation errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of the structure of a generative model provided by the prior art;

[0026] Figure 2 A schematic diagram of the structure of another generative model provided by the prior art;

[0027] Figure 3 A schematic diagram of the structure of the dialogue model provided in the embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram of conversation data provided by an embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0030] Figure 6 A flowchart of a method for training a dialogue model according to an embodiment of the present disclosure;

[0031] Figure 7 A flowchart of a method for training a dialogue model provided in another embodiment of the present disclosure;

[0032] Figure 8 A flowchart of a method for training a dialogue model provided in another embodiment of the present disclosure;

[0033] Figure 9 A flowchart of a method for training a dialogue model provided in another embodiment of the present disclosure;

[0034] Figure 10 A flowchart of a method for training a dialogue model provided in another embodiment;

[0035] Figure 11 A flowchart of a method for training a dialogue model provided in another embodiment;

[0036] Figure 12 A schematic diagram of the structure of a training device for a dialogue model provided in an embodiment of the present disclosure;

[0037] Figure 13 A schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0040] In the prior art, standard reinforcement learning algorithms, such as policy gradient and reinforcement learning (Q-learning), are commonly used to optimize dialogue strategies. Optimizing dialogue strategies can be understood as training dialogue models. The dialogue model can be a generative model, such as a long short-term memory network (LSTM). The structural diagram of the generative model is shown in the figure below. Figure 1As shown, the generative model includes an encoder and a decoder. This generative model is a sequence-to-sequence generative model. Its task is to generate a system response given a conversation history. For example, the generative model can generate a first system response based on a first conversation history, where the first system response includes words w1, w2, and w3. The generative model can also generate a second system response based on a second conversation history, where the second system response includes words w4, w5, and w6. The generative model can undergo multiple rounds of iterative training. The first conversation history can include user statements from the first round, and the first system response can be the output of the decoder in the first round. The second conversation history can include user statements from the second round, and the second system response can be the output of the decoder in the second round. For example, the encoder encodes the first conversation history and then decodes it through a decoder to generate the first system response. The system response can be generated word by word, and the system response can be a text sequence consisting of each word. The generative model is optimized using a reinforcement learning algorithm within a vocabulary space. The vocabulary space can be a preset dictionary. The vocabulary space can also be referred to as a vocabulary set. Specifically, the method of optimizing the generative model using the reinforcement learning algorithm can be: using the vocabulary set as the action space, the latent vector of the generative model as the state, and the semantic similarity score between the generated system response and the correct text, such as the Bilingual Evaluation Understudy (BLEU) score, as the feedback function, and optimizing the generative model using the standard reinforcement learning algorithm. However, the vocabulary space is large, so in order to simplify the dialogue strategy optimization, Figure 1 The generative model shown is modified to Figure 2 The generative model shown.

[0041] like Figure 2 As shown, the action space is no longer the entire vocabulary, but rather a separately constructed latent random variable. The feedback function directly uses the dialogue completion rate. This approach has the advantage that the space of latent random variables can be customized compared to the size of the vocabulary. Furthermore, the latent random variable is much smaller than the vocabulary, making dialogue policy optimization simpler.

[0042] However, if Figure 1 and Figure 2A common problem with the methods shown is that task-oriented dialogue systems based on reinforcement learning algorithms require not only labeled dialogue data but also an additional user simulator to simulate users and enable interaction between the user and the dialogue system. In many practical task-oriented dialogue scenarios, a large amount of unlabeled human-to-human logs often exist. Unlabeled human-to-human logs refer to unlabeled human-to-human logs, which are real conversations between humans. Human-to-human logs can be textual conversations or textual conversations transcribed from speech. However, labeling unlabeled human-to-human logs consumes significant manpower and resources. Furthermore, because the correspondence between states and actions in existing unlabeled human-to-human logs is fixed, directly using unlabeled human-to-human logs to optimize dialogue policies can lead to extrapolation errors. Extrapolation errors can be understood as the extrapolation error caused by the fact that, when the training data is fixed and no new training data is added (i.e., in a batch reinforcement learning setting), reinforcement learning algorithms face extrapolation errors. This refers to the inconsistency between the probability distribution given by the dialogue policy optimized by the reinforcement learning algorithm and the data distribution in the fixed dataset, causing conventional reinforcement learning algorithms to fail.

[0043] To address this issue, embodiments of the present disclosure provide a method for training a conversation model using a batch reinforcement learning algorithm. Batch reinforcement learning is an offline reinforcement learning method used to train conversation models when the conversation model is unable to interact with users. This method utilizes existing fixed datasets, such as unlabeled person logs, to train the conversation model. Specifically, embodiments of the present disclosure train the conversation model using the Bootstrapping Error Accumulation Reduction Q-Learning (BEAR-QL) algorithm, a technique used in the field of batch reinforcement learning.

[0044] The dialogue model provided by the embodiment of the present disclosure includes: a first policy network, a second policy network, a first value network, and a second value network. The first policy network can be Figure 3 The original policy network shown, the second policy network can be as follows Figure 3 The target strategy network shown in the figure, the first value network can be as follows Figure 3 The original value network shown, the second value network can be as follows Figure 3 The target value network shown in . In addition, Figure 3As shown, the dialogue model also includes a first encoder, a second encoder, and a decoder. The first value network can be one or more. The second value network can also be one or more. During the iterative training of the dialogue model, the process shown at 31 represents the previous training round, and the process shown at 32 represents the current training round. During the use or prediction phase of the dialogue model, the process shown at 31 represents the previous use or prediction phase, and the process shown at 32 represents the current use or prediction phase.

[0045] For example, taking the training phase of the dialogue model as an example, Figure 3 The process shown in 31 is the training process of the t-1th round, and the process shown in 32 is the training process of the tth round. In the tth round of training, the input of the dialogue model is sys t-1 and usr t ,usr t Indicates the user statement used in the current round. For example, several conversation data in the Wubiaorenren log are as follows Figure 4 As shown. Figure 3 The usr shown t-1 It can be Figure 4 AAA,sys shown t-1 It can be Figure 4 The BBB number that the customer service replied to in the Wubiao Renren log, or sys t-1 It can be that during the t-1 round of training, the dialogue model is based on usr t-1 and sys t-2 Generated system reply. sys t-2 The meaning is similar to sys t-1 The meaning of, for example, sys t-2 It can be the system response generated by the dialogue model during the t-2 round of training, or sys t-2 It can be Figure 4 AAA's previous customer service response as shown. t It can be Figure 4 Specifically, during the training process of the tth round, the first encoder in the dialogue model can be based on sys t-1 and usr t Generate a sentence vector. This sentence vector can be input into the second encoder, which can obtain a continuous hidden state vector based on the sentence vector. The hidden state vector is recorded as s t The Caintic state vector s t After processing by a policy network such as the second policy network or the first policy network, the latent action vector a can be obtained. t The probability distribution of a hidden state vector s t Can correspond to multiple latent action vectors at . Further, according to the latent action vector a t The probability distribution of one or more hidden action vectors a t The one or more latent action vectors a t After processing by the decoder, the system response sys for round t can be obtained t The first encoder can be specifically an LSTM encoder. The second encoder can be specifically a context encoder. The LSTM encoder, context encoder, and decoder can each be an LSTM, with the difference being that the LSTM encoder operates at a turn level, while the context encoder operates at a dialog level. Both the LSTM encoder and the context encoder use LSTM to encode a sequence into a vector. The decoder is used for decoding, for example, to decode a vector into a sequence.

[0046] Specifically, the dialogue model trained using the batch reinforcement learning algorithm described in this embodiment may include several sub-models, such as a context encoder, a value network, a policy network, an LSTM encoder, and a decoder. The context encoder can be denoted as f ctt (·), the value network can be recorded as Q θ (s t ,a t ), the policy network can be recorded as π φ (a t |s t ), the LSTM encoder can be recorded as f en (·), the decoder can be recorded as f de (·). Both the policy network and the value network are fully connected networks, used to optimize the reinforcement learning algorithm. The difference between the policy network and the value network is that the output dimension of the policy network is greater than 1, for example, it is generally 100. The output dimension of the value network is 1, which represents a specific value. Figure 3 The dialogue model shown can include multiple value networks, such as multiple second value networks and multiple first value networks, thereby accelerating the convergence of the algorithm.

[0047] Specifically, the training phase of the dialogue model can be offline, and the use phase of the dialogue model can be online. In the use phase of the dialogue model, sys t-1 and sys t-2 They are the system responses generated by the dialogue model during the online dialogue between the dialogue model and the user. t-1 and usr tIt can be what the user said online. Given the conversation history, the conversation model can generate a system response. The conversation history can be the conversation history between the user and the conversation model. For example, Figure 4 The dialogue between the user and the system above the dotted line is the dialogue history, and the DDD below the dotted line can be the system response. Figure 3 Among the first strategy network, the second strategy network, the first value network, and the second value network shown, only the second strategy network participates in the calculation.

[0048] After the dialogue model training is completed, the dialogue model can be used in the application scenario of human-computer dialogue. Figure 5 As shown, the application scenario includes a server 50 and a terminal, and the terminal is not limited to Figure 5 The smartphone 51, desktop 52, and smart speaker 53 are shown. In addition, the terminal may also include a PDA, a tablet computer, a wearable device with a display screen, a laptop computer, an all-in-one computer, a smart home device, etc. The smartphone 51, the desktop 52, and the smart speaker 53 are used as examples for schematic illustration. The server 50 may be a cloud server or a server cluster. Specifically, the server 50 may include a trained dialogue model, and the server 50 may conduct a human-computer dialogue with the user of the terminal through the dialogue model. For example, the user may input a user question to the terminal through human-computer interaction. Furthermore, the terminal may send the user question to the server 50, and the server 50 may generate a system response based on the dialogue model and feed the system response back to the user's terminal.

[0049] It is understandable that this embodiment does not limit the human-computer interaction method between the user and the terminal. For example, the user can enter a text-type question in the browser installed on the smartphone 51 or the desktop computer 52. Alternatively, the user can interact with the smartphone 51 or the smart speaker 53 through voice interaction. The smartphone 51 or the smart speaker 53 can collect the user's voice questions and send the voice questions to the server 50, and the server 50 converts the voice questions into text-type questions. Alternatively, when the smartphone 51 or the smart speaker 53 collects the voice questions, it can convert the voice questions into text-type questions and send the text-type questions to the server 50. The server 50 uses the text-type question as the input of the dialogue model, and the dialogue model generates a system response. The server 50 further feeds back the system response to the user's terminal.

[0050] It is understandable that, in some embodiments, the trained dialogue model can also be deployed in the terminal, so that the terminal can generate a system response based on the user statement or user question.

[0051] The following combination Figure 3The training phase of the dialogue model is schematically illustrated, and the method of human-computer dialogue is introduced in the subsequent embodiments in conjunction with the trained dialogue model.

[0052] During the training phase, Figure 3 The first policy network, second policy network, first value network, and second value network shown all participate in the calculation. For example, the first policy network assists in optimizing the second policy network, the first value network assists in optimizing the second value network, and the first value network also assists in optimizing the first policy network. The sample data used to train the conversation model can be an unlabeled human-to-human conversation dataset, denoted as D. This unlabeled human-to-human conversation dataset can be, for example, the unlabeled human-to-human logs described above, or a dataset composed of unlabeled human-to-human logs.

[0053] The dialogue model in the initialization state may include: a first encoder (such as an LSTM encoder), a second encoder (such as a context encoder), a decoder, multiple first value networks, a first policy network, multiple second value networks, and a second policy network. Among them, the multiple first value networks correspond to the multiple second value networks one by one, and one first policy network corresponds to one second policy network. The multiple first value networks can be recorded as A first policy network can be denoted as π φ , multiple second value networks can be recorded as A second policy network can be denoted as π φ′ The initial parameters of the second policy network are consistent with the initial parameters of the first policy network, and the initial parameters of the second value network are consistent with the initial parameters of the first value network.

[0054] Figure 6 Flowchart of the training method of the dialogue model provided in the embodiment of the present disclosure. Figure 6 As shown, the specific steps of this method are as follows:

[0055] S601: Obtain current round dialogue data, where the current round dialogue data includes the system response from the previous round and the user statements from the current round.

[0056] For example, obtain the current round of dialogue data from the unlabeled human dialogue dataset. The current round of dialogue data includes sys t-1 and usr t ,sys t-1 Indicates the system reply of the previous round, usr t Indicates the user statement of this round. For example, sys t-1 It can be Figure 4 The customer service response shown is BBB, usr t It can be Figure 4The user statement CCC is shown. For example, Figure 4 The sys shown t-2 and usr t-1 Can be used in the t-1 round of training of the dialogue model, sys t-1 and usr t It can be used in the tth round of training of the dialogue model. t The first statement of the user can be the user statement taken by the dialogue model in the tth round of training. t-1 It can be recorded as the historical reply statement corresponding to the user's first statement.

[0057] S602: Encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector.

[0058] For example, sys t-1 and usr t As the input of the dialogue model, sys t-1 and usr t After the encoding process of the first encoder and the second encoder in the dialogue model, the continuous hidden state vector s is obtained t , Cain state vector s t Denoted as the first hidden state vector.

[0059] S603: Pass the first hidden state vector through a first policy network or a second policy network to obtain a first probability distribution of a hidden action vector.

[0060] For example, the first hidden state vector is input into the first policy network or the second policy network. The second policy network or the first policy network can obtain the hidden action vector a according to the first hidden state vector t The first probability distribution of .

[0061] S604. Optimize the first value network according to the first probability distribution of the latent action vector to obtain an optimized first value network; optimize the first strategy network according to the optimized first value network to obtain an optimized first strategy network; optimize the second value network according to the optimized first value network to obtain an optimized second value network; optimize the second strategy network according to the optimized first strategy network to obtain an optimized second strategy network.

[0062] For example, according to the latent action vector a t The first probability distribution is used to optimize one or more first value networks.

[0063] Optionally, the one or more first value networks are optimized according to the first probability distribution of the latent action vector, including: calculating evaluation indicators corresponding to one or more latent action vectors in the first probability distribution through the one or more second value networks; and optimizing the one or more first value networks according to the evaluation indicators.

[0064] For example, the second policy network can be used to calculate the latent action vector a t The first probability distribution is sampled to obtain p latent action vectors, which can also be recorded as p latent action variables. The p latent action variables can be recorded as Among them, s′ represents the t-1 and usr t The first hidden state vector obtained. Further, one or more second value networks can be used to calculate the evaluation index of the p hidden action variables, and the evaluation index can specifically be a Q value. Thus, multiple first value networks are optimized according to the Q values ​​corresponding to the p hidden action variables to obtain multiple optimized first value networks. It can be understood that when there are 1 first value network and 1 second value network respectively, the second value network can calculate the Q values ​​corresponding to the p hidden action variables, and further, the first value network is optimized according to the Q values ​​corresponding to the p hidden action variables to obtain an optimized first value network.

[0065] Further, the first strategy network is optimized according to the optimized first value network to obtain the optimized first strategy network; the second value network is optimized according to the optimized first value network to obtain the optimized second value network; the second strategy network is optimized according to the optimized first strategy network to obtain the optimized second strategy network.

[0066] For example, the network parameters of the second strategy network can be optimized according to the network parameters of the optimized first strategy network, and the network parameters of the second value network corresponding to each optimized first value network can be optimized according to the network parameters of the first value network.

[0067] It is understood that the process from S601 to S604 can be a training process of the dialogue model. The dialogue model can undergo multiple rounds of iterative training. When the network parameters of the second policy network and the network parameters of each second value network meet preset conditions, the training of the dialogue model can be stopped, thereby obtaining a trained dialogue model.

[0068] In the disclosed embodiment, an encoder encodes the system responses from the previous round and the user statements from the current round to obtain a continuous first hidden state vector. Furthermore, the first hidden state vector is passed through the first policy network or the second policy network to obtain a first probability distribution of the latent action vector. Based on the first probability distribution of the latent action vector, the first value network is optimized to obtain an optimized first value network; the first policy network is optimized based on the optimized first value network to obtain an optimized first policy network; the second value network is optimized based on the optimized first value network to obtain an optimized second value network; and the second policy network is optimized based on the optimized first policy network to obtain an optimized second policy network. This completes the training process for the dialogue model. The probability distribution of the system responses given by the trained dialogue model is consistent with the data distribution in the fixed dataset, thereby avoiding extrapolation errors.

[0069] In summary, this embodiment proposes an end-to-end model based on batch reinforcement learning, namely a dialogue model. Without using any dialogue state labels or dialogue action labels, the dialogue model can be directly trained using unlabeled human logs. Specifically, the dialogue model can be trained using the BEAR-QL algorithm to solve the problem of extrapolation errors.

[0070] On the basis of the above embodiment, the first value network is optimized according to the first probability distribution of the latent action vector, including: Figure 7 The following steps are shown:

[0071] S701: Calculate evaluation indicators corresponding to one or more latent action vectors in the first probability distribution through the second value network.

[0072] For example, the second policy network can be used to calculate the latent action vector a t The first probability distribution is sampled to obtain p latent action vectors, which can also be recorded as p latent action variables. The p latent action variables can be recorded as Among them, s′ represents the t-1 and usr t The obtained hidden state vector s t That is the first hidden state vector. The second strategy network is responsible for the hidden action vector a t The specific process of sampling the first probability distribution of can be: modifying the network parameters of the second strategy network or the first strategy network so that the hidden action vector a output by the second strategy network or the first strategy network is t The first probability distribution of the second strategy network can collect one or more hidden action vectors a from the first probability distribution before the change. t, and collecting one or more latent action vectors a from the changed first probability distribution t Thus, the p latent action vectors are obtained. That is, the sampling of the p latent action vectors is achieved by a method similar to the dropout method in the training process of a deep learning network, in which neural network units are temporarily discarded from the network according to a certain probability.

[0073] When p latent action vectors are collected, evaluation indicators of the p latent action vectors can be calculated through one or more second value networks. The evaluation indicator can be specifically a Q value, that is, the expected benefit of taking a certain action under a given state at a certain moment. For example, in this embodiment, K second value networks are used as an example. The first latent state vector calculated by the jth second value network in the K second value networks and the i-th latent action vector a in the p latent action vectors are calculated as i The Q value can be expressed as Where s′ represents the first hidden state vector. Since the first hidden state vector corresponds to p hidden action vectors, the Q value corresponding to each of the p hidden action vectors can be the Q value of the first hidden state vector and a certain hidden action vector calculated when the first hidden state vector is fixed.

[0074] S702. Optimize the first value network based on the evaluation metrics and feedback function, where the feedback function includes a conversation completion rate and a semantic similarity between a system response generated in this round and a next response statement of a user statement in this round in an unlabeled human-to-human conversation dataset.

[0075] Furthermore, according to the Q value calculated by each second value network in the K second value networks for the p hidden action vectors, y(s,a) is obtained. y(s,a) can be expressed as the following formula (1):

[0076]

[0077] in, The calculation process can be to fix i first, and then traverse j so that the value of j is from 1 to K. After j is from 1 to K, the minimum value can be obtained. and the biggest one Further, according to the weight coefficient λ, the smallest and the biggest one Perform weighted summation. After the weighted summation obtains the sum value, update i so that i=i+1. Traverse j again and repeat the subsequent calculations. Since the value of i in formula (1) is from 1 to p, the sum value obtained by p weighted summations can be calculated. y(s,a) represents the maximum sum value among the sum values ​​obtained by the p weighted summations. The s in y(s,a) represents the first hidden state vector. The a in y(s,a) represents the a corresponding to the maximum sum value obtained by the weighted summation. i .

[0078] Furthermore, the one or more first value networks are optimized based on the evaluation indicators and feedback function, where the feedback function includes the conversation completion rate and the semantic similarity between the system response generated in this round and the next response statement of the user statement in this round in the unlabeled human-to-human conversation dataset.

[0079] For example, K second value networks may correspond to K first value networks, and the K first value networks may be optimized according to the following formula (2):

[0080]

[0081] Wherein, i in formula (2) represents the i-th first value network among the K first value networks. i represents the network parameters of the i-th first value network. Represents the Q value output by the i-th first value network based on s and a. The s in y(s,a) has the same meaning as the s in y(s,a). The a in y(s,a) has the same meaning as that in y(s,a), and will not be repeated here. γ represents a parameter. r represents a feedback function, which includes the dialogue completion rate and BLEU, wherein the BLEU is used to represent the semantic similarity between the system response generated in this round and the next response statement of the user statement in this round in the unlabeled human dialogue dataset. The system response generated in this round can be the system response obtained by the dialogue model based on the user's first statement and the historical response statement corresponding to the user's first statement. For example Figure 4 The user statement CCC is the user's first statement, and the customer service reply BBB is the historical reply statement corresponding to the user's first statement. During the dialogue model training process, the dialogue model can output the system reply generated in this round based on CCC and BBB. In addition, according to Figure 4It can be seen that in the unlabeled human-to-human dialogue dataset, the next reply statement of the user sentence CCC is, for example, DDD replied by the customer service. DDD replied by the customer service may be the correct answer. Therefore, the system reply generated by the dialogue model in this round based on the output of CCC and BBB may have a certain degree of similarity with the correct answer. In this embodiment, the semantic similarity between the system reply and the correct answer can be represented by BLEU.

[0082] This embodiment uses a more complex feedback function, such as one that includes the conversation completion rate and the semantic similarity between the system response generated by the conversation model based on the user's first statement and the historical responses to the user's first statement, and the next response to the user's first statement in the unlabeled human-to-human conversation dataset. This allows the decoder in the conversation model to be optimized, thereby resolving the issue of poor decoding performance.

[0083] Based on the above embodiment, the method further includes: Figure 8 The following steps are shown:

[0084] S801: Acquire first historical conversation data, where the first historical conversation data includes a first historical sentence and a historical reply sentence corresponding to the first historical sentence.

[0085] For example, first historical conversation data can be obtained from an unlabeled human-to-human conversation dataset. The first historical conversation data includes a first historical sentence and a historical reply sentence corresponding to the first historical sentence. The first historical sentence can be recorded as the user's second sentence, which can be the user's first sentence. Alternatively, the user's second sentence can be another user sentence in addition to the user's first sentence.

[0086] For example, the second user statement can be a user statement randomly collected from an unlabeled human-to-human conversation dataset, and the second user statement can be one or more. Figure 4 The FFF shown is a second statement of the user. The historical reply statement corresponding to the second statement of the user FFF is GGG.

[0087] S802: Acquire second historical conversation data, where the second historical conversation data includes a second historical sentence and a historical reply sentence corresponding to the second historical sentence, and a semantic similarity between the second historical sentence and the first historical sentence is greater than or equal to a preset threshold.

[0088] Alternatively, a second historical conversation data set may be obtained from an unlabeled human-to-human conversation dataset. The second historical conversation data may include a second historical sentence and a historical reply sentence corresponding to the second historical sentence. The semantic similarity between the second historical sentence and the first historical sentence is greater than or equal to a preset threshold. In this embodiment, the second historical sentence may be recorded as a target sentence similar to the first historical sentence.

[0089] For example, based on the user's second statement, n target statements similar to the user's second statement can be determined from the unlabeled human-to-human conversation dataset D, where n is greater than or equal to 1, and the semantic similarity between each of the n target statements and the user's second statement is greater than or equal to a preset threshold. It is understandable that each target statement can correspond to a historical reply statement. For example, Figure 4 The FFF shown is the user's second sentence, HHH is the target sentence similar to the user's second sentence, and III is the historical reply sentence corresponding to the target sentence. It is understandable that there may be multiple target sentences similar to the user's second sentence, and one is used here as an example for schematic illustration. For example, based on FFF, multiple target sentences can be determined from an unlabeled human-to-human conversation dataset, and the semantic similarity between each target sentence and FFF is greater than or equal to a preset threshold. Specifically, the semantic similarity between the user's second sentence and a target sentence can be the similarity between the sentence vector of the user's second sentence calculated using tools such as bert-as-service and the sentence vector of the target sentence.

[0090] S803: Encode the second historical conversation data using an encoder to obtain a continuous second hidden state vector, and pass the second hidden state vector through a second policy network to obtain a second probability distribution of the hidden action vector.

[0091] For example Figure 4 As shown, the user's second sentence FFF corresponds to n target sentences, and each of these n target sentences can correspond to a historical response sentence. Therefore, a target sentence and its corresponding historical response sentence can be recorded as a second historical conversation data. Furthermore, the second historical conversation data is used as input to the conversation model and encoded using the encoder in the conversation model to obtain a continuous second hidden state vector. The encoder in the conversation model includes a first encoder and a second encoder. It can be understood that n target sentences correspond to n second historical conversation data, and therefore, n second hidden state vectors can be obtained based on the n second historical conversation data.

[0092] Specifically, each of the n second hidden state vectors can be used as an input of the second policy network, and the second policy network can output a second probability distribution of the latent action vector based on each second hidden state vector, thereby obtaining n second probability distributions of the latent action vector, that is, one second hidden state vector can correspond to one second probability distribution, and the n second hidden state vectors and the n second probability distributions have a one-to-one correspondence.

[0093] For example, the target sentence HHH and the historical response sentence III corresponding to the target sentence are used as the input of the dialogue model. After processing by the first encoder and the second encoder, a second hidden state vector can be obtained. The second hidden state vector is input to the second policy network, which can obtain a second probability distribution of the hidden action vector based on the second hidden state vector.

[0094] S804: Encode the first historical conversation data using an encoder to obtain a continuous third hidden state vector, and pass the third hidden state vector through a first policy network to obtain a third probability distribution of the hidden action vector.

[0095] For example, if the user's second sentence FFF corresponds to the historical reply sentence GGG, FFF and GGG are used as the input of the dialogue model. After processing by the first encoder and the second encoder, a third hidden state vector can be obtained. The third hidden state vector is input into the first policy network, which can then obtain a third probability distribution of the hidden action vector based on the third hidden state vector.

[0096] S805: If the difference between the second probability distribution and the third probability distribution is less than a preset difference, obtain third historical dialogue data, where the third historical dialogue data includes a third historical sentence and a historical reply sentence corresponding to the third historical sentence.

[0097] For example, the difference between the second probability distribution and the third probability distribution is calculated. When the difference is less than the preset difference, the third historical conversation data can continue to be collected from the unlabeled human-to-human conversation dataset D. The third historical conversation data includes a third historical sentence and a historical reply sentence corresponding to the third historical sentence. Among them, the third historical sentence can be recorded as the user's third sentence. Similarly, the user's third sentence can be one or more. Since the data volume of the unlabeled human-to-human conversation dataset D is large, and the user's third sentence and the user's second sentence may be collected at different times, the user's third sentence may be the same as the user's second sentence as described above, or it may be different from the user's second sentence. However, the distribution of the user's third sentence in the unlabeled human-to-human conversation dataset D is the same as the distribution of the user's second sentence in the unlabeled human-to-human conversation dataset D.

[0098] Optionally, before S805, the following steps may also be included: Figure 9 The following steps are shown:

[0099] S901: Obtain a latent action vector with the maximum probability from each of the plurality of second probability distributions to obtain a first set.

[0100] For example, when n second probability distributions are obtained, a latent action vector with the largest probability can be collected from each second probability distribution, thereby obtaining n latent action vectors. The n latent action vectors can also be recorded as n latent action variables, and the n latent action variables can be recorded as Here, s represents a third hidden state vector obtained based on the user's second statement and the historical reply statements corresponding to the user's second statement. The n hidden action variables can constitute a first set. It can be understood as a function whose input is a user's second sentence randomly collected from the unlabeled human-to-human dialogue dataset D, and whose output is n latent action variables.

[0101] S902: Acquire multiple latent action vectors from the third probability distribution to obtain a second set.

[0102] For example, the historical reply sentence corresponding to the user's second sentence FFF is GGG. FFF and GGG are used as the input of the dialogue model. After processing by the first encoder and the second encoder, a third latent state vector can be obtained. The first policy network can obtain a third probability distribution of the latent action vector based on the third latent state vector. Furthermore, m latent action vectors can be collected from the third probability distribution. The m latent action vectors can be recorded as m latent action variables, and the m latent action variables can be recorded as Here, s represents the third hidden state vector obtained based on the user's second statement and the historical reply statement corresponding to the user's second statement. In addition, the m hidden action vectors can constitute a second set. φ (·|s) can be understood as a function whose input is a user's second sentence randomly collected from the unlabeled human-to-human dialogue dataset D, and whose output is m latent action variables.

[0103] S903: Calculate a difference between the second probability distribution and the third probability distribution according to the multiple latent motion vectors in the first set and the multiple latent motion vectors in the second set.

[0104] For example, are the sample values ​​from the n second probability distributions, is the sampling value in the third probability distribution. The n second probability distributions can represent the distribution of n target sentences in the unlabeled human-to-human dialogue dataset. The distribution of the n target sentences in the unlabeled human-to-human dialogue dataset can be recorded as The third probability distribution can be recorded as π φ (·|s). Further, according to and It can be calculated and π φ The difference between (·|s) can be calculated based on the Maximum Mean Discrepancy (MMD). For example, the difference can be expressed as The calculation formula of MMD is as follows (3):

[0105]

[0106] Among them, {x1,...,x n}Specifically {y1,…,y m}Specifically k(,) represents the Gaussian kernel.

[0107] Optionally, the first strategy network is optimized according to the optimized first value network to obtain the optimized first strategy network, including: Figure 10 The following steps are shown:

[0108] S1001. Encode the third historical conversation data using an encoder to obtain a continuous fourth hidden state vector, and pass the fourth hidden state vector through a first policy network to obtain a fourth probability distribution of the hidden action vector.

[0109] For example, in and π φ If the difference between (·|s) is less than or equal to a preset difference, the user's third statement and the corresponding historical replies can be collected from the unlabeled human-to-human conversation dataset D. Furthermore, the user's third statement and the corresponding historical replies are used as input to the conversation model. The encoder then encodes the user's third statement and the corresponding historical replies to obtain a fourth latent state vector. Furthermore, the fourth latent state vector is input to the first policy network, which can then derive a fourth probability distribution of the latent action vector based on the fourth latent state vector.

[0110] S1002. Optimize the first policy network according to the fourth hidden state vector, the fourth probability distribution, and the optimized first value network to obtain an optimized first policy network.

[0111] Furthermore, the first policy network is optimized according to the fourth hidden state vector, the fourth probability distribution of the hidden action vector, and the optimized one or more first value networks, thereby obtaining an optimized first policy network.

[0112] Optionally, optimizing the first policy network according to the fourth hidden state vector, the fourth probability distribution and the optimized one or more first value networks includes optimizing the first policy network according to the fourth hidden state vector, one or more hidden action vectors in the fourth probability distribution and the optimized one or more first value networks.

[0113] The specific optimization process of the first strategy network can be expressed as the following formula (4):

[0114]

[0115] in, Represents a constraint condition, where s in the constraint condition represents the third hidden state vector obtained based on the user's second statement and the historical reply statements corresponding to the user's second statement after randomly collecting a user's second statement from the unlabeled human-to-human conversation dataset D. Specifically, the user's second statement can be randomly collected multiple times from the unlabeled human-to-human conversation dataset D, and the user's second statement collected each time may be the same or different. According to It can be seen that every time a user's second statement is collected, a difference value can be calculated, that is, Therefore, when collecting the second statement of the user multiple times, multiple difference values ​​can be calculated. The constraint condition can be to calculate the average value of the multiple difference values ​​so that the average value is less than or equal to the threshold ε. Further, when the constraint condition is satisfied, Optimize the first policy network.

[0116] In addition, for formula (4), s in the equation represents the fourth latent state vector obtained by randomly sampling a user's third statement from the unlabeled human-to-human conversation dataset D and then basing it on the user's third statement and the historical responses to it. D is the unlabeled human-to-human conversation dataset. It is understandable that since the user's third statement is collected under the constraints, the user's second statement is collected before the user's third statement. Therefore, the user's third statement may be the same as or different from the user's second statement. Specifically, under the constraints, the user's third statement can be randomly sampled multiple times from the unlabeled human-to-human conversation dataset D. The third statements collected each time may be the same or different.

[0117] In addition, for formula (4), π(·|s) in the formula represents the hidden action vector a output by the first policy network according to the fourth hidden state vector given the fourth hidden state vector. t The fourth probability distribution of a:π(·|s) represents the hidden action vector a t Multiple latent action vectors a are collected from the fourth probability distribution t Since each user's third statement can correspond to a fourth latent state vector, multiple fourth latent state vectors can be obtained by randomly collecting user third statements multiple times from the unlabeled human-to-human conversation dataset D. S represents the set of the multiple fourth latent state vectors. π represents the fourth probability distribution of the latent action vector given any fourth latent state vector. π φ Represents the optimized first policy network. Indicates that given the fourth hidden state vector s, the fourth hidden state vector and a hidden action vector a calculated by the optimized j-th first value network t The corresponding Q value. t is the latent action vector a t Multiple latent action vectors a collected from the fourth probability distribution t Any one of .

[0118] This example optimizes a generative dialogue model with a hidden state vector using the classic BEAR-QL algorithm in batch reinforcement learning. This addresses the extrapolation errors often encountered in conventional reinforcement learning due to a fixed dialogue dataset. Furthermore, the feedback function includes not only the dialogue completion rate but also the sentence-level BLEU score, addressing the issue of poor decoder performance.

[0119] Based on the above embodiment, the method further includes: passing the first probability distribution of the latent action vector through the decoder to obtain a system response generated in this round.

[0120] In a feasible implementation, the first probability distribution of the latent action vector is passed through the decoder to obtain the system response generated in this round, including: determining the latent action vector with the highest probability in the first probability distribution; using the latent action vector with the highest probability in the first probability distribution as the input of the decoder, and obtaining the system response generated in this round through the decoder.

[0121] For example, the second policy network can select the hidden action vector a with the largest probability from the first probability distribution of hidden action vectors. t , and the hidden action vector a with the largest probability t Input to the decoder, that is, the hidden action vector a with the maximum probability t It can be used as the input of the decoder. Further, the decoder can use the hidden action vector a with the maximum probability to t Get the user statement usr of this round t The corresponding system reply sys t That is, the system response generated in this round.

[0122] In another feasible implementation, the first probability distribution of the latent action vector is passed through the decoder to obtain the system response generated in this round, including: collecting multiple latent action vectors with probabilities greater than or equal to a preset probability from the first probability distribution; using multiple latent action vectors with probabilities greater than or equal to a preset probability in the first probability distribution as inputs to the decoder, and obtaining multiple system responses corresponding to the user statements in this round through the decoder; and determining the system response generated in this round from the multiple system responses.

[0123] For example, the second policy network can collect multiple latent action vectors a from the first probability distribution of latent action vectors t , the multiple latent action vectors a t The hidden action vector a with the largest probability in the first probability distribution may be included. t , and the hidden action vector a with the highest probability t The surrounding hidden action vector a t . Further, the second strategy network can convert the multiple latent action vectors a t Input to the decoder, the decoder can be used according to the multiple latent action vectors a t Get multiple system responses, specifically, a hidden action vector a t It can correspond to a system response. The decoder can randomly select a system response from the multiple system responses as the system response sys generated in this round t .

[0124] It is understandable that the system responds to the dialogue model sys t In the case oft Feedback to the user's terminal device.

[0125] Based on the above embodiment, the second strategy network is optimized according to the optimized first strategy network, including: optimizing the network parameters of the second strategy network according to the network parameters of the optimized first strategy network; the one or more second value networks are optimized according to the optimized one or more first value networks, including: optimizing the network parameters of the second value network corresponding to the first value network according to the network parameters of each first value network in the optimized one or more first value networks.

[0126] For example, the network parameters of the second value network corresponding to each first value network can be optimized according to the optimized network parameters of the first value network using the following formula (5).

[0127] θ′ i ←τθ i +(1-τ)θ′ i (5)

[0128] Among them, θ i θ′ represents the network parameters of the optimized i-th first value network. i represents the network parameters of the second value network. τ represents a hyperparameter between (0, 1).

[0129] The network parameters of the second strategy network can be optimized according to the optimized network parameters of the first strategy network through the following formula (6).

[0130] φ′←τφ+(1-τ)φ′ (6)

[0131] Where φ represents the network parameters of the optimized first policy network. φ′ represents the network parameters of the second policy network. τ represents the hyperparameter between (0, 1).

[0132] When the dialogue model training is completed, this embodiment can also provide a dialogue processing method, which includes: Figure 11 The following steps are shown, where S1101-S1104 may be methods executed by the dialogue model during the use phase. Specifically, the dialogue model use phase and the dialogue model training phase may be executed on the same device or on different devices.

[0133] S1101. Obtain the user's current statement and the historical reply statements corresponding to the user's current statement.

[0134] For example, the use phase of the dialogue model can be online. Figure 3The usr shown t It can be the user's current statement, which can be as follows Figure 5 The user shown in FIG. 1 enters a question into the terminal through human-computer interaction. Figure 3 The sys shown t-1 It can be the historical reply sentence corresponding to the user's current sentence. The historical reply sentence is the system reply generated by the dialogue model in the previous round.

[0135] S1102. Generate a target hidden state vector based on the user's current sentence and the historical reply sentences corresponding to the user's current sentence.

[0136] For example, the user's current sentence and the historical reply sentences corresponding to the user's current sentence can be used as inputs of the dialogue model. The first encoder and the second encoder in the dialogue model can generate corresponding hidden state vectors, which are recorded as target hidden state vectors.

[0137] Optionally, generating a target hidden state vector based on the user's current sentence and the historical reply sentences corresponding to the user's current sentence includes: generating a sentence vector based on the user's current sentence and the historical reply sentences corresponding to the user's current sentence; generating the target hidden state vector based on the sentence vector.

[0138] For example, the first encoder can generate a sentence vector based on the user's current sentence and the historical reply sentences corresponding to the user's current sentence. The second encoder can generate a target hidden state vector based on the sentence vector. The target hidden state vector can be as follows: Figure 3 The s shown t .

[0139] S1103: Obtain a target probability distribution of the latent action vector according to the target latent state vector and the second policy network.

[0140] During the usage phase of the dialogue model, the second policy network participates in the calculation. Specifically, the target latent state vector can be used as the input of the second policy network, and the second policy network can output the target probability distribution of the latent action vector based on the target latent state vector.

[0141] S1104. Determine a target system response corresponding to the user's current statement based on the target probability distribution.

[0142] This embodiment optimizes a generative dialogue model with a hidden state vector using the classic BEAR-QL algorithm in batch reinforcement learning. This addresses the extrapolation errors often encountered in conventional reinforcement learning due to fixed dialogue datasets. Furthermore, the feedback function includes not only the dialogue completion rate at the dialogue level but also the sentence-level BLEU score, addressing the issue of poor decoder performance. This enables the trained dialogue model to generate accurate system responses, improving the accuracy of intelligent question answering.

[0143] Figure 12 This is a structural diagram of the training device for the dialogue model provided in the embodiment of the present disclosure. The training device for the dialogue model provided in the embodiment of the present disclosure can execute the processing flow provided in the embodiment of the training method for the dialogue model, wherein the dialogue model at least includes: an encoder, a decoder, a first policy network, a second policy network, a first value network, and a second value network. Figure 12 As shown, the dialogue model training device 120 includes:

[0144] An acquisition module 121 is configured to obtain current round conversation data, wherein the current round conversation data includes the system response from the previous round and the user statement from the current round;

[0145] An encoding module 122 is configured to encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector;

[0146] A processing module 123 is configured to pass the continuous first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector;

[0147] The optimization module 124 is used to optimize the first value network according to the first probability distribution of the latent action vector to obtain an optimized first value network; optimize the first policy network according to the optimized first value network to obtain an optimized first policy network; optimize the second value network according to the optimized first value network to obtain an optimized second value network; and optimize the second policy network according to the optimized first policy network to obtain an optimized second policy network.

[0148] Optionally, the processing module 123 is further configured to:

[0149] The first probability distribution of the latent action vector is passed through the decoder to obtain the system response generated in this round.

[0150] Optionally, the acquisition module 121 is also used to: acquire first historical conversation data, the first historical conversation data includes a first historical sentence and a historical reply sentence corresponding to the first historical sentence; acquire second historical conversation data, the second historical conversation data includes a second historical sentence and a historical reply sentence corresponding to the second historical sentence, and the semantic similarity between the second historical sentence and the first historical sentence is greater than or equal to a preset threshold; the encoding module 122 is also used to: encode the second historical conversation data using an encoder to obtain a continuous second hidden state vector; the processing module 123 is also used to pass the second hidden state vector through a second policy network to obtain a second probability distribution of the hidden action vector; the encoding module 122 is also used to: encode the first historical conversation data using an encoder to obtain a continuous third hidden state vector; the processing module 123 is also used to pass the third hidden state vector through a first policy network to obtain a third probability distribution of the hidden action vector; the acquisition module 121 is also used to: if the difference between the second probability distribution and the third probability distribution is less than the preset difference, acquire third historical conversation data, the third historical conversation data including a third historical sentence and a historical reply sentence corresponding to the third historical sentence.

[0151] Optionally, the optimization module 124 optimizes the first policy network according to the optimized first value network to obtain the optimized first policy network, specifically for:

[0152] Encoding the third historical conversation data using an encoder to obtain a continuous fourth hidden state vector, and passing the fourth hidden state vector through the first policy network to obtain a fourth probability distribution of the hidden action vector;

[0153] The first policy network is optimized according to the fourth hidden state vector, the fourth probability distribution, and the optimized first value network to obtain an optimized first policy network.

[0154] Optionally, when optimizing the first value network according to the first probability distribution of the latent action vector, the optimization module 124 is specifically configured to: calculate, through the second value network, evaluation indicators corresponding to one or more latent action vectors in the first probability distribution;

[0155] The first value network is optimized based on the evaluation metrics and feedback function, where the feedback function includes a conversation completion rate and a semantic similarity between a system response generated in this round and a next response statement of a user statement in this round in an unlabeled human-to-human conversation dataset.

[0156] Optionally, the acquisition module 121 is also used to: before obtaining the third historical conversation data, obtain the implicit action vector with the largest probability from each of the multiple second probability distributions to obtain a first set; obtain multiple implicit action vectors from the third probability distribution to obtain a second set; the conversation model training device 120 also includes a calculation module 125, which is used to calculate the difference between the second probability distribution and the third probability distribution based on the multiple implicit action vectors in the first set and the multiple implicit action vectors in the second set.

[0157] Optionally, when the processing module 123 passes the first probability distribution of the latent action vector through the decoder to obtain the system response generated in this round, it is specifically configured to:

[0158] Determining a hidden action vector with the highest probability in the first probability distribution;

[0159] The latent action vector with the maximum probability in the first probability distribution is used as the input of the decoder, and the system response generated in this round is obtained through the decoder.

[0160] Optionally, when the processing module 123 passes the first probability distribution of the latent action vector through the decoder to obtain the system response generated in this round, it is specifically configured to:

[0161] Collecting a plurality of latent action vectors having a probability greater than or equal to a preset probability from the first probability distribution;

[0162] Using multiple latent action vectors with probabilities greater than or equal to a preset probability in the first probability distribution as inputs to the decoder, and obtaining multiple system responses corresponding to the user sentences in the current round through the decoder;

[0163] A system response generated in this round is determined from the multiple system responses.

[0164] Figure 12 The training device for the dialogue model of the illustrated embodiment can be used to implement the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0165] The above describes the internal functions and structure of the dialogue model training device, which can be implemented as an electronic device. Figure 13 This is a schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. Figure 13 As shown, the electronic device includes a memory 131 and a processor 132 .

[0166] Memory 131 is used to store programs. In addition to the aforementioned programs, memory 131 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, images, videos, etc.

[0167] The memory 131 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0168] The processor 132 is coupled to the memory 131 and executes the program stored in the memory 131 to:

[0169] Obtaining current round conversation data, where the current round conversation data includes the system response from the previous round and the user's statement from the current round;

[0170] Encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector;

[0171] Passing the first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector;

[0172] According to the first probability distribution of the latent action vector, the first value network is optimized to obtain an optimized first value network; according to the optimized first value network, the first strategy network is optimized to obtain an optimized first strategy network; according to the optimized first value network, the second value network is optimized to obtain an optimized second value network; according to the optimized first strategy network, the second strategy network is optimized to obtain an optimized second strategy network.

[0173] Further, if Figure 13 As shown, the electronic device may further include: a communication component 133, a power component 134, an audio component 135, a display 136 and other components. Figure 13 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 13 Components shown.

[0174] The communication component 133 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 133 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 133 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0175] The power supply assembly 134 provides power to various components of the electronic device. The power supply assembly 134 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0176] The audio component 135 is configured to output and / or input audio signals. For example, the audio component 135 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 131 or transmitted via the communication component 133. In some embodiments, the audio component 135 also includes a speaker for outputting audio signals.

[0177] The display 136 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0178] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the training method of the dialogue model described in the above embodiment.

[0179] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0180] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a dialogue model, characterized in that: The dialogue model at least includes: an encoder, a decoder, a first policy network, a second policy network, a first value network, and a second value network; the method includes: Obtaining current round conversation data, where the current round conversation data includes the system response from the previous round and the user's statement from the current round; Encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector; Passing the first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector; According to the first probability distribution of the latent action vector, the first value network is optimized to obtain an optimized first value network; according to the optimized first value network, the first strategy network is optimized to obtain an optimized first strategy network; according to the optimized first value network, the second value network is optimized to obtain an optimized second value network; according to the optimized first strategy network, the second strategy network is optimized to obtain an optimized second strategy network.

2. The method according to claim 1, characterized in that The method further comprises: The first probability distribution of the latent action vector is passed through the decoder to obtain the system response generated in this round.

3. The method according to claim 2, characterized in that The method further comprises: Acquire first historical conversation data, where the first historical conversation data includes a first historical sentence and a historical reply sentence corresponding to the first historical sentence; Acquiring second historical conversation data, where the second historical conversation data includes a second historical sentence and a historical reply sentence corresponding to the second historical sentence, wherein a semantic similarity between the second historical sentence and the first historical sentence is greater than or equal to a preset threshold; Encoding the second historical conversation data using an encoder to obtain a continuous second hidden state vector, and passing the second hidden state vector through a second policy network to obtain a second probability distribution of the hidden action vector; Encoding the first historical conversation data using an encoder to obtain a continuous third hidden state vector, and passing the third hidden state vector through a first policy network to obtain a third probability distribution of the hidden action vector; If the difference between the second probability distribution and the third probability distribution is less than a preset difference, third historical dialogue data is obtained, where the third historical dialogue data includes a third historical sentence and a historical reply sentence corresponding to the third historical sentence.

4. The method according to claim 3, characterized in that The first strategy network is optimized according to the optimized first value network to obtain an optimized first strategy network, including: Encoding the third historical conversation data using an encoder to obtain a continuous fourth hidden state vector, and passing the fourth hidden state vector through the first policy network to obtain a fourth probability distribution of the hidden action vector; The first policy network is optimized according to the fourth hidden state vector, the fourth probability distribution, and the optimized first value network to obtain an optimized first policy network.

5. The method according to claim 1, wherein Optimizing the first value network according to the first probability distribution of the latent action vector includes: Calculate, by the second value network, evaluation indicators corresponding to one or more latent action vectors in the first probability distribution; The first value network is optimized based on the evaluation metrics and feedback function, where the feedback function includes a conversation completion rate and a semantic similarity between a system response generated in this round and a next response statement of a user statement in this round in an unlabeled human-to-human conversation dataset.

6. The method according to claim 3, characterized in that If the difference between the second probability distribution and the third probability distribution is less than a preset difference, before obtaining the third historical conversation data, the method further includes: Obtaining the hidden action vector with the maximum probability from each of the plurality of second probability distributions to obtain a first set; Obtaining a plurality of latent action vectors from the third probability distribution to obtain a second set; The difference between the second probability distribution and the third probability distribution is calculated based on the multiple latent motion vectors in the first set and the multiple latent motion vectors in the second set.

7. The method according to claim 2, characterized in that Passing the first probability distribution of the latent action vector through the decoder to obtain a system response generated in this round includes: Determining a hidden action vector with the highest probability in the first probability distribution; The latent action vector with the maximum probability in the first probability distribution is used as the input of the decoder, and the system response generated in this round is obtained through the decoder.

8. The method according to claim 2, characterized in that Passing the first probability distribution of the latent action vector through the decoder to obtain a system response generated in this round includes: Collecting a plurality of latent action vectors having a probability greater than or equal to a preset probability from the first probability distribution; Using multiple latent action vectors with probabilities greater than or equal to a preset probability in the first probability distribution as inputs to the decoder, and obtaining multiple system responses corresponding to the user sentences in the current round through the decoder; A system response generated in this round is determined from the multiple system responses.

9. A training device for a dialogue model, characterized in that: The dialogue model at least includes: an encoder, a decoder, a first policy network, a second policy network, a first value network, and a second value network; the device includes: An acquisition module is used to obtain the current round of dialogue data, which includes the system response of the previous round and the user statement of the current round; An encoding module, configured to encode the system response of the previous round and the user statement of the current round using an encoder to obtain a continuous first hidden state vector; a processing module, configured to pass the continuous first hidden state vector through the first policy network or the second policy network to obtain a first probability distribution of a hidden action vector; The optimization module is used to optimize the first value network according to the first probability distribution of the latent action vector to obtain an optimized first value network; optimize the first strategy network according to the optimized first value network to obtain an optimized first strategy network; optimize the second value network according to the optimized first value network to obtain an optimized second value network; optimize the second strategy network according to the optimized first strategy network to obtain an optimized second strategy network.

10. An electronic device, wherein: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images