Human-computer conversation model training method, human-computer conversation method and device
By training the dialogue state tracking module and dialogue strategy module asynchronously, the problem of model convergence difficulty caused by synchronous training is solved, thereby improving the intelligence and user experience of human-computer dialogue.
Patent Information
- Application Number
- CN202211697905.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2042-12-28
AI Technical Summary
In existing human-computer dialogue model training methods, the synchronous training of the dialogue state tracking module and the dialogue strategy module leads to mutual interference, making model convergence difficult and affecting the intelligence of human-computer dialogue.
An asynchronous training method is adopted, which controls the amount of sample data for the dialogue state tracking module and the dialogue strategy module by introducing a first threshold and a second threshold. The modules are trained by course learning and reinforcement learning respectively, avoiding synchronous updates.
It improves the convergence and robustness of the model, enhances the intelligence of human-computer dialogue, and improves the user experience.
Smart Images
Figure CN115952267B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial intelligence technology, and in particular to a human-computer dialogue model training method and device, and a human-computer dialogue method and device. Background Technology
[0002] In many customer service scenarios, customer service personnel need to converse with users to collect information and process orders. During peak periods, user demand increases dramatically, making it difficult for human customer service representatives to cope. Therefore, the industry has widely developed machine-based customer service systems to alleviate the shortage of human representatives. Existing research on training human-machine dialogue systems generally uses supervised learning or reinforcement learning methods.
[0003] Currently, human-computer dialogue systems are widely used. In each round, the dialogue state tracking module first understands the user's intent and records the dialogue state from its inception to the present. Then, the dialogue strategy module categorizes system intents to achieve multi-round dialogue. The dialogue strategy module determines the action selected by the machine agent in the current round and is a crucial component of the human-computer dialogue system.
[0004] In the process of developing this invention, the inventors discovered that existing human-computer dialogue model training methods suffer from poor accuracy and difficulty in convergence, thus affecting the intelligence of human-computer dialogue. Through research and analysis, the specific reasons for these problems were found to be as follows:
[0005] In the training process of existing human-computer dialogue models, a common approach is to train the dialogue state tracking and dialogue policy modules simultaneously. This means that both the dialogue state tracking module and the dialogue policy module are updated concurrently during model training. However, the dialogue state tracking module and the dialogue policy module can interfere with each other.
[0006] On the one hand, during training, the dialogue strategy module is affected by the dialogue state tracking module, leading to biases in the training of the dialogue strategy. Reinforcement learning is a learning-through-exploration method where the agent selects a strategy and interacts with the environment. As the environment changes, a reward is given, which is used to evaluate the quality of the chosen action. Then, parameters are updated based on the reward. In a dialogue system, the dialogue system and the user model interact and sample dialogue, acquiring rewards during the sampling process, and finally updating model parameters based on the rewards. However, during the interaction, due to errors in the dialogue state tracking module, the dialogue strategy module may make decisions based on the erroneous output of the dialogue state tracking module. Thus, in some cases, even if the dialogue strategy module's decision is correct, the dialogue state tracking module may fail to update to the correct dialogue state, causing the dialogue to reach its maximum number of turns, resulting in dialogue failure. Therefore, errors in the dialogue state tracking module can lead to dialogue failure even with correct dialogue strategy decisions, thus "penalizing" the correct dialogue strategy.
[0007] On the other hand, the dialogue strategy module also indirectly affects the dialogue state tracking module. Different dialogue decisions in a dialogue system often lead to different user responses. These responses are input into the system and then processed by the dialogue state tracking module to understand the user statements and update the system's dialogue state. However, the accuracy of the dialogue state tracking module's understanding varies for different types of user statements, thus introducing different errors. Furthermore, reinforcement learning is a sampling process. Due to the strong randomness of dialogue strategy sampling, the distribution of different types of user statements in the sampled data becomes uneven, resulting in unstable training of the dialogue state tracking module and an inability to fully learn difficult and rare user statement categories.
[0008] Therefore, in the training process of existing human-computer dialogue models, due to the synchronous training of dialogue state tracking and dialogue strategy modules, the dialogue strategy module is affected by the dialogue state tracking module, resulting in deviations. At the same time, the dialogue state tracking module is also affected by the dialogue strategy module, leading to unstable results and insufficient training. Ultimately, this makes it difficult for the entire human-computer dialogue model to converge, thus affecting the intelligence of human-computer dialogue. Summary of the Invention
[0009] In view of this, the main objective of the present invention is to provide a human-computer dialogue model training method, a human-computer dialogue method and device, which can enhance model convergence and improve the intelligence of human-computer dialogue.
[0010] To achieve the above objectives, the technical solution proposed in this embodiment of the invention is as follows:
[0011] A method for training a human-computer dialogue model includes:
[0012] a. Using a human-computer dialogue model, sample data of a single dialogue process is sampled to obtain dialogue state tracking sample data and dialogue strategy sample data corresponding to each round of dialogue, and the comprehension difficulty level corresponding to the dialogue state tracking sample data is determined. The dialogue state tracking sample data is added to the first sample data pool, and the dialogue strategy sample data is added to the second sample data pool.
[0013] b. If the number of sampled dialogue processes reaches an integer multiple of a preset first threshold, then the first sample data pool is split according to the comprehension difficulty level to obtain a sample data sub-pool corresponding to each comprehension difficulty level. Using the sample data sub-pool, the dialogue state tracking module of the human-computer dialogue model is trained using a course learning method, and the first sample data pool is cleared after training. If the number of sampled dialogue processes reaches an integer multiple of a preset second threshold, then the dialogue strategy module of the human-computer dialogue model is trained using a reinforcement learning method using the sample data in the second sample data pool, and the second sample data pool is cleared after training. Wherein, the first threshold is greater than the second threshold.
[0014] c. If the number of sampled dialogue processes has not reached the preset threshold for the number of dialogue processes, then return to step a.
[0015] This invention also proposes a human-computer dialogue method, including:
[0016] During human-computer dialogue, a pre-trained human-computer dialogue model is used to generate corresponding system response statements for each round of user dialogue; wherein, the human-computer dialogue model is obtained based on the training method of the human-computer dialogue model described above.
[0017] This invention also proposes a training device for a human-computer dialogue model, including a processor and a memory;
[0018] The memory stores an application program that can be executed by the processor, which enables the processor to perform the human-computer dialogue model training method described above.
[0019] This invention also proposes a human-computer dialogue device, including a processor and a memory;
[0020] The memory stores an application program that can be executed by the processor, which enables the processor to perform the human-computer dialogue method as described above.
[0021] In summary, the human-computer dialogue model training scheme and human-computer dialogue scheme proposed in this embodiment of the invention introduce a first threshold and a second threshold to control the amount of sample data used to train the dialogue state tracking module and the dialogue strategy module, respectively. This causes the dialogue state tracking module and the dialogue strategy module to update asynchronously instead of synchronously. Furthermore, the first threshold is greater than the second threshold, and a course learning method is used to train the dialogue state tracking module. On the one hand, compared to the dialogue strategy module, a larger sample data pool is used to train the dialogue state tracking module. Moreover, the course learning method allows for targeted training of the dialogue state tracking module based on sample data of different levels of comprehension difficulty, thereby improving the accuracy of the dialogue state tracking module. On the other hand, for the dialogue strategy module, training is performed each time based on a smaller number of dialogue process sample data, thus updating parameters with fewer interactions and continuously learning the optimal dialogue strategy. Therefore, by adopting the embodiments of the present invention, the problem of mutual influence between the dialogue state tracking module and the dialogue strategy module when updating the dialogue state tracking module and the dialogue strategy module at the same time can be avoided, the mutual influence between the dialogue state tracking module and the dialogue strategy module can be alleviated, thereby enhancing the model's convergence and robustness, making the model have a stronger fault tolerance, improving the intelligence of human-computer dialogue, and thus enhancing the user experience. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the human-computer dialogue model training method according to an embodiment of the present invention;
[0023] Figure 2 This is a schematic diagram of the human-computer dialogue model training method architecture according to an embodiment of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0025] Figure 1 This is a schematic diagram of the human-computer dialogue model training method according to an embodiment of the present invention, as follows: Figure 1 As shown, this embodiment mainly includes the following steps:
[0026] Step 101: Using a human-computer dialogue model, sample data of a single dialogue process is collected to obtain dialogue state tracking sample data and dialogue strategy sample data corresponding to each round of dialogue. The comprehension difficulty level corresponding to the dialogue state tracking sample data is determined, and the dialogue state tracking sample data is added to the first sample data pool, and the dialogue strategy sample data is added to the second sample data pool.
[0027] Step 101 is used to generate sample data for a single dialogue process. For each round of dialogue in the process, corresponding dialogue state tracking sample data and dialogue strategy sample data are generated to enhance the training of the human-computer dialogue model.
[0028] It should be noted that the dialogue state tracking module and dialogue strategy module in the human-computer dialogue model need to be initialized using pre-trained model parameters. That is, before executing step 101, the human-computer dialogue model needs to be pre-trained on real dialogue data using supervised learning in order to obtain the initial model parameters for reinforcement learning.
[0029] In step 101, in order to facilitate the subsequent training of the dialogue state tracking module using the course learning method, it is necessary to determine the corresponding comprehension difficulty level for each sampled dialogue state tracking data. This will allow the dialogue state tracking data to be classified based on different comprehension difficulty levels in subsequent steps, and the dialogue state tracking module to be trained using dialogue state tracking data with different comprehension difficulty levels according to a certain course learning strategy.
[0030] In one embodiment, the dialogue state tracking sample data may include: input data and theoretical output data of the dialogue state tracking module in the current round of dialogue. In another embodiment, to facilitate the classification of the dialogue state tracking sample data in subsequent steps, the comprehension difficulty level determined in this step can be stored in the dialogue state tracking sample data.
[0031] In one implementation, the dialogue strategy sample data may include: input data, actual output data, and corresponding reward data of the dialogue strategy module in the current round of dialogue.
[0032] In one implementation, step 101 may specifically employ the following steps to sample data of a single dialogue process using a human-computer dialogue model:
[0033] Step a1: The dialogue strategy module of the human-computer dialogue model on the machine side selects system actions and outputs corresponding system statements based on the current training task.
[0034] The training task is a pre-set training task for a human-computer dialogue model, so that the model can perform the corresponding task through training.
[0035] Step a2: In response to the user's current reply statement on the user side, the human-computer dialogue model uses the dialogue state tracking module to determine the current dialogue state based on the user statement and the current dialogue history; uses the dialogue strategy module to select a system action based on the determined dialogue state; uses the natural language generation module to output a system statement based on the selected system action; triggers the user side to determine the theoretical output data of the dialogue state tracking module; and triggers the machine side's machine perspective module to determine the preset understanding difficulty level of the user action corresponding to the user statement as the understanding difficulty level corresponding to the dialogue state tracking sample data of the current round of dialogue.
[0036] In practical applications, user-side operations can be performed manually or simulated using a pre-set user model. The user model is used to simulate user-generated dialogue statements. Using a user model can reduce costs and improve training efficiency, but it requires sampling the user model simultaneously with the sample data to train the model using the corresponding sample data.
[0037] In this step, the comprehension difficulty level corresponding to the dialogue state tracking sample data will be executed by the machine-side machine perspective module. This machine perspective module is independent of the human-computer dialogue model on the machine side. It can know the user action corresponding to the user's statement, and thus can determine the comprehension difficulty level corresponding to the dialogue state tracking sample data based on the preset comprehension difficulty level of the user action.
[0038] In practical applications, those skilled in the art can set specific levels of difficulty in understanding according to the needs of the application. For example, the levels of difficulty in understanding can be divided into three types: easy, medium, and difficult.
[0039] When you understand that the difficulty levels include easy, medium, and hard, you can refer to the formula beforehand. Set the difficulty level for understanding each user action. Here, dst_acc is the accuracy of the dialogue state tracking module output for each user action after the model is pre-trained on a certain amount of dialogue data; a and b are the preset easy threshold and hard threshold, respectively.
[0040] It should be noted that due to the randomness of reinforcement learning, the distribution of user statements of different categories in the sample data pool of the dialogue state tracking module will be unbalanced. Therefore, in subsequent steps, course learning will be used to adjust the training data of the dialogue state tracking module. In this embodiment, user statements of different categories are divided according to user actions. The human-computer dialogue model trained using reinforcement learning is not started from scratch, but rather pre-trained on real dialogue data using supervised learning. The pre-trained model parameters are then used as the initial model parameters for reinforcement learning. Therefore, it is easy to statistically determine the accuracy (dst_acc) of the dialogue state tracking module output under different user actions in the dialogue data. Then, based on the accuracy of the dialogue state tracking module under each user action, all training data in the sample data pool of the dialogue state tracking module are divided into three levels: easy, medium, and difficult, according to user actions. The accuracy of the dialogue state tracking module varies significantly for different levels of training data.
[0041] The specific user action corresponding to the user statement can be determined by the user model or manually and then communicated to the machine side.
[0042] Step a3: If the current dialogue round has not reached the preset maximum dialogue round, and the intent of the system action selected in the current dialogue round is not to end the dialogue, then the dialogue process is determined to be not over; otherwise, the dialogue process is determined to be over, triggering the user side to determine whether the current dialogue state and the user's purpose are consistent. If the following conditions are met simultaneously: the dialogue round has not reached the maximum dialogue round, the intent of the system action selected in the current dialogue round is to end the dialogue, and the dialogue state determined at the end of the dialogue process is consistent with the user's purpose, then the dialogue is determined to be successful; otherwise, the dialogue is determined to be unsuccessful. The user's purpose is randomly selected by the user side from a preset set of dialogue purposes.
[0043] In practical applications, when the user side is a user model, a perspective module independent of that user model can determine whether the current dialogue state and the user's purpose are consistent. When the user side is a real person, the direct trigger can be determined by that person.
[0044] Step a4: Determine the reward data for the dialogue strategy module based on whether the dialogue process has ended, whether the dialogue was successful, and whether the system action selected in the current round of dialogue conforms to the preset dialogue flow.
[0045] The specific method for determining the reward data in this step is the same as the existing technology, and will not be repeated here.
[0046] Step a5: If the dialogue process has not ended, return to step a2 to continue collecting sample data for the next round of dialogue.
[0047] In this step, if it is determined in step a4 that the dialogue process has ended, then step 102 is executed to determine whether it is necessary to trigger the training of the dialogue state tracking module or the dialogue strategy module.
[0048] In one implementation, when the user side is a pre-trained user model, the following steps b1 to b6 can be used to sample data for a single dialogue process using the human-computer dialogue model:
[0049] Step b1: The user model initializes the user purpose by randomly selecting a user; the machine-side human-computer dialogue model initiates a dialogue, including using the dialogue strategy module to select system actions and output corresponding system statements based on the current training task.
[0050] Step b2: The user model selects user actions based on the system statements output by the human-computer dialogue model, generates user statements based on the user actions, and outputs them.
[0051] Step b3: In response to the user's current reply statement on the user side, the human-computer dialogue model uses the dialogue state tracking module to determine the current dialogue state based on the user statement and the current dialogue history; uses the dialogue strategy module to select a system action based on the determined dialogue state; uses the natural language generation module to output a system statement based on the selected system action; triggers the user side to determine the theoretical output data of the dialogue state tracking module; and triggers the machine side's machine perspective module to determine the preset comprehension difficulty level of the user action corresponding to the user statement as the comprehension difficulty level corresponding to the dialogue state tracking sample data of the current round of dialogue; and determines the theoretical output data of the user model based on the user's purpose and the system action selected in the current round of dialogue.
[0052] In step b3, the user side can specifically determine the theoretical output data of the dialogue state tracking module using the following method:
[0053] Based on the user action corresponding to the current round of dialogue and the dialogue state determined in the previous round of dialogue, the theoretical output data of the dialogue state tracking module is determined.
[0054] In practical applications, the theoretical output data can be determined by a machine-view module on the user side that is independent of the user model.
[0055] Step b4: If the current dialogue round has not reached the preset maximum dialogue round, and the intent of the system action selected in the current dialogue round is not to end the dialogue, then the dialogue process is determined to be not over; otherwise, the dialogue process is determined to be over, triggering the user side to determine whether the current dialogue state and the user's purpose are consistent. If the following conditions are met simultaneously: the dialogue round has not reached the maximum dialogue round, the intent of the system action selected in the current dialogue round is to end the dialogue, and the dialogue state determined at the end of the dialogue process is consistent with the user's purpose, then the dialogue is determined to be successful; otherwise, the dialogue is determined to be unsuccessful. The user's purpose is randomly selected by the user side from a preset set of dialogue purposes.
[0056] Step b5: Determine the reward data for the dialogue strategy module based on whether the dialogue process has ended, whether the dialogue is successful, and whether the system action selected in the current round of dialogue conforms to the preset dialogue flow; determine the reward data for the user model based on whether the dialogue process has ended, whether the dialogue is successful, and whether the user action selected by the user model in the current round of dialogue conforms to the preset dialogue flow.
[0057] Step b6: If the dialogue process has not ended, return to step b3 to continue collecting sample data for the next round of dialogue.
[0058] Based on steps b1 to b6 above, user model sample data corresponding to each round of dialogue during the dialogue process can be obtained. Accordingly, in step 101, these user model sample data can be added to the corresponding sample data pool. Specifically, step 101 may further include the following:
[0059] The user model sample data corresponding to each round of dialogue obtained by sampling is added to the third sample data pool; the user model sample data includes the user model's input data, actual output data, theoretical output data, and user model's reward data.
[0060] Step 102: If the number of sampled dialogue processes reaches an integer multiple of a preset first threshold, then the first sample data pool is split according to the comprehension difficulty level to obtain a sample data sub-pool corresponding to each comprehension difficulty level. Using the sample data sub-pool, the dialogue state tracking module of the human-computer dialogue model is trained using a course learning method, and the first sample data pool is cleared after training. If the number of sampled dialogue processes reaches an integer multiple of a preset second threshold, then the dialogue strategy module of the human-computer dialogue model is trained using a reinforcement learning method using the sample data in the second sample data pool, and the second sample data pool is cleared after training. Wherein, the first threshold is greater than the second threshold.
[0061] It's important to note that in this step, after each second threshold of dialogue processes, reinforcement learning is used to train the dialogue policy module. The network parameters are then updated using the sample data generated from these second threshold of dialogue processes, and the resource pool is cleared afterward. After each first threshold (first threshold > second threshold) of dialogue processes, curriculum learning is used to update the dialogue state tracking module of the human-computer dialogue model. Since the first threshold is not equal to the second threshold, the updates of the dialogue policy module and the dialogue state tracking module are not synchronous. The benefits of this asynchronous update are: for the dialogue state tracking module, due to the large amount of stored sample data from the first threshold of dialogue processes, curriculum learning can be used to adjust the distribution of training data and to regulate the training process, allowing for targeted training; for the dialogue policy module, updating its parameters based on a smaller number of dialogue processes allows it to continuously learn the optimal dialogue policy from the sample data pool. Therefore, this mitigates the mutual influence between the dialogue state tracking module and the dialogue policy module, thereby enhancing model convergence and robustness, and giving the model stronger fault tolerance.
[0062] The first threshold and the second threshold can be set by those skilled in the art according to actual needs, as long as the first threshold is greater than the second threshold.
[0063] In step 102, considering the randomness of reinforcement learning, which may lead to an imbalance in the distribution of user statements of different categories in the sample data pool of the dialogue state tracking module, course learning is used here to adjust the training data of the dialogue state tracking module.
[0064] In one implementation, the dialogue state tracking module of the human-computer dialogue model can be trained using the sample data sub-pool and a course learning method:
[0065] Based on the sample data sub-pool, the dialogue state tracking module is trained sequentially using general learning, focused learning, and review methods.
[0066] Specifically, when the difficulty level is categorized into three levels: easy, medium, and hard, the corresponding sample data sub-pool includes an easy sample data sub-pool, a medium sample data sub-pool, and a hard sample data sub-pool. Accordingly, the dialogue state tracking module can be trained using a course-based learning approach:
[0067] The dialogue state tracking module is trained by using all sample data from each sample data sub-pool in the preset sub-pool traversal order.
[0068] The sub-pool traversal order is: simple sample data sub-pool, medium sample data sub-pool, difficult sample data sub-pool, medium sample data sub-pool, difficult sample data sub-pool, difficult sample data sub-pool, simple sample data sub-pool, medium sample data sub-pool, and difficult sample data sub-pool.
[0069] When training the dialogue state tracking module using the aforementioned sub-pool traversal order, firstly, a general learning process is performed from easy to difficult, that is, the model is trained sequentially using the simple sample data sub-pool, the medium sample data sub-pool, and the difficult sample data sub-pool, using all data in sequence. Secondly, focused learning is performed, that is, the model is trained sequentially using the data from the medium sample data sub-pool and the difficult sample data sub-pool. Since the accuracy of the dialogue state tracking module is high with simple data, followed by medium data, and worst with difficult data, this approach aims to have the model learn specifically for medium and difficult sample data, and then learn for difficult sample data. Finally, a review process is performed, where training proceeds from easy to difficult, that is, training is performed sequentially using the simple sample data sub-pool, the medium sample data sub-pool, and the difficult sample data sub-pool, ensuring that the model does not forget the simple data. Experimental results show that the curriculum learning method can effectively mitigate the influence between different modules in the human-computer dialogue model, and compared with traditional update strategies, it achieves the best dialogue success rate.
[0070] In the above training, whenever a sample data sub-pool is traversed, a batch of sample data is obtained from that pool according to the preset batch processing number. The training is carried out using this batch of sample data. After the training of this batch of sample data is completed, the network parameters of the module are optimized and adjusted using the training results.
[0071] In one implementation, when sampling sample data for a single dialogue process is performed in step 101, and the user side generates user dialogue using a user model, step 102 can also utilize sample data from the third sample data pool to train the user model at the same training frequency as the dialogue strategy module. The specific method is as follows:
[0072] If the number of sampled dialogue processes reaches an integer multiple of the second threshold, the user model is trained using the sample data in the third sample data pool, and the third sample data pool is cleared after training is completed.
[0073] Step 103: If the number of sampled dialogue processes has not reached the preset threshold for the number of dialogue processes, return to step 101.
[0074] This step is used to determine whether the training process of the human-computer dialogue model needs to be terminated. That is, when the number of sampled dialogue processes reaches the preset threshold for the number of dialogue processes, the model training process is terminated. Otherwise, return to step 101 to sample the dialogue process data and use the sampled data to train the model.
[0075] The threshold for the number of dialogue processes is used to limit the termination condition of human-computer dialogue model training, and the appropriate value can be set by those skilled in the art according to actual needs.
[0076] Figure 2 A schematic diagram of the model training method architecture according to an embodiment of the present invention is provided. Figure 2 As shown, in this model training method, during the reinforcement learning interaction sampling process, the machine client (i.e., the human-computer dialogue model) and the user model interact through natural language. After completing a dialogue process, the model's input, output, and reward are saved to the dialogue state tracking module sample data pool, the dialogue policy module sample data pool, and the user model sample data pool, respectively. Then, for every first threshold (k2) dialogue processes, the model parameters of the dialogue state tracking module are updated using course learning. For every second threshold (k1) dialogue processes, the model parameters of the dialogue policy module are updated using reinforcement learning. For every k1 dialogue processes, the model parameters of the user model are updated using reinforcement learning, where k1 is less than k2.
[0077] The dialogue state tracking module and dialogue strategy module can be implemented using existing methods. Preferably, they can also be implemented as follows:
[0078] The dialogue state tracking module maintains the dialogue state (user-provided information) from the start of the dialogue to the current round (round t). In this embodiment of the invention, two encoders will be used to encode the dialogue history. Status of the conversation in the previous round in This is the system response from the previous round (round t-1). This refers to the user's statement in the current round. Then, based on the latent vectors from the dialogue history encoder, a multilayer perceptron is used to classify the user's actions. The latent vectors of the two encoders are concatenated and decoded by a decoder to obtain the dialogue state for the current round. The formula for the dialogue state tracking module can be defined as follows:
[0079] The dialogue strategy module selects the system action and system slot value for the current round based on the dialogue state, guiding the direction of the dialogue and determining its success or failure. The input of the dialogue strategy module in round t includes (1) the system action of the previous round. (2) Current dialogue state (3) Latent vectors of the dialogue history encoder (4) Database query result q t The system's dialogue strategy module consists of a multi-layered perception mechanism, and its formula can be defined as follows:
[0080] In the aforementioned dialogue state tracking module, the input is the dialogue state and dialogue history of the previous round (the system response of the previous round and the user statement of the current round). Compared with the traditional input of the entire dialogue history (the concatenation of all system statements and user statements from the first round of dialogue to the current round), the processing speed is faster, the training time is shorter, and the response time is also shorter when deployed online.
[0081] As can be seen from the above technical solution, the above method embodiment introduces a first threshold and a second threshold to control the amount of sample data used to train the dialogue state tracking module and the dialogue strategy module, respectively. This makes the dialogue state tracking module and the dialogue strategy module no longer update synchronously, but asynchronously. Furthermore, the first threshold is greater than the second threshold, and a course learning method is used to train the dialogue state tracking module. Thus, on the one hand, compared to the dialogue strategy module, a larger sample data pool is used to train the dialogue state tracking module. Moreover, the course learning method allows for targeted training of the dialogue state tracking module based on sample data of different levels of comprehension difficulty, thereby improving the accuracy of the dialogue state tracking module. On the other hand, for the dialogue strategy module, training is performed each time based on a smaller number of dialogue process sample data, thus updating parameters with fewer interactions and continuously learning the optimal dialogue strategy. Therefore, by adopting the embodiments of the present invention, the problem of mutual influence between the dialogue state tracking module and the dialogue strategy module when updating the dialogue state tracking module and the dialogue strategy module at the same time can be avoided, the mutual influence between the dialogue state tracking module and the dialogue strategy module can be alleviated, thereby enhancing the model's convergence and robustness, making the model have a stronger fault tolerance, improving the intelligence of human-computer dialogue, and thus enhancing the user experience.
[0082] Based on the above embodiments of the human-computer dialogue model training method, correspondingly, this embodiment of the invention proposes a human-computer dialogue method, including:
[0083] During human-computer dialogue, a pre-trained human-computer dialogue model is used to generate corresponding system response statements for each round of user dialogue; wherein, the human-computer dialogue model is obtained based on the human-computer dialogue model training method described above.
[0084] Based on the above-described training method embodiment for human-computer dialogue model, the present invention proposes a training device for human-computer dialogue model, including a processor and a memory.
[0085] The memory stores an application program that can be executed by the processor, which enables the processor to execute the training method for the human-computer dialogue model as described above.
[0086] The training methods and devices for the above-mentioned human-computer dialogue models are based on the same inventive concept. Since the methods and devices solve problems in similar ways, the implementation of the devices and methods can refer to each other, and the repeated parts will not be described again.
[0087] Based on the above embodiments of the human-computer dialogue method, the present invention proposes a human-computer dialogue device, including a processor and a memory;
[0088] The memory stores an application program that can be executed by the processor, which enables the processor to perform the human-computer dialogue method as described above.
[0089] The above-mentioned human-computer dialogue method and device are based on the same inventive concept. Since the methods and devices solve problems in similar ways, the implementation of the devices and methods can refer to each other, and the repeated parts will not be described again.
[0090] Furthermore, each embodiment of the present invention can be implemented by a data processing program executed by a data processing device such as a computer. Clearly, the data processing program constitutes the present invention. Moreover, the data processing program, typically stored in a storage medium, is executed by directly reading the program from the storage medium or by installing or copying the program to the storage device (such as a hard disk and / or memory) of the data processing device. Therefore, such a storage medium also constitutes the present invention. The storage medium can use any type of recording method, such as paper storage media (e.g., paper tape), magnetic storage media (e.g., floppy disks, hard disks, flash memory), optical storage media (e.g., CD-ROMs), magneto-optical storage media (e.g., MOs), etc.
[0091] Therefore, the present invention also discloses a storage medium storing a data processing program for performing any of the above-described embodiments of the methods of the present invention.
[0092] Furthermore, the steps described in this invention can be implemented not only by a data processing program but also by hardware, such as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Therefore, such hardware capable of implementing the methods described in this invention can also constitute this invention.
[0093] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.
[0094] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for training a human-computer dialogue model, characterized in that, include: a. Using a human-computer dialogue model, sample data of a single dialogue process is sampled to obtain dialogue state tracking sample data and dialogue strategy sample data corresponding to each round of dialogue, and the comprehension difficulty level corresponding to the dialogue state tracking sample data is determined. The dialogue state tracking sample data is added to the first sample data pool, and the dialogue strategy sample data is added to the second sample data pool. b. If the number of sampled dialogue processes reaches an integer multiple of the preset first threshold, the first sample data pool is split according to the understanding difficulty level to obtain a sample data sub-pool corresponding to each understanding difficulty level. Using the sample data sub-pool, the dialogue state tracking module of the human-computer dialogue model is trained using the course learning method, and the first sample data pool is cleared after training is completed. If the number of sampled dialogue processes reaches an integer multiple of a preset second threshold, then the dialogue strategy module of the human-computer dialogue model is trained using the sample data in the second sample data pool and a reinforcement learning method is adopted, and the second sample data pool is cleared after training is completed; wherein, the first threshold is greater than the second threshold. c. If the number of sampled dialogue processes has not reached the preset threshold for the number of dialogue processes, then return to step a.
2. The method according to claim 1, characterized in that, The dialogue state tracking sample data includes: the input data and theoretical output data of the dialogue state tracking module in the current round of dialogue; The dialogue strategy sample data includes: the input data, actual output data, and corresponding reward data of the dialogue strategy module in the current round of dialogue.
3. The method according to claim 2, characterized in that, The sampling of sample data for a single dialogue process using a human-computer dialogue model includes: a1. The dialogue strategy module of the human-computer dialogue model on the machine side selects system actions and outputs corresponding system statements based on the current training task. a2. The human-computer dialogue model responds to the user's current reply statement on the user side by using the dialogue state tracking module to determine the current dialogue state based on the user statement and the current dialogue history; using the dialogue strategy module to select a system action based on the determined dialogue state; using the natural language generation module to output a system statement based on the selected system action; triggering the user side to determine the theoretical output data of the dialogue state tracking module; and triggering the machine side's machine perspective module to determine the preset understanding difficulty level of the user action corresponding to the user statement as the understanding difficulty level corresponding to the dialogue state tracking sample data of the current round of dialogue; a3. If the current dialogue round has not reached the preset maximum dialogue round, and the intent of the system action selected in the current dialogue round is not to end the dialogue, then the dialogue process is determined to be not over; otherwise, the dialogue process is determined to be over, triggering the user side to determine whether the current dialogue state and the user's purpose are consistent. If the following conditions are met simultaneously: the dialogue round has not reached the maximum dialogue round, the intent of the system action selected in the current dialogue round is to end the dialogue, and the dialogue state determined at the end of the dialogue process is consistent with the user's purpose, then the dialogue is determined to be successful; otherwise, the dialogue is determined to be unsuccessful. The user's purpose is randomly selected by the user side from a preset set of dialogue purposes. a4. Based on whether the dialogue process has ended, whether the dialogue is successful, and whether the system action selected in the current round of dialogue conforms to the preset dialogue flow, determine the reward data of the dialogue strategy module; a5. If the dialogue process has not ended, return to step a2 to continue the sample data collection for the next round of dialogue.
4. The method according to claim 3, characterized in that, The user side is a pre-trained user model, which is used to simulate users generating dialogue statements; Between step a1 and step a2, the following is further included: The user model selects user actions based on the system statements output by the human-computer dialogue model, generates user statements based on the user actions, and outputs them. The determination of the theoretical output data of the dialogue state tracking module in step a2 includes: Based on the user action corresponding to the current round of dialogue and the dialogue state determined in the previous round of dialogue, the theoretical output data of the dialogue state tracking module is determined. Step a2 further includes: Based on the user's purpose and the system action selected in the current round of dialogue, determine the theoretical output data of the user model; Step a4 further includes: The reward data for the user model is determined based on whether the dialogue process has ended, whether the dialogue has been successful, and whether the user action selected by the user model in the current round of dialogue conforms to the preset dialogue flow. Step a further includes: The user model sample data corresponding to each round of dialogue obtained by sampling is added to the third sample data pool; the user model sample data includes the user model's input data, actual output data, theoretical output data, and user model's reward data.
5. The method according to claim 4, characterized in that, Step b further includes: If the number of sampled dialogue processes reaches an integer multiple of the second threshold, the user model is trained using the sample data in the third sample data pool, and the third sample data pool is cleared after training is completed.
6. The method according to claim 1, characterized in that, The dialogue state tracking module, which uses the sample data sub-pool and a course learning method to train the human-computer dialogue model, includes: Based on the sample data sub-pool, the dialogue state tracking module is trained sequentially using general learning, focused learning, and review methods.
7. The method according to claim 6, characterized in that, The difficulty levels for understanding are categorized as easy, medium, and hard. The sample data sub-pool includes a simple sample data sub-pool, a medium sample data sub-pool, and a difficult sample data sub-pool; The dialogue state tracking module for training the human-computer dialogue model includes: According to the preset sub-pool traversal order, the dialogue state tracking module is trained by using all sample data in each sample data sub-pool that has been traversed. The sub-pool traversal order is: simple sample data sub-pool, medium sample data sub-pool, difficult sample data sub-pool, medium sample data sub-pool, difficult sample data sub-pool, difficult sample data sub-pool, simple sample data sub-pool, medium sample data sub-pool, difficult sample data sub-pool.
8. A human-computer dialogue method, characterized in that, include: During human-computer dialogue, a pre-trained human-computer dialogue model is used to generate corresponding system response statements for each round of dialogue user statements; wherein, the human-computer dialogue model is obtained based on the training method of the human-computer dialogue model according to any one of claims 1 to 7.
9. A training device for a human-computer dialogue model, characterized in that, Including processor and memory; The memory stores an application program that can be executed by the processor, which causes the processor to execute the human-computer dialogue model training method as described in any one of claims 1 to 7.
10. A human-computer interaction device, characterized in that, Including processor and memory; The memory stores an application program that can be executed by the processor, which causes the processor to execute the human-computer dialogue method as described in claim 8.