Dialogue processing method, device and equipment and storage medium

Abstract

The application relates to a dialogue processing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: performing at least one round of dialogue through two agent networks in a dialogue learning system; obtaining state transition operators corresponding to each round of dialogue in the at least one round of dialogue of the two agent networks in a manner of reinforcement learning; obtaining loss function values of the two agent networks based on the state transition operators corresponding to each round of dialogue in the at least one round of dialogue of the two agent networks; and performing parameter updating on the two agent networks respectively based on the loss function values of the two agent networks. According to the scheme, the rewards generated by the dialogue are decomposed by introducing the states corresponding to the two agent networks in the dialogue learning system, different agent networks can be trained more targetedly, and the accuracy of the trained dialogue system agent networks is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a dialogue processing method and device, equipment and a storage medium. BACKGROUND

[0002] An intelligent dialogue system is an intelligent system that provides auxiliary information in a relevant field to a user through dialogue interaction with the user.

[0003] In related technologies, an intelligent dialogue system usually performs reinforcement learning training through a pre-set user simulator. That is, a developer pre-sets a user simulator that simulates user dialogue, outputs user dialogue through the user simulator in the process of training the intelligent dialogue system, and responds through the intelligent dialogue system, and updates parameters of the intelligent dialogue system according to a response result of the intelligent dialogue system.

[0004] However, the above scheme needs to pre-set a user simulator, and in some fields that require a large amount of professional knowledge (such as the medical field), an accurate user simulator cannot usually be constructed, thereby resulting in low accuracy of the trained intelligent dialogue system. SUMMARY

[0005] Embodiments of the present application provide a dialogue processing method, device, equipment and storage medium, which can improve the accuracy of an intelligent dialogue system. The technical scheme is as follows.

[0006] In one aspect, a dialogue processing method is provided, and the method comprises:

[0007] performing at least one round of dialogue through two agent networks in a dialogue learning system;

[0008] obtaining, in a manner of reinforcement learning, a state transition operator corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks; the state transition operator comprises an observation value of the agent network on a dialogue environment in a round of dialogue, a dialogue action of the agent network in a round of dialogue, and a reward value of the dialogue actions of the two agent networks in a round of dialogue; the dialogue environment is used to indicate a dialogue situation between the two agent networks;

[0009] based on the state transition operator corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks, obtaining a loss function value of each of the two agent networks;

[0010] based on the loss function value of each of the two agent networks, respectively updating parameters of the two agent networks.

[0011] In another aspect, a dialogue processing device is provided, and the device comprises:

[0012] a dialogue module, configured to perform at least one round of dialogue between two agent networks in a dialogue learning system;

[0013] an operator obtaining module, configured to obtain, in a manner of reinforcement learning, a state transition operator corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks; the state transition operator includes an observation value of the dialogue environment for the agent network in a round of dialogue, a dialogue action of the agent network in the round of dialogue, and a reward value of the dialogue actions of the two agent networks in the round of dialogue; the dialogue environment is used to indicate a dialogue situation between the two agent networks;

[0014] a loss obtaining module, configured to obtain, based on the state transition operator corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks, a loss function value of each of the two agent networks;

[0015] a parameter updating module, configured to perform parameter updating on the two agent networks respectively based on the loss function value of each of the two agent networks.

[0016] In a possible implementation, the loss obtaining module is configured to,

[0017] for the i-th round of dialogue in the at least one round of dialogue, obtain, based on the state transition operator corresponding to the i-th round of dialogue for each of the two agent networks, a value function approximation value corresponding to the i-th round of dialogue for each of the two agent networks;

[0018] obtain, based on the value function approximation value corresponding to the i-th round of dialogue for each of the two agent networks, a hybrid value function approximation value of the i-th round of dialogue;

[0019] obtain, based on the hybrid value function approximation value of the i-th round of dialogue and the value function approximation value corresponding to the i-th round of dialogue for each of the two agent networks, the loss function value corresponding to the i-th round of dialogue for each of the two agent networks.

[0020] In a possible implementation, the loss obtaining module is configured to fuse, in a manner of a hypernetwork, the value function approximation value corresponding to the i-th round of dialogue for each of the two agent networks to obtain the hybrid value function approximation value of the i-th round of dialogue.

[0021] In a possible implementation, the hypernetwork is a network based on a QMIX framework.

[0022] In a possible implementation, the two agent networks include a user agent network and a dialogue system agent network; a round of dialogue includes one dialogue action of the user agent network and one dialogue action of the dialogue system agent network.

[0023] The operator obtaining module is configured to,

[0024] For the i th round of dialogue in the at least one round of dialogue, based on a first observation value of the dialogue environment at the end of the i-1 th round of dialogue by the user agent network, a first dialogue action of the user agent network is obtained by a user policy in the user agent network;

[0025] Based on a second observation value of the dialogue environment after the first dialogue action is executed by the dialogue system agent network, a second dialogue action of the dialogue system agent network is obtained by a system policy in the dialogue system agent network;

[0026] A reward value corresponding to the i th round of dialogue is obtained, which is output by the environment system after the dialogue action of the dialogue system agent network in the i th round of dialogue is executed;

[0027] Based on a third observation value of the dialogue environment after the second dialogue action is executed by the user agent network, a third dialogue action of the user agent network is obtained by the user policy;

[0028] A fourth observation value of the dialogue environment after the third dialogue action is executed by the dialogue system agent network is obtained;

[0029] A first state transition operator corresponding to the i th round is obtained by the user agent network, and the first state transition operator includes the first observation value, the first dialogue action, the reward value corresponding to the i th round of dialogue, and the third observation value;

[0030] A second state transition operator corresponding to the i th round is obtained by the dialogue system agent network, and the second state transition operator includes the second observation value, the second dialogue action, the reward value corresponding to the i th round of dialogue, and the fourth observation value.

[0031] In a possible implementation, the operator obtaining module is configured to,

[0032] Based on the first observation value, the user dialogue information is queried from a user target database by the user policy;

[0033] The first dialogue action is obtained based on the queried user dialogue information.

[0034] In a possible implementation, in response to that the dialogue learning system is applied to the medical field, the user target database includes user constraints and user requests;

[0035] The user constraint is used to indicate a named entity in the medical field; and the user request is used to indicate a dialogue request in the medical field.

[0036] In a possible implementation, the first observation value includes: a dialogue action of the dialogue system agent network in the i-1th round of dialogue, a dialogue action of the user agent network in the i-1th round of dialogue, a target state used to indicate remaining constraints and requests, and an inconsistency vector between a response of the dialogue system agent network and the user constraint.

[0037] In a possible implementation, the operator obtaining module is configured to,

[0038] Based on the second observation value, query system dialogue information from the system database through the system policy;

[0039] Obtain the second dialogue action based on the queried system dialogue information.

[0040] In a possible implementation, in response to the dialogue learning system being applied to the medical field, the system database contains named entities in the medical field, and related information of the named entities in the medical field.

[0041] In a possible implementation, the second observation value includes:

[0042] The first dialogue action, the dialogue action of the user agent network in the i-1th round of dialogue, a belief state of a constraint time slot and a request time slot provided by the user agent network, and an embedding vector of a number of query results from the system database.

[0043] In a possible implementation, the apparatus further includes:

[0044] The pre-training module is configured to, before the dialogue module performs at least one round of dialogue through two agent networks in the dialogue learning system, pre-train the two agent networks through dialogue sample texts.

[0045] The dialogue sample texts contain texts of at least one round of dialogue.

[0046] In still another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the dialogue processing method described above.

[0047] In still another aspect, a computer readable storage medium is provided, which stores at least one computer instruction, which is loaded and executed by a processor to implement the dialogue processing method described above.

[0048] In yet another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the above dialogue processing method.

[0049] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0050] By setting the dialogue learning system with two agent networks, at least one round of dialogue generated between the two agent networks is used to obtain the state transition operator of the two agent networks in each round of dialogue through reinforcement learning. When calculating the loss function value, the loss function value of each of the two agent networks is obtained through the state transition operator of each of the two agent networks in each round of dialogue, and each of the two agent networks is trained through the loss function value of each of the two agent networks. Therefore, by introducing the state corresponding to each of the two agent networks in the dialogue learning system, the reward generated by the dialogue can be decomposed, and different agent networks can be trained more targetedly, thereby improving the accuracy of the trained dialogue system agent network.

[0051] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0053] Figure 1 is a system configuration diagram of a dialogue processing system related to various embodiments of the present application;

[0054] Figure 2 is a flowchart of a dialogue processing method according to an exemplary embodiment;

[0055] Figure 3 is a flowchart of a dialogue processing method according to an exemplary embodiment;

[0056] Figure 4 is Figure 3 is a whole scheme diagram of the interaction between the two agent networks related to the embodiments shown;

[0057] Figure 5 is a structural block diagram of a dialogue processing apparatus according to an exemplary embodiment;

[0058] Figure 6 is a structural schematic diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] The exemplary embodiments will be described in detail herein below with reference to the drawings. The following description is with reference to the drawings, wherein like numerals refer to like elements throughout. The embodiments described in the following exemplary embodiments do not represent all of the implementations in accordance with this application. Rather, they are merely examples of apparatus and methods in accordance with aspects of this application as detailed in the appended claims.

[0060] Before various embodiments shown in this application are described, several concepts related to this application are introduced.

[0061] Reference is made to Figure 1 , which shows a system structure diagram of a dialogue processing system related to various embodiments of this application. As shown in Figure 1 , the system includes a terminal 140 and a server 160; optionally, the system can also include a database 180.

[0062] The terminal 140 can be a terminal device with certain processing capabilities and interface display functions, for example, the terminal 140 can be a mobile phone, a tablet computer, an e-book reader, smart glasses, a laptop computer, a desktop computer, a dialogue terminal, etc.

[0063] The terminal 140 can include a terminal used by a developer or a user, for example, in the medical field, the terminal 140 can be a terminal used by a developer or a patient.

[0064] When the terminal 140 is implemented as a terminal used by a developer, the developer can develop a machine learning model for dialogue through the terminal 140, and deploy the machine learning model to the server 160 or a terminal used by a user.

[0065] When the terminal 140 is implemented as a terminal used by a user (such as a patient), a dialogue application program can be installed in the terminal 140, and after the terminal 140 obtains a dialogue input by a user through voice or text, the terminal 140 can call a dialogue model through the above-mentioned application program to respond to the dialogue of the user.

[0066] The server 160 can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0067] For example, when the scheme shown in the present application is applied to the medical field, it can be implemented as part of a medical cloud. The medical cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on cloud computing, mobile technology, multimedia, wireless communication, big data, and Internet of Things technologies, and combining medical technology to achieve sharing of medical resources and expansion of medical scope. Because of the combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, the current hospital appointment registration, electronic medical records, medical insurance, and the like are all products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and global layout.

[0068] The server 160 described above can be a server that provides background services for the application installed in the terminal 140. The background server can be version management of the application, background processing of user conversations obtained by the application and returning of response results, background training of machine learning models developed by developers, and the like.

[0069] The database 180 described above can be a Redis database, or it can also be other types of databases. The database 180 is used to store various types of data.

[0070] Optionally, the terminal 140 and the server 160 are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0071] Optionally, the system can further include a management device (not shown) connected to the server 160 through a communication network. Optionally, the communication network is a wired network or a wireless network. Figure 1

[0072] ​Optionally, the wireless networks or wired networks described above use standard communications technologies and / or protocols. The networks typically carry Internet traffic, but can also include private networks, such as LANs (Local Area Network), MANs (Metropolitan Area Network), WANs (Wide Area Network), wireless networks, wired networks, virtual private networks, etc. In some embodiments, the data exchanged over the networks is represented using technologies and / or formats including, but not limited to, HTML (Hyper Text Mark-up Language), XML (Extensible Markup Language), etc. In addition, the data can be encrypted using any of several techniques, such as SSL (Secure Socket Layer), TLS (Transport Layer Security), VPN (Virtual Private Network), IPsec (Internet Protocol Security), etc. In other embodiments, technologies other than those specifically described can be employed.

[0073] Figure 2 is a flowchart of a dialogue processing method according to an example embodiment. The method can be performed by a computer device, such as a server or a terminal, or a combination of a server and a terminal. The server can be the server 160 in the embodiment shown above, and the terminal can be the terminal 140 in the embodiment shown above. As shown in the flowchart, the dialogue processing method can include the following steps. Figure 1 Figure 1 Figure 2

[0074] Step 201, performing at least one round of dialogue between two agent networks in a dialogue learning system.

[0075] Step 202, obtaining, in a manner of reinforcement learning, a state transition operator corresponding to each of the at least one round of dialogue for each of the two agent networks; the state transition operator includes an observation value of the dialogue environment by the agent network in a round of dialogue, a dialogue action of the agent network in the round of dialogue, and a reward value of the dialogue action of the two agent networks in the round of dialogue; the dialogue environment is used to indicate a dialogue situation between the two agent networks.

[0076] ​​​At step 203, loss function values of the two agent networks are obtained based on the state transition operators of the two agent networks corresponding to each of the at least one round of dialogue.

[0077] In a possible implementation, the computer device can obtain the loss function values of the two agent networks based on system state information and the state transition operators of the two agent networks corresponding to each of the at least one round of dialogue, wherein the system state information is used to indicate a global state of the dialogue learning system.

[0078] At step 204, the two agent networks are respectively updated in parameters based on the loss function values of the two agent networks.

[0079] To sum up, the scheme shown in the embodiments of the present application can obtain the state transition operators of the two agent networks in each round of dialogue through the at least one round of dialogue generated between the two agent networks by using the reinforcement learning, and can obtain the loss function values of the two agent networks in each round of dialogue through the state transition operators of the two agent networks in each round of dialogue when calculating the loss function values, and can train the two agent networks respectively through the loss function values of the two agent networks, so as to decompose the rewards generated by the dialogue through the states corresponding to the two agent networks in the dialogue learning system, and can train the different agent networks more targetedly, thereby improving the accuracy of the trained dialogue system agent networks.

[0080] In a possible implementation, the scheme shown in the embodiments of the present application can be implemented based on the AI (Artificial Intelligence) technology, and the reinforcement learning can be used to train the agent networks.

[0081] AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions.

[0082] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning technologies.

[0083] The scheme shown in the embodiments of the present application is mainly applied to artificial neural networks, reinforcement learning, speech processing technology, natural language processing technology and other technologies in the field of artificial intelligence.

[0084] The scheme shown in the embodiments of the present application can be applied to any scenario that needs to provide services for users through dialogues, such as medical dialog scenarios (e.g., AI guide diagnosis, intelligent pre-diagnosis, intelligent medication assistant and other intelligent medical dialog scenarios).

[0085] For example, taking the AI guide diagnosis scenario in the medical field as an example, through the scheme shown in the embodiments of the present application, a patient agent and a medical dialog system agent can be set, and an AI guide diagnosis related dialogue can be performed between the patient agent and the medical dialog system agent, and the loss functions of the two agents are calculated according to the dialogue results and the global state, and the two agents are trained respectively. After training is completed, the medical dialog system agent is deployed to a server or a terminal, and subsequent user's inquiry dialogue can be answered by the medical dialog system agent to provide AI guide diagnosis services, such as recommending relevant medical departments to the user through the user's input dialogue information such as symptoms.

[0086] In the related art, a dialogue system is usually reinforced training through a user simulator, and it is often challenging to design a reliable user simulator, which is equivalent to building a good dialogue agent. Among them, with the growing demand for dialogue systems to handle more complex tasks, it is very difficult to build a user simulator based entirely on rules, which requires a lot of domain expertise. Especially in the auxiliary diagnosis scene, the user simulator needs to comprehensively and realistically simulate all kinds of patients, but building a realistic patient requires a lot of medical knowledge, such as the symptoms of different diseases, the degree of manifestation of symptoms, etc. The complex dialogue logic in the auxiliary diagnosis process not only makes it difficult to design a rule-based patient simulator, but also requires high maintenance costs. At the same time, the multi-agent joint training based solution is difficult to directly transplant to the medical auxiliary diagnosis dialogue scene. An important reason is that different roles in the auxiliary diagnosis scene have different task goals, for example, doctors need to efficiently inquire about symptoms and make accurate diagnoses, while patients focus on accurately answering the doctor's inquiries. Therefore, it is difficult to accurately model the patient using multi-agent with shared rewards, thereby hindering the system's learning strategy performance.

[0087] To solve the above problems, the present application uses a multi-agent framework based on a role-aware reward decomposition mechanism to jointly model the doctor-patient consultation system to reduce the artificial dependence of the model:

[0088] 1) The present application proposes a multi-agent dialogue strategy learning, which regards the user as another dialogue agent rather than a user simulator. The dialogue between the user and the dialogue system is modeled as a collaborative interaction process, in which both agents learn their own strategies simultaneously.

[0089] 2) The present application estimates the overall reward value as a complex nonlinear combination of each agent value, which is conditioned only on local observations, automatically decomposes the reward for the two agents, and promotes them to coordinate their behaviors while acting in a decentralized manner. At the same time, the two agents learn in a centralized manner to obtain global state information and remove communication restrictions.

[0090] 3) At the same time, in order to further alleviate the non-stationary problem in multi-agent learning, the present application realizes the proposed consultation system based on shared hidden layers and input mutual transmission.

[0091] Figure 3 is a flowchart of a dialogue processing method according to an example embodiment. The method can be executed by a computer device, for example, the computer device can be a server, or the computer device can also be a terminal, or the computer device can include a server and a terminal, wherein the server can be the server described above Figure 1The terminal can be the user terminal 140 in the embodiment shown above Figure 1 The terminal 140 in the embodiment shown above. As Figure 3 The dialogue processing method can include the following steps.

[0092] Step 301, pre-training two agent networks in the dialogue learning system through dialogue sample text; wherein the dialogue sample text contains text of at least one round of dialogue.

[0093] The scheme shown in the embodiments of the present application is applied to the training scene of a doctor-patient dialogue system. When processing a complex auxiliary diagnosis dialogue task of multiple diseases, the action space of the strategy can be very large. Taking the above two agent networks as examples, the patient agent (i.e. the user agent) and the system agent (i.e. the agent network corresponding to the dialogue system), the training process can be divided into three stages: first, pre-training the dialogue strategy using a conversation corpus, then interacting with the doctor-patient agent based on simple rules and collecting experience for preliminary iteration, and finally improving the pre-trained strategy and the rule-based optimization strategy using reinforcement learning.

[0094] Among them, the goal of pre-training is to make the state-action trajectory distribution generated by the model match the trajectory distribution of the labeled corpus (i.e. the dialogue sample text). For example, in the embodiments of the present application, the maximum entropy loss function can be used for model parameter update in the pre-training process, and the optimization goal of the pre-training process of the present application can be:

[0095]

[0096]

[0097] Where L BC is the maximum entropy loss function of behavior cloning, and are the labeled corpus of the system agent and the patient agent, represent the prediction model, and θ · is its parameter. s p ,a p are the state and action of the patient agent in the pre-training process, s s ,a s are the state and action of the system agent in the pre-training process.

[0098] Step 302, at least one round of dialogue is performed through the two agent networks in the dialogue learning system.

[0099] Taking a training scene of a doctor-patient conversation system as an example, when a pre-training strategy and a rule-based optimization strategy are improved by using reinforcement learning, at least one round of conversation can be performed by a patient agent and a system agent, which can also be referred to as at least one round of conversation, wherein the patient agent generates a sentence (i.e., a conversation action), and the system agent generates a sentence in reply to the sentence, which can be referred to as one round of conversation.

[0100] In step 303, state transition operators corresponding to each round of conversation in the at least one round of conversation are obtained for the two agent networks in a manner of reinforcement learning; the state transition operators include an observation value of the agent networks on a conversation environment in a round of conversation, a conversation action of the agent networks in the round of conversation, and a reward value of the conversation actions of the two agent networks in the round of conversation; the conversation environment is used to indicate a conversation situation between the two agent networks.

[0101] In a possible implementation, the two agent networks include a user agent network and a conversation system agent network; a round of conversation includes one conversation action of the user agent network and one conversation action of the conversation system agent network.

[0102] The state transition operators corresponding to each round of conversation in the at least one round of conversation are obtained for the two agent networks in a manner of reinforcement learning, including:

[0103] For the i th round of conversation in the at least one round of conversation, a first conversation action of the user agent network is obtained based on a first observation value of the user agent network on a conversation environment at the end of the (i-1) th round of conversation by a user policy in the user agent network;

[0104] A second conversation action of the conversation system agent network is obtained based on a second observation value of the conversation system agent network on a conversation environment after the first conversation action is performed by a system policy in the conversation system agent network;

[0105] A reward value corresponding to the i th round of conversation is obtained, which is output by an environment system after the conversation action of the conversation system agent network in the i th round of conversation is performed;

[0106] A third conversation action of the user agent network is obtained based on a third observation value of the user agent network on a conversation environment after the second conversation action is performed by the user policy;

[0107] A fourth observation value of the conversation system agent network on a conversation environment after the third conversation action is performed is obtained;

[0108] A first state transition operator corresponding to the i th round of conversation is obtained for the user agent network, and the first state transition operator includes the first observation value, the first conversation action, the reward value corresponding to the i th round of conversation, and the third observation value;

[0109] obtain a second state transition operator corresponding to the i th round of the dialogue system agent network, the second state transition operator comprising a second observation value, a second dialogue action, a reward value corresponding to the i th round of dialogue, and a fourth observation value.

[0110] In a possible implementation, based on the first observation value of the dialogue environment at the end of the i-1 th round of dialogue by the user agent network, the first dialogue action of the user agent network is obtained through a user policy in the user agent network, comprising:

[0111] Based on the first observation value, the user dialogue information is queried from the user target database through the user policy;

[0112] Based on the queried user dialogue information, the first dialogue action is obtained.

[0113] In a possible implementation, in response to the dialogue learning system being applied to the medical field, the user target database comprises user constraints and user requests;

[0114] The user constraints are used to indicate named entities in the medical field, and the user requests are used to indicate dialogue requests in the medical field.

[0115] In a possible implementation, the first observation value comprises: a dialogue action of the dialogue system agent network in the i-1 th round of dialogue, a dialogue action of the user agent network in the i-1 th round of dialogue, a target state used to indicate remaining constraints and requests, and an inconsistency vector between a response of the dialogue system agent network and the user constraints.

[0116] In a possible implementation, based on the second observation value of the dialogue environment after the first dialogue action is executed by the dialogue system agent network, the second dialogue action of the dialogue system agent network is obtained through a system policy in the dialogue system agent network, comprising:

[0117] Based on the second observation value, the system dialogue information is queried from the system database through the system policy;

[0118] Based on the queried system dialogue information, the second dialogue action is obtained.

[0119] In a possible implementation, in response to the dialogue learning system being applied to the medical field, the system database comprises named entities in the medical field and related information of the named entities in the medical field.

[0120] In a possible implementation, the second observation value comprises:

[0121] the first dialogue action, the dialogue action of the user agent network in the i-1th round of dialogue, the belief state of the constraint time slot and the request time slot provided by the user agent network, and the embedding vector of the number of query results from the system database.

[0122] In step 304, the loss function values of the two agent networks are obtained based on the state transition operators of the two agent networks corresponding to each of the at least one round of dialogue.

[0123] In a possible implementation, the computer device can obtain the loss function values of the two agent networks based on system state information and the state transition operators of the two agent networks corresponding to each of the at least one round of dialogue; and the system state information is used to indicate the global state of the dialogue learning system.

[0124] In the embodiments of the present application, in order to simplify the complexity of the training process, the system state information can be set as a constant.

[0125] In a possible implementation, the loss function values of the two agent networks are obtained based on system state information and the state transition operators of the two agent networks corresponding to each of the at least one round of dialogue, and the method comprises the following steps.

[0126] For the i th round of dialogue in the at least one round of dialogue, the value function approximation values of the two agent networks corresponding to the i th round of dialogue are obtained based on the state transition operators of the two agent networks corresponding to the i th round of dialogue.

[0127] The hybrid value function approximation value of the i th round of dialogue is obtained based on the system state information corresponding to the i th round of dialogue and the value function approximation values of the two agent networks corresponding to the i th round of dialogue.

[0128] The loss function values of the two agent networks corresponding to the i th round of dialogue are obtained based on the hybrid value function approximation value of the i th round of dialogue and the value function approximation values of the two agent networks corresponding to the i th round of dialogue.

[0129] In a possible implementation, the hybrid value function approximation value of the i th round of dialogue is obtained based on the system state information corresponding to the i th round of dialogue and the value function approximation values of the two agent networks corresponding to the i th round of dialogue, and the method comprises the following steps.

[0130] The system state information corresponding to the i th round of dialogue and the value function approximation values of the two agent networks corresponding to the i th round of dialogue are fused through a super network to obtain the hybrid value function approximation value of the i th round of dialogue.

[0131] In a possible implementation, the super network is a network based on a QMIX (hybrid Q network) framework.

[0132] Step 305, based on the respective loss function values of the two agent networks, respectively updating the parameters of the two agent networks.

[0133] Taking the training of a dialogue system in the medical field as an example, the technical solution can be applied to a dialogue policy learning scene based on sequence decision. The present application regards the dialogue policy learning of the doctor-patient pair as a Markov decision process problem, and improves the strategy by using the multi-agent reinforcement learning framework QMIX. First, the patient goal G=(C, R) composed of patient constraints C (such as various explicit and implicit symptom descriptions) and requests R (such as disease queries) is given, and an external database containing all candidate entities and corresponding information is given, and the patient agent and the system agent interact with each other in the dialogue session to achieve the patient goal. There can be multiple domains in G, and the two agents can complete all subtasks in each domain. Both agents can partially observe the environment, i.e. only the patient agent uses the above-mentioned patient goal G, and only the dialogue system agent uses the above-mentioned external database. The method of understanding each other's information is through dialogue interaction. Unlike ordinary multi-agent task settings, the two agents in the dialogue box of the present application are executed asynchronously. In one dialogue round, the patient agent first issues a query, and then the system agent returns a response, and the two agents communicate alternately. Therefore, according to the patient agent strategy μ(a p |s p ) and the system agent strategy π(a s |s s ), each dialogue session can be regarded as a trajectory of state-action pairs:

[0134]

[0135] In the present application, an interaction unit can be activated by the patient agent. For example, first, the patient agent observes the environment (wherein the definition of the environment in reinforcement learning is the physical world describing the Markov decision process; for example, in the embodiment of the present application, the patient agent observing the environment here can mean that the patient agent observes the patient in the physical world of the Markov decision process) to obtain the initial state Then perform the action using the ε-greedy greedy exploration This exploration selects a random action with a probability of ε, or the patient agent can also adopt a greedy strategy Determine the action, where Q(s, a; θ p ) is the approximate value function of the patient agent, realized as a multilayer perceptron parameterized by θ p .

[0136] Then the system agent starts to update the state of the environment according to the state Give a random action with a probability of ε Or action Where Q(s,a; θ) s () is an approximate function of the system agent. The environment then updates its state based on the responses from both parties. And give the reward r for the current round. 0 Agents handle state transitions separately. and This loop continues until the dialogue ends. Q(s,a′:θ) s ) and Q(s,a′:θ p The parameters of ) are updated by minimizing the following squared loss using stochastic gradient descent:

[0137]

[0138] (s s a s ,r,s' s )~D s

[0139]

[0140] Where E is the expected value, r is the reward value, γ∈[0,1] is the discount factor, and Q′(·θ) t ) is the QMIX function, Q′(·) is the objective value function that is updated only periodically, and θ p yes (Patient network) parameters, θ s These are parameters of the system agent, θ t It is the reward distribution network in QMIX; s p It refers to the status of the patient network, a p It refers to the dialogue behavior of patients on the network, s s It refers to the status of the system's proxy network, a s It refers to the dialogue behavior of the system's proxy network.

[0141] This application provides a multi-agent reinforcement learning-based strategy for jointly learning patient and system policies. First, a small number of real doctor-patient dialogues are collected and stored in a data pool D. i Then, imitation learning is used to pre-train Q(s,a;θ). s ) and Q(s,a;θ p The action space for patient and system strategies can be very large, and this application deals with multi-domain and complex auxiliary diagnostic dialogue tasks. Therefore, in addition to pre-training, this application uses a rule-based doctor-patient agent for warm-start before reinforcement learning optimization. Subsequently, multi-agent reinforcement learning of the policies of both parties is employed. Please refer to [reference needed]. Figure 4 This diagram illustrates the overall scheme of two agents interacting through dialogue behavior, as described in an embodiment of this application. Figure 4As shown, in each round of dialogue, the patient agent 41 and the system agent 42 observe the state of the environment 43 respectively, and make corresponding actions (i.e. generate the dialogue sentence of this round), at the same time, the environment also outputs the reward value r, the state observed by the two agents in each round of dialogue, the action performed by each agent and the reward value r are input into a hybrid network 44 for reward distribution, and the loss function value of each agent is obtained, and each agent is updated through the loss function value.

[0142] The present application can select a deep Q network (DQN) framework to learn a deterministic dialogue strategy. DQN is a deep reinforcement learning algorithm that uses an experience replay mechanism to store training data, then randomly samples from it for training, and updates the Q network. Since the two agents succeed by cooperation, but their roles in the dialogue are asymmetric, the present application can combine a QMIX network Q(s, a; θ t ) to guide the decomposition of task rewards into different parts for better policy learning. Please note that the method of the present application is completely data-driven, without the need to build a user simulator in advance, and does not require any supervision from others during training.

[0143] where the system policy decides the system operation a to provide appropriate responses to the user agent according to the system dialogue state s s . The system dialogue state s s is the concatenation of the following information: 1) the last system action (i.e. the last system action); 2) the last patient action (i.e. the last patient action); 3) the goal state representing the remaining constraints and the remaining requests that need to be sent; where the remaining constraints refer to the constraints in the patient goal G that have not been used, and the remaining requests refer to the requests in the patient goal G that have not been used; 4) the inconsistency vector, which represents the inconsistency between the system response (i.e. the response of the system agent) and the user constraints C. Each system action a is a subset of the dialogue action set A, because there can be multiple intents in one dialogue round. The dialogue act is an abstract representation of the intent, which can be represented by a four-tuple consisting of domain, intent, slot type and slot value (e.g. [disease diagnosis, notification, temperature, 38°]). In practice, the dialogue act is delexicalized in the dialogue policy. The present application replaces the slot value with a placeholder and refills it with a true value according to the entity selected from the external database, so that the system can operate on the invisible value. The system state of the dialogue round t includes: the user operation of the current round, the system operation of the last round (the last round), the belief state of tracking the constraint slots and request slots provided by the user agent, and the concatenation of the embedding vector of the number of query results from the database.

[0144] The user policy The patient operation is determined according to the patient dialogue state, to express the constraints and requests to the system agent. Similar to the system strategy, the patient strategy uses the delexicalized pair of dialogue actions, and refills the values according to the user target G. In addition to predicting dialogue behavior, the user strategy also outputs a termination signal T (which is used to determine whether the dialogue is terminated).

[0145] In the embodiments of the present application, in the multi-agent doctor-patient agent joint learning framework, the doctor and the patient coordinate their behaviors according to each other, and make decisions in a decentralized manner. In order to share global state information and remove communication restrictions, a centralized training agent can be used. Learning a joint state action value function conditioned on each state information is an effective way of centralized learning, and the solution of the present application is to use a value-based policy network QMIX for centralized learning to obtain distributed doctor-patient strategies. QMIX uses a network that estimates the joint action value as a complex nonlinear combination of each agent's value, and the combination is only conditioned on local observations. The present application enforces the monotonicity of the joint action value in each agent's value in structure, which allows the maximization of the joint action value in non-strategic learning and guarantees the consistency between centralized and decentralized strategies. Because argmax of the joint action value function is equivalent to argmax of each local action value function, its monotonicity is the same. Therefore, the distributed strategy is greedy to obtain the optimal action through Q p and Q s QMIX is based on the following monotonicity constraint:

[0146]

[0147] where, is the partial derivative symbol, and the above Q p i.e. Q s i.e. Q t i.e.

[0148] In order to realize the above constraint to ensure the non-negativity of the weight, QMIX uses a super network to input the constant vector V, and uses a linear network and an absolute value activation function to ensure that the output parameter is not negative. The same way is used for the offset, but there is no constraint of non-negativity, and the offset of the last layer of the mixed network is obtained through a two-layer network and a ReLU activation function. The non-linear mapping network. Among them, the QMIX cost function is as shown in the above formula (1).

[0149] In one exemplary embodiment of this application, the user agent network and the system agent network are implemented by sharing a hidden layer. That is, for the user agent network and the system agent network, the output layers of the two agent networks are different, and the part before the output layer of the two agent networks is the same, or the two agent networks share the network part before the output layer.

[0150] The training of the aforementioned model is divided into three stages: first, pre-training with behavior cloning; second, hot-starting based on rule-based agents; and finally, further optimization of the policy using reinforcement learning. This application uses DQN as the basic reinforcement learning model. First, an ε-greedy greedy exploration policy is run to collect experience samples, and then QMIX is used to train the Q-network of the doctor and patient. The ε-greedy greedy exploration policy means that when the agent makes a decision, there is a small positive probability ε (less than 1) of randomly selecting an unknown action, and a probability of 1- of selecting the action with the highest value among the actions already considered.

[0151] The training process for the model can be as follows:

[0152] Input: N, M, ε, θ s θ p θ t L, D s D p D i ,r,Z;

[0153] Output:

[0154] 1: Initialize experience pool D p D p Empty.

[0155] 2: Initialize the labeled corpus D i Fill in the data with real doctor-patient dialogues marked with status and actions.

[0156] 3: In D i Pre-training

[0157] 4: Initialization For θ′ s =θ s , θ′ t =θ t , θ′ p =θ p .

[0158] 5: forn = 1: M, then,

[0159] 6: Use rule-based doctor-patient agents for warm starts.

[0160] 7: endfor

[0161] 8: for n = 1 : N, then,

[0162] 9: initialize patient goal C and patient state s p .

[0163] 10: with probability ε randomly select an action a p .

[0164] 11: else, select action a using a greedy strategy p .

[0165] 12: patient agent performs action a p , system agent observes s s .

[0166] 13: repeat

[0167] 14: with probability ε randomly select an action a s .

[0168] 15: else, select action a using a greedy strategy s .

[0169] 16: system agent performs a s , patient agent observes s' p .

[0170] 17: agents jointly observe a reward value r.

[0171] 18: store transition operator (s p , a p , r, s' p ) to D p .

[0172] 19: with probability ε randomly select an action a p .

[0173] 20: else select action a using a greedy strategy p .

[0174] 21: patient agent performs action a p , system agent observes s' s .

[0175] 22: store transition operator (s s , a s , r, s' s ) to D s .

[0176] 23: observe a termination signal from the environment.

[0177] 24: until T is a termination signal.

[0178] 25: sample a batch (s s , a s , r, s' s ) from D s .

[0179] 26: sample a batch (s p , a p , r, s' p ) from D p .

[0180] 27: update

[0181] 28: update

[0182] 29: endfor

[0183] 30: return

[0184] In the evaluation stage, the success rate is mainly used for evaluation. A successful dialogue is defined as the dialogue agent answering all the patient's questions and making the correct disease diagnosis. On the one hand, the dynamic change of the success rate during the training process, i.e., the learning rate, is used as a test index, and on the other hand, the success rate of the final convergence of the model is looked at. In addition, the average number of dialogue rounds is also tested, and the better the dialogue agent can complete the dialogue task in a shorter number of rounds. For all the above test indexes, the application is tested multiple times to obtain the mean value.

[0185] The above scheme is implemented based on the reinforcement learning dialogue model and related medical knowledge set, wherein the medical knowledge needs to be continuously accumulated and labeled, and therefore, when the data is insufficient, a small amount of data can be used to train a prototype system, and the medical and patient agents in the system simultaneously update their respective strategies in the interaction process. At the same time, real medical dialogue data is continuously accumulated, which can be used to build and improve more detailed and comprehensive scenarios and strategy spaces, and then the method provided by the present application is used to gradually expand the scenarios covered by the model, and finally an AI medical intelligent inquiry assistant with wide coverage and high accuracy is formed.

[0186] The technical solution aims at the problem that the existing reinforcement learning method needs to construct a patient agent model to learn the auxiliary diagnosis dialogue strategy, and proposes a multi-agent dialogue strategy scheme for simultaneously training the patient strategy and the system strategy, and the main beneficial effects include:

[0187] 1) The Multi-Agent Diagnostic Dialog Policy Learning (MADDPL) system based on the multi-agent reinforcement learning framework proposed in this solution enables developers to quickly build an auxiliary diagnostic dialog system from scratch. Two agents interact and learn together without the need for an explicit user simulator to be built in advance. MADDPL enhances the scalability of both medical and patient strategies and accelerates the landing of large-scale complex auxiliary diagnostic systems.

[0188] 2) MADDPL uses behavior cloning to pre-train based on a small amount of annotated dialog behavior corpus and uses a rule-based double-agent warm start. This allows the reinforcement learning model to obtain a better initial value before optimization, avoiding a large amount of invalid exploration, improving learning efficiency and convergence speed.

[0189] 3) This application introduces the QMIX mixed value network for role-aware reward automatic decomposition to integrate the specific role and domain knowledge of each agent in the task-oriented dialog. QMIX allows learning of a rich joint action value function that allows an automatically learnable reward decomposition mechanism by imposing monotonicity constraints on the mixing network, making the learning goal of each agent more accurate and accelerating the learning speed of the agent strategy.

[0190] 4) In order to further alleviate the non-stationary problem in multi-agent learning, the application uses a shared hidden layer and input mutual transmission based training method to train the proposed consultation system in the engineering implementation.

[0191] For example, the application collects some samples of doctor-patient conversations in a hospital, obtains a set of state-action sequences through the parser, and can quickly deploy a dialogue system for auxiliary diagnosis. The model first uses the collected expert data to perform behavior cloning and quickly learn a part of the knowledge, and then uses the scheme proposed in the patent to train the doctor-patient agent simultaneously, and then the application puts the system in a hospital or an online hospital scene, and lets it dialogue with real patients and learn online. During the dialogue process, the system agent continuously explores and learns better response strategies. At the same time, the system continuously collects evaluation information from patients, which continuously updates the learning goal of the system agent. As more and more patients are contacted, the system agent gradually learns more and more rich strategies, and gradually makes decisions and answers closer to the expert. At the same time, as the real doctor-patient data is continuously accumulated, the annotation data pool of the application is continuously enriched, and the application can correct the patient agent on the one hand, and the application can let the more mature system agent interactively guide the update iteration of the patient agent, so that it can cover a larger action space. Ideally, the doctor-patient agent can gradually explore and learn all strategies, and these strategies can be supported in practice. Finally, the model designed in the patent can "cultivate" an excellent general practitioner and an experienced patient.

[0192] The patent uses a deterministic policy model based on DQN, and other stochastic policy models can also be considered, such as actor-critic (actor-critic) algorithm, policy gradient algorithm, etc.

[0193] During the training process of the model, behavior cloning is only used to assist exploration in the initial stage, and multiple behavior cloning can also be considered to seek better initialization results. And the pre-training stage can also use other supervised learning models such as beta weight logistic regression to pre-train the network parameters.

[0194] QMIX is used in this paper to decompose the perception rewards of different agents, and other centralized multi-agent reward decomposition modes can also be used, such as Value-Decomposition Networks (VDN) for multi-agent cooperative learning, Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement learning (QTRAN), etc.

[0195] In summary, the scheme shown in the embodiments of the present application sets a dialogue learning system with two agent networks, uses at least one round of dialogue generated between the two agent networks, obtains state transition operators of the two agent networks in each round of dialogue through reinforcement learning, obtains loss function values of the two agent networks respectively through the state transition operators of the two agent networks in each round of dialogue when calculating the loss function values, and trains the two agent networks respectively through the loss function values of the two agent networks respectively, so as to decompose the rewards generated by the dialogue through the states corresponding to the two agent networks in the dialogue learning system respectively, and more accurately train different agent networks, thereby improving the accuracy of the trained dialogue system agent network.

[0196] Figure 5 is a structural block diagram of a dialogue processing apparatus according to an exemplary embodiment. The apparatus can implement Figure 2 or Figure 3 The dialogue processing apparatus includes all or part of the steps in the method provided by the embodiments shown in

[0197] The dialogue module 501 is configured to perform at least one round of dialogue through two agent networks in a dialogue learning system.

[0198] The operator obtaining module 502 is configured to obtain state transition operators of the two agent networks respectively corresponding to each round of dialogue in the at least one round of dialogue in a manner of reinforcement learning. The state transition operators include observation values of the agent networks on a dialogue environment in a round of dialogue, dialogue actions of the agent networks in a round of dialogue, and reward values of dialogue actions of the two agent networks in a round of dialogue. The dialogue environment is used to indicate a dialogue situation between the two agent networks.

[0199] The loss obtaining module 503 is configured to obtain loss function values of the two agent networks respectively based on the state transition operators of the two agent networks respectively corresponding to each round of dialogue in the at least one round of dialogue.

[0200] The parameter updating module 504 is configured to update parameters of the two agent networks respectively based on the loss function values of the two agent networks respectively.

[0201] In a possible implementation, the loss obtaining module 503 is configured to,

[0202] For the i th round of dialogue in the at least one round of dialogue, obtain value function approximation values of the two agent networks respectively corresponding to the i th round of dialogue based on the state transition operators of the two agent networks respectively corresponding to the i th round of dialogue.

[0203] obtain a hybrid value function approximation value of the i th round of dialogue based on the value function approximation values of the two agent networks corresponding to the i th round of dialogue respectively;

[0204] obtain loss function values of the two agent networks corresponding to the i th round of dialogue respectively based on the hybrid value function approximation value of the i th round of dialogue and the value function approximation values of the two agent networks corresponding to the i th round of dialogue respectively.

[0205] In a possible implementation, the loss obtaining module 503 is configured to obtain a hybrid value function approximation value of the i th round of dialogue by fusing the value function approximation values of the two agent networks corresponding to the i th round of dialogue respectively in a manner of a hypernetwork.

[0206] In a possible implementation, the hypernetwork is a network based on a QMIX framework.

[0207] In a possible implementation, the two agent networks include a user agent network and a dialogue system agent network; and one round of dialogue includes one dialogue action of the user agent network and one dialogue action of the dialogue system agent network.

[0208] The operator obtaining module 502 is configured to,

[0209] For an i th round of dialogue in the at least one round of dialogue, obtain a first dialogue action of the user agent network based on a first observation value of the dialogue environment at the end of an (i-1) th round of dialogue by a user policy in the user agent network;

[0210] obtain a second dialogue action of the dialogue system agent network based on a second observation value of the dialogue environment after the first dialogue action is executed by a system policy in the dialogue system agent network;

[0211] obtain a reward value corresponding to the i th round of dialogue output by the environment system after the dialogue action of the dialogue system agent network in the i th round of dialogue is executed;

[0212] obtain a third dialogue action of the user agent network based on a third observation value of the dialogue environment after the second dialogue action is executed by the user policy;

[0213] obtain a fourth observation value of the dialogue environment after the third dialogue action is executed by the dialogue system agent network;

[0214] obtain a first state transition operator corresponding to the i-th round of the user agent network, the first state transition operator comprising the first observation value, the first dialogue action, a reward value corresponding to the i-th round of dialogue, and the third observation value;

[0215] obtain a second state transition operator corresponding to the i-th round of the dialogue system agent network, the second state transition operator comprising the second observation value, the second dialogue action, a reward value corresponding to the i-th round of dialogue, and the fourth observation value.

[0216] In a possible implementation, the operator obtaining module 502 is configured to,

[0217] query user dialogue information from a user target database based on the first observation value and by using the user policy;

[0218] obtain the first dialogue action based on the queried user dialogue information.

[0219] In a possible implementation, in response to the dialogue learning system being applied to the medical field, the user target database comprises user constraints and user requests;

[0220] The user constraints are used to indicate named entities in the medical field, and the user requests are used to indicate dialogue requests in the medical field.

[0221] In a possible implementation, the first observation value comprises a dialogue action of the dialogue system agent network in the i-1-th round of dialogue, a dialogue action of the user agent network in the i-1-th round of dialogue, a target state used to indicate remaining constraints and requests, and an inconsistency vector between a response of the dialogue system agent network and the user constraints.

[0222] In a possible implementation, the operator obtaining module 502 is configured to,

[0223] query system dialogue information from a system database based on the second observation value and by using the system policy;

[0224] obtain the second dialogue action based on the queried system dialogue information.

[0225] In a possible implementation, in response to the dialogue learning system being applied to the medical field, the system database comprises named entities in the medical field and related information of the named entities in the medical field.

[0226] In a possible implementation, the second observation value comprises:

[0227] The first dialogue action, the dialogue action of the user agent network in the i-1th round of dialogue, the belief state of the constraint time slot and the request time slot provided by the user agent network, and the embedding vector of the number of query results from the system database.

[0228] In a possible implementation, the apparatus further includes:

[0229] A pre-training module is configured to pre-train two agent networks in a dialogue learning system through dialogue sample texts before the dialogue module 501 performs at least one round of dialogue between the two agent networks.

[0230] The dialogue sample texts include texts of at least one round of dialogue.

[0231] In summary, the scheme shown in the embodiments of the present application sets a dialogue learning system with two agent networks, uses at least one round of dialogue generated between the two agent networks, obtains the state transition operator of each agent network in each round of dialogue through reinforcement learning, obtains the loss function value of each agent network through the state transition operator of each agent network in each round of dialogue when calculating the loss function value, and trains each agent network through the loss function value of each agent network, thereby decomposing the reward generated by the dialogue through the state corresponding to each agent network in the dialogue learning system, and more accurately training different agent networks, thereby improving the accuracy of the trained dialogue system agent network.

[0232] Figure 6 is a structural schematic diagram of a computer device according to an example embodiment. The computer device 600 includes a central processing unit (CPU) 601, a system memory 604 including a random access memory (RAM) 602 and a read-only memory (ROM) 603, and a system bus 605 connecting the system memory 604 and the central processing unit 601. The computer device 600 further includes a basic input / output system 606 to help transfer information between various devices in the computer, and a mass storage device 607 for storing an operating system 613, application programs 614 and other program modules 615.

[0233] The mass storage device 607 is connected to the central processing unit 601 through a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer readable medium provide nonvolatile storage for the computer device 600. That is, the mass storage device 607 can include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0234] In the interest of not unduly repeating history, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media described herein can represent any type of computer readable medium for storing data that is accessible by a computer.

[0235] The computer device 600 can be connected to the Internet or other network through a network interface unit 611 connected to the system bus 605.

[0236] The memory also includes one or more programs stored therein, which one or more programs, when executed by the central processing unit 601, implement Figure 2 or Figure 3 any of the methods shown.

[0237] In exemplary embodiments, there is also provided a non-transitory computer readable storage medium including instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to perform the methods shown in the various embodiments of the present application. For example, the non-transitory computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0238] In an example embodiment, a computer program product or computer program including computer instructions stored on a computer readable storage medium is also provided. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods illustrated in the various embodiments described above.

[0239] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0240] It is to be understood that the application is not limited to the precise details of design or construction that have been described above and illustrated in the drawings. Various modifications and changes can be made thereunto without departing from the scope of the application. The scope of the application is indicated by the appended claims, rather than by the foregoing description.

Claims

1. A dialogue processing method, characterized in that, The method includes: At least one round of dialogue is conducted through two agent networks in the dialogue learning system; The state transition operators for each of the two agent networks in each of the at least one round of dialogue are obtained using reinforcement learning. The state transition operators include the agent network's observation of the dialogue environment in a round of dialogue, the agent network's dialogue actions in a round of dialogue, and the reward values ​​for the dialogue actions of the two agent networks in a round of dialogue. The dialogue environment is used to indicate the dialogue situation between the two agent networks. Based on the state transition operators corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks, obtain the loss function values ​​of the two agent networks respectively; Based on the loss function values ​​of the two agent networks, the parameters of the two agent networks are updated respectively.

2. The method according to claim 1, characterized in that, The step of obtaining the loss function values ​​of the two proxy networks based on the state transition operators corresponding to each round of the at least one round of dialogue includes: For the i-th round of dialogue in the at least one round of dialogue, based on the state transition operators of the two agent networks corresponding to the i-th round of dialogue, the approximate values ​​of the value functions of the two agent networks corresponding to the i-th round of dialogue are obtained. Based on the approximate value of the value function corresponding to the i-th round of dialogue for each of the two agent networks, the approximate value of the mixed value function for the i-th round of dialogue is obtained; Based on the approximate value of the hybrid value function of the i-th round of dialogue, and the approximate value of the value function of each of the two agent networks corresponding to the i-th round of dialogue, the loss function values ​​of each of the two agent networks corresponding to the i-th round of dialogue are obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the mixed value function approximation value of the i-th round of dialogue based on the value function approximation values ​​of the two agent networks corresponding to the i-th round of dialogue includes: By using a hypernetwork approach, the approximate value of the value function corresponding to the i-th round of dialogue is fused between the two agent networks to obtain the approximate value of the mixed value function for the i-th round of dialogue.

4. The method according to claim 3, characterized in that, The hypernetwork is a network based on the QMIX framework.

5. The method according to claim 1, characterized in that, The two proxy networks include a user agent network and a dialogue system proxy network; a round of dialogue includes a dialogue action of the user agent network and a dialogue action of the dialogue system proxy network. The step of obtaining the state transition operators for each of the two agent networks corresponding to each round of the at least one round of dialogue using reinforcement learning includes: For the i-th round of the at least one round of dialogue, based on the first observation of the dialogue environment at the end of the (i-1)-th round of dialogue by the user agent network, the first dialogue action of the user agent network is obtained through the user policy in the user agent network; Based on the second observation value of the dialogue environment after the first dialogue action is executed by the dialogue system agent network, the second dialogue action of the dialogue system agent network is obtained through the system policy in the dialogue system agent network. Obtain the reward value corresponding to the i-th round of dialogue, which is output by the environment system after the dialogue action of the dialogue system agent network is executed in the i-th round of dialogue. Based on the third observation of the dialogue environment after the second dialogue action is executed by the user agent network, the third dialogue action of the user agent network is obtained through the user policy. Obtain a fourth observation of the dialogue environment by the dialogue system agent network after the third dialogue action is executed; Obtain the first state transition operator corresponding to the i-th round of the user agent network. The first state transition operator includes the first observation value, the first dialogue action, the reward value corresponding to the i-th round of dialogue, and the third observation value. Obtain the second state transition operator corresponding to the i-th round of the dialogue system agent network. The second state transition operator includes the second observation value, the second dialogue action, the reward value corresponding to the i-th round of dialogue, and the fourth observation value.

6. The method according to claim 5, characterized in that, The first dialogue action of the user agent network, obtained based on the first observation of the dialogue environment at the end of the (i-1)th round of dialogue by the user agent network and through the user policy in the user agent network, includes: Based on the first observation, user dialogue information is queried from the user target database using the user strategy; The first dialogue action is obtained based on the retrieved user dialogue information.

7. The method according to claim 6, characterized in that, In response to the application of the dialogue learning system in the medical field, the user target database contains user constraints and user requests; The user constraints are used to indicate named entities in the medical field; the user requests are used to indicate dialogue requests in the medical field.

8. The method according to claim 7, characterized in that, The first observation includes: the dialogue actions of the dialogue system agent network in the (i-1)th round of dialogue, the dialogue actions of the user agent network in the (i-1)th round of dialogue, the target state for indicating the remaining constraints and remaining requests, and the inconsistency vector between the response of the dialogue system agent network and the user constraints.

9. The method according to claim 5, characterized in that, The second observation of the dialogue environment after the first dialogue action is executed, based on the dialogue system proxy network, and the acquisition of the second dialogue action of the dialogue system proxy network through the system policy in the dialogue system proxy network, includes: Based on the second observation, the system dialogue information is queried from the system database using the system strategy. The second dialogue action is obtained based on the retrieved system dialogue information.

10. The method according to claim 9, characterized in that, In response to the application of the dialogue learning system in the medical field, the system database contains named entities in the medical field, as well as related information of the named entities in the medical field.

11. The method according to claim 10, characterized in that, The second observation includes: The first dialogue action, the dialogue action of the user agent network in the (i-1)th round of dialogue, the belief states of the constraint slots and request slots provided by the user agent network, and the embedding vector of the number of query results from the system database.

12. The method according to any one of claims 1 to 11, characterized in that, Before conducting at least one round of dialogue through two agent networks in the dialogue learning system, the process also includes: The two agent networks were pre-trained using dialogue sample text; The dialogue sample text contains text from at least one round of dialogue.

13. A dialogue processing apparatus, characterized in that, The device includes: The dialogue module is used to conduct at least one round of dialogue through two agent networks in the dialogue learning system. The operator acquisition module is used to acquire, in a reinforcement learning manner, the state transition operators corresponding to each of the two agent networks in each round of the at least one round of dialogue; the state transition operators include the agent network's observation of the dialogue environment in a round of dialogue, the agent network's dialogue actions in a round of dialogue, and the reward values ​​of the dialogue actions of the two agent networks in a round of dialogue; the dialogue environment is used to indicate the dialogue situation between the two agent networks. The loss acquisition module is used to acquire the loss function values ​​of the two agent networks based on the state transition operators corresponding to each round of dialogue in the at least one round of dialogue for each of the two agent networks. The parameter update module is used to update the parameters of the two agent networks respectively based on the loss function values ​​of the two agent networks.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to implement the dialogue processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the dialogue processing method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor of a computer device, causing the computer device to perform the dialogue processing method as described in any one of claims 1 to 12.

Images