Dialogue processing method, dialogue system, electronic device, and computer storage medium

Abstract

The application provides a dialogue processing method, a dialogue system, an electronic device and a computer storage medium, comprising converting user input speech into corresponding dialogue states; determining corresponding initial dialogue skills, processing the initial dialogue skills, and determining target dialogue skills; selecting initial speech actions based on the target dialogue skills; processing the initial speech actions to determine target speech actions; and converting the target speech actions into corresponding dialogue speech and playing it. The scheme proposes a method of using hierarchical reinforcement learning to construct a two-layer dialogue strategy of top-level strategy and bottom-level strategy, selects initial dialogue skills through the top-level strategy, and then processes to obtain target dialogue skills; selects initial speech actions based on the target dialogue skills through the bottom-level strategy, and then processes to obtain target speech actions. Through the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so as to achieve the purpose of smooth dialogue.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a dialogue processing method, a dialogue system, an electronic device and a computer storage medium. BACKGROUND

[0002] Generally, dialogue can be understood as a sequential decision-making process, and the machine needs to update and maintain the internal dialogue state by understanding the user's statement during the dialogue process, and then select the optimal action (such as confirming the demand, asking the limit condition, providing the result, etc.) according to the current dialogue state, so as to complete the task.

[0003] The existing dialogue system is based on a single dialogue type, and does not consider how to process multiple dialogue types at the same time, such as chatting, question and answer, task, and recommendation. However, in actual human dialogue, multiple different types of dialogue segments are usually covered; if multiple types of dialogue data are simply combined, it will lead to the problem of unsmooth dialogue in the dialogue process.

[0004] Therefore, how to provide a dialogue processing method to manage multiple types of dialogue data to achieve smooth dialogue is a problem to be solved by the present application. SUMMARY

[0005] Therefore, the embodiments of the present application provide a dialogue processing method, a dialogue system, an electronic device and a computer storage medium to solve the problem of unsmooth dialogue in the prior art.

[0006] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the embodiments of the present application shows a dialogue processing method applied to a dialogue system, and the method comprises:

[0008] Obtaining the voice input by the user, and converting the voice into corresponding dialogue state;

[0009] Determining the initial dialogue skill corresponding to the dialogue state from the skill set, wherein the skill set is obtained by processing historical dialogue data based on dialogue type;

[0010] Processing the initial dialogue skill of the dialogue state to determine the target dialogue skill of the dialogue state;

[0011] Selecting the initial voice action based on the target dialogue skill of the dialogue state;

[0012] Processing the initial voice action of the target dialogue skill to determine the target voice action of the dialogue state;

[0013] convert the target voice action of the dialogue state into a corresponding dialogue voice and play.

[0014] Optionally, the method further comprises:

[0015] tracking the voice state to determine a corresponding historical voice state;

[0016] analyzing and processing the historical voice state and the voice state to determine a target voice action.

[0017] Optionally, the processing of the historical dialogue data based on dialogue types to obtain a skill set comprises:

[0018] dividing historical dialogue data in a dialogue system into dialogue data of different dialogue types according to dialogue types;

[0019] for each dialogue type, dividing dialogue data of the dialogue type according to different action types to determine dialogue data corresponding to each skill.

[0020] Optionally, the processing of the initial dialogue skill of the dialogue state to determine a target dialogue skill of the dialogue state comprises:

[0021] inputting the initial dialogue skill of the dialogue state into a first reinforcement learning DQN network as an input;

[0022] calculating, by the first DQN network, the input initial dialogue skill of the dialogue state to obtain a corresponding first value function;

[0023] updating network parameters of the first DQN network based on the first value function to reconstruct the first DQN network;

[0024] processing the initial dialogue skill of the dialogue state by using the updated first DQN network to output a target dialogue skill.

[0025] Optionally, the processing of the initial voice action of the target dialogue skill to determine a target voice action of the dialogue state comprises:

[0026] inputting the initial voice action of the target dialogue skill into a second reinforcement learning DQN network as an input;

[0027] calculating, by the second DQN network, the input initial voice action of the target dialogue skill to obtain a corresponding second value function;

[0028] updating network parameters of the second DQN network based on the second value function to reconstruct the second DQN network;

[0029] The initial voice action of the target dialogue skill is processed by using the updated second DQN network, and a target voice action is output.

[0030] The second aspect of the embodiment of the application shows a dialogue system, which comprises:

[0031] A language understanding module is configured to acquire a voice input by a user, and convert the voice into a corresponding dialogue state.

[0032] A dialogue management module is configured to determine an initial dialogue skill corresponding to the dialogue state from a skill set, wherein the skill set is obtained by processing historical dialogue data based on a dialogue type; process the initial dialogue skill of the dialogue state to determine a target dialogue skill of the dialogue state; select an initial voice action based on the target dialogue skill of the dialogue state; process the initial voice action of the target dialogue skill to determine a target voice action of the dialogue state.

[0033] A natural language generation module is configured to convert the target voice action of the dialogue state into a corresponding dialogue voice and play the dialogue voice.

[0034] Optionally, the dialogue management module is further configured to track the voice state to determine a corresponding historical voice state; and analyze and process the historical voice state and the voice state to determine a target voice action.

[0035] Optionally, the dialogue management module configured to process the initial dialogue skill of the dialogue state to determine the target dialogue skill of the dialogue state is specifically configured to take the initial dialogue skill of the dialogue state as an input of a first deep Q-network (DQN) network.

[0036] The first DQN network is configured to calculate the input initial dialogue skill of the dialogue state to obtain a corresponding first value function.

[0037] The network parameters of the first DQN network are updated based on the first value function to reconstruct the first DQN network.

[0038] The initial dialogue skill of the dialogue state is processed by using the updated first DQN network to output a target dialogue skill.

[0039] The third aspect of the embodiment of the application shows an electronic device configured to run a program, wherein the program is configured to execute the dialogue processing method shown in the first aspect of the embodiment of the application.

[0040] The fourth aspect of the embodiment of the present application shows a computer storage medium, the storage medium comprises a program, wherein the program controls the device where the storage medium is located to execute the dialogue processing method shown in the first aspect of the embodiment of the present application when the program is running.

[0041] Based on the dialogue processing method, the dialogue system, the electronic device and the computer storage medium provided by the embodiment of the present application, the method comprises: acquiring a voice input by a user, converting the voice into a corresponding dialogue state; determining an initial dialogue skill corresponding to the dialogue state, the skill set is obtained by processing historical dialogue data based on dialogue types; processing the initial dialogue skill of the dialogue state to determine the target dialogue skill of the dialogue state; selecting an initial voice action based on the target dialogue skill of the dialogue state; processing the initial voice action of the target dialogue skill to determine the target voice action of the dialogue state; converting the target voice action of the dialogue state into a corresponding dialogue voice and playing. The embodiment of the present application proposes a method of using hierarchical reinforcement learning to construct a two-layer dialogue strategy of top strategy and bottom strategy, select an initial dialogue skill based on the converted dialogue state through the top strategy, then process the initial dialogue skill to determine the target dialogue skill of the dialogue state; select an initial voice action based on the target dialogue skill through the bottom strategy, process the initial voice action to determine the target voice action of the dialogue state; finally, convert the target voice action into a corresponding dialogue voice and play. Through the method shown above, the corresponding dialogue skill can be called according to different dialogue scenes, and the corresponding topic guidance can be performed, so as to achieve the purpose of smooth dialogue. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0043] Figure 1 The structure diagram of the dialogue system shown in the embodiment of the present application;

[0044] Figure 2 The flowchart of the dialogue processing method shown in the embodiment of the present application;

[0045] Figure 3 The flowchart of determining the target dialogue skill shown in the embodiment of the present application;

[0046] Figure 4A flowchart of a process for determining a target voice action according to an embodiment of the present application is shown in FIG. 1.

[0047] Figure 5 A flowchart of another dialog processing method according to an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0049] The terms "first", "second", "third", "fourth" and the like in the specification of the application and the claims, and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or devices.

[0050] It should be noted that the description of "first", "second" and the like in the present application is only for description purpose, and cannot be understood as indicating or implying the relative importance of the indicated technical features, or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection required by the present application.

[0051] In the present application, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0052] In the embodiment of the present application, the common dialogue system can be divided into three types: chat type, task-oriented type and question-answer type. The chat type dialogue aims to generate interesting and informative natural replies so that the human-computer dialogue can continue; the question-answer type dialogue refers to one question and one answer, that is, the user puts forward a question, and the system returns the correct answer by analyzing the question and searching the knowledge base; the task-oriented dialogue refers to the multi-round dialogue driven by tasks, that is, the machine needs to determine the user's goal by understanding, active inquiry, clarification and other means, and then returns the correct result by calling the corresponding API query to complete the user's demand.

[0053] Referring to Figure 1 A structural schematic diagram of a dialogue system is shown in the embodiment of the present application.

[0054] The dialogue system shown in the embodiment of the present application can be deployed in a robot or other application device, and the dialogue system is a robot integrating multiple dialogue skills.

[0055] The dialogue system includes an input end 10, a language understanding module NLU 20, a natural language generation module NLG 30 and a dialogue management module DM 40.

[0056] The dialogue management module DM 40 stores a skill set composed of multiple skills. Specifically, the construction process of the skill set is as follows: the historical dialogue data in the dialogue system is divided according to dialogue types, and the dialogue data of different dialogue types is divided; for each dialogue type, the dialogue data of the dialogue type is divided according to different action types, and the dialogue data corresponding to each skill is determined, that is, the multiple skills,

[0057] It should be noted that each skill is a set of each major category of basic actions; the skills under each dialogue type form a skill set G, and the number of skills under each dialogue type is multiple.

[0058] G can be represented as {task-oriented type Task-Domain1, Task-Domain2, …, chat type Chat-Topic1, Chat-Topic2, …, question-answer type QA1, QA2, …}, wherein the task-oriented type Task-Domain1, Task-Domain2, etc. are task-oriented skills; the chat type Chat-Topic1, Chat-Topic2, etc. are chat skills; and the question-answer type QA1, QA2, etc. are question-answer skills.

[0059] Further, it should be noted that the dialogue type at least includes the task-oriented type, the chat type and the question-answer type.

[0060] The mapping relationship between different dialogue states and dialogue types is constructed.

[0061] The dialogue management module DM40 further comprises a global dialogue state tracking module global DST, a top-level policy top-level policy, and a low-level policy low-level policy.

[0062] The global dialogue state tracking module global DST is configured to track and record the global dialogue state.

[0063] The top-level policy top-level policy is configured to select a skill for responding according to the current dialogue state.

[0064] The low-level policy low-level policy is configured to complete a specific dialogue action of a skill.

[0065] The process of the dialogue processing method implemented by the dialogue system according to the embodiments of the present application includes:

[0066] The language understanding module NLU20 is configured to acquire a voice input by a user, and convert the voice into a corresponding dialogue state.

[0067] The dialogue management module DM40 is configured to determine an initial dialogue skill corresponding to the dialogue state from a skill set based on the top-level policy top-level policy, the skill set being obtained by processing historical dialogue data based on dialogue types, and process the initial dialogue skill of the dialogue state to determine a target dialogue skill of the dialogue state; select an initial voice action from the target dialogue skill of the dialogue state based on the low-level policy low-level policy, and process the initial voice action of the target dialogue skill to determine a target voice action of the dialogue state.

[0068] The natural language generation module NLG30 is configured to convert the target voice action of the dialogue state into a corresponding dialogue voice and play the dialogue voice.

[0069] In a specific implementation, the target voice action of the dialogue state is converted into a voice and played to the user to form a dialogue.

[0070] In the embodiment of the present application, a hierarchical reinforcement learning method is used to construct a two-layer dialogue strategy of a top-layer strategy and a bottom-layer strategy. The top-layer strategy is used to select an initial dialogue skill based on a converted dialogue state, and then the initial dialogue skill is processed to determine a target dialogue skill of the dialogue state. The bottom-layer strategy is used to select an initial voice action based on the target dialogue skill, and then the initial voice action is processed to determine a target voice action of the dialogue state. Finally, the target voice action is converted into a corresponding dialogue voice and played. Through the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0071] Optionally, based on the dialogue system shown in the embodiment of the present application, the dialogue management module DM40 is further configured to track the voice state to determine a corresponding historical voice state, and analyze and process the historical voice state and the voice state to determine a target voice action.

[0072] In a specific implementation, the voice state is tracked by using global dialogue tracking to determine a historical voice state related to the voice state, and the historical voice state and the voice state are processed to determine a dialogue intent, a slot filling state, a dialogue topic, and a called dialogue module of each historical voice state and voice state. Based on the semantics of the dialogue intent, the slot filling state, the dialogue topic, and the called dialogue module of each historical voice state and voice state, a corresponding dialogue strategy, i.e., a target voice action, is selected and output.

[0073] In the embodiment of the present application, the voice state is tracked by using a global dialogue state tracking method to determine a historical voice state related to the voice state, and then the historical voice state is analyzed to determine a dialogue intent, a slot filling state, a dialogue topic, and a called dialogue module, a corresponding dialogue strategy, i.e., a target voice action, is selected and output. Through the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0074] Optionally, based on the dialogue system shown in the embodiment of the present application, the dialogue management module DM40 that processes the initial dialogue skill of the dialogue state to determine a target dialogue skill of the dialogue state is specifically configured to:

[0075] The initial dialogue skill of the dialogue state is input as an input of a first reinforcement learning DQN network; the first DQN network calculates the initial dialogue skill of the input dialogue state to obtain a corresponding first value function; network parameters of the first DQN network are updated based on the first value function to reconstruct the first DQN network; and the initial dialogue skill of the dialogue state is processed by using the updated first DQN network to output a target dialogue skill.

[0076] It should be noted that different reward signals, i.e., different reward discount factors and external rewards, are set according to multiple experiences or experiments for different types of dialogue skills.

[0077] In a specific implementation, the initial dialogue skill of the dialogue state is input into the first reinforcement learning DQN network by a top-level policy. A reward discount factor and an external reward corresponding to the initial dialogue skill are obtained, and the reward discount factor and the external reward corresponding to the initial dialogue skill, the initial dialogue skill of the dialogue state, and a recorded historical target dialogue skill g ’ The first value function Q1(S, g) is obtained by substituting formula (1) for calculation, that is, the expectation of the reward function when the skill g is executed in the dialogue state S, i.e., the sum of the maximum external rewards actually obtained. The first value function obtained by calculation is used as feedback of the reward function at each step, i.e., the network parameters of the first DQN network are updated, so as to adjust the first DQN network, and then the first DQN network is retrained based on the updated network parameters. The first DQN network obtained by performing retraining is executed to process the initial dialogue skill of the dialogue state to output a target dialogue skill.

[0078] Formula (1):

[0079]

[0080] Wherein, γ is a reward discount factor, r e is an external reward, N refers to a time step required for skill completion or ending, that is, the external environmental incentives obtained in N steps are taken as a whole as a reward of the top-level policy, g is a recorded historical target dialogue skill of the last dialogue state, S is a dialogue state, S t is a dialogue state at time t, and E is an expectation.

[0081] Further, the dialogue state S at the current time is affected by the dialogue state at the last time.

[0082] In the embodiment of the present application, a hierarchical reinforcement learning method is used to construct a two-layer dialogue strategy of a top-layer strategy and a bottom-layer strategy. The top-layer strategy adjusts an initial dialogue skill by using a first reinforcement learning DQN network to determine a corresponding target dialogue skill. The bottom-layer strategy selects an initial speech action based on the target dialogue skill to process the initial speech action and determine a target speech action of the dialogue state. Finally, the target speech action is converted into a corresponding dialogue speech and played. Through the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0083] Optionally, based on the dialogue system shown in the embodiment of the present application, the dialogue management module DM40 for processing the initial speech action of the target dialogue skill and determining the target speech action of the dialogue state is specifically used for:

[0084] inputting the initial speech action of the target dialogue skill into an input of a second reinforcement learning DQN network; the second DQN network calculates the inputted initial speech action of the target dialogue skill to obtain a corresponding second value function; the network parameters of the second DQN network are updated based on the second value function to reconstruct the second DQN network; and the initial speech action of the target dialogue skill is processed by using the updated second DQN network to output a target speech action.

[0085] In a specific implementation, the initial speech action of the target dialogue skill is inputted into an input of a second reinforcement learning DQN network by the bottom-layer strategy. A discount factor of a reward corresponding to the target dialogue skill and a reward of the target dialogue skill at a current time t are obtained, and the discount factor of the reward corresponding to the initial dialogue skill and the reward of the target dialogue skill at the current time t, the initial speech action and the target dialogue skill g are substituted into formula (2) to calculate a speech action a of the initial dialogue skill at t+1 time, that is, a speech action a of the initial dialogue skill at t+1 time. t+1 t+1 The formula (2) is calculated to obtain a second value function Q2(S, a, g), that is, the expectation of the reward function when the initial speech action of the target skill g is executed at the dialogue state S, that is, the sum of the actual obtained external rewards is maximized. The second value function calculated by using is used as the feedback of each step to update the reward function, that is, the network parameters of the second DQN network are updated, so as to adjust the second DQN network, and then the second DQN network is retrained based on the updated network parameters. The second DQN network obtained by performing the retraining is executed to process the initial speech action of the dialogue state and output a target speech action.

[0086] Formula (2):

[0087] ​

[0088] wherein, gamma is a discount factor of the reward, r i t is a reward of the target dialogue skill at a current time t, N refers to a time step required for an action to be completed or ended, H is a history target dialogue skill recording a last dialogue state, S is a dialogue state, S t+1 is a dialogue state of the dialogue state at a time t+1, a t+1 is a speech action of the initial speech action a at the time t+1, g is a target dialogue skill, and E is an expectation.

[0089] Optionally, the method further comprises: recording the current dialogue state, the target dialogue skill corresponding to the dialogue state, and a target speech action in the target dialogue skill.

[0090] In the embodiment of the present application, a two-layer dialogue strategy of a top-layer strategy and a bottom-layer strategy is constructed by using a hierarchical reinforcement learning method. The top-layer strategy adjusts an initial dialogue skill by using a first reinforcement learning DQN network to determine a corresponding target dialogue skill. The bottom-layer strategy adjusts an initial speech action by using a second DQN to determine a target speech action of the dialogue state. Finally, the target speech action is converted into a corresponding dialogue speech and played. By using the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0091] Referring to Figure 2 is a flowchart of a dialogue processing method according to an embodiment of the present application. The method comprises:

[0092] Step S201: obtaining a language sentence input by a user, and converting the language sentence into a corresponding dialogue state.

[0093] In the process of implementing step S201, a sentence generated by the user when talking to a robot is obtained, a language sentence corresponding to the sentence is recognized by a language understanding module to determine a user intention of the sentence and extract slot information, and then a corresponding dialogue state is determined.

[0094] It should be noted that the dialogue state can determine a corresponding dialogue type.

[0095] Step S202: determining an initial dialogue skill corresponding to the dialogue state.

[0096] In the process of implementing step S202, the skill set is traversed to find a dialogue type corresponding to the dialogue state, and then an action type corresponding to the dialogue type is determined; and a skill is selected from the skill set as the initial dialogue skill based on the action type.

[0097] It should be noted that the initial dialogue skill g belongs to G.

[0098] Step S203: processing the initial dialogue skill of the dialogue state, determining the target dialogue skill of the dialogue state.

[0099] It should be noted that the process of implementing step S203 of processing the initial dialogue skill of the dialogue state to determine the target dialogue skill of the dialogue state includes the following steps, as shown in Figure 3

[0100] Step S301: taking the initial dialogue skill of the dialogue state as the input of the first reinforcement learning DQN network.

[0101] In the process of implementing step S301, the initial dialogue skill of the dialogue state is input into the input of the first reinforcement learning DQN network.

[0102] Step S302: the first DQN network calculates the input initial dialogue skill of the dialogue state to obtain the corresponding first value function.

[0103] It should be noted that for different types of dialogue skills, different reward signals are set according to multiple experiences or experiments, that is, different reward discount factors and external rewards.

[0104] In the process of implementing step S302, the discount factor and external reward corresponding to the initial dialogue skill are obtained, and the discount factor and external reward corresponding to the initial dialogue skill, the initial dialogue skill of the dialogue state, and the recorded historical target dialogue skill g ’ are substituted into formula (1) for calculation to obtain the first value function Q1(S, g), that is, the expected reward function when skill g is executed in dialogue state S, that is, the sum of the actual obtained external rewards is maximized.

[0105] Further, the dialogue state S at the current time is affected by the dialogue state at the last time.

[0106] Optionally, it also includes recording the current dialogue state and the target dialogue skill corresponding to the dialogue state.

[0107] Step S303: updating the network parameters of the first DQN network based on the first value function to reconstruct the first DQN network.

[0108] ​In the implementation of step S303, the first value function obtained by calculation is used to update the reward function as the feedback of each step, that is, the network parameters of the first DQN network are updated, so as to adjust the first DQN network, and then retrain the first DQN network based on the updated network parameters.

[0109] Step S304: processing the initial dialogue skill of the dialogue state by using the updated first DQN network, and outputting a target dialogue skill.

[0110] In the implementation of step S304, the first DQN network obtained by retraining is executed to process the initial dialogue skill of the dialogue state, and output a target dialogue skill.

[0111] In the embodiment of the application, for the consistency of user interest and the mention of a new topic by the user, the dialogue system in the robot bot should adjust the corresponding topic accordingly.

[0112] Based on the above steps S301 to S304, the dialogue topic is specified based on the top-level strategy, that is, a series of dialogues are performed on the dialogue state.

[0113] Step S204: selecting an initial speech action based on the target dialogue skill of the dialogue state.

[0114] In the implementation of step S204, the target dialogue skill g conveyed by the previous layer and the current state s are received, and then a possible system action a, that is, an initial speech action, is selected.

[0115] Step S205: processing the initial speech action of the target dialogue skill to determine the target speech action of the dialogue state.

[0116] It should be noted that in the implementation of step S205 of processing the initial speech action of the target dialogue skill to determine the target speech action of the dialogue state, as shown in Figure 4 , the following steps are included:

[0117] Step S401: inputting the initial speech action of the target dialogue skill into the input of the second reinforcement learning DQN network.

[0118] It should be noted that the implementation of step S401 is the same as the process of step S301 described above, and can be referred to each other.

[0119] Step S402: the second DQN network calculates the input target dialogue skill and the initial speech action to obtain a corresponding second value function.

[0120] In the implementation process of step S402, the discount factor of the reward corresponding to the target dialogue skill and the reward of the target dialogue skill at the current time t are obtained, the discount factor of the reward corresponding to the initial dialogue skill and the reward of the target dialogue skill at the current time t, the initial voice action and the target dialogue skill g, and the dialogue state S at the time t+1 of the dialogue state at the time t are substituted into formula (2) for calculation to obtain the second value function Q2(S, a, g), that is, the expectation of the reward function when the initial voice action of the target skill g is performed at the dialogue state S, that is, the sum of the actual obtained external rewards is maximized. t+1 The initial voice action a at the time t+1 t+1 is substituted into formula (2) for calculation to obtain the second value function Q2(S, a, g), that is, the expectation of the reward function when the initial voice action of the target skill g is performed at the dialogue state S, that is, the sum of the actual obtained external rewards is maximized.

[0121] Optionally, the current dialogue state, the target dialogue skill corresponding to the dialogue state, and the target voice action in the target dialogue skill are recorded.

[0122] Step S403: updating the network parameters of the second DQN network based on the second value function to reconstruct the second DQN network.

[0123] Step S404: processing the initial voice action of the target dialogue skill by using the updated second DQN network to output the target voice action.

[0124] It should be noted that the implementation processes of steps S403 to S404 are the same as those of steps S303 to S304.

[0125] Optionally, if the target voice action is a knowledge type question and answer action, the target voice action directly corresponds to a response module.

[0126] Step S206: converting the target voice action of the dialogue state into voice and playing.

[0127] In the implementation process of step S206, the output target voice action of the dialogue state is converted into voice and played to the user to form a dialogue.

[0128] In the embodiment of the application, the method of hierarchical reinforcement learning is used to construct two-layer dialogue strategies of a top-layer strategy and a bottom-layer strategy, the initial dialogue skill is selected based on the converted dialogue state by the top-layer strategy, then the initial dialogue skill is processed to determine the target dialogue skill of the dialogue state, the initial voice action is selected based on the target dialogue skill by the bottom-layer strategy, then the initial voice action is processed to determine the target voice action of the dialogue state, and finally the target voice action is converted into corresponding dialogue voice and played. Through the above method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0129] Based on the above-mentioned dialogue processing method, the embodiment of the present application further discloses another dialogue processing method, as shown in the flowchart of a dialogue processing method shown in the embodiment of the present application, the method comprises the following steps: Figure 5

[0130] Step S501: obtaining a language sentence input by a user, and converting the language sentence into a corresponding dialogue state.

[0131] It should be noted that the process of implementing step S501 is the same as the process of implementing step S201 described above, and the two can be referred to each other.

[0132] Step S502: tracking the dialogue state and determining a corresponding historical dialogue state.

[0133] In the process of implementing step S502, the global dialogue tracking is used to track the dialogue state, so as to determine the historical dialogue state related to the dialogue state.

[0134] Step S503: analyzing and processing the historical dialogue state and the dialogue state, and determining a target dialogue action.

[0135] In the process of implementing step S503, the global dialogue tracking is used to process the historical dialogue state and the dialogue state, so as to determine the dialogue intention, the slot filling state, the dialogue topic, and the called dialogue module of each historical dialogue state and dialogue state. Based on the semantics of the dialogue intention, the slot filling state, the dialogue topic, and the called dialogue module of each historical dialogue state and dialogue state, a corresponding dialogue strategy, i.e., a target dialogue action, is selected and output.

[0136] Optionally, for the coherence of the dialogue topic sequence, the embodiment of the present application uses additional knowledge graph and other information to measure the dialogue.

[0137] Step S504: converting the target dialogue action of the dialogue state into speech and playing.

[0138] It should be noted that the process of implementing step S504 is the same as the process of implementing step S206 described above, and the two can be referred to each other.

[0139] ​In the embodiment of the present application, the speech state is tracked based on a global dialogue state tracking manner to determine a history speech state related to the speech state, and then the history speech state is analyzed to determine a dialogue intention, a slot filling state, a dialogue topic, and a called dialogue module, and so on, a corresponding dialogue strategy, i.e., a target speech action, is selected and output. Through the above-mentioned method, corresponding dialogue skills can be called according to different dialogue scenes, and corresponding topic guidance can be performed, so that the purpose of smooth dialogue is achieved.

[0140] The embodiment of the present application also discloses an electronic device for running a database stored procedure, wherein the database stored procedure is executed to perform the above-mentioned Figure 2 disclosed data processing method.

[0141] The embodiment of the present application also discloses a computer storage medium, the storage medium comprising a database stored procedure, wherein the database stored procedure is executed to control a device where the storage medium is located to perform the above-mentioned Figure 2 disclosed data processing method.

[0142] In the context of the present disclosure, the computer storage medium can be a tangible medium, which can contain or store a program for use by or in connection with an instruction execution system, apparatus or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of machine-readable storage media can include one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dialog processing method characterized by, The method is applied to a dialogue system, and comprises: obtaining a voice input by a user, and converting the voice into a corresponding dialogue state; traversing a skill set to find a dialogue type corresponding to the dialogue state, to determine an action type corresponding to the dialogue type, the skill set being obtained based on processing historical dialogue data according to dialogue types; selecting a skill from the skill set as an initial dialogue skill based on the action type; inputting the initial dialogue skill of the dialogue state into a first deep Q-network (DQN) network for reinforcement learning; the first DQN network performing calculation on the inputted initial dialogue skill of the dialogue state to obtain a first value function; updating network parameters of the first DQN network based on the first value function, to reconstruct the first DQN network; processing the initial dialogue skill of the dialogue state by using the updated first DQN network, to output a target dialogue skill; selecting an initial voice action based on the target dialogue skill of the dialogue state; processing the initial voice action of the target dialogue skill to determine a target voice action of the dialogue state; alternatively, tracking a voice state to determine a corresponding historical voice state; processing the historical voice state and the voice state by using global dialogue tracking, to determine a dialogue intent, a slot filling state, a dialogue topic, and a called dialogue module of each historical voice state and voice state; selecting a target voice action based on semantics of the dialogue intent, the slot filling state, the dialogue topic, and the called dialogue module of each historical voice state and voice state; converting the target voice action into a corresponding dialogue voice and playing the dialogue voice.

2. The method of claim 1, wherein, The skill set obtained based on processing historical dialogue data according to dialogue types comprises: dividing historical dialogue data in a dialogue system into dialogue data of different dialogue types according to dialogue types; for each dialogue type, dividing dialogue data of the dialogue type according to different action types to determine dialogue data corresponding to each skill.

3. The method of claim 1, wherein, The processing of the initial voice action of the target dialogue skill to determine the target voice action of the dialogue state comprises: inputting the initial voice action of the target dialogue skill into an input of a second DQN network for reinforcement learning; the second DQN network performing calculation on the inputted initial voice action of the target dialogue skill to obtain a second value function; updating network parameters of the second DQN network based on the second value function, to reconstruct the second DQN network; processing the initial voice action of the target dialogue skill by using the updated second DQN network, to output a target voice action.

4. A dialogue system, comprising: a language understanding module configured to obtain a voice input by a user, and convert the voice into a corresponding dialogue state; a dialogue management module configured to traverse a skill set to find a dialogue type corresponding to the dialogue state, to determine an action type corresponding to the dialogue type, the skill set being obtained based on processing historical dialogue data according to dialogue types; selecting a skill from the skill set as an initial dialogue skill based on the action type; the skill set is obtained by processing historical dialogue data based on dialogue types; taking the initial dialogue skill of the dialogue state as an input of a first reinforcement learning DQN network; the first DQN network calculates the initial dialogue skill of the input dialogue state to obtain a first value function, and updates network parameters of the first DQN network based on the first value function to reconstruct the first DQN network; processing the initial dialogue skill of the dialogue state by using the updated first DQN network to output a target dialogue skill; selecting an initial voice action based on the target dialogue skill of the dialogue state; processing the initial voice action of the target dialogue skill to determine a target voice action of the dialogue state, or tracking a voice state to determine a corresponding historical voice state; processing the historical voice state and the voice state by using global dialogue tracking to determine a dialogue intent, a slot filling state, a dialogue topic, and a called dialogue module of each historical voice state and voice state; selecting a target voice action based on semantics of the dialogue intent, the slot filling state, the dialogue topic, and the called dialogue module of each historical voice state and voice state; a natural language generation module configured to convert the target voice action into a corresponding dialogue voice and play the dialogue voice.

5. An electronic device, comprising: The electronic device is configured to run a program, and the program is configured to perform the dialogue processing method of any one of claims 1-3 when running.

6. A computer storage medium, characterized in that, The storage medium includes a program, and the program is configured to control a device where the storage medium is located to perform the dialogue processing method of any one of claims 1-3 when running.

Images