Conversation processing method, apparatus, device, medium, and product

Abstract

Embodiments of the present application disclose a dialogue processing method, device, equipment, medium and product, the method comprising: generating a business service agent and at least two types of object agents according to historical dialogue data between a business service object and each object; obtaining a multi-round dialogue result of the business service agent and each object agent in a preset reinforcement learning environment; obtaining dialogue quality of the business service agent and each object agent in the multi-round dialogue result, and generating a reward value of interaction of the business service agent and each object agent according to the dialogue quality; selecting a target object agent from each object agent according to the reward value, and training a target business service agent for dialogue with an object according to interaction of the business service agent and the target object agent. The technical solution of the embodiments of the present application can flexibly, accurately and naturally carry out dialogue with the object.

Description

Technical Field

[0001] This application relates to the field of dialogue technology, and more specifically, to a dialogue processing method, a dialogue processing apparatus, an electronic device, a computer-readable storage medium, and a computer product. Background Technology

[0002] Many platforms offer services such as consultation, Q&A, problem handling, and complaints. As the demand for these services continues to increase, more and more platforms will provide intelligent customer service to handle the needs of customers. However, currently, intelligent customer service usually adopts fixed strategies, such as using keyword matching or predefined rules to generate responses, which lacks flexibility and is difficult to handle complex needs. Summary of the Invention

[0003] The embodiments of this application provide a dialogue processing method, a dialogue processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product that dynamically adapt to the needs of different types of objects and can flexibly, accurately, and naturally engage in dialogue with objects.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a dialogue processing method is provided, comprising: generating a business service agent and at least two types of object agents based on historical dialogue data between a business service object and various objects; obtaining multi-turn dialogue results between the business service agent and each object agent in a preset reinforcement learning environment; the reinforcement learning environment being used to describe the scene factors required for agent dialogue interaction; obtaining the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generating reward values ​​for the interaction between the business service agent and each object agent based on the dialogue quality; selecting a target object agent from the object agents based on the reward values, and training a target business service agent based on the interaction between the business service agent and the target object agent, the target business service agent being used to engage in dialogue with objects.

[0006] According to one aspect of the embodiments of this application, a dialogue processing apparatus is provided, comprising: a generation module, configured to generate a business service agent and at least two types of object agents based on historical dialogue data between a business service object and various objects; an acquisition module, configured to acquire multi-turn dialogue results between the business service agent and each object agent in a preset reinforcement learning environment; the reinforcement learning environment being used to describe the scene factors required for agent dialogue interaction; the acquisition module being further configured to acquire the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generate reward values ​​for the interaction between the business service agent and each object agent based on the dialogue quality; and a training module, configured to select a target object agent from the various object agents based on the reward values, and train a target business service agent based on the interaction between the business service agent and the target object agent, the target business service agent being used to engage in dialogue with objects.

[0007] In one embodiment of this application, the generation module is further configured to identify the object type of each object based on the historical dialogue data between the business service object and each object; obtain representative dialogue samples corresponding to the business service object and each object type; and adjust the base model based on the representative dialogue samples to obtain the business service agent and the object agent corresponding to each object type.

[0008] In one embodiment of this application, the generation module is further configured to perform data cleaning processing on the historical dialogue data to obtain target historical dialogue data, and to perform feature extraction on the target historical dialogue data to obtain dialogue features; to perform cluster analysis on the dialogue features to determine the cluster center of the cluster to which each object belongs; and to obtain the object type of each object based on the feature distribution of each cluster center and the historical dialogue content of each object under the cluster center.

[0009] In one embodiment of this application, the generation module is further configured to obtain the dialogue style of objects of each object type; construct an object question based on the dialogue style, determine the expected response example corresponding to the object question from the knowledge base, and generate a representative dialogue sample corresponding to the business service object based on the object question and the expected response example; construct a service question for the business service object, construct an object response example corresponding to the service question based on the dialogue style, and generate a representative dialogue sample corresponding to each object type based on the service question and the object response example.

[0010] In one embodiment of this application, the preset reinforcement learning environment includes executable actions and a dialogue state describing the current situation of the agent in the dialogue; the acquisition module is further configured to randomly select candidate object agents from the various object agents and generate the current dialogue state; acquire the executable actions selected by the business service agent according to the current dialogue state, and update the dialogue context according to the executable actions; acquire the object responses generated by the candidate object agents according to the dialogue context to obtain the dialogue result of the current dialogue round; generate the next dialogue state according to the dialogue result of the current dialogue round, and re-execute the steps of acquiring the executable actions selected by the business service agent and updating the dialogue context to obtain the multi-round dialogue results between the business service agent and the candidate object agents.

[0011] In one embodiment of this application, the acquisition module is further configured to identify the emotional state and intent distribution of the candidate object agent based on the dialogue result of the current dialogue round; construct personality parameters for the candidate object agent based on the dialogue result of the current dialogue round, the personality parameters being used to simulate a personalized object; and generate the next dialogue state based on the dialogue context, the emotional state, the intent distribution, and the personality parameters.

[0012] In one embodiment of this application, the acquisition module is further configured to update the dialogue rounds between the business service agent and the candidate object agent; acquire the dialogue satisfaction of the object agent in the dialogue results up to the latest dialogue round; if it is determined that the dialogue between the business service agent and the candidate object agent has ended based on at least one of the dialogue satisfaction and the updated dialogue rounds, then generate the multi-round dialogue results based on the dialogue results up to the latest dialogue rounds.

[0013] In one embodiment of this application, the acquisition module is further configured to identify the emotional changes in the responses generated by the object agent and the correctness of the responses from the business service agent in the multi-turn dialogue results; determine the dialogue efficiency in the multi-turn dialogue results based on the response content and dialogue rounds of the object agent in the multi-turn dialogue results; and use the emotional changes, the correctness of the responses, and the dialogue efficiency as the dialogue quality.

[0014] In one embodiment of this application, the dialogue quality further includes dialogue coherence calculated by the similarity between the current response and the previous response of the business service agent; the acquisition module is further configured to set dialogue weights for the emotion change, the correctness of the response, the dialogue efficiency, the dialogue coherence and the dialogue consistency according to the business needs corresponding to the business service agent; and to calculate the reward value based on the emotion change, the correctness of the response, the dialogue efficiency and the dialogue coherence, and the corresponding dialogue weights.

[0015] In one embodiment of this application, the training module is further configured to obtain the average reward value of the business service agent interacting with each object agent multiple times; set a random selection probability, and determine a target selection probability based on the random selection probability, wherein the target selection probability is greater than the random selection probability; and select the target object agent with the lowest average reward value based on the target selection probability.

[0016] In one embodiment of this application, the training module is further configured to obtain the performance differences of the business service agent on different object agents based on the training state of the business service agent; and adjust the selection probability of selecting a target object agent from the various object agents based on the performance differences.

[0017] In one embodiment of this application, the apparatus further includes an application module, configured to acquire dialogue content input by a target object, and identify the object type of the target object based on the dialogue content; generate a target response through the target business service agent based on the object type and the dialogue content; acquire a dialogue context generated based on the target response, and re-identify the object type based on the dialogue context; and repeatedly execute the step of generating the target response until the dialogue ends.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the dialogue processing method as described above.

[0019] According to one aspect of the embodiments of this application, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the dialogue processing method as described above.

[0020] According to one aspect of the embodiments of this application, an embodiment of this application provides a computer program product, including a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the dialogue processing method described above.

[0021] In the technical solution provided by the embodiments of this application, a business service agent and at least two types of object agents are generated based on historical dialogue data between the business service object and each object. In a preset reinforcement learning environment, the business service agent engages in multi-round dialogues with each object agent, thereby obtaining the multi-round dialogue results between the business service agent and each object agent in the preset reinforcement learning environment. The reinforcement learning environment is used to describe the scene factors required for agent dialogue interaction. The dialogue quality between the business service agent and each object agent in the multi-round dialogue results is obtained, and reward values ​​are generated based on the dialogue quality. These reward values ​​not only reflect the performance of the business service agent but also help adjust its adaptability to different object agents. Furthermore, rewards are generated based on the reward values ​​from each object agent. The system selects a target intelligent agent from among several intelligent agents and, based on the interaction between the business service intelligent agent and the target intelligent agent, enables the business service intelligent agent to progressively optimize its dialogue strategy in a targeted manner to train a target business service intelligent agent. In other words, the technical solution provided in this application, on the one hand, allows the business service intelligent agent to dynamically adjust its dialogue strategy in a reinforcement learning environment to adapt to different object styles; on the other hand, by selecting a target intelligent agent through reward values, the business service intelligent agent learns the behavioral characteristics corresponding to a specific intelligent agent in a targeted manner, constructing a target business service intelligent agent that can dynamically adapt to different object types. Thus, in practical applications, the target business service intelligent agent dynamically adapts to the needs of different types of objects, and can flexibly, accurately, and naturally engage in dialogue with objects, significantly improving dialogue effectiveness and object experience.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0024] Figure 1 This is a schematic diagram of one implementation environment involved in this application;

[0025] Figure 2 This is a flowchart illustrating a dialogue processing method in an exemplary embodiment of this application;

[0026] Figure 3 This is a schematic diagram of a dialogue processing interface shown in an exemplary embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating another dialogue processing method as shown in an exemplary embodiment of this application;

[0028] Figure 5 This is a flowchart illustrating another dialogue processing method as shown in an exemplary embodiment of this application;

[0029] Figure 6 This is a schematic diagram illustrating an intelligent agent interaction as an exemplary embodiment of this application;

[0030] Figure 7 This is a flowchart illustrating another dialogue processing method as shown in an exemplary embodiment of this application;

[0031] Figure 8 This is a schematic diagram illustrating a dialogue processing system as shown in an exemplary embodiment of this application;

[0032] Figure 9 This is a schematic diagram illustrating the clustering process of the clustering module in an exemplary embodiment of this application;

[0033] Figure 10 This is a schematic diagram illustrating the training fine-tuning process of the fine-tuning training module in an exemplary embodiment of this application;

[0034] Figure 11 This is a schematic diagram illustrating a reinforcement learning process as shown in an exemplary embodiment of this application;

[0035] Figure 12 This is a schematic diagram illustrating the process of the opponent selection module selecting an opponent, as shown in an exemplary embodiment of this application;

[0036] Figure 13 This is a structural block diagram of a dialogue processing apparatus illustrated in an exemplary embodiment of this application;

[0037] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying diagrams are merely illustrative and do not necessarily include all content and operations, nor do they necessarily have to be executed in the described order. For example, some operations may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0041] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] The technical solutions of the embodiments of this application will be described in detail below.

[0043] Please see Figure 1 , Figure 1 This is a schematic diagram of an implementation environment involved in this application. The implementation environment includes a terminal and a server. The server runs a server-side dialogue system, and the terminal runs a terminal-side dialogue system. The object requesting the service can use the terminal to communicate with the service object, which is the object providing the service.

[0044] The server is used to generate business service agents and at least two types of object agents based on the historical dialogue data between the business service object and each object. For example, it can generate one business service agent and three object agents of different types. An agent is a proxy that can perceive the environment and take actions to achieve a specific goal, and has autonomy, adaptability and interaction capabilities.

[0045] Next, the system obtains the multi-turn dialogue results between the business service agent and each object agent in a preset reinforcement learning environment. The reinforcement learning environment describes the scenario factors required for the agent's dialogue interaction, such as the multi-turn dialogue result 1 between the business service agent and object agent A1, the multi-turn dialogue result 2 between the business service agent and object agent A2, and the multi-turn dialogue result 3 between the business service agent and object agent A3. Then, the system obtains the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generates reward values ​​for the interaction between the business service agent and each object agent based on the dialogue quality. Then, the system selects a target object agent from among the object agents based on the reward values, such as selecting object agent A3 as the target object agent based on reward values ​​1 to 3. Then, the system trains a target business service agent based on the interaction between the business service agent and the target object agent. The target business service agent is used to conduct dialogue with the object, that is, it can be used to conduct dialogue with the object in the future.

[0046] In some embodiments, the terminal can also implement the dialogue processing process independently, that is, the terminal generates a business service agent and an object agent, and then generates a reward value based on the results of multiple rounds of dialogue, thereby obtaining the target business service agent.

[0047] The aforementioned terminals can be electronic devices such as smartphones, tablets, laptops, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This document does not impose any restrictions on these.

[0048] The terminal and server establish a communication connection beforehand via a network, enabling them to communicate with each other. The network can be wired or wireless, and this document does not impose any restrictions.

[0049] It should be noted that, in the specific embodiments of this application, at least one of the dialogue data involves an object. When the embodiments of this application are applied to specific products or technologies, permission or consent from the object is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0050] The following describes in detail the various implementation details of the technical solutions in the embodiments of this application.

[0051] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a dialogue processing method according to an embodiment of this application. This method can be applied to... Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by both a terminal and a server. In this embodiment, the method is described as being executed by a server. The dialogue processing method may include S210 to S240, which are described in detail below.

[0052] S210. Based on the historical dialogue data between the business service object and each object, generate a business service intelligent agent and at least two types of object intelligent agents.

[0053] In this embodiment of the application, the business service object is the object that provides business services, such as customer service that provides e-commerce consultation, problem answering, problem handling and complaints. The historical dialogue data between the business service object and each object includes the entire dialogue data from the start of the dialogue between the business service object and the object to the end of the dialogue, such as the questions raised by the object and the response of the business service object, the number of dialogue rounds and the object's feedback (such as satisfaction rating).

[0054] It should be noted that there is a large amount of historical dialogue data, which includes various types of object requirements and response styles of business service objects. Therefore, based on this large amount of historical dialogue data, at least two types of object agents can be generated. For example, different object requirements will result in different types of object agents. A business service agent can also be generated, which covers various types of response styles.

[0055] In one example, a large language model is trained based on historical dialogue data to obtain business service agents and at least two types of object agents.

[0056] S220. Obtain the results of multi-turn dialogues between the business service agent and each object agent in a preset reinforcement learning environment; the reinforcement learning environment is used to describe the scenario factors required for agent dialogue interaction.

[0057] In this embodiment, the business service agent is the intelligent customer service role, and there are at least two types of object agents, which are different object roles. Therefore, the business service agent can interact with each object agent in a preset reinforcement learning environment. The object agent can simulate the behavior of objects in actual applications and generate input content. The business service agent can generate natural language responses to the input of the object agent. Each time the object agent initiates a dialogue and receives a valid response, it constitutes a dialogue round. Then, the multi-round dialogue results between the business service agent and each object agent are obtained. The multi-round dialogue results include the entire dialogue data from the start of the dialogue between the business service agent and the object agent to the end of the dialogue.

[0058] The reinforcement learning environment is used to describe the scenario factors required for dialogue interaction between the business service agent and the object agent. This reinforcement learning environment changes with the interaction of the agent's dialogue. Thus, the business service agent can continuously optimize its behavior through dialogue interaction with different types of object agents. The reinforcement learning environment can provide the business service agent with a scenario for continuously optimizing its strategy, enabling the business service agent to better serve different types of objects and improve the quality of dialogue. In other words, in the reinforcement learning environment, the dialogue strategy of the business service agent is optimized based on the dialogue interaction between the business service agent and different types of object agents, and the results of multiple rounds of dialogue between the business service agent and each object agent in the preset reinforcement learning environment are obtained.

[0059] In one example, the reinforcement learning environment not only simulates the specific scenario of dialogue interaction, but also covers environmental factors in the dialogue, such as dialogue state and actions. In reinforcement learning, the state is a description of the agent's interaction with the environment and is the basis for the agent's decision-making; the state needs to be able to fully reflect the current dialogue context; while the action is the decision behavior made by the agent based on the current state.

[0060] S230. Obtain the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generate the reward value for the interaction between the business service agent and each object agent based on the dialogue quality.

[0061] In this embodiment of the application, by identifying the dialogue content in the results of multi-turn dialogue, the dialogue quality between the business service agent and the object agent can be obtained. This dialogue quality is used to describe the overall performance of the dialogue, including multiple dimensions such as the emotional atmosphere, communication speed, and accuracy of the answers.

[0062] In one example, obtaining dialogue quality includes: identifying the emotional changes in the responses generated by the object agent and the correctness of the responses from the business service agent in the multi-turn dialogue results; determining the dialogue efficiency in the multi-turn dialogue results based on the content of the object agent's responses and the number of dialogue turns; and using emotional changes, the correctness of responses, and dialogue efficiency as the criteria for dialogue quality.

[0063] Specifically, it identifies the emotional state of the responses generated by the object agent in each round of multi-round dialogue, and generates emotional changes based on the emotional state of each round, such as changing from negative emotions to positive emotions, or maintaining negative emotions.

[0064] The correctness of each response from the business service agent is verified, including: Matching with the knowledge base: For each answer given by the business service agent, check whether it conforms to a preset standard answer or an answer stored in the knowledge base. Whether the answer effectively solves the object agent's question: Determine whether the business service agent's answer effectively solves the question posed by the object agent. Context consistency: Determine whether the business service agent's answer is consistent with the dialogue context, and whether there are any contradictions or inconsistencies in information.

[0065] The efficiency of the business service agent's dialogue is determined by analyzing the responses and dialogue rounds of the target agent. For example, it's checked whether the target agent's needs can be resolved within a set number of dialogue rounds. If the needs are resolved within the set number of rounds, the dialogue is considered efficient. The responses of the target agent in each round reflect its understanding and feedback to the business service agent's answers. For instance, if the target agent's responses are always clear and the problem is not further complicated, the dialogue can be considered efficient. If the target agent repeatedly asks similar questions or does not respond to previous answers, it indicates that the business service agent is not communicating effectively and the dialogue efficiency is low.

[0066] In this embodiment, the dialogue quality is obtained by combining emotion changes, response correctness, and dialogue efficiency.

[0067] In one example, to improve the long-term performance of the agent, in addition to emotion changes, correctness of responses, and dialogue efficiency, dialogue coherence can be introduced as a measure of dialogue quality. Dialogue coherence is used to detect whether the business service agent can maintain logically clear dialogue without jumps or abrupt responses. Specifically, the similarity between the current response and the previous response of the business service agent is calculated, and the dialogue coherence is determined based on this similarity. Feature extraction can be performed on the current and previous responses, and the extracted vector cosine similarity can be calculated. If the vector cosine similarity is greater than a first preset similarity threshold, it indicates dialogue coherence; the higher the similarity, the higher the dialogue coherence. However, if the similarity is greater than a second preset similarity threshold, it indicates that the business service agent may be repeating responses, and the dialogue coherence is set to a default value, such as 0.

[0068] Reward values ​​are generated based on the dialogue quality, showing the interaction between the business service agent and each object agent. Dialogue quality and reward value are positively correlated. In one example, a pre-built mapping between different dialogue qualities and reward values ​​is constructed, allowing the acquired dialogue quality to be matched against this mapping to determine the corresponding reward value.

[0069] Optionally, generating reward values ​​can involve assigning different weights to each dimension of dialogue quality and calculating the final dialogue quality based on those weights. For example, dialogue weights can be set for emotion changes, response correctness, dialogue efficiency, dialogue coherence, and dialogue consistency based on the business requirements corresponding to the business service agent; and reward values ​​can be calculated based on emotion changes, response correctness, dialogue efficiency, dialogue coherence, and the corresponding dialogue weights.

[0070] Assuming that, based on the business needs corresponding to the business service agent, the weight of emotion change is 0.3, the weight of response correctness is 0.35, the weight of dialogue efficiency is 0.2, and the weight of dialogue coherence is 0.15, then the reward value score range for emotion change is 0-30, the reward value score range for response correctness is 0-30, the reward value score range for dialogue efficiency is 0-25, and the reward value score range for dialogue coherence is 0-15, as shown in Table 1 below, which illustrates the correspondence between different dialogue quality dimensions and different reward values.

[0071]

[0072]

[0073] Table 1

[0074] Based on Table 1, the reward values ​​corresponding to emotion change, response correctness, dialogue efficiency, and dialogue coherence are added together to obtain the final reward value for the business service agent and the object agent.

[0075] S240. Select a target object agent from among the object agents based on the reward value, and train a target business service agent based on the interaction between the business service agent and the target object agent. The target business service agent is used to communicate with the object.

[0076] In this embodiment, the reward value can be used to obtain the interaction performance of the business service agent when interacting with different object agents. The higher the reward value, the higher the corresponding interaction performance. Therefore, the reward value can be used to determine the weaknesses of the business service agent when dealing with different types of objects. Then, a target object agent can be selected from various object agents as a training opponent. The business service agent interacts with the target object agent. That is, based on the optimization of the business service agent's dialogue strategy according to the dialogue interaction with different types of object agents, it further interacts with a specific target object agent to specifically improve the business service agent's dialogue strategy in its weaknesses, thereby obtaining the target business service agent.

[0077] In one example, selecting a target intelligent agent based on a reward value and interacting with the business service intelligent agent can be done cyclically. For example, first select target intelligent agent 1 based on the reward value, then interact the business service intelligent agent with target intelligent agent 1 to update the reward value, then select target intelligent agent 3 based on the updated reward value, then interact the business service intelligent agent with target intelligent agent 3 to update the reward value, and then reselect the target intelligent agent to obtain the target business service intelligent agent.

[0078] In another example, selecting a target agent based on reward value and interacting with the business service agent can be determined according to the training rounds. For example, in the first round of training, the target agent is randomly selected with a 20% probability; in subsequent rounds of training, the target agent with the lowest reward is selected with an 80% probability to improve the weak capabilities of the business service agent.

[0079] In this embodiment, the target business service agent can dynamically adapt to the needs of different types of objects. The target business service agent can be applied to a dialogue system to engage in dialogue with objects. Therefore, in one example, after obtaining the target business service agent, the process further includes: acquiring the dialogue content input by the target object and identifying the object type of the target object based on the dialogue content; generating a target response through the target business service agent based on the object type and dialogue content; acquiring the dialogue context based on the target response and re-identifying the object type based on the dialogue context; and repeating the step of generating the target response until the dialogue ends.

[0080] The target object refers to the object interacting with the target business service agent. The dialogue content input by the target object is provided in text form (such as through a chat window, speech recognition to text, etc.), including specific questions, needs, and other information. Based on the dialogue content, the agent analyzes the object's needs, behavioral patterns, and emotional state to determine the object type. The target business service agent then determines different response strategies based on this type and generates appropriate responses to the target object's needs or questions based on the response strategies and dialogue content. For example, for target objects with vague needs, the business service agent may adopt a questioning and clarification strategy, asking for more details to clarify the object's needs and then generating a response; for target objects with clear needs, the business service agent will directly provide relevant answers to generate a response. The response can be information extracted from a pre-set knowledge base, answers generated by a model, or content generated in combination with the context.

[0081] In the embodiments of this application, within the same dialogue, the target business service agent can continuously adapt to changes in the target object's needs. For example, the target object can input new dialogue content based on the target response of the target business service agent, and then generate a dialogue context based on the target response and the new dialogue content. Subsequently, it is necessary to re-evaluate the target object's needs and emotional state based on the dialogue context. If the target object shows a vague need type in the first round, but clarifies its needs in the second round, it is necessary to re-identify the target object as having a clear need type. The response strategy is adjusted according to the dynamic changes in the re-evaluated target object's needs and emotional state. By continuously executing the steps of generating responses, updating the dialogue state, and re-identifying the target object type, multiple rounds of dialogue are continuously conducted until the conditions for ending the dialogue are met (such as problem resolution, target object satisfaction, etc.), providing more personalized services, ensuring that the interaction with the target object is flexible and effective, and able to respond quickly to changes in the target object's needs.

[0082] The target business service agent can continuously adapt to changes in object requirements, such as Figure 3 As shown, it illustrates the interaction interface between the target object Zhang San and the target business service agent X. The target object first inputs the first round of dialogue content 310 in the interaction interface diagram. The target business service agent identifies the target object as having a vague need based on the dialogue content and generates a response 320 using a dialogue strategy of clarifying questions. The target object inputs the second round of dialogue content 330 in the interaction interface. The target business service agent identifies the new dialogue content as having a clear need based on the new dialogue content and generates a response 340 using a dialogue strategy of direct answering.

[0083] In this embodiment, a business service agent and at least two types of object agents are generated based on historical dialogue data between the business service object and various objects. In a preset reinforcement learning environment, the business service agent engages in multi-turn dialogues with each object agent, thereby obtaining the multi-turn dialogue results between the business service agent and each object agent in the preset reinforcement learning environment. The reinforcement learning environment is used to describe the scene factors required for agent dialogue interaction. The dialogue quality between the business service agent and each object agent in the multi-turn dialogue results is obtained, and reward values ​​are generated based on the dialogue quality. These reward values ​​not only reflect the performance of the business service agent but also help adjust its adaptability to different object agents. Furthermore, based on the reward values, rewards are generated from each object agent... The system selects a target intelligent agent and, based on the interaction between the business service intelligent agent and the target intelligent agent, enables the business service intelligent agent to progressively optimize its dialogue strategy in a targeted manner to train a target business service intelligent agent. In other words, the technical solution provided in this application, on the one hand, allows the business service intelligent agent to dynamically adjust its dialogue strategy in a reinforcement learning environment to adapt to different object styles; on the other hand, by selecting a target intelligent agent through reward values, the business service intelligent agent learns the behavioral characteristics corresponding to a specific intelligent agent and constructs a target business service intelligent agent that can dynamically adapt to different object types. Thus, in practical applications, the target business service intelligent agent dynamically adapts to the needs of different types of objects and can flexibly, accurately, and naturally engage in dialogue with objects, significantly improving dialogue effectiveness and object experience.

[0084] In one embodiment of this application, another dialogue processing method is provided, which can be applied to... Figure 1 The implementation environment shown is illustrated using the example of the method being executed by a server. Figure 4 As shown, this dialogue processing method is in Figure 2 Based on S210 to S240 shown, S210 is extended to S410 to S430; S410 to S430 are described in detail below.

[0085] S410. Identify the object type of each object based on the historical dialogue data between the business service object and each object.

[0086] As described above, historical dialogue data includes various types of object requirements, so different object types can be identified based on historical dialogue data. In particular, historical dialogue data can be clustered to determine object types.

[0087] In one example, identifying the object type of each object includes: cleaning the historical dialogue data to obtain the target historical dialogue data, and extracting features from the target historical dialogue data to obtain dialogue features; performing cluster analysis on the dialogue features to determine the cluster center of the cluster to which each object belongs; and obtaining the object type of each object based on the feature distribution of each cluster center and the historical dialogue content of each object under the cluster center.

[0088] First, the historical dialogue data is cleaned to remove redundant information and noise, retaining the data useful for the clustering task. This cleaning process includes eliminating duplicate dialogue records to ensure that the analyzed data is unique and valid, removing noise data that may interfere with clustering (such as irrelevant text or meaningless dialogue), and ensuring the accuracy of the clustering results.

[0089] The target historical dialogue data obtained from data cleaning is used to extract dialogue features, which include text semantic features, including high-dimensional vectors transformed from each dialogue and TF-IDF (term frequency-inverse document frequency) word vectors obtained by calculating important words in the dialogue, as well as grammatical and syntactic features, such as sentence length and sentence complexity; the dialogue features also include sentiment features extracted by sentiment analysis model and intent features identified by intent recognition model.

[0090] Clustering algorithms are used to perform cluster analysis on dialogue features. Algorithms such as K-Means and DBSCAN are used to obtain an array containing labels for the cluster each object belongs to, indicating which category each object is assigned to (a number from 0 to 3). For example, user_clusters = [0,1,2,0,3] means that object 1 and object 4 belong to category 0, object 2 belongs to category 1, object 3 belongs to category 2, and so on.

[0091] Next, the cluster centers of each object are determined. Each cluster center represents the "typical" characteristics of that class of objects. Therefore, the characteristics of each class can be determined by analyzing the feature distribution of these cluster centers. In one example, the feature distribution of the cluster centers can be compared with predefined object types, and the object type corresponding to the cluster center can be determined based on the comparison.

[0092] In one example, after analyzing the cluster centers, the dialogue content of objects belonging to the same cluster can be examined to analyze their language style, sentiment, etc., to further confirm the characteristics of the objects. If the object type determined by the cluster centers is the same as the object type determined by the dialogue content under the same cluster, then the object type is taken as the final object type; if the object type determined by the cluster centers is different from the object type determined by the dialogue content under the same cluster, a different clustering algorithm can be used for cluster analysis.

[0093] S420. Obtain the business service object and the representative dialogue samples corresponding to each object type.

[0094] In this application embodiment, representative dialogue samples corresponding to business service objects are obtained, as well as representative dialogue samples corresponding to objects of each object type. The representative dialogue samples refer to typical dialogues that can reflect the characteristics of different object types.

[0095] The process of obtaining representative dialogue samples from cluster analysis results includes: obtaining the dialogue style of objects for each object type; constructing object questions based on the dialogue style, determining the expected response examples corresponding to the object questions from the knowledge base, and generating representative dialogue samples corresponding to business service objects based on the object questions and expected response examples; constructing service questions for business service objects, constructing object response examples corresponding to the service questions based on the dialogue style, and generating representative dialogue samples corresponding to objects for each object type based on the service questions and object response examples.

[0096] Dialogue style, including tone, language expression, emotional changes, and questioning methods, can be obtained from cluster analysis results. Based on the dialogue style of different object types, questions that objects of that type might ask are constructed, and expected response examples corresponding to these questions are extracted from the knowledge base. This generates a set of complete and representative dialogue samples, serving as the basis for subsequent training and model optimization. The expected response refers to the best answer that the business service object should give in a specific situation; the expected response will vary depending on the dialogue style. For example, a representative dialogue sample might be: Object question: "I want to buy a mobile phone, can you recommend one?", with an expected response example: "Are you more concerned about battery life or camera performance?".

[0097] The representative dialogue samples for each target audience consist of service questions posed by the business service target. These service questions are crucial for understanding the target audience's needs and guiding the dialogue. The responses are designed based on different dialogue styles, with examples tailored to each type of target audience. For instance, for a representative dialogue sample for a target audience with clear needs, the service question might be: "This phone has a 5000mAh battery and supports fast charging." An example response might be: "How long does it take to charge?" For a representative dialogue sample for a target audience with strong emotions, the service question might be: "How do you feel about the current price?" An example response might be: "Why is this phone still so expensive?"

[0098] In one example, to ensure the diversity of the samples, the representative dialogue samples not only include common problems, but also anomalies or edge cases to enhance the agent's ability to respond.

[0099] In one example, after obtaining representative dialogue samples, the richness of the representative dialogue samples can be enhanced through data expansion, such as multimodal enhancement, such as text + speech or image, synonym replacement, tone adjustment, etc.

[0100] S430. Adjust the base model based on representative dialogue samples to obtain business service agents and object agents corresponding to each object type.

[0101] In this embodiment, the base model is a large language model with certain language understanding and dialogue generation capabilities. The base model is supervised and fine-tuned through representative dialogue samples. During the training process, the model learns based on questions and correct responses, adjusts its parameters, and obtains a business service agent that can better cope with the needs and dialogue styles of different types of objects, as well as an object agent that can simulate the behavior of a certain type of object.

[0102] In this embodiment, by analyzing the dialogue content between the business service object and the dialogue, the object is classified according to the characteristics of the dialogue. Through this classification, by constructing representative dialogue samples and performing supervised fine-tuning (SFT), business object intelligent agents that can cope with various dialogue styles and various types of object intelligent agents that simulate different object behaviors can be trained. This allows for a better understanding of the object's needs and emotions, thereby providing more personalized and efficient services.

[0103] This application provides another dialogue processing method, which can be applied to... Figure 1 The implementation environment shown is illustrated using the example of the method being executed by a server. Figure 5 As shown, this dialogue processing method is in Figure 2Based on what is shown, Figure 2 The S220 shown is extended to S510 to S540, wherein the preset reinforcement learning environment includes executable actions and a dialogue state used to describe the agent's current situation in the dialogue. S510 to S540 are described in detail below.

[0104] S510. Randomly select candidate object agents from various object agents and generate the current dialogue state.

[0105] S520: Obtain the executable action selected by the business service agent based on the current dialogue state, and update the dialogue context based on the executable action.

[0106] S530. Obtain the object response generated by the candidate object agent based on the dialogue context to obtain the dialogue result of the current dialogue round.

[0107] S540. Generate the next dialogue state based on the dialogue result of the current dialogue round, and re-execute the steps of obtaining the executable actions selected by the business service agent and updating the dialogue context to obtain the multi-round dialogue results between the business service agent and the candidate object agent.

[0108] In this embodiment of the application, a candidate object agent is randomly selected from various object agents. Initially, an initial dialogue state can be generated. This initial dialogue state includes an empty dialogue context, an initial object sentiment state (e.g., neutral), and a current dialogue round of 0 to initiate the dialogue process. Subsequently, the current dialogue state is generated based on the dialogue style of the candidate object agent and the dialogue result with the business service agent.

[0109] The business service agent makes decisions based on the current dialogue state. The selectable actions for the business service agent include: Direct Answer: Providing a direct answer based on a knowledge base to resolve the object's specific question; this action is typically suitable for objects with explicit needs. Clarifying Question: Asking the object a question to further clarify its needs or problem; suitable for objects with ambiguous needs, helping the agent obtain more information to provide a more accurate answer. Recommended Help Content: Providing relevant help links or documentation; suitable for scenarios where the object requests more information or seeks self-service problem-solving. End Conversation: Proactively ending the conversation; suitable when the object clearly expresses satisfaction or the problem has been resolved.

[0110] Each executable action corresponds to a different dialogue strategy. The business service agent needs to make appropriate choices based on the feedback from the object agent and the progress of the dialogue. It updates the dialogue context based on the executable actions, that is, updates the response of the business service agent to the dialogue context. Then, the candidate object agent generates the object response based on the updated dialogue context, completes the current dialogue round, and obtains the dialogue result of the current dialogue round. The dialogue result includes at least the executable actions selected by the business service agent and the responses generated by the candidate object agents.

[0111] like Figure 6 As shown, the dialogue process between a business service agent and a fuzzy demand-type object agent is illustrated in reinforcement learning environment 1, and the dialogue process between a business service agent and an emotion-strong object agent is illustrated in reinforcement learning environment 2.

[0112] The next dialogue state is generated based on the dialogue result of the current dialogue round, and training or execution of multiple dialogue rounds continues. That is, S520 to S530 are repeatedly executed to obtain the dialogue result of the next dialogue round, and thus the multi-round dialogue results between the business service agent and the candidate object agent are obtained.

[0113] Then, another candidate object agent is randomly selected from each object agent, and S510 to S540 are executed repeatedly to obtain the multi-round dialogue results between the business service agent and other object agents.

[0114] In one example, the dialogue states corresponding to different dialogue rounds can be different, but each dialogue state includes basic information such as dialogue context, emotional state, and dialogue round. To enhance the agent's understanding of the dialogue environment and make decisions more intelligent and context-sensitive, finer-grained state modeling can be introduced based on the dialogue results of different dialogue rounds. This includes object intent distribution (e.g., purchase consultation, complaint feedback, product recommendation, technical support) and simulated object personality. Therefore, generating the next dialogue state based on the dialogue result of the current dialogue round includes: identifying the emotional state and intent distribution of candidate object agents based on the dialogue result of the current dialogue round; constructing personality parameters for candidate object agents based on the dialogue result of the current dialogue round, with personality parameters used to simulate personalized objects; and generating the next dialogue state based on the dialogue context, emotional state, intent distribution, and personality parameters.

[0115] Specifically, the dialogue result of the current dialogue round is input into the sentiment analysis model to obtain the sentiment state of the candidate agent in the current dialogue round, and the dialogue result is input into the intent classifier to obtain the current intent distribution of the candidate agent.

[0116] To enable business service agents to use objects with different personalities, different personality parameters are constructed for each object agent in a reinforcement learning environment. These personality parameters are used to simulate personalized objects. For example, personality parameters include at least one of personality, patience, and knowledge level. For instance, for an object agent with clear needs, the knowledge level is high, and for an object agent with strong emotions, the patience level is low. The next dialogue state includes the dialogue context, emotional state, dialogue turn, intent distribution, and personality parameters. Thus, the business service agent can optimize its decision-making behavior based on the next dialogue state.

[0117] In other embodiments, when introducing more granular state modeling based on the dialogue results of different dialogue rounds, contextual information, such as knowledge base availability (whether the current question can be directly queried) and policy history (whether previous policies were effective), can also be introduced.

[0118] In this embodiment of the application, obtaining the multi-round dialogue results between the business service agent and the candidate object agent includes: updating the dialogue rounds between the business service agent and the candidate object agent; obtaining the dialogue satisfaction of the object agent in the dialogue results up to the latest dialogue round; if it is determined that the dialogue between the business service agent and the candidate object agent has ended based on at least one of the dialogue satisfaction and the updated dialogue rounds, then generating multi-round dialogue results based on the dialogue results up to the latest dialogue round.

[0119] Specifically, after the business service agent executes an executable action, the candidate object agent generates an object response, and a back-and-forth dialogue is conducted. The dialogue round between the business service agent and the candidate object agent is updated by incrementing the dialogue round by 1. The dialogue satisfaction of the object agent in the dialogue results up to the latest dialogue round is obtained. This dialogue satisfaction can be determined based on the emotional state of the object agent and the dialogue content of the object agent. For example, if the emotional state of the object agent is good and the dialogue content of the object agent includes preset keywords such as "good" or "satisfied", then the dialogue satisfaction is determined to be greater than the preset satisfaction threshold.

[0120] In one example, if the dialogue satisfaction is greater than a preset satisfaction threshold, the dialogue between the business service agent and the candidate agent is determined to have ended. If the updated dialogue rounds exceed the preset maximum number of rounds (e.g., 10 rounds), the dialogue is determined to have ended. After the dialogue is determined to have ended, all dialogue results up to the latest dialogue round are taken as the multi-round dialogue results.

[0121] In this embodiment, a multi-agent reinforcement learning dialogue environment is implemented, in which a business service agent interacts with an object agent, selects an appropriate action based on the current dialogue state, and promotes the dialogue by updating the dialogue context, emotional state, and turn order. In this way, the dialogue training of multiple agents can be simulated, enabling the business service agent to continuously optimize its strategy in the reinforcement learning environment, so that it can better serve different types of objects and improve the dialogue quality in practical applications.

[0122] This application also provides another dialogue processing method, which can be applied to... Figure 1 The implementation environment shown is illustrated using the example of the method being executed by a server. Figure 7 As shown, in Figure 2 Based on what is shown, Figure 2 The S240 shown is extended to S710 to S730. S710 to S730 are described in detail below.

[0123] S710: Obtain the average reward value of the business service agent's multiple interactions with each object agent.

[0124] S720. Set the random selection probability and determine the target selection probability based on the random selection probability. The target selection probability is greater than the random selection probability.

[0125] S730: Select the target agent with the lowest average reward value based on the target selection probability.

[0126] In this embodiment, the target agent is dynamically selected as the adversary, enabling the business service agent to learn diverse dialogue styles and optimize itself to address its weaknesses.

[0127] Specifically, the average reward value of each object agent and the business service agent in n interactions is calculated. Based on the average reward value, it can be determined whether the corresponding object type is a weakness of the business service agent. For example, if the average reward obtained by the business service agent in the interaction with the object agent with strong emotions is much lower than that of other types, it indicates that the business service agent is insufficient in dealing with emotional objects and needs to be trained.

[0128] Set the random selection probability, which indicates the probability of randomly selecting the target intelligent agent from various intelligent agents. For example, if the random selection probability is 10% to 30%, such as 20%, then the target selection probability is 1-20%=80%. Therefore, with an 80% probability, the intelligent agent with the lowest average reward value is selected as the target intelligent agent for the next round of interaction.

[0129] In this embodiment of the application, in order to avoid the business service agent from focusing too much on certain specific types of objects and ignoring the adaptability of other object types, a random number can be generated. If the random number is greater than the random selection probability, the target object agent with the lowest average reward value is selected according to the target selection probability. If the random number is greater than the random selection probability, a random object agent is selected as the target object agent according to the random selection probability.

[0130] In this embodiment of the application, in order to ensure that the business service agent can adapt to a wide range of object types, it is necessary to periodically reset the agent's opponent selection rules. The method further includes:

[0131] The performance differences of the business service agent on different object agents are obtained based on the training state of the business service agent; the selection probability of selecting the target object agent from each object agent is adjusted based on the performance differences.

[0132] The training state of the business service agent can be the training rounds. For example, if the target agent with the lowest average reward value is selected with an 80% probability, and the target agent is randomly selected with a 20% probability, and the number of interactions between the business service agent and the target agent (training rounds) reaches a preset threshold, then the performance difference of the business service agent on different target agents is obtained. This performance difference can be the reward convergence. For example, if the change in the average reward value within the most recent set number of interactions is less than a set threshold (e.g., 0.5), then the business service agent is considered to have stabilized and a reset is triggered. This performance difference can also be the balance of the business service agent across all object types. For example, if the mean variance of the reward is too large (indicating that some object types are not adaptable enough), then a reset is triggered.

[0133] In one example, after triggering the opponent selection rule reset, the random selection probability can be adjusted, such as increasing the random selection probability, for example, randomly selecting the target agent with a 70% probability.

[0134] In one example, the number of training rounds of the business service agent on different object agents can be counted. If the number of training rounds of an object agent of a certain object type is too small (<20%), the probability of selecting an object agent of that specific object type will be increased. If the number of training rounds of an object agent of a certain object type is too large (>50%), the probability of selecting an object agent of that specific object type will be decreased.

[0135] In one example, for the object agent corresponding to the object type with the lowest reward value, its corresponding selection probability can be increased to increase the training weight (i.e., the number of training times for the business service agent on this object type increases); for the object agent corresponding to the object type with the highest reward value, its corresponding selection probability can be decreased to reduce the training weight and prevent overfitting.

[0136] In this embodiment, by dynamically selecting target agents as adversaries, the business service agents can learn diverse dialogue styles and optimize themselves to address their weaknesses. The training process balances efficient learning and generalization capabilities, preventing the business service agents from relying on a fixed pattern.

[0137] For ease of understanding, this application also provides a dialogue processing method applied to a dialogue processing system. Taking an e-commerce company that wants to optimize its customer service system and handle customer issues more efficiently through intelligent customer service as an example, the company provides a large amount of historical dialogue data, which includes various types of object needs and customer service response styles. The goal is to build a customer service intelligent agent (i.e., the aforementioned business service intelligent agent) that can dynamically adapt to different object needs.

[0138] The dialogue processing system includes a Base Agent training module and a reinforcement learning training module, such as... Figure 8 As shown, the BaseAgent training module includes a clustering module and a fine-tuning training module.

[0139] The clustering module aims to cluster customer service and object dialogue data, determine the types of customer service representatives and objects, and generate corresponding agent configurations. Input data includes cleaned customer service and object dialogue records; output data consists of customer service type (1 type) and object type (4 types).

[0140] like Figure 9 As shown, the clustering process of the clustering module includes: S910, loading historical dialogue data; S920, data cleaning; S930, feature extraction using NLP; S940, cluster analysis using a clustering algorithm; and S950, saving representative dialogue samples for customer service and each object type.

[0141] The data cleaning process includes: removing noise (such as spelling errors), deduplication (repeated conversations), and extracting text fields (object questions, customer service answers).

[0142] The feature extraction process includes: generating text embeddings using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model; BERT embeddings are used to capture semantic features, including contextual understanding and dialogue intent; and extracting dialogue features, such as semantics, intent, and sentiment, from the text embeddings.

[0143] The clustering analysis process includes selecting the K-means algorithm for clustering. The K-means algorithm can divide conversations into different categories to capture the conversational styles of customer service representatives and individuals. The clustering parameter is set to 5, and the clustering analysis is performed.

[0144] Clustering Results Output: Customer Service Type: Unified into 1 category, catering to diverse customer styles. Customer Type: Divided into 4 categories based on dialogue style and intent (vague need type, explicit need type, strong emotion type, casual chat type); Vague need type: The customer's question is unclear and requires further clarification. For example, the customer may not know the type of information they want, such as, "How is this product?"; Explicit need type: The customer has a clear question and need, expecting a quick and direct answer, such as, "What is the battery life of these headphones?"; Strong emotion type: The customer expresses strong emotions (such as anxiety, dissatisfaction, or excitement), requiring customer service to have a certain level of emotional understanding and reassurance, such as, "Why hasn't my order arrived yet?"; Casual chat type: The customer has not clearly stated a question and prefers casual or aimless conversation, such as, "What products have you recommended recently?"

[0145] The goal of the fine-tuning training module is to train basic agents by constructing an instruction dataset, i.e., representative dialogue samples, and performing SFT based on qwen (e.g., qen2). The input is the instruction dataset; the output is the basic agents trained by SFT (1 customer service agent and 4 object agents).

[0146] like Figure 10 As shown, the training fine-tuning process of the fine-tuning module includes:

[0147] S1010 Receive clustering module output. S1020 Construct instruction data. S1030 SFT fine-tuning training. S1040 Save the fine-tuned agent.

[0148] The clustering results generate instruction data (i.e., the aforementioned representative dialogue samples), including question-response pairs. For example, an instruction for a vague need-based question: Customer Service Question: What specific aspect would you like to know? Response: I'm not really sure, just looking around. An instruction for a clear need-based question: Customer Service Question: The battery life is 8 hours. Response: Understood, thank you. An instruction for a strongly emotional question: Customer Service Question: I'm sorry to have kept you waiting. Response: Okay, please process it as soon as possible. An instruction for a casual conversation-based question: Customer Service Question: This product is pretty good. Response: Hmm, really? Do you have any other recommendations?

[0149] Using large language models (LLMs) such as Qen as a foundation, the agent is fine-tuned using SFT based on the instructed data. Output: Customer service agent: capable of handling dialogue styles of various objects; Object agent: simulates dialogue behavior of different types of objects.

[0150] The reinforcement learning training module includes an Env building module and an opponent selection module. The goal of the Env building module is to construct a reinforcement learning environment based on the Gym interface (the standard environment interface in reinforcement learning frameworks, containing `reset()` and `step()` methods for simulating multi-round agent interactions) to achieve multi-agent dialogue training. Input data includes the current state and action; output data includes the next state, reward value, and whether to terminate the dialogue.

[0151] In reinforcement learning, states include: dialogue context (object input, customer service response); object emotional state (positive, neutral, negative); dialogue turn (recording the current turn); and object type (current object type, such as vague demand type).

[0152] Actions include the set of actions that the customer service agent can perform, such as answering questions directly, clarifying questions, recommending help content, and ending the conversation.

[0153] Rewards include: detecting changes in the subject's emotional state in subsequent inputs through a sentiment analysis model, and the subject's emotional change (positive / neutral / negative) in response to the answer. For example, a positive change in the subject's emotion results in +10 points, while a negative change results in -10 points. Rewards also include: dialogue efficiency, such as resolving the problem within 3 rounds: +15 points; failing to resolve the problem within 10 rounds: -10 points. Rewards further include: answer quality, i.e., whether the answer to the subject's question is correct and whether a matching answer can be found in the knowledge base or model. Judgment criteria: matching and verification against the knowledge base or preset standard answers. A completely correct answer: +15 points; a partially correct answer: +5 points; an incorrect answer: -10 points.

[0154] Terminating a dialogue includes: the other party explicitly expressing satisfaction or ending the dialogue, or the number of dialogue rounds exceeding the maximum number (e.g., 10 rounds).

[0155] like Figure 11 As shown, the reinforcement learning process of the reinforcement learning training module includes:

[0156] S1110, Call the `env.reset()` method. S1120, Initialize the object agent. S1130, Initialize the dialogue state. S1140, Call the `env.step(action)` method. S1150, Update the dialogue. S1160, Calculate the reward value based on the dialogue result. S1170, Determine whether to terminate the dialogue.

[0157] The `env.reset()` function initializes the environment, performing the following tasks: randomly selecting an object agent as the adversary; initializing the object's state, including an empty dialogue context; initializing the object's emotional state (neutral) and the current dialogue turn (0); and using `env.step(action)` to select an action by the customer service agent within the current state.

[0158] The dialogue update includes: updating the dialogue context based on the action, where the dialogue context records the customer service response and the object's reply, and then the object agent generates a reply based on the current dialogue context and emotional state, and uses the sentiment analysis module to update the object's emotional state and update the dialogue round.

[0159] The goal of the opponent selection module is to dynamically select target agents as opponents, ensuring that the customer service agent learns diverse dialogue styles. Input data includes reward records and object types; output data is the opponent for the next round.

[0160] The opponent selection module evaluates the performance of all target agents, recording the win rate or reward value in interactions with the customer service agent. Target agents with low win rates or poor reward values ​​are considered weaknesses of the customer service agent. Based on the evaluation results, the target agent with the lowest reward value is selected as the training opponent.

[0161] Optionally, the agent with the lowest average reward can be selected with a probability of 1-p using the reward.

[0162] like Figure 12 As shown, the opponent selection module's process for selecting an opponent includes:

[0163] S1210, Set of target agents. S1220, Define the random selection probability p. S1230, Record the average reward for each target agent. S124, Execute the selection policy.

[0164] The selection strategy includes: generating a random number r; if the random number is less than the random selection probability, then randomly selecting an object agent from the object agent set with probability p; if the random number is greater than the random selection probability, then selecting an object agent based on the reward, that is, selecting the object agent with the lowest average reward with probability 1-p.

[0165] In this embodiment, it is necessary to periodically reset the object agent's opponent selection rules to ensure the customer service agent's ability to handle a wide range of object types.

[0166] In this embodiment, users are divided into four categories (fuzzy demand type, explicit demand type, strong emotion type, and casual chat type) using a clustering algorithm, and instruction data is constructed for each type of user for training, comprehensively covering diverse behaviors; the customer service agent is trained through SFT fine-tuning and reinforcement learning, which can dynamically adjust the dialogue strategy to adapt to different user styles; the use of a Gym environment and a multi-opponent selection mechanism ensures that the customer service agent can optimize its weaknesses and enhance its robustness in multiple rounds of training.

[0167] In this embodiment, if an opponent is selected for training via the opponent selection module, and the training rounds reach the set number, a trained customer service agent is output and deployed to the customer service system to engage in dialogue with the target. The customer service agent implicitly analyzes the input content to identify the target type. Within the same dialogue, the customer service agent can continuously adapt to changes in the target's needs. For example: vaguely defined user → clarifying question → explicitly defined user → direct answer.

[0168] This application describes an apparatus embodiment that can be used to execute the dialogue processing method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the dialogue processing method described above.

[0169] This application provides a dialogue processing device, such as... Figure 13 As shown, the device includes...

[0170] The generation module 1310 is used to generate business service agents and at least two types of object agents based on the historical dialogue data between the business service object and each object.

[0171] The acquisition module 1320 is used to acquire the multi-round dialogue results between the business service agent and each object agent in a preset reinforcement learning environment; the reinforcement learning environment is used to describe the scene factors required for the agent's dialogue interaction.

[0172] The acquisition module 1320 is also used to acquire the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generate reward values ​​for the interaction between the business service agent and each object agent based on the dialogue quality.

[0173] The training module 1330 is used to select a target object agent from the various object agents according to the reward value, and to train a target business service agent based on the interaction between the business service agent and the target object agent. The target business service agent is used to communicate with the object.

[0174] In one embodiment of this application, based on the foregoing scheme, the generation module is further configured to identify the object type of each object based on the historical dialogue data between the business service object and each object; obtain representative dialogue samples corresponding to the business service object and each object type; adjust the base model based on the representative dialogue samples to obtain the business service agent and the object agent corresponding to each object type.

[0175] In one embodiment of this application, based on the foregoing scheme, the generation module is further configured to perform data cleaning processing on the historical dialogue data to obtain target historical dialogue data, and to extract features from the target historical dialogue data to obtain dialogue features; to perform cluster analysis on the dialogue features to determine the cluster center of the cluster to which each object belongs; and to obtain the object type of each object based on the feature distribution of each cluster center and the historical dialogue content of each object under the cluster center.

[0176] In one embodiment of this application, based on the foregoing scheme, the generation module is further configured to obtain the dialogue style of objects of each object type; construct object questions based on the dialogue style, determine the expected response examples corresponding to the object questions from the knowledge base, and generate representative dialogue samples corresponding to the business service objects based on the object questions and expected response examples; construct service questions for the business service objects, construct object response examples corresponding to the service questions based on the dialogue style, and generate representative dialogue samples corresponding to objects of each object type based on the service questions and object response examples.

[0177] In one embodiment of this application, the preset reinforcement learning environment includes executable actions and a dialogue state describing the current situation of the agent in the dialogue; the acquisition module is further configured to randomly select candidate object agents from the various object agents and generate the current dialogue state; acquire the executable actions selected by the business service agent according to the current dialogue state, and update the dialogue context according to the executable actions; acquire the object responses generated by the candidate object agents according to the dialogue context to obtain the dialogue result of the current dialogue round; generate the next dialogue state according to the dialogue result of the current dialogue round, and re-execute the steps of acquiring the executable actions selected by the business service agent and updating the dialogue context to obtain the multi-round dialogue results between the business service agent and the candidate object agents.

[0178] In one embodiment of this application, based on the foregoing scheme, the acquisition module is further configured to identify the emotional state and intent distribution of the candidate object agent according to the dialogue result of the current dialogue round; construct personality parameters for the candidate object agent according to the dialogue result of the current dialogue round, the personality parameters being used to simulate a personalized object; and generate the next dialogue state according to the dialogue context, the emotional state, the intent distribution, and the personality parameters.

[0179] In one embodiment of this application, based on the foregoing scheme, the acquisition module is further configured to update the dialogue rounds between the business service agent and the candidate object agent; acquire the dialogue satisfaction of the object agent in the dialogue results up to the latest dialogue round; if it is determined that the dialogue between the business service agent and the candidate object agent has ended based on at least one of the dialogue satisfaction and the updated dialogue rounds, then generate the multi-round dialogue results based on the dialogue results up to the latest dialogue rounds.

[0180] In one embodiment of this application, based on the foregoing scheme, the acquisition module is further configured to identify the emotional changes in the responses generated by the object agent and the correctness of the responses from the business service agent in the multi-turn dialogue results; determine the dialogue efficiency in the multi-turn dialogue results based on the response content and dialogue rounds of the object agent in the multi-turn dialogue results; and use the emotional changes, the correctness of the responses, and the dialogue efficiency as the dialogue quality.

[0181] In one embodiment of this application, based on the foregoing scheme, the dialogue quality further includes dialogue coherence calculated by the similarity between the current response and the previous response of the business service agent; the acquisition module is further configured to set dialogue weights for the emotion change, the correctness of the response, the dialogue efficiency, the dialogue coherence and the dialogue consistency according to the business needs corresponding to the business service agent; and calculate the reward value based on the emotion change, the correctness of the response, the dialogue efficiency and the dialogue coherence, and the corresponding dialogue weights.

[0182] In one embodiment of this application, based on the foregoing scheme, the training module is further configured to obtain the average reward value of the business service agent interacting with each object agent multiple times; set a random selection probability, and determine a target selection probability based on the random selection probability, wherein the target selection probability is greater than the random selection probability; and select the target object agent with the lowest average reward value based on the target selection probability.

[0183] In one embodiment of this application, based on the foregoing scheme, the training module is further configured to obtain the performance differences of the business service agent on different object agents according to the training state of the business service agent; and adjust the selection probability of selecting the target object agent from the various object agents according to the performance differences.

[0184] In one embodiment of this application, based on the foregoing scheme, the device further includes an application module, configured to acquire dialogue content input by a target object, and identify the object type of the target object based on the dialogue content; generate a target response through the target business service agent based on the object type and dialogue content; acquire a dialogue context generated based on the target response, and re-identify the object type based on the dialogue context; and repeatedly execute the step of generating the target response until the dialogue ends.

[0185] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0186] The device provided in the above embodiments can be located in a terminal or in a server.

[0187] Embodiments of this application also provide an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the dialogue processing method described above.

[0188] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0189] It should be noted that, Figure 14 The computer system 1400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0190] like Figure 14As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in read-only memory (ROM) 1402 or a program loaded from storage portion 1408 into random access memory (RAM) 1403. The RAM 1403 also stores various programs and data required for system operation. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0191] In some embodiments, the following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1414, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1410 as needed so that computer programs read from it can be installed into the storage section 1408 as needed.

[0192] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as a computer program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1409, and / or installed from removable medium 1414. When the computer program is executed by processor (CPU) 1401, it performs various functions defined in the system of this application.

[0193] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0195] The units or modules described in the embodiments of this application can be implemented in software or hardware, and can also be located in a processor. The names of these units or modules do not necessarily limit the specific unit or module itself.

[0196] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the dialogue processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0197] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the dialogue processing method provided in the various embodiments described above.

[0198] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0199] Other embodiments of this application will readily conceive of by considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0200] The above content is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A dialog processing method characterized by, include: Based on the historical dialogue data between the business service object and each object, generate a business service intelligent agent and at least two types of object intelligent agents; Obtain the multi-round dialogue results between the business service agent and each object agent in a preset reinforcement learning environment; The reinforcement learning environment is used to describe the scenario factors required for intelligent agent dialogue interaction. The dialogue quality between the business service agent and each object agent in the multi-turn dialogue results is obtained, and the reward value of the interaction between the business service agent and each object agent is generated according to the dialogue quality. Based on the reward value, a target object agent is selected from the various object agents, and a target business service agent is trained based on the interaction between the business service agent and the target object agent. The target business service agent is used to communicate with the object.

2. The method of claim 1, wherein, The process of generating business service agents and at least two types of object agents based on historical dialogue data between business service objects and various objects includes: Based on the historical dialogue data between the business service object and each object, identify the object type of each object; Obtain the business service object and representative dialogue samples corresponding to each object type; The base model is adjusted based on the representative dialogue samples to obtain the business service agent and the object agent corresponding to each object type.

3. The method of claim 2, wherein, The step of identifying the object type of each object based on the historical dialogue data between the business service object and each object includes: The historical dialogue data is cleaned to obtain target historical dialogue data, and features are extracted from the target historical dialogue data to obtain dialogue features. Cluster analysis is performed on the dialogue features to determine the cluster center of each object's cluster. The object type of each object is obtained based on the characteristic distribution of each cluster center and the historical dialogue content of each object under each cluster center.

4. The method of claim 2, wherein, The step of obtaining the business service object and representative dialogue samples corresponding to each object type includes: Obtain the dialogue style of the object for each of the specified object types; Based on the dialogue style, an object question is constructed, and the expected response example corresponding to the object question is determined from the knowledge base. Based on the object question and the expected response example, a representative dialogue sample corresponding to the business service object is generated. Construct service questions for business service objects, and construct object response examples corresponding to the service questions based on the dialogue style. Generate representative dialogue samples corresponding to each object type based on the service questions and object response examples.

5. The method of claim 1, wherein, The preset reinforcement learning environment includes executable actions and a dialogue state used to describe the agent's current situation in the dialogue; The step of obtaining the multi-turn dialogue results between the business service agent and each object agent in a preset reinforcement learning environment includes: Randomly select candidate object agents from the various object agents and generate the current dialogue state; Obtain the executable action selected by the business service agent based on the current dialogue state, and update the dialogue context based on the executable action; Obtain the object response generated by the candidate object agent based on the dialogue context to obtain the dialogue result of the current dialogue round; The next dialogue state is generated based on the dialogue result of the current dialogue round, and the steps of obtaining the executable action selected by the business service agent and updating the dialogue context are re-executed to obtain the multi-round dialogue results between the business service agent and the candidate object agent.

6. The method of claim 5, wherein, The step of generating the next dialogue state based on the dialogue result of the current dialogue round includes: Identify the emotional state and intent distribution of the candidate agent based on the dialogue results of the current dialogue round; Based on the dialogue results of the current dialogue round, personality parameters are constructed for the candidate intelligent agent, and the personality parameters are used to simulate a personalized object; The next dialogue state is generated based on the dialogue context, the emotional state, the intent distribution, and the personality parameters.

7. The method of claim 5, wherein, The process of obtaining the multi-turn dialogue results between the business service agent and the candidate object agent includes: Update the dialogue rounds between the business service agent and the candidate object agent; Obtain the dialogue satisfaction of the target agent in the dialogue results up to the latest dialogue round; If the dialogue between the business service agent and the candidate agent is determined to have ended based on at least one of the dialogue satisfaction and the updated dialogue round, then the multi-round dialogue result is generated based on the dialogue result up to the latest dialogue round.

8. The method of claim 1, wherein, The step of obtaining the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results includes: Identify the emotional changes in the responses generated by the object agent and the correctness of the responses from the business service agent in the results of the multi-turn dialogue; The dialogue efficiency in the multi-turn dialogue results is determined based on the response content of the target agent and the number of dialogue turns in the multi-turn dialogue results. The emotional changes, the correctness of the responses, and the efficiency of the dialogue are considered as the quality of the dialogue.

9. The method of claim 8, wherein, The dialogue quality also includes dialogue coherence calculated by the similarity between the current response and the previous response of the business service agent. Based on the dialogue quality, reward values ​​are generated for the interactions between the business service agent and each object agent, including: Based on the business requirements corresponding to the business service agent, set dialogue weights for the emotion change, the correctness of the response, the dialogue efficiency, the dialogue coherence and the dialogue consistency. The reward value is calculated based on the emotional changes, the correctness of the response, the efficiency of the dialogue, the coherence of the dialogue, and the corresponding dialogue weights.

10. The method according to any one of claims 1 to 9, characterized in that, The step of selecting a target intelligent agent from the various intelligent agents based on the reward value includes: Obtain the average reward value of the business service agent's multiple interactions with each of the object agents; A random selection probability is set, and a target selection probability is determined based on the random selection probability, wherein the target selection probability is greater than the random selection probability; The target agent with the lowest average reward value is selected based on the target selection probability.

11. The method of claim 10, wherein, The method further includes: The performance differences of the business service agent on different object agents are obtained based on the training state of the business service agent. The selection probability of choosing the target object agent from the various object agents is adjusted based on the performance differences.

12. The method according to any one of claims 1 to 9, characterized in that, After obtaining the target business service agent based on the interaction between the business service agent and the target object agent, the method further includes: Obtain the dialogue content input by the target object, and identify the object type of the target object based on the dialogue content; The target business service agent generates a target response based on the object type and dialogue content. Obtain the dialogue context generated based on the target response, and re-identify the object type based on the dialogue context; Repeat the steps to generate the target response until the conversation ends.

13. A dialog processing apparatus characterized by comprising: include: The generation module is used to generate business service agents and at least two types of object agents based on the historical dialogue data between the business service object and each object. The acquisition module is used to acquire the multi-round dialogue results between the business service agent and each object agent in a preset reinforcement learning environment; The reinforcement learning environment is used to describe the scenario factors required for intelligent agent dialogue interaction. The acquisition module is also used to acquire the dialogue quality between the business service agent and each object agent in the multi-turn dialogue results, and generate reward values ​​for the interaction between the business service agent and each object agent based on the dialogue quality. The training module is used to select a target object agent from the various object agents based on the reward value, and to train a target business service agent based on the interaction between the business service agent and the target object agent. The target business service agent is used to communicate with the object.

14. An electronic device, comprising: include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method of any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from and executes the computer program to cause the electronic device to perform the method of any one of claims 1 to 12.

Images