Session simulation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a session simulation method and device, a storage medium and a program product, and the method comprises the steps: obtaining a session test case generated based on real session data between a real user and a customer service, the session test case comprising a real session intention and a user response behavior in the real session data; and inputting the session test case into the simulation robot, so that the simulation robot performs a simulation session with the customer service robot based on the real session intention, and reproduces the user response behavior in the simulation session process. In the scheme, the customer service robot can accumulate more experiences and learn more problem solving methods through a mode of enabling the customer service robot to carry out various simulation sessions, so that various problems of real users can be better understood and coped with, accurate answers are provided for the real users, and the user experience is improved. And the self reliability and the user satisfaction degree are improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a session simulation method, device, storage medium and program product. Background Art

[0002] With the development of artificial intelligence technology, various machine learning models, such as Large Language Model (LLM), have been widely used in various customer service robots. Based on the natural language processing capabilities of these machine learning models, customer service robots can simulate the behavior of human customer service and provide users with instant, efficient and low-cost customer service, such as understanding and responding to users' inquiries and questions, which greatly alleviates the shortage of human customer service resources.

[0003] However, in actual applications, due to the randomness and uncertainty of machine learning models when generating content, the reliability of customer service robots is limited, resulting in inaccurate or misleading answers when handling user questions. This unstable performance of customer service robots directly affects the user experience and reduces user satisfaction with customer service. Summary of the invention

[0004] The embodiments of the present application provide a conversation simulation method, device, storage medium, and program product to improve the reliability of a customer service robot and user satisfaction with customer service.

[0005] In a first aspect, the present application provides a session simulation method, the method comprising:

[0006] Obtaining a session test case generated based on real session data between a real user and customer service, wherein the session test case includes real session intentions and user response behaviors in the real session data;

[0007] The conversation test case is input into the simulation robot so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention and reproduces the user response behavior during the simulation conversation.

[0008] In a second aspect, the present application provides a conversation simulation device, the device comprising:

[0009] An acquisition module, used to acquire a session test case generated based on real session data between a real user and customer service, wherein the session test case includes real session intentions and user response behaviors in the real session data;

[0010] The simulation module is used to input the conversation test case into the simulation robot so that the simulation robot can conduct a simulation conversation with the customer service robot based on the real conversation intention and reproduce the user response behavior during the simulation conversation.

[0011] In a third aspect, the present application provides a computing device comprising a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the session simulation method as described in the first aspect.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a computer to implement the session simulation method as described in the first aspect.

[0013] In a fifth aspect, the present application provides a computer program product storing a computer program, which implements the session simulation method as described in the first aspect when executed by a computer.

[0014] In the conversation simulation method provided in the embodiment of the present application, a conversation test case generated based on the real conversation data between a real user and a customer service is first obtained, wherein the conversation test case contains the real conversation intention and user response behavior in the real conversation data; then, the conversation test case is input into the simulation robot, so that the simulation robot conducts a simulated conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation. In this solution, by allowing the customer service robot to conduct a variety of simulated conversations, the customer service robot can accumulate more experience and learn more problem-solving methods, so that it can better understand and respond to various problems of real users, provide accurate answers to real users, and improve its own reliability and user satisfaction. In addition, in this solution, a conversation test case is generated based on real conversation data, and a simulation robot is used to conduct a simulated conversation with the customer service robot based on the conversation test case, which not only allows the simulated conversation to cover the real conversation scenario, but also improves the efficiency of conversation simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a conversation simulation method provided in an embodiment of the present application;

[0017] Figure 2 A flowchart of another session simulation method provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a solution decision tree provided in an embodiment of the present application;

[0019] Figure 4 A flowchart of another session simulation method provided in an embodiment of the present application;

[0020] Figure 5 A flowchart of a model training method provided in an embodiment of the present application;

[0021] Figure 6 A flowchart of another session simulation method provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of a conversation simulation system provided in an embodiment of the present application;

[0023] Figure 8 A structural diagram of a conversation simulation device provided in an embodiment of the present application;

[0024] Fig. 9 For Figure 8 A schematic diagram of the structure of an electronic device corresponding to the conversation simulation device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] It should be noted that the technical solution of the embodiment of the present application is applicable to a network virtual environment, and the user described generally refers to a "virtual user". A real user can register a user account in the server through a registration method to obtain a user identity in the network environment. Optionally, the user can be any role that can interact with the customer service, such as a consumer, a merchant, etc.

[0029] With the development of artificial intelligence technology, customer service robots are widely used in various industries such as e-commerce, financial services, telecommunications services, medical health and public services. The application of customer service robots can alleviate the problem of tight human customer service resources and improve customer service efficiency and response speed.

[0030] However, in actual applications, the reliability of customer service robots is limited due to the randomness and uncertainty of the machine learning models (such as large language models) they are configured with when generating content, which may lead to inaccurate or misleading answers when handling user questions, which is commonly known as irrelevant answers and inappropriate replies. This unstable performance of customer service robots directly affects the user experience, reduces user satisfaction with customer service, and may even pose a risk of triggering negative public opinion.

[0031] To solve at least one of the above technical problems, an embodiment of the present application provides a conversation simulation method. In this conversation simulation method, by allowing the customer service robot to conduct a variety of simulated conversations, the customer service robot can accumulate more experience and learn more problem-solving methods, so that it can better understand and respond to various problems of real users, provide accurate answers to real users, and improve its own reliability and user satisfaction.

[0032] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0033] The session simulation method provided in the embodiment of the present application can be executed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, or a server. The server can be a physical server including an independent host, or a virtual server, or a cloud server or a server cluster.

[0034] The large language model mentioned in the embodiments of the present application includes but is not limited to the bidirectional encoder model of Transformers (Bid i rect iona l Encoder Representation from Transformers, referred to as BERT) and the like.

[0035] Figure 1 A flow chart of a session simulation method provided in an embodiment of the present application, such as Figure 1 As shown, the following steps may be included:

[0036] 101. Obtain a session test case generated based on real session data between a real user and customer service, where the session test case includes real session intent and user response behavior in the real session data.

[0037] 102. Input the conversation test case into the simulation robot, so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation.

[0038] In the embodiment of the present application, in order to improve the efficiency of the simulated conversation, a simulated conversation is conducted between a simulated robot and a customer service robot. The simulated robot is used to simulate the behavior and language expression of a real user, to consult the customer service robot about the intention, and to respond to the answer of the customer service robot. The simulated robot is configured with a large language model, and the simulation of the real user by the simulated robot can be realized by inputting information used to simulate the real user into the large language model in the simulated robot.

[0039] It is understandable that the simulated conversation between the simulation robot and the customer service robot is actually a simulation of the real conversation between the real user and the customer service, that is, the real conversation process between the real user and the customer service is reproduced through a virtual dialogue environment.

[0040] Based on this, in the embodiment of the present application, the information used to simulate the real user is determined from the conversation dimension. Specifically, for a real conversation between a real user and customer service, as an optional way to determine the information used to simulate the real user, a conversation test case corresponding to the real conversation can be generated based on the real conversation data corresponding to the real conversation.

[0041] Among them, the real conversation data contains various information used to understand and analyze the interaction characteristics between real users and customer service, including but not limited to: real conversation records between real users and customer service, real conversation intentions of real users, user response behaviors, etc.

[0042] It should be noted that in order to ensure the information security of real users and avoid the impact of session simulation on real users, the real session data involved in the embodiments of the present application have been desensitized to the privacy information of real users, that is, the real session data does not contain the privacy information of real users.

[0043] The conversation test case is used to guide the simulation robot to simulate the real user and conduct a simulated conversation with the customer service robot to reproduce the real conversation in a virtual dialogue environment. The conversation test case includes information used to simulate the real user, such as the real conversation intention and user response behavior of the real user in the real conversation data.

[0044] The real conversation intention is the purpose or goal of the real conversation between the real user and the customer service. Optionally, the real conversation intention can be described by at least one of the following intention description information related to the conversation: conversation scenario, user demand, etc. Among them, the conversation scenario is the specific situation in which the real user has a conversation with the customer service. For example, in an e-commerce shopping scenario, the conversation scenario may include: online shopping consultation, logistics inquiry, member rights consultation, merchant problem complaint, etc. User demand refers to the needs, questions or requests raised by the real user in the conversation with the customer service, such as: refund of shipping fee, complaint of false shipment by the merchant, logistics information consultation, request for invoice, etc.

[0045] User response behavior refers to the reaction and response of a real user to the information, questions or prompts provided by the customer service during a real conversation with the customer service. The user response behavior can be verbal, such as text reply, voice answer, etc., or non-verbal, such as clicking a button, selecting an option, etc. The specific form of the user response behavior is not limited in this embodiment.

[0046] For ease of understanding, the following first describes the process of obtaining session test cases.

[0047] It is understandable that in real customer service conversation scenarios, different real users will have various real conversations with customer service and generate corresponding real conversation data, which may correspond to the same or different real conversation intentions. In simulated conversation scenarios, in order to improve the reliability of the customer service function provided by the customer service robot from different dimensions, it may be necessary to obtain conversation test cases generated based on specific real conversation data (for example, real conversation data corresponding to a certain real conversation intention), and based on the obtained conversation test cases, enable the simulation robot to conduct a simulated conversation with the customer service robot, thereby improving the reliability of the customer service function provided by the customer service robot from dimensions such as answer accuracy, answer consistency, and complex problem handling.

[0048] Among them, the customer service functions provided by the customer service robot can be the functions of a customer service robot that has been deployed in an actual environment and provides services to users, or it can be the functions of a customer service robot that is still in the development or optimization process and has not yet been officially deployed in the actual environment.

[0049] As an optional method for obtaining conversation test cases, you can first determine the conversation simulation task based on the actual simulation conversation requirements; then, based on the conversation simulation task description information corresponding to the conversation simulation task, obtain the conversation test case that matches the conversation simulation task, that is, the conversation test case generated based on specific real conversation data, for use in the simulation conversation between the simulation robot and the customer service robot.

[0050] In an optional embodiment, acquiring a session test case matching the session simulation task according to the session simulation task description information corresponding to the session simulation task includes:

[0051] First, the conversation simulation task description information is obtained, wherein the conversation simulation task description information includes the target conversation intention. The target conversation intention may be: training or testing the customer service robot's ability to solve a user's request in a certain conversation scenario.

[0052] Afterwards, according to the target session intent, the real session data between the real user and the customer service is obtained, and the real session intent in the real session data matches the target session intent. For the convenience of description, the real session data is hereinafter referred to as the real session data matching the target session intent, so as to distinguish it from other real session data that is not related to the target session intent.

[0053] As mentioned above, in a real customer service conversation scenario, different real users will have different real conversations with customer service every day, and generate corresponding real conversation data. In fact, the real conversation data corresponding to any real conversation will be stored in the real conversation database after the real conversation ends. Each real conversation data in the real conversation database contains the corresponding real conversation intent, where the real conversation intent and the target conversation intent use the same description rules when describing, for example: the conversation scenario, user demands and other intention description information used to describe the conversation intent correspond to the same classification method.

[0054] Based on this, in the simulation session scenario, real session data that matches the target session intent can be screened out from the real session database. Specifically, according to the real session intent contained in each real session data in the real session database, the real session data whose real session intent matches the target session intent is screened out from the real session data contained in the real session database.

[0055] For example, the correlation between the real conversation intent corresponding to each real conversation data in the real conversation database and the target conversation intent can be determined by keyword matching, text similarity calculation, semantic similarity calculation, etc. Then, based on the relationship between the correlation and a set correlation threshold, the real conversation data corresponding to at least one real conversation intent whose correlation is greater than the set correlation threshold is determined to be the real conversation data matching the target conversation intent.

[0056] Optionally, if there are multiple real conversation intentions whose correlation is greater than the set correlation threshold, the multiple real conversation intentions whose correlation is greater than the set correlation threshold may be further screened to narrow the acquisition scope of the real conversation data matching the target conversation intention.

[0057] For example, the number of real sessions corresponding to multiple real session intentions with correlation greater than a set correlation threshold can be obtained first, wherein the number of real sessions corresponding to each real session intention can be counted from different time dimensions, such as one day, one week, or one month. Then, according to the number of real sessions corresponding to these multiple real session intentions in a certain same time dimension, the target real session intention is screened out from the multiple real session intentions, such as determining the real session intention with the largest number of corresponding real sessions among the multiple real session intentions as the target real session intention. It is easy to understand that the more real sessions a real session intention corresponds to, the more real sessions the real session intention corresponds to, indicating that the conversation scenarios and user demands corresponding to the real session intention are generally concerned by real users. By using the real session data corresponding to the real session intention with high frequency of real sessions, better information for simulating the conversation interaction of real users in real customer service conversation scenarios can be obtained. Finally, the real session data corresponding to the target real session intention is used as the real session data matching the target session intention.

[0058] In fact, the real session data that matches the target session intent may include various types of real session data, such as real session data of real users being satisfied / unsatisfied with customer service, real session data of real users having more / fewer session turns with customer service, etc.

[0059] Among them, in the real conversation data of real users who are dissatisfied with the customer service, there may be some information that is not conducive to the simulation user robot simulating the real user and the customer service robot to conduct a simulated conversation, such as: emotional conversation messages output by the real user or the customer service. Such information may interfere with the simulation effect of the simulation robot and is not conducive to improving the reliability of the customer service robot. In addition, if the conversation rounds of the real conversation data are small, it may not be possible to obtain accurate information for simulating real users.

[0060] In order to extract accurate information that does not interfere with the simulation robot's simulation of real users, optionally, according to preset conversation screening conditions, target real conversation data that meets the preset conversation screening conditions can be determined from the real conversation data that matches the target conversation intention. Optionally, the conversation screening conditions include at least one of the following: the real user's satisfaction with the customer service is greater than a first set threshold, and the conversation rounds between the real user and the customer service are greater than a second set threshold.

[0061] In the above process, based on the correlation between the real conversation intention corresponding to each real conversation data in the real conversation database and the target conversation intention corresponding to the conversation simulation task, at least one real conversation intention whose correlation with the target conversation intention is greater than the set correlation threshold is screened out; based on the number of real conversations corresponding to at least one real conversation intention, the target real conversation intention with a large number of corresponding real conversations is screened out; based on the preset conversation screening conditions, the target real conversation data whose satisfaction with the customer service of the real user is greater than the first set threshold and the number of conversation rounds between the real user and the customer service is greater than the second set threshold is screened out from the real conversation data matching the target conversation intention. Through the above multiple screening methods, high-quality real conversation data that matches the target conversation intention, corresponds to the real conversation intention with high-frequency real conversations, and meets the preset conversation screening conditions can be obtained from the real conversation database; based on the obtained real conversation data, conversation test cases can be better generated, so that the simulation robot has a better conversation simulation effect when conducting a simulation conversation with the customer service robot.

[0062] Finally, based on the acquired real session data that matches the target session intent, session test cases are generated.

[0063] It is understandable that if the conversation test case is used to guide the simulation robot to simulate the real user and conduct a simulated conversation with the customer service robot, then at least the real conversation intention and user response behavior corresponding to the real user should be obtained based on the real conversation data that matches the target conversation intention. Among them, the real conversation intention and user response behavior determine the conversation direction and content when the simulation robot simulates the real user and conducts a simulated conversation with the customer service robot.

[0064] As mentioned above, the real conversation data contains various information used to understand and analyze the interaction characteristics between real users and customer service, including but not limited to: real conversation records between real users and customer service, real conversation intentions of real users, user response behaviors, etc. Among them, the real conversation intentions and user response behaviors in the real conversation data can be manually determined based on the real conversation records between real users and customer service after the corresponding real conversation ends, or can be generated by natural language processing of the real conversation records between real users and customer service through a large language model.

[0065] Optionally, after obtaining the real session data matching the target session intent, a session test case can be directly generated based on the real session intent and user response behavior in the real session data. The session test case corresponds one-to-one to the real session corresponding to the real session data, and generating a session test case means determining the information corresponding to a real session for simulating a real user.

[0066] In some application scenarios, the simulation robot and the customer service robot need to have a consultation object or a conversation object for the simulation conversation. For example, in an e-commerce scenario, an order can be used as the consultation object. The following takes the consultation object as an example to illustrate the generation process of conversation test cases when the simulation conversation requires a consultation object.

[0067] In the specific implementation process, the real session intention and user response behavior corresponding to the real user can be obtained from the real session data that matches the target session intention. Then, the privacy information of the real user in the real order corresponding to the real session data is desensitized to obtain the target real order, and a test order is generated according to the order information of the target real order, for example: the type or quantity of the goods in the test order is determined according to the product information in the target real order, or the order generation time of the test order is determined according to the order status in the target real order. Among them, the order status is a description of the order processing progress, which is used to help users understand the processing status of the current order and possible next actions, such as: waiting for payment, paid, confirmed, in transit, delivered, return / exchange, etc. Finally, based on the real session intention, user response behavior and test order, a session test case is generated, that is, the association relationship between the real session intention, user response behavior and test order is established, so that the simulation robot and the customer robot can use the test order as the consultation object when conducting a simulation conversation.

[0068] The above embodiment provides a solution for first acquiring real session data matching the target session intent according to the target session intent in the session simulation task description information, and then generating a session test case matching the session simulation task based on the acquired real session data.

[0069] In another optional embodiment, session test cases corresponding to different real session data may be generated in advance. Then, when a session simulation task needs to be executed, that is, when a simulation robot needs to conduct a simulated conversation with a customer service robot, at least one session test case matching the session simulation task is obtained from the pre-generated session test cases according to the target session intent in the session simulation task description information, wherein the real session intent corresponding to at least one session test case matches the target session intent. In this embodiment, the process of generating session test cases according to real session data is the same as that in the above embodiment, and will not be repeated here.

[0070] It is worth noting that, in addition to the above-mentioned real conversation intentions, user response behaviors and test orders, the conversation test case may also include other custom information related to the simulation conversation. For example, the custom information may include: the end conditions of the simulation conversation, such as: ending the simulation conversation after a certain number of conversation rounds, ending the simulation conversation after preset keywords appear in the simulation conversation message, ending the simulation conversation after the simulation conversation reaches the expected conversation goal, etc. For another example, the custom information may include: the difficulty of the simulation conversation, wherein the difficulty of the simulation conversation may be determined based on the real conversation chat records in the real conversation data used to generate the conversation test case, such as: determining the difficulty of the simulation conversation corresponding to the conversation test case based on the number of conversations corresponding to the real conversation chat records or the number of demands of the real user, etc. The more the number of conversations or the number of demands, the greater the difficulty of the simulation conversation.

[0071] Optionally, in the application scenario of filtering at least one session test case matching the session simulation task from pre-generated session test cases, any information contained in the session test case can be used as a session test case screening condition and configured in the session simulation task description information, including but not limited to the above-mentioned target session intention, user response behavior, simulation session difficulty, etc. By configuring the corresponding session test case screening condition in the session simulation task description information, at least one session test case matching the session simulation task can be quickly and accurately screened out from the pre-generated session test cases, thereby achieving the session simulation goal corresponding to the session simulation task. The above is an explanation of the process of obtaining the session test case.

[0072] After obtaining the session test case, the session test case is input into the simulation robot so that the simulation robot and the customer service robot establish an instant communication connection through a communication protocol such as the WebSocket protocol, and based on the instant communication connection, conduct a simulation session with the customer service robot that matches the real session intention in the session test case, and reproduce the user response behavior contained in the session test case during the simulation session.

[0073] In actual applications, multiple simulation conversations can be conducted simultaneously between the simulation robot and the customer service robot. In order to ensure that each simulation conversation can proceed normally, the simulation robot can regularly obtain the instant communication connection status between the simulation robot and the customer service robot, and send heartbeat messages to the customer service robot to maintain the instant communication connection with the customer service robot before the simulation session ends.

[0074] In summary, in the conversation method provided in the embodiment of the present application, a conversation test case generated based on the real conversation data between a real user and a customer service is first obtained, wherein the conversation test case contains the real conversation intention and user response behavior in the real conversation data, and then the conversation test case is input into the simulation robot, so that the simulation robot conducts a simulated conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation, which enables the customer service robot to conduct a variety of simulated conversations, so that the customer service robot can accumulate more experience and learn more problem-solving methods, so that it can better understand and respond to various problems of real users, provide accurate answers to real users, and improve its own reliability and user satisfaction. In addition, in this solution, a conversation test case is generated based on real conversation data, and a simulation robot is used to conduct a simulated conversation with a customer service robot based on the conversation test case, which not only enables the simulated conversation to cover real conversation scenarios, but also improves the efficiency of conversation simulation.

[0075] Figure 2 An interactive flow chart of another session simulation method provided in an embodiment of the present application, such as Figure 2 As shown, the following steps may be included:

[0076] 201. Obtain a session test case generated based on real session data between a real user and customer service, where the session test case includes real session intentions and user response behaviors in the real session data.

[0077] 202. Input the conversation test case into the simulation robot, so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation.

[0078] 203. The real decision factors in the real conversation data are stored in the customer service robot. The real decision factors are the basis for the customer service to make decisions at the decision nodes of the solution decision tree when determining the real intent solution that matches the real conversation intent.

[0079] 204. During the simulation conversation, a simulation identifier is added to the simulation conversation message sent by the simulation robot to the customer service robot.

[0080] 205. When the customer service robot recognizes the simulation identifier, it obtains the target decision factor that matches the context of the simulation session message from the real decision factor. The target decision factor is used to replace the simulation decision factor determined based on the context of the simulation session message to make a decision judgment on the decision node, so as to generate a target intention solution corresponding to the simulation session.

[0081] The specific implementation process of step 201 and step 202 may refer to the aforementioned embodiment and will not be described in detail here.

[0082] It is easy to understand that the customer service robot's ability to provide users with the right solution is an important manifestation of its reliability. Generally, human customer service or customer service robots can determine the solution corresponding to the user's intention based on the Standard Operating Procedure (SOP). Among them, SOP corresponds to a multi-level solution decision tree.

[0083] For ease of understanding, combined Figure 3 To illustrate with an example, Figure 3 A schematic diagram of a solution decision tree provided in an embodiment of the present application. Figure 3 As shown, the solution decision tree contains multiple decision nodes, each decision node represents a decision judgment condition, each branch represents a decision path corresponding to a decision judgment result, and the final leaf node represents a specific solution (for example: solution 1, solution 2, ..., solution 4).

[0084] It should be noted that the actual solution decision tree may contain more Figure 3 More or fewer decision nodes or solutions are indicated. The solution decision tree is not based on Figure 3 The circumstances indicated are limited.

[0085] In the process of determining the solution, the human customer service or customer service robot obtains the corresponding decision factors and makes decisions at each decision node of the solution decision tree based on the decision factors to determine the final solution.

[0086] Among them, the decision factor refers to the feature or attribute used to make a decision at the decision node. In other words, the decision factor provides the input information required for making a decision. For example, a decision node is used to determine whether the time difference between the time when the user signs for the goods and the current time is more than seven days. The decision factor can be the actual time difference between the time when the user signs for the goods and the current time, such as 5 days.

[0087] In actual applications, decision factors include system decision factors and manual decision factors. Among them, system decision factors are determined by preset algorithms based on user behavior data. Among them, user behavior data includes but is not limited to user history data (for example: user's past consultation records, complaint records, browsing records, purchase records, search records, purchase preferences, etc.), user order records (for example: current order status, logistics information, etc.) and other static or dynamic behavior data. Manual decision factors are determined by observing and inquiring users through customer service (including manual customer service and customer service robots) who are in conversation with users, including but not limited to the user's tone, emotions, specific demands, etc.

[0088] In the embodiment of the present application, for the sake of ease of description, in the process of a real conversation between a real user and customer service, the solution determined by the customer service that matches the real conversation intention of the real user is called the real intention solution; the basis for making decisions at the decision nodes of the solution decision tree in the process of determining the real intention solution, that is, the decision factor, is called the real decision factor, and the real decision factor includes the real system decision factor and the real manual decision factor.

[0089] Correspondingly, in the process of conducting a simulated conversation between the simulation robot and the customer service robot, the solution determined by the customer service robot that matches the real conversation intention contained in the conversation test case is called the target intention solution; in the process of conducting a simulated conversation, the decision factor determined by the customer service robot according to the context of the simulation conversation message corresponding to the simulation conversation and used to make a decision at the decision node of the solution decision tree is called a simulation decision factor, and the simulation decision factor includes a simulation system decision factor and a simulation manual decision factor.

[0090] It is understandable that if the information used to determine the simulation decision factor obtained by the customer service robot during the simulation conversation with the customer service robot is consistent with the information used to determine the real decision factor obtained by the customer service when the real user has a real conversation with the customer service, then the simulation decision factor determined by the customer service robot will be consistent with the real decision factor to a large extent. Therefore, during the simulation conversation, the target intent solution determined by the customer service robot will be relatively consistent with the real intent solution, that is, the simulation conversation can better reproduce the real conversation process between the real user and the customer service.

[0091] However, in simulated conversation scenarios, the information obtained by customer service robots is difficult to be consistent with the information obtained in real conversation scenarios. For example, in some simulated conversation scenarios involving orders, due to the restrictions on order generation conditions, some real orders with large amounts, orders that have exceeded the protection period, or orders involving logistics status information may not be able to generate matching test orders, or the generated test orders may not meet the actual simulation conversation requirements.

[0092] For example, suppose that the real order corresponding to the real session data used to generate the session test case is an order involving a large amount of money, and the order status of the order is: in transit, and the real session intent in the real session data is: query the logistics information of the goods. For orders with large amounts of money, usually after the user completes the payment, further manual confirmation is required before the order is shipped. During this process, the order status of the order may be converted from paid to confirmed, from confirmed to shipped, and from shipped to in transit, etc. If you want to reproduce the real conversation process between the real user and the customer service, the order status of the test order should also be in transit. Then, after generating the test order based on the real order, you need to perform the above series of operations to make the order status of the test order also in transit.

[0093] For another example, suppose the order status of the real order corresponding to the real session data used to generate the session test case is: the goods have been signed for 7 days, and the real session intent in the real session data is: return and refund. It is understandable that if the real session process between the real user and the customer service is to be reproduced, the order status of the test order should also be 7 days after the goods have been signed for, so it is necessary to wait 7 days after generating the test order based on the real order so that the order status of the test order is also 7 days after the goods have been signed for.

[0094] It is understandable that if the order status of the test order of the customer service robot is inconsistent with the order status corresponding to the real user's real conversation with the customer service, the determined simulation decision factor may not be consistent with the real decision factor, and the generated target intent solution is also inconsistent with the real intent solution, so it is impossible to reproduce the real conversation process between the real user and the customer service. It may take a long time to make the order status of the test order consistent with the order status of the real order, which cannot meet the needs of some simulation conversations with high immediacy requirements.

[0095] In an optional embodiment, in order to reproduce the real conversation process between the real user and the customer service and meet various simulation conversation requirements, such as Figure 3As shown, the real decision factors in the real conversation data can be stored in the customer service robot in advance. In the process of the simulation conversation, the simulation identifier is added to the simulation conversation message sent by the simulation robot to the customer service robot. Therefore, when the customer service robot recognizes the simulation identifier, it obtains the target decision factor that matches the context of the simulation conversation message from the real decision factor. For example, if the context of the simulation conversation message is discussing information related to the order status, the target decision factor related to the order status is obtained from the real decision factor. After that, when the customer service robot makes a decision judgment at the decision node of the solution decision tree, the target decision factor is used to replace the simulation decision factor determined by the customer service robot based on the context of the simulation conversation message to generate a target intention solution corresponding to the simulation conversation.

[0096] It can be understood that since the true intent solution is determined based on the true decision factors, then based on the target decision factors obtained from the true decision factors and matching the context of the simulated session message, it is naturally possible to generate a target intent solution that has a high consistency with the true intent solution.

[0097] For ease of understanding, combined Figure 3 For example, suppose the decision node on the right side of the first row in the solution decision tree is used to determine whether the product receipt time exceeds 7 days. If it does not exceed 7 days, enter the left branch, and further, solution 3 can be obtained. If it exceeds 7 days, enter the right branch, and further, solution 4 can be obtained.

[0098] Assume that the order status of the real order is: the goods have been signed for 8 days, and the current order status of the test order is: the goods have been signed for 5 days. Based on this assumption, it can be determined that the real decision factor is 8 days and the simulation decision factor is 5 days. If this solution is not adopted, the real intent solution based on the real decision factor is Solution 4, and the target intent solution based on the simulation decision factor is Solution 3. The two are inconsistent, that is, they cannot reproduce the real conversation process between real users and customer service. If this solution is adopted, the real decision factor can be used to intervene in the simulation decision factor. For example: in the simulation conversation scenario, when making a decision on the decision node on the right side of the first row in the solution decision tree, the real decision factor of 8 days is used instead of the simulation decision factor of 5 days to make a decision, thereby obtaining the target intent solution which is Solution 4, the same as the real intent solution.

[0099] It is worth mentioning that, based on the above introduction to system decision factors and manual decision factors, in the process of determining solutions, system decision factors are essentially objectively determined based on the behavioral data generated by users; while manual decision factors reflect the service capabilities of customer service. In the process of simulation conversations, whether the manual decision factors determined by customer service are accurate directly affects whether the correct solution can be obtained.

[0100] Therefore, in order to ensure that the target intent solution generated by the customer service robot matches its own service capabilities in the process of reproducing the real conversation between the real user and the customer service, when the customer service robot recognizes the simulation identifier, it obtains the target decision factor that matches the context of the simulation session message from the real decision factor, including: obtaining the target real system decision factor that matches the context of the simulation session message from the real system decision factor contained in the real decision factor; then, replacing the simulation system decision factor determined based on the context of the simulation session message with the target real system decision factor, and combining it with the simulation artificial decision factor determined based on the context of the simulation session message, making a decision judgment on the decision node of the solution decision tree to generate the target intent solution corresponding to the simulation session.

[0101] In another optional embodiment, in order to ensure that the simulation session can better reproduce the real conversation process between the real user and the customer service, before the simulation session is conducted, the information associated with the real conversation corresponding to the conversation test case can also be pre-stored in the customer service robot, so that the customer service robot determines the simulation decision factor based on the associated information during the simulation session. Optionally, the information associated with the real conversation can be the historical conversation messages between the real user and the customer service before the real conversation, and / or the historical consultation information of the real user on the real order corresponding to the real conversation, etc.

[0102] In an embodiment of the present application, by storing the real decision factors in the real conversation data to the customer service robot, and adding a simulation identifier to the simulated conversation message sent by the simulation robot to the customer service robot, the customer service robot, when identifying the simulation identifier, obtains the target decision factor that matches the context of the simulated conversation message from the real decision factor, instead of the simulated decision factor determined based on the context of the simulated conversation message to make a decision judgment on the decision node, thereby generating a target intention solution corresponding to the simulated conversation. On the one hand, it can effectively avoid the inability to reproduce the real conversation process between the real user and the customer service because the information obtained by the customer service robot for determining the simulation decision factor is consistent with the information obtained by the customer service for determining the real decision factor when the real user has a real conversation with the customer service. On the other hand, it can also meet the needs of simulation conversations with high immediacy requirements.

[0103] Figure 4 A flowchart of another session simulation method provided in an embodiment of the present application, such as Figure 4 As shown, the following steps may be included:

[0104] 401. Obtain a session test case generated based on real session data between a real user and customer service, where the session test case includes real session intent and user response behavior in the real session data.

[0105] 402. Input the conversation test case into the simulation robot, so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation.

[0106] 403. During the simulation conversation, a first simulation conversation message sent by a customer service robot is received by the simulation robot. If the first conversation simulation message is a simulation conversation message expressed in a non-natural language, the first conversation simulation message is converted into a second simulation conversation message expressed in a natural language according to a preset mapping rule.

[0107] 404. Generate a response message corresponding to the second simulation session message according to the user response behavior, and send the response information to the customer service robot.

[0108] The specific implementation process of step 401 and step 402 may refer to the aforementioned embodiment and will not be described in detail here.

[0109] It is understandable that during the simulation conversation, the types of messages sent by the customer service robot to the simulation robot can be various, for example, they can be expressed in natural language (such as text, etc.) or in non-natural language (such as cards, pictures, etc.). In order to improve the fluency of the simulation conversation between the simulation robot and the customer service robot, the embodiment of the present application provides a conversation simulation method for helping the simulation robot to accurately understand the simulation conversation messages sent by the customer service robot.

[0110] Specifically, during the simulation conversation, first, a first simulation conversation message sent by a customer service robot is received by the simulation robot. Afterwards, if the first conversation simulation message is a simulation conversation message expressed in a non-natural language, the first conversation simulation message is converted into a second simulation conversation message expressed in a natural language according to a preset mapping rule. For example, if the first conversation simulation message is a card with options, the card information expressed in a natural language corresponding to the card can be determined according to the preset mapping rule, and the card information is used as the second simulation conversation message. Among them, the card information may include information such as a card title, option content, and a corresponding operation to be performed (for example, selecting an option from multiple options).

[0111] Optionally, the preset mapping rule may be a user-defined function, etc., and the present embodiment does not limit the expression form of the preset mapping rule.

[0112] Afterwards, the customer service robot correctly understands the first session simulation message sent by the customer service robot based on the second simulation session message. Thus, a response message corresponding to the second simulation session message can be generated according to the user response behavior included in the session test case, and the response information can be sent to the customer service robot.

[0113] As mentioned above, user response behavior is the reaction and response to the information, questions or prompts provided by the customer service robot, which can be verbal or non-verbal.

[0114] Optionally, based on the user response behavior included in the session test case, a response message corresponding to the second simulation session message is generated, including: based on the question and answer response behavior in the user response behavior in the session test case, a text response message corresponding to the second simulation session message is generated; or, based on the operation response behavior in the user response behavior in the session test case, a user interface operation simulation instruction corresponding to the second simulation session message is generated.

[0115] It is worth noting that during the simulation conversation, the simulation robot and the customer service robot do not perform real user interface interactions like real users. In the embodiment of the present application, various user interface interaction behaviors such as clicking buttons, selecting options, filling in information, opening pages, and initiating consultations are simulated through user interface operation simulation instructions. Based on these user interface operation simulation instructions, the customer service robot can identify the operation response behaviors of the real user simulated by the simulation robot and make corresponding reactions or responses.

[0116] In the embodiment of the present application, the first simulation conversation message expressed in non-natural language sent by the customer service robot is converted into the second simulation conversation message expressed in natural language through the preset mapping rule, so that the simulation robot can accurately understand the simulation conversation message sent by the customer service robot and conduct a smooth simulation conversation with the customer service robot. In addition, in the process of generating the response message corresponding to the second simulation conversation message according to the user response behavior contained in the conversation test case, the simulation robot can generate a text response message according to the question-and-answer response behavior in the user response behavior on the one hand, and generate a user interface operation simulation instruction according to the operation response behavior in the user response behavior on the other hand, that is, it can realize the simulation of the conversation interaction of the real user language class in the real conversation process, and can also realize the simulation of the conversation interaction of the real user non-language class (i.e., user interface interaction operation class) in the real conversation process, which can realize a high degree of simulation of the user response behavior and improve the simulation degree of the simulation conversation.

[0117] Figure 5A flowchart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the following steps may be included:

[0118] 501. Obtain conversation messages contained in a real conversation data sample between a real user and customer service.

[0119] 502. According to a preset mapping rule, convert a first conversation message sample expressed in a non-natural language in the conversation message into a second conversation message sample expressed in a natural language.

[0120] 503. Input the second conversation message sample into the large language model configured in the simulation robot to perform fine-tuning training on the large language model.

[0121] As mentioned above, the simulation robot is configured with a large language model. By inputting information used to simulate real users into the large language model in the simulation robot, the simulation robot can simulate the real user and conduct a simulated conversation with the customer service robot. During the simulation conversation process, that is, the use stage of the large language model, in order to achieve a smooth simulation conversation with the customer service robot, the first simulation conversation message expressed in a non-natural language sent by the customer service robot can be converted into a second simulation conversation message expressed in a natural language through the above-mentioned preset mapping rules, so that the large language model in the simulation robot accurately understands the simulation conversation message sent by the customer service robot.

[0122] To further improve the accuracy of the large language model in the simulation robot in understanding the simulation conversation messages expressed in non-natural languages, optionally, the large language model can be fine-tuned based on preset mapping rules during the training phase of the large language model.

[0123] During the specific implementation process, the conversation messages contained in the real conversation data sample between the real user and the customer service can be obtained first; then, according to the preset mapping rule, the first conversation message sample expressed in non-natural language in the conversation message is converted into a second conversation message sample expressed in natural language; finally, the second conversation message sample is input into the large language model configured in the simulation robot to fine-tune the large language model.

[0124] In an embodiment of the present application, during the training phase and the use phase of the large language model corresponding to the simulation robot, non-natural language conversation messages are converted into natural language conversation messages based on the same preset mapping rules, and the meaning of the non-natural language conversation messages is understood based on the converted natural language conversation messages, thereby ensuring the correctness of the simulation robot's understanding of the non-natural language conversation messages.

[0125] Figure 6 A flowchart of another session simulation method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the following steps may be included:

[0126] 601. Obtain a session test case generated based on real session data between a real user and customer service, where the session test case includes real session intentions and user response behaviors in the real session data.

[0127] 602. Input the conversation test case into the simulation robot, so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation.

[0128] 603. In response to the end of the simulation session, the session message corresponding to the simulation session is input into the service evaluation model, so that the service evaluation model outputs an evaluation result corresponding to at least one service evaluation item as the service evaluation result of the customer service robot.

[0129] The specific implementation process of step 601 and step 602 may refer to the aforementioned embodiment, and will not be described in detail in this embodiment.

[0130] In actual applications, there is usually a need to evaluate the service capabilities of customer service robots. For example, the service capabilities of customer service robots that have been deployed in actual environments and provide services to users can be evaluated; or the service capabilities of customer service robots that have not yet been formally deployed in actual environments can be evaluated; or the service capabilities of customer service robots of different versions can be evaluated.

[0131] By evaluating the service capabilities of customer service robots, on the one hand, when the service capabilities of customer service robots are poor (that is, the reliability of customer service robots is low), the customer service robots can be optimized in a timely manner to ensure that the customer service robots deployed in the actual environment and providing services to users are customer service robots with better service capabilities, thereby improving user satisfaction; on the other hand, by evaluating the service capabilities corresponding to different versions of customer service robots, it can be determined whether the corresponding service capabilities of the customer service robots are improved before and after the version upgrade, that is, whether the optimization operations of the customer service robots have improved the reliability of the customer service robots.

[0132] The service capability of the customer service robot can be represented by at least one service evaluation item. For example, the service evaluation item can be: whether the customer service robot's reply is a wrong answer, whether it is irrelevant to the question, whether it is an inappropriate reply, etc. The service evaluation item used to represent the service capability can be customized.

[0133] In the specific implementation process, a service evaluation model that can evaluate the service capability of the customer service robot can be pre-trained. Optionally, the service evaluation model can be various machine learning models that can realize service capability evaluation, including but not limited to large language models. For any simulation session, in response to the end of the simulation session, the session message corresponding to the simulation session can be input into the service evaluation model, so that the service evaluation model outputs an evaluation result corresponding to at least one service evaluation item, and then the service evaluation result of the customer service robot can be determined based on the evaluation result corresponding to at least one service evaluation item, and the service evaluation result is used to reflect the service capability of the customer service robot.

[0134] It is understandable that when the service capability of a customer service robot needs to be evaluated, it is often necessary for a simulation robot and a customer service robot to conduct multiple simulation conversations based on multiple conversation test cases. Afterwards, the service capability of the customer service robot is determined by performing statistical analysis on the service evaluation results corresponding to the multiple simulation conversations.

[0135] In an optional embodiment, if the simulation robot and the customer service robot are required to conduct multiple simulation conversations based on multiple conversation test cases, the conversation simulation task description information determined according to the actual simulation conversation requirements may also include: scheduling information corresponding to the multiple conversation test cases.

[0136] Optionally, the scheduling information may include: the timing for the simulation robot and the customer service robot to conduct multiple simulation conversations based on multiple conversation test cases, such as: periodically conducting multiple simulation conversations based on multiple conversation test cases, conducting multiple simulation conversations based on multiple conversation test cases at a certain point in time, etc.; or, the execution strategy for the simulation robot and the customer service robot to conduct multiple simulation conversations based on multiple conversation test cases, such as: dividing multiple conversation test cases into multiple batches, each batch containing a preset number of conversation test cases, and the simulation robot and the customer service robot conduct a simulation conversation based on the conversation test cases contained in a single batch each time, etc.

[0137] In an embodiment of the present application, by using the scheduling information corresponding to the multiple conversation test cases included in the conversation simulation task description information, on the one hand, it can ensure that the simulation robot and the customer service robot conduct multiple simulation conversations in an orderly manner based on the multiple conversation test cases; on the other hand, by controlling the simulation robot and the customer service robot to conduct multiple simulation conversations based on the multiple conversation test cases periodically or once through the scheduling information, it can be discovered in time whether the customer service robot has reliability problems, ensuring that the service capability of the customer service robot can remain stable in different time periods or at specific time points; by controlling the simulation robot and the customer service robot to conduct a simulation conversation based on the conversation test cases included in a single batch each time through the scheduling information, it can be avoided that the simulation robot handles too many conversation test cases at one time, resulting in poor processing capability.

[0138] Figure 7 A schematic diagram of a conversation simulation system provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the conversation simulation system includes: a simulation configuration module, a simulation robot module and a simulation conversation evaluation module.

[0139] During the specific implementation process, the simulation configuration module is used to configure information related to the simulation session, including but not limited to: generating session test cases based on real session data between real users and customer service, determining session test cases that match the simulation session requirements (i.e., session simulation tasks), determining scheduling information corresponding to multiple session test cases for simulation sessions, and determining at least one service evaluation item for evaluating the service capabilities of the customer service robot.

[0140] The simulation robot module is used to provide a simulation robot for conducting simulation conversations with the customer service robot. In actual applications, the simulation robot module can pre-configure a variety of simulation robots suitable for different scenarios, such as simulation robots suitable for e-commerce scenarios, simulation robots suitable for medical and health scenarios, etc. When conducting a simulation conversation, based on the usage scenario of the customer service robot, a simulation robot that matches it can be selected to simulate a real user in the usage scenario and conduct a simulation conversation with the customer service robot.

[0141] The simulation session evaluation module is used to evaluate the service capability of the customer service robot that conducts the simulation session according to the session message corresponding to the simulation session, and to feedback the service evaluation result, for example: through the pre-configured service evaluation model, based on the session message corresponding to the simulation session, the service capability of the customer service robot is evaluated, and the service evaluation result is output in the form of a service evaluation report, etc. Among them, the service evaluation report may include the evaluation result of at least one service evaluation item corresponding to each simulation session, and the statistical results of the service evaluation results corresponding to multiple sessions, etc.

[0142] In actual applications, the conversation simulation system can provide a service interface to the outside, and the service interface includes but is not limited to a software development kit (SDK), an application programming interface (API), etc. The user can call the service interface through the user device to enable the modules in the conversation simulation system to cooperate with each other to implement the conversation simulation method provided in the above embodiment. The detailed execution process and technical effects are described in the above embodiment, which will not be repeated here.

[0143] The conversation simulation device of one or more embodiments of the present application will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0144] Figure 8 A structural diagram of a conversation simulation device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes: an acquisition module 11 and a simulation module 12.

[0145] The acquisition module 11 is used to acquire a session test case generated based on real session data between a real user and customer service, wherein the session test case includes real session intentions and user response behaviors in the real session data.

[0146] The simulation module 12 is used to input the conversation test case into the simulation robot, so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention, and reproduces the user response behavior during the simulation conversation.

[0147] Optionally, the acquisition module 11 is specifically used to obtain session simulation task description information, wherein the session simulation task description information includes a target session intent; based on the target session intent, obtain real session data between a real user and customer service, wherein the real session intent in the real session data matches the target session intent; and generate the session test case based on the real session data.

[0148] Optionally, the acquisition module 11 is further specifically used to determine target real session data in the real session data that meets a preset session screening condition, where the session screening condition includes at least one of the following: the real user's satisfaction with the customer service is greater than a first set threshold, and the number of conversation rounds between the real user and the customer service is greater than a second set threshold; and generate the session test case according to the target real session data.

[0149] Optionally, the acquisition module 11 is also specifically used to obtain the real session intention and user response behavior corresponding to the real user from the real session data; desensitize the privacy information of the real user in the real order corresponding to the real session data to obtain the target real order; generate a test order according to the order information of the target real order; generate a session test case according to the real session intention, the user response behavior and the test order.

[0150] Optionally, the conversation simulation device also includes: a processing module, used to store real decision factors in the real conversation data to the customer service robot, and the real decision factors are the basis for the customer service to make decisions at the decision nodes of the solution decision tree when determining the real intention solution that matches the real conversation intention.

[0151] Correspondingly, the simulation module 12 is specifically used to add a simulation identifier in the simulation session message sent by the simulation robot to the customer service robot during the simulation conversation, so that when the customer service robot identifies the simulation identifier, it obtains a target decision factor that matches the context of the simulation session message from the real decision factors. The target decision factor is used to replace the simulation decision factor determined based on the context of the simulation session message to make a decision judgment on the decision node, so as to generate a target intention solution corresponding to the simulation conversation.

[0152] Optionally, the simulation module 12 is also specifically used to, during the simulation conversation, receive, through the simulation robot, a first simulation conversation message sent by the customer service robot; if the first conversation simulation message is a simulation conversation message expressed in a non-natural language, convert the first conversation simulation message into a second simulation conversation message expressed in a natural language according to a preset mapping rule; generate a response message corresponding to the second simulation conversation message according to the user response behavior; and send the response information to the customer service robot.

[0153] Optionally, the conversation simulation device also includes: a training module, used to obtain conversation messages contained in the real conversation data sample between the real user and the customer service; according to the mapping rule, converting the first conversation message sample expressed in non-natural language in the conversation message into a second conversation message sample expressed in natural language; inputting the second conversation message sample into the large language model configured in the simulation robot to fine-tune the large language model.

[0154] Optionally, the simulation module 12 is also specifically used to generate a text response message corresponding to the second simulation session message based on the question and answer response behavior in the user response behavior; or to generate a user interface operation simulation instruction corresponding to the second simulation session message based on the operation response behavior in the user response behavior.

[0155] Optionally, the conversation simulation device also includes: an evaluation module, which is used to input the conversation message corresponding to the simulation conversation into a service evaluation model in response to the end of the simulation conversation, so that the service evaluation model outputs an evaluation result corresponding to at least one service evaluation item as the service evaluation result of the customer service robot.

[0156] Figure 8 The device shown can execute the steps introduced in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0157] In one possible design, the above Figure 8 The structure of the conversation simulation device shown can be implemented as an electronic device, such as Fig. 9 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable codes, and when the executable codes are executed by the processor 22, the processor 22 can at least implement the session simulation method provided in the above embodiments.

[0158] In addition, an embodiment of the present application provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the session simulation method provided in the aforementioned embodiment.

[0159] An embodiment of the present application provides a computer program product, including: a computer program, when the computer program is executed by a processor of an electronic device, the processor is caused to execute the session simulation method provided in the aforementioned embodiment.

[0160] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0161] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A conversation simulation method, characterized in that: include: Obtaining a session test case generated based on real session data between a real user and customer service, wherein the session test case includes real session intentions and user response behaviors in the real session data; The conversation test case is input into the simulation robot so that the simulation robot conducts a simulation conversation with the customer service robot based on the real conversation intention and reproduces the user response behavior during the simulation conversation.

2. The method according to claim 1, characterized in that The acquiring of a session test case generated based on real session data between a real user and customer service comprises: Acquire conversation simulation task description information, wherein the conversation simulation task description information includes a target conversation intention; According to the target session intent, real session data between a real user and customer service is acquired, wherein the real session intent in the real session data matches the target session intent; The session test case is generated according to the real session data.

3. The method according to claim 2, characterized in that The generating the session test case according to the real session data includes: Determine target real conversation data that meets a preset conversation screening condition in the real conversation data, where the conversation screening condition includes at least one of the following: satisfaction of the real user with the customer service is greater than a first set threshold, and the conversation rounds between the real user and the customer service are greater than a second set threshold; The session test case is generated according to the target real session data.

4. The method according to claim 2, characterized in that: The generating the session test case according to the real session data includes: Acquire the real session intention and user response behavior corresponding to the real user from the real session data; Desensitizing the privacy information of the real user in the real order corresponding to the real session data to obtain the target real order; Generate a test order according to the order information of the target real order; A session test case is generated according to the real session intention, the user response behavior and the test order, and the test order is used to determine the consulting object corresponding to the simulation session.

5. The method according to claim 1, characterized in that The method further comprises: The real decision factor in the real conversation data is stored in the customer service robot, wherein the real decision factor is the basis for the customer service to make a decision at a decision node of a solution decision tree when determining a real intent solution matching the real conversation intent; During the simulation conversation, a simulation identifier is added to the simulation conversation message sent by the simulation robot to the customer service robot, so that when the customer service robot identifies the simulation identifier, it obtains a target decision factor that matches the context of the simulation conversation message from the real decision factor. The target decision factor is used to replace the simulation decision factor determined based on the context of the simulation conversation message to make a decision judgment on the decision node, so as to generate a target intention solution corresponding to the simulation conversation.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: During the simulation conversation, receiving, by the simulation robot, a first simulation conversation message sent by the customer service robot; If the first conversation simulation message is a simulation conversation message expressed in a non-natural language, converting the first conversation simulation message into a second simulation conversation message expressed in a natural language according to a preset mapping rule; Generate a response message corresponding to the second simulation session message according to the user response behavior; The response information is sent to the customer service robot.

7. The method according to claim 6, characterized in that The method further comprises: Acquire the conversation messages contained in the real conversation data sample between the real user and the customer service; According to the mapping rule, converting the first conversation message sample in the conversation message which is not expressed in a natural language into a second conversation message sample which is expressed in a natural language; The second conversation message sample is input into the large language model configured in the simulation robot to perform fine-tuning training on the large language model.

8. The method according to claim 6, characterized in that The step of generating a response message corresponding to the second simulation session message according to the user response behavior includes: generating a text response message corresponding to the second simulation session message according to the question-answer response behavior in the user response behavior; or A user interface operation simulation instruction corresponding to the second simulation session message is generated according to the operation response behavior in the user response behavior.

9. The method according to claim 1, characterized in that: The method further comprises: In response to the end of the simulation session, the session message corresponding to the simulation session is input into the service evaluation model, so that the service evaluation model outputs an evaluation result corresponding to at least one service evaluation item as the service evaluation result of the customer service robot.

10. A computing device, characterized in that It comprises a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the session simulation method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is executed by a computer to implement the conversation simulation method according to any one of claims 1 to 9.

12. A computer program product, characterized in that A computer program is stored, and when the computer program is executed by a computer, the conversation simulation method according to any one of claims 1 to 9 is implemented.

Images