Agent evaluation method and apparatus, and computer-readable storage medium

Abstract

The present disclosure relates to the technical field of computers, and relates to an agent evaluation method and apparatus, and a computer-readable storage medium. The agent evaluation method comprises: on the basis of current interaction information between an agent and a user side, historical interaction information, and system setting information of the agent, using an evaluation model to predict whether the user side is satisfied with the current interaction information; and on the basis of the prediction result, evaluating the performance of the agent.

Description

Agent evaluation method and device, and computer-readable storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an agent evaluation method, an agent evaluation device, a computer-readable storage medium and a computer program product. BACKGROUND

[0002] An agent is an artificial intelligence technology for simulating human thinking and behavior. The agent can understand the instructions and needs of a user, and can generate meaningful feedback. With the development of computer technology, the agent technology is widely applied to human-computer interaction application scenarios such as intelligent terminals and virtual reality.

[0003] SUMMARY

[0004] According to some embodiments of the present disclosure, an agent evaluation method is provided, comprising: predicting, according to current interaction information between an agent and a user side, historical interaction information and system setting information of the agent, whether the user side is satisfied with the current interaction information by using an evaluation model; and evaluating performance of the agent according to a prediction result.

[0005] In some embodiments, predicting, according to the current interaction information between the agent and the user side, the historical interaction information and the system setting information of the agent, whether the user side is satisfied with the current interaction information by using the evaluation model comprises: predicting, by using the evaluation model, a probability that the user side is satisfied with the current interaction information; and evaluating the performance of the agent according to the prediction result comprises: in response to the probability exceeding a probability threshold, determining that the agent meets a performance evaluation condition.

[0006] In some embodiments, the system setting information comprises role attribute information and / or function information of the agent.

[0007] In some embodiments, the agent evaluation method further comprises: in response to the agent not meeting the performance evaluation condition, determining, according to the system setting information and the current interaction information, a performance defect type of the agent by using a classification model.

[0008] In some embodiments, in response to the agent not meeting the performance evaluation condition, determining, according to the system setting information and the current interaction information, the performance defect type of the agent by using the classification model comprises: determining, according to the historical interaction information, the system setting information and the current interaction information, the performance defect type of the agent by using the classification model.

[0009] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side and a current reply information fed back by the agent, and the performance defect type comprises at least one of the following: the current reply information does not conform to the fact, the current reply information cannot solve a problem raised in the current interaction sentence, and the current interaction sentence comprises a sentence with a repetition number exceeding a quantity threshold.

[0010] In some embodiments, the performance defect type of the agent is a basis for optimizing the agent.

[0011] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side, and the historical interaction information comprises a historical interaction sentence initiated by the user side, and the current interaction sentence and the historical interaction sentence are simulated by a generation model to be generated by a real user.

[0012] According to another embodiment of the present disclosure, an agent evaluation device is provided, comprising: a prediction unit configured to predict, according to current interaction information of an agent with a user side, historical interaction information, and system setting information of the agent, whether the user side is satisfied with the current interaction information by using an evaluation model; and an evaluation unit configured to evaluate a performance of the agent according to a prediction result.

[0013] In some embodiments, the prediction unit predicts, by using the evaluation model, a probability that the user side is satisfied with the current interaction information, and the evaluation unit determines that the agent satisfies a performance evaluation condition in response to the probability exceeding a probability threshold.

[0014] In some embodiments, the system setting information comprises role attribute information and / or function information of the agent.

[0015] In some embodiments, the evaluation unit determines, in response to the agent not satisfying the performance evaluation condition, a performance defect type of the agent by using a classification model according to the system setting information and the current interaction information.

[0016] In some embodiments, the evaluation unit determines, according to the historical interaction information, the system setting information, and the current interaction information, the performance defect type of the agent by using the classification model.

[0017] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side and a current reply information fed back by the agent, and the performance defect type comprises at least one of the following: the current reply information does not conform to the fact, the current reply information cannot solve a problem raised in the current interaction sentence, and the current interaction sentence comprises a sentence with a repetition number exceeding a quantity threshold.

[0018] In some embodiments, the performance defect type of the agent is a basis for optimizing the agent.

[0019] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side, and the historical interaction information comprises a historical interaction sentence initiated by the user side, the current interaction sentence and the historical interaction sentence simulating real user generation by using a generation model.

[0020] According to yet some embodiments of the present disclosure, an agent evaluation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the agent evaluation method in any of the above embodiments based on instructions stored in the memory.

[0021] According to still some embodiments of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to implement the agent evaluation method in any of the above embodiments.

[0022] According to still some embodiments of the present disclosure, a computer program product is also provided, comprising instructions which, when executed by a processor, cause the processor to perform the agent evaluation method in any of the above embodiments.

[0023] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application, illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the present disclosure. In the drawings:

[0025] FIG. 1 shows a flowchart of an agent evaluation method according to some embodiments of the present disclosure;

[0026] FIG. 2 shows a schematic diagram of an agent evaluation method according to some embodiments of the present disclosure;

[0027] FIG. 3 shows a schematic diagram of an agent evaluation method according to some other embodiments of the present disclosure;

[0028] FIG. 4 shows a block diagram of an agent evaluation apparatus according to some embodiments of the present disclosure;

[0029] FIG. 5 shows a block diagram of some other embodiments of an agent evaluation apparatus of the present disclosure;

[0030] FIG. 6 shows a block diagram of yet some other embodiments of an agent evaluation apparatus of the present disclosure. DETAILED DESCRIPTION

[0031] With reference to the drawings, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. The following description of at least one example embodiment is merely illustrative in nature and not intended to be limiting on the present disclosure and its applications or uses. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present disclosure.

[0032] Unless specifically stated, the relative arrangement of the components and steps, numerical expressions, and values shown in the embodiments are not meant to limit the scope of the present disclosure. At the same time, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportion relationship. The techniques, methods and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered as part of the authorized description. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, not as a limitation. Therefore, other examples of the example embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0033] The inventors of the present disclosure found that the above-mentioned related art has the following problems: the performance of the intelligent agent cannot be effectively evaluated, resulting in poor human-computer interaction effect. In view of this, the present disclosure proposes an intelligent agent evaluation technical solution, which can effectively evaluate the performance of the intelligent agent, thereby improving the effect of human-computer interaction.

[0034] In view of the above technical problems, the technical solution of the present disclosure combines the interaction between the intelligent agent and the user side and the setting of the intelligent agent itself, predicts whether the user side is satisfied with the interaction, and realizes the performance evaluation of the intelligent agent, thereby improving the effect of human-computer interaction.

[0035] In addition, the performance of the intelligent agent cannot be effectively evaluated, so that the technical defects existing in the intelligent agent cannot be found and solved, resulting in poor human-computer interaction effect.

[0036] In view of the above technical problems, the technical solution of the present disclosure analyzes the performance defects of the intelligent agent in combination with the interaction between the intelligent agent and the user side and the setting of the intelligent agent itself, to determine the performance defect type, for the intelligent agent that does not meet the performance evaluation condition. In this way, the cause of the performance defect of the intelligent agent can be accurately found, thereby improving the effect of human-computer interaction.

[0037] For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0038] FIG. 1 shows a flowchart of an agent evaluation method according to some embodiments of the present disclosure.

[0039] As shown in FIG. 1, in step 110, according to the current interaction information between the agent and the user side, the historical interaction information, and the system setting information of the agent, an evaluation model is used to predict whether the user side is satisfied with the current interaction information.

[0040] In step 120, according to the prediction result, the performance of the agent is evaluated.

[0041] In this way, by combining the interaction between the agent and the user side and the setting of the agent itself, and by predicting whether the user side is satisfied with the interaction, a reliable basis is provided for the performance evaluation of the agent, thereby improving the effect of human-computer interaction.

[0042] In some embodiments, the system setting information includes role attribute information and / or function information of the agent.

[0043] For example, the role attribute information can include personality characteristics, identity characteristics, and other role characteristics of the agent; and the function information can include function types that the agent can implement, such as translation function, query function, writing function, chat function, etc.

[0044] For example, the system setting information can be system prompt information input by the producer of the agent when generating the agent.

[0045] In this way, it can be analyzed whether the reply information fed back by the agent in the interaction with the user conforms to the setting of the agent itself, a reliable basis is provided for the performance evaluation of the agent, and the effect of human-computer interaction is improved.

[0046] In some embodiments, the evaluation model is used to predict a probability that the user side is satisfied with the current interaction information; and in response to the probability exceeding a probability threshold, it is determined that the agent meets the performance evaluation condition. For example, the evaluation model can be a machine learning model based on artificial intelligence technology.

[0047] For example, the evaluation model can be a binary classification model, which predicts whether the user side is satisfied with the current interaction information or whether the user side is not satisfied with the current interaction information. For example, the current interaction information between the agent and the user side, the historical interaction information, and the system setting information of the agent are input into the binary classification model; the binary classification model predicts a probability of like behavior or a probability of dislike behavior of the user side with respect to the current interaction information, which is used as a basis for performance evaluation of the agent.

[0048] For example, a probability that the user side likes the current interaction information can be selected as a basis for performance evaluation of the agent; in response to the probability that the user side likes the current interaction information exceeding a probability threshold, it is determined that the agent meets the performance evaluation condition; and in response to the probability that the user side likes the current interaction information not exceeding the probability threshold, it is determined that the agent does not meet the performance evaluation condition.

[0049] In this way, by predicting whether the user side is satisfied with the current interaction information in combination with the interaction between the agent and the user side, the setting of the agent itself, and the context relationship between the historical interaction information and the current interaction information, performance evaluation of the agent can be realized, thereby improving the effect of human-computer interaction.

[0050] The performance evaluation of the agent will be described below by way of example in conjunction with the embodiment in FIG. 2.

[0051] FIG. 2 shows a schematic diagram of an agent evaluation method according to some embodiments of the present disclosure.

[0052] As shown in FIG. 2, the current interaction information and the historical interaction information can be QA (Question Answering) dialogues generated by the user and the agent in the interaction process; the current interaction information can be a current QA dialogue to be evaluated, and the historical interaction information can be a historical QA dialogue. The system setting information can be system prompt information input by the producer of the agent when generating the agent.

[0053] For example, the like behavior 221 or the dislike behavior 222 of the user online can be taken as a prediction target 22, the system prompt 211 of the agent, the historical QA dialogue 212, and the current QA dialogue 213 can be taken as input features 21, and a binary classification model can be constructed as an evaluation model 2a; the evaluation model 2a is used to predict the probability that the current reply information of the agent in the current QA dialogue 213 is liked by the user; and the probability of liking is taken as an evaluation index of whether the performance evaluation condition of the agent is required.

[0054] In some embodiments, for an agent that has been online, an interaction sentence initiated by the user online can be used as input of the evaluation model; and for an agent that has not been online, a generated model can be used to generate an interaction sentence initiated by the user side as input of the evaluation model. For example, the current interaction information includes a current interaction sentence initiated by the user side, and the historical interaction information includes a historical interaction sentence initiated by the user side, and the current interaction sentence and the historical interaction sentence are simulated by a generated model to generate real users.

[0055] For example, the generation model can be a machine learning model based on artificial intelligence technology, etc.; the generation model can be trained based on the interaction information samples of the agent, through a pre-training process and a fine-tuning training process, so that the generation model can simulate the interaction between the real user and the agent.

[0056] As shown in FIG. 2, the generation model 2b is trained based on the interaction information samples of the agent, through a pre-training process and a fine-tuning training process; the generation model 2b generates the historical interaction sentences initiated by the user side in the historical QA conversation 212 and the current interaction sentences initiated by the user side in the current QA conversation 213, so that the agent feeds back the corresponding historical reply sentences and current reply sentences; the evaluation model 20 predicts the probability that the current reply information of the agent in the current QA conversation 213 is liked by the user side; and the probability of being liked is taken as an evaluation index of whether the performance evaluation condition of the agent is required.

[0057] In the above embodiment, considering the agent that has not been online and the lack of data support for performance evaluation of the agent, the generation model simulates the interaction between the real user and the agent, which can provide data support for performance evaluation of the agent, so as to realize performance evaluation of the agent that has not been online. In this way, the performance defects of the agent can be found in time before the agent is online, thereby improving the effect of human-computer interaction.

[0058] In the following, some embodiments are exemplarily described to analyze the performance defects of the agent that does not meet the performance evaluation condition, so as to determine the performance defect type.

[0059] In some embodiments, in response to the agent not meeting the performance evaluation condition, the performance defect type of the agent is determined by using a classification model according to system setting information and current interaction information. For example, the system setting information can be system prompt information input by the agent producer when generating the agent, etc.

[0060] For example, the system setting information includes role attribute information and / or function information of the agent; the role attribute information can include character features, identity features and other role features of the agent; the function information can include translation function, query function, writing function, chatting function and other function types that can be implemented by the agent.

[0061] In the above embodiment, for the agent that does not meet the performance evaluation condition, the performance defects of the agent are analyzed in combination with the interaction between the agent and the user side and the setting condition of the agent itself, so as to determine the performance defect type. In this way, the reason for the performance defects of the agent can be accurately found, thereby improving the effect of human-computer interaction.

[0062] In some embodiments, the performance defect type of the agent can also be determined by the classification model according to the historical interaction information, the system setting information, and the current interaction information.

[0063] In this way, the performance defect of the agent can be accurately analyzed in combination with the contextual relationship between the historical interaction information and the current interaction information, so that the cause of the performance defect of the agent can be found, and the effect of human-computer interaction can be improved.

[0064] In some embodiments, the current interaction information includes a current interaction sentence initiated by the user side and current reply information fed back by the agent, and the performance defect type includes at least one of the current reply information not conforming to the fact, the current reply information being unable to solve a problem raised by the current interaction sentence, and the current interaction sentence including a statement with a number of repetitions (such as the number of times a same statement appears in a plurality of historical interaction sentences) exceeding a quantity threshold. For example, the performance defect type of the agent is used as a basis for optimizing the agent.

[0065] In the above embodiments, for the agent that does not meet the performance evaluation condition, the performance defect of the agent is analyzed to determine the performance defect type as a basis for optimizing the agent. In this way, the agent that does not meet the performance evaluation condition can be optimized in a targeted manner, the performance of the agent can be improved, and the effect of human-computer interaction can be improved.

[0066] The determination of the performance defect type will be described below by way of example with reference to the embodiments in FIG. 3.

[0067] FIG. 3 shows a schematic diagram of an agent evaluation method according to some other embodiments of the present disclosure.

[0068] As shown in FIG. 3, the current interaction information and the historical interaction information can be QA conversations generated by the user and the agent in the interaction process; the current interaction information can be a current QA conversation to be evaluated, and the historical interaction information can be historical QA conversations. The system setting information can be system prompt information input by the producer of the agent when generating the agent.

[0069] For example, various performance defect types such as the performance defect type 321 or the performance defect type 322 can be taken as the classification result 32, the system prompt 311 of the agent, the historical QA conversation 312, and the current QA conversation 313 can be taken as the input feature 31, and a multi-classification model can be taken as the classification model 3; the classification model 3 is used to determine the performance defect type of the agent that does not meet the performance evaluation condition.

[0070] For example, a classification system of performance defects can be constructed in advance; the existing user-agent interaction information is labeled according to the classification system to generate training samples for training the classification model 3. The training samples can also be expanded by using a machine learning model to generate more training samples to improve the training effect.

[0071] In the above embodiments, a complete and automated agent performance evaluation technical solution is constructed. The performance defects are classified in combination with the interaction between the agent and the user side and the setting of the agent itself to analyze the causes of the agent performance defects. In this way, the agent can be optimized in a targeted manner, the agent performance is observable and improvable, and the human-computer interaction effect is improved.

[0072] FIG. 4 shows a block diagram of an agent evaluation apparatus according to some embodiments of the present disclosure.

[0073] As shown in FIG. 4, the agent evaluation apparatus 4 includes a prediction unit 41 configured to predict, by using an evaluation model, whether the user side is satisfied with the current interaction information according to the current interaction information, the historical interaction information and the system setting information of the agent; and an evaluation unit 42 configured to evaluate the performance of the agent according to the prediction result.

[0074] In some embodiments, the prediction unit 41 predicts, by using the evaluation model, a probability that the user side is satisfied with the current interaction information; and the evaluation unit 42 determines that the agent satisfies the performance evaluation condition in response to the probability exceeding a probability threshold.

[0075] In some embodiments, the system setting information includes role attribute information and / or function information of the agent.

[0076] In some embodiments, the evaluation unit 42 determines, in response to the agent not satisfying the performance evaluation condition, a performance defect type of the agent by using a classification model according to the system setting information and the current interaction information.

[0077] In some embodiments, the evaluation unit 42 determines, by using the classification model, the performance defect type of the agent according to the historical interaction information, the system setting information and the current interaction information.

[0078] In some embodiments, the current interaction information includes a current interaction sentence initiated by the user side and current reply information fed back by the agent, and the performance defect type includes at least one of the current reply information not conforming to the fact, the current reply information being unable to solve a problem raised in the current interaction sentence, and the current interaction sentence including a sentence with a repetition number exceeding a quantity threshold.

[0079] In some embodiments, the performance defect type of the agent is a basis for optimizing the agent.

[0080] In some embodiments, the current interaction information includes a current interaction sentence initiated by the user side, and the historical interaction information includes a historical interaction sentence initiated by the user side, and the current interaction sentence and the historical interaction sentence are simulated to be generated by a real user by using a generation model.

[0081] In the above embodiments, in combination with the interaction between the agent and the user side and the setting of the agent itself, whether the user side is satisfied with the interaction is predicted to provide a reliable basis for performance evaluation of the agent, thereby improving the effect of human-computer interaction.

[0082] FIG. 5 shows a block diagram of another embodiment of the agent evaluation apparatus of the present disclosure.

[0083] As shown in FIG. 5, the agent evaluation apparatus 5 of this embodiment includes a memory 51 and a processor 52 coupled to the memory 51, and the processor 52 is configured to execute the agent evaluation method in any one of the embodiments of the present disclosure based on instructions stored in the memory 51.

[0084] The memory 51 may, for example, include a system memory, a fixed nonvolatile storage medium, etc. The system memory may, for example, store an operating system, an application program, a Boot Loader, a database, and other programs, etc.

[0085] FIG. 6 shows a block diagram of still another embodiment of the agent evaluation of the present disclosure.

[0086] As shown in FIG. 6, the agent evaluation apparatus 6 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the agent evaluation method in any one of the preceding embodiments based on instructions stored in the memory 610.

[0087] The memory 610 may, for example, include a system memory, a fixed nonvolatile storage medium, etc. The system memory may, for example, store an operating system, an application program, a Boot Loader, and other programs, etc.

[0088] The agent evaluation apparatus 6 may further include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650 and the memory 610 and the processor 620 may be connected through a bus 660, for example. The input / output interface 630 provides a connection interface for display, mouse, keyboard, touch screen, microphone, speaker, and other input / output devices. The network interface 640 provides a connection interface for various networking devices. The storage interface 650 provides a connection interface for external storage devices such as SD card and U disk.

[0089] According to some embodiments of the present disclosure, an agent evaluation method is provided, comprising: predicting, according to current interaction information of an agent and a user side, historical interaction information, and system setting information of the agent, whether the user side is satisfied with the current interaction information by using an evaluation model; and evaluating performance of the agent according to a prediction result.

[0090] In some embodiments, predicting, according to current interaction information of an agent and a user side, historical interaction information, and system setting information of the agent, whether the user side is satisfied with the current interaction information by using an evaluation model comprises: predicting, by using the evaluation model, a probability that the user side is satisfied with the current interaction information; and evaluating performance of the agent according to the prediction result comprises: determining that the agent meets a performance evaluation condition in response to the probability exceeding a probability threshold.

[0091] In some embodiments, the system setting information comprises role attribute information and / or function information of the agent.

[0092] In some embodiments, the agent evaluation method further comprises: in response to the agent not meeting the performance evaluation condition, determining, according to the system setting information and the current interaction information, a performance defect type of the agent by using a classification model.

[0093] In some embodiments, in response to the agent not meeting the performance evaluation condition, determining, according to the system setting information and the current interaction information, a performance defect type of the agent by using a classification model comprises: determining, according to the historical interaction information, the system setting information, and the current interaction information, the performance defect type of the agent by using the classification model.

[0094] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side and current reply information fed back by the agent, and the performance defect type comprises at least one of the current reply information not conforming to facts, the current reply information being unable to solve a problem raised by the current interaction sentence, and the current interaction sentence comprising a sentence with a repetition number exceeding a quantity threshold.

[0095] In some embodiments, the performance defect type of the agent is a basis for optimizing the agent.

[0096] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side, and the historical interaction information comprises a historical interaction sentence initiated by the user side, the current interaction sentence and the historical interaction sentence being simulated to be generated by a real user by using a generation model.

[0097] According to another embodiment of the present disclosure, an agent evaluation apparatus is provided, comprising: a prediction unit configured to predict, according to current interaction information of an agent with a user side, historical interaction information, and system setting information of the agent, whether the user side is satisfied with the current interaction information by using an evaluation model; and an evaluation unit configured to evaluate performance of the agent according to a prediction result.

[0098] In some embodiments, the prediction unit is configured to predict, by using the evaluation model, a probability that the user side is satisfied with the current interaction information; and the evaluation unit is configured to determine that the agent meets a performance evaluation condition in response to the probability exceeding a probability threshold.

[0099] In some embodiments, the system setting information comprises role attribute information and / or function information of the agent.

[0100] In some embodiments, the evaluation unit is configured to determine, in response to the agent not meeting the performance evaluation condition, a performance defect type of the agent by using a classification model according to the system setting information and the current interaction information.

[0101] In some embodiments, the evaluation unit is configured to determine, by using the classification model, the performance defect type of the agent according to the historical interaction information, the system setting information, and the current interaction information.

[0102] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side and a current reply information fed back by the agent, and the performance defect type comprises at least one of the current reply information not conforming to facts, the current reply information being unable to solve a problem raised by the current interaction sentence, and the current interaction sentence comprising a sentence with a repetition number exceeding a quantity threshold.

[0103] In some embodiments, the performance defect type of the agent is a basis for optimizing the agent.

[0104] In some embodiments, the current interaction information comprises a current interaction sentence initiated by the user side, and the historical interaction information comprises a historical interaction sentence initiated by the user side, the current interaction sentence and the historical interaction sentence being simulated to be generated by a real user by using a generation model.

[0105] According to still another embodiment of the present disclosure, an agent evaluation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute an agent evaluation method in any of the above embodiments based on instructions stored in the memory.

[0106] According to yet another embodiment of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to implement an agent evaluation method in any of the above embodiments.

[0107] According to still some embodiments of the present disclosure, a computer program product is also provided, comprising instructions which, when executed by a processor, cause the processor to perform the agent evaluation method according to any one of the above embodiments.

[0108] In the above embodiments, in combination with the interaction between the agent and the user side and the setting of the agent itself, a reliable basis is provided for the performance evaluation of the agent by predicting whether the user side is satisfied with the interaction, thereby improving the effect of human-computer interaction.

[0109] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0110] So far, the agent evaluation method, the agent evaluation device, the computer-readable storage medium, and the computer program product according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0111] The methods and systems of the present disclosure can be implemented in a number of ways. For example, the methods and systems of the present disclosure can be implemented using software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely for illustration, and the steps of the methods of the present disclosure are not limited to the above specifically described order, unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as programs recorded in recording media, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers the recording media storing the programs for executing the methods according to the present disclosure.

[0112] Although some specific embodiments of the present disclosure have been described in detail through examples, those skilled in the art should understand that the above examples are merely for illustration, and are not intended to limit the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for evaluating an agent, comprising: Based on the current interaction information between the agent and the user, the historical interaction information, and the system settings information of the agent, the evaluation model is used to predict whether the user is satisfied with the current interaction information. The performance of the agent is evaluated based on the prediction results.

2. The agent evaluation method according to claim 1, wherein, The step of predicting whether the user is satisfied with the current interaction information based on the current interaction information between the agent and the user, historical interaction information, and the system settings information of the agent, using an evaluation model, includes: Using the evaluation model, predict the probability that the user is satisfied with the current interaction information; The evaluation of the agent's performance based on the prediction results includes: When the probability of satisfaction exceeds a probability threshold, the agent is determined to meet the performance evaluation conditions.

3. The agent evaluation method according to claim 1 or 2, wherein, The system settings information includes the role attribute information and / or function information of the intelligent agent.

4. The agent evaluation method according to any one of claims 1-3, further comprising: In response to the agent failing to meet the performance evaluation conditions, the type of performance defect of the agent is determined using a classification model based on the system settings and current interaction information.

5. The agent evaluation method according to claim 4, wherein, In response to the agent failing to meet performance evaluation conditions, the type of performance defect of the agent is determined using a classification model based on the system settings and current interaction information, including: Based on the historical interaction information, the system setting information, and the current interaction information, the classification model is used to determine the performance defect type of the intelligent agent.

6. The agent evaluation method according to claim 4 or 5, wherein, The current interaction information includes the current interaction statement initiated by the user and the current response information fed back by the agent. The performance defect type includes at least one of the following: the current response information is not factual, the current response information cannot solve the problem raised by the current interaction statement, and the current interaction statement includes a statement that is repeated more than a certain number of times.

7. The agent evaluation method according to any one of claims 4-6, wherein, The performance defect type of the agent serves as the basis for optimizing the agent.

8. The agent evaluation method according to any one of claims 1-7, wherein, The current interaction information includes the current interaction statement initiated by the user, and the historical interaction information includes the historical interaction statements initiated by the user. The current interaction statement and the historical interaction statements are generated using a generative model to simulate real user interactions.

9. An agent evaluation device, comprising: The prediction unit is used to predict, based on the current interaction information between the agent and the user, historical interaction information, and the system setting information of the agent, whether the user is satisfied with the current interaction information using an evaluation model; An evaluation unit is used to evaluate the performance of the agent based on the prediction results.

10. An agent evaluation device, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the agent evaluation method of any one of claims 1-8 based on instructions stored in the memory device.

11. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the agent evaluation method as described in any one of claims 1-8.

12. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the agent evaluation method according to any one of claims 1-8.

Images