Agent testing method and apparatus, electronic device, medium, and program product

Abstract

The present disclosure relates to the technical field of computers, and relates to an agent testing method and apparatus, an electronic device, a medium, and a program product. The agent testing method of the present disclosure comprises: generating a user simulator on the basis of character setting information; on the basis of function information of an agent to be tested, interacting with said agent by means of the user simulator; on the basis of the interaction with said agent, generating target information for the interaction by means of the user simulator; and generating a test result for said agent on the basis of the target information.

Description

Method and device for testing agent, electronic device, medium and program product TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a method and device for testing agent, electronic device, medium and program product. BACKGROUND

[0002] With the development of AI (Artificial Intelligence) technology, AI applications such as agents or robots that can interact smoothly with users have begun to appear in large numbers.

[0003] Creators can create agents (or robots) in some network platforms, but there are some agents in these agents that users do not feel good. In order to enable users to use the agent more effectively, and enable the agent to bring users a more convenient and efficient use experience, the network platform will recommend some agents with better test results to the user. Usually, test personnel interact with each agent, score each agent according to the content of the interaction to obtain the test result.

[0004] SUMMARY

[0005] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0006] According to some embodiments of the present disclosure, a method for testing an agent is provided, comprising: generating a user simulator according to character setting information; interacting with a to-be-tested agent according to function information of the to-be-tested agent through the user simulator; generating target information for the interaction according to the interaction with the to-be-tested agent through the user simulator; and generating a test result of the to-be-tested agent according to the target information.

[0007] According to some embodiments of the present disclosure, a method for testing an agent is provided, comprising: generating a user simulator according to character setting information; interacting with a to-be-tested agent according to function information of the to-be-tested agent through the user simulator; generating target information for the interaction according to the interaction with the to-be-tested agent through the user simulator; and generating a test result of the to-be-tested agent according to the target information.

[0008] According to still some embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, configured to store instructions, which, when executed by the processor, cause the processor to perform the test method of the agent of any one of the embodiments of the present disclosure.

[0009] According to yet some embodiments of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, performs the test method of the agent of any one of the embodiments of the present disclosure.

[0010] According to still some embodiments of the present disclosure, a computer program product is provided, comprising: instructions, which, when executed by a processor, implement the test method of the agent of any one of the embodiments of the present disclosure.

[0011] According to yet some embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, implement the test method of the agent of any one of the embodiments of the present disclosure.

[0012] Other features, aspects, and advantages of the present disclosure will become apparent from the following detailed description of the exemplary embodiments of the present disclosure with reference to the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] The preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. The accompanying drawings, which are included to provide a further understanding based on the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve the purpose of explaining the present disclosure. It should be understood that the drawings are for illustration only and are not solely intended to limit the present disclosure. In the drawings:

[0014] FIG. 1 shows a flowchart of the test method of the agent according to some embodiments of the present disclosure;

[0015] FIG. 2 shows a schematic diagram of an interaction interface corresponding to the agent according to some embodiments of the present disclosure;

[0016] FIG. 3 shows a schematic diagram of the test method of the agent according to some embodiments of the present disclosure;

[0017] FIG. 4 shows a structural schematic diagram of a test device of the agent according to some embodiments of the present disclosure;

[0018] FIG. 5 shows a structural schematic diagram of an electronic device according to some embodiments of the present disclosure;

[0019] FIG. 6 shows a structural schematic diagram of a computer system according to some embodiments of the present disclosure.

[0020] It should be understood that the dimensions of the various parts shown in the drawings are not necessarily to scale. Identical or similar components are identified throughout the various figures with identical or similar reference numerals. Therefore, when a component is identified in one figure, it can not be further discussed in subsequent figures. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The description of the embodiments below is actually only illustrative, and should not be construed as any limitation on the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein.

[0022] It should be understood that the various steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect. Unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these embodiments should be interpreted as merely exemplary, not limiting the scope of the present disclosure.

[0023] The term "comprise" and variations of the term, such as "comprising," "comprises," and "comprised of" as used in the present disclosure are open-ended, meaning that additional elements / features can be included. The term "based on" means "based, at least in part, on."

[0024] Reference throughout this specification to "an embodiment," "some embodiments," or "embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. For example, the term "an embodiment" means "at least one embodiment." The terms "another embodiment" or "some embodiments" means "at least one additional embodiment." The terms "one embodiment" or "an embodiment" or "some embodiments" do not necessarily refer to the same embodiment. Furthermore, the phrase "in one embodiment" or "in some embodiments" or "in embodiments" as used throughout this specification, does not necessarily refer to the same embodiment, although it can.

[0025] It should be noted that the terms "first", "second", and the like in the present disclosure are merely intended to distinguish different devices, modules, or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules, or units. Unless otherwise specified, the terms "first", "second", and the like are not intended to imply a given order or any other manner of given order in time, space, ranking, or any other manner.

[0026] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise explicitly specified in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0028] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments. In addition, in one or more embodiments, specific features, structures, or characteristics can be combined by any suitable manner from the present disclosure which is clear to those skilled in the art.

[0029] The completed intelligent agent (or robot) needs to be tested by human before being recommended to users for use. Human testing is usually simulated by a tester to simulate the actual use of the user, prepare some questions to interact with the intelligent agent, and then the tester scores the intelligent agent according to the reply of the intelligent agent and his own feelings to get the test result. However, the efficiency of manual testing is relatively low.

[0030] In view of the low efficiency of manual testing, the present disclosure proposes an intelligent agent testing method, generates a user simulator that can simulate a real user, interacts with the intelligent agent to be tested through the user simulator, generates target information for the interaction, and then generates a test result of the intelligent agent to be tested according to the target information, thereby ensuring the effectiveness of the test while improving the testing efficiency of the intelligent agent.

[0031] Some embodiments of the intelligent agent testing method of the present disclosure will be described below with reference to FIGS. 1-3. The intelligent agent of the present disclosure can include an AI application such as a robot (Bot).

[0032] FIG. 1 is a flowchart of some embodiments of the intelligent agent testing method of the present disclosure. As shown in FIG. 1, the method of this embodiment includes steps S102-S108.

[0033] In step S102, a user simulator is generated according to the character setting information.

[0034] The character setting information is used to describe the characteristics of the character to be simulated, and the user simulator generated according to the character setting information can simulate the interactive behavior of the real user (character) based on the characteristics. For example, the character setting information includes one or more of the character type setting information, the character attribute setting information, and the interaction mode setting information.

[0035] In some embodiments, the user simulator can be generated by a machine learning model according to the character setting information. For example, the machine learning model is an LLM (Large Language Model) or other generative model, not limited to the examples shown.

[0036] In step S104, the user simulator interacts with the to-be-tested agent according to the function information of the to-be-tested agent.

[0037] The function information of the to-be-tested agent can include a description of the basic function of the to-be-tested agent. For example, the to-be-tested agent is an emotionally intelligent reply agent, and the description of the basic function of the to-be-tested agent can be "generating a piece of emotionally intelligent reply content according to the conversation content provided by the user". The function information of the to-be-tested agent can also include one or more of the type, field, and supported auxiliary function of the to-be-tested agent.

[0038] In order to facilitate the use of users, the name or brief description of the agent displayed in the network platform or application can reflect the basic function of the agent, and the user will interact with the agent according to the basic function of the agent. Therefore, the user simulator interacts with the to-be-tested agent according to the function information of the to-be-tested agent, which can more accurately simulate the interactive behavior of the real user, thereby generating more effective interactive content to make the test of the to-be-tested agent more accurate.

[0039] In some embodiments, the function information of the to-be-tested agent is input into the user simulator, and the user simulator interacts with the to-be-tested agent according to the function information of the to-be-tested agent. The user simulator can interact with different to-be-tested agents, and the process of generating the user simulator can be executed in advance. When using the user simulator, only the function information of different to-be-tested agents needs to be input into the user simulator, thereby improving the test efficiency.

[0040] In step S106, the target information for the interaction with the to-be-tested agent is generated by the user simulator according to the interaction with the to-be-tested agent.

[0041] The real user will generate various emotions (feelings) such as happiness, anger, satisfaction and dissatisfaction during the interaction with the agent, which reflects whether the agent can effectively achieve its function and whether it meets the user's needs. During the testing of an agent, the emotional information of the user using the agent is the most important evaluation element. Therefore, by generating the emotional information (feeling information) of the interaction with the agent to be tested as target information through the user simulator, the real user can be simulated more accurately, and the test result can be more accurate.

[0042] For example, the target information includes emotional information, and the emotional information includes at least one of emotion information and satisfaction information. The emotion information is, for example, anger, anger, calmness, happiness, surprise, etc., and the satisfaction information is, for example, satisfaction, dissatisfaction, etc.

[0043] The target information can be determined according to the testing requirements of the agent to be tested, for example, the target information can also be the number of interaction rounds, the number of different types of feedback operations (such as "like", "dislike", etc.), and the like, which are not limited to the examples shown.

[0044] In step S108, a test result of the agent to be tested is generated according to the target information.

[0045] The user simulator can interact with the agent to be tested for one or more rounds, and target information can be generated after each round of interaction, so the target information includes one or more. For example, the test result can be divided into qualified and unqualified, and the proportion of target information in which the emotional value is a preset value such as anger, anger or dissatisfaction in one or more target information can be determined, and the proportion is compared with a threshold value. In the case where the proportion exceeds the threshold value, the test result of the agent to be tested is determined to be unqualified. For another example, the corresponding evaluation score of each emotional value can be set, and for one or more target information, the evaluation score can be determined according to the emotional value corresponding to each target information, and all evaluation scores are summed to obtain the evaluation score of the agent to be tested as the test result. The embodiment form of the test result of the agent to be tested can be set according to actual requirements, and in addition to referring to emotional information, other information such as the number of interactions and interaction content can also be referred to, which is not limited to the examples shown above.

[0046] In the above embodiments, the user simulator is generated according to the character setting information, the user simulator can simulate a real user and interact with the to-be-tested agent according to the function information of the to-be-tested agent, and target information for the interaction is generated, and then the test result of the to-be-tested agent is determined according to the target information. The scheme based on the above embodiments can realize automatic testing of the agent, without the need for test personnel to participate, thereby improving the testing efficiency of the agent. Especially for the scene where a large number of agents need to be tested, the user simulator can be reused, and the testing of multiple agents can be quickly completed, thereby further improving the testing efficiency. In addition, compared with the scheme of automatically asking questions and scoring the to-be-tested agent according to some preset questions and scoring rules, the user simulator can accurately simulate the interaction behavior of a real user, thereby improving the effectiveness and accuracy of the test.

[0047] Next, how to determine the character setting information and how to generate the user simulator will be described in combination with some embodiments.

[0048] For the to-be-tested agent, the character setting information can be randomly generated, and the generated user simulator is also random for the to-be-tested agent.

[0049] Alternatively, the determination or selection of the character setting information or the user simulator can be associated with the test information of the to-be-tested agent. In this case, a plurality of user simulators can be generated according to a plurality of character setting information, one or more user simulators are selected from the plurality of user simulators according to the test information of the to-be-tested agent and the character setting information of each user simulator, and the selected one or more user simulators are used for subsequent interaction with the to-be-tested agent. The character setting information or the user simulator can also be determined according to the test information of the to-be-tested agent, and then the user simulator is generated. The determined character setting information or user simulator is related to the test information of the to-be-tested agent, which can be more targeted and more effective in testing the to-be-tested agent.

[0050] In the case of generating a plurality of user simulators according to a plurality of character setting information, one or more dimensions of features can be selected from a character feature library for combination to generate a plurality of character setting information, and then a plurality of user simulators are generated.

[0051] The character feature library includes character feature information of multiple dimensions, for example, character type, character attribute, interaction mode and the like. The character type is classified according to the use degree of the intelligent agent, for example, the character type can be divided into new user, ordinary user, experienced user and the like. The character attribute can include one or more sub-dimensions, for example, one or more of region, personality, gender, age range, occupation and psychological state. The interaction mode can include one or more sub-dimensions, for example, one or more of interaction strategy, interaction round number, interaction professional degree, interaction demand feature and tolerance. The interaction strategy can be used to describe the number of questions, the type of questions, the relevance of questions and the like during interaction; the interaction round number can correspond to several levels such as few, medium and many; the interaction professional degree can correspond to several levels such as low, medium and high; the interaction demand feature can be used to describe the interaction demand for different types of functions; and the tolerance is used to describe the tolerance for the reply of the intelligent agent not meeting the demand, for example, which can be divided into several levels such as low, medium and high. The specific features in the character feature library can be configured according to actual needs, which are not limited to the examples shown.

[0052] Different types of users will have different interaction modes, so different character types can correspond to different interaction modes. For example, as shown in Table 1, taking the character type as new user, ordinary user and experienced user as an example, each character type corresponds to different values of interaction demand feature, interaction strategy, tolerance, interaction round number and interaction professional degree.

[0053] Table 1

[0054] In the character feature library, each dimension or sub-dimension corresponds to multiple values, and when determining the character setting information, one or more specific values under the dimension can be selected. The more dimensions selected, the more specific the character features, and the closer the generated user simulator can be to the real user.

[0055] After generating the plurality of user simulators according to the plurality of character setting information, one or more user simulators matching the test information of the agent to be tested are selected from the plurality of user simulators. For example, the test information includes one or more of function information, test target, and target user group. The function information of the agent to be tested includes one or more of basic function, type, field, and supported auxiliary function of the agent to be tested. The type of the agent to be tested includes, for example, tool, role playing, and general purpose. The field includes, for example, entertainment, medical treatment, education, and sports. The supported auxiliary function includes, for example, whether to install a plug-in, whether to configure a knowledge base, whether to configure a workflow, and whether to support cooperation with other agents. The supported auxiliary function can further include at least one of the type or identity of the installed plug-in, the type or identity of the configured knowledge base, the specific content of the configured workflow, and the type or identity of the other agent to be cooperated with.

[0056] The test target is used to represent an expected effect or a target to be achieved by the test. For example, the test target is to test the interaction performance of the agent to be tested with a new user, or the test target is to test the interaction performance of the agent to be tested for a professional problem. For example, the target user group is teenagers or people who often write.

[0057] The test information of the agent to be tested is different, the selected user simulators are different, and the corresponding character setting information is different. Based on the test information of the agent to be tested and the character setting information, the user simulators are selected, so that the test is more effective and accurate.

[0058] The character setting information can also be determined according to the test information of the agent to be tested, and then the user simulators are generated. The user simulators generated in this way can be more targeted. It can also be determined from the existing user simulators whether there is a user simulator matching the test information of the agent to be tested. If there is, the user simulator is directly applied, otherwise, the character setting information is determined according to the test information of the agent to be tested, and then the user simulators are generated.

[0059] In some embodiments, the character type setting information is determined according to at least one of the function information and the test target of the agent to be tested.

[0060] The character type setting information can be determined by referring to part or all of the function information of the agent to be tested. For example, for the agent to be tested with more supported auxiliary functions, the character type setting information can be an experienced user. For example, for the agent to be tested of the general purpose type, the character type setting information can include a new user, an ordinary user, and an experienced user. If there are multiple character type setting information, there can be multiple user simulators generated.

[0061] For example, the test target is to test the interaction performance of the to-be-tested intelligent agent with a new user, and the setting information of the character type can be for the new user. According to at least one of the function information of the to-be-tested intelligent agent and the test target, the setting information of the character type is determined, which can make the user type simulated by the user simulator more matched with the function of the to-be-tested intelligent agent and the test target, and improve the effectiveness of the test.

[0062] In some embodiments, the character setting information includes setting information of character attributes, and the setting information of the character attributes is determined according to characteristics of a target user group corresponding to the to-be-tested intelligent agent, or is randomly selected from a character feature library as the setting information of the character attributes, wherein the character feature library includes character attribute features in multiple dimensions.

[0063] For example, the target user group corresponding to the to-be-tested intelligent agent is teenagers, and the age range in the setting information of the character attributes can be set as the age range corresponding to the teenagers. For another example, the target user group corresponding to the to-be-tested intelligent agent is a group of people who often write, and the occupation in the setting information of the character attributes can be set as a writer, an editor, etc. The characteristics of the target user group can be matched with the characteristics in the character feature library to determine the setting information of the character attributes. In addition, in order to make the test more comprehensive, the character setting information matched with the characteristics of a non-target group can also be set, so that the target user and the non-target user can be simulated to interact with the to-be-tested intelligent agent, which is more in line with the actual application scenario.

[0064] In the case where the character type corresponds to the interaction mode, at least one of the setting information of the interaction strategy, the interaction round number, the interaction professionalism, the interaction demand characteristics and the tolerance can be determined according to the setting information of the character type. The more dimensions included in the character setting information, the closer the user simulator can be to the real user.

[0065] According to the characteristics of the target user group corresponding to the to-be-tested intelligent agent, the setting information of the character attributes is determined, which can make the user simulator simulate the user in the target user group, so that the test result is more targeted and more effective.

[0066] After the character setting information is determined, the character setting information can be input into the machine learning model to obtain the user simulator. In some embodiments, according to the character setting information and the description information of the task corresponding to the user simulator, prompt information is generated; and the prompt information is input into the machine learning model to generate the user simulator.

[0067] For example, the description information of the task is used to instruct the user simulator to interact with the agent to be tested according to the function information of the agent to be tested, and generate target information for the interaction. For example, the description information of the task is "please return the current emotion / last round of conversation satisfaction / new question according to the basic function information of the agent and the answer content". The machine learning model can realize the corresponding function by understanding the prompt information (Prompt), that is, the function of the user simulator.

[0068] In some embodiments, the prompt information further includes at least one of a value range of the target information, an example, and a constraint condition, the example includes at least one of an example of the first interaction information generated by the user simulator and an example of the target information, and the constraint condition is used to constrain the first interaction information generated by the user simulator to conform to the character setting information and / or be related to the second interaction information of the agent to be tested in the last round of interaction.

[0069] For example, the target information includes at least one of emotion information and satisfaction information, the value range of the emotion information includes anger, anger, calmness, happiness, surprise, etc., and the value range of the satisfaction includes satisfaction and dissatisfaction, etc., and is not limited to the examples shown. The constraint condition can make the interaction information generated by the user simulator more consistent with the real user. The constraint condition can also include constraints on the expression method, the number of first interaction information generated each time. For example, the expression method is concise and concise, and one first interaction information is generated each time.

[0070] The prompt information can also include an output format, a basis for generating target information, a role of the user simulator, a value range of an operation, a trigger condition of the operation, etc. For example, the basis for generating the target information is whether the question and the answer in the last round of interaction match, whether the user demand is met. For example, the role of the user simulator is configured as "you are the user of the agent". For example, the operation includes one or more operations that the real user can perform on the agent to be tested. In addition to inputting text, voice, etc., the real user can also perform other operations in the process of interacting with the agent, such as exiting, triggering a feedback function, etc.

[0071] The value range of the operation in the prompt information can be configured, for example, exiting, interrupting the reply of the agent, instructing the agent to regenerate the second interaction information, "liking" or "disliking" the second interaction information generated by the agent, and the like, without being limited to the examples. In the interaction process between the user simulator and the to-be-tested agent, the user simulator can simulate the operation of a real user, and one of the more important operations is a feedback operation. The feedback operation can enable the to-be-tested agent to adjust the interaction strategy and the generated reply information in real time, and can test the adaptability of the to-be-tested agent to the user. Through the configuration of the feedback operation, the user simulator can more accurately simulate a real user, and the testing effect can be improved. The feedback operation can include interrupting the reply of the agent, instructing the agent to regenerate the second interaction information, "liking" or "disliking" the second interaction information generated by the agent, and the like, without being limited to the examples. The trigger condition of the operation is, for example, "if you don't want to chat, return [exit]" and the like.

[0072] The content of the prompt information is not limited to the examples described above. The user simulator can be generated according to the prompt information, and the generation efficiency can be improved.

[0073] After the user simulator is generated, the user simulator is used to interact with the to-be-tested agent. The interaction method of the user simulator and the to-be-tested agent is described below in combination with some examples.

[0074] The first interaction information can be generated by the user simulator according to at least one of the character setting information, the function information of the to-be-tested agent, and the historical interaction content with the to-be-tested agent, and the first interaction information is input into the to-be-tested agent to obtain the second interaction information output by the to-be-tested agent.

[0075] In some examples, the user simulator interacts with the to-be-tested agent for one or more rounds. In each round of interaction, the first interaction information is generated by the user simulator according to the character setting information, the function information of the to-be-tested agent, and the historical interaction content with the to-be-tested agent; the first interaction information is sent to the to-be-tested agent to obtain the second interaction information output by the to-be-tested agent.

[0076] Each round of interaction can include the user simulator sending the first interaction information to the to-be-tested agent and the to-be-tested agent replying with the second interaction information. The first interaction information can be in various media forms such as text and voice, and correspondingly, the second interaction information can be in various media forms such as text and voice, without being limited herein. In some interaction rounds, the user simulator can also generate an operation and send it to the to-be-tested agent. In some cases, the to-be-tested agent can perform a corresponding action according to the operation, and the to-be-tested agent can not be configured with a corresponding action.

[0077] The historical interaction content includes second interaction information output by the to-be-tested intelligent agent in the last round of interaction, and can also include first interaction information generated by the user simulator in the last round of interaction, and can also include first interaction information generated by the user simulator and second interaction information output by the to-be-tested intelligent agent in a preset number of rounds of interaction before the last round of interaction. In addition, the historical interaction content can also include information of a feedback operation. In combination with the context of the interaction, the first interaction information generated by the user simulator can be more accurate and real.

[0078] The user simulator generates the first interaction information in need of conforming to the character setting information, in need of referring to the function information of the to-be-tested intelligent agent, and in possession of the memory capability to refer to the historical interaction content of the to-be-tested intelligent agent. In this way, the first interaction information generated is more in line with the interaction information of a real user, and the test efficiency is improved while the authenticity and effectiveness of the test are improved. However, the first interaction information generated by the user simulator can be a question for the function information of the to-be-tested intelligent agent, and can also be irrelevant to the function of the to-be-tested intelligent agent, for example, can be chatting with the to-be-tested intelligent agent, and can be related to the last round of interaction or a new topic, which is determined by the user simulator itself.

[0079] In some embodiments, in each round of interaction, whether to perform a feedback operation is determined by the user simulator according to the character setting information, the function information of the to-be-tested intelligent agent, and the second interaction information output by the to-be-tested intelligent agent in the current round of interaction, wherein the feedback operation is used to trigger a feedback function corresponding to the second interaction information; in response to determining to perform the feedback operation, a target type of the feedback operation is determined by the user simulator according to the character setting information, the function information of the to-be-tested intelligent agent, and the second interaction information output by the to-be-tested intelligent agent in the current round of interaction; and a feedback operation of the target type is generated by the user simulator and sent to the to-be-tested intelligent agent.

[0080] Other operations in addition to the feedback operation can also be generated by the user simulator, for example, an operation of exiting and then entering, and the like. The feedback operation can be used to trigger a feedback control corresponding to the second interaction information or the to-be-tested intelligent agent. As shown in FIG. 2, an interaction interface of a real user and an intelligent agent, in the reply information generated by the intelligent agent, controls such as “like” 201, “dislike” 202 (indicating dissatisfaction, dislike, and the like), “regenerate” 203, and the like are set, and the user simulator can simulate the operation of the real user, that is, simulate the operation of the real user triggering these controls. Different types of feedback operations can correspond to different feedback functions or feedback controls.

[0081] The user simulator determines and generates a feedback operation in accordance with the character setting information, in reference to the function information of the to-be-tested intelligent agent and the second interaction information output by the to-be-tested intelligent agent in the current round of interaction, and in reference to the first interaction information generated by the user simulator in the current round of interaction, and the content of the interaction in a preset number of rounds before the current round of interaction. For example, the user simulator determines a matching degree of the first interaction information and the second interaction information according to the function information of the to-be-tested intelligent agent, the first interaction information generated by the user simulator in the current round of interaction, and the second interaction information output by the to-be-tested intelligent agent, and generates a feedback operation according to the matching degree and the character setting information. For example, the first interaction information and the second interaction information have a low matching degree, and the tolerance level in the character setting information is set to a low level, and the user simulator may generate a feedback operation of exiting, “kicking”, etc.

[0082] The user simulator can also generate a feedback operation based on the target information. For example, the target information is happy or satisfied, and the user simulator can generate a “like” operation.

[0083] By simulating the feedback operation of a real user through the user simulator, it can be tested whether the adjustment of the to-be-tested intelligent agent based on the feedback is effective and reasonable, and it is more in line with the real test scenario, thereby improving the effectiveness and accuracy of the test.

[0084] In some embodiments, after each round of interaction, the user simulator determines whether to end the interaction according to the character setting information, the function information of the to-be-tested intelligent agent, and the historical interaction content with the to-be-tested intelligent agent.

[0085] When to end the interaction is determined by the user simulator itself, and can be determined according to the character setting information, the function information of the to-be-tested intelligent agent, and the historical interaction content with the to-be-tested intelligent agent. The historical interaction content of the to-be-tested intelligent agent at least includes the second interaction information generated by the to-be-tested intelligent agent in the last round of interaction. For example, the user simulator determines that it does not want to chat anymore based on the above information, and can determine to end the interaction and generate an exit operation.

[0086] By determining whether to end the interaction by the user simulator itself, it is more in line with the real scenario compared to manually specifying the number of rounds or the number of questions, thereby improving the effectiveness and accuracy of the test.

[0087] In some embodiments, after each round of interaction in one or more rounds of interaction, the user simulator generates target information according to the character setting information, the function information of the to-be-tested intelligent agent, and the historical interaction content with the to-be-tested intelligent agent.

[0088] The foregoing embodiment mentions that the target information can include at least one of the emotion information and the satisfaction information. The historical interaction content includes at least first interaction information generated by the user simulator in the last round of interaction and second interaction information output by the to-be-tested intelligent agent. In a real scene, different users may have different emotions and satisfaction for the same reply. Some users are easy to get angry, and some users have high tolerance. Different users may have different emotions and satisfaction for the same question of the to-be-tested intelligent agent, and the last round of conversation is a direct factor affecting the emotions and satisfaction of the user. Therefore, the user simulator generates the target information according to the character setting information, the function information of the to-be-tested intelligent agent, and the historical interaction content with the to-be-tested intelligent agent, so that the emotions of the real user can be simulated more accurately, and the effectiveness and accuracy of the test are improved.

[0089] In some embodiments, a matching degree of the first interaction information generated by the user simulator in the last round of interaction and the second interaction information output by the to-be-tested intelligent agent is determined according to the function information of the to-be-tested intelligent agent and the historical interaction content with the to-be-tested intelligent agent; and the target information is generated according to the matching degree and the character setting information.

[0090] The user simulator needs to refer to the character setting information in addition to the matching degree of the first interaction information and the second interaction information, so that the target information generated can be more consistent with the emotions of the real user, and the to-be-tested intelligent agent can be tested more effectively.

[0091] The target information corresponding to each round of interaction can be obtained through the user simulator, and then the test result of the to-be-tested intelligent agent can be generated according to the target information.

[0092] In some embodiments, the test result of the to-be-tested intelligent agent is generated according to at least one of the number of rounds of interaction between the user simulator and the to-be-tested intelligent agent and the historical interaction content between the user simulator and the to-be-tested intelligent agent, and the target information.

[0093] The user simulator can determine whether to end the interaction, so the more the number of rounds of interaction, the better the performance of the to-be-tested intelligent agent in the interaction process can be reflected. The historical interaction content between the user simulator and the to-be-tested intelligent agent can also include some expressions such as “I am angry” and “too good”. Through machine learning simulation, the historical interaction content can be understood, and the performance of the to-be-tested intelligent agent in the interaction can be further evaluated.

[0094] For example, a first score of the to-be-tested intelligent agent is determined according to the target information, a second score is determined according to the number of rounds of interaction, and a third score is determined according to the historical interaction content. The three scores are weighted and summed to obtain a total score as the test result of the to-be-tested intelligent agent.

[0095] The test result of the to-be-tested intelligent agent can be determined by using the above method, and the accuracy can be improved.

[0096] In some embodiments, the character setting information includes a plurality of different character setting information, and the plurality of different character setting information corresponds to a plurality of user simulators. The test result of the to-be-tested intelligent agent is generated according to target information of the plurality of user simulators.

[0097] By setting a plurality of user simulators, the to-be-tested intelligent agent can be tested more comprehensively, and the effectiveness and accuracy of the test can be improved. For example, the total score of the to-be-tested intelligent agent obtained according to each user simulator can be summed up as the test result of the to-be-tested intelligent agent. The scores of the to-be-tested intelligent agent can also be counted for different user simulators of different character types. The form and determination method of the specific test result can be determined according to actual test requirements, and are not limited to the examples shown.

[0098] In some embodiments, a target intelligent agent is selected according to the test result of each to-be-tested intelligent agent and recommended to a user.

[0099] By testing each to-be-tested intelligent agent through a user simulator to obtain a test result, and then recommending an intelligent agent with good test effect to a user, the use of the user can be more convenient, and the user interaction experience can be improved.

[0100] As shown in FIG. 3, the intelligent agent set can include a plurality of to-be-tested intelligent agents. For each to-be-tested intelligent agent, a user simulator can be used to interact with the to-be-tested intelligent agent. A plurality of user simulators can be generated according to a plurality of person setting information (character setting information). The person setting information can be generated according to a person setting library (character feature library), and the person setting library can include a plurality of dimensions of features. The number and order of interactions (or questions) are determined by the user simulator and are uncertain. The user simulator can generate information in multiple modalities such as text and operations. The user simulator has a planning function, that is, to generate the first interaction information (for example, a question) of the next round of interaction. The user simulator has a memory, that is, to refer to historical interaction content for interaction. The user simulator generates the first interaction information based on the person setting information during the interaction process. The first interaction information (question) is sent to the to-be-tested intelligent agent, and the to-be-tested intelligent agent answers. The user simulator can also generate some operations. The user simulator evaluates the conversation to generate target information, which is used as the basis for recommending or screening intelligent agents.

[0101] The present disclosure also provides an intelligent agent testing device, which is described below in conjunction with FIG. 4.

[0102] FIG. 4 is a structural diagram of some embodiments of the test device of the agent of the present disclosure. As shown in FIG. 4, the test device 40 of the agent of this embodiment includes a first generation module 410, a user simulation module 420, and a second generation module 430.

[0103] The first generation module 410 is configured to generate a user simulator according to the character setting information.

[0104] The user simulation module 420 is configured to interact with the agent to be tested according to the function information of the agent to be tested through the user simulator, and generate target information for the interaction according to the interaction with the agent to be tested through the user simulator.

[0105] The second generation module 430 is configured to generate a test result of the agent to be tested according to the target information.

[0106] In some embodiments, the user simulator interacts with the agent to be tested for one or more rounds, and the user simulation module 420 is configured to, in each round of interaction, generate first interaction information according to the character setting information, the function information of the agent to be tested, and the historical interaction content with the agent to be tested through the user simulator; send the first interaction information to the agent to be tested to obtain second interaction information output by the agent to be tested.

[0107] In some embodiments, the user simulation module 420 is configured to, in each round of interaction, determine whether to perform a feedback operation according to the character setting information, the function information of the agent to be tested, and the second interaction information output by the agent to be tested in the current round of interaction through the user simulator, wherein the feedback operation is used to trigger a feedback function corresponding to the second interaction information; in response to determining to perform the feedback operation, determine a target type of the feedback operation according to the character setting information, the function information of the agent to be tested, and the second interaction information output by the agent to be tested in the current round of interaction through the user simulator; and send a feedback operation of the target type to the agent to be tested through the user simulator.

[0108] In some embodiments, the user simulation module 420 is configured to, after each round of interaction, determine whether to end the interaction according to the character setting information, the function information of the agent to be tested, and the historical interaction content with the agent to be tested through the user simulator.

[0109] In some embodiments, the user simulator interacts with the agent to be tested for one or more rounds, and the user simulation module 420 is configured to, after each round of interaction of the one or more rounds, generate target information according to the character setting information, the function information of the agent to be tested, and the historical interaction content with the agent to be tested through the user simulator.

[0110] In some embodiments, the historical interaction content includes first interaction information generated by the user simulator in the previous round of interaction and second interaction information output by the to-be-tested agent, the user simulation module 420 is configured to determine a matching degree of the first interaction information generated by the user simulator in the previous round of interaction and the second interaction information output by the to-be-tested agent according to the function information of the to-be-tested agent and the historical interaction content of the to-be-tested agent; and generate target information according to the matching degree and the character setting information.

[0111] In some embodiments, the character setting information includes setting information of a character type, the character type is divided based on a usage degree of the to-be-tested agent, and the first generation module 410 is further configured to determine the setting information of the character type according to at least one of the function information of the to-be-tested agent and the test target.

[0112] In some embodiments, the character setting information further includes setting information of a character attribute, and the first generation module 410 is further configured to determine the setting information of the character attribute according to a feature of a target user group corresponding to the to-be-tested agent, or randomly select a character attribute feature from a character feature library as the setting information of the character attribute, wherein the character feature library includes character attribute features in multiple dimensions.

[0113] In some embodiments, the character setting information further includes setting information of at least one of an interaction strategy, an interaction round number, an interaction professionalism, an interaction demand feature, and a tolerance.

[0114] In some embodiments, the first generation module 410 is further configured to generate prompt information according to the character setting information and description information of a task corresponding to the user simulator, and input the prompt information into a machine learning model to generate the user simulator.

[0115] In some embodiments, the prompt information further includes at least one of a value range of the target information, an example, and a constraint condition, wherein the example includes at least one of an example of the first interaction information generated by the user simulator and an example of the target information, and the constraint condition is used to constrain that the first interaction information generated by the user simulator conforms to the character setting information and / or is related to the second interaction information of the to-be-tested agent in the previous round of interaction.

[0116] In some embodiments, the character setting information includes a plurality of different character setting information, the plurality of different character setting information corresponds to a plurality of user simulators, and the second generation module 430 is configured to generate a test result of the to-be-tested agent according to target information of the plurality of user simulators.

[0117] In some embodiments, the second generation module 430 is configured to generate the test result of the to-be-tested agent according to at least one of a number of rounds in which the user simulator interacts with the to-be-tested agent and historical interaction content of the user simulator with the to-be-tested agent, and the target information.

[0118] In some embodiments, the target information includes at least one of emotion information and satisfaction information.

[0119] It should be noted that the above-mentioned various units (modules) are only logical modules according to the specific functions they implement, and are not used to limit the specific implementation manners, for example, they can be implemented in software, hardware or a combination of software and hardware. In actual implementation, the above-mentioned various units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned various units are indicated by dashed lines in the drawings, indicating that these units can not actually exist, and the operations / functions they implement can be implemented by the processing circuit itself.

[0120] In addition, although not shown, the device can also include a memory, which can store various information generated by the device, the various units included in the device in operation, programs and data for operation, data to be sent by the communication unit, etc. The memory can be a volatile memory and / or a non-volatile memory. For example, the memory can include but is not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), flash memory. Of course, the memory can also be located outside the device. Alternatively, although not shown, the device can also include a communication unit, which can be used for communication with other devices. In one example, the communication unit can be implemented in a suitable manner known in the art, for example, including communication components such as antenna array and / or radio frequency link, various types of interfaces, communication units, etc. Here will not be described in detail. In addition, the device can also include other components not shown, such as radio frequency link, baseband processing unit, network interface, processor, controller, etc. Here will not be described in detail.

[0121] Some embodiments of the present disclosure also provide an electronic device. FIG. 5 shows a block diagram of some embodiments of the electronic device of the present disclosure. For example, in some embodiments, the electronic device 50 can be various types of devices, for example, can include but are not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet PCs), PMPs (portable multimedia players), car terminals (for example, car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. For example, the electronic device 50 can include a display panel for displaying data and / or execution results utilized in the schemes according to the present disclosure. For example, the display panel can be various shapes, for example, a rectangular panel, an oval panel, or a polygonal panel, and the like. In addition, the display panel can not only be a flat panel, but also a curved panel, or even a spherical panel.

[0122] As shown in FIG. 5, the electronic device 50 of this embodiment includes a memory 51 and a processor 52 coupled to the memory 51. It should be noted that the components of the electronic device 50 shown in FIG. 5 are only exemplary and are not limiting, and the electronic device 50 can also have other components according to actual application needs. The processor 52 can control other components in the electronic device 50 to perform desired functions.

[0123] In some embodiments, the memory 51 is configured to store one or more computer readable instructions. When the processor 52 executes the computer readable instructions, the computer readable instructions are executed by the processor 52 to implement the method according to any of the above embodiments. For specific implementation of each step of the method and related explanations, please refer to the above embodiments, and repeated parts will not be described here.

[0124] For example, the processor 52 and the memory 51 can directly or indirectly communicate with each other. For example, the processor 52 and the memory 51 can communicate through a network. The network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 52 and the memory 51 can also communicate with each other through a system bus, and the present disclosure does not limit this.

[0125] For example, the processor 52 can be embodied as various appropriate processors, processing devices, and the like, such as a central processing unit (CPU), a graphics processing unit (GPU), a network processing unit (NP), and the like; also can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The central processing unit (CPU) can be an X86 or ARM architecture, etc. For example, the memory 51 can include any combination of various forms of computer readable storage media, such as a volatile memory and / or a non-volatile memory. The memory 51 may, for example, include a system memory, such as storing an operating system, an application program, a boot loader, a database, and other programs, etc. Various application programs and various data, etc. can also be stored in the storage medium.

[0126] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, in the case of being implemented by software and / or firmware, can install programs constituting the software from a storage medium or a network to a computer system having a dedicated hardware structure, such as the computer system (or electronic device) 60 shown in FIG. 6, which, when various programs are installed, can perform various functions, including such as the functions of the foregoing, and the like. FIG. 6 is a block diagram showing an example structure of a computer system that can be employed according to embodiments of the present disclosure.

[0127] In FIG. 6, the central processing unit (CPU) 601 performs various processes according to programs stored in a read only memory (ROM) 602 or programs loaded from a storage section 608 to a random access memory (RAM) 603. In the RAM 603, data required when the CPU 601 performs various processes, etc. is also stored as necessary. The central processing unit is merely exemplary, and can also be other types of processors, such as the various processors of the foregoing. The ROM 602, the RAM 603, and the storage section 608 can be various forms of computer readable storage media, as follows. Note that, although the ROM 602, the RAM 603, and the storage section 608 are shown separately in FIG. 6, one or more of them can be combined or located in the same or different memory or storage module.

[0128] The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0129] The following components are connected to the input / output interface 605: an input portion 606, such as a touch panel, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output portion 607, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage portion 608, including a hard disk, a magnetic tape, and the like; and a communication portion 609, including a network interface card, such as a LAN card, a modem, and the like. The communication portion 609 allows communication processing to be performed via a network, such as the Internet. It is easily understood that, although the respective devices or modules in the computer system 60 are shown in FIG. 6 as communicating through the bus 604, they can also communicate through a network or other means, wherein the network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0130] The drive 610 is also connected to the input / output interface 605 as necessary. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is attached to the drive 610 as necessary, so that a computer program read therefrom is installed in the storage portion 608 as necessary.

[0131] In the case where the above series of processes are implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 611.

[0132] According to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program carried on a computer-readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network by the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the CPU 601, the above-described functions defined in the methods of the embodiments of the present disclosure are executed.

[0133] Note that in the context of the present disclosure, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a computer-readable program code transmitted in baseband or as part of a carrier wave over a transmission medium, in which the computer-readable program code can be embodied. Such a transmitted computer-readable signal medium can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or transport a computer-readable program code for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0134] The above computer-readable medium can be included in the above electronic device; or can exist separately from the electronic device.

[0135] In some embodiments, a computer program is also provided, comprising instructions which, when executed by a processor, cause the processor to perform the method of any of the above embodiments. For example, the instructions can be embodied as computer program code.

[0136] Computer program code for carrying out operations of the present disclosure can be written in any one or more of a variety of programming languages or combinations of languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0137] The flow diagrams and the block diagrams in the drawings are meant only to illustrate possible architectures, functions and operations for a system, method and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0138] The modules, components or units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the module, component or unit does not constitute a limitation on the module, component or unit itself.

[0139] The functions described above in the detailed description of embodiments of the present disclosure can be implemented in one or more hardware logic components, or any combination thereof. For example, non-limiting examples of hardware logic components include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0140] According to some embodiments of the present disclosure, a test method of an agent is provided, including: generating a user simulator according to character setting information; interacting with a to-be-tested agent according to function information of the to-be-tested agent through the user simulator; generating target information for the interaction according to the interaction with the to-be-tested agent through the user simulator; and generating a test result of the to-be-tested agent according to the target information.

[0141] In some embodiments, the user simulator interacts with the to-be-tested agent for one or more rounds, and the interaction with the to-be-tested agent according to the function information of the to-be-tested agent through the user simulator includes: in each round of the one or more rounds, generating first interaction information according to the character setting information, the function information of the to-be-tested agent, and historical interaction content with the to-be-tested agent through the user simulator; and sending the first interaction information to the to-be-tested agent to obtain second interaction information output by the to-be-tested agent.

[0142] In some embodiments, the interaction with the to-be-tested agent according to the function information of the to-be-tested agent through the user simulator further includes: in each round, determining whether to perform a feedback operation according to the character setting information, the function information of the to-be-tested agent, and the second interaction information output by the to-be-tested agent in the round through the user simulator, wherein the feedback operation is used to trigger a feedback function corresponding to the second interaction information; in response to determining to perform the feedback operation, determining a target type of the feedback operation according to the character setting information, the function information of the to-be-tested agent, and the second interaction information output by the to-be-tested agent in the round through the user simulator; and generating the feedback operation of the target type and sending the feedback operation to the to-be-tested agent through the user simulator.

[0143] In some embodiments, the interaction with the to-be-tested agent according to the function information of the to-be-tested agent through the user simulator further includes: after each round of interaction, determining whether to end the interaction according to the character setting information, the function information of the to-be-tested agent, and the historical interaction content with the to-be-tested agent through the user simulator.

[0144] In some embodiments, the user simulator interacts with the to-be-tested agent for one or more rounds, and the generation of the target information for the interaction according to the interaction with the to-be-tested agent through the user simulator includes: after each round of the one or more rounds, generating the target information according to the character setting information, the function information of the to-be-tested agent, and the historical interaction content with the to-be-tested agent through the user simulator.

[0145] In some embodiments, the historical interaction content includes first interaction information generated by the user simulator in the previous round of interaction and second interaction information output by the to-be-tested agent, and the target information is generated by the user simulator according to the character setting information, the function information of the to-be-tested agent, and the historical interaction content of the to-be-tested agent. The generating of the target information includes: determining a matching degree of the first interaction information generated by the user simulator in the previous round of interaction and the second interaction information output by the to-be-tested agent according to the function information of the to-be-tested agent and the historical interaction content of the to-be-tested agent; and generating the target information according to the matching degree and the character setting information.

[0146] In some embodiments, the character setting information includes setting information of a character type, and the character type is divided based on a usage degree of the to-be-tested agent. The test method further includes: determining the setting information of the character type according to at least one of the function information of the to-be-tested agent and the test target.

[0147] In some embodiments, the character setting information further includes setting information of a character attribute. The test method further includes: determining the setting information of the character attribute according to a feature of a target user group corresponding to the to-be-tested agent; or randomly selecting a character attribute feature from a character feature library as the setting information of the character attribute, wherein the character feature library includes character attribute features in multiple dimensions.

[0148] In some embodiments, the character setting information further includes setting information of at least one of an interaction strategy, an interaction round number, an interaction professional degree, an interaction demand feature, and a tolerance degree.

[0149] In some embodiments, the generating of the user simulator according to the character setting information includes: generating prompt information according to the character setting information and description information of a task corresponding to the user simulator; and inputting the prompt information into a machine learning model to generate the user simulator.

[0150] In some embodiments, the prompt information further includes at least one of a value range of the target information, an example, and a constraint condition, wherein the example includes at least one of an example of the first interaction information generated by the user simulator and an example of the target information, and the constraint condition is used to constrain that the first interaction information generated by the user simulator conforms to the character setting information and / or is related to the second interaction information of the to-be-tested agent in the previous round of interaction.

[0151] In some embodiments, the character setting information includes a plurality of different character setting information, the plurality of different character setting information corresponds to a plurality of user simulators, and the generating of the test result of the to-be-tested agent according to the target information includes: generating the test result of the to-be-tested agent according to the target information of the plurality of user simulators.

[0152] In some embodiments, the generating the test result of the to-be-tested agent according to the target information comprises: generating the test result of the to-be-tested agent according to at least one of the number of rounds of interaction between the user simulator and the to-be-tested agent and historical interaction content between the user simulator and the to-be-tested agent, and the target information.

[0153] In some embodiments, the target information comprises at least one of emotion information and satisfaction information.

[0154] According to another embodiment of the present disclosure, a test device of an agent is provided, comprising: a first generating module configured to generate a user simulator according to character setting information; a user simulation module configured to interact with a to-be-tested agent according to function information of the to-be-tested agent through the user simulator, and generate target information for the interaction according to the interaction with the to-be-tested agent through the user simulator; and a second generating module configured to generate a test result of the to-be-tested agent according to the target information.

[0155] According to still another embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, configured to store instructions, the instructions being executed by the processor to cause the processor to perform the test method of the agent according to any one of the embodiments of the present disclosure.

[0156] According to yet another embodiment of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to perform the test method of the agent according to any one of the embodiments of the present disclosure.

[0157] According to still another embodiment of the present disclosure, a computer program product is provided, comprising: instructions, the instructions being executed by a processor to implement the test method of the agent according to any one of the embodiments of the present disclosure.

[0158] According to yet another embodiment of the present disclosure, a computer program is provided, comprising: instructions, the instructions being executed by a processor to implement the test method of the agent according to any one of the embodiments of the present disclosure.

[0159] The above description is merely some embodiments of the present disclosure and a description of the principles of the technology applied. Those skilled in the art should understand that the disclosure range involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above disclosure concept. For example, the above features are replaced with each other to form a technical solution with similar functions disclosed in the present disclosure (but not limited to).

[0160] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0161] In addition, while operations are depicted in a particular, chronological sequence, this should not be understood as requiring such order unless otherwise specifically indicated. In some cases, the order of steps can be changed. In some cases, multiple steps can be performed concurrently. In some cases, various elements have been shown as being in contact or connected with one another. In actual practice, however, the elements can be spaced apart from one another in some cases. In addition, although a particular number of steps and / or elements have been shown, in some cases, more or less steps and / or elements can be utilized. In some cases, not all steps and / or elements are utilized. In some cases, additional steps and / or elements can be utilized.

[0162] While certain features of the disclosure have been illustrated and described, many modifications, substitutions, changes, and elaborations have been devised by those skilled in the art. Therefore, it is expressly intended that the claims not be limited by the foregoing description. Rather, it is the claims including any amendments thereto that are intended to define the scope of the disclosure.

Claims

1. A method for testing an agent, comprising: generating a user simulator according to character setting information; interacting with a to-be-tested agent according to function information of the to-be-tested agent through the user simulator; generating target information for the interaction according to the interaction with the to-be-tested agent through the user simulator; generating a test result of the to-be-tested agent according to the target information.

2. The test method of claim 1, wherein, The user simulator interacts with the to-be-tested agent for one or more rounds, and the interacting with the to-be-tested agent according to function information of the to-be-tested agent through the user simulator comprises: in each round of the one or more rounds of interaction, generating first interaction information according to the character setting information, the function information of the to-be-tested agent and historical interaction content with the to-be-tested agent through the user simulator; sending the first interaction information to the to-be-tested agent to obtain second interaction information output by the to-be-tested agent.

3. The test method of claim 2, wherein, The interacting with the to-be-tested agent according to function information of the to-be-tested agent through the user simulator further comprises: in each round of interaction, determining whether to perform a feedback operation according to the character setting information, the function information of the to-be-tested agent and the second interaction information output by the to-be-tested agent in the current round of interaction through the user simulator, wherein the feedback operation is used to trigger a feedback function corresponding to the second interaction information; in response to determining to perform the feedback operation, determining a target type of the feedback operation according to the character setting information, the function information of the to-be-tested agent and the second interaction information output by the to-be-tested agent in the current round of interaction through the user simulator; generating the feedback operation of the target type through the user simulator and sending it to the to-be-tested agent.

4. The test method of claim 2 or 3, wherein, The interacting with the to-be-tested agent according to function information of the to-be-tested agent through the user simulator further comprises: after each round of interaction, determining whether to end the interaction according to the character setting information, the function information of the to-be-tested agent and the historical interaction content with the to-be-tested agent through the user simulator. The user simulator interacts with the to-be-tested agent for one or more rounds, and the generating target information for the interaction according to the interaction with the to-be-tested agent through the user simulator comprises:

5. The test method according to any one of claims 1 to 4, wherein, after each round of interaction of the one or more rounds of interaction, generating the target information according to the character setting information, the function information of the to-be-tested agent and the historical interaction content with the to-be-tested agent through the user simulator. The historical interaction content comprises first interaction information generated by the user simulator and second interaction information output by the to-be-tested agent in the previous round of interaction, and the generating the target information according to the character setting information, the function information of the to-be-tested agent and the historical interaction content with the to-be-tested agent through the user simulator comprises:

6. The test method of claim 5, wherein, ​ determine a matching degree of first interaction information generated by the user simulator and second interaction information output by the intelligent agent in the last round of interaction according to the function information of the intelligent agent to be tested and historical interaction content with the intelligent agent to be tested; generate the target information according to the matching degree and the character setting information.

7. The test method of any one of claims 1-6, wherein, The character setting information includes character type setting information, and the character type is divided based on a use degree of the intelligent agent to be tested. The test method further includes: determine character type setting information according to at least one of the function information of the intelligent agent to be tested and a test target.

8. The test method of claim 7, wherein, The character setting information further includes character attribute setting information. The test method further includes: determine the character attribute setting information according to a feature of a target user group corresponding to the intelligent agent to be tested; or randomly select character attribute features from a character feature library as the character attribute setting information, wherein the character feature library includes character attribute features in multiple dimensions.

9. The test method of claim 7 or 8, wherein, The character setting information further includes setting information of at least one of an interaction strategy, an interaction round number, an interaction professional degree, an interaction demand feature, and a tolerance.

10. The test method of any one of claims 1-9, wherein, The generation of the user simulator according to the character setting information includes: generate prompt information according to the character setting information and description information of a task corresponding to the user simulator; input the prompt information into a machine learning model to generate the user simulator.

11. The test method of claim 10, wherein, The prompt information further includes at least one of a value range, an example, and a constraint condition of the target information, wherein the example includes at least one of an example of the first interaction information generated by the user simulator and an example of the target information, and the constraint condition is used to constrain that the first interaction information generated by the user simulator conforms to the character setting information and / or is related to the second interaction information of the intelligent agent to be tested in the last round of interaction.

12. The test method of any one of claims 1-11, wherein, The character setting information includes multiple different character setting information, and the multiple different character setting information corresponds to multiple user simulators. The generation of a test result of the intelligent agent to be tested according to the target information includes: generate a test result of the intelligent agent to be tested according to target information of the multiple user simulators.

13. The test method of any one of claims 1-12, wherein, The generation of a test result of the intelligent agent to be tested according to the target information includes: generate a test result of the intelligent agent to be tested according to at least one of a round number of interaction between the user simulator and the intelligent agent to be tested and historical interaction content between the user simulator and the intelligent agent to be tested, and the target information.

14. The test method of any one of claims 1-13, wherein, The target information includes at least one of emotion information and satisfaction information.

15. An intelligent agent testing device, comprising: a first generation module configured to generate a user simulator according to character setting information; a user simulation module configured to interact with an intelligent agent to be tested through the user simulator according to function information of the intelligent agent to be tested, and generate target information for the interaction through the user simulator according to the interaction with the intelligent agent to be tested. A second generating module, configured to generate a test result of the to-be-tested agent according to the target information. 16.An electronic device, comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform the method of testing an agent according to any one of claims 1-14.

17. A computer readable storage medium having stored thereon a computer program, wherein, A program which, when executed by a processor, implements the method of testing an agent according to any one of claims 1-14.

18. A computer program product, comprising: instructions which, when executed by a processor, implement the method of testing an agent according to any one of claims 1-14.

19. A computer program comprising: instructions which, when executed by a processor, implement the method of testing an agent according to any one of claims 1-14.

Images