Intelligent agent evaluation method and device, equipment, medium and program product

Abstract

The invention provides an agent evaluation method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a test data set; the test data set comprises a plurality of label agent templates and a plurality of to-be-evaluated agent templates, and determining a first output result of each to-be-evaluated agent template for a plurality of input tasks and a second output result of each label agent template for the input tasks according to capability indexes of different dimensions of each to-be-evaluated agent template, and determining a capability score, and determining a comprehensive performance score of the to-be-evaluated proxy template according to the capability score of the capability index of each dimension and the evaluation weight. By comparing the label agent template with the agent template to be evaluated, the agent template automatically generated by the large language model is evaluated from multiple dimensions, the actual performance of the agent is truly reflected, the accuracy and comprehensiveness of evaluation are improved, human intervention is reduced, and the objectivity of evaluation is improved.

Description

Background Art

[0002] With the development of artificial intelligence, intelligent agent generation has become an important topic. In the field of intelligent agent generation, due to the extreme diversity and complexity of application scenarios, traditional evaluation methods cannot fully capture the performance of the generated intelligent agents and cannot accurately quantify the actual performance of the intelligent agents.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides an intelligent agent evaluation method, apparatus, device, medium and program product, which at least to some extent overcome the problem that the related art cannot accurately evaluate the generated intelligent agent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] In a first aspect, an embodiment of the present disclosure provides an agent evaluation method, the method comprising:

[0007] Acquire a test data set; the test data set includes: a plurality of label proxy templates, and a plurality of proxy templates to be evaluated of agents automatically output by a large language model;

[0008] For each of the to-be-evaluated proxy templates, respectively determining a first output result of each of the to-be-evaluated proxy templates for a plurality of input tasks and a second output result of each of the label proxy templates for an input task, with respect to capability indicators of different dimensions of each to-be-evaluated proxy template;

[0009] Determine, according to the first output result and the second output result, a capability score of the agent template to be evaluated under the capability indicator of each dimension;

[0010] The comprehensive performance score of the agent template to be evaluated is determined based on the capability score and evaluation weight of the capability indicator of each dimension.

[0011] In a possible embodiment, the capability indicator includes: planning capability; the output result includes: the tools used and the planning thinking process;

[0012] Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes:

[0013] For each of the input tasks, determining a first matching degree between a first tool corresponding to any to-be-evaluated proxy template and a second tool corresponding to any label proxy template;

[0014] determining a second degree of matching between the corresponding first planning thinking process and the second planning thinking process;

[0015] Determining a comprehensive score for each of the input tasks according to the first matching degree and the second matching degree under each of the input tasks;

[0016] According to the comprehensive score of each of the input tasks, the planning capability score of any one of the agent templates to be evaluated is determined.

[0017] In a possible embodiment, determining the planning capability score of any one of the to-be-evaluated agent templates according to the comprehensive score of each of the input tasks includes:

[0018] Determining a maximum comprehensive score from the comprehensive scores of each of the input tasks;

[0019] For any of the proxy templates to be evaluated, determining the maximum comprehensive score of each of the label proxy templates;

[0020] The planning capability score of any one of the proxy templates to be evaluated is determined according to the maximum comprehensive score of each of the label proxy templates and the number of the label proxy templates.

[0021] In a possible embodiment, the capability indicator includes: tool use capability; the output result includes: the used tool, the parameter corresponding to the tool, and the parameter value corresponding to the parameter;

[0022] Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes:

[0023] For each of the input tasks, determining a third matching degree between a third tool corresponding to any to-be-evaluated proxy template and a fourth tool corresponding to any label proxy template;

[0024] determining a fourth degree of match between the corresponding first parameter and the second parameter;

[0025] determining a fifth degree of match between the corresponding first parameter value and the second parameter value;

[0026] Determining a comprehensive score for each of the input tasks according to the third matching degree, the fourth matching degree, and the fifth matching degree under each of the input tasks;

[0027] According to the comprehensive score of each of the input tasks, a tool usage capability score of any one of the agent templates to be evaluated is determined.

[0028] In a possible embodiment, the capability indicators include: planning capability, tool use capability and diversity generation capability;

[0029] Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes:

[0030] Obtain planning ability scores and tool use ability scores;

[0031] Counting the cumulative number of times the first output result and the second output result are the same;

[0032] A diversity generation ability score is determined according to the ratio of the accumulated number of times to the total number of input tasks, the planning ability score, and the tool use ability score.

[0033] In a possible embodiment, the capability indicators include: role-playing capability;

[0034] Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes:

[0035] According to the predefined multiple roles and the input task, respectively obtain a first output result corresponding to any proxy template to be evaluated and a second output result corresponding to any label proxy template;

[0036] Determining a sixth matching degree between the first output result corresponding to each of the roles and the second output result;

[0037] From the sixth matching degrees corresponding to each of the roles, a maximum matching degree is determined as the role playing ability score.

[0038] In a second aspect, an embodiment of the present disclosure provides an intelligent agent evaluation device, comprising:

[0039] An acquisition unit is used to acquire a test data set; the test data set includes: a plurality of label proxy templates, and a plurality of proxy templates to be evaluated of intelligent agents automatically output by a large language model;

[0040] A first determining unit is used to determine, for each proxy template to be evaluated, a first output result of each proxy template to be evaluated for a plurality of input tasks and a second output result of each label proxy template for an input task, respectively, with respect to capability indicators of different dimensions of each proxy template to be evaluated;

[0041] A second determining unit, configured to determine a capability score of the agent template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result;

[0042] The evaluation unit is used to determine the comprehensive performance score of the agent template to be evaluated according to the ability score and evaluation weight of the ability indicator of each dimension.

[0043] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method described in the first aspect above by executing the executable instructions.

[0044] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0045] In a fifth aspect, according to another aspect of the present disclosure, a computer program product or a computer program is also provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes any of the above-mentioned methods.

[0046] The embodiment of the present disclosure provides an agent evaluation method, device, equipment, medium and program product, the method comprising: obtaining a test data set; the test data set includes: multiple label proxy templates, and multiple agent proxy templates to be evaluated automatically output by a large language model, respectively determining the first output results of each agent template to be evaluated for multiple input tasks and the second output results of each label proxy template for the input task for the ability indicators of different dimensions of each agent template to be evaluated, determining the ability score of the agent template to be evaluated under the ability indicator of each dimension according to the first output result and the second output result, and determining the comprehensive performance score of the agent template to be evaluated according to the ability score and evaluation weight of the ability indicator of each dimension. The label proxy template of the mature label agent is compared with the agent template to be evaluated of the agent to be evaluated, and then the agent template automatically generated by the large language model is evaluated from multiple dimensions, so as to truly reflect the actual performance of the agent in complex tasks, optimize its adaptability and efficiency in diversified application scenarios, improve the accuracy and comprehensiveness of the evaluation, reduce human intervention, and improve the objectivity of the evaluation.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0049] Figure 1 A flow chart showing an agent evaluation method according to an embodiment of the present disclosure is shown;

[0050] Figure 2 A flow chart for determining a planning capability score in an embodiment of the present disclosure is shown;

[0051] Figure 3 A flowchart for determining a tool usage ability score in an embodiment of the present disclosure is shown;

[0052] Figure 4 A flow chart for determining a diversity generation capability score according to an embodiment of the present disclosure is shown;

[0053] Figure 5 A flow chart for determining a role-playing ability score in an embodiment of the present disclosure is shown;

[0054] Figure 6 A schematic diagram showing the structure of an intelligent agent evaluation device in an embodiment of the present disclosure is shown;

[0055] Figure 7 A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0057] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0058] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0059] First, an intelligent agent evaluation method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities. In the following process, the electronic device is taken as a server as an example.

[0060] Figure 1 A flow chart of an agent evaluation method in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the agent evaluation method provided in the embodiment of the present disclosure includes the following steps:

[0061] S102: Acquire a test data set, where the test data set includes a plurality of label proxy templates and a plurality of proxy templates to be evaluated of intelligent agents automatically output by a large language model.

[0062] The label agent template is the agent template corresponding to the label agent, which is used to compare with the agent template to be evaluated and to evaluate the agent template to be evaluated. The label agent can be a high-quality agent selected by humans, or a mature high-quality agent selected by a computer according to task requirements and application scenarios.

[0063] S104: for the capability indicators of different dimensions of each proxy template to be evaluated, respectively determine a first output result of each proxy template to be evaluated for multiple input tasks and a second output result of each label proxy template for the input task.

[0064] S106: Determine the capability score of the agent template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result.

[0065] In a possible embodiment, the capability indicators of different dimensions may include multiple ones, for example: planning capability, tool use capability, diversity generation capability, role-playing capability, context understanding capability, etc.

[0066] S108: Determine the comprehensive performance score of the agent template to be evaluated based on the capability score and evaluation weight of the capability indicator of each dimension.

[0067] In a possible embodiment, the evaluation weight of the capability indicator of each dimension may be preset manually or determined by a computer.

[0068] The determination method may include: determining the evaluation weight of the capability indicator of each dimension according to the task requirements and task importance of the input task. Differentiated weight allocation methods can make the evaluation results more accurate.

[0069] In a possible embodiment, after obtaining the comprehensive performance score of the proxy template to be evaluated, the performance of the proxy template to be evaluated is determined according to the comprehensive performance score to obtain the evaluation result of the corresponding intelligent agent.

[0070] In a possible embodiment, the capability indicators are: planning capability, tool use capability, diversity generation capability, and role-playing capability.

[0071] The comprehensive performance score is determined by the following formula (1), as shown below:

[0072] S overall =ωS plan +ω′S tool +ω″S adaptability +ω″′S profile (1)

[0073] Among them, S overall represents the comprehensive performance score; ω, ω′, ω″, ω″′ represent the corresponding evaluation weights; S plan represents the planning ability score; S tool Indicates the tool use ability score; S adaptability represents the diversity generation ability score; S profile Indicates the role-playing ability score.

[0074] Through the above method, a comprehensive test data set is first constructed for the performance of multiple agents automatically output by the large language model. The test data set includes proxy templates to be evaluated corresponding to multiple agents automatically generated by the large language model for diverse question queries, as well as the corresponding mature proxy templates as standard references, i.e., label proxy templates.

[0075] By comparing the proxy template to be evaluated with the labeled proxy template, the performance and effectiveness of the proxy template to be evaluated in different task scenarios are comprehensively measured.

[0076] Through the pre-set capability indicators of different dimensions, the final evaluation results can reflect the comprehensive capabilities in many aspects, comprehensively evaluate the agent templates to be evaluated, ensure that the automatically generated multiple intelligent agents can show high adaptability, efficiency and accuracy when performing complex tasks, accurately evaluate the performance of the intelligent agents, reduce human intervention, and improve the objectivity of the evaluation.

[0077] In one possible embodiment, the purpose of the evaluation indicators is to provide a comprehensive evaluation result of the intelligent agent performance. In the embodiment of the present disclosure, the performance view can also be determined based on the comprehensive performance score and the ability score of each different dimension of the ability indicator, and displayed on the front-end interface.

[0078] Through the above method, the proportion of capability indicator scores of different dimensions in the comprehensive performance score can be reflected, and the strength of capability indicators of different functional dimensions can be intuitively displayed. Since the capability indicator score can directly display its ability in this dimension, the capabilities of different agent templates to be evaluated can be directly displayed and judged through the performance view.

[0079] For example, if the planning ability score of agent template A to be evaluated is greater than the set threshold, it can be retained. When it is detected that the task type of the user's input task is a planning task, agent template A to be evaluated can be selected. Agent template A to be evaluated is only used for task planning, and it does not need to call tools. Agent template A to be evaluated can complete the task and has a higher tendency.

[0080] For other templates, the above method can also be adopted, not only to evaluate the ability indicators of the agent template to be evaluated under multiple dimensions, but also to obtain directional intelligent agents whose ability scores of ability indicators in a certain dimension are greater than the set threshold during the evaluation process.

[0081] The specific dimensional indicators are not limited in the embodiments of the present disclosure, and the following embodiments exemplify the methods for determining the ability scores of several specific ability indicators.

[0082] Example 1: Planning capabilities.

[0083] The ability to plan the input task is evaluated. The output results, namely the first output result and the second output result, may include: the tools used and the planning thinking process.

[0084] When the intelligent agent processes an input task, it needs to consider what kind of tools to use, namely the first tool and the second tool, and when it obtains the output result of the input task, it needs to output the planning thinking process for processing the input task, namely the first planning thinking process and the second planning thinking process.

[0085] The first output result includes: a first tool and a first planning thinking process; the second output result includes: a second tool and a second planning thinking process.

[0086] Figure 2 A flow chart of determining a planning capability score in an embodiment of the present disclosure is shown. Figure 2 As shown, the following steps are included:

[0087] S202: For each input task, determine a first matching degree between a first tool corresponding to any to-be-evaluated proxy template and a second tool corresponding to any label proxy template.

[0088] In a possible embodiment, the first matching degree may be determined by editing examples or cosine matching degree.

[0089] S204: Determine a second matching degree between the corresponding first planning thinking process and the second planning thinking process.

[0090] The second matching degree can be determined by cosine matching degree or a scoring mechanism based on a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model.

[0091] S206: Determine a comprehensive score for each input task according to the first matching degree and the second matching degree under each input task.

[0092] In a possible embodiment, the comprehensive score may be determined according to a preset weight of the first matching degree and a preset weight of the second matching degree.

[0093] S208: Determine the planning capability score of any agent template to be evaluated according to the comprehensive score of each input task.

[0094] In a possible embodiment, after determining the comprehensive score of each input task, the average comprehensive score of multiple input tasks is determined, and the average comprehensive score between any proxy template to be evaluated and each label proxy template is determined respectively, and then the planning ability score of any proxy template to be evaluated is determined based on the multiple average comprehensive scores and the number of label proxy templates.

[0095] In another possible embodiment, it can also be determined in the following manner:

[0096] From the comprehensive scores of each input task, determine the maximum comprehensive score; for any proxy template to be evaluated, determine the maximum comprehensive score of each label proxy template respectively; according to the maximum comprehensive score of each label proxy template and the number of label proxy templates, determine the planning ability score of any proxy template to be evaluated.

[0097] In another possible embodiment, the planning capability score may be determined according to the first output result and the second output result by using the following formula (2), as shown below:

[0098]

[0099] Among them, S plan represents the planning ability score, M represents the number of label proxy templates; j represents the index of the label proxy template; N represents the number of input tasks; i represents the index of the input task; ω1 represents the first weight; ω2 represents the second weight; Tool iIndicates the first tool when the input task is i; Tool′ i,j Indicates the second tool when the input task is i; T1M (Tool i ,Tool′ i,j ) represents the first matching degree; S i represents the first planning thinking process when task is i; S′ i,j represents the second planning thinking process when task is i; TtM(S i ,S′ i,j ) represents the second matching degree.

[0100] Through the above method, when the intelligent agent processes the input task, the tools used by the intelligent agent and the planning thinking process of the input task processing flow can be clarified, and the planning ability of the intelligent agent's agent template to be evaluated can be accurately evaluated.

[0101] In the above function for determining the degree of matching, the degree of matching of the planning thinking process can be determined using the TtM function.

[0102] In the above function for determining the matching degree, the matching degree between tools can be determined by first using exact matching to quickly exclude unmatched tool names, then using edit distance or cosine matching to quantify the matching degree, and finally adjusting weights and thresholds as needed to optimize the evaluation results.

[0103] Example 2: Tool usage capabilities.

[0104] When judging the tool to use, the tool parameters and specific tool parameter values ​​selected when the tool is used can be evaluated.

[0105] Figure 3 A flow chart of determining a tool usage capability score in an embodiment of the present disclosure is shown. Figure 3 As shown, the following steps are included:

[0106] S302: For each input task, determine a third matching degree between a third tool corresponding to any to-be-evaluated proxy template and a fourth tool corresponding to any label proxy template.

[0107] In a possible embodiment, in the process of determining the planning ability score, the tool use ability score can be determined simultaneously, so the first tool and the third tool can be the same, and the fourth tool and the second tool can also be the same.

[0108] It should be noted that if the capability indicator in the processing method for a certain input task only includes the tool use capability but does not include the planning capability score, then the first tool and the second tool may be directly determined.

[0109] S304: Determine a fourth matching degree between the corresponding first parameter and the second parameter.

[0110] S306: Determine a fifth matching degree between the corresponding first parameter value and the second parameter value.

[0111] Among them, parameter refers to the parameter information corresponding to the tool, and parameter value refers to the specific setting value of the parameter information.

[0112] For example, the tool used is a certain tool. The parameters corresponding to the tool may include: data source connection parameters, calculation-related parameters, etc. If, in the process of processing the input task, the proxy template to be evaluated uses calculation-related parameters, and the label proxy template also uses calculation-related parameters, it means that the two proxy templates have a high degree of matching in terms of tool parameters.

[0113] For example, when setting parameter values, the calculation parameters include aggregation functions, grouping fields, etc. The proxy template to be evaluated uses aggregation functions, and sets specific processes including summation and maximum value, etc. The label proxy template uses grouping fields for calculation, which means that the two proxy templates have a low matching degree in terms of tool parameters.

[0114] When calculating the matching degree, a variety of matching degree or similarity calculation methods can be selected, which are not limited in the present disclosure, such as cosine similarity.

[0115] S308: Determine a comprehensive score for each input task according to the third matching degree, the fourth matching degree, and the fifth matching degree under each input task.

[0116] In a possible embodiment, a weight may be assigned to each matching degree and a comprehensive score may be determined.

[0117] In another possible embodiment, the inventors have considered that there is an obvious progressive setting method between parameters and parameter values, and if the parameter settings are different, there is no parameter value matching. Therefore, the determination method may include: based on the fourth matching degree and the fifth matching degree, first determine the intermediate matching degree of the parameters, then assign weights to the intermediate matching degree and the third matching degree, and determine the comprehensive score.

[0118] S310: Determine a tool usage capability score of any agent template to be evaluated based on the comprehensive score of each input task.

[0119] In a possible embodiment, after the comprehensive score is determined, the method for determining the tool usage ability score may be similar to the method of step S208 in Example 1, which will not be described in detail here.

[0120] In another possible embodiment, the tool usage capability score may be determined by the following formula (3), as shown below:

[0121]

[0122] Among them, S tool represents the tool usage ability score; M1 represents the number of label proxy templates; N1 represents the number of input tasks, i and j represent the index of input tasks and the index of label proxy template, respectively, indicating the meaning of the i-th and j-th; TlM(Tool1 i ,Tool2′ i,j ) indicates the third matching degree; Tool1 i Indicates the third tool; Tool2′ i,j represents the fourth tool; ω3 and ω4 represent the third weight and the fourth weight respectively; TlM(a k,i ,a′ k,j ) represents the fourth matching degree, PM(v k,i ,v′ k,j ) represents the fifth matching degree; K represents the number of parameters; k represents the index of the parameter; a k,i Represents the first parameter; a′ k,i,j Indicates the second parameter; v k,i represents the first parameter value; v′ k,i,j The second parameter value.

[0123] By means of the above, the correlation between parameters and parameter values ​​can be improved, and the used tools can be further evaluated to determine the performance of the agent template to be evaluated with respect to the used tools, the set parameters and the parameter values.

[0124] In addition, based on the characteristic that parameter values ​​are texts, the edit distance matching method can be used to quickly exclude unmatched parameter values, and then the cosine matching method can be used to quantify the degree of matching. This is because the combination of the edit distance and cosine matching method can quickly exclude obviously unmatched candidates when calculating the matching degree of parameter values, while carefully measuring the semantic matching. This is an efficient and accurate evaluation method, and is particularly suitable for processing scenarios that need to consider both literal and semantic matching, thereby optimizing computing resources and improving the robustness and accuracy of the evaluation.

[0125] Example 3: Diversity generation capabilities.

[0126] It is necessary to determine not only the scores for planning ability and tool use ability, but also the score for variety generation ability, which can characterize the flexibility and diversity in performing tasks.

[0127] Figure 4 A flow chart of determining a diversity generation capability score in an embodiment of the present disclosure is shown. Figure 4 As shown, the following steps are included:

[0128] S402: Obtaining a planning capability score and a tool use capability score.

[0129] S404: Count the cumulative number of times the first output result and the second output result are the same.

[0130] S406: Determine a diversity generation capability score based on the ratio of the cumulative number of times to the total number of input tasks, the planning capability score, and the tool use capability score.

[0131] In this way, a penalty mechanism is introduced into the determination process to identify and reduce repetitive behaviors of the agent when performing tasks, repeatedly using the same strategies or tools in matching tasks without adjusting them according to the context or task requirements, which may indicate a lack of adaptability or innovation.

[0132] In a possible embodiment, the diversity generation capability score can be determined by formula (4), as shown below:

[0133]

[0134] Among them, S adaptability represents the diversity generation ability score; r represents the cumulative number of times the first output result and the second output result are the same; N represents the total number of input tasks; ω5 and ω6 represent the fifth weight and the sixth weight, respectively.

[0135] Example 4: Role-playing capabilities.

[0136] The intelligent agent also needs to generate appropriate output results according to different roles or contexts. An ability indicator called role-playing ability can be defined to measure role adaptability and context understanding ability.

[0137] Figure 5 A flow chart of determining a role playing ability score in an embodiment of the present disclosure is shown. Figure 5 As shown, the following steps are included:

[0138] S502: According to a plurality of predefined roles and input tasks, a first output result corresponding to any to-be-evaluated proxy template and a second output result corresponding to any label proxy template are obtained respectively.

[0139] In a possible embodiment, L different roles may be predefined, each role having a pre-set corresponding context and expected behavior. According to the proxy template to be evaluated and the label proxy template, the first output result and the second output result of the automatically generated multiple agents and the label agent output corresponding to the input task may be obtained respectively.

[0140] S504: Determine a sixth matching degree between the first output result and the second output result corresponding to each role.

[0141] In a possible embodiment, for each role, a sixth matching degree between the first output result and the second output result may be determined.

[0142] S506: Determine the maximum matching degree from the sixth matching degrees corresponding to each role as the role playing ability score.

[0143] The purpose of role-playing ability is to ensure that the agent can generate appropriate outputs according to different roles or contexts, thereby measuring its role adaptability and context understanding ability. Through this indicator, it is possible to identify which roles the intelligent agent performs best in and which roles need further improvement. This helps the developers of the agent understand its performance in different scenarios and guide its optimization and upgrade to improve the accuracy of the evaluation.

[0144] In a possible embodiment, the role-playing ability score can be determined by formula (5), as shown below:

[0145]

[0146] Among them, S profile Indicates the role-playing ability score; OM (O i ,O′ j ) represents the sixth matching degree; O i Represents the first output result; O' k Indicates the second output result.

[0147] Based on the same inventive concept, the present disclosure also provides an agent evaluation device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0148] Figure 6 A schematic diagram of the structure of an intelligent agent evaluation device in an embodiment of the present disclosure is shown. Figure 6As shown, the agent evaluation device 60 includes: an acquisition unit 601, which is used to acquire a test data set; the test data set includes: multiple label proxy templates, and multiple agent proxy templates to be evaluated automatically output by a large language model; a first determination unit 602, which is used to determine, for each agent template to be evaluated, a first output result for multiple input tasks and a second output result for each label proxy template for the input task according to the capability indicators of different dimensions of each agent template to be evaluated; a second determination unit 603, which is used to determine the capability score of the agent template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result; an evaluation unit 604, which is used to determine the comprehensive performance score of the agent template to be evaluated according to the capability score and evaluation weight of the capability indicator of each dimension.

[0149] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0150] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0152] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform any step of the above method embodiments.

[0153] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0154] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0155] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0156] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0157] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0158] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the above embodiment.

[0159] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0160] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0162] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0163] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0164] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0165] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0166] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An agent evaluation method, characterized in that: The method comprises: Acquire a test data set; the test data set includes: a plurality of label proxy templates, and a plurality of proxy templates to be evaluated of agents automatically output by a large language model; For each of the to-be-evaluated proxy templates, determining a first output result of each of the to-be-evaluated proxy templates for a plurality of input tasks and a second output result of each of the label proxy templates for an input task, respectively; Determine, according to the first output result and the second output result, a capability score of the agent template to be evaluated under the capability indicator of each dimension; The comprehensive performance score of the agent template to be evaluated is determined based on the capability score and evaluation weight of the capability indicator of each dimension.

2. The method according to claim 1, characterized in that The capability indicators include: planning capability; the output results include: tools used and planning thinking process; Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes: For each of the input tasks, determining a first matching degree between a first tool corresponding to any to-be-evaluated proxy template and a second tool corresponding to any label proxy template; determining a second degree of matching between the corresponding first planning thinking process and the second planning thinking process; Determining a comprehensive score for each of the input tasks according to the first matching degree and the second matching degree under each of the input tasks; According to the comprehensive score of each of the input tasks, the planning capability score of any one of the agent templates to be evaluated is determined.

3. The method according to claim 2, characterized in that Determining the planning capability score of any one of the agent templates to be evaluated according to the comprehensive score of each of the input tasks includes: Determining a maximum comprehensive score from the comprehensive scores of each of the input tasks; For any of the proxy templates to be evaluated, determining the maximum comprehensive score of each of the label proxy templates; The planning capability score of any one of the proxy templates to be evaluated is determined according to the maximum comprehensive score of each of the label proxy templates and the number of the label proxy templates.

4. The method according to claim 1, characterized in that: The capability indicators include: tool use capability; the output results include: the used tools, the parameters corresponding to the tools, and the parameter values ​​corresponding to the parameters; Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes: For each of the input tasks, determining a third matching degree between a third tool corresponding to any to-be-evaluated proxy template and a fourth tool corresponding to any label proxy template; determining a fourth degree of match between the corresponding first parameter and the second parameter; determining a fifth degree of match between the corresponding first parameter value and the second parameter value; Determining a comprehensive score for each of the input tasks according to the third matching degree, the fourth matching degree, and the fifth matching degree under each of the input tasks; According to the comprehensive score of each of the input tasks, a tool usage capability score of any one of the agent templates to be evaluated is determined.

5. The method according to claim 1, characterized in that The capability indicators include: planning capability, tool use capability and diversity generation capability; Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes: Obtain planning ability scores and tool use ability scores; Counting the cumulative number of times the first output result and the second output result are the same; A diversity generation ability score is determined according to the ratio of the accumulated number of times to the total number of input tasks, the planning ability score, and the tool use ability score.

6. The method according to claim 1, characterized in that The ability indicators include: role-playing ability; Determining the capability score of the proxy template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result includes: According to the predefined multiple roles and the input task, respectively obtain a first output result corresponding to any proxy template to be evaluated and a second output result corresponding to any label proxy template; Determining a sixth matching degree between the first output result corresponding to each of the roles and the second output result; From the sixth matching degrees corresponding to each of the roles, a maximum matching degree is determined as the role playing ability score.

7. An intelligent agent evaluation device, characterized in that: include: An acquisition unit, used to acquire a test data set; The test data set includes: multiple label proxy templates, and proxy templates to be evaluated of multiple agents automatically output by the large language model; A first determining unit is used to determine, for each proxy template to be evaluated, a first output result of each proxy template to be evaluated for a plurality of input tasks and a second output result of each label proxy template for an input task, respectively, with respect to capability indicators of different dimensions of each proxy template to be evaluated; A second determining unit, configured to determine a capability score of the agent template to be evaluated under the capability indicator of each dimension according to the first output result and the second output result; The evaluation unit is used to determine the comprehensive performance score of the agent template to be evaluated according to the ability score and evaluation weight of the ability indicator of each dimension.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images