Assessment method and device of intelligent agent and large language model, medium and electronic equipment

Abstract

The invention discloses an agent and large language model evaluation method and device, a medium and electronic equipment, and relates to the technical field of large models and agents, and the evaluation method comprises the steps: creating an evaluation task which is used for evaluating at least one dialogue of a plurality of to-be-evaluated objects in a dialogue process from at least one evaluation dimension, the plurality of to-be-evaluated objects are used for realizing dialogue tasks in the same field; the evaluation task is executed to obtain an evaluation result, and the evaluation result is used for comparing the dialogue ability among the multiple to-be-evaluated objects; displaying the evaluation result on the first interface, and displaying a second interface; in response to the configuration operation for the evaluator of the target type in the second interface, a configured evaluator is obtained, the configured evaluator is used for evaluating the dialogue from the corresponding evaluation dimension, a user can conveniently analyze the building effect of the agents of different configurations, in addition, user-defined configuration can be carried out based on the configuration operation, and the user experience is improved. And thus, evaluators of different evaluation dimensions are obtained.

Description

Technical Field

[0001] The present disclosure relates to the fields of large model technology and intelligent agent technology, and in particular, to an evaluation method, device, medium, and electronic device for an intelligent agent and a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence technology has been used in many fields. In practical applications, it is necessary to evaluate smart objects based on artificial intelligence technology in order to optimize the ability of smart objects to process downstream tasks in practical applications based on the evaluation results. Summary of the Invention

[0003] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for evaluating an intelligent agent and a large language model, comprising: Creating an evaluation task, wherein the evaluation task is used to evaluate at least one dialogue between multiple objects to be evaluated during a dialogue process from at least one evaluation dimension, the multiple objects to be evaluated including at least an agent, and the multiple objects to be evaluated are used to implement a dialogue task in the same field; Executing the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of the multiple objects to be evaluated; Displaying the evaluation result on the first interface; The evaluation method further comprises: Displaying a second interface, wherein the second interface displays different types of evaluators, the different types of evaluators including at least one of an evaluator based on prompt words, an evaluator based on codes, an evaluator based on basic matching rules, an evaluator based on search enhancement generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; In response to the configuration operation of the evaluator for the target type in the second interface, a configured evaluator is obtained, wherein the configured evaluator is used to evaluate the conversation from the corresponding evaluation dimension.

[0005] In a second aspect, the present disclosure provides an evaluation device for an intelligent agent and a large language model, comprising: A creation module, configured to create an evaluation task, wherein the evaluation task is configured to evaluate at least one dialogue between multiple objects to be evaluated during a dialogue process from at least one evaluation dimension, the multiple objects to be evaluated including at least an agent, and the multiple objects to be evaluated are configured to implement a dialogue task in the same field; An evaluation module, configured to execute the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of the plurality of objects to be evaluated; A first display module, configured to display the evaluation result on a first interface; The evaluation device for the agent and the large language model also includes: a second display module, configured to display a second interface, wherein the second interface displays different types of evaluators, the different types of evaluators including at least one of an evaluator based on prompt words, an evaluator based on codes, an evaluator based on basic matching rules, an evaluator based on search-enhanced generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; The configuration module is used to obtain a configured evaluator in response to a configuration operation of the evaluator for the target type in the second interface, wherein the configured evaluator is used to evaluate the conversation from a corresponding evaluation dimension.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the evaluation method described in the first aspect when executed by a processing device.

[0007] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the evaluation method of the first aspect.

[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the evaluation method described in the first aspect when executed by a processor.

[0009] Through the above technical solution, multiple intelligent agents that realize dialogue tasks in the same field will be compared and evaluated, so that the dialogue capabilities of different intelligent agents can be intuitively compared from the evaluation results displayed on the first interface, which is convenient for users to analyze the construction effects of intelligent agents with different configurations. In addition, custom configurations can be performed based on configuration operations to obtain evaluators with different evaluation dimensions.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flowchart of a method for evaluating an intelligent agent and a large language model according to an embodiment of the present disclosure; Figure 2 is a schematic diagram illustrating a method for determining evaluation results of a conversation under all evaluation dimensions according to an embodiment of the present disclosure; Figure 3 is a schematic diagram of a first interface according to an embodiment of the present disclosure; Figure 4 is a schematic diagram showing a configuration of a second interface according to an embodiment of the present disclosure; Figure 5 is a schematic diagram showing a configuration interface according to an embodiment of the present disclosure; Figure 6 is a schematic diagram of a third interface according to an embodiment of the present disclosure; Figure 7 is a schematic diagram illustrating a process of creating an evaluation task according to an embodiment of the present disclosure; Figure 8 is a schematic diagram of a fifth interface according to an embodiment of the present disclosure; Figure 9 1 is a block diagram of an evaluation device for an intelligent agent and a large language model according to an embodiment of the present disclosure; Figure 10 It is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0013] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0014] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0016] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0018] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] At the same time, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] Intelligent objects based on artificial intelligence (AI), such as large language models (LLMs) and agents, have demonstrated outstanding performance in a variety of fields. However, existing technologies often use a relatively simple evaluation method for LLMs and agents. This method only provides the evaluation results of a specific LLM or agent in a visual interface, making it impossible to intuitively compare the evaluation results of different intelligent objects.

[0024] In view of this, an embodiment of the present disclosure provides an evaluation method for an intelligent agent and a large language model, which is explained and illustrated below in conjunction with the accompanying drawings.

[0025] Figure 1 This is a flow chart of an evaluation method for an intelligent agent and a large language model according to an embodiment of the present disclosure. The evaluation method for an intelligent agent and a large language model can be applied to an electronic device. The evaluation method for an intelligent agent and a large language model can be performed by an evaluation device for an intelligent agent and a large language model, wherein the evaluation device for an intelligent agent and a large language model can be implemented by software and / or hardware, and the software and / or hardware can be configured in an electronic device. Figure 1 The evaluation method of the intelligent agent and the large language model may include steps 110, 120, 130, 140 and 150.

[0026] In step 110, an evaluation task is created, wherein the evaluation task is used to evaluate at least one dialogue of multiple objects to be evaluated during a dialogue process from at least one evaluation dimension, the multiple objects to be evaluated include at least intelligent agents, and the multiple objects to be evaluated are used to implement dialogue tasks in the same field.

[0027] The types of the multiple objects to be evaluated can be the same. For example, the multiple objects to be evaluated are all agents. It is understandable that different agents have different configurations, for example, they rely on different large language models, thereby enabling comparative evaluation of multiple agents.

[0028] The types of multiple objects to be evaluated can be diverse. As an example, in addition to the intelligent agent, the multiple objects to be evaluated can also include a large language model, specifically a first large language model, and the intelligent agent is built based on at least the second large language model, the first large language model and the second large language model are different or the same, and the dialogue task implemented by the first large language model is the same as the dialogue task implemented by the intelligent agent. For example, a dialogue task refers to providing an input question to an intelligent object and obtaining a corresponding reply from the intelligent object. An input question and a reply constitute a dialogue. The following takes the example of multiple objects to be evaluated including an intelligent agent and a large language model to explain and illustrate the present disclosure.

[0029] As can be seen from the above, the intelligent agent is built based on at least the second largest language model. In other words, the intelligent agent is a system built using the large language model as a component and combining it with other components. For example, a plug-in with search capabilities can be configured; another example is a knowledge base with professional domain knowledge; another example is a plug-in with text-to-image capabilities, and so on.

[0030] From the above content, we can see that the first and second language models can be different, and the performance of the first language model is better than that of the second language model. On this basis, based on the evaluation results of the first language model and the intelligent agent that relies on the second language model, it is evaluated whether the intelligent agent obtained by configuring additional plug-ins for the second language model with poor performance can achieve or exceed the performance of the first language model. This is suitable for business needs that want to use a more cost-effective model to build an intelligent agent.

[0031] From the above content, we can see that the first language model and the second language model can be the same, that is, the bare model and the intelligent agent are evaluated, so that the construction effect of the intelligent agent can be evaluated.

[0032] The evaluation dimensions can be manually configured or default. It is worth noting that the default evaluation dimensions are mandatory. For example, the default evaluation dimensions can include the amount of tokens used in a conversation and the time it takes for a smart object to generate a reply. For more information on manually configured evaluation dimensions, please refer to the following related embodiments and will not be elaborated on in this embodiment.

[0033] In step 120 , the evaluation task is performed to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of multiple objects to be evaluated.

[0034] Among them, since the evaluation task may involve evaluation of multiple evaluation dimensions, the evaluation of different evaluation dimensions can be performed in parallel, thereby improving the evaluation efficiency.

[0035] In step 130 , the evaluation result is displayed on the first interface.

[0036] The evaluation result may include at least one of a first evaluation result of each object to be evaluated, a second evaluation result of each object to be evaluated, and a third evaluation result of each object to be evaluated.

[0037] The first evaluation result is used to represent the evaluation result of each conversation corresponding to the object to be evaluated under each evaluation dimension.

[0038] The second evaluation result is used to represent the fusion result of the first evaluation result. The first evaluation result is used to represent the evaluation results of all conversations corresponding to the object to be evaluated under the same evaluation dimension. The second evaluation result can support filtering of the evaluation dimension to be displayed, so that only the second evaluation results corresponding to the selected evaluation dimension are displayed. The fusion result of the first evaluation result can be obtained based on weights. Due to the different focuses of the conversations, the weights of each conversation under different evaluation dimensions may be different. These weights can be manually configured by the user or automatically recommended by the model. As an example, in the case of automatic model recommendation, the model can automatically generate recommended weights for each evaluation dimension for the conversation based on the conversation and the evaluation dimension. For example, if the conversation is about the weather, if the evaluation dimension is accuracy, a higher weight can be set for accuracy. If the evaluation dimension is creativity, a lower weight can be set for creativity, thereby improving the accuracy of the evaluation.

[0039] The third evaluation result represents the fusion of the second evaluation result. The second evaluation result represents the evaluation results of all conversations with the corresponding object under all evaluation dimensions. For example, the third evaluation result, taking the evaluation task-related evaluation dimensions including RAG (Retrieval-Augmented Generation) and other dimensions (such as accuracy), can be determined as follows: First, calculate the evaluation results of the conversation in other dimensions; Then, if the Trace data shows that the object to be evaluated has called the RAG capability during the reasoning process for the conversation, the evaluation results of other dimensions can be integrated with the evaluation results of the RAG dimension to obtain the evaluation results of the conversation in all evaluation dimensions. Figure 2 FIG. 1 is a schematic diagram showing a method for determining the evaluation results of a conversation under all evaluation dimensions according to an embodiment of the present disclosure. Figure 2The object to be evaluated performs the evaluation task, completes the reasoning of the input question, and obtains the dialogue; determines whether the RAG capability is called during the reasoning process. If the RAG capability is called, the context of the retrieval and recall in the Trace data is extracted, and the RAG evaluator determines the evaluation result of the dialogue in the RAG dimension based on the context, and further integrates the evaluation results of the dialogue in the RAG dimension and other dimensions to obtain the evaluation results of the dialogue in all evaluation dimensions, that is, the second target evaluation result of the dialogue; if the RAG capability is not called, the evaluation results of the dialogue in other evaluation dimensions are integrated to obtain the evaluation results of the dialogue in all evaluation dimensions, that is, the second target evaluation result of the dialogue.

[0040] Finally, the second target evaluation results corresponding to all conversations are fused to obtain the third evaluation result.

[0041] As an example, step 130 may include: displaying a first evaluation result for each object to be evaluated in a first area of the first interface; displaying a second evaluation result for each object to be evaluated in a chart and / or table in a second area of the first interface; and displaying a third evaluation result for each object to be evaluated in a third area of the first interface. The first area, the second area, and the third area are different from each other.

[0042] Among them, tables can display specific values, while charts are used to intuitively reflect the differences between different objects to be evaluated. As examples, charts can be radar charts and bar charts. Taking the radar chart as an example, the radar chart includes N category axes, each category axis is used to represent an evaluation dimension, and N is an integer greater than 1. When the radar chart includes the second evaluation results of different objects to be evaluated, each object to be evaluated can be identified by a different legend. In addition, for charts, zooming is supported, that is, the chart can be enlarged or reduced by performing a zoom operation on the chart.

[0043] Figure 3 is a schematic diagram of the first interface according to an embodiment of the present disclosure. It is worth noting that, Figure 3 The evaluation set in consists of the input questions provided to the object to be evaluated (i.e. Figure 3 The reference content may be a manually configured reference response to the input question, which may assist in the evaluation of the object to be evaluated.

[0044] Continue to refer to Figure 3 The first area 301 of the first interface displays the first evaluation results of each conversation of different objects to be evaluated in each evaluation dimension, for example Figure 3 Dimensions 1, 2, 3, 4, and 5 are shown. Dimensions 1, 2, and 5 are manually configured, while Dimensions 3 and 4 are default assessment dimensions, which are automatically added to the assessment task when the assessment task is configured.

[0045] Continue to refer to Figure 3 The second evaluation result is displayed in the second area 302 of the first interface, and the evaluation dimensions corresponding to the second evaluation result may include dimension 1, dimension 2, dimension 3 and dimension 4, and the second evaluation results of all objects to be evaluated in dimension 1, dimension 2, dimension 3 and dimension 4 are displayed in the form of radar charts and tables.

[0046] Continue to refer to Figure 3 The first interface can also display information on whether the conversation is successful.

[0047] In step 140 , a second interface is displayed, wherein different types of evaluators are displayed in the second interface.

[0048] In step 150 , in response to the configuration operation of the evaluator for the target type in the second interface, a configured evaluator is obtained, wherein the configured evaluator is used to evaluate the conversation from the corresponding configured evaluation dimension.

[0049] The above evaluation can be implemented based on evaluators, each of which corresponds to an evaluation dimension. Users can pre-configure evaluators and then select the corresponding evaluator when configuring the evaluation task to evaluate the conversation based on the corresponding evaluation dimension.

[0050] In this embodiment, different types of evaluators may include at least one of a prompt word-based evaluator, a code-based evaluator, a basic matching rule-based evaluator, a retrieval-enhanced generation-based evaluator, a manual evaluation-based evaluator, and a natural language processing-based evaluator. These evaluators can be regarded as internal evaluators, and these internal evaluators can pre-configure basic content. These basic contents can be content shared by different evaluation dimensions to improve configuration efficiency.

[0051] In addition, when configuring an evaluation task, you can also provide an external evaluator for selection. The external evaluator is an evaluator provided by a third party. The external evaluator can be connected through a gateway or other means to meet advanced and complex evaluation needs.

[0052] Through the above technical solution, multiple agents that realize dialogue tasks in the same field are comparatively evaluated, so as to intuitively compare the dialogue capabilities of different agents from the evaluation results displayed on the first interface, so that users can analyze the construction effects of agents with different configurations. In addition, the evaluation of two types of objects to be evaluated is realized, namely the comparative evaluation of large language models and agents, and the evaluation results of multiple objects to be evaluated are visualized to facilitate comparative analysis and viewing by evaluators. When the large language models corresponding to the two types of objects to be evaluated are consistent, it is convenient to evaluate the construction effect of the agent. When the large language models corresponding to the two types of objects to be evaluated are inconsistent, it is convenient to evaluate whether the agent built based on the second large language model can achieve or exceed the performance of the bare model. In addition, custom configuration can be performed based on the configuration operation to obtain evaluators with different evaluation dimensions.

[0053] Figure 4 : is a schematic diagram showing a configuration of a second interface according to an embodiment of the present disclosure. Figure 4 The second interface displays different types of evaluators. The evaluators based on the prompt words correspond to Figure 4 The Prompt type in [ ]. A prompt-based evaluator can be implemented using an LLM, which uses the LLM to score the output of the evaluation object based on predefined prompts. Prompts can be preset or user-defined, and different prompts correspond to different evaluation dimensions, such as correctness, creativity, depth, balance, and precision.

[0054] Code-based evaluator correspondence Figure 4 The code type in the code-based evaluator is implemented based on code. Different codes can correspond to different evaluation rules, and different evaluation rules correspond to different evaluation dimensions. For example, it supports configuring evaluation rules by writing code such as Python and Java.

[0055] Evaluator corresponding to basic matching rules Figure 4 Basic matching rule-based evaluators are implemented based on rules such as regular expressions and fuzzy matching. Users can configure these rules to implement evaluations based on specific keywords or format errors.

[0056] Evaluator corresponding to retrieval-enhanced generation Figure 4The RAG type in . The evaluator generated based on retrieval enhancement can be implemented through LLM, and evaluation is achieved based on evaluation dimensions such as retrieval relevance, response accuracy, and knowledge coverage. Retrieval relevance is used to describe the cosine similarity between the input question and the recalled document; response accuracy is used to describe the similarity (such as ROUGE-L) and entity matching between the inference result (i.e., response) of the object to be evaluated and the retrieved document; knowledge coverage is used to describe whether the inference result of the object to be evaluated covers the key information of the retrieved document. Knowledge coverage can be achieved through TF-IDF (Term Frequency-Inverse Document Frequency).

[0057] Evaluator based on manual evaluation Figure 4 The manual rule type in the manual evaluation. The evaluator based on manual evaluation allows users to define the evaluation criteria through text, scores or options, and then manually evaluate them.

[0058] Evaluator based on natural language processing Figure 4 The NLP (Natural Language Processing) type in [1]. The NLP-based evaluator is implemented using NLP technology and evaluates dimensions such as user satisfaction and conversation coherence.

[0059] From the above content, we can see that one type of evaluator can be configured with multiple evaluation logics, and each evaluation logic corresponds to an evaluation dimension. Therefore, based on manual custom configuration, an evaluator that evaluates from various evaluation dimensions can be obtained.

[0060] Among them, the configuration operation of the evaluator may include at least one of the configuration operation of the prompt words on which the evaluator depends, the configuration operation of the variables associated with the prompt words on which the evaluator depends, the configuration operation of the evaluation model on which the evaluator depends, the configuration operation of the model parameters of the evaluation model, and the configuration operation of the evaluation logic of the evaluator.

[0061] For a preset evaluator (eg, an evaluator based on prompt words), the prompt words can be modified according to actual needs.

[0062] For objects to be evaluated that are configured with multiple variables, the preset evaluator (e.g., an evaluator based on prompt words) cannot support adjustments to the evaluator's input fields or the format of the input fields based on the changing variables. Therefore, the evaluator cannot adapt to the objects to be evaluated with dynamic variables, resulting in errors in the evaluation results and inability to identify them, which in turn affects the accuracy of the evaluation results and the judgment of the evaluators. To solve this problem, support is provided for the configuration of variables associated with the prompt words that the preset evaluator relies on, so that the input parameters of the evaluator can be adjusted according to actual conditions. It is understandable that the adjustment can be an operation to add new variables and an operation to change the format of variables, etc.

[0063] For estimators that are bound to a specific underlying model, it's impossible to switch the model they rely on. This results in limited evaluation results and limited flexibility in adapting to different scenarios. Therefore, we can provide configuration operations for the evaluation model that the estimator relies on and its model parameters.

[0064] For rule-based or code-based evaluators, the evaluation logic of the evaluator can be configured according to actual needs to adapt to different scenarios.

[0065] Figure 5 This is a schematic diagram of a configuration interface according to an embodiment of the present disclosure. The configuration interface can be displayed on the second interface in the form of a floating layer. Figure 4 After selecting the target type, you can configure the evaluator to obtain an evaluator that belongs to the target type and can evaluate the conversation from the corresponding configured evaluation dimension. When the configuration of the evaluator for the target type is triggered, it is displayed Figure 5 The configuration interface shown. Continue to refer to Figure 5 By configuring the parameters of the evaluator such as the model and prompt words, the configured evaluator can evaluate the dialogue from the evaluation dimension of correctness.

[0066] For code-based evaluators, testing capabilities can also be provided. Therefore, in some embodiments, the above-mentioned evaluation method for intelligent agents and large language models can also include: in response to a debugging operation on a configured evaluator, displaying a third interface, wherein the third interface displays the evaluator's program code and an input parameter editing area, the program code is used to represent the evaluator's evaluation logic, and the input parameter editing area is used to edit the input parameters of the test program code; in response to a test request for the program code, running the program code based on the input parameters to obtain a test result; and displaying the test result in the test result area of the third interface.

[0067] Figure 6 is a schematic diagram of a third interface according to an embodiment of the present disclosure. Figure 6, displays the evaluator's program code and supports online editing of the program code. The parameter editing area 601 is used to edit the test program code's input parameters. When a test request is initiated for a test control 602, the program code is run based on the parameters edited in the parameter editing area 601, and the test results are displayed in the test result area 603. Users can adjust the program code based on the test results to configure the evaluator. This improves the configuration capabilities of custom evaluators.

[0068] Because users can configure different estimators based on their needs, and some pre-built estimators rely on different model architectures, this can lead to issues like cross-framework communication and resource scheduling. Furthermore, related technologies don't support dynamic parsing of estimator input parameters. To address this, we propose encapsulating each configured estimator as an independent plug-in and managing it through a predefined unified interface.

[0069] A predefined unified interface, such as an RPC (Remote Procedure Call) interface or an HTTP (Hypertext Transfer Protocol) interface, is used to communicate with plugins. This addresses cross-framework communication and resource scheduling issues, as well as support for dynamic parsing of evaluator input parameters. Input parameters to the interface include the conversation of the evaluated object, plugin metadata, and configuration parameters. Plugin metadata determines the invoked plugin, i.e., the invoked evaluator. Configuration parameters enable hot updates of plugins. Output parameters include evaluation results and scoring rationale.

[0070] In some embodiments, each plug-in can be run as a separate microservice. As an example, the plug-in runtime can be deployed through the Docker container engine.

[0071] In some embodiments, when multiple plug-ins are called, parallel calling of multiple plug-ins can be supported to improve evaluation efficiency.

[0072] In some embodiments, the evaluation task is also used to instruct multiple objects to be evaluated to reason about input questions in a selected target data set to generate at least one dialogue. The above-mentioned steps of creating an evaluation task can be implemented in the following manner: displaying a fourth interface, the fourth interface is used to create an evaluation task; in response to a selection operation for a data set in the fourth interface, determining a target data set; in response to a selection operation for a candidate object to be evaluated in the fourth interface, determining multiple objects to be evaluated; in response to a selection operation for a configured candidate evaluator and / or candidate evaluator group in the fourth interface, determining an evaluator, which is used to evaluate at least one dialogue from a corresponding evaluation dimension; in response to a creation operation in the fourth interface, creating an evaluation task based on the evaluator, multiple objects to be evaluated and the target data set.

[0073] It can be understood that the evaluation task in this embodiment is not only used to instruct each object to be evaluated to reason about the input questions in the selected target data set to generate at least one dialogue, but also used to evaluate at least one dialogue of each object to be evaluated during the dialogue process from at least one evaluation dimension, and the evaluation is implemented by calling the evaluator.

[0074] The candidate evaluator group can be created in advance. When creating the candidate evaluator group, the group name and corresponding group description can be configured, and the evaluators in the candidate evaluator group can be configured.

[0075] Figure 7 This is a schematic diagram illustrating a process for creating an evaluation task according to an embodiment of the present disclosure. The target dataset includes an input question and reference content. The candidate objects to be evaluated include those belonging to a large language model and those belonging to an agent. Selection is made from these candidate objects to obtain the object to be evaluated. In this embodiment, when executing a configured evaluation task, the object to be evaluated is first used to reason about the input question, obtain a response, and then obtain a corresponding dialogue. Next, the evaluator is called to evaluate the dialogue based on the corresponding evaluation dimension to obtain an evaluation result.

[0076] Continue to refer to Figure 7 When selecting an evaluator for evaluating at least one conversation from at least one evaluation dimension, a pre-configured candidate evaluator group can be directly reused to avoid repeated configuration of evaluators for periodically performed evaluation tasks.

[0077] In other embodiments, when creating an evaluation task, a selection can be made in real time from provided evaluators 1, 2, ..., M to determine the evaluator used to evaluate at least one conversation based on at least one evaluation dimension. It is understood that M is an integer greater than 2.

[0078] In other embodiments, a pre-configured candidate evaluator group may be directly reused and fine-tuned to obtain an evaluator for evaluating at least one conversation from at least one evaluation dimension.

[0079] As can be seen from the above content, evaluation dimensions can include not only manually configured dimensions but also mandatory evaluation dimensions. Therefore, when creating an evaluation task, the evaluators associated with the evaluation task include not only the evaluators selected by the user from the candidate evaluators, but also the evaluators corresponding to the mandatory evaluation dimensions.

[0080] In some embodiments, the evaluation method of the above-mentioned intelligent agent and large language model may also include: in response to the selection operation of the target dialogue in the first interface, displaying a fifth interface, wherein the fifth interface is at least used to display the evaluation results of the target dialogue under each object to be evaluated and under each evaluation dimension; in response to the hover operation of the evaluation result of the target dialogue in the fifth interface, displaying the scoring reason, wherein the evaluation result is obtained by evaluation based on the prompt word evaluator.

[0081] The target dialogue belongs to a group of dialogues in a round of dialogues performed by the object to be evaluated. A round of dialogues can include multiple groups of dialogues.

[0082] Figure 8 is a schematic diagram of a fifth interface according to an embodiment of the present disclosure. Figure 8 The fifth interface 801 may be displayed in the form of a floating layer on top of the first interface. The content displayed on the fifth interface 801 may include the target dialogue (including input questions and the reasoning results of the intelligent object for the dialogue), the evaluation results of the target dialogue under various evaluation dimensions, the deep thinking process of the large language model, and so on.

[0083] Continue to refer to Figure 8 The scoring reasons 802 can be displayed on the fifth interface 801 in the form of a floating layer. By displaying the scoring reasons, the evaluator can intuitively understand the evaluation logic of the evaluator.

[0084] Based on the same concept, the present disclosure provides an evaluation device for an intelligent agent and a large language model. Figure 9 is a block diagram of an evaluation device for an intelligent agent and a large language model according to an embodiment of the present disclosure, with reference to Figure 9 , the evaluation device 900 for the agent and the large language model includes: A creation module 901 is configured to create an evaluation task, wherein the evaluation task is configured to evaluate at least one dialogue between multiple objects to be evaluated, including at least agents, during a dialogue process, based on at least one evaluation dimension. The multiple objects to be evaluated are configured to implement a dialogue task in the same domain. An evaluation module 902 is configured to execute the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of the plurality of objects to be evaluated; A first display module 903 is used to display the evaluation result on a first interface; The evaluation device 900 for the agent and the large language model further includes: A second display module 904 is configured to display a second interface, wherein the second interface displays different types of evaluators, including at least one of an evaluator based on prompt words, an evaluator based on codes, an evaluator based on basic matching rules, an evaluator based on search enhancement generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; The configuration module 905 is configured to obtain a configured evaluator in response to the configuration operation of the evaluator for the target type in the second interface, wherein the configured evaluator is used to evaluate the conversation from the corresponding evaluation dimension.

[0085] Optionally, the configuration operation includes at least one of the following: Configuration operations on the prompt words that the evaluator relies on; Configuration operations on variables associated with the cue words on which the evaluator depends; Configuration operations on the evaluation model relied upon by the evaluator; Configuration operations on model parameters of the evaluation model; Configuration operations on the evaluation logic of the evaluator.

[0086] Optionally, the agent and large language model evaluation device 900 further includes: a third display module, configured to display a third interface in response to a debugging operation on a configured evaluator, wherein the third interface displays the program code of the evaluator and an input parameter editing area, the program code being used to represent the evaluation logic of the evaluator, and the input parameter editing area being used to edit input parameters for testing the program code; A testing module, configured to respond to a test request for the program code, execute the program code based on the input parameters, and obtain a test result; The fourth display module is used to display the test results in the test result area of the third interface.

[0087] Optionally, each configured evaluator is encapsulated as an independent plug-in, and management of each plug-in is achieved through a predefined unified interface.

[0088] Optionally, the evaluation task is further configured to instruct the multiple objects to be evaluated to reason about input questions in the selected target data set to generate the at least one dialogue, and the creation module 901 is further configured to: Displaying a fourth interface, wherein the fourth interface is used to create an assessment task; In response to a selection operation on a data set in the fourth interface, determining the target data set; In response to a selection operation on a candidate object to be evaluated in the fourth interface, determining the plurality of objects to be evaluated; In response to a selection operation on a configured candidate evaluator and / or candidate evaluator group in the fourth interface, determining an evaluator, wherein the evaluator is used to evaluate the at least one conversation based on the corresponding evaluation dimension; In response to the creation operation in the fourth interface, the evaluation task is created based on the evaluator, the multiple objects to be evaluated, and the target data set.

[0089] Optionally, the first display module 903 is further configured to: Displaying, in a first area of the first interface, a first evaluation result of each of the objects to be evaluated, wherein the first evaluation result is used to represent an evaluation result of each of the conversations corresponding to the objects to be evaluated under each of the evaluation dimensions; Displaying, in a second area of the first interface, a second evaluation result of each of the objects to be evaluated in each of the evaluation dimensions in a chart or table, wherein the second evaluation result is used to represent a fusion result of the first target evaluation result, and the first target evaluation result is used to represent the evaluation results of all the conversations corresponding to the object to be evaluated under the same evaluation dimension; The third evaluation result of each object to be evaluated is displayed in the third area of the first interface, wherein the third evaluation result is used to represent the fusion result of the second target evaluation result, and the second target evaluation result is used to represent the evaluation results of all the conversations corresponding to the object to be evaluated under all the evaluation dimensions.

[0090] Optionally, the chart includes a radar chart, which identifies each object to be evaluated by a different legend, and the radar chart includes N category axes, each category axis is used to represent one evaluation dimension, and N is an integer greater than 1.

[0091] Optionally, the agent and large language model evaluation device 900 further includes: a fifth display module, configured to display a fifth interface in response to a selection operation on the target conversation in the first interface, wherein the fifth interface is configured to display at least an evaluation result of the target conversation under each of the objects to be evaluated and under each of the evaluation dimensions; The sixth display module is used to display the scoring reason in response to the hover operation on the evaluation result of the target dialogue in the fifth interface, wherein the evaluation result is obtained by evaluating the evaluator based on the prompt word.

[0092] Optionally, the multiple objects to be evaluated also include a first large language model, which is used to implement the dialogue task. The intelligent agent is built based on at least a second large language model, and the first large language model and the second large language model are different or the same.

[0093] Among them, the implementation of each module in the above-mentioned evaluation device 900 for intelligent agents and large language models can refer to the above-mentioned relevant embodiments, and this embodiment will not be described in detail here.

[0094] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the above-mentioned evaluation method for an intelligent agent and a large language model.

[0095] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned evaluation method for intelligent agents and large language models.

[0096] Based on the same concept, an embodiment of the present disclosure further provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the above-mentioned evaluation method for intelligent agents and large language models.

[0097] Reference below Figure 10 , which shows a schematic structural diagram of an electronic device 1000 suitable for implementing an embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0098] like Figure 10As shown, electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of electronic device 1000 are also stored in RAM 1003. Processing device 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0099] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Figure 10 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0100] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0101] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0102] In some embodiments, electronic devices can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can interconnect with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0103] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0104] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: creates an evaluation task, wherein the evaluation task is used to evaluate at least one conversation between multiple objects to be evaluated during a conversation process from at least one evaluation dimension, wherein the multiple objects to be evaluated include at least an intelligent agent, and the multiple objects to be evaluated are used to implement conversation tasks in the same field; executes the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversation capabilities between the multiple objects to be evaluated; and displays the evaluation result on a first interface. The evaluation method further includes: displaying a second interface, wherein different types of evaluators are displayed in the second interface, wherein the different types of evaluators include at least one of an evaluator based on prompt words, an evaluator based on code, an evaluator based on basic matching rules, an evaluator based on retrieval enhancement generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; and in response to a configuration operation for an evaluator of a target type in the second interface, obtains a configured evaluator, wherein the configured evaluator is used to evaluate the conversation from the corresponding evaluation dimension.

[0105] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of the present disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0108] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0111] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0112] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A method for evaluating an agent and a large language model, characterized in that: The evaluation method includes: Creating an evaluation task, wherein the evaluation task is used to evaluate at least one dialogue between multiple objects to be evaluated during a dialogue process from at least one evaluation dimension, the multiple objects to be evaluated including at least an agent, and the multiple objects to be evaluated are used to implement a dialogue task in the same field; Executing the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of the multiple objects to be evaluated; Displaying the evaluation result on the first interface; The evaluation method further comprises: Displaying a second interface, wherein the second interface displays different types of evaluators, the different types of evaluators including at least one of an evaluator based on prompt words, an evaluator based on codes, an evaluator based on basic matching rules, an evaluator based on search enhancement generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; In response to the configuration operation of the evaluator for the target type in the second interface, a configured evaluator is obtained, wherein the configured evaluator is used to evaluate the conversation from the corresponding evaluation dimension.

2. The evaluation method according to claim 1, wherein: The configuration operation includes at least one of the following: Configuration operations on the prompt words that the evaluator relies on; Configuration operations on variables associated with the cue words on which the evaluator depends; Configuration operations on the evaluation model relied upon by the evaluator; Configuration operations on model parameters of the evaluation model; Configuration operations on the evaluation logic of the evaluator.

3. The evaluation method according to claim 1, wherein: The evaluation method further comprises: In response to a debugging operation on a configured evaluator, a third interface is displayed, wherein the third interface displays the program code of the evaluator and an input parameter editing area, the program code is used to represent the evaluation logic of the evaluator, and the input parameter editing area is used to edit the input parameters of the program code; In response to a test request for the program code, running the program code based on the input parameters to obtain a test result; The test result is displayed in the test result area of the third interface.

4. The evaluation method according to claim 1, wherein: Each configured evaluator is encapsulated as an independent plug-in, and management of each plug-in is achieved through a predefined unified interface.

5. The evaluation method according to claim 1, wherein: The evaluation task is further used to instruct the multiple objects to be evaluated to reason about input questions in the selected target data set to generate the at least one dialogue. Creating the evaluation task includes: Displaying a fourth interface, wherein the fourth interface is used to create an assessment task; In response to a selection operation on a data set in the fourth interface, determining the target data set; In response to a selection operation on a candidate object to be evaluated in the fourth interface, determining the plurality of objects to be evaluated; In response to a selection operation on a configured candidate evaluator and / or candidate evaluator group in the fourth interface, determining an evaluator, wherein the evaluator is used to evaluate the at least one conversation based on the corresponding evaluation dimension; In response to the creation operation in the fourth interface, the evaluation task is created based on the evaluator, the multiple objects to be evaluated, and the target data set.

6. The evaluation method according to claim 1, wherein: The displaying of the evaluation result on the first interface includes: Displaying, in a first area of the first interface, a first evaluation result of each of the objects to be evaluated, wherein the first evaluation result is used to represent an evaluation result of each of the conversations corresponding to the objects to be evaluated under each of the evaluation dimensions; Displaying, in a second area of the first interface, a second evaluation result of each of the objects to be evaluated in each of the evaluation dimensions in a chart or table, wherein the second evaluation result is used to represent a fusion result of the first target evaluation result, and the first target evaluation result is used to represent the evaluation results of all the conversations corresponding to the object to be evaluated under the same evaluation dimension; The third evaluation result of each object to be evaluated is displayed in the third area of the first interface, wherein the third evaluation result is used to represent the fusion result of the second target evaluation result, and the second target evaluation result is used to represent the evaluation results of all the conversations corresponding to the object to be evaluated under all the evaluation dimensions.

7. The evaluation method according to claim 6, characterized in that The chart includes a radar chart, which identifies each object to be evaluated through different legends. The radar chart includes N category axes, each category axis is used to represent one evaluation dimension, and N is an integer greater than 1.

8. The evaluation method according to claim 6, wherein: The evaluation method further comprises: In response to a selection operation on the target conversation in the first interface, a fifth interface is displayed, wherein the fifth interface is used to at least display the evaluation results of the target conversation under each of the objects to be evaluated and under each of the evaluation dimensions; In response to a hover operation on the evaluation result of the target conversation in the fifth interface, a scoring reason is displayed, wherein the evaluation result is obtained by evaluating an evaluator based on prompt words.

9. The evaluation method according to any one of claims 1 to 8, characterized in that: The multiple objects to be evaluated also include a first large language model, which is used to implement the dialogue task. The intelligent agent is built based on at least a second large language model, and the first large language model and the second large language model are different or the same.

10. An evaluation device for an intelligent agent and a large language model, characterized in that: include: A creation module, configured to create an evaluation task, wherein the evaluation task is configured to evaluate at least one dialogue between multiple objects to be evaluated during a dialogue process from at least one evaluation dimension, the multiple objects to be evaluated including at least an agent, and the multiple objects to be evaluated are configured to implement a dialogue task in the same field; An evaluation module, configured to execute the evaluation task to obtain an evaluation result, wherein the evaluation result is used to compare the conversational abilities of the plurality of objects to be evaluated; A first display module, configured to display the evaluation result on a first interface; The evaluation device for the agent and the large language model also includes: a second display module, configured to display a second interface, wherein the second interface displays different types of evaluators, the different types of evaluators including at least one of an evaluator based on prompt words, an evaluator based on codes, an evaluator based on basic matching rules, an evaluator based on search-enhanced generation, an evaluator based on manual evaluation, and an evaluator based on natural language processing; The configuration module is used to obtain a configured evaluator in response to a configuration operation of the evaluator for the target type in the second interface, wherein the configured evaluator is used to evaluate the conversation from a corresponding evaluation dimension.

11. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the evaluation method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the evaluation method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the evaluation method according to any one of claims 1 to 9 are implemented.

Images