Evaluators generating method, apparatus and storage medium

Abstract

The present disclosure provides an evaluator generation method, device and storage medium, and relates to the technical field of artificial intelligence. The evaluator generation method comprises the following steps: obtaining description information of an evaluation object; determining evaluation indexes based on a first machine learning model according to the description information; and generating an evaluator for each evaluation index based on a second machine learning model according to the description information.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an evaluator generation method, apparatus and storage medium. Background Technology

[0002] With the rapid development of large-scale model technology, large-scale model intelligent agents, as AI (Artificial Intelligence) systems with autonomous decision-making, environmental interaction, and task execution capabilities, have been widely applied in government affairs, public services, and other fields. The performance of the intelligent agent directly affects its reliability and user experience. Scientific and comprehensive evaluation of the intelligent agent during the R&D process can effectively improve its service quality, reliability, and user experience.

[0003] By evaluating agents, we can identify the problems that agents currently have, thereby promoting their rapid iteration and improvement. Summary of the Invention

[0004] One objective of this disclosure is to improve the comprehensiveness and accuracy of the evaluation of intelligent agent evaluators.

[0005] According to one aspect of some embodiments of this disclosure, an evaluator generation method is proposed, comprising: obtaining descriptive information of an evaluation object; determining an evaluation index based on a first machine learning model according to the descriptive information; and generating an evaluator for each evaluation index based on a second machine learning model and the descriptive information.

[0006] In some embodiments, determining evaluation metrics based on a first machine learning model according to the description information includes: determining intermediate links of the evaluation object according to the description information; and determining multiple evaluation metrics based on the first machine learning model when the number of intermediate links is greater than a predetermined number, wherein each evaluation metric corresponds to at least one intermediate link, and the multiple evaluation metrics correspond to all intermediate links.

[0007] In some embodiments, the description information includes a complete description of the evaluation object and evaluation requirements.

[0008] In some embodiments, determining the evaluation metrics based on the first machine learning model according to the description information includes: determining the intermediate links of the evaluation object based on the first machine learning model according to the full-link description of the evaluation object; determining the evaluation targets for one or more intermediate links based on the first machine learning model according to the evaluation requirements; and generating the evaluation metrics based on the first machine learning model according to the intermediate links and the evaluation targets.

[0009] In some embodiments, generating an evaluator for each evaluation metric based on a second machine learning model includes: determining prompt words for each evaluation metric based on the second machine learning model and description information, wherein the prompt words include the input requirements of the intermediate link corresponding to the evaluation metric, the output requirements of the intermediate link corresponding to the evaluation metric, the evaluation criteria, the scoring rules, and the output requirements of the evaluator, and the evaluator includes the evaluation metric and the prompt words.

[0010] In some embodiments, the evaluator generation method further includes: generating evaluation metrics and prompt words based on a third machine learning model and description information when the number of intermediate links is less than or equal to a predetermined number, wherein the evaluator includes evaluation metrics and prompt words.

[0011] In some embodiments, the evaluator generation method further includes: obtaining feedback from the evaluator; and updating at least one of the evaluation metrics or the evaluator's prompts based on the feedback.

[0012] In some embodiments, updating the evaluator's prompt based on feedback includes adding feedback to the prompt for the corresponding evaluator.

[0013] In some embodiments, updating the evaluator's prompts based on feedback includes: updating the evaluator's prompts based on the feedback and the evaluator's prompts, using a fourth machine learning model.

[0014] According to one aspect of some embodiments of this disclosure, an evaluator generation apparatus is proposed, comprising: an information acquisition unit configured to acquire descriptive information of an evaluation object; a processing unit configured to determine an evaluation index based on the descriptive information and a first machine learning model; and a generation unit configured to generate an evaluator for each evaluation index based on a second machine learning model and the descriptive information.

[0015] In some embodiments, the evaluator generation apparatus further includes an update unit configured to obtain feedback on the evaluator and update at least one of the evaluation metrics or the evaluator's prompts based on the feedback.

[0016] According to one aspect of some embodiments of this disclosure, an evaluator generation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the evaluator generation methods described above based on instructions stored in the memory.

[0017] According to one aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement any of the evaluator generation methods described above.

[0018] According to one aspect of some embodiments of this disclosure, a computer program product is proposed, including a computer program or instructions that, when executed by a processor, implement any of the evaluator generation methods described above. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure.

[0020] Figure 1 Flowcharts are provided for some embodiments of the evaluator generation method of this disclosure.

[0021] Figure 2 Flowcharts for some other embodiments of the evaluator generation method of this disclosure.

[0022] Figure 3 These are schematic diagrams of some embodiments of the evaluator generation apparatus of this disclosure.

[0023] Figure 4 These are schematic diagrams of other embodiments of the evaluator generation apparatus of this disclosure.

[0024] Figure 5 This is a schematic diagram of some further embodiments of the evaluator generation apparatus of this disclosure. Detailed Implementation

[0025] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments.

[0026] In related technologies, agent evaluation mainly focuses on end-to-end performance evaluation, such as the GAIA (General AI Assistants) benchmark, or some general evaluations related to security and compliance.

[0027] The inventors discovered that for specific business application agents, such as RAG (Retrieval-augmented Generation) intelligent question answering, complex process calls are often involved. The evaluators in related technologies are difficult to help locate the weak points of the agent's performance and have low reference value for agent improvement.

[0028] To address the aforementioned issues, this disclosure proposes an evaluator generation method, apparatus, and storage medium. This method considers the processing flow of the agent to be evaluated, first determining evaluation metrics, and then generating an evaluator corresponding to those metrics. This achieves an evaluation of the agent's processing flow, improving the comprehensiveness and accuracy of the evaluation. Using the evaluator in the embodiments shown in this disclosure to evaluate the corresponding agent, the generated evaluation results can help developers pinpoint the agent's performance weaknesses, thereby improving the speed of agent development and service quality.

[0029] Flowcharts of some embodiments of the evaluator generation method disclosed herein are as follows: Figure 1 As shown.

[0030] In step S11, descriptive information of the evaluation object is obtained. The evaluation object described in this disclosure can be an intelligent agent. In some embodiments, the descriptive information includes a functional introduction of the evaluation object. When the processing of the evaluation object includes multiple intermediate links (or intermediate modules), the descriptive information includes an introduction to the processing, such as a full-link description of the evaluation object, which includes descriptive information of the functions performed by each intermediate link of the evaluation object.

[0031] In some embodiments, the description information includes evaluation requirements for the evaluation object. These evaluation requirements may match the functional description of the evaluation object; for example, they may include evaluation requirements for intermediate links in the processing of the evaluation object, or requirements for the evaluation object as a whole.

[0032] In some embodiments, users can provide evaluation requirements as needed. For example, users can specify that multiple intermediate links should be evaluated as a whole, or that some or all intermediate links should be evaluated independently. This approach achieves a balance between the evaluator's efficiency and comprehensiveness, meeting the user's needs and increasing the user's control over the evaluator's generation process.

[0033] In some embodiments, the description information may also include an identifier of the evaluation object, such as the name of the evaluation object, so as to identify and distinguish the evaluation objects corresponding to different evaluators, thereby facilitating the accurate use of the evaluator when different evaluation objects exist, and improving the accuracy of the evaluation.

[0034] In step S13, an evaluation metric is determined based on the description information and the first machine learning model. The evaluation metric corresponds to part or all of the processing steps (intermediate links) of the evaluated object. Each evaluation metric acts as a relatively independent evaluation operation, targeting part or all of the intermediate links of the evaluated object. In some embodiments, the evaluation metric can be the name of the evaluator to be generated subsequently, such as a retrieval completeness evaluator or a reordering relevance evaluator.

[0035] For evaluation tasks, a key issue is how to design evaluation metrics. From the perspective of response type, current agents can be divided into generative and numerical responses. For numerical responses, there are standard quantitative judgments of whether they are correct or not. However, agents that respond with generative information lack clear, quantitative metrics. The method in this disclosure can automatically generate evaluation metrics based on descriptive information using a machine learning model, and is suitable for evaluating generative agents.

[0036] The aforementioned first machine learning model can be a model with semantic analysis capabilities, thereby improving the processing efficiency of evaluation metrics for descriptive information. For example, the first machine learning model can be a large language model, thus eliminating the need for targeted training operations. By leveraging the large-scale knowledge reserves and semantic understanding capabilities of the large language model, the implementation difficulty can be reduced and the deployment efficiency improved.

[0037] In some embodiments, such as Figure 2 As shown, determining the evaluation metrics based on the first machine learning model, according to the description information, includes steps 231-233.

[0038] In step 231, the intermediate links of the evaluation object are determined based on the description information, and the number of intermediate links is obtained.

[0039] In step 232, if the number of intermediate links is greater than a predetermined number, step 233 is executed.

[0040] The predetermined quantity is a positive integer greater than 1, for example, 2. For the same evaluation object, the smaller the predetermined quantity, the more detailed the evaluation result of the generated evaluator, and the better the effect of locating defects; the larger the predetermined quantity, the more general the evaluation result of the generated evaluator, but the higher the generation efficiency of the evaluator.

[0041] If the number of intermediate links is less than or equal to the predetermined number, skip step 233 and proceed to step 250, whereby an evaluator is generated.

[0042] In step 233, multiple evaluation metrics are determined based on the first machine learning model. Each evaluation metric corresponds to at least one intermediate link, and multiple evaluation metrics correspond to all intermediate links. This method allows the evaluation object to be broken down into multiple stages for evaluation, achieving evaluation of each stage, avoiding omissions and duplications, and improving the comprehensiveness and accuracy of the evaluation.

[0043] In some embodiments, the first machine learning model can determine the intermediate links of the evaluation object based on the full-link description of the evaluation object; the first machine learning model can determine the evaluation targets for one or more intermediate links based on the evaluation requirements. The first machine learning model integrates the intermediate links and the evaluation targets to generate evaluation metrics.

[0044] The first machine learning model can first determine an evaluation metric for each intermediate link based on the intermediate links, and then merge or split the evaluation metrics for each intermediate link according to the evaluation objectives, and update the evaluation metrics.

[0045] For example, the intermediate links of the evaluation object include three intermediate links: retrieval, sorting, and generation. The evaluation requirements describe that retrieval and sorting should be accurate as a whole, and generation should be accurate and complete. In addition, the evaluation indicators should be updated to meet the requirement of effective collaborative work throughout the entire evaluation object.

[0046] The first machine learning model first determines evaluation metrics for retrieval, ranking, and generation based on the description in the end-to-end specification. Then, it merges retrieval and ranking to determine an evaluation metric for the retrieval and ranking link. For the generation link, it determines two evaluation metrics: generation accuracy and generation completeness. In addition, it evaluates the effectiveness of the entire link's collaborative work.

[0047] This method enables the generation of evaluators from different dimensions, improving the comprehensiveness of the evaluation and meeting the evaluation requirements of different dimensions, thereby increasing the effective utilization rate of information.

[0048] In some embodiments, the analytical capabilities of large language models can be used to automatically generate evaluation indicators. The generated results should cover the capabilities and indicators of the evaluated object, and the generated evaluation indicators should use clear, professional naming to reflect the dimensions and objectives of the evaluation. This method leverages the processing power of large language models to generate evaluation indicators, reducing human intervention and improving processing efficiency.

[0049] In step S15, based on the second machine learning model and according to the description information, an evaluator is generated for each evaluation metric generated in step S13 or step 233. The second machine learning model can be a model with semantic analysis capabilities, thereby improving the processing efficiency of evaluation metrics obtained from the description information. For example, the second machine learning model can be a large language model, thus eliminating the need for targeted training operations and leveraging the large-scale knowledge reserve and semantic understanding capabilities of the large language model to reduce implementation difficulty and improve deployment efficiency. In some embodiments, the first and second machine learning models are the same model, thereby reducing the number of machine learning models required in the evaluator generation process and simplifying device configuration requirements. In some embodiments, based on the second machine learning model, prompt words for each evaluation metric are determined according to the description information. The prompt words include the input requirements of the intermediate link corresponding to the evaluation metric, the output requirements of the intermediate link corresponding to the evaluation metric, the evaluation criteria, the scoring rules, and the output requirements of the evaluator. The evaluator includes the evaluation metric and the prompt words.

[0050] The input and output requirements of the intermediate link can be expressed through textual descriptions and function calls. For example, the input of the intermediate link is: user question {query}, and the output of the intermediate link is: retrieved content {retrieved_context}.

[0051] The evaluation criteria can include multiple categories of standards, along with descriptive information for each category and scoring rules. For example, evaluation criteria might include coverage and comprehensiveness. Coverage descriptions include whether the retrieved content relates to the core elements of the problem, with evaluation rules including 0 = no coverage, 1 = partial coverage, and 2 = complete coverage. Comprehensiveness descriptions include whether multiple relevant perspectives were retrieved, with evaluation rules including 0 = single perspective, 1 = partially multiple perspectives, and 2 = comprehensive multiple perspectives. This method allows for the generation of scores for each category of standards based on the evaluation criteria, improving the intuitiveness of the evaluator's output.

[0052] The scoring rules include rules for summarizing scores from multiple categories, such as weighted averages or score conversions. This approach allows for the aggregation of evaluation results from different perspectives, building upon individual assessments and further enhancing the intuitiveness of the output results.

[0053] The evaluator's output requirements include both content and format requirements. For example, the output content includes a score and evaluation reason, output in a format such as {{"score": score, "reason": "detailed evaluation reason"}}. Detailed evaluation reasons can be generated based on the evaluation results for each category in the evaluation criteria. This approach not only provides users with intuitive evaluation scores but also facilitates analysis of the reasons behind those scores, further helping users pinpoint performance weaknesses, improve the effective utilization of information, and increase the efficiency of improvement for the evaluated object.

[0054] Based on the methods in the embodiments shown above, the knowledge reserves and semantic understanding capabilities of machine learning models can be utilized to break down the evaluation of the evaluation object into multiple evaluation indicators, and then an evaluator can be generated for each evaluation indicator, thereby improving the comprehensiveness and accuracy of the evaluation. Using the evaluator in the embodiments shown in this disclosure to evaluate the corresponding evaluation object, the generated evaluation results can help developers locate the performance weaknesses of the intelligent agent, which is beneficial to improving the development speed and service quality of the intelligent agent.

[0055] In some embodiments, the evaluator generation method of this disclosure further includes: when the number of intermediate links is less than or equal to a predetermined number, simultaneously generating evaluation metrics and prompt words based on a third machine learning model according to description information, thereby obtaining an evaluator. For example... Figure 2 As shown, if the number of intermediate links is determined to be less than or equal to a predetermined number in step 232, the step of generating evaluation metrics is skipped.

[0056] In step 250, if the preceding step is step 233, then as shown in step S15 above, based on the second machine learning model, prompt words corresponding to each evaluation indicator are generated according to the description information and the evaluation indicators obtained in step 233; if the preceding step is step 232, then the evaluation indicators and prompt words are generated simultaneously based on the third machine learning model, thereby obtaining the evaluator. The number of evaluation indicators (evaluators) generated by the third machine learning model can be one or more.

[0057] This method reduces the number of model calls and skips the operation of generating evaluation metrics when the processing of the evaluation object is relatively simple. It generates an evaluator including evaluation metrics and prompt words in a single call, thereby improving processing efficiency without affecting the performance of the evaluator.

[0058] The aforementioned third machine learning model can be a model with semantic analysis capabilities, thereby improving the efficiency of processing evaluation metrics for descriptive information. For example, the first machine learning model can be a large language model, thus eliminating the need for targeted training operations. By leveraging the large-scale knowledge reserves and semantic understanding capabilities of the large language model, the implementation difficulty can be reduced and the deployment efficiency improved.

[0059] In some embodiments, based on the evaluator generated in step S15, a machine learning model, such as a large language model, is used to evaluate the evaluation object, and the resulting output is then analyzed and processed by experts. Further, such as... Figure 1 As shown, the evaluator generation method of this disclosure further includes steps S17 and S19.

[0060] In step S17, feedback from the evaluator is obtained. In some embodiments, the feedback may include feedback on evaluation metrics, such as adding, deleting, or modifying them. In some embodiments, the feedback may include feedback on prompt words, such as adding, deleting, or modifying them.

[0061] In step S19, at least one of the evaluation metrics or evaluator prompts is updated based on the feedback.

[0062] In some embodiments, to avoid bias from individual cases, the opinions of experts on multiple cases are first summarized to remove some inconsistent opinions, and then at least one of the evaluation indicators or evaluator prompts is updated based on the summary results.

[0063] Taking feedback on prompt words as an example, firstly, the consistent feedback from experts is summarized, and the inconsistent parts are removed to obtain a summary of opinions. Further, this summary of opinions is integrated with the prompt words generated in step S15.

[0064] In some embodiments, the opinion summary can be directly appended to the prompt words generated by the original evaluator, improving processing efficiency. In some embodiments, based on the fourth machine learning model, the prompt words of the evaluator are updated according to the opinion summary and the prompt words of the evaluator generated in step S15, thereby obtaining more complete and more readable prompt words. The aforementioned fourth machine learning model can be a large language model, thereby leveraging the large-scale knowledge reserves and semantic understanding capabilities of the large language model to reduce implementation difficulty and improve deployment efficiency.

[0065] In some embodiments, if the feedback includes comments on the evaluation metrics, then as follows: Figure 2As shown, based on the summary of opinions, the process returns to step 233, where the evaluation indicators are updated using the opinions in the summary. Then, in step 250, an evaluator is generated for the updated evaluation indicators. This method allows for improvement at the evaluation indicator level based on expert opinions, enhancing the evaluator's alignment with user needs. In some embodiments, the processes of generating, using, and updating the evaluator based on feedback can be performed in multiple rounds. Through iterative feedback and evaluator updates, the evaluator is fine-tuned as data accumulates, further improving its alignment with the evaluation object and user needs, and enhancing the comprehensiveness and accuracy of the evaluation.

[0066] Based on the method in the embodiments shown above, it is possible to achieve automatic generation of human-machine collaborative evaluators. On the basis of automatically generated evaluators, combined with a small amount of expert feedback, continuous iterative optimization can be carried out to obtain business-approved and standardized evaluators, thereby improving the performance of the evaluators.

[0067] The following section uses a business knowledge question-answering agent as an example to introduce the evaluator generation method disclosed in this publication.

[0068] 1. Input the agent's description information. The description information includes a basic description of the agent, a complete description of the agent's lifecycle, and a brief description of the evaluation requirements.

[0069] Basic description of the intelligent agent: a business knowledge question-answering intelligent agent.

[0070] Full-link description: This agent is a RAG-type agent. After the user inputs a question, it retrieves relevant content from the knowledge base, rearranges it, and then generates an answer based on the relevant content using a large model.

[0071] Evaluation Requirements: Evaluate the business knowledge question-answering AI agent by designing comprehensive evaluation metrics to assess the agent's overall performance, including three stages: retrieval, reordering, and generation. The retrieval results should comprehensively retrieve relevant content. The reordering stage should further select the most relevant content set. The generation stage should generate comprehensive, accurate, and user-friendly answers based on the reordering results.

[0072] 2. Based on the input agent description, a large model is used to generate an evaluator set, including evaluator names, evaluator prompts, and input parameters. Depending on the problem complexity, if there are few intermediate links, evaluators are generated directly. If there are many intermediate links, to improve performance, they are generated in steps: first, evaluation metrics (i.e., evaluator names) are generated, and then prompts are generated for each metric.

[0073] Based on the above requirements, the commands for calling the large model are as follows. These commands are merely examples and do not constitute an undue limitation on this disclosure.

[0074] You are a professional AI evaluator design expert. Please design multiple high-quality evaluators for the "Business Knowledge Question Answering Agent".

[0075] ## Agent Information

[0076] - Basic description: { request.agent_name}

[0077] - Full-chain description: {request.agent despy}

[0078] - Evaluation requirement: {frequest.generate_desp}

[0079] ##Design Requirements

[0080] ### 1. Number of evaluators

[0081] Please generate multiple evaluators with different dimensions, covering the agent's core capabilities and key metrics.

[0082] ### 2. Evaluator Naming

[0083] - Use clear and professional naming conventions, such as "Accuracy Evaluator," "Logical Reasoning Evaluator," and "Verbal Fluency Evaluator."

[0084] - The name should directly reflect the dimensions and objectives of the assessment.

[0085] ### 3. Cue Word Design

[0086] - Use Python string formatting syntax: {{variable name}} represents the parameter to be filled in.

[0087] -The prompt should include:

[0088] * Clear evaluation criteria and dimensions

[0089] *Specific scoring rules (e.g., 0, 1, 2-point scale)

[0090] *Detailed evaluation steps and considerations

[0091] *Output format requirements

[0092] ### 4. Input Field Design

[0093] - Design reasonable input parameters based on the characteristics of the intelligent agent.

[0094] - Commonly used fields: query (user question), answer (agent answer), context, etc.

[0095] - Field names should be in English, and descriptions should be in Chinese.

[0096] ### 5. Output Field Specifications

[0097] - score: rating (0, 1, or 2 points, where 0 is the worst and 2 is the best)

[0098] - Reason: Detailed evaluation reasons and analysis

[0099] ###Please ensure:

[0100] - Each evaluator has unique evaluation dimensions.

[0101] - Detailed and actionable prompts

[0102] - The input field name is exactly the same as the variable name in the prompt.

[0103] - Standardized output format

[0104] ## Output Format

[0105] Please return the JSON in strict accordance with the following format:

[0106] {{

[0107] "evaluators":[

[0108] {{"name": "Specific evaluator name",

[0109] "prompt": "You are a professional evaluation expert. Please conduct an evaluation of the following: User issue: {{query}}"

[0110] "input_fields": {{

[0111] "query": "user question"

[0112] "answer": The agent's response.

[0113] }},

[0114] "output_flelds": {{

[0115] "score": "score (a floating-point number between 0 and 1)"

[0116] "reason": "Detailed evaluation reasons and analysis"

[0117] }}

[0118] Based on the above input examples, the evaluators are shown in Tables 1 to 6.

[0119] Table 1 Examples of evaluators

[0120]

[0121] Table 2 Examples of evaluators

[0122]

[0123] Table 3 Examples of evaluators

[0124]

[0125] Table 4 Examples of Evaluators

[0126]

[0127] Table 5 Examples of Evaluators

[0128]

[0129] Table 6 Examples of Evaluators

[0130]

[0131] In the examples above, the expressions in curly braces "{}" represent content that needs to be called during the process. For example, during the generation of the evaluator, the user's basic description, end-to-end description, and evaluation requirements will be called through {request.agent_name}, {request.agent despy}, and {frequest.generate_desp}, respectively. Similarly, during the evaluation of the agent using the evaluator, {query}, {knowledge_base}, {retrieved_context}, {reranked_context}, and {answer} will be called to obtain the user's question, knowledge base content, retrieved content, reranked content, and agent's answer, respectively.

[0132] The above generated results are merely examples and do not constitute an undue limitation on this disclosure.

[0133] Based on the method disclosed herein, it is convenient to evaluate complex business intelligent agents, especially suitable for evaluating generation-type tasks, and can obtain the specific performance of the entire link module of the intelligent agent. The method is easy to implement and highly efficient, and is suitable for evaluation scenarios with cold start.

[0134] In some embodiments, the evaluator generation apparatus of this disclosure is such as Figure 3 As shown in the image.

[0135] The information acquisition unit 31 is capable of acquiring descriptive information about the evaluation object. In some embodiments, the information acquisition unit 31 can perform the method in any embodiment of step S11 above.

[0136] Processing unit 32 is capable of determining evaluation metrics based on a first machine learning model according to the description information. In some embodiments, processing unit 32 is capable of performing the methods in any of the embodiments of steps S13 above and steps 231-233.

[0137] The generation unit 33 is capable of generating an evaluator for each evaluation metric based on a second machine learning model and description information. In some embodiments, the generation unit 33 can execute the method in any embodiment of step S15 above. In some embodiments, the generation unit 33 can also, when the processing unit 32 determines that the number of intermediate links is less than or equal to a predetermined number, synchronously generate evaluation metrics and prompt words based on a third machine learning model and description information, thereby improving the evaluator generation efficiency for simple evaluation objects.

[0138] Based on the apparatus in the embodiments shown above, the knowledge reserves and semantic understanding capabilities of machine learning models can be utilized to break down the evaluation of the evaluation object into multiple evaluation indicators, and then an evaluator can be generated for each evaluation indicator, thereby improving the comprehensiveness and accuracy of the evaluation. Using the evaluator in the embodiments shown in this disclosure to evaluate the corresponding intelligent agent, the generated evaluation results can help researchers locate the performance weaknesses of the intelligent agent, which is beneficial to improving the development speed and service quality of the intelligent agent.

[0139] In some embodiments, such as Figure 3 As shown, the evaluator generation apparatus further includes an update unit 34, capable of acquiring feedback on the evaluator and updating at least one of the evaluation metrics or the evaluator's prompts based on the feedback. In some embodiments, the feedback may include feedback on the evaluation metrics, such as adding, deleting, or modifying them. In some embodiments, the feedback may include feedback on the prompts, such as adding, deleting, or modifying them.

[0140] In some embodiments, to avoid bias from individual cases, the opinions of experts on multiple cases are first summarized to remove some inconsistent opinions, and then at least one of the evaluation indicators or evaluator prompts is updated based on the summary results.

[0141] Taking feedback on prompt words as an example, the consensus feedback from experts is first summarized, and the inconsistent parts are removed to obtain a summary of opinions. Further, this summary of opinions is integrated with the prompt words generated by generation unit 33. In some embodiments, the summary of opinions can be directly appended to the prompt words generated by the original evaluator, improving processing efficiency. In some embodiments, based on the fourth machine learning model, the prompt words of the evaluator are updated according to the summary of opinions and the prompt words generated by the evaluator in generation unit 33, resulting in more complete and readable prompt words. The aforementioned fourth machine learning model can be a large language model, thereby leveraging the large-scale knowledge reserves and semantic understanding capabilities of large language models to reduce implementation difficulty and improve deployment efficiency.

[0142] In some embodiments, if the feedback includes opinions on the evaluation indicators, the evaluation indicators can be updated based on the summary of opinions. Then, an evaluator can be generated in generation unit 33 based on the updated evaluation indicators. This allows for improvement at the evaluation indicator level based on expert opinions, enhancing the evaluator's fit with user needs. In some embodiments, the operations of generating the evaluator, using the evaluator, and updating the evaluator based on feedback can be performed in multiple rounds. Through iterative feedback and evaluator updates, as data accumulates, the evaluator can be fine-tuned, further improving its fit with the evaluation object and user needs, and enhancing the comprehensiveness and accuracy of the evaluation.

[0143] Based on the apparatus in the embodiments shown above, it is possible to continuously iterate and optimize the automatically generated evaluator, combined with a small amount of expert feedback, to obtain a business-approved and standardized evaluator, thereby improving the performance of the evaluator.

[0144] A schematic diagram of an embodiment of the evaluator generation apparatus disclosed herein is shown below. Figure 4 As shown, the evaluator generation apparatus includes a memory 401 and a processor 402. The memory 401 can be a disk, flash memory, or any other non-volatile storage medium. The memory stores instructions from the corresponding embodiments of the evaluator generation method described above. The processor 402 is coupled to the memory 401 and can be implemented as one or more integrated circuits, such as a microprocessor or microcontroller. The processor 402 executes the instructions stored in the memory, thereby improving the comprehensiveness and accuracy of the agent evaluator.

[0145] In one embodiment, it can also be as follows: Figure 5As shown, the evaluator generation apparatus 500 includes a memory 501 and a processor 502. The processor 502 is coupled to the memory 501 via a BUS bus 503. The evaluator generation apparatus 500 can also be connected to an external storage device 505 via a storage interface 504 to access external data, and can also be connected to a network or another computer system (not shown) via a network interface 506. Further details are omitted here.

[0146] In this embodiment, storing data instructions in a memory and then processing those instructions with a processor can improve the comprehensiveness and accuracy of the agent evaluator.

[0147] In another embodiment, a computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the steps of the method in the corresponding embodiment of the evaluator generation method. Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0152] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0153] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them; although this disclosure has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of this disclosure or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of this disclosure, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in this disclosure.

Claims

1. An evaluator generation method, comprising: Obtain descriptive information about the evaluation object; Based on the described information, evaluation metrics are determined using a first machine learning model. Based on the second machine learning model, an evaluator is generated for each evaluation metric according to the described information.

2. The evaluator generation method according to claim 1, wherein, The step of determining the evaluation index based on the first machine learning model according to the described information includes: Based on the described information, determine the intermediate links of the evaluation object; If the number of intermediate links is greater than a predetermined number, multiple evaluation metrics are determined based on a first machine learning model, wherein each evaluation metric corresponds to at least one intermediate link, and the multiple evaluation metrics correspond to all the intermediate links.

3. The evaluator generation method according to claim 1 or 2, wherein, The descriptive information includes a complete description of the evaluation object and evaluation requirements.

4. The evaluator generation method according to claim 3, wherein, The step of determining the evaluation index based on the first machine learning model according to the described information includes: Based on the full-link description of the evaluation object, the intermediate links of the evaluation object are determined based on the first machine learning model; Based on the evaluation requirements, evaluation targets for one or more of the intermediate links are determined using a first machine learning model; The evaluation metrics are generated based on the intermediate link and the evaluation objective, using a first machine learning model.

5. The evaluator generation method according to claim 1, wherein, The generation of evaluators for each evaluation metric based on the second machine learning model includes: Based on the second machine learning model, prompt words for each evaluation indicator are determined according to the description information. The prompt words include the input requirements of the intermediate link corresponding to the evaluation indicator, the output requirements of the intermediate link corresponding to the evaluation indicator, the evaluation criteria, the scoring rules, and the output requirements of the evaluator. The evaluator includes the evaluation indicator and the prompt words.

6. The evaluator generation method according to claim 2, further comprising: If the number of intermediate links is less than or equal to the predetermined number, an evaluation index and a prompt word are generated based on the description information using a third machine learning model, wherein the evaluator includes the evaluation index and the prompt word.

7. The evaluator generation method according to claim 1, further comprising: Get feedback; The evaluator's prompts are updated based on the feedback received.

8. The evaluator generation method according to claim 7, wherein, The prompts for updating the evaluator based on the feedback include: Add the feedback to the corresponding evaluator's prompt; or Based on the fourth machine learning model, the prompts of the evaluator are updated according to the feedback and the prompts of the evaluator.

9. An evaluator generation apparatus, comprising: The information acquisition unit is configured to acquire descriptive information about the evaluation object; The processing unit is configured to determine an evaluation metric based on the description information and a first machine learning model. The generation unit is configured to generate an evaluator for each evaluation metric based on the second machine learning model and the description information.

10. The evaluator generation apparatus according to claim 9, further comprising: The update unit is configured to obtain feedback and update the evaluator's prompts based on the feedback.

11. An evaluator generation apparatus, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the evaluator generation method as described in any one of claims 1 to 8 based on instructions stored in the memory.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the evaluator generation method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the evaluator generation method according to any one of claims 1 to 8.

Images