Universal agent evaluation method, device, equipment, medium and product

By using a large language model to evaluate the basic parameters of an agent from multiple dimensions, this approach solves the problems of insufficient accuracy and lack of universality in existing agent evaluation technologies, and achieves accurate, efficient, and universal evaluation of agent capabilities.

CN121301150APending Publication Date: 2026-01-09WUHAN AI RES
0 Cites 5 Cited by

Patent Information

Application Number
CN202511378226.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-25
Publication Date
2026-01-09

AI Technical Summary

Technical Problem

Existing intelligent agent evaluation methods lack accuracy and universality. Human subjective evaluation cannot be scaled up, and automated evaluation based on rules or simple indicators cannot accurately assess the multi-dimensional capabilities of intelligent agents.

Method used

The basic parameters of the intelligent agent are evaluated using a large language model. The evaluation is based on five dimensions: long-term memory accuracy, task arrangement rationality, character consistency, tool call effectiveness, and content compliance. A corresponding evaluation set is constructed and scored, including the generation of evaluation tasks using retrieval enhancement generation algorithms and test case generation algorithms.

Benefits of technology

It achieves accurate and efficient evaluation of intelligent agents, can evaluate the capabilities of intelligent agents in multiple dimensions, and has universality and versatility, making it suitable for evaluating different intelligent agents.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a universal agent evaluation method, device and equipment, a medium and a product, and belongs to the technical field of agent evaluation, and the method comprises the steps: obtaining basic parameters of an agent; based on the basic parameters, the agents are evaluated in five dimensions of long-term memory accuracy, task arrangement rationality, human setting consistency, tool calling effectiveness and content compliance, so that evaluation of the agents is achieved; wherein in the process of evaluating the intelligent agent in any one of the five dimensions, a corresponding evaluation set is constructed based on the basic parameters, the large language model is adopted to score the output of the intelligent agent on the evaluation set, and the score of the intelligent agent in the dimension is obtained. According to the method, the intelligent agent can be evaluated universally and accurately.
Need to check novelty before this filing date? Find Prior Art