Intelligent agent evaluation method and device and storage medium
By obtaining the execution information of the agent's task execution task, combining preset process and result information, the task execution process and results of the agent are evaluated, and the problem of inaccurate evaluation of the agent in the existing technology is solved, and the comprehensiveness of the agent is achieved , accurate assessment.
Patent Information
- Application Number
- CN202411409613.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology lacks systematic agent evaluation methods, which limits the application and optimization of agents, making it difficult to accurately understand the operation effect of agents.
By obtaining the execution information of the agent's tasks, evaluating and processing based on the preset process information and result information, combining business needs and semantic needs, a comprehensive evaluation of the agent's task execution process and results are generated to generate target evaluation results.
Accurate and reliable evaluation of the agent is achieved, and the task execution effect of the agent can be comprehensively judged based on multiple dimensions, improving the accuracy and reliability of the evaluation.
Smart Images

Figure CN118916662A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an intelligent agent evaluation method, device and storage medium. Background Art
[0002] With the rapid development of artificial intelligence technology, especially the continuous breakthroughs in deep learning and big data analysis technology, large models have become an effective way to realize intelligent agents. These intelligent agents can be widely used in many fields such as medicine, finance, and education, aiming to support decision-making by simulating and predicting complex interactions and changes in the industry.
[0003] Although the computing power and data processing capabilities of large models have reached historical highs, there are few standards and models to follow for systematically evaluating and verifying the performance of these agents.
[0004] At present, the lack of detailed evaluation methods not only limits the application and improvement of intelligent agents, but also restricts the further intelligentization of related industries. Therefore, there is a lack of an accurate intelligent agent evaluation method to effectively understand the operation effect of intelligent agents, and then adjust and optimize the design and operation of intelligent agents. Summary of the invention
[0005] The present application at least provides an intelligent agent evaluation method, apparatus, device and computer-readable storage medium.
[0006] The first aspect of the present application provides an intelligent agent evaluation method, including: obtaining execution information obtained by the intelligent agent when executing a target task, the execution information including execution process information and execution result information; performing process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; performing result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; and determining a target evaluation result of the intelligent agent based on the first evaluation result and the second evaluation result.
[0007] In one embodiment, the preset process information includes business requirements and semantic requirements preset in the process of the intelligent agent performing a task, the first evaluation result includes a business evaluation result and a semantic evaluation result, and the step of performing process evaluation processing on the execution process information according to the preset process information to obtain the first evaluation result includes: evaluating and processing the execution process information according to the business requirements to obtain the business evaluation result; evaluating and processing the execution process information according to the semantic requirements to obtain the semantic evaluation result.
[0008] In one embodiment, the execution process information includes subtasks split during the process of the agent executing the target task, and the step of evaluating and processing the execution process information according to the business requirements to obtain the business evaluation result includes: comparing the task content of each subtask with the preset subtask content in the business requirements to obtain the completeness of the task content; and / or, comparing the execution order between each subtask with the preset subtask execution order in the business requirements to obtain the rationality of the task sequence; and / or, comparing the business characteristics of each subtask with the preset business characteristics in the business requirements to obtain the task relevance; and / or, comparing each subtask with the reference subtask corresponding to the target task to obtain the task accuracy; determining the business evaluation result based on at least one of the completeness of the task content, the rationality of the task sequence, the task relevance and the task accuracy.
[0009] In one embodiment, the step of determining the business evaluation result based on at least one of the task content completeness, the task sequence rationality, the task relevance and the task accuracy includes: selecting target data from the task content completeness, the task sequence rationality, the task relevance and the task accuracy; performing weighted sum processing on the target data and the data weight corresponding to the target data to obtain the business evaluation result.
[0010] In one embodiment, the step of evaluating the execution process information according to the semantic requirements to obtain the semantic evaluation result includes: obtaining the target semantic features obtained by the agent in the process of executing the target task, comparing the target semantic features with the preset semantic features to obtain the semantic understanding accuracy; and / or, performing fluency detection processing on the natural language text generated by the agent to obtain semantic representation fluency; and / or, performing text matching processing between the natural language text generated by the agent and preset keywords to obtain text information richness; and / or, performing coherence detection processing on the natural language text generated by the agent to obtain text coherence; determining the semantic evaluation result according to at least one of the semantic understanding accuracy, the semantic representation fluency, the text information richness and the text coherence.
[0011] In one embodiment, the step of determining the semantic evaluation result based on at least one of the semantic understanding accuracy, the semantic representation fluency, the text information richness and the text coherence includes: selecting target data from the semantic understanding accuracy, the semantic representation fluency, the text information richness and the text coherence; performing weighted sum processing on the target data and the data weight corresponding to the target data to obtain the semantic evaluation result.
[0012] In one embodiment, the step of performing result evaluation processing on the execution result information according to the preset result information to obtain a second evaluation result includes: performing feature comparison processing on the preset result information and the execution result information to obtain a feature matching degree; and determining the second evaluation result according to the feature matching degree.
[0013] In one embodiment, the step of determining the target evaluation result of the intelligent agent based on the first evaluation result and the second evaluation result includes: obtaining the evaluation weight of the first evaluation result and the evaluation weight of the second evaluation result; performing weighted summation processing based on the first evaluation result, the second evaluation result and the corresponding evaluation weights to obtain the target evaluation result.
[0014] The second aspect of the present application provides an intelligent agent evaluation device, including: an acquisition module, used to obtain execution information obtained by the intelligent agent when executing a target task, the execution information including execution process information and execution result information; a first evaluation module, used to perform process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; a second evaluation module, used to perform result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; and a comprehensive evaluation module, used to determine the target evaluation result of the intelligent agent based on the first evaluation result and the second evaluation result.
[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned intelligent agent evaluation method.
[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned intelligent agent evaluation method when the program instructions are executed by a processor.
[0017] The above scheme obtains the execution information obtained by the intelligent agent when executing the target task, can analyze the task execution process of the intelligent agent according to the execution process information in the execution information, and analyze the task execution result of the intelligent agent through the execution result information in the execution information; performs process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; performs result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; comprehensively evaluates the task execution process of the intelligent agent and the task execution result of the intelligent agent, thereby obtaining the target evaluation result of the intelligent agent and realizing accurate and reliable evaluation of the intelligent agent.
[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0020] Figure 1 is a flowchart of an exemplary embodiment of the agent evaluation method of the present application; Figure 2 It is a schematic diagram of the result verification interface of the intelligent agent evaluation system in the intelligent agent evaluation method of the present application; Figure 3 is a block diagram of an intelligent agent evaluation device shown in an exemplary embodiment of the present application; Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0023] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.
[0024] For ease of understanding, one of the applicable scenarios of this application is now exemplified. The intelligent agent of this application, also known as industry intelligent agents, refers to an intelligent system with the ability to combine large models. It mainly uses advanced machine learning and artificial intelligence technologies to extract and analyze industry-related information and knowledge from large amounts of data. Unlike traditional industry intelligent systems, industry intelligent agents can not only perform basic data analysis and prediction, but also use large models for more advanced cognition and reasoning, thereby generating more accurate and comprehensive industry insights and decision-making recommendations.
[0025] The intelligent agent evaluation method of the present application can be applied to an intelligent agent evaluation system, in which a large model can be preset for evaluating the intelligent agent, and the intelligent agent evaluation system can be set in a terminal device or server connected to the intelligent agent for communication, or can be directly set in the intelligent agent, which is not limited here.
[0026] See also Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the agent evaluation method of the present application. Specifically, it may include the following steps: Step S110, obtaining execution information obtained by the agent executing the target task, the execution information includes execution process information and execution result information.
[0027] It should be noted that the target task can be preset and sent to the agent in advance, and the agent executes the target task to obtain execution information. The execution information may include the execution process information obtained by the agent in the process of executing the target task and the execution result information obtained by the agent after executing the target task. It can be understood that the execution process information represents the execution process of the target task by the agent, and the execution result information represents the execution result of the target task by the agent.
[0028] For example, for common target tasks, they can usually be some questions and / or instructions issued to the agent, that is, the target task can be represented in natural language and / or in computer language, which will not be described here. Then the agent will perform task planning and task execution according to the received target task (question) to obtain the final execution result (answer to the question).
[0029] For example, if the target task currently received by the agent is "planning a travel route from area A to area B", the agent may split the target task planning into subtasks such as "querying transportation methods", "comparing the time and cost of different transportation methods", "selecting the best transportation method", and "booking tickets or air tickets". After the agent completes each subtask, it generates the corresponding task execution results, such as "selecting high-speed rail as a means of transportation", "booking a second-class seat ticket for the G series high-speed rail", etc. These task execution results can be used for subsequent agent effect evaluation.
[0030] Therefore, the agent evaluation method of the present application can be evaluated by obtaining the agent's task planning steps, intermediate results of execution, and final task execution results according to the target task, so as to comprehensively cover all operating dimensions of the agent. Secondly, by designing a reasonable evaluation scheme, the rationality of the execution results of the agent's task planning steps and intermediate steps can be autonomously evaluated based on the large model. Then, from the perspective of information elements, a comparison and verification can be performed based on the given reference information elements and the information elements answered by the model. Combined with specific evaluation indicators, a comprehensive evaluation of the agent's task execution results is performed. Finally, it is also possible to propose an interactive system design for agent effect evaluation to achieve auxiliary manual secondary verification and improve the accuracy of the evaluation of the overall effect of the agent.
[0031] Step S120: performing process evaluation on the execution process information according to the preset process information to obtain a first evaluation result.
[0032] In combination with the above steps, in order to ensure the accuracy of the evaluation of the intelligent agent, the intelligent agent evaluation method of the present application can evaluate the task execution process and task execution results of the intelligent agent.
[0033] Therefore, in this step, the execution process information can be evaluated according to the preset process information to obtain a first evaluation result, that is, an evaluation result of the task execution process of the intelligent agent.
[0034] Step S130, performing result evaluation processing on the execution result information according to the preset result information to obtain a second evaluation result.
[0035] In conjunction with the above steps, in this step, the execution result information can be evaluated according to the preset result information to obtain a second evaluation result, that is, an evaluation result of the task execution result of the intelligent agent.
[0036] It should be noted that the evaluation and processing method in the present application may include but is not limited to a threshold comparison method based on preset logical judgment, a conditional judgment method, etc., and / or a model evaluation method based on a neural network, etc., which will not be elaborated here.
[0037] Step S140, determining a target evaluation result of the agent according to the first evaluation result and the second evaluation result.
[0038] In conjunction with the above steps, the method for determining the target evaluation result in the present application may include, but is not limited to, a method of making a single judgment based on any one of the first evaluation result and the second evaluation result, or a method of making a comprehensive judgment based on the first evaluation result and the second evaluation result. Preferably, a comprehensive judgment method can be selected to perform data fusion processing on the first evaluation result and the second evaluation result, so that a comprehensive evaluation result of the intelligent agent can be obtained, that is, the target evaluation result. A comprehensive judgment method based on multiple dimensions can more accurately characterize the task execution effect of the intelligent agent.
[0039] Among them, the data fusion processing method of the present application may include but is not limited to summation processing, weighted summation processing, etc. For weighted summation processing, a first evaluation weight corresponding to the first evaluation result and a second evaluation weight corresponding to the second evaluation result may be preset. It should be noted that the present application does not limit the size relationship between the first evaluation weight and the second evaluation weight, which can be adaptively set according to the actual application scenario, which will not be elaborated here.
[0040] It can be seen that the present application can analyze the task execution process of the intelligent agent according to the execution process information in the execution information, and analyze the task execution result of the intelligent agent according to the execution result information in the execution information by acquiring the execution information obtained when the intelligent agent executes the target task; perform process evaluation processing on the execution process information according to the preset process information to obtain a first evaluation result; perform result evaluation processing on the execution result information according to the preset result information to obtain a second evaluation result; and comprehensively evaluate the task execution process of the intelligent agent and the task execution result of the intelligent agent, thereby obtaining the target evaluation result of the intelligent agent and realizing accurate and reliable evaluation of the intelligent agent.
[0041] Based on the above embodiments, the embodiment of the present application describes the steps of performing process evaluation processing on the execution process information according to the preset process information to obtain the first evaluation result. The preset process information includes the business requirements and semantic requirements preset in the process of the agent performing the task, and the first evaluation result includes the business evaluation result and the semantic evaluation result. Specifically, the method of this embodiment includes the following steps: The execution process information is evaluated and processed according to business requirements to obtain business evaluation results; the execution process information is evaluated and processed according to semantic requirements to obtain semantic evaluation results.
[0042] In conjunction with the above-mentioned embodiments, in the actual application scenario of the intelligent agent, in addition to ensuring that the intelligent agent can accurately perform the target task, it also involves the need to interpret and generate natural language. Therefore, in the intelligent agent evaluation method of the present application, when evaluating the intelligent agent's task execution process, it includes evaluating the business needs and semantic needs involved in the intelligent agent's task execution process.
[0043] Specifically, the execution process information is evaluated and processed according to the business requirements to obtain the business evaluation result of the agent, thereby indicating whether the agent meets the preset business requirements during the task execution. The execution process information is evaluated and processed according to the semantic requirements to obtain the semantic evaluation result of the agent, thereby indicating whether the agent meets the preset semantic requirements during the task execution, which is equivalent to indicating whether the agent can accurately understand the semantic information contained in the target task and can accurately generate text information.
[0044] It is understandable that in the special case of an agent performing a task, the agent may not be able to accurately understand the semantic information of the target task, but may accidentally obtain and output the correct answer. However, such cases are very rare and one-sided. Therefore, in the agent evaluation method of this application, it is necessary to evaluate not only whether the agent can meet the business needs, but also whether the agent meets the semantic needs (semantic understanding needs and semantic generation needs, etc.).
[0045] Optionally, for some agents that do not involve natural language processing, during the process of agent evaluation, the evaluation method of the present application may further include: determining whether the target task received by the agent includes natural language requirements (for example, the target task received by the agent includes natural language data or the target task received by the agent requires output of natural language data, etc.). If so, the steps of business evaluation processing and semantic evaluation processing in this embodiment are enabled; if not, the steps of semantic evaluation processing in this embodiment may be suspended, and only the steps of business evaluation processing are enabled.
[0046] Based on the above embodiments, the embodiment of the present application describes the steps of evaluating and processing the execution process information according to business needs to obtain the business evaluation results. Among them, the execution process information includes the subtasks obtained by splitting the process of the intelligent agent performing the target task. Specifically, the method of this embodiment includes the following steps: Compare the task content of each subtask with the preset subtask content in the business requirements to obtain the completeness of the task content; and / or, compare the execution order between each subtask with the preset subtask execution order in the business requirements to obtain the rationality of the task sequence; and / or, compare the business characteristics of each subtask with the preset business characteristics in the business requirements to obtain the task relevance; and / or, compare each subtask with the reference subtask corresponding to the target task to obtain the task accuracy; determine the business evaluation result based on at least one of the completeness of the task content, the rationality of the task sequence, the task relevance and the task accuracy.
[0047] With reference to the foregoing embodiments, in the business demand assessment process of the present application, at least one of the task content completeness, task sequence rationality, task relevance, and task accuracy is mainly assessed to determine the business assessment result.
[0048] It should be noted that when the agent receives the target task (problem), it usually plans the overall target task into several subtasks, executes each subtask to obtain the task execution results of each subtask, and finally obtains the task execution results of the target task. Therefore, this application uses the process of the agent executing the target task in steps as the basis for effect evaluation. Only after clarifying the agent's planning method for the task and the results of executing each subtask, can the agent's execution effect be further accurately evaluated.
[0049] Specifically, when the agent receives one or more target tasks, it will plan the overall target task into multiple executable subtasks according to its own preset related algorithms and models. Among them, agents applied to different industries may have different related algorithms and models, depending on the actual application scenario of the agent, which will not be elaborated here.
[0050] Exemplarily, the task planning methods adopted by the intelligent agent in the present application may include but are not limited to ReAct, ReWOO, etc. Among them, ReAct is a method for enhancing large language models, which can enhance the effects of reasoning and decision-making by combining reasoning (Reasoning) and action (Acting). ReWOO (Reasoning WithOut Observation) can solve the redundancy problem of ReAct, reduce the use of tokens, and in more complex situations, it can show better robustness than frameworks such as ReAct. These methods have certain effects in realizing the task planning of intelligent agents, but the technical solution of this application is not limited to specific task planning methods. Regardless of the method used for task planning, the business requirement is mainly to ensure that the planned subtasks can clearly and effectively reflect the core requirements of the overall task, thereby providing accurate basic data for subsequent effect evaluation.
[0051] When the agent starts planning and executing the received target task, it can obtain the corresponding execution process information (such as the subtasks planned by the agent, the execution order between subtasks, etc.). When the agent executes task planning, different business scenarios may have different requirements and emphases on the planning steps. Therefore, the primary task of this embodiment is to determine the key task planning evaluation dimensions according to the specific business scenario and obtain business requirements. For example, dimensions related to the business scenario may include but are not limited to the completeness, order rationality, relevance, and accuracy of the task planning.
[0052] Among them, the task content completeness evaluation of this embodiment is mainly to ensure that the target task is planned into various subtasks as comprehensively and without omission as possible, and the execution of each subtask by the agent can fully meet the needs represented by the target task. Specifically, it can be determined by comparing whether the various subtasks planned by the agent match the complete preset task framework (one of the data representations of business needs), for example, whether the various subtasks planned by the agent are the same as the various preset subtasks in the preset task framework, and the task content completeness is determined according to the ratio of the number of identical tasks in each subtask planned by the agent and each preset subtask in the preset task framework (for example, there are 5 preset subtasks in the preset task framework, and there are 4 subtasks in the subtasks planned by the agent that are the same as the preset subtasks, then the task content completeness of the agent can be 80%). Among them, whether the subtasks and the preset subtasks match the same can be determined based on the relevant data involved in the task, the semantic understanding of the task, the number of tasks, etc., which are not limited here.
[0053] The evaluation of the rationality of the task sequence in this embodiment is mainly to check whether the logical relationship between the subtasks planned by the agent is reasonable, so as to ensure the rationality, fluency and efficiency of the task execution. Specifically, the rationality of the task sequence can be obtained by comparing the execution order of each subtask by the agent and the preset subtask execution order in the business requirements. Among them, the preset subtask execution order can be directly preset or obtained from the preset task framework (the preset task framework can not only be provided with each preset subtask, but also the execution order of each subtask can be preset). The ratio of the number of subtasks whose execution order matches the preset execution order to the total number of subtasks planned by the agent can be determined as the rationality of the task sequence.
[0054] The task relevance evaluation of this embodiment is mainly to evaluate whether each planned subtask is closely related to the specific business scenario and whether it can meet the original requirements of the target task. Specifically, the business features of each subtask can be extracted and compared with the preset business features in the business requirements, and the feature similarity is used as the task relevance. Business features may refer to, but are not limited to, data features of business-related data involved in the execution of each subtask (such as data requested and output during task execution), text features extracted from the task name of the subtask, etc., as well as determining the corresponding business scenario based on the business-related data involved in the task execution of the intelligent agent, and matching the business scenario with the preset business scenario, etc., which are not limited here.
[0055] The task accuracy assessment of this embodiment is mainly to verify whether each planned subtask accurately reflects the intention and requirements of the original target task. In order to achieve the assessment of this dimension, it is necessary to preset the reference subtasks corresponding to the target task, so that the large model of the agent participating in the assessment can refer to the preset reference subtasks and conduct accuracy assessment on the subtasks planned by the agent, and then judge the difference between the subtasks split out by the agent and the reference subtasks (which may include but are not limited to the difference in the number of subtasks, the difference in the content of subtasks, etc.), thereby determining the task accuracy of the agent. Different task accuracy assessment criteria can be set for different business scenarios, which will not be elaborated here.
[0056] To summarize the above example: if the target task received by the agent involves the data analysis business of the financial scenario, the related business will involve the generation of query SQL, and the agent can be evaluated based on the task accuracy, task relevance and other related indicators of the generated SQL statement. If the target task involves the traffic flow control business of the smart transportation scenario, the related business will involve the generation of traffic management actions, and the agent can be evaluated based on indicators such as the completeness of the task content of the traffic management action and the rationality of the task sequence, etc., which will not be elaborated here.
[0057] Optionally, during the evaluation process of this application, the large model used to evaluate the intelligent agent in this application can also be enabled to have the ability of autonomous evaluation through prompt engineering or model fine-tuning. The specific introduction method can refer to the existing method, and this application will not elaborate on it.
[0058] Based on the above embodiments, the present embodiment of the present application describes the steps of determining the business evaluation result based on at least one of the completeness of the task content, the rationality of the task sequence, the task relevance, and the task accuracy. Specifically, the method of this embodiment includes the following steps: The target data is selected from the completeness of the task content, the rationality of the task sequence, the task relevance and the task accuracy; the target data and the data weight corresponding to the target data are weighted and summed to obtain the business evaluation result.
[0059] In combination with the above embodiments, one or more of the methods provided in the above embodiments can be used to obtain one or more data of task content completeness, task sequence rationality, task relevance, and task accuracy. According to the specific application scenario of the agent, one or more data can be flexibly selected from these data as target data for determining the business evaluation result.
[0060] Specifically, each data is preset with a corresponding data weight. For the selected target data, in the process of business evaluation processing, the corresponding target data can be weighted according to the data weight of each target data, and then the target data after each weighted processing can be summed to obtain the business evaluation result. Exemplarily, the evaluation result of the present application can be represented in the form of an evaluation score, or in the form of a natural language, or in the form of a binary judgment (for example, 0 and 1, true and false, yes and no, etc.).
[0061] Based on the above embodiments, the present embodiment of the present application describes the steps of evaluating the execution process information according to the semantic requirements to obtain the semantic evaluation result. Specifically, the method of this embodiment includes the following steps: Obtain target semantic features obtained by the intelligent agent in the process of executing the target task, compare the target semantic features with preset semantic features to obtain semantic understanding accuracy; and / or, perform fluency detection processing on the natural language text generated by the intelligent agent to obtain semantic representation fluency; and / or, perform text matching processing between the natural language text generated by the intelligent agent and preset keywords to obtain text information richness; and / or, perform coherence detection processing on the natural language text generated by the intelligent agent to obtain text coherence; determine the semantic evaluation result according to at least one of the semantic understanding accuracy, semantic representation fluency, text information richness and text coherence.
[0062] The description is made in conjunction with the foregoing embodiment. The foregoing embodiment describes the process of service evaluation. The semantic evaluation process of this embodiment can still refer to the description of the foregoing service evaluation process in the same way.
[0063] It should be noted that the semantic evaluation process of this application is mainly an evaluation of the semantic processing process involved in the execution of tasks by the agent. Usually, the task execution process of the agent involves the understanding and generation of natural language. Therefore, whether the agent can accurately perform the target task and accurately output the task execution result is one of the important indicators for judging the operation effect of the agent.
[0064] In view of the importance of intelligent agents in processing natural language, this embodiment mainly describes general evaluation dimensions related to the natural language understanding and generation process. These dimensions are intended to measure the performance of the intelligent agent in understanding and generating natural language, including semantic understanding accuracy, semantic representation fluency, text information richness, and text coherence. The semantic evaluation result is determined according to at least one of the semantic understanding accuracy, semantic representation fluency, text information richness, and text coherence.
[0065] The semantic understanding accuracy evaluation of this embodiment is mainly to evaluate whether the agent correctly understands the semantics of the received natural language data (such as natural language instructions or natural language questions, etc.). Therefore, the agent can be required to accurately capture the user's intention and avoid misunderstanding or ambiguity of natural language. Specifically, the target semantic features obtained by the agent in the process of performing the target task (or in the process of performing the subtask) and the preset semantic features can be compared, and the obtained semantic feature similarity can be determined as the semantic understanding accuracy.
[0066] The semantic representation fluency evaluation of this embodiment is mainly to detect whether the natural language text generated by the intelligent agent is smooth and natural, without grammatical errors or unclear expressions, which helps to improve user experience and communication efficiency. Among them, the evaluation method of the semantic representation fluency of this application may include but is not limited to evaluation through a pre-trained language model (the language model can be integrated in the large model of this application), N-gram evaluation, comparative evaluation of the generated text and the preset original text, and displaying the natural language text generated by the intelligent agent and evaluating it according to the evaluation instructions received from the user input, etc., which are not limited here.
[0067] The text information richness evaluation of this embodiment is mainly to measure whether the text generated by the intelligent agent contains enough information, and then determine whether the intelligent agent can meet the information needs of the user, so as to ensure that the intelligent agent is concise and comprehensive when generating text. Specifically, the method for judging the richness of text information can be to perform text matching processing between the natural language text generated by the intelligent agent and the preset keywords, and the proportion of the number of natural language words successfully matched in the natural language text to the number of words in the natural language text is determined as the richness of text information. Therefore, the richness of text information can be used to characterize the proportion of the number of valid words in the natural language text generated by the intelligent agent relative to the full text, wherein the preset keywords refer to the expected words included by the intelligent agent when generating the natural language text in an ideal state. It should be noted that different target tasks can correspond to different preset keywords, which will not be elaborated here.
[0068] The text coherence evaluation of this embodiment is mainly to evaluate whether the intelligent agent can maintain the coherence and consistency of the context in application scenarios such as generating multiple rounds of dialogue and / or long texts, which helps to maintain the fluency of the dialogue and the integrity of the text. Specific text coherence detection methods may include, but are not limited to, using word vector models (such as Word2Vec, GloVe, etc.), sentence vector models (combining or encoding word vectors in a sentence to obtain a vector representation of the sentence. Common methods include average pooling, recurrent neural networks (RNN), long short-term memory networks (LSTM) and Transformer, etc.), similarity calculation (for example, by calculating the similarity between context sentence vectors to measure the semantic coherence between sentences. Common similarity calculation methods include cosine similarity, Euclidean distance, Manhattan distance, etc.), semantic matching models (using deep learning models (such as Siamese networks, BERT) for semantic matching to determine the semantic relationship between two sentences, including similarity, opposite, implication, etc.), and context models (considering context information, by introducing context representation or attention mechanism, to improve the understanding of semantic coherence between sentences) and other methods to implement and obtain the text coherence of the intelligent agent, which will not be elaborated here.
[0069] Optionally, due to the differences in the expression methods and grammatical rules of different languages, etc. In the semantic evaluation process of the present application, the same natural language text can also be translated into natural language texts of different languages for evaluation respectively, to obtain the semantic evaluation results of the agent when processing different languages respectively, and then the semantic evaluation results corresponding to different languages are integrated to obtain the final semantic evaluation results. For example, for the text coherence of the agent, if the natural language text generated by the agent is in Chinese, the context coherence evaluation can be performed based on the Chinese text generated by the agent to obtain the Chinese coherence evaluation result; then the natural language text is translated into English, and the agent performs context coherence evaluation based on the English text to obtain the English coherence evaluation result; the natural language text can also be translated into Japanese, and the agent performs context coherence evaluation based on the Japanese text to obtain the Japanese coherence evaluation result, etc. Finally, the Chinese coherence evaluation result, the English coherence evaluation result, and the Japanese coherence evaluation result can be weighted and summed to obtain the text coherence of the agent. In addition, the semantic understanding accuracy, semantic representation fluency, and text information richness in this embodiment can also refer to the above description, and will not be repeated here.
[0070] In the semantic evaluation method of this embodiment, the large model can also be equipped with the ability of self-evaluation through prompt word engineering or model fine-tuning. The specific introduction method is not repeated here.
[0071] Based on the above embodiments, the present embodiment of the present application describes the steps of determining the semantic evaluation result according to at least one of the semantic understanding accuracy, semantic representation fluency, text information richness, and text coherence. Specifically, the method of this embodiment includes the following steps: The target data is selected from the semantic understanding accuracy, semantic representation fluency, text information richness and text coherence; the target data and the data weight corresponding to the target data are weighted and summed to obtain the semantic evaluation result.
[0072] With reference to the foregoing embodiments, the method for determining the semantic evaluation result can be similarly referred to the method for determining the business evaluation result. Each data can be preset with a corresponding data weight (the data weights of different data can be the same or different). For the selected target data, in the process of semantic evaluation processing, the corresponding target data can be weighted according to the data weights of each target data, and then the target data after each weighted processing can be summed to obtain the semantic evaluation result. Exemplarily, the evaluation result of the present application can be represented in the form of an evaluation score, or in the form of a natural language, or in the form of a binary judgment (for example, 0 and 1, true and false, yes and no, etc.).
[0073] Based on the above embodiment, the embodiment of the present application describes the steps of performing result evaluation processing on the execution result information according to the preset result information to obtain the second evaluation result. Specifically, the method of this embodiment includes the following steps: Perform feature comparison processing on the preset result information and the execution result information to obtain a feature matching degree; and determine a second evaluation result based on the feature matching degree.
[0074] The above embodiments are described in conjunction with the method for evaluating the task execution process of the agent, while the present embodiment mainly describes the method for evaluating the task execution result of the agent. Since in the actual application scenario of the agent, there may be situations where an incorrect task execution result is output according to a correct task execution process, or a correct task execution result is output according to an incorrect task execution process, the agent evaluation method of the present application combines the task execution process and the task execution result of the agent for evaluation, thereby improving the accuracy of the evaluation of the agent.
[0075] Specifically, this embodiment is mainly to ensure the quality of the task execution results (answers) generated by the intelligent agent, which may include evaluating the task execution results corresponding to each subtask, or evaluating the final target task execution results, which is not limited here.
[0076] Exemplarily, the execution result information includes information elements of the task execution results, which may include but are not limited to data indicators, operation steps, conclusion statements and other information obtained by the agent after executing the target task, which characterize the core components of the task execution results of the agent. It is equivalent to extracting information elements (execution result information) from the task execution results obtained by the agent and comparing them with preset information elements (preset result information), and then judging whether the execution result information output by the agent is accurate. Among them, the method for obtaining execution result information in this application may include but is not limited to being implemented through a pre-set information recognition algorithm. The information recognition algorithm uses natural language processing (NLP) technology, combined with a specific industry knowledge base (preset database), to perform semantic understanding and information tagging on the task execution results (answers).
[0077] Furthermore, in the present application, the process of performing feature comparison processing on the preset result information and the execution result information may be to use a difference analysis algorithm, which can compare the execution result information in the task execution result with the preset result information in the expected execution result one by one, and score the feature matching degree of each execution result information to obtain the second evaluation result. Among them, it is also possible to extract whether there is corresponding execution result information from the task execution result according to the preset result information, and if there is, extract it for feature comparison, and if not, judge it as empty and set it to 0.
[0078] In addition, the present embodiment can also be implemented through an integrity judgment module, which integrates the identified execution result information with the preset industry standards and / or the requirements of the actual application scenarios to check whether the task execution results (answers) of the intelligent agent comprehensively cover all specified aspects of the target task (problem), thereby determining the task execution completeness rate of the intelligent agent for the target task. The task execution completeness rate can be the ratio between the number of correct extractions of the execution result information that matches the preset result information and the total number of preset result information, etc.
[0079] For example, in a certain financial industry application scenario, the agent needs to answer questions about market risk assessment. According to the method provided in the above embodiment, the information elements (execution result information) in the agent's answer are first extracted, such as "risk factor", "potential loss", and "market volatility". Subsequently, these information elements are compared with the output of the predefined risk assessment model (preset result information) to determine the validity of the agent's answer. If only the "potential loss" and "market volatility" information elements exist in the agent's answer output, the integrity assessment module will indicate that the agent's task execution completeness rate is 2 / 3 = 66.7%.
[0080] Based on the above embodiment, the embodiment of the present application describes the steps of determining the target evaluation result of the agent according to the first evaluation result and the second evaluation result. Specifically, the method of this embodiment includes the following steps: Obtain an evaluation weight of the first evaluation result and an evaluation weight of the second evaluation result; perform weighted summation processing based on the first evaluation result, the second evaluation result and the corresponding evaluation weights to obtain a target evaluation result.
[0081] In conjunction with the above-mentioned embodiment, the first evaluation result may correspond to a first evaluation weight, and the second evaluation result may correspond to a second evaluation weight, wherein the first evaluation weight and the second evaluation weight may be the same or different, which is not limited here.
[0082] In the above-mentioned embodiment, it has been explained that the first evaluation result can represent the evaluation dimension generated based on the business scenario and natural language, and the second evaluation result can represent the evaluation dimension based on the information element of the task execution result. In this embodiment, these dimensions can be effectively integrated to form a comprehensive evaluation system. For example, by adopting methods such as weighted average, each dimension is quantified and assigned corresponding weights, so as to achieve an overall effect evaluation of the process of the intelligent agent performing the target task.
[0083] The weighted average method of the present application may include but is not limited to: assigning different evaluation weights to each evaluation dimension according to its importance, and then calculating the sum of the weighted averages of the scores (evaluation results) of each dimension as the comprehensive evaluation result. This method is simple and easy to implement, but the allocation of evaluation weights needs to be based on sufficient experience and data analysis.
[0084] For example, when using the weighted average method for comprehensive evaluation, the evaluation weights of each evaluation dimension can be determined first. For example, the first evaluation weight of the first evaluation result is set to 35%, and the second evaluation weight of the second evaluation result is set to 65%. Among them, the weight of the completeness of the task content that constitutes the first evaluation result can be 40%, the rationality of the task sequence is 30%, the task relevance is 10%, and the task accuracy is 20%. Then, each planning step can be scored in each dimension (such as using a 10-point system), and finally the evaluation score of the agent in executing each step can be calculated based on the weights and the corresponding evaluation results. In this way, the effectiveness of the agent in different planning steps can be intuitively compared, providing strong support for subsequent optimization and improvement.
[0085] Based on the above embodiments, the present application also provides an agent evaluation system, which may include a result verification interface, and may provide interaction for users in the process of evaluating the agent, so as to verify the evaluation results through the received user instructions, thereby improving the accuracy of the evaluation. Figure 2 ,like Figure 2 As shown, Figure 2 It is a schematic diagram of the result verification interface of the intelligent agent evaluation system in the intelligent agent evaluation method of the present application.
[0086] The result verification interface of this application can clearly and intuitively display the various intermediate steps (subtasks) and task execution results of the agent when performing the target task, which can include the original description of the target task (problem), the planning steps of the agent, the output results of each step, etc. Through the visual interface, users can easily view and understand the processing flow of the agent, and can also understand the overall average indicators of the agent in the process of performing the target task and various related effect indicators (such as completeness, rationality, relevance, accuracy, etc.).
[0087] Then, for each intermediate step, the system provides detailed evaluation dimensions, including evaluation value and evaluation reasons. The evaluation value is a quantitative representation of the execution result of the step, while the evaluation reason provides a qualitative analysis of the execution effect of the step. These evaluation dimensions help users to have a more comprehensive understanding of the pros and cons of each step and provide guidance for subsequent optimization.
[0088] In addition, the system supports information element comparison of the final results of the intelligent agent. By comparing the preset standard results with the actual results, the system can automatically identify the correct and incorrect parts of the comparison. This function helps users quickly locate the problem and improve the efficiency of verification.
[0089] Furthermore, in addition to the system's autonomous evaluation function in the above example, this system can also introduce a manual secondary verification function through the interactive interface. Users can check the planning steps and evaluation results displayed by the system in combination with their own professional knowledge and experience, and issue user instructions. During the verification process, users can refer to the evaluation results provided by the system for modification, and the system will also adjust the evaluation results according to the received user instructions. Through this type of human-machine combined evaluation method, the efficiency of the automated evaluation of large models is retained, and the accuracy and flexibility of manual verification are incorporated. The evaluation methods and evaluation standards of large models can also be modified through manual verification.
[0090] In summary, the agent evaluation method and agent evaluation system provided by this application can analyze the agent's task planning of each subtask for the target task (problem) and perform a large-scale autonomous effect evaluation at the step level. Then, the information elements of the task execution result (answer) are further compared with the preset information elements, and finally the target evaluation result is obtained, so as to achieve a systematic and comprehensive effect evaluation of the agent.
[0091] It should be further explained that the execution subject of the intelligent agent evaluation method may be an intelligent agent evaluation device, for example, the intelligent agent evaluation method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the intelligent agent evaluation method may be implemented by a processor calling a computer-readable instruction stored in a memory.
[0092] Figure 3 FIG. 1 is a block diagram of an intelligent agent evaluation device shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary agent evaluation device 300 includes: an acquisition module 310, a first evaluation module 320, a second evaluation module 330 and a comprehensive evaluation module 340. Specifically: The acquisition module 310 is used to obtain the execution information obtained by the intelligent agent when executing the target task, and the execution information includes the execution process information and the execution result information.
[0093] The first evaluation module 320 is used to perform process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result.
[0094] The second evaluation module 330 is used to perform result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result.
[0095] The comprehensive evaluation module 340 is used to determine the target evaluation result of the agent according to the first evaluation result and the second evaluation result.
[0096] In this exemplary intelligent agent evaluation device, by acquiring the execution information obtained when the intelligent agent performs the target task, it is possible to analyze the task execution process of the intelligent agent according to the execution process information in the execution information, and analyze the task execution result of the intelligent agent according to the execution result information in the execution information; perform process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; perform result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; and comprehensively evaluate the task execution process of the intelligent agent and the task execution result of the intelligent agent, thereby obtaining a target evaluation result of the intelligent agent and achieving accurate and reliable evaluation of the intelligent agent.
[0097] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0098] Among them, the functions of each module can be found in the embodiment of the intelligent agent evaluation method, and will not be repeated here.
[0099] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned agent evaluation method embodiments. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.
[0100] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned agent evaluation method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.
[0101] In this exemplary electronic device, by acquiring the execution information obtained when the intelligent agent performs the target task, it is possible to analyze the task execution process of the intelligent agent according to the execution process information in the execution information, and analyze the task execution result of the intelligent agent according to the execution result information in the execution information; perform process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; perform result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; and comprehensively evaluate the task execution process of the intelligent agent and the task execution result of the intelligent agent, thereby obtaining a target evaluation result of the intelligent agent and achieving accurate and reliable evaluation of the intelligent agent.
[0102] See also Figure 5 , Figure 5 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned agent evaluation method embodiments.
[0103] In this exemplary storage medium, by running the program instructions in the storage medium to obtain the execution information obtained by the intelligent agent performing the target task, it is possible to analyze the task execution process of the intelligent agent according to the execution process information in the execution information, and analyze the task execution result of the intelligent agent according to the execution result information in the execution information; perform process evaluation processing on the execution process information according to the preset process information to obtain a first evaluation result; perform result evaluation processing on the execution result information according to the preset result information to obtain a second evaluation result; and comprehensively evaluate the task execution process of the intelligent agent and the task execution result of the intelligent agent, thereby obtaining a target evaluation result of the intelligent agent and achieving accurate and reliable evaluation of the intelligent agent.
[0104] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0105] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0106] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0107] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
Claims
1. An agent evaluation method, characterized in that: The method comprises: Acquire execution information obtained by the agent executing the target task, wherein the execution information includes execution process information and execution result information; Performing process evaluation processing on the execution process information according to preset process information to obtain a first evaluation result; Performing result evaluation processing on the execution result information according to preset result information to obtain a second evaluation result; A target evaluation result of the agent is determined according to the first evaluation result and the second evaluation result.
2. The method according to claim 1, characterized in that The preset process information includes business requirements and semantic requirements preset in the process of the agent performing the task, the first evaluation result includes a business evaluation result and a semantic evaluation result, and the step of performing process evaluation processing on the execution process information according to the preset process information to obtain the first evaluation result includes: Evaluate and process the execution process information according to the business requirements to obtain the business evaluation result; The execution process information is evaluated and processed according to the semantic requirements to obtain the semantic evaluation result.
3. The method according to claim 2, characterized in that The execution process information includes subtasks obtained by splitting the agent in the process of executing the target task, and the step of evaluating the execution process information according to the business requirements to obtain the business evaluation result includes: Compare the task content of each subtask with the preset subtask content in the business requirement to obtain the completeness of the task content; and / or, comparing the execution order of each subtask with the preset subtask execution order in the business requirement to obtain the rationality of the task sequence; and / or, comparing the business characteristics of each subtask with the preset business characteristics in the business requirements to obtain task relevance; and / or, comparing each subtask with a reference subtask corresponding to the target task to obtain task accuracy; The business evaluation result is determined based on at least one of the task content completeness, the task sequence rationality, the task relevance, and the task accuracy.
4. The method according to claim 3, characterized in that The step of determining the business evaluation result based on at least one of the task content completeness, the task sequence rationality, the task relevance, and the task accuracy comprises: Select target data from the task content completeness, the task sequence rationality, the task relevance, and the task accuracy; The target data and the data weight corresponding to the target data are weighted and summed to obtain the business evaluation result.
5. The method according to claim 2, characterized in that: The step of evaluating the execution process information according to the semantic requirement to obtain the semantic evaluation result includes: Acquire target semantic features obtained by the agent in the process of performing the target task, compare the target semantic features with preset semantic features, and obtain semantic understanding accuracy; and / or, performing fluency detection processing on the natural language text generated by the agent to obtain the fluency of semantic representation; and / or, performing text matching processing between the natural language text generated by the agent and preset keywords to obtain text information richness; and / or, performing coherence detection processing on the natural language text generated by the agent to obtain text coherence; The semantic evaluation result is determined according to at least one of the semantic understanding accuracy, the semantic representation fluency, the text information richness, and the text coherence.
6. The method according to claim 5, characterized in that The step of determining the semantic evaluation result according to at least one of the semantic understanding accuracy, the semantic representation fluency, the text information richness, and the text coherence comprises: Selecting target data from the semantic understanding accuracy, the semantic representation fluency, the text information richness, and the text coherence; The target data and the data weight corresponding to the target data are weighted and summed to obtain the semantic evaluation result.
7. The method according to claim 1, characterized in that The step of performing result evaluation processing on the execution result information according to the preset result information to obtain a second evaluation result includes: Perform feature comparison processing on the preset result information and the execution result information to obtain a feature matching degree; The second evaluation result is determined according to the feature matching degree.
8. The method according to claim 1, characterized in that The step of determining the target evaluation result of the agent according to the first evaluation result and the second evaluation result comprises: Obtaining an evaluation weight of the first evaluation result and an evaluation weight of the second evaluation result; The target evaluation result is obtained by performing weighted summation processing based on the first evaluation result, the second evaluation result and the corresponding evaluation weights.
9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Natural language text generation method and device, electronic equipment and readable storage medium
CN116796708A
Quality evaluation method and device of multi-modal knowledge graph, storage medium and equipment
CN116805012A
Quality evaluation method, system and equipment of multi-modal large model and storage medium
CN117909702A
SQL statement generation capability evaluation method and device, equipment and medium
CN118520007A
Answer quality evaluation method and system based on divide-and-conquer agent
CN118551851A
Cited By
Multi-agent-based SQL statement end-to-end batch generation method
CN119179711A
Task processing method, computing device and computer readable storage medium
CN119806850A
Multi-agent-driven multi-mode cognitive method and device, electronic equipment and medium
CN119961683A
Intelligent agent evaluation processing method and device, storage medium and electronic equipment
CN120144417A
An agent evaluation processing method and device, a storage medium and an electronic device
CN120144417B
