Problem detection method and device for task system, equipment, medium and product

Abstract

The invention discloses a problem detection method and device for a task system, equipment, a medium and a product in the technical field of computers. The problem detection method comprises the steps that task execution information generated when the task system (such as an AI Agent) executes a first task is obtained; and analyzing the task execution information by using the first model based on the cue word corresponding to the first model to obtain an analysis result, so that the analysis result can describe a problem occurring when the task system executes the first task, and the problem of the AI Agent is automatically detected by means of a machine learning model. Wherein the cue word corresponding to the first model is iteratively optimized based on the analysis result of the historical task, so that the cue word can accurately and comprehensively describe the constraint which needs to be met when the problem detection is carried out on the AI Agent as much as possible, and the first model shows better problem detection performance under the guidance of the cue word; therefore, problem detection performance self-evolution is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, medium, or product for detecting problems in a task system. Background Technology

[0002] With the rapid development of artificial intelligence (AI) agents, they have evolved from simple encapsulations of large language models (LLMs) to complex task systems achieved through multi-agent collaboration and the invocation of diverse tool pools. These systems employ a "React-Act-Observe" model, enabling them to plan, execute, and reflect to accomplish complex tasks.

[0003] Furthermore, while increasingly complex AI agent architectures can raise the upper limit of capabilities, they also introduce the unknowability of their internal behavior, and this "unknowability" becomes an obstacle to iteration. For example, if the output generated by an AI agent for a certain task has minor flaws, these flaws are very likely due to a problem (such as a decision-making error) in some part of the AI ​​agent's processing of the task. However, how to uncover such problems has become a pressing technical issue that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method, apparatus, device, medium, and product for detecting problems in a task system.

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides a problem detection method for a task system. The method includes: acquiring task execution information generated by the task system executing a first task; analyzing the task execution information using a first model based on prompt words corresponding to the first model to obtain analysis results, the analysis results including problems encountered by the task system in executing the first task; and updating the prompt words based on the analysis results.

[0007] In one possible implementation, the method further includes: updating the prompt word based on the updated system description data of the task system in response to detecting an update to the task system.

[0008] In one possible implementation, the first model is configured to execute an attribution workflow to obtain the analysis results; the first model includes multiple evaluators, each of which is configured to execute a different stage of the attribution workflow; the prompt words corresponding to the first model include prompt words corresponding to each of the evaluators.

[0009] In one possible implementation, the method further includes: updating the labeling system corresponding to the second evaluator in the first model based on the output data of the first evaluator in the first model, wherein the first evaluator is configured to analyze the task execution information to obtain the output data based on the labeling system corresponding to the first evaluator, and the second evaluator is configured to analyze the output data and the task execution information based on the labeling system corresponding to the second evaluator.

[0010] In one possible implementation, updating the label system corresponding to the second evaluator in the first model based on the output data of the first evaluator in the first model includes: performing semantic clustering based on the question indicated by the output data to obtain at least one cluster; updating the label system corresponding to the second evaluator based on the at least one cluster, wherein the updated label system includes the labels corresponding to each cluster.

[0011] In one possible implementation, obtaining the task execution information generated by the task system executing the first task includes: responding to receiving the original log generated by the task system executing the first task according to a preset mode, performing structured processing on the original log to obtain a structured log; mapping the structured log according to the preset mode to obtain an execution trace; and determining the task execution information based on the execution trace.

[0012] In one possible implementation, updating the prompt word based on the analysis result includes: in response to receiving an audit result provided for the analysis result, updating the prompt word based on the audit result, wherein the audit result indicates the correspondence between different contents in the analysis result and different parts of the prompt word.

[0013] In one possible implementation, the method satisfies at least the following constraints: the first task is any task in the task set; the task execution information includes the execution trace of the first task and at least a portion of the product file of the first task; each evaluator in the first model is a large language model; the prompt words are represented according to a preset template, the preset template includes at least one variable part, the at least one variable part includes tag information, system information, exemption policy, and at least a portion of the business scenario, the tag information includes at least a portion of a first tag system, a second tag system, and a third tag system, and the business scenario includes at least a portion of positive examples and negative examples.

[0014] This application provides a problem detection device for a task system, comprising: an acquisition unit for acquiring task execution information generated by the task system executing a first task; an analysis unit for analyzing the task execution information using a first model based on prompt words corresponding to the first model to obtain analysis results, the analysis results including problems encountered by the task system in executing the first task; and an update unit for updating the prompt words based on the analysis results.

[0015] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the problem detection method for a task system provided in this application.

[0016] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the problem detection method for a task system provided in this application.

[0017] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the problem detection method for a task system provided in this application.

[0018] Compared with related technologies, this application has at least the following advantages:

[0019] The technical solution provided in this application first obtains task execution information (such as logs, artifacts, etc.) generated by the task system (such as an AI Agent) in executing the first task, so that the task execution information can describe the characteristics presented by the AI ​​Agent in executing the first task, such as what processes were experienced and what data was generated in each process; then, a first model (such as LLM) is used to analyze the task execution information based on the prompt words corresponding to the first model to obtain analysis results, so that the analysis results can at least describe the problems that occurred in the task system in executing the first task (such as decision errors, redundant steps, etc.), so as to realize the automatic detection of AI Agent problems (such as process problems) by means of machine learning models. The prompt words corresponding to the first model are iteratively optimized based on the analysis results of historical tasks. This ensures that the prompt words can describe as accurately and comprehensively as possible the constraints (such as problem recognition scope, positive examples, negative examples, exemption strategies, etc.) that need to be met when performing problem detection (such as process problem detection) on the AI ​​Agent. As a result, the first model exhibits better problem detection performance under the guidance of the prompt words. In this way, the problem detection performance presented by the first model can be automatically iteratively optimized through the iterative optimization of the prompt words, so as to achieve self-evolution of problem detection performance and better improve the problem detection effect. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a problem detection method for a task system provided in this application embodiment;

[0022] Figure 2 This is a schematic diagram of the structure of a self-evolving problem detection system provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram illustrating an AI Agent tool usage capability assessment framework provided in this application embodiment;

[0024] Figure 4 A schematic diagram of an AI Agent process efficiency evaluation framework provided for an embodiment of this application;

[0025] Figure 5 A schematic diagram of an AI Agent planning reflection capability assessment framework provided for an embodiment of this application;

[0026] Figure 6 A schematic diagram of a problem detection device for a task system provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] Research has found that AI Agents have the following advantages over pre-built workflows: AI Agents handle uncertainty, enabling them to autonomously address open problems that fixed workflows cannot handle, dynamic tool calls, and interactive decision-making.

[0029] Research also revealed that the unknowability of the internal behavior of AI agents hinders their performance iteration. Therefore, to overcome this obstacle, process problems in AI agents can be detected. This allows for subsequent analysis of the AI ​​agent's inherent defects, leading to iterative optimization. These process problems refer to various issues that arise at each stage of the AI ​​agent's autonomous closed-loop process of "perception-planning-execution-feedback" to complete a task, affecting task progress and reducing the accuracy of results. Thus, process problems can encompass all internal capability issues of the AI ​​agent that affect the final output, system efficiency, or cost.

[0030] Research has also revealed that in certain scenarios, pre-defined evaluation benchmarks can be used to detect process problems in AI agents. This approach is conducted in a static, idealized environment. Specifically, it involves: first, relevant personnel manually designing a standard, correct execution trajectory as the ground truth for a given task; then, using an AI agent to execute the task and obtain the actual execution trajectory; finally, comparing the actual execution trajectory with the ground truth and using metrics such as success rate and step matching degree to evaluate the AI ​​agent's planning, reflection, and tool usage capabilities, thereby detecting process problems in the AI ​​agent.

[0031] The study also found that the above-mentioned scheme has the following defects: (1) Because the scheme relies heavily on the Ground Truth designed manually by relevant personnel, the scheme is only applicable to AI Agents with fixed architecture, fixed tasks and unique optimal solutions for tasks. As a result, the scheme cannot be applied to complex AI Agents with dynamic architecture, open tasks and no unique optimal solutions in real production environments; (2) Because the evaluation capability of the scheme is static and fixed, it cannot discover new and emerging unknown problems beyond the preset.

[0032] Research revealed that, in order to at least overcome some of the deficiencies mentioned above, this application attempted the following approach: First, relevant personnel manually designed evaluation logic for the AI ​​Agent through hard coding or fixed configuration, so that the evaluation logic could detect application programming interface (API) calls in the AI ​​Agent, record execution logs, and provide some basic product evaluation capabilities; then, the evaluation logic was used to perform end-to-end product evaluation of the AI ​​Agent (e.g., evaluating the quality of the final result) and fixed module output evaluation, in order to detect problems in the AI ​​Agent process.

[0033] Research has revealed that the solution described above has at least the following drawbacks: because the evaluation logic in this solution is obtained through hard coding or fixed configuration, it cannot automatically adapt when the architecture or behavior pattern of the AI ​​Agent changes. Instead, it requires a lot of manual effort to update and maintain the evaluation logic.

[0034] Based on the above research, in order to overcome the problems of static and rigid evaluation systems and the inability to evolve on their own in the above four solutions, this application provides a problem detection method for task systems. The method includes: firstly, acquiring task execution information (such as logs, products, etc.) generated by the task system (such as an AI Agent) when performing a first task, so that the task execution information can describe the characteristics of the AI ​​Agent performing the first task, such as what processes were involved and what data was generated in each process; then, using a first model (such as LLM) to analyze the task execution information based on the prompt words corresponding to the first model to obtain analysis results, so that the analysis results can at least describe the problems that occurred when the task system performed the first task (such as decision errors, redundant steps, etc.), so as to realize the automatic detection of AI Agent problems (such as process problems) by means of machine learning models. The prompts for the first model are iteratively optimized based on the analysis results of historical tasks. This ensures that the prompts accurately and comprehensively describe the constraints (such as problem awareness scope, positive examples, negative examples, exemption strategies, etc.) that need to be met when detecting problems (such as process problems) for the AI ​​Agent. As a result, the first model exhibits better problem detection performance under the guidance of these prompts. This iterative optimization of the prompts enables automatic iterative optimization of the problem detection performance of the first model, achieving self-evolution of problem detection performance. This overcomes the adverse effects caused by the static and fixed problem detection performance and inability to self-evolve of some solutions (such as the solutions shown in the four paragraphs above), thus improving the problem detection effect.

[0035] Furthermore, this application does not limit the executing entity of the problem detection method for the task system. For example, the method can be applied to a terminal device or a server. Alternatively, the method can be implemented through data interaction between the terminal device and the server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0037] To better understand the technical solution provided in this application, the problem detection method for task systems provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the problem detection method for task systems provided in this application includes S1-S3 below.

[0038] S1: Obtain the task execution information generated by the task system when executing the first task.

[0039] Among them, a task system refers to any system with task execution capabilities, such as an AI Agent intelligent system.

[0040] The first task refers to any task performed by the AI ​​Agent during process problem detection (such as...). Figure 2 (Any task recorded in the evaluation set shown).

[0041] It should be noted that, for Figure 2 The evaluation set shown has the following characteristics: it records a large number of tasks; and it is not static, but is periodically updated by pulling tasks from a large number of online tasks executed by the AI ​​Agent to ensure that the evaluation set can record as many diverse tasks as possible, so that the problem detection process based on the evaluation set can be better applied to the AI ​​Agent with the characteristic of task openness.

[0042] Therefore, in one possible implementation, the first task described above can be a set of tasks (such as...). Figure 2 The evaluation set shown is any task in the set. This task set is updated regularly based on online tasks performed by the AI ​​Agent to ensure it comprehensively records all tasks processed by the AI ​​Agent. This results in a diverse range of tasks recorded in the set, enabling the problem detection process based on this set to detect as many process problems as possible presented by the AI ​​Agent under different tasks, thereby improving the comprehensiveness of problem detection.

[0043] The aforementioned task execution information refers to the information generated by the task system when executing the first task, so that the task execution information can indicate the characteristics presented by the task system in executing the first task, such as which tools were used and whether the execution was successful.

[0044] As can be seen, the task execution information for the first task can be obtained by a task system (such as an AI Agent) executing the first task, so that the task execution information can describe the characteristics exhibited by the task system when executing the first task, thus enabling the task execution information to at least describe how the task system executed the first task. Here, the task system refers to a system that executes the first task according to the ReAct mode, such as any AI Agent or... Figure 2 The task execution platform shown is an example. It should be noted that ReAct is an abbreviation for "Reflect-Act-Observe".

[0045] Furthermore, this application does not limit the implementation method of the aforementioned task execution information. For example, the task execution information may refer to some or all of the data generated by the AI ​​Agent performing the first task, such as... Figure 2 The data shown includes raw logs, output files, etc. It can be seen that, in one possible implementation, the task execution information of the first task may include the raw logs and output files generated by the AI ​​Agent executing the first task.

[0046] Research has found that the raw logs generated by the AI ​​Agent performing the first task are of limited value because they record events that occurred during the execution of the first task in a raw and unstructured (or semi-structured) manner.

[0047] Based on the above research, in order to overcome the shortcomings shown in the previous paragraph, the process of determining the task execution information of the first task may include: in response to receiving the original log generated by the task system executing the first task according to a preset mode (such as the "Think-Action-Observe" ReAct mode), making the original log (such as...) Figure 2 The original log shown can indicate the characteristics exhibited by the task system during the execution of the first task in a structured or unstructured manner; the original log is processed into a structured log, so that the structured log does not contain unstructured or semi-structured data compared to the original log, thus making the structured log easier for the machine to understand, overcoming the defects caused by the presence of unstructured or semi-structured data in the original log; the structured log is mapped according to this preset mode to obtain the execution trace, so that the execution trace (such as...) Figure 2The structured trace shown can represent the results of structured analysis of the thought chain in the structured log, so that the execution trace can present the data recorded by the structured log according to the thought chain structure. In turn, the execution trace can present the characteristics of the task system during the execution of the first task in the sequence of "think-action-observation". In this way, the execution trace can restore the execution flow of the task system for the first task as accurately as possible. Based on the execution trace, the task execution information of the first task is determined so that the task execution information includes the execution trace. In this way, the task execution information can indicate the execution flow of the first task in the task system as accurately as possible. In this way, the task execution information can describe the execution flow of the first task in a way that is easier for the large language model to understand, thereby reducing the difficulty of log understanding and better overcoming the defects caused by the large language model's inability to thoroughly understand the log.

[0048] For the aforementioned preset mode, the preset mode refers to the mode adopted by the task system to perform the task (such as the "Think-Action-Observe" ReAct mode); and this application does not limit the implementation of the preset mode. For example, if the task system is an AI Agent, the preset mode can be the "Think-Action-Observe" ReAct mode.

[0049] The structured log mentioned above is obtained by converting unstructured or semi-structured log data (such as strings and text) recorded in the original log into structured data in a fixed format that can be parsed by machines. Moreover, this application does not limit the method of obtaining the structured log. For example, it can adopt any method that can convert the original log into structured data, such as pre-constructing scripts, rules, or machine learning models with this conversion function for implementation.

[0050] Regarding the above execution traces (such as) Figure 2 For the structured trace shown, the execution trace is obtained by mapping the structured log to the ReAct pattern framework so that the execution trace belongs to the standard "think-act-observe" sequence. This allows the execution trace to describe the execution process of the AI ​​Agent for the first task in a way that is easier for the large language model to understand, thereby reducing the difficulty of log understanding.

[0051] Furthermore, this application does not limit the implementation of the above-mentioned execution trace. For example, the execution trace can at least satisfy the following constraints: (1) The execution trace is a complete array of calling agents (call_agents) so that the execution trace can form an interaction sequence arranged in time order, thereby enabling the execution trace to completely present the agent calling situation presented during the execution of the first task; (2) Each call_agents entry in the execution trace and its internal tool calls constitute one or more ReAct loops; (3) The first field (such as the agent_name field) in each call_agents entry indicates its acting subject (Acting (4) The second field (such as the output field) in each call_agents entry indicates the thought, so that the second field can describe the Agent's internal reasoning, decision-making and analysis; (5) The call tool information (call_tool_info) array corresponding to each call_agents entry indicates the action, so that the call_tool_info array describes the specific operation performed by the Agent after the decision. The third field (such as the name field) in the call_tool_info array indicates the tool name, and the fourth field (such as the input field) in the call_tool_info array indicates the tool input parameters; (6) The fifth field (such as the output field) in the call_tool_info array indicates the observation, so that the fifth field can describe the result returned by the environment after the action is executed, so that the fifth field can be used as the sixth field (such as the input field) in the next call_agents entry.

[0052] Furthermore, this application does not limit the method of obtaining the aforementioned execution traces. For example, it can employ pre-built scripts, rules, machine learning models, or tools (such as...) capable of mapping structured logs to the ReAct pattern framework. Figure 2 The log parsing tool shown is used for implementation.

[0053] Based on the aforementioned task execution information for the first task, it is known that, in one possible implementation, this task execution information may include the execution trace of the first task (e.g., Figure 2 The structured trace shown) and the output files of the first task (such as...) Figure 2 At least a portion of the output file shown is used to ensure that the task execution information can describe the execution characteristics of the AI ​​Agent under the first task as comprehensively and accurately as possible.

[0054] Based on the above content related to S1, it can be seen that in some scenarios, the AI ​​Agent first performs the first task to obtain the raw logs and output files; then, some processing is performed on the raw logs (such as...). Figure 2 The thought chain structure analysis and other processing shown are used to obtain the execution trace represented in the sequence of "thinking-action-observation". Then, based on the execution trace and the product file, the task execution information of the first task is determined so that the task execution information can describe the execution characteristics of the AI ​​Agent under the first task in a way that is easier for LLM to understand. This will enable the LLM to detect process problems of the AI ​​Agent under the first task from the task execution information.

[0055] S2: Using the first model and the corresponding prompt words, analyze the task execution information to obtain the analysis results, which include the problems that occurred when the task system executed the first task.

[0056] The first model refers to a machine learning model with AI Agent process problem detection function; and this application does not limit the implementation method of the first model. For example, the first model can be implemented using LLM.

[0057] The prompt words corresponding to the first model can indicate the characteristics of the first model (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions for judging the first model to constrain the performance of the first model; and this application does not limit the implementation method of the prompt words.

[0058] The aforementioned analysis results refer to the results obtained by the first model analyzing the task execution information of the first task (such as problem detection and processing), so that the analysis results at least include the problems that occur when the task system executes the first task, such as problems existing during the execution of the first task; moreover, this application does not limit the implementation method of the analysis results. For example, in some scenarios, in order to better improve the problem detection effect, the analysis results may also include the associated information of the problem (such as the identification of typical cases, replay links of typical cases, etc.), so that the analysis results can not only describe what kind of problems exist in the execution of the first task, but also describe the samples that occurred during the execution of the first task and were selected as typical cases. Here, the typical case refers to the sample that occurred during the execution of the first task, actually had the problem, and was selected as a case associated with the problem, so that the typical case can describe how the AI ​​Agent behaves when the problem occurs.

[0059] Furthermore, this application does not limit the implementation of the above-mentioned S2. For example, S2 may specifically be: inputting the prompt word corresponding to the first model and the task execution information of the first task into the first model, so that the first model can analyze the task execution information (such as problem detection and processing) under the guidance of the prompt word and obtain the analysis result, so that the analysis result can describe what problems (such as tool usage errors) occurred when the AI ​​Agent performed the first task and its typical cases, thereby enabling the analysis result to describe the process problems presented by the AI ​​Agent under the first task.

[0060] S3: Update the prompt words based on the analysis results.

[0061] In this application, after obtaining the above analysis results, the prompt words corresponding to the first model can be updated based on the analysis results so that the updated prompt words include some or all of the content in the analysis results (such as some problems and their typical cases). This allows the updated prompt words to more accurately and comprehensively describe the process problems that the AI ​​Agent may encounter when performing tasks compared to the previous prompt words. Consequently, the updated prompt words can more accurately and comprehensively describe the constraints that need to be met when detecting process problems for the AI ​​Agent compared to the previous prompt words. Thus, under the guidance of the updated prompt words, the first model exhibits better process problem recognition performance.

[0062] Furthermore, this application does not limit the implementation of the above-mentioned S3. For example, in some scenarios, S3 may specifically be: filling the prompt word with some or all of the problems recorded in the analysis results (such as new problems not recorded in the prompt word corresponding to the first model) to achieve the update processing for the prompt word.

[0063] For example, in some scenarios, S3 above can specifically be: generating a new prompt word based on the above analysis results and the prompt word corresponding to the first model at the current time. This application does not limit the process of generating the prompt word; for example, it can utilize any machine learning model with prompt word generation capabilities (such as LLM or...). Figure 2 The prompt word generator shown is used for implementation. Therefore, in one possible implementation, S3 can specifically be: utilizing a second model (such as LLM or...) Figure 2 The prompt word generator shown updates the prompt words corresponding to the first model based on the analysis results, so that the first model can continue to perform problem detection processing on the task description information of other tasks under the guidance of the prompt words.

[0064] Based on the above S1 to S3, the self-evolving problem detection method for task systems provided in this application includes: first, acquiring task execution information (such as logs, products, etc.) generated by the task system (such as an AI Agent) executing a first task, so that the task execution information can describe the characteristics presented by the AI ​​Agent executing the first task, such as what processes were experienced and what data was generated in each process; then, using a first model (such as LLM) to analyze the task execution information based on the prompt words corresponding to the first model to obtain analysis results, so that the analysis results can at least describe the problems that occurred in the task system executing the first task (such as decision errors, redundant steps, etc.), so as to realize the automatic detection of AI Agent problems (such as process problems) by means of machine learning models. The prompt words corresponding to the first model are iteratively optimized based on the analysis results of historical tasks. This ensures that the prompt words can describe as accurately and comprehensively as possible the constraints (such as problem recognition scope, positive examples, negative examples, exemption strategies, etc.) that need to be met when performing problem detection (such as process problem detection) on the AI ​​Agent. As a result, the first model exhibits better problem detection performance under the guidance of the prompt words. In this way, the problem detection performance presented by the first model can be automatically iteratively optimized through the iterative optimization of the prompt words, so as to achieve self-evolution of problem detection performance and better improve the problem detection effect.

[0065] Research has found that in some scenarios, in order to improve the iteration effect, the prompt words corresponding to the first model can be templated. Specifically, a standardized prompt word template is defined so that the template includes not only some fixed parts but also some variable parts, so that subsequent iteration optimization of the prompt words can be achieved by adjusting these variable parts, thereby improving the controllability of prompt word iteration updates.

[0066] Based on the above research, in one possible implementation, the prompt words corresponding to the first model can at least satisfy the following constraints: the prompt words are represented according to a preset template (such as a standardized prompt word template predefined by relevant personnel), the preset template includes at least one variable part, the at least one variable part includes tag information, system information, exemption strategy, and at least a part of the business scenario, the tag information includes at least a part of the first tag system, the second tag system, and the third tag system, and the business scenario includes at least a part of the positive examples and negative examples, so that subsequent iterative optimization of the prompt words can be achieved by adjusting these variable parts, so as to avoid defects when regenerating the whole prompt words (such as some content that should not be changed being changed), and improve the controllability of the iterative optimization of the prompt words.

[0067] Regarding the aforementioned label information, this label information can describe the process problems that the AIAgent may encounter when performing tasks in a structured manner (such as a tree diagram); and this label information can include at least a portion of a first label system, a second label system, and a third label system. Furthermore, this application does not limit the representation of this label information. For example, when the label information includes a first label system, a second label system, and a third label system, the label information can be represented in a hierarchical structure, and this hierarchical structure can specifically be: the first label system is located in the first layer (such as layer L0); the second label system is located in the second layer (such as layer L1); and the third label system is located in the third layer (such as layer L2).

[0068] For the aforementioned first labeling system, different labels in this system indicate different task states (such as task execution success). This ensures that the first labeling system not only records labels representing "task execution success," but also labels representing various reasons for task failure, such as labels representing "rejection - missing input information," "rejection - non-compliance with preset security rules," "rejection - insufficient capability," "rejection - unavailable third-party dependencies," "product - severely missing content," "product - mismatched attachment format," "interruption - task timeout," "interruption - user interruption," and "interruption - system error," etc. This allows the first labeling system to comprehensively cover various task states that the AI ​​Agent may encounter during task execution. Furthermore, this application does not limit the method of obtaining the first labeling system; for example, the first labeling system can be manually labeled by relevant personnel. For example, this first labeling system can be detected from some files (such as system description data of AI Agents). In addition, since the first labeling system is at the L0 level, the labels existing in the first labeling system can be called L0 labels.

[0069] In the aforementioned second labeling system, different labels indicate different first problems (such as general problems), enabling the second labeling system to indicate potential process problems that may arise when the AI ​​Agent successfully completes a task. The second labeling system is obtained through a preliminary, macro-level classification of AI Agent process problems, resulting in a relatively coarse-grained classification of problem categories represented by each label. Furthermore, the first problem refers to the problem represented by the labels present in the second labeling system. Moreover, since the second labeling system operates at the L1 level, the labels present in the second labeling system can be referred to as L1 labels.

[0070] Research has revealed that because the trajectory of an AI Agent during task execution is a dynamic and non-linear "think-and-do" process, it is difficult to statically decompose and map this trajectory to multiple independent modules, as the capabilities are highly coupled in actual use. To overcome this difficulty, this application abstracts and summarizes the core workflow of the AI ​​Agent into three dimensions: tool usage, process efficiency, and planning reflection. Planning reflection is derived by integrating the highly dependent and interdependent capabilities of the AI ​​Agent, such as intent understanding, task planning, process reflection, context processing, and user interaction. Tool usage serves as the entity link in the AI ​​Agent's task execution. Process efficiency is configured to analyze non-engineering efficiency bottlenecks, such as invalid loops, redundant steps, or unnecessary user interactions during task execution, complementing conventional end-to-end time consumption and other engineering tracking point analyses.

[0071] Based on the foregoing, in one possible implementation, the aforementioned second tagging system may include tags corresponding to the tool usage dimension, tags corresponding to the process efficiency dimension, and tags corresponding to the planning reflection dimension. This allows the second tagging system to comprehensively describe the potential problems that may arise when the AI ​​Agent performs a task. Specifically, the tags corresponding to the tool usage dimension can comprehensively indicate potential tool invocation problems that may arise when the AI ​​Agent performs a task, such as issues related to tool selection, parameter construction, result processing, and interaction with external tools. The tags corresponding to the process efficiency dimension can comprehensively indicate potential process efficiency problems that may arise when the AI ​​Agent performs a task, such as redundant steps, unnecessary waiting, and resource waste that affect execution efficiency. The tags corresponding to the planning reflection dimension can comprehensively indicate potential planning reflection problems that may arise when the AI ​​Agent performs a task, such as issues related to the logical aspects of intent understanding, task planning, process reflection, and adjustment. Furthermore, this application does not limit the method of obtaining the second tagging system. For example, the second tagging system may be manually labeled by relevant personnel. Alternatively, the second tagging system may be automatically generated using a certain method (such as LLM).

[0072] For the aforementioned third labeling system (such as...) Figure 2Taking the L2 label tree shown as an example, different labels in this third label system indicate different second problems (such as fine-grained problems, modular problems, problems of particular concern to relevant personnel, etc.), so that the third label system can indicate the type of problem that may occur when the AI ​​Agent performs the task. Among them, the problem categories indicated by the labels in the third label system have advantages over the problem categories indicated by the labels in the second label system, such as more refined classification granularity, more semantic problem types, and problem types that are easier for LLM to understand. In addition, since the third label system is at the L2 level, the labels existing in the third label system can be called L2 labels. Furthermore, the labels in the third label system can be determined based on some problem clusters obtained by semantic clustering of a large number of problem cases carrying L1 labels, so that the third label system includes the labels corresponding to these problem clusters (such as L2 labels). It should be noted that this application does not limit the method of obtaining the labels corresponding to the problem clusters. For example, the labels corresponding to the problem clusters can be obtained by semantic extraction of the problem clusters using a third model (such as LLM), so that the labels can represent the semantics common to each problem in the problem cluster (such as the semantics of "image missing"). For example, the tags corresponding to the problem cluster can be manually labeled by relevant personnel. Therefore, in one possible implementation, some tags in this third labeling system can be manually labeled by relevant personnel, while the remaining tags in the third labeling system can be obtained through offline detection methods (such as...). Figure 2 The L2 label tree update method shown is used to obtain the result.

[0073] Regarding the aforementioned system information, this system information can describe some characteristics of the task system (AI Agent) itself; moreover, this system information is based on the task system's system description data (such as...). Figure 2 The system description data is determined within the system design context shown, so that the system information includes some or all of the content in the system description data. This system description data can describe all or most of the characteristics of the task system; and this application does not limit the implementation method of the system description data. For example, the system description data may include system design documents, product requirement documents (PRDs), etc. The system design document can describe the design characteristics of the task system, such as the definition of agent prompts (e.g., role settings, task execution workflow, thinking rules, tool execution rules, input / output descriptions, etc.), tool function description information (e.g., input / output parameters), and the mapping relationship between agents and tools. The PRD can describe some requirement information of the task system, such as business scenario definitions and product style requirements.

[0074] The aforementioned exemption strategy identifies aspects of the task system that don't require attention during task execution, such as what may appear to be problems but are actually design features or technical limitations (e.g., redundant checks designed to ensure robustness). This allows for the identification of issues not inherent to the task system itself, ensuring the validity of the detected problems. Furthermore, this application does not limit the method of obtaining the exemption strategy. For example, it can be manually annotated by relevant personnel. Alternatively, it can be extracted from the task system's system description data (such as system design documents). Or, it can be detected from the analysis results of historical tasks.

[0075] For the aforementioned business scenario, this business scenario can indicate the characteristics of the application scenario of the task system; moreover, this application does not limit the business scenario. For example, the business scenario can satisfy at least one of the following constraints: the business scenario can describe the core task types of the task system and their definitions (e.g., the scope of the writing scenario includes in-depth research content, office writing, learning and education content, literary and artistic creation, etc.); the business scenario can describe some typical examples of the task system (e.g., positive examples and negative examples, etc.), the positive examples can describe typical behavioral patterns that the task system needs to learn, and the negative examples can indicate typical behavioral patterns that the task system needs to avoid. In addition, this application does not limit the method of obtaining the business scenario. For example, the business scenario can be manually labeled by relevant personnel. Or, the business scenario can be extracted from the system description data of the task system (e.g., extracting the core task types and their definitions from the PRD, etc.). Or, the business scenario can be mined from the analysis results of historical tasks (e.g., mining negative examples from the analysis results, etc.).

[0076] Based on the aforementioned prompts, in one possible implementation, S3 can specifically be: updating the first part of the prompts corresponding to the first model according to the analysis results. The first part includes at least one of the exemption strategy, positive examples, and negative examples, so that the updated first part adds some new content compared to the original first part (such as new problems that need to be exempted, behavioral patterns that can be used as positive examples, behavioral patterns that can be used as negative examples, etc.). This allows the prompts including the updated first part to more comprehensively describe the constraints that need to be met when using the first model to detect problems in the AI ​​Agent process, thereby improving the problem detection effect.

[0077] Research has found that in some scenarios, to improve accuracy, after obtaining the analysis results provided by the first model, relevant personnel can review these results to select valuable content, such as new issues that need to be exempted, behavioral patterns that can serve as positive examples, and behavioral patterns that can serve as negative examples. This allows for subsequent iterative optimization of prompts based on the review results, thereby improving the controllability of prompt optimization.

[0078] Based on the above research, in one possible implementation, S3 can specifically be: in response to receiving the review result provided for the above analysis result, updating the prompt word corresponding to the first model according to the review result, wherein the review result indicates the correspondence between different contents in the analysis result and different parts in the prompt word, so that the review result can more accurately indicate the update direction of the prompt word, thereby making the prompt word update based on the review result more controllable, so as to better avoid the defects caused by the uncontrollable update direction.

[0079] Regarding the aforementioned review results, these results refer to those provided by relevant personnel through human-computer interaction regarding the analysis results output by the first model. These results demonstrate how to use the analysis results to update the corresponding prompts in the first model. For example, updating the exemption strategy using issue 1 described by the analysis results, or using typical cases corresponding to issue 2 described by the analysis results as negative examples. This allows the review results to describe the correspondence between different content in the analysis results and different parts of the prompts, enabling subsequent updates to the prompts based on this correspondence. This updating can utilize a second model (such as...). Figure 2 The application may implement the prompt word generator shown in the figure, or it may be implemented in other ways, and this application does not make any specific limitation on this.

[0080] Research has revealed that in some scenarios, AI agents are not static but rather iteratively updated based on business needs. Therefore, to better meet this requirement, the aforementioned problem detection method for task systems can include at least the following steps: In response to the detection of an update to the task system, the prompt words corresponding to the first model are updated based on the updated system description data of the task system, so that the updated prompt words can describe the characteristics of the updated task system. This allows the first model to better detect problems (such as process problems) in the updated task system under the guidance of the updated prompt words, thereby enabling the first model to automatically adapt to the latest state of the task system. This achieves self-evolution of problem detection performance with the iterative updates of the AI ​​agent, thereby improving the problem detection effect.

[0081] Furthermore, this application does not limit the implementation of the above-mentioned step of "updating the prompt words corresponding to the first model based on the updated system description data of the task system". For example, it can specifically be: updating at least part of the content (such as system information, positive examples, etc.) of the prompt words corresponding to the first model based on the system description data.

[0082] As can be seen, in one possible implementation, the above-mentioned problem detection method for a task system may include at least the following steps: in response to detecting an update to the task system, updating the second part of the prompt word corresponding to the first model based on the updated system description data of the task system, the second part including system information and at least one positive example, the system information describing at least one feature of the task system, so that the updated second part can better describe the features of the updated task system, so as to better achieve the self-evolution of problem detection performance and improve the problem detection effect.

[0083] Furthermore, to better improve problem detection performance, the aforementioned first model should at least satisfy the following constraints: the first model is configured to execute an attribution workflow (such as...). Figure 2 The attribution workflow shown enables the first model to process task execution information (such as...) of the first task in a pipeline manner. Figure 2 The structured traces shown are analyzed in depth layer by layer to better identify process problems presented by the AI ​​Agent in the first task. This attribution workflow refers to the execution process of conducting in-depth analysis of the task execution information of the first task.

[0084] Furthermore, to better improve the problem detection effect, the first model mentioned above can at least satisfy the following constraints: the first model is configured to execute the attribution workflow to obtain the above analysis results; the first model includes multiple evaluators (such as evaluator 1 to evaluator 5), and different evaluators in the first model are configured to execute different stages of the attribution workflow; the prompt words corresponding to the first model include the prompt words corresponding to each evaluator (such as the prompt words corresponding to evaluator 1 to evaluator 5), so that the task execution information of the first task can be analyzed in depth layer by layer with the help of these evaluators.

[0085] For any evaluator in the first model described above, the evaluator is configured to implement the corresponding stage in the attribution workflow under the guidance of the prompt word corresponding to the evaluator; and the implementation of the prompt word corresponding to the evaluator is similar to the implementation of the prompt word corresponding to the first model described above, and will not be repeated here for the sake of brevity.

[0086] Furthermore, this application does not limit the implementation method of each evaluator in the first model described above. For example, it can be implemented using any machine learning model (such as LLM). Therefore, in one possible implementation, each evaluator in the first model can be a large language model, enabling these evaluators to better complete the corresponding stages in the attribution workflow, thereby improving the problem detection effect.

[0087] Furthermore, this application does not limit the working principle of the first model described above; for example, it can adopt any pre-built workflow (such as...). Figure 2 The attribution workflow shown is implemented.

[0088] Therefore, in one possible implementation, when the first model includes evaluators 1 to 5, the working principle of the first model may include steps 11 to 15 below.

[0089] Step 11: Using evaluator 1, based on the prompt words corresponding to evaluator 1, analyze the above task execution information to obtain a first result. The first result indicates the state of the execution process of the first task under the first label system. Different labels in the first label system indicate different task states.

[0090] The evaluator 1 is configured to classify the task description information of any task into task status categories (such as success, failure, etc.) to quickly filter out tasks that require in-depth analysis.

[0091] Therefore, evaluator 1 can refer to an evaluator with task state classification capabilities (such as LLM or...). Figure 2 The task evaluation diverter shown); and the evaluator 1 is configured to: under the guidance of the prompt words corresponding to the evaluator 1, classify the execution status of the first task according to the task execution information of the first task to obtain a first result, so that the first result can describe the execution status of the first task (such as the status of task execution success), thereby enabling the first result to indicate the status of the execution process of the first task under the first label system (such as corresponding to the label representing "task execution success").

[0092] Based on the above paragraph, the first result can at least include the label corresponding to the first task under the first label system, so that the first result can represent the task state (such as the task being successfully executed) presented by the AI ​​Agent when executing the first task.

[0093] In addition, the prompt words corresponding to evaluator 1 can indicate the characteristics of evaluator 1 (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions for evaluator 1 to constrain the classification performance of evaluator 1.

[0094] Furthermore, this application does not limit the implementation of the prompt words corresponding to the evaluator 1. For example, in one possible implementation, the prompt words corresponding to the evaluator 1 can at least satisfy the following constraints: the prompt words are represented according to a preset template, the preset template includes at least one variable part, the at least one variable part includes a first tag system, the system information corresponding to the evaluator 1, the exemption policy corresponding to the evaluator 1, and at least a portion of the business scenario corresponding to the evaluator 1. Wherein, the system information corresponding to the evaluator 1 refers to some characteristics (such as various task states, etc.) existing in the task system itself and related to the evaluator 1. The exemption policy corresponding to the evaluator 1 refers to some problem exemption rules related to the evaluator 1. The business scenario corresponding to the evaluator 1 refers to business scenario information (such as positive examples, negative examples, etc.) related to the evaluator 1.

[0095] Furthermore, this application does not limit the input data of the evaluator 1. For example, the input data may include at least the prompt word corresponding to the evaluator 1 and the task execution information of the first task. Alternatively, the input data may also include user-inputted instructions that can trigger the first task. Furthermore, the input data may include message notifications generated during the execution of the first task (such as task completion notifications). And furthermore, the input data may include system error logs generated during the execution of the first task.

[0096] Furthermore, this application does not limit the implementation of the evaluator 1 described above. For example, the evaluator 1 can be implemented using LLM.

[0097] Based on the relevant content of step 11 above, after obtaining the task execution information of the first task, the task execution information is input into evaluator 1 so that evaluator 1 can analyze the task execution information under the guidance of the corresponding prompt words to obtain a first result. This first result can represent the task status presented by the AI ​​Agent when executing the first task, and thus can indicate to a certain extent whether the first task belongs to the task that requires in-depth analysis. Specifically: if the first result indicates that the AI ​​Agent is in a task execution failure state when executing the first task, the problem of the first task can be located directly based on the system error log, without the need for in-depth analysis of the first task (such as the analysis and processing shown in steps 12 to 14 below), thus determining that the first task does not belong to the task that requires in-depth analysis; however, if the first result indicates that the AI ​​Agent is in a task execution success state when executing the first task, it can be determined that the problem of the first task has not yet been found, and in-depth analysis of the first task is required, thus determining that the first task belongs to the task that requires in-depth analysis.

[0098] Step 12: In response to the first result indicating that the first task was successfully executed, the evaluator 2 is used to analyze the task execution information based on the prompt words corresponding to the evaluator 2 to obtain the second result. The second result indicates the state of the execution process of the first task under the second label system. Different labels in the second label system indicate different first problems.

[0099] The evaluator 2 is configured to perform a deep scan of the task description information (such as execution traces) of the first task in order to identify all potential problems from a macro perspective.

[0100] Therefore, evaluator 2 can refer to an evaluator with general problem identification capabilities (such as LLM or...). Figure 2 The general problem scanner shown); and the evaluator 2 is configured to: under the guidance of the prompt words corresponding to the evaluator 2, analyze the task execution information of the first task (such as general problem scanning processing) to obtain a second result, so that the second result can describe what general problems occurred during the execution of the first task by the AI ​​Agent (such as... Figure 2 The L1 anomaly shown makes the second result able to indicate the state of the execution process of the first task under the second labeling system.

[0101] Based on the above, the second result can at least include the label corresponding to the first task under the second label system, so that the second result can indicate what common problems may have occurred when the AI ​​Agent performed the first task.

[0102] In addition, the prompt words corresponding to evaluator 2 can indicate the characteristics of evaluator 2 (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions of evaluator 2 to constrain the problem scanning performance of evaluator 2.

[0103] Furthermore, this application does not limit the implementation of the prompt words corresponding to the evaluator 2. For example, in one possible implementation, the prompt words corresponding to the evaluator 2 can at least satisfy the following constraints: the prompt words are represented according to a preset template, the preset template includes at least one variable part, the at least one variable part includes a second tag system, the system information corresponding to the evaluator 2, the exemption policy corresponding to the evaluator 2, and at least a portion of the business scenario corresponding to the evaluator 2. Wherein, the system information corresponding to the evaluator 2 refers to some characteristics existing in the task system itself and related to the evaluator 2. The exemption policy corresponding to the evaluator 2 refers to some problem exemption rules related to the evaluator 2. The business scenario corresponding to the evaluator 2 refers to business scenario information related to the evaluator 2 (such as positive examples, negative examples, etc.).

[0104] Furthermore, this application does not limit the input data of the evaluator 2. For example, the input data may include at least the prompt word corresponding to the evaluator 2 and the task execution information of the first task. Alternatively, the input data may also include instructions input by the user that can trigger the first task. Furthermore, the input data may also include the identifier (trace_id) of the execution trajectory recorded in the task execution information.

[0105] Furthermore, this application does not limit the output data of the evaluator 2 (such as the second result mentioned above). For example, the output data may include the label corresponding to the first task under the second labeling system (such as the L1 label), so that the output data can represent the issue type (issue_type) of the common problem that occurred when the first task was executed. Alternatively, the output data may also include the associated information of the problem, such as the name of the called Agent (agent_name), the name of the called tool (tool_name), the analysis result of the problem (analysis), the log evidence of the problem (evidence), and the impact of the problem on the final product (product_impact). The analysis result should at least indicate the cause of the problem (such as the root cause). The log evidence refers to the logs related to the problem, so that the log evidence can accurately describe what operations the AI ​​Agent performed (such as what tools were selected) led to the occurrence of the problem, so that the log evidence can be used as a typical case corresponding to the problem to prove that the AI ​​Agent did indeed encounter the problem. Furthermore, the output data may also include the aforementioned trace_id.

[0106] Therefore, in one possible implementation, the second result may include problem description information and log evidence of a first problem (such as a general problem) that exists during the execution of the first task. The problem description information can indicate the characteristics of the first problem, such as issue_type (e.g., the corresponding tag under the second tagging system), agent_name, tool_name, analysis (e.g., root cause description), product_impact (e.g., problem phenomenon), etc.

[0107] Furthermore, this application does not limit the implementation of the evaluator 2 described above. For example, the evaluator 2 can be implemented using LLM.

[0108] In addition, to better improve the problem detection effect, the aforementioned evaluator 2 may include a tool usage evaluator, a process efficiency evaluator, and a planning reflection evaluator; and each evaluator in the evaluator 2 follows a four-layer evaluation framework from basic to advanced to achieve generalization and deepening of problem discovery.

[0109] For the aforementioned tool usage evaluator, the problems discovered using it fall under the category of issues indicated by the tags corresponding to the tool usage dimension recorded in the second tagging system. Furthermore, this tool usage evaluator is configured to assess the agent's ability to invoke external tools to complete sub-tasks, and the tool invocation lifecycle spans the agent's complete "think-act-observe" closed-loop process in the ReAct mode. The evaluation framework employed by this tool usage evaluator (e.g., ...) Figure 3 The assessment framework shown is an abstraction of capabilities from the four dimensions of "execution-planning-adaptation-cognition" to enable the tool to use an assessor that includes a basic execution layer, a strategic planning layer, a dynamic adaptation layer, and a co-evolutionary layer.

[0110] For the basic execution layer in the tool evaluation tool, it is configured to evaluate the correctness of the basic syntax, parameters, and environment of a single tool call to ensure that the tool can be successfully triggered and executed. Specifically, the call syntax correctness evaluation module in this basic execution layer is configured to evaluate whether the tool call instructions generated by the Agent conform to the specifications and can be correctly parsed and triggered by the engine. The parameter completeness and format evaluation module in this basic execution layer is configured to evaluate whether all necessary parameters are provided during the call and whether the parameter format is correct. The parameter content accuracy evaluation module in this basic execution layer is configured to evaluate whether the parameter values ​​are consistent with the context information, user instructions, or the basic facts of the task. The output transmission integrity evaluation module in this basic execution layer is configured to evaluate whether the output results are completely and losslessly transmitted to the next step of the workflow after successful tool execution. The environment compatibility evaluation module in this basic execution layer is configured to evaluate whether tool execution fails due to environmental problems (such as missing dependencies, insufficient permissions, or configuration errors).

[0111] For the strategy planning layer in the tool usage evaluator mentioned above, this layer is configured to evaluate the rationality and efficiency of the Agent's selection and organization of tool usage in task planning, focusing on "how to use" and "how well it is used." Specifically, the tool selection optimality evaluation module in this layer is configured to assess whether the most functionally suitable and efficient tool has been selected to solve the current subtask, rather than using general or unsuitable tools. The call path efficiency evaluation module in this layer is configured to evaluate whether the tool call sequence is concise and efficient, and whether there are unnecessary duplicate calls, redundant operations caused by state forgetting, or inefficient paths that can be optimized. The parameter strategy rationality evaluation module in this layer is configured to evaluate whether the parameter values ​​themselves reflect a good execution strategy, such as the selection of search keywords and the planning of code logic. The tool reliability awareness evaluation module in this layer is configured to evaluate the AI ​​Agent system's identification and handling of tool defects, such as whether a tool defect is not the Agent's fault, but a good task system should be able to identify and avoid the defect, or record the problem.

[0112] For the dynamic adaptation layer in the tool usage evaluator mentioned above, this layer is configured to evaluate the agent's ability to perform real-time diagnosis, debugging, and correction when faced with tool call failures or unexpected results. Specifically, the error identification and diagnosis evaluation module within this dynamic adaptation layer is configured to evaluate whether the agent can identify the problem and accurately pinpoint the root cause when a tool call returns an error or unexpected result. The remediation strategy effectiveness evaluation module within this dynamic adaptation layer is configured to evaluate the effectiveness of the agent's subsequent actions (such as retrying, correcting parameters, replacing tools, and installing dependencies) in resolving errors, and whether they can ultimately resolve the problem. The post-failure replanning capability evaluation module within this dynamic adaptation layer is configured to evaluate whether the agent can adjust its overall task planning and find alternative paths to achieve the final goal when a single remediation fails or the tool is unavailable.

[0113] For the co-evolutionary layer in the aforementioned tool usage evaluator, this layer is configured to evaluate the collaborative efficiency of tool usage among agents in a multi-agent task system, as well as the strategy optimization capability of the entire task system over time. The agent-to-agent collaborative efficiency evaluation module in this co-evolutionary layer is configured to evaluate the synergy between different types of agents (such as Coordinator Agent and Worker Agent) or multiple agents of the same type (such as multiple Worker Agents) in tool invocation and information transmission, such as whether there are issues like information loss, duplication of work, or inconsistent goals. The shared state and resource management evaluation module in this co-evolutionary layer is configured to evaluate whether there are conflicts, competition, or state pollution issues when multiple agents use tools that affect the shared environment (such as file systems and databases). The long-term strategy optimization evaluation module in this co-evolutionary layer is configured to evaluate whether the AI ​​Agent system can learn from historical task tool usage experience (especially failure experience) to form better tool selection and usage strategies. The tool capability boundary perception assessment module in this co-evolution layer is configured to assess whether the AI ​​Agent system can accurately understand the capability boundaries and limitations of each tool, avoid "forcing the tool to do something it cannot do", or proactively seek user help or decompose tasks when the tool's capabilities are insufficient.

[0114] For the aforementioned process efficiency evaluator, the problems discovered by it belong to the categories indicated by the labels corresponding to the process efficiency dimension recorded in the second labeling system. Furthermore, the process efficiency evaluator is configured to assess whether the agent consumes unnecessary resources (such as time, computing power, and information) while achieving the same quality output. Additionally, the process efficiency evaluator is configured to evaluate according to six principles: eliminating redundancy (doing only once), seeking the optimal path (taking the shortest route), matching resources and tasks (using the right tools), maximizing parallel processing (doing things together if possible), eliminating non-value-adding steps (doing only useful things), and reducing switching and waiting (maintaining fluidity). Moreover, the evaluation framework adopted by the process efficiency evaluator (such as...) Figure 4 The evaluation framework shown is derived by abstracting capabilities from the four dimensions of "execution-process-collaboration-cognition" to make the process efficiency evaluator include a basic execution layer, a process strategy layer, a collaboration strategy layer, and a metacognitive evolution layer.

[0115] For the basic execution layer in the aforementioned process efficiency evaluator, this layer is configured to evaluate the necessity of individual actions. Specifically, the action necessity evaluation module within this layer is configured to assess whether there are redundant or omitting actions that do not contribute value to the final goal or generate redundant information. The tool selection appropriateness evaluation module within this layer is configured to assess whether the Agent has chosen the most suitable tool for the current subtask, avoiding problems such as "using a sledgehammer to crack a nut" or "using a small knife to cut a large tree." The tool interface design quality evaluation module within this layer is configured to assess whether the API (parameters, functions) design of the tool is clear and unambiguous, avoiding Agent confusion, redundant thinking, or incorrect calls. The system basic performance evaluation module within this layer is configured to evaluate the objective performance of the system when performing basic operations (such as file read / write), identifying bottlenecks caused by improper implementation methods.

[0116] For the process strategy layer in the aforementioned process efficiency evaluator, this layer is configured to evaluate the optimality of the overall step sequence. Specifically, the path optimality evaluation module within this layer is configured to assess whether the step sequence for completing the task is the optimal path, and whether there are detours, unnecessary backtracking, or inefficient "generate-correct" patterns. The internal parallelism evaluation module within this layer is configured to identify independent subtasks in the process that can be executed in parallel but are incorrectly processed sequentially, and to detect acceleration potential. The information utilization efficiency evaluation module within this layer is configured to evaluate whether the information generated in the previous step is fully and correctly utilized in the next step, avoiding information loss, duplicate generation, or neglect.

[0117] For the collaboration strategy layer in the aforementioned process efficiency evaluator, this layer is configured to evaluate the efficiency of multi-agent collaboration. Specifically, the task decomposition and assignment evaluation module within this layer is configured to evaluate whether the controlling agent can reasonably decompose complex user intentions into subtasks and assign them to the most suitable specialist agents. The shared state management evaluation module within this layer is configured to evaluate the efficiency of the mechanism for sharing information (state) between agents, and whether it prioritizes memory transfer over unnecessary physical input / output. The resource scheduling rationality evaluation module within this layer is configured to evaluate whether the controlling agent implements the optimal scheduling strategy (such as parallelization) when scheduling multiple agents or parallelizable tools.

[0118] For the metacognitive evolution layer in the aforementioned process efficiency evaluator, this layer is configured to assess the AI ​​Agent system's ability to learn from experience and optimize efficiency strategies. Specifically, the behavior correction capability assessment module within this layer is configured to evaluate whether the AI ​​Agent system can summarize patterns from past execution experiences and proactively avoid repeating mistakes in future tasks. The self-assessment capability assessment module within this layer is configured to assess whether the Agent can evaluate the effectiveness and efficiency of its own plans before or during execution and make timely adjustments. The environmental adaptability assessment module within this layer is configured to assess whether the Agent can flexibly adjust its strategies based on different task contexts (such as the clarity of user instructions and task type), rather than adopting a rigid process.

[0119] For the aforementioned planning reflection evaluator, the problems discovered using it belong to the categories indicated by labels corresponding to the planning reflection dimensions recorded in the second labeling system. Furthermore, this planning reflection evaluator is configured to evaluate the entire mental activity of the AI ​​Agent from receiving user instructions to delivering output, assessing its dynamic cycle capability of "understanding-planning-execution-analysis-adjustment." Additionally, the evaluation framework employed by this planning reflection evaluator (such as...) Figure 5 The assessment framework shown is derived from the ability abstraction of the four dimensions of "execution-planning-adaptation-cognition", so that the planning reflection assessment device includes the execution correctness layer, the planning rationality layer, the dynamic adaptation layer and the metacognitive evolution layer.

[0120] For the execution correctness layer in the aforementioned planning reflection evaluator, this layer is configured to evaluate whether the Agent executes planned atomic operations accurately and without error. This is the foundation for the success of all complex tasks, focusing on "doing it right" rather than "doing it well." Specifically, the tool invocation accuracy evaluation module in this layer is configured to evaluate whether the Agent can provide correct, valid, and formatted parameters to the selected tool and correctly process the tool's output. The sub-Agent invocation accuracy evaluation module in this layer is configured to evaluate whether the leader Agent (such as the CoordinatorAgent) can correctly select and invoke the appropriate sub-Agents to execute specific sub-tasks. The context and state management evaluation module in this layer is configured to evaluate the Agent's ability to maintain, transmit, and utilize key information (state, intermediate products, constraints) in a multi-step task process.

[0121] For the planning rationality layer in the aforementioned planning reflection evaluator, this layer is configured to assess whether the plan formulated by the agent is logically clear, efficient, and closely aligned with the user's goals, focusing on the quality of the transformation from "understanding" to "planning," i.e., "doing it smartly." Specifically, the intent understanding and decomposition evaluation module within this layer is configured to assess whether the agent can accurately and completely understand the explicit and implicit requirements of user instructions and decompose them into a logically reasonable sequence of sub-tasks. The path optimality evaluation module within this layer is configured to assess whether the execution path formulated by the agent (i.e., the sequence of calls between the agent and tools) is the most efficient, and whether there are redundant, circuitous, or parallel steps executed sequentially. The tool optimality evaluation module within this layer is configured to assess whether, for a specific sub-task, the agent has chosen the "most suitable," rather than merely "usable," tool to achieve optimal quality and efficiency. The goal consistency evaluation module within this layer is configured to assess whether, from the start to the end of the task, the behavior of all agents closely revolves around the initial user goal, and whether there is a "goal drift" phenomenon.

[0122] For the dynamic adaptation layer in the aforementioned planning reflection evaluator, this layer is configured to evaluate the agent's ability to cope with unexpected situations (such as tool errors or poor results), focusing on the system's robustness and intelligent adjustment capabilities in uncertain environments, i.e., "knowing how to correct errors." Specifically, the error identification and recovery evaluation module in this dynamic adaptation layer is configured to evaluate whether the agent can identify errors and take effective corrective measures when the tool explicitly returns error or failure signals. The correction strategy correctness evaluation module in this dynamic adaptation layer is configured to evaluate whether the correction strategies adopted by the agent after identifying problems are reasonable, effective, and do not introduce new problems. The reflection effectiveness evaluation module in this dynamic adaptation layer is configured to evaluate whether the agent can proactively evaluate and reflect when there are no explicit error signals but the intermediate results are unsatisfactory, and to identify potential problems. The result quality self-check evaluation module in this dynamic adaptation layer is configured to evaluate whether the leading agent performs a global quality check on the final product in the final stage of the task process to ensure that it meets the initial global goals.

[0123] For the metacognitive evolution layer in the aforementioned planning reflection evaluator, this layer is configured to assess whether the AIAgent system possesses the ability to learn, summarize, and refine better strategies from past successes and failures. This goes beyond single-task fixes, focusing on long-term, cross-task performance improvements—that is, "learning to learn." Specifically, the strategy optimization capability assessment module within this metacognitive evolution layer is configured to evaluate whether the system can dynamically optimize its subsequent behavioral strategies based on continuous feedback (success or failure) during a single task execution. The pattern generalization capability assessment module within this metacognitive evolution layer is configured to assess whether the system can summarize successful patterns or pitfalls from multiple task experiences and generalize them to new, similar tasks, continuously improving long-term performance. The self-awareness accuracy assessment module within this metacognitive evolution layer is configured to assess whether the agent has an accurate understanding of its own capability boundaries, the advantages and disadvantages of its tools, and the limitations of its knowledge.

[0124] Based on the relevant content of step 12 above, when the evaluator 1 detects that the AI ​​Agent has successfully completed the first task, it can be determined that no problems have been found in the first task. Therefore, the task execution information of the first task is input into the evaluator 2, so that the evaluator 2 can perform a general problem analysis on the task execution information under the guidance of the corresponding prompt words to obtain a second result. This second result can indicate the general problems that the AI ​​Agent may have encountered in executing the first task and their log evidence. Thus, the second result can indicate the state of the execution process of the first task under the second label system, and further, the second result can indicate which labels under the second label system indicate the problems that the execution process of the first task may have encountered and their log evidence.

[0125] Step 13: Using evaluator 3, based on the prompt words corresponding to evaluator 3, analyze the task execution information and the second result to obtain the third result. The third result indicates the state of the execution process of the first task under the third label system. Different labels in the third label system indicate different second problems.

[0126] Evaluator 3 is configured to refine the detected common problems.

[0127] Therefore, evaluator 3 can refer to an evaluator with fine-grained problem identification capabilities (such as LLM or...). Figure 2(as shown in the fine-grained problem identifyer); and the evaluator 3 is configured to: under the guidance of the prompt words corresponding to the evaluator 3, analyze the output data of the evaluator 2 based on the task execution information of the first task to obtain a third result, so that the third result can indicate which label each general problem identified by the evaluator 2 is matched to under the third label system, thereby enabling the third result to indicate the state of the execution process of the first task under the third label system, such as the state of {<name, fine-grained label 1>, <state, occurrence>, <reason, problem description information and log evidence of all general problems matching fine-grained label 1>}.

[0128] As can be seen, the aforementioned third result may include the labels matched by each of the first problems indicated by the second result under the third label system, so that the third result can indicate the label corresponding to the first task under the third label system, thereby enabling the third result to indicate what fine-grained problems may have occurred when the AI ​​Agent performed the first task.

[0129] In addition, the prompt words corresponding to evaluator 3 can indicate the characteristics of evaluator 3 (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions of evaluator 3 to constrain the refinement performance of evaluator 3.

[0130] Furthermore, this application does not limit the implementation of the prompt words corresponding to the evaluator 3. For example, in one possible implementation, the prompt words corresponding to the evaluator 3 can at least satisfy the following constraints: the prompt words are represented according to a preset template, which includes at least one variable part, including a third tagging system, system information corresponding to the evaluator 3, exemption policy corresponding to the evaluator 3, and at least a portion of the business scenario corresponding to the evaluator 3. Wherein, the system information corresponding to the evaluator 3 refers to some characteristics existing in the task system itself and related to the evaluator 3. The exemption policy corresponding to the evaluator 3 refers to some problem exemption rules related to the evaluator 3. The business scenario corresponding to the evaluator 3 refers to business scenario information related to the evaluator 3 (such as positive examples, negative examples, etc.).

[0131] Furthermore, this application does not limit the input data of the evaluator 3. For example, the input data may include at least the prompt words corresponding to the evaluator 3, the task execution information of the first task, and the output data of the evaluator 2 (such as the second result mentioned above). Alternatively, the input data may also include the identifier of the execution trajectory recorded in the task execution information.

[0132] Furthermore, this application does not limit the output data of the evaluator 3 (such as the third result mentioned above). For example, the output data may include the label corresponding to the first task under the third labeling system, so that the output data can indicate what kind of fine-grained problem occurred when the first task was executed. Also, the output data may include some attributes of the "corresponding label," such as the attribute that uses "problem description information and log evidence of all general problems matching fine-grained label 1" as its attribute value. Therefore, the third result can indicate what kind of fine-grained problem occurred when the first task was executed and its reasons.

[0133] Furthermore, this application does not limit the implementation of the evaluator 3 described above. For example, the evaluator 3 can be implemented using LLM.

[0134] Based on the relevant content of step 13 above, after using evaluator 2 to determine the problem description information and log evidence of the general problem that the AI ​​Agent may have encountered in performing the first task, evaluator 3 infers from the problem description information and log evidence that, in the third labeling system (such as...), the problem description information and log evidence are used to infer the problem in the third labeling system (such as...). Figure 2 The labels in the L2 label tree shown are matched with the general problem to obtain a third result, so that the third result can represent the fine-grained problems that the AI ​​Agent may have encountered in performing the first task and their related content (such as the reasons for identifying the fine-grained problem).

[0135] Step 14: Using evaluator 4, the third result is analyzed based on the prompt words corresponding to evaluator 4 to obtain a fourth result, which indicates the relationship (such as causal relationship) between the different problems described by the third result.

[0136] The evaluator 4 is configured to: verify the accuracy of detected problems (such as fine-grained problems indicated by the third result mentioned above) to filter false alarms (such as problems specified by the exemption policy); further attribute the causes by combining the system description data of the task system to better determine the root causes of some problems; and support the fusion and construction of causal analysis chains to determine the causal relationships between different problems.

[0137] As can be seen, evaluator 4 refers to an evaluator with root cause analysis and fusion functions (such as LLM or...). Figure 2(The root cause analysis and fusion apparatus shown); and the evaluator 4 is configured to: under the guidance of the prompt words corresponding to the evaluator 4, analyze the output data (such as the third result) of the evaluator 3 to obtain a fourth result, so that the fourth result can describe whether the fine-grained problem indicated by the third result is a true defect of the AI ​​Agent, the root cause of the fine-grained problem, and the causal relationship between different fine-grained problems, so that the fourth result can describe the root cause of some problems that occur during the AI ​​Agent's execution of the first task and the causal relationship between different problems.

[0138] Based on the foregoing, in one possible implementation, the fourth result may include a problem verification result, a root cause analysis result, and a problem fusion result. The problem verification result indicates whether the detected problem (e.g., a fine-grained problem indicated by the third result) is a valid problem (e.g., a problem that describes a true defect in the AI ​​Agent) or an invalid problem (e.g., a problem specified by an exemption policy). This verification result is determined based on the system information and exemption policy recorded in the prompts corresponding to the evaluator 4. The root cause analysis result indicates the root cause of the detected valid problems (e.g., unclear prompts for the Agent, insufficient LLM platform capabilities used in the AI ​​Agent, or inherent tool defects) and provides modification suggestions. The problem fusion result indicates the causal relationship between the detected valid problems. This fusion result refers to the evaluator 4 deduplicating multiple valid problems discovered in the first task and combining them with the execution traces of the first task (e.g., ...). Figure 2 The attribution chain constructed by the structured trace shown is used to present the effective questions in the order of Agent call trajectories, so that the fusion result can accurately describe the relationship between different effective questions (such as the relationship that question 2 occurred after question 1 occurred).

[0139] In addition, the prompt words corresponding to evaluator 4 can indicate the characteristics of evaluator 4 (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions of evaluator 4 to constrain the classification performance of evaluator 4.

[0140] Furthermore, this application does not limit the implementation of the prompt words corresponding to the evaluator 4. For example, in one possible implementation, the prompt words corresponding to the evaluator 4 can at least satisfy the following constraints: the prompt words are represented according to a preset template, which includes at least one variable part. The at least one variable part includes the system information corresponding to the evaluator 4, the exemption policy corresponding to the evaluator 4, and at least a portion of the business scenario corresponding to the evaluator 4. Wherein, the system information corresponding to the evaluator 4 refers to some characteristics existing in the task system itself and related to the evaluator 4. The exemption policy corresponding to the evaluator 4 refers to some problem exemption rules related to the evaluator 4. The business scenario corresponding to the evaluator 4 refers to business scenario information related to the evaluator 4.

[0141] Furthermore, this application does not limit the input data of the evaluator 4. For example, the input data may include at least the prompt words corresponding to the evaluator 4 and the output data of the evaluator 3 (such as the third result). Alternatively, the input data may also include the execution trace in the task execution information of the first task. Furthermore, the input data may also include instructions input by the user that can trigger the first task. And furthermore, the input data may also include the identifier (trace_id) of the execution trajectory recorded in the task execution information.

[0142] Furthermore, this application does not limit the output data of the evaluator 4 (such as the fourth result mentioned above). For example, the output data may include a valid question chain. This valid question chain not only indicates the relationship between different valid questions (such as causal relationships), but also indicates some relevant content of each valid question (such as root cause, its position in the valid question chain, log evidence, etc.). As another example, the output data may also include the aforementioned trace_id.

[0143] Furthermore, this application does not limit the implementation of the evaluator 4 described above. For example, the evaluator 4 can be implemented using LLM.

[0144] Based on the relevant content of step 14 above, after obtaining the fine-grained problems output by evaluator 3, evaluator 4 can verify the accuracy of these problems, analyze the root causes, and construct a causal analysis chain to obtain a fourth result, so that the fourth result can better describe the problems that occurred when the AI ​​Agent performed the first task.

[0145] Step 15: Using evaluator 5, the fourth result is analyzed based on the prompt words corresponding to evaluator 5 to obtain the analysis result.

[0146] The evaluator 5 is configured to: find the top-ranked issues, generate an automated analysis report, and add typical cases. It should be noted that this application does not limit the method for determining this ranking; for example, the method for determining the ranking can be determined based on the issue detection requirements of the actual application scenario.

[0147] Research has revealed that in some scenarios, the granularity of problem classification described by the third-party labeling system may still be relatively coarse. Therefore, to further improve problem detection performance, semantic clustering can be used to obtain finer-grained problem classification results. For example, when the third-party labeling system includes the label "image missing," all valid problems matching this label can be semantically clustered to obtain problem clusters, allowing different problem clusters to represent different types of image missing (such as missing barcode images, missing high-resolution images, etc.).

[0148] Based on the above research, in one possible implementation, the evaluator 5 can be configured to at least: first, semantically cluster all valid questions matching the same L2 label to obtain some question clusters; then, based on the relevant information of each question recorded in each question cluster (such as question description information, log evidence, etc.), determine the relevant information of each question cluster (such as question type, question description information, typical cases, replay links, etc.); and then automatically generate an analysis report based on the relevant information of these question clusters to push to relevant objects (such as...). Figure 2 (Objects 201 and 202 shown).

[0149] Therefore, evaluator 5 can refer to an evaluator with problem clustering capabilities (such as LLM or...). Figure 2 The problem clusterer shown); and the evaluator 5 is configured to: under the guidance of the prompt words corresponding to the evaluator 5, analyze the valid problems output by the evaluator 4 to obtain analysis results (such as analysis reports), so that the analysis results can describe what problems mainly occurred during the AI ​​Agent's execution of the first task.

[0150] In addition, the prompt words corresponding to evaluator 5 can indicate the characteristics of evaluator 5 (such as the functions implemented, the constraints satisfied, etc.), so that the prompt words can serve as the core instructions of evaluator 5 to constrain the performance of evaluator 5.

[0151] Furthermore, this application does not limit the implementation of the prompt words corresponding to the evaluator 5. For example, in one possible implementation, the prompt words corresponding to the evaluator 5 can at least satisfy the following constraints: the prompt words are represented according to a preset template, the preset template includes at least one variable part, the at least one variable part includes the system information corresponding to the evaluator 5, the exemption policy corresponding to the evaluator 5, and at least a portion of the business scenario corresponding to the evaluator 5. Wherein, the system information corresponding to the evaluator 5 refers to some characteristics existing in the task system itself and related to the evaluator 5. The exemption policy corresponding to the evaluator 5 refers to some problem exemption rules related to the evaluator 5. The business scenario corresponding to the evaluator 5 refers to business scenario information related to the evaluator 5.

[0152] Furthermore, this application does not limit the implementation of the evaluator 5 described above. For example, the evaluator 5 can be implemented using LLM.

[0153] Based on the relevant content of steps 11 to 15 above, the attribution workflow provided by this application for evaluating a task system (such as an AI Agent) may include: first, using evaluator 1 to analyze the task status from the task execution information of the first task to determine whether the first task was successfully executed; then, using evaluator 2 to analyze the general problems that the task system may have encountered when executing the first task from the task execution information when the first task is determined to be successfully executed; next, using evaluator 3 to determine the L2 labels matching each general problem to determine the fine-grained problems that the task system may have encountered when executing the first task; then, using evaluator 4 to analyze the accuracy, root causes, and construct the causal chain of these fine-grained problems; finally, using evaluator 5 to perform semantic clustering on the problems output by evaluator 4 to generate an analysis report, thus enabling automatic detection of problems in the AI ​​Agent process.

[0154] It should be noted that, for Figure 2In the attribution workflow shown, in some scenarios, the fine-grained problem identifyr and the root cause analysis and fusion unit can be combined into one stage, and a more powerful LLM can be used to complete this stage. This allows a more powerful LLM to complete the entire process from labeling to root cause analysis in one go, but this may place higher demands on the model's capabilities and reduce interpretability. Alternatively, in some scenarios, the fine-grained problem identifyr can be removed, allowing the root cause analysis and fusion unit to be configured to analyze common problems that have already been detected. Furthermore, in some scenarios, the method of injecting the system characteristics of the AI ​​Agent into each evaluator using system information from prompt words can be replaced with Retrieval-augmented Generation (RAG). Specifically, the system description data of the AI ​​Agent is stored in a database, so that each evaluator can subsequently retrieve its associated content from the database using the RAG method. In this case, the prompt words for each evaluator no longer record system information, overcoming the shortcomings caused by excessive content in the prompt words.

[0155] In addition, in some scenarios (such as scenarios where new problems constantly emerge), in order to better improve the problem detection effect, the above-mentioned problem detection method for task systems may also include the following steps: based on the output data of the first evaluator in the first model (such as the second result above or... Figure 2 The L1 anomaly description shown updates the label system (such as the third label system) corresponding to the second evaluator in the first model. The first evaluator (such as evaluator 2) is configured to analyze the above task execution information based on the label system corresponding to the first evaluator to obtain the output data. The second evaluator (such as evaluator 3) is configured to analyze the output data and the task execution information based on the label system corresponding to the second evaluator, so that the updated label system contains labels that match the problems indicated by the output data. In this way, the third label system can be automatically updated with a large number of detected common problems to ensure that the third label system can cover all kinds of problems as comprehensively as possible, so as to overcome the problems caused by the continuous emergence of new problems, so as to realize the self-evolution of the third label system and better adapt to the problem detection needs in the scenario of rapid iteration of problem patterns.

[0156] It should be noted that this application does not limit the implementation of the above-mentioned step of "updating the label system corresponding to the second evaluator in the first model based on the output data of the first evaluator in the first model". For example, it can specifically be: firstly, semantically clustering all the questions indicated by the output data to obtain some question clusters; then, determining the labels (such as L2 labels) corresponding to these question clusters automatically or manually; and then, updating the third label system based on these labels so that the updated third label system includes these labels.

[0157] As can be seen, in one possible implementation, the update process of the label system (such as the third label system) corresponding to the second evaluator in the first model may include: after obtaining the output data of the first evaluator in the first model, performing semantic clustering based on the problems indicated by the output data (such as some or all general problems) to obtain at least one cluster, so that the semantics of different problems in the same cluster are as similar as possible, and the semantics of problems belonging to different clusters are as dissimilar as possible; updating the label system corresponding to the second evaluator based on the at least one cluster, the updated third label system including the labels corresponding to each cluster, so that a large number of detected general problems can be classified in a more granular way by means of semantic clustering, so as to ensure that the third label system can cover all detected problems as comprehensively as possible.

[0158] It should also be noted that this application does not limit the timing of the update of the aforementioned third labeling system; for example, it can be updated according to a certain period. Thus, in one possible implementation, the update process of the third labeling system can be as follows: In response to detecting that the time interval between the current moment and the last update time of the third labeling system reaches a preset threshold, it can be determined that the update condition of the third labeling system has been met. Therefore, all general questions (such as questions carrying L1 labels) detected by evaluator 2 within the time interval [last update time, current moment] can be semantically clustered to obtain some question clusters; then, the labels corresponding to these question clusters can be determined automatically or manually; then, the third labeling system is updated based on these labels so that the updated third labeling system includes these labels.

[0159] Furthermore, to improve flexibility, when the output data of the first evaluator (e.g., evaluator 2) in the first model indicates that there is at least one first problem (e.g., a general problem) in the execution process of the first task, the update process of the label system corresponding to the second evaluator (e.g., evaluator 3) in the first model may include: in response to the existence of a third problem that meets preset conditions (e.g., a problem that cannot be matched with an L2 label) in the at least one first problem, it can be determined that a second problem matching the third problem cannot be found in the label system (e.g., a third label system) corresponding to the second evaluator via the second evaluator, thereby determining that a new problem has emerged. Therefore, the label system corresponding to the second evaluator is updated according to the third problem so that the updated label system includes the label matching the third problem, thereby adding the L2 label corresponding to the new problem to the updated label system compared to the previous label system. This enables timely self-evolution of the third label system to ensure that the third label system can adapt to the iteration speed of the problem pattern as quickly as possible.

[0160] Regarding the aforementioned preset conditions, these preset conditions refer to pre-set conditions used to screen newly emerging questions; and this application does not limit these preset conditions. For example, when the third question detected by evaluator 2 meets the preset conditions, the preset conditions can indicate that the evaluator 3 cannot find a second question that matches the third question in the third label system.

[0161] Furthermore, this application does not limit the update method of the aforementioned third label system. For example, it can utilize a pre-built machine learning model with this update function (such as...). Figure 2 The implementation involves a dynamic tagging system generator (also known as the "third model"). This generator is configured to: first, semantically cluster a large number of problem cases with L1 tags to obtain problem clusters; then, obtain the L2 tags corresponding to each problem cluster through a certain method (such as automatic generation by an LLM or human-computer interaction by relevant personnel); and update the third tagging system using these L2 tags to achieve self-evolution of the problem cognition scope. The third model can be an LLM.

[0162] Research has found that when using an LLM (Limited Ledger Model) to implement a refereeing strategy indicated by its prompts, the strategy typically passively depends on the intelligence of the LLM. A more intelligent LLM results in a better strategy, while a less intelligent LLM results in a worse strategy. Furthermore, as shown in Table 1 below, different refereeing strategies have varying degrees of complexity, which may lead to different requirements on the LLM's capabilities. This, in turn, results in varying degrees of dependence on the LLM's intelligence, making any judgment accuracy independent of the complexity of the refereeing strategy unreasonable.

[0163] The research also found that in some scenarios, cue word engineering can be used to decompose and reduce the dimensionality of relatively complex refereeing strategies indicated by cue words, thereby reducing the dependence of the refereeing strategy on the intelligence level of LLM, so as to achieve controllable and reliable accuracy on refereeing strategies of different complexities.

[0164] Based on the above research, to further improve the problem detection effect, the problem detection method for task systems can also include: after updating the prompt word (such as the prompt word corresponding to the first model or the prompt word corresponding to any evaluator), decomposing the prompt word into multiple referee strategies, so as to decompose the relatively complex referee strategy indicated by the prompt word into some relatively simple referee strategies; determining the model capability requirements corresponding to each referee strategy according to the strategy type of each referee strategy; determining the optimization strategy corresponding to each referee strategy (such as the optimization strategy shown in Table 1 above) according to the model capability requirements corresponding to each referee strategy; and processing the prompt word according to the optimization strategy, so as to optimize the content corresponding to each referee strategy in the prompt word by applying the corresponding optimization strategy, so that the referee strategy indicated by the optimized prompt word is less dependent on the intelligence of the LLM than the referee strategy indicated by the unoptimized prompt word, so that the LLM can better reduce false recalls and generalization ability under the guidance of the optimized prompt word, thereby making the problem detection process under the guidance of the optimized prompt word exhibit better performance and improving the problem detection effect.

[0165]

[0166] Table 1. Relevant content of different types of referee strategies

[0167] Based on the problem detection method for task systems provided in the embodiments of this application, the embodiments of this application also provide a problem detection system, which includes a first engine (such as...). Figure 2 The meta-support system shown) and the second engine (such as Figure 2 (As shown in the attribution workflow), the second engine is configured to execute at least S2 as described above, and the first engine is configured to execute at least S3 as described above. The first engine is configured to perform self-evolution (e.g., prompt word updates, L2 label tree updates, etc.) in response to the second engine. The second engine is configured to evaluate the AI ​​Agent (e.g., task status, labels at each level, etc.). Furthermore, the implementation methods of these two engines are described in the relevant content above; for brevity, they will not be repeated here.

[0168] As can be seen, this application provides a self-evolving problem detection system that can automatically detect and process problems in a self-evolving AI Agent process, and the system has the features shown in (I) to (VII) below.

[0169] (i) The above-mentioned problem detection system can automatically evolve its problem detection capability and problem cognition scope with the iteration of AI Agent through a feedback loop.

[0170] (ii) The aforementioned problem detection system includes a meta-support system for dynamically generating evaluation capabilities and an attribution pipeline for performing multi-stage analysis. The two form a closed loop through a dynamically generated evaluator and a feedback-updated labeling system. It can be seen that the meta-support system functions as an evolutionary engine (such as the first engine mentioned above); the attribution pipeline functions as an execution engine (such as the second engine mentioned above).

[0171] (iii) The above-mentioned problem detection system can receive data (such as logs, product files, etc.) generated by the AI ​​Agent to perform tasks as input, process it through the evaluators in the attribution pipeline, and output a structured problem report (such as the analysis results mentioned above). These evaluators themselves are dynamically maintained by the meta support system according to the latest system design and knowledge of the AI ​​Agent.

[0172] (iv) The meta-support system in the aforementioned problem detection system is the core of achieving the "self-evolution" characteristic. Its responsibility is to dynamically manufacture and upgrade the capabilities required by each evaluator in the attribution pipeline. Specifically, the evaluator automatic construction module in this meta-support system is configured to automatically generate prompts for each evaluator, so that these prompts serve as the core instructions for each evaluator's judgment, thereby achieving self-evolution of evaluation capabilities. Furthermore, the automatic optimization of evaluator capabilities achieved through this meta-support system has at least the following characteristics: ① It reduces false recalls through a static approach of evidence-driven methods and linking process with product performance; it reduces false recalls through a dynamic approach of refining problem pattern definitions, introducing a complete system context, and improving exemption rules; ② It improves generalization capabilities through a static approach of covering the problem definition to the entire lifecycle; and it improves generalization capabilities through a dynamic approach of fallback labels and dynamically learning new rules. In addition, the dynamic label system generation module in this meta-support system is configured to solve the problem of the fixed label system becoming ineffective due to the continuous emergence of new problems by performing semantic clustering on a large number of problem cases with L1 labels, thereby achieving self-evolution of the problem recognition scope. It is evident that this problem detection system learns and identifies emerging problem patterns by semantically clustering unknown problems discovered in the attribution pipeline and dynamically creating and updating a problem labeling system (such as a third labeling system) in conjunction with manual review and naming.

[0173] (V) The attribution pipeline in the above-mentioned problem detection system is the core execution process for in-depth analysis of a single task log, which includes the following five stages: Stage 1: Use a task status classifier to classify the final result of the task (such as success or failure) and quickly filter out the tasks that need in-depth analysis; Stage 2: Use a general problem scanner to perform a deep scan of the entire link log to find all suspected problems at the macro level, so as to obtain a list of suspected problems carrying L1 labels and log evidence; Stage 3: Use a fine-grained problem identifier to further refine the general problem, so as to infer the L2 label that best matches the general problem with the L1 label from the problem description information and log evidence corresponding to the L1 label; Stage 4: Use root cause analysis and fusion to verify the accuracy of the detected fine-grained problems, analyze the root cause, and construct the problem causal relationship chain; Stage 5: Use a problem clusterer to perform semantic clustering on all verified problems of a batch of tasks to obtain a more refined problem category than the L2 label, so as to generate an analysis report.

[0174] (vi) The formation of the evolutionary closed loop in the above-mentioned problem detection system is as follows: After the problem clusterer completes the report generation, the output data of the problem clusterer (especially newly discovered problem clusters and typical cases, etc.) will be fed back to the meta-support system as input, so that relevant personnel can review the report to update the exemption rules, positive examples, negative examples, and name new problem clusters to update the dynamic labeling system (such as...). Figure 2 (As shown in the L2 label tree). This forms a complete closed loop of "analysis → discovery → feedback → learning → evolution". It is evident that this problem detection system can feed back the problem clustering analysis results from the attribution pipeline to the meta-support system, serving as the basis for updating its knowledge base (including labeling system, exemption rules, positive examples, negative examples, etc.) to drive the iterative upgrade of the next round of evaluation capabilities.

[0175] (vii) The above-mentioned problem detection system defines an evaluator template and dynamically injects external knowledge such as the latest system design documents of the AI ​​Agent, exemption rules and anti-pattern examples (such as negative examples) into the template at runtime, thereby automatically generating an evaluator that can adapt to the latest state of the AI ​​Agent.

[0176] Based on the aforementioned content regarding the problem detection system, it can be concluded that this system possesses the following advantages: ① Because the system's evaluation capabilities can automatically evolve with the iteration of the AI ​​Agent itself, there is no need to rewrite the evaluation code for each change, ensuring the system's continuous availability and high scalability. This not only achieves self-evolution of the evaluation system but also significantly reduces maintenance costs; ② By dynamically generating evaluation logic, the system eliminates its reliance on static ground truth, effectively evaluating complex AI Agents in real production environments and achieving benchmark-free evaluation of dynamic and complex AI Agents; ③ The system can shorten the manual log analysis process of several hours or even days to an automated process completed within hours, directly pinpointing the root cause of problems with an accuracy rate of over 80%, thereby significantly improving attribution efficiency and accuracy; ④ Through a self-evolving tagging system, the system can continuously expand its problem cognition boundaries, continuously discovering and classifying unknown and emerging problems, thus achieving continuous discovery and learning of unknown problems; ⑤ The system targets AI... The system leverages the characteristics of rapid iteration in agent architecture and problem patterns to provide a method for automatically evolving its evaluation capabilities and knowledge system, overcoming the constraints of manual maintenance of evaluation logic and addressing the problem of static and inflexible evaluation systems. ⑥ The system can effectively evaluate the execution process of AI agents in an open environment without pre-setting a fixed Ground Truth, thus addressing the issue of dynamic evaluation benchmarks. ⑦ The system changes the reliance on manual review of AI agent internal execution logs, enabling automated and accurate process attribution for failed or inefficient tasks, thus addressing the problem of inefficient attribution. ⑧ The system can detect process efficiency and quality issues that, while resulting in qualified final products, contain redundant steps, invalid attempts, or other problems that cannot be detected through result evaluation, thus addressing the problem of overlooked hidden issues.

[0177] Based on the problem detection method for task systems provided in the embodiments of this application, the embodiments of this application also provide a problem detection device for task systems, which will be described below in conjunction with... Figure 6 Explanation and clarification will be provided. Among them, Figure 6 This is a schematic diagram of a problem detection device for a task system provided in an embodiment of this application. It should be noted that for technical details of the problem detection device for a task system provided in this embodiment, please refer to the relevant content of the problem detection method for task systems described above.

[0178] like Figure 6 As shown, the problem detection device 600 for a task system provided in this application embodiment includes:

[0179] Acquisition unit 601 is used to acquire task execution information generated by the task system executing the first task;

[0180] Analysis unit 602 is used to analyze the task execution information using a first model based on the prompt words corresponding to the first model to obtain analysis results, the analysis results including problems encountered by the task system in executing the first task;

[0181] The update unit 603 is used to update the prompt words based on the analysis results.

[0182] In one possible implementation, the updating unit 603 is further configured to: update the prompt word according to the updated system description data of the task system in response to detecting that the task system has been updated.

[0183] In one possible implementation, the first model is configured to execute an attribution workflow to obtain the analysis results; the first model includes multiple evaluators, each of which is configured to execute a different stage of the attribution workflow; the prompt words corresponding to the first model include prompt words corresponding to each of the evaluators.

[0184] In one possible implementation, the updating unit 603 is further configured to: update the label system corresponding to the second evaluator in the first model based on the output data of the first evaluator in the first model, wherein the first evaluator is configured to analyze the task execution information to obtain the output data based on the label system corresponding to the first evaluator, and the second evaluator is configured to analyze the output data and the task execution information based on the label system corresponding to the second evaluator.

[0185] In one possible implementation, the update unit 603 is specifically configured to: perform semantic clustering based on the first question indicated by the output data to obtain at least one cluster; and update the label system corresponding to the second evaluator based on the at least one cluster, wherein the updated label system includes the labels corresponding to each cluster.

[0186] In one possible implementation, the acquisition unit 601 is specifically configured to: in response to receiving the original log generated by the task system executing the first task according to a preset mode, perform structured processing on the original log to obtain a structured log; map the structured log according to the preset mode to obtain an execution trace; and determine the task execution information based on the execution trace.

[0187] In one possible implementation, the updating unit 603 is specifically configured to: update the prompt word according to the audit result provided for the analysis result in response to receiving the audit result, wherein the audit result indicates the correspondence between different contents in the analysis result and different parts in the prompt word.

[0188] In one possible implementation, the problem detection device 600 for the task system satisfies at least the following constraints: the first task is any task in the task set; the task execution information includes the execution trace of the first task and at least a portion of the product file of the first task; each evaluator in the first model is a large language model; the prompt words are represented according to a preset template, the preset template includes at least one variable part, the at least one variable part includes tag information, system information, exemption policy, and at least a portion of the business scenario, the tag information includes at least a portion of a first tag system, a second tag system, and a third tag system, and the business scenario includes at least a portion of positive examples and negative examples.

[0189] Based on the above-mentioned content regarding the problem detection device 600 for the task system, the working principle of the device 600 includes: firstly, acquiring task execution information (such as logs, products, etc.) generated by the task system (such as an AI Agent) executing the first task, so that the task execution information can describe the characteristics presented by the AI ​​Agent executing the first task, such as what processes were experienced and what data was generated in each process; then, using a first model (such as LLM) to analyze the task execution information based on the prompt words corresponding to the first model to obtain analysis results, so that the analysis results can at least describe the problems that occurred in the task system executing the first task (such as decision errors, redundant steps, etc.), so as to realize the automatic detection of AI Agent problems (such as process problems) by means of machine learning models. The prompt words corresponding to the first model are iteratively optimized based on the analysis results of historical tasks. This ensures that the prompt words can describe as accurately and comprehensively as possible the constraints (such as problem recognition scope, positive examples, negative examples, exemption strategies, etc.) that need to be met when performing problem detection (such as process problem detection) on the AI ​​Agent. As a result, the first model exhibits better problem detection performance under the guidance of the prompt words. In this way, the problem detection performance presented by the first model can be automatically iteratively optimized through the iterative optimization of the prompt words, so as to achieve self-evolution of problem detection performance and better improve the problem detection effect.

[0190] In addition, this application also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the problem detection method for a task system provided in this application.

[0191] See Figure 7The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0192] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0193] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0194] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0195] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0196] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the problem detection method for a task system provided in this application.

[0197] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0198] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0199] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0200] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0201] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0203] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0204] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0205] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0206] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0207] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0208] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0209] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0210] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A problem detection method for a task system, characterized in that, The method includes: Obtain the task execution information generated by the task system executing the first task; Using a first model based on the prompt words corresponding to the first model, the task execution information is analyzed to obtain analysis results, including problems encountered by the task system in executing the first task; Update the prompt words based on the analysis results.

2. The method according to claim 1, characterized in that, The method further includes: In response to the detection of an update to the task system, the prompt word is updated based on the updated system description data of the task system.

3. The method according to claim 1, characterized in that, The first model is configured to execute an attribution workflow to obtain the analysis results; The first model includes multiple evaluators, each configured to perform a different stage of the attribution workflow. The prompt words corresponding to the first model include the prompt words corresponding to each of the evaluators.

4. The method according to claim 1, characterized in that, The method further includes: The labeling system corresponding to the second evaluator in the first model is updated based on the output data of the first evaluator in the first model. The first evaluator is configured to analyze the task execution information to obtain the output data based on the labeling system corresponding to the first evaluator. The second evaluator is configured to analyze the output data and the task execution information based on the labeling system corresponding to the second evaluator.

5. The method according to claim 4, characterized in that, The step of updating the label system corresponding to the second evaluator in the first model based on the output data of the first evaluator in the first model includes: Based on the questions indicated by the output data, semantic clustering is performed to obtain at least one cluster; Based on the at least one cluster, the label system corresponding to the second evaluator is updated, and the updated label system includes the labels corresponding to each cluster.

6. The method according to claim 1, characterized in that, The step of obtaining the task execution information generated by the task system executing the first task includes: In response to receiving the raw log generated by the task system executing the first task according to a preset mode, the raw log is processed into a structured log. The structured logs are mapped according to the preset pattern to obtain execution traces; Based on the execution trace, the task execution information is determined.

7. The method according to claim 1, characterized in that, Updating the prompt words based on the analysis results includes: In response to receiving an audit result for the analysis results, the prompt word is updated based on the audit result, wherein the audit result indicates the correspondence between different contents in the analysis results and different parts of the prompt word.

8. The method according to any one of claims 1-7, characterized in that, The method must satisfy at least the following constraints: The first task is any task in the task set; The task execution information includes the execution trace of the first task and at least a portion of the product file of the first task; In the first model, each evaluator is a large language model; The prompt words are represented according to a preset template, which includes at least one variable part. The at least one variable part includes tag information, system information, exemption policy, and at least a part of the business scenario. The tag information includes at least a part of a first tag system, a second tag system, and a third tag system. The business scenario includes at least a part of positive examples and negative examples.

9. A problem detection device for a task system, characterized in that, include: The acquisition unit is used to acquire task execution information generated by the task system executing the first task; The analysis unit is used to analyze the task execution information using a first model based on the prompt words corresponding to the first model to obtain analysis results, the analysis results including problems encountered by the task system in executing the first task; An update unit is used to update the prompt words based on the analysis results.

10. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-8.

11. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method of any one of claims 1-8.

Images