Intelligent agent hallucination positioning method, device, medium and program product

Abstract

The application relates to the technical field of artificial intelligence, and discloses an agent hallucination positioning method, equipment, a medium and a program product. The method comprises the following steps: acquiring trajectory data of an agent, wherein the trajectory data comprises step information of N execution steps, N is an integer greater than 1; performing abnormality detection on each execution step in the N execution steps through multiple abnormality detection modes respectively to generate detection results of each execution step, and the object types detected by different abnormality detection modes are different; calculating an abnormality score of each step based on the detection results of different abnormality detection modes corresponding to each step; and determining an abnormal step in the N execution steps based on the abnormality score of each step, wherein the abnormal step comprises a hallucination starting step. In this way, the hallucination source can be positioned more accurately, and the multiple abnormality detection modes detect different object types respectively, so that the missed detection caused by a single detection mode can be avoided, and the accuracy of hallucination source positioning is further improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, medium, and program product for illusory localization of intelligent agents. Background Technology

[0002] With the development of large language models (LLM), agents based on large language models have been widely used in scenarios involving automated task execution and multi-turn tool invocation.

[0003] However, large language models suffer from the illusion problem, which involves generating content that is inconsistent with reality or prior evidence. Since the agent's execution trajectory includes multiple interdependent steps, errors in one round can propagate along the trajectory, affecting decisions in subsequent steps and ultimately leading to task failure.

[0004] Therefore, in order to improve the accuracy of task execution by intelligent agents, it is necessary to identify and correct the steps that generate hallucinations. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, medium, and program product for locating intelligent agent illusions, so as to improve the accuracy of locating the source of intelligent agent illusions.

[0006] The first aspect of this application provides a method for locating hallucinations in an intelligent agent, applied to an electronic device. The method includes: acquiring trajectory data of the intelligent agent, wherein the trajectory data includes step information for N execution steps, where N is an integer greater than 1; performing anomaly detection on each of the N execution steps using multiple anomaly detection methods, generating a detection result for each execution step, wherein different anomaly detection methods detect different object types; calculating anomaly scores for each step based on the detection results of different anomaly detection methods corresponding to each step; and determining the anomalous steps among the N execution steps based on the anomaly scores for each step, wherein the anomalous steps include the hallucination initiation step.

[0007] In the embodiments of this application, each execution step in the trajectory is independently detected by multiple anomaly detection methods, and an anomaly score is calculated based on the detection method that determines the anomaly of each step. Finally, the abnormal step (the hallucination initiation step) is determined based on the anomaly score, which can more accurately locate the hallucination source.

[0008] Furthermore, multiple anomaly detection methods detect different object types, and different signal dimensions complement each other. Any detection method can generate a signal when it captures an anomaly, and any type of hallucination can be captured by at least one detection method, avoiding missed detections caused by a single detection method.

[0009] In one possible implementation of the first aspect above, based on the detection results of different anomaly detection methods corresponding to each step, the anomaly score of each step is calculated, including: determining the anomaly score of the execution step according to the number of anomaly detection methods that are judged as anomalies in each execution step; and / or, determining the anomaly score of the execution step according to the preset weight corresponding to the anomaly detection method that is judged as anomalies in each execution step.

[0010] In this embodiment, an anomaly score is determined based on the number of anomaly detection methods and / or preset weights, providing a flexible and reliable anomaly quantification mechanism. A step that is identified as anomaly by more detection methods is more likely to be a genuine source of illusion, effectively avoiding false alarms from a single detection method. The weighted mode allows different weights to be assigned based on the confidence or importance of each detection method, further improving positioning accuracy. The two modes can be used in combination to adapt to the actual needs of different deployment scenarios.

[0011] In one possible implementation of the first aspect above, the illusion initiation step is the execution step with the highest exception score among the exception steps.

[0012] In this embodiment, the execution step with the highest anomaly score is determined as the hallucination initiation step. The highest anomaly score indicates that the step is pointed to by the most or most important detection methods. The superposition of multiple signals significantly improves the reliability and accuracy of the localization results.

[0013] In one possible implementation of the first aspect above, each of the N execution steps includes a step number; and, the number of execution steps with the highest abnormal score among the abnormal steps is greater than 1, and the abnormal step with the smallest step number among the multiple execution steps with the highest abnormal score among the abnormal steps is taken as the illusion initiation step.

[0014] In this embodiment, when there are multiple instances of the highest abnormal score, the execution step with the smallest step number is taken as the hallucination initiation step. By utilizing the characteristic of hallucination propagating backward along the trajectory, when multiple steps have the same abnormal score, the earliest step is most likely to be the true source, and subsequent abnormalities are likely to be secondary errors that have been propagated and contaminated. This ensures accurate location of the first cause in the trajectory propagation chain and avoids mistakenly identifying the contaminated steps downstream of the propagation chain as the source.

[0015] In one possible implementation of the first aspect mentioned above, the multiple anomaly detection methods include at least two of the following: a detection method based on parameter specification verification; a detection method based on return anomaly matching; a detection method based on fact statements and prior trajectories for implication determination; a detection method based on prior conditions and prior trajectories for implication determination; a detection method based on reasoning conclusions and prior intermediate results for consistency verification; and a detection method based on plan data and prior trajectories for evidence verification.

[0016] In this embodiment, six anomaly detection methods are set up to target tool usage illusions, retrieval illusions, reasoning illusions, human-computer interaction illusions, and planning illusions, forming a complete illusion detection system that covers various illusion forms that may occur in multi-turn tool invocation tasks by intelligent agents. Different detection methods detect different object types, are independent and complementary to each other, and any detection method that determines an anomaly will generate a marker, ensuring no omissions. Furthermore, each detection method can be independently enabled or disabled, supporting customized deployment according to actual needs.

[0017] In one possible implementation of the first aspect above, each of the N execution steps includes a tool call list, which includes at least one tool call. Each tool call includes at least one of the following parameters: tool name, tool parameter name, parameter type, and a list of required parameters. Furthermore, anomaly detection is performed on each of the N execution steps using a parameter specification verification-based detection method, generating a detection result for each execution step. This includes: obtaining a preset tool parameter pattern library, wherein the tool parameter pattern library includes at least one of the following tool parameters: tool name, tool parameter name, parameter type, and a list of required parameters; and checking item by item whether the parameters of each tool call in the tool call list of each execution step conform to the tool parameter... The tool parameters are corresponding to those in the parameter pattern library. For each execution step, if a parameter anomaly condition is detected, an anomaly is determined to exist, and an anomaly flag based on the corresponding parameter specification verification detection method is generated. The parameter anomaly condition includes at least one of the following: the tool name of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library; the tool parameter name of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library; the parameter type of at least one tool call in the tool call list of the execution step does not match the expected type corresponding to it in the tool parameter pattern library; at least one required parameter of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library.

[0018] In this embodiment, an anomaly detection method based on parameter specification verification utilizes a pre-built tool parameter pattern library to verify the name, parameter name, parameter type, and required parameters of tool calls item by item. This method can quickly capture tool usage illusions caused by large model illusions, such as incorrect tool selection, misspelled parameter names, mismatched parameter types, or missing required parameters. This method is driven by a pure rule engine, does not rely on large model inference, and has the advantages of high detection efficiency, zero inference cost, and fast response speed.

[0019] In one possible implementation of the first aspect above, each of the N execution steps includes a tool return result list, which includes at least one tool return result; and, anomaly detection is performed on each of the N execution steps using a detection method based on return anomaly matching to generate a detection result for each execution step, including: obtaining a preset anomaly keyword library, wherein the anomaly keyword library includes at least one anomaly keyword used to characterize anomalies in tool return results; matching each tool return result in the tool return result list of each execution step with the anomaly keyword library; for each execution step, if an anomaly condition is detected, it is determined that an anomaly has been detected, and an anomaly marker corresponding to the detection method based on return anomaly matching is generated; wherein the anomaly condition includes: at least one tool return result in the tool return result list of the execution step includes any anomaly keyword in the anomaly keyword library.

[0020] In this embodiment, an anomaly detection method based on return anomaly matching utilizes a pre-defined anomaly keyword library to match and detect the tool's returned results, enabling rapid capture of various explicit errors during tool execution. This method is also driven by a pure rule engine, requiring no large model involvement, and can efficiently detect tool execution layer anomalies with zero inference cost.

[0021] In one possible implementation of the first aspect above, each of the N execution steps includes agent output content; and anomaly detection is performed on each of the N execution steps by a detection method based on fact statements and prior trajectories for implication judgment, generating a detection result for each execution step, including: extracting fact statements from the agent output content of each execution step based on a large language model; inputting the fact statements of each execution step and the prior trajectories of each execution step in the agent's trajectory data into the large language model, and having the large language model output the implication judgment result of the fact statements relative to the prior trajectories, wherein the implication judgment result includes at least one of the following: support, contradiction, and lack of basis; for each execution step, if an anomaly condition is detected that satisfies the fact statement anomaly condition, it is determined that an anomaly has been detected, and an anomaly marker corresponding to the detection method based on fact statements and prior trajectories for implication judgment is generated; wherein the fact statement anomaly condition includes at least one of the following: there is at least one contradictory judgment in the implication judgment result; the proportion of unfounded judgments in the implication judgment result exceeds a preset threshold.

[0022] In this embodiment, an anomaly detection method based on the implication determination of fact statements and prior trajectories is employed. This method utilizes a large language model to extract fact statements and performs implication relationship determination with the prior trajectories. This enables the capture of retrieval-related illusions and human-computer interaction-related illusions in the agent's output that contradict prior evidence or lack supporting evidence. This approach leverages the semantic understanding capabilities of the large language model, enabling it to handle complex semantic inconsistencies. Furthermore, the combined design of contradictory and unsupported anomaly conditions ensures detection sensitivity while avoiding misjudgments caused by individual noisy statements by limiting the proportion of unsupported statements to a certain threshold.

[0023] In one possible implementation of the first aspect above, factual statements in each execution step are extracted from the agent's output content at each execution step based on a large language model, including: for each execution step, corresponding to the agent's output content including text output content, atomic factual statements are extracted from the text output content based on a large language model; or, corresponding to the agent's output content only including tool calls, atomic factual statements are extracted based on the parameters of the tool calls based on a large language model; wherein, atomic factual statements are related to specific results, state values, or numerical values.

[0024] In this embodiment, by extracting atomized fact statements, the extraction rules are clearly defined (only factual assertions about specific results, state values, or numerical values ​​are extracted, while task plans, coordination instructions, or common-sense descriptions are not extracted), ensuring that the extracted statements are verifiable factual assertions rather than subjective descriptions or non-factual content, thus improving the accuracy and efficiency of subsequent implication determination. Furthermore, different scenarios with and without text output are handled separately, ensuring complete coverage of the method.

[0025] In one possible implementation of the first aspect above, each of the N execution steps includes a tool call list; and anomaly detection is performed on each of the N execution steps using a detection method based on preconditions and pre-trajectory implications, generating a detection result for each execution step, including: extracting the precondition declarations implicit in the tool calls in each execution step based on a large language model, wherein the precondition declarations are used to characterize the execution gap between the user instruction and the tool call, and the precondition declarations include: environmental states or data conditions that should have been established before the tool call execution but have not yet been verified; and comparing the precondition declarations of each execution step with each... Each execution step inputs a preceding trajectory from the agent's trajectory data into a large language model. The large language model outputs an implication judgment result of the preceding condition declaration relative to the preceding trajectory. The implication judgment result includes at least one of the following: support, contradiction, or lack of basis. For each execution step, if an abnormal condition satisfying the preceding condition is detected, an anomaly is determined, and an anomaly label corresponding to the detection method of implication judgment based on the preceding condition and the preceding trajectory is generated. The abnormal condition of the preceding condition includes: the implication judgment result is contradictory or lacking basis, and the execution step has been labeled by at least one anomaly detection method other than the detection method of implication judgment based on the preceding condition and the preceding trajectory.

[0026] In this embodiment, an anomaly detection method based on implication determination using preconditions and pre-trajectory is employed. This method utilizes a large language model to extract the implicit precondition declarations from the tool call and performs implication determination with the pre-trajectory. This enables the capture of planning illusions where the agent fails to verify necessary preconditions before executing the tool call. This method complements the factual declaration implication determination, where the latter detects whether the output content is supported by evidence, while the former detects whether the operational premise is supported by evidence. Anomalies require contradictory or unfounded conditions that have already been flagged by at least one other detection method. This cross-validation mechanism avoids false alarms caused by the uncertainty of a single detection method, significantly improving detection accuracy.

[0027] In one possible implementation of the first aspect above, each of the N execution steps includes a tool call list and agent output content; and an anomaly detection is performed on each of the N execution steps by a detection method that verifies the consistency between the reasoning conclusion and the preceding intermediate results, generating a detection result for each execution step, including: for each execution step, corresponding to an agent output content including a computational derivation operation, obtaining the derivation conclusion corresponding to the computational derivation operation, wherein the computational derivation operation includes at least one of the following: computational tool call, text content containing mathematical derivation, text content containing computational summary type. The process involves: obtaining the user task description and the tool name corresponding to the computational derivation operation; inputting the derivation conclusion, user task description, tool name, and the previously verified intermediate results of the previous trajectory in the trajectory data of each execution step into the large language model; using the previously verified intermediate results to re-derive the derivation result; comparing the consistency between the re-derived result and the derivation conclusion; detecting a condition that satisfies the inference anomaly, determining that an anomaly has been detected, and generating an anomaly marker corresponding to the detection method based on the consistency verification between the inference conclusion and the previous intermediate results; where the inference anomaly condition includes: the re-derived result is inconsistent with the derivation conclusion.

[0028] In this embodiment, an anomaly detection method based on consistency verification between the reasoning conclusion and the preceding intermediate results is employed. This method utilizes a large language model to recalculate using the previously verified intermediate results, and then compares the recalculated result with the original reasoning conclusion. This approach can capture reasoning illusions in computational tool calls or mathematical derivation-related text content. Essentially, this is a counterfactual recalculation, relying solely on existing intermediate results within the trajectory without external knowledge. If the conclusions are inconsistent, it indicates an error in the original reasoning process. This method effectively detects subtle reasoning errors where the intermediate results are correct but the derivation process is flawed, or where the derivation conclusion logically contradicts the preceding results.

[0029] In one possible implementation of the first aspect above, obtaining the derivation conclusion corresponding to the computational derivation operation includes: for computational derivation operations that are computational tool calls, taking the output result of the computational tool call as the derivation conclusion; for computational derivation operations that are text content containing mathematical derivations or text content containing computational summaries, taking the conclusion expressed in the text content as the derivation conclusion.

[0030] In this embodiment of the application, by defining the method of obtaining the derivation conclusion corresponding to the computational derivation operation, it is clarified that the computational tool call directly retrieves the output result and the textual reasoning content retrieves the conclusion expressed therein, ensuring that subsequent consistency verification can be based on accurate conclusions and avoiding misjudgment caused by incorrect conclusion extraction.

[0031] In one possible implementation of the first aspect above, each of the N execution steps includes agent output content; and anomaly detection is performed on each of the N execution steps by a detection method based on plan data and prior trajectory verification, generating a detection result for each execution step, including: identifying whether the agent output content of each execution step contains content of plan or task decomposition type; for each execution step, corresponding to the agent output content containing content of plan or task decomposition type, extracting specific factual data from the agent output content, the specific factual data including at least one of the following: numbers, assignment expressions, dates, names of persons, names of organizations; detecting that the plan data anomaly condition is met, determining that an anomaly has been detected, and generating an anomaly marker corresponding to the detection method based on plan data and prior trajectory verification; wherein, the plan data anomaly condition includes: the existence of target specific factual data in the specific factual data of the agent output content, wherein there is no basis for the target specific factual data in the prior trajectory of the agent's trajectory data for each execution step.

[0032] In this embodiment, an anomaly detection method based on verification of planned data and prior trajectories identifies content of planned or task decomposition types, extracts specific factual data, and verifies whether there is any basis for it in the prior trajectories. This method can capture planning illusions, i.e., when the agent fabricates specific data that should be obtained through tool calls in subsequent steps during the planning phase. This method can effectively identify typical illusion patterns where plans and data are decoupled, preventing the agent from executing erroneous operational plans without sufficient evidence.

[0033] A second aspect of this application provides an electronic device, comprising: a memory for storing instructions executable by one or more processors of the electronic device; and a processor, one of the processors of the electronic device, for executing the instructions stored in the memory to implement any of the methods of the first aspect described above.

[0034] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed, implement any of the methods described in the first aspect.

[0035] The fourth aspect of this application provides a computer program product including instructions that, when executed, implement any of the methods described in the first aspect above.

[0036] In the embodiments of this application, when the second to fourth aspects implement any one of the methods in the first aspect, they can achieve the same or similar technical effects as any one of the methods in the first aspect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 According to an embodiment of this application, an interface diagram of a smart assistant application corresponding to a scenario in which a user uses a smart assistant on a mobile phone to make a travel plan is shown; Figure 2 A flowchart illustrating an intelligent agent illusion localization method is shown according to an embodiment of this application; Figure 3 A logical diagram illustrating the localization and verification of an agent's trajectory illusion source is shown according to an embodiment of this application; Figure 4 A logical diagram of a detection method based on return anomaly matching is shown according to an embodiment of this application; Figure 5 A logical diagram of a detection method for implication determination based on fact statements and prior trajectories is shown according to an embodiment of this application; Figure 6 A schematic diagram of a detection method for implication determination based on preconditions and pre-trajectory is shown according to an embodiment of this application; Figure 7 A logical schematic diagram of a detection method based on parameter specification verification is shown according to an embodiment of this application; Figure 8 A logical diagram illustrating a detection method for consistency verification based on reasoning conclusions and preliminary intermediate results is shown in the embodiments of this application. Figure 9 A schematic diagram of a detection method based on plan data and prior trajectory verification is shown according to an embodiment of this application; Figure 10 A schematic diagram of the hardware structure of an electronic device is shown according to an embodiment of this application. Detailed Implementation

[0039] The illustrative embodiments of this application include, but are not limited to, an intelligent agent illusion localization method, device, medium, and program product.

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described clearly and in detail below with reference to the accompanying drawings.

[0041] As mentioned earlier, agents based on large language models have been widely used in automated task execution and multi-round tool invocation scenarios. Large language models can suffer from the illusion problem. Since the execution trajectory of an agent contains multiple steps that depend on each other, an error generated in one round will propagate along the trajectory, affecting the decision-making of subsequent steps and ultimately leading to task failure.

[0042] For example, refer to Figure 1 The image shown depicts the interface of the Smart Assistant app, corresponding to a scenario where a user uses the Smart Assistant on their mobile phone to plan a trip.

[0043] like Figure 1 As shown, the user interface 10a includes a session area and a model output area. The session area 101 displays the user-inputted text message "Please create a detailed plan for my trip through the water towns of Jiangnan." The model output area 102 displays the response content generated by the model based on the tool's call results, "A two-day in-depth plan for a trip through the water towns of Jiangnan...".

[0044] The execution trajectory of the intelligent agent can include: Step 1, the intelligent agent receives the user instruction "Please create a detailed travel plan for the Jiangnan water towns," and parses out the destination as the "Jiangnan water towns" region and the task type as travel plan generation. Step 2, the intelligent agent calls a weather query tool to obtain local weather information. Step 3, based on the weather information obtained in Step 2, the intelligent agent calls a scenic spot query tool to obtain scenic spot information that meets the weather conditions. Step 4, based on the scenic spot information obtained in Step 3, the intelligent agent calls a plan generation tool to generate a travel plan and output it to the model output area 102.

[0045] If an error occurs at any stage of the above trajectory, it will propagate along the execution path and affect subsequent steps. For example, if in step 2 the agent incorrectly sets the location parameter to "Jiangnan Water Town" (the weather query tool requires a specific city name, but Jiangnan Water Town is not a city name), the call will return an incorrect parameter. Based on this incorrect return information, step 3 cannot obtain the correct weather conditions, which may lead to setting the attraction query condition to "indoor attractions" instead of the user's intended "wandering" theme, causing the query object to deviate from the user's needs. Step 4 generates a travel plan based on the attraction information obtained in step 3 that deviates from the intended purpose. Although the final output plan is complete in form, it deviates from the user's true intention.

[0046] This shows that small errors in the initial steps can propagate along the trajectory and amplify step by step, eventually leading to mission failure.

[0047] Therefore, in order to improve the accuracy of task execution by intelligent agents, it is necessary to identify and correct the steps that generate hallucinations.

[0048] In some cases, to determine the authenticity of an agent's output (i.e., whether a hallucination has occurred), a fact-checking-based detection method is typically used. This involves verifying the agent's final output against known knowledge or prior evidence. If they match, no hallucination has occurred; if they do not match, a hallucination has occurred.

[0049] However, this type of method only makes judgments at the final output level and cannot pinpoint the specific steps that cause the hallucination, making it difficult to make corrections to prevent the hallucination from recurring.

[0050] In other cases, in order to locate the specific steps in the trajectory where the illusion is generated, an end-to-end detection method can be used. That is, the complete execution trajectory of the agent is input into a large language model, and the large language model directly outputs the step number where the illusion source is located.

[0051] However, experiments show that the accuracy of existing mainstream open-source and closed-source large language models (such as GPT series, Gemini series, Claude series, Qwen series, Llama series, etc.) in locating the source of illusion is generally low.

[0052] For example, in the planning illusion task, the best-performing closed-source model achieved an accuracy of only 31.3%, while the open-source model achieved only 14.9%; in the retrieval illusion task, the closed-source model achieved an accuracy of 45.1%, while the open-source model achieved 28.1%; in the reasoning illusion task, the closed-source model achieved an accuracy of 64.4%, while the open-source model achieved 28.8%; in the human-interaction illusion task, the closed-source model achieved an accuracy of 61.6%, while the open-source model achieved 41.1%; and in the tool-use illusion task, the closed-source model achieved an accuracy of only 19.4%, while the open-source model achieved only 8.7%.

[0053] However, in this type of method, different types of hallucinations are distributed across different signal dimensions, making it difficult for a single model to cover all signal dimensions simultaneously in a single inference, which easily leads to missed detections and false positives. Furthermore, this type of method only outputs step numbers and cannot distinguish between the "hallucination source (i.e., the initial step of the hallucination)" and "subsequent steps that have been contaminated by propagation (i.e., abnormal steps other than the initial step of the hallucination)".

[0054] In view of this, embodiments of this application provide a method for locating hallucinations in an intelligent agent, applied to an electronic device. The method includes: acquiring trajectory data of an intelligent agent, wherein the trajectory data includes step information of N execution steps, where N is an integer greater than 1; performing anomaly detection on each of the N execution steps using multiple anomaly detection methods, generating a detection result for each execution step, wherein different anomaly detection methods detect different object types; calculating anomaly scores for each step based on the detection results of different anomaly detection methods corresponding to each step; and determining the abnormal steps among the N execution steps based on the anomaly scores for each step, wherein the abnormal steps include the hallucination initiation step.

[0055] It is understandable that by independently detecting each execution step in the trajectory using multiple anomaly detection methods, and calculating anomaly scores based on the detection methods used to determine the anomalies of each step, the abnormal step (the hallucination initiation step) can be identified based on the anomaly scores, thus more accurately locating the source of the hallucination.

[0056] Furthermore, multiple anomaly detection methods detect different object types, and different signal dimensions complement each other. Any detection method can generate a signal when it captures an anomaly, and any type of hallucination can be captured by at least one detection method, avoiding missed detections caused by a single detection method.

[0057] The intelligent agent illusion localization method provided in the embodiments of this application is described in detail below with reference to the accompanying drawings.

[0058] For example, Figure 2 A flowchart illustrating an intelligent agent illusion localization method is shown according to an embodiment of this application. It can be understood that... Figure 2 The execution entity in the flowchart shown is the electronic device 100 (such as the aforementioned mobile phone 10 or a server connected to the mobile phone 10). For ease of description, the following will refer to... Figure 2 When describing the flowchart shown, the execution subject of the flowchart will not be repeated.

[0059] like Figure 2 As shown, this process includes, but is not limited to: S201: Obtain the trajectory data of the agent, wherein the trajectory data includes step information of N execution steps, where N is an integer greater than 1.

[0060] In some embodiments, the step information for each execution step includes at least one of the following: a tool call list, a tool return result list (or tool call return result list), and agent output content.

[0061] The tool call list includes at least one tool call, and each tool call includes a tool name, parameter name, parameter type, and a list of required parameters; the tool return result list includes at least one tool return result; and the agent output content includes the text output content generated by the agent when performing this step.

[0062] S202: Perform anomaly detection on each of the N execution steps using various anomaly detection methods, and generate a detection result for each execution step. The different anomaly detection methods detect different types of objects.

[0063] In some embodiments, the multiple anomaly detection methods include at least two of the following: a detection method based on parameter specification verification; a detection method based on return anomaly matching; a detection method based on fact statements and prior trajectories for implication determination; a detection method based on prior conditions and prior trajectories for implication determination; a detection method based on reasoning conclusions and prior intermediate results for consistency verification; and a detection method based on plan data and prior trajectories for evidence verification.

[0064] It is understandable that by setting up six anomaly detection methods—specifically targeting tool usage illusions, retrieval illusions, reasoning illusions, human-computer interaction illusions, and planning illusions—a complete illusion detection system is formed, covering various illusion forms that may occur in multi-turn tool invocation tasks by intelligent agents. Different detection methods detect different object types, are independent and complementary to each other, and any detection method that determines an anomaly will generate a label, ensuring no omissions. Furthermore, each detection method can be independently enabled or disabled, supporting tailored deployment according to actual needs.

[0065] The specific detection process of each anomaly detection method will be described in detail below, and will not be repeated here.

[0066] In some embodiments, for each execution step, the detection result includes: an anomaly flag corresponding to the anomaly detection method if any anomaly detection method determines that the execution step is abnormal.

[0067] In other words, if an execution step is identified as abnormal by multiple anomaly detection methods, the detection result of that execution step will include multiple anomaly markers; if an execution step is not identified as abnormal by any anomaly detection method, the detection result of that execution step will not include anomaly markers.

[0068] S203: Calculate the anomaly score for each step based on the detection results of different anomaly detection methods corresponding to each step.

[0069] In some embodiments, the electronic device 100 calculates the anomaly score for each step by: determining the anomaly score of the execution step based on the number of anomaly detection methods that are determined to be abnormal in each execution step; and / or, determining the anomaly score of the execution step based on the preset weight corresponding to the anomaly detection method that is determined to be abnormal in each execution step.

[0070] In some embodiments, the preset weight of each anomaly detection method is related to the detection accuracy and reliability of each anomaly detection method.

[0071] For example, the implication determination method based on factual statements and preceding trajectories, aided by a large language model, can perform deep semantic analysis and has relatively high detection accuracy; while the detection method based on returning anomaly matching relies on keyword matching and may have a certain false positive rate. Therefore, the preset weight corresponding to the implication determination method based on factual statements and preceding trajectories is greater than the preset weight corresponding to the detection method based on returning anomaly matching.

[0072] It is understood that the preset weights of different detection methods can be preset by those skilled in the art based on factors such as the actual application scenario, the accuracy requirements of each detection method, or the historical detection accuracy rate. This application does not limit this.

[0073] In some embodiments, the electronic device 100 may use the number of anomaly detection methods that are determined to be abnormal in each execution step as an anomaly score.

[0074] For example, in conjunction with the preceding text Figure 1 The scenario depicted illustrates an agent making a travel plan. It assumes multiple anomaly detection methods, including those based on parameter specification verification, those based on factual statements and prior trajectories for implication determination, and those based on consistency verification between inference conclusions and prior intermediate results. If... Figure 1 Step 2 (calling the weather query tool) in the trajectory shown is identified as abnormal by the parameter specification verification detection method (e.g., the location parameter "Jiangnan Water Town" does not conform to the city name format required by the tool), but is not identified as abnormal by the other two detection methods. Therefore, the electronic device 100 determines the abnormality score of this execution step to be 1 based on the number of abnormality detection methods. If step 4 (generating a travel plan) is identified as abnormal by both the factual statement and the preceding trajectory implication judgment detection method and the consistency verification based on the reasoning conclusion and the preceding intermediate results detection method (e.g., the weather data referenced in the plan contradicts the weather information returned in step 2, and the final plan content is logically inconsistent with the preceding intermediate results), then the electronic device 100 determines the abnormality score of this execution step to be 2. Since step 4 is identified as abnormal by more detection methods, its abnormality score is higher than that of step 2, indicating that step 4 has more dimensions of abnormal representation and is more likely to be a real source of illusion.

[0075] In some embodiments, when each step is marked by multiple detection methods, the electronic device 100 may also use the sum of preset weights corresponding to the anomaly detection methods that determine each execution step as an anomaly as an anomaly score.

[0076] For example, suppose the preset weight for the detection method based on parameter specification verification is 0.4, the preset weight for the detection method based on factual statements and prior trajectories for implication determination is 0.4, and the preset weight for the detection method based on consistency verification between inference conclusions and prior intermediate results is 0.2. Figure 1 Step 3 (calling the attraction query tool) in the trajectory shown is identified as anomalous by the detection method based on the implication judgment of factual statements and prior trajectories (for example, the attraction query basis output by step 3 contradicts the weather information returned by step 2). Therefore, the electronic device 100 determines the anomalous score of this execution step to be 0.4 based on the preset weights corresponding to each detection method. If step 4 is identified as anomalous by both the detection method based on the implication judgment of factual statements and prior trajectories and the detection method based on the consistency verification between the inference conclusion and the preceding intermediate results, then the electronic device 100 determines the anomalous score of this execution step to be 0.6. Although both steps are marked by the detection methods, step 4 is marked by more detection methods, resulting in a higher anomalous score, indicating that this step has a higher anomalous confidence in the weighted dimension.

[0077] Understandably, determining anomaly scores based on the number of anomaly detection methods and / or preset weights provides a flexible and reliable anomaly quantification mechanism. A step identified as anomaly by more detection methods is more likely to be a genuine source of illusion, effectively avoiding false alarms from a single detection method. The weighted mode allows assigning different weights based on the confidence or importance of each detection method, further improving positioning accuracy. The two modes can be used in combination to adapt to the actual needs of different deployment scenarios.

[0078] S204: Based on the abnormal score of each step, identify the abnormal steps among the N execution steps, where the abnormal steps include the hallucination initiation step.

[0079] In some embodiments, the hallucination initiation step is the execution step with the highest anomalous score among the anomalous steps.

[0080] For example, in conjunction with the preceding text Figure 1The scenario depicts an agent making a travel plan. Assume the agent's execution trajectory includes steps 1 through 4. After various anomaly detection methods, step 1 is not marked by any method; step 2 is marked by a method based on parameter specification verification; step 3 is marked by a method based on factual statements and prior trajectories for implication determination; and step 4 is marked by both a method based on factual statements and prior trajectories for implication determination and a method based on consistency verification between the inference conclusion and prior intermediate results. Therefore, step 1 has an anomaly score of 0, step 2 has an anomaly score of 1, step 3 has an anomaly score of 1, and step 4 has an anomaly score of 2. The electronic device 100 identifies step 4, with the highest anomaly score, as the anomaly step, i.e., the hallucination initiation step.

[0081] It is understandable that by determining the execution step with the highest anomaly score as the hallucination initiation step, the highest anomaly score indicates that the step is jointly pointed to by the most or most important detection methods. The superposition of multiple signals significantly improves the reliability and accuracy of the localization results.

[0082] In some embodiments, each of the N execution steps includes a step number. Furthermore, if the number of execution steps with the highest abnormal score among the abnormal steps is greater than 1, the electronic device 100 may use the abnormal step with the smallest step number among the multiple execution steps with the highest abnormal score as the hallucination initiation step.

[0083] For example, still in conjunction with the previous text Figure 1 In the scenario depicting an agent making a travel plan, if step 3 is jointly flagged by two detection methods—one based on factual statements and prior trajectories for implication determination, and the other based on prior conditions and prior trajectories for implication determination—the anomaly score is 2. Similarly, if step 4 is jointly flagged by two detection methods—one based on factual statements and prior trajectories for implication determination, and the other based on consistency verification between the reasoning conclusion and prior intermediate results—the anomaly score is also 2. In this case, steps 3 and 4 have the highest anomaly scores, with a count greater than 1. Since the step number of step 3 is lower than that of step 4, the electronic device 100 identifies step 3 as the hallucination initiation step.

[0084] It is understandable that by taking the execution step with the smallest step number when there are multiple abnormal scores, the hallucination is started by taking advantage of the characteristic of the hallucination propagating backward along the trajectory. When multiple steps have the same abnormal score, the earliest step is most likely to be the real source, and subsequent abnormalities are likely to be secondary errors that have been propagated and contaminated. This ensures that the first cause is accurately located in the trajectory propagation chain and avoids mistakenly identifying the contaminated steps downstream of the propagation chain as the source.

[0085] Understandable. Figure 2 The data processing procedure shown is only one example; in other embodiments, Figure 2The illustrated intelligent agent hallucination localization method may further include a post-processing step for the detection results, such as normalizing abnormal scores or filtering abnormal steps according to a preset threshold. Those skilled in the art can also provide further details based on actual needs. Figure 2 The process shown may include adding, deleting, or merging other content, and this application does not impose any restrictions on this.

[0086] For example, in some embodiments, after the electronic device 100 determines the illusion initiation step among the N execution steps, the illusion initiation step can be corrected, for example, by replacing the tool call parameter of the illusion initiation step with the correct value and then re-executing the illusion initiation step.

[0087] For example, in other embodiments, to further improve the accuracy of hallucination localization, after the electronic device 100 identifies abnormal steps among the N execution steps, counterfactual verification can be performed on all the identified abnormal steps. For instance, counterfactual verification can be performed on each abnormal step in descending order of abnormality score. The input or output data of each abnormal step can be replaced with the corrected value and the step can be re-executed. If the result output by the agent is accurate, the abnormal step is confirmed as the hallucination initiation step, and the hallucination initiation step can be further corrected for subsequent use.

[0088] For example, Figure 3 According to an embodiment of this application, a logical diagram is shown for locating and verifying the source of an agent's trajectory illusion.

[0089] like Figure 3 As shown, the input data is the agent's multi-step tool call trajectory {step, content, tool_calls, tool_responses}. Each step in the trajectory includes a step number (step), agent output content (content), tool call list (tool_calls), and tool return result list (tool_responses).

[0090] Phase 1 is the multi-signal candidate screening phase, which includes six parallel signal extraction lines, R1 to R6. R1, error detection, is a signal extraction line based on anomaly matching. It matches abnormal keywords in the tool's returned results using pure code rules. If anomaly keywords such as "error," "exception," or "timeout" are matched, a corresponding anomaly tag is generated. R2, declaration implication, is a signal extraction line based on factual statements and prior trajectories for implication determination. It extracts factual statements from the agent's output using a large language model and performs implication determination against prior trajectories. If the proportion of contradictory or unfounded judgments exceeds a threshold, a corresponding anomaly tag is generated. R3, precondition implication, is a signal extraction line based on preconditions and prior trajectories for implication determination. It extracts the implicit preconditions from tool calls using a large language model and performs implication determination against prior trajectories. If the judgment result is contradictory or unfounded, and this step has already been marked by at least one other line, a corresponding anomaly tag is generated. R4 schema verification is a signal extraction line based on parameter specification verification. It verifies the parameter specifications of tool calls based on pure code rules. If the tool name is not in the schema library, parameter name is misspelled, parameter type is mismatched, or required parameters are missing, a corresponding anomaly flag is generated. R5 inference consistency verification is a signal extraction line based on the consistency between inference conclusions and intermediate results. It verifies the consistency between the inference conclusions and intermediate results of calculation derivation steps based on a large language model. If the re-inference result is inconsistent with the inference conclusion, a corresponding anomaly flag is generated. R6 plan grounding verification is a signal extraction line based on the basis verification between plan data and the preceding trajectory. It verifies the basis verification between specific data in plan decomposition steps and the preceding trajectory based on a large language model. If specific data appearing in the plan has no basis in the preceding trajectory, a corresponding anomaly flag is generated.

[0091] The six lines are detected independently, generating corresponding anomaly tags for steps deemed abnormal. Then, they are merged and scored: the union of the anomaly tags generated by each line is used to generate a candidate abnormal step set. For each step in the candidate set, a score is awarded based on the number of signal lines that were tagged, or a weighted score based on the preset weights of each signal line. The step with the highest score and the earliest execution order is determined as the primary responsible step (i.e., the hallucination initiation step).

[0092] The output data for Phase 1 includes: the determination of whether the trajectory contains hallucinations, a list of candidate abnormal steps sorted by score, the step number of the hallucination initiation step, and a detailed record of which signal lines marked each step.

[0093] The above output results can be used for counterfactual verification in Phase Two.

[0094] Specifically, after receiving the list of candidate anomalous steps from Phase 1, Phase 2 does not need to iterate through and verify all steps in the trajectory. Instead, it only needs to perform counterfactual verification on each anomalous step in the candidate list, ranked from highest to lowest score. For each candidate step to be verified, Phase 2 replaces the input or output data of that step with the corrected value, keeping other steps unchanged, and re-executes the subsequent trajectory. If replacing the step eliminates the anomalousness in subsequent steps or restores the trajectory execution result to normal, then that step is confirmed as the true illusion initiation step; if the anomalousness still exists after replacement, the next candidate step is verified.

[0095] In this way, the verification scope of Phase 2 is limited by the output of Phase 1 to the candidate steps with anomaly scores of non-zero (i.e., the identified anomalous steps), avoiding the huge computational overhead caused by exhaustive verification and improving the efficiency and accuracy of illusion source localization.

[0096] Understandably, Figure 3 The logic diagram shown is merely an exemplary implementation provided by the embodiments of this application. Those skilled in the art can add, delete, or merge the signal lines in the diagram according to actual needs, and this application does not impose any restrictions on this.

[0097] The following section, with reference to the accompanying diagrams, details the specific process by which the above-mentioned anomaly detection methods perform anomaly detection on each of the N execution steps and generate the detection results for each execution step.

[0098] In some embodiments, each of the N execution steps includes a tool return result list, which includes at least one tool return result.

[0099] Furthermore, the process of detecting anomalies in each of the N execution steps using a detection method based on return anomaly matching includes: obtaining a preset anomaly keyword library, wherein the anomaly keyword library includes at least one anomaly keyword used to characterize the abnormality of the tool's return result; matching each tool return result in the tool return result list of each execution step with the anomaly keyword library; for each execution step, if the result anomaly condition is detected, it is determined that an anomaly has been detected, and an anomaly tag corresponding to the detection method based on return anomaly matching is generated; wherein the result anomaly condition includes: at least one tool return result in the tool return result list of the execution step includes any anomaly keyword in the anomaly keyword library.

[0100] For example, Figure 4 According to an embodiment of this application, a logical diagram of a detection method based on return anomaly matching (R1 error detection) is shown.

[0101] like Figure 4 As shown, the input data for this detection method is the tool return result text for each execution step of the agent. The detection method includes the following: First, the tool return result for each execution step is matched using regular expressions against a pre-defined database of abnormal keywords, which contains more than 20 types of abnormal keywords such as error, exception, failed, not found, and timeout. Second, for any execution step, if its tool return result matches any of the above abnormal keywords, that step is marked as a candidate abnormal step, and a corresponding abnormal tag is generated.

[0102] like Figure 4 As shown in the example output data, step 1 is marked with the exception reason "error", and the candidate exception step set is {1}. This detection method is based on pure code rule-driven, does not require calling a large language model, and has the advantage of zero inference cost.

[0103] It is understandable that by using an anomaly detection method based on matching returned anomalies, and utilizing a pre-built anomaly keyword library to match and detect the tool's returned results, various explicit errors during tool execution can be quickly captured. This method is also driven by a pure rule engine, requiring no large model involvement, and can efficiently complete the detection of tool execution layer anomalies with zero inference cost.

[0104] In some embodiments, each of the N execution steps includes agent output content.

[0105] Furthermore, the process of anomaly detection for each of the N execution steps using an implication judgment method based on fact statements and prior trajectories includes: extracting fact statements from the agent's output content for each execution step using a large language model; inputting the fact statements for each execution step and the prior trajectories for each execution step in the agent's trajectory data into the large language model, and having the large language model output the implication judgment result of the fact statements relative to the prior trajectories, wherein the implication judgment result includes at least one of the following: support, contradiction, or lack of basis; for each execution step, if an anomaly condition is detected that satisfies the fact statement anomaly condition, an anomaly is determined to be detected, and an anomaly marker corresponding to the implication judgment method based on fact statements and prior trajectories is generated; wherein the fact statement anomaly condition includes at least one of the following: the implication judgment result contains at least one contradictory judgment; the proportion of unfounded judgments in the implication judgment result exceeds a preset threshold.

[0106] In some embodiments, the electronic device 100 extracts fact statements from the agent's output content for each execution step based on a large language model using the following method: for each execution step, if the agent's output content includes text output content, atomic fact statements are extracted from the text output content based on a large language model; or, if the agent's output content only includes tool calls, corresponding atomic fact statements are extracted based on the parameters of the tool calls using a large language model; wherein, the atomic fact statements are related to specific results, state values, or numerical values.

[0107] Understandably, by extracting atomized fact statements, the extraction rules are clearly defined (only factual assertions about specific results, state values, or numerical values ​​are extracted, while task plans, coordination instructions, or common-sense descriptions are not extracted), ensuring that the extracted statements are verifiable factual assertions rather than subjective descriptions or non-factual content, thus improving the accuracy and efficiency of subsequent implication determination. Furthermore, the method handles scenarios with and without text output separately, guaranteeing complete coverage.

[0108] For example, Figure 5 According to an embodiment of this application, a logical diagram of a detection method (R2 declaration implication) based on fact statements and prior trajectories for implication determination is shown.

[0109] like Figure 5 As shown, the input data for this detection method is the text output content of each execution step of the agent. The detection method includes the following: First, atomic fact statements are extracted from the agent's output content at each execution step, with no more than one statement extracted per step (i.e., the number of atomic fact statements is less than or equal to 1). These statements are factual assertions about specific results, state values, or numerical values. Second, the extracted fact statements, along with the preceding trajectories, are input into a shared implication engine. The large language model outputs implication judgment results, which include three types: support (A), contradiction (B), or lack of basis (C). If the proportion of contradictory or lack-of-basis judgments exceeds a preset threshold, the step is marked as a candidate anomalous step, and a corresponding anomalous label is generated.

[0110] like Figure 5 The example output data shows that the declaration number is s8_c1, the judgment result is contradictory (B), the evidence is that the rear left tire pressure is 30 pound-force per square inch (PSI), which is lower than the normal value of 32 PSI-35 PSI, and the candidate anomalous step set is {8}.

[0111] It is understandable that an anomaly detection method based on the implication determination of fact statements and prior trajectories can capture retrieval-related illusions and human-computer interaction-related illusions in the agent's output that contradict or lack supporting evidence. This method leverages the semantic understanding capabilities of large language models to handle complex semantic inconsistencies. Furthermore, the combined design of contradictory and unsupported anomaly conditions ensures both detection sensitivity and avoids misjudgments caused by individual noisy statements by limiting the proportion of unsupported statements to a certain threshold.

[0112] In some embodiments, each of the N execution steps includes a list of tool calls.

[0113] Furthermore, the process of anomaly detection for each of the N execution steps using an implication judgment method based on preconditions and pre-trajectory includes: extracting the precondition declarations implicit in the tool call in each execution step based on a large language model, wherein the precondition declarations are used to characterize the execution gap between the user instruction and the tool call, and the precondition declarations include: environmental states or data conditions that should have been established before the tool call execution but have not yet been verified; inputting the precondition declarations of each execution step and the pre-trajectory of each execution step in the agent's trajectory data into the large language model, and outputting the implication judgment result of the precondition declarations relative to the pre-trajectory, wherein the implication judgment result includes at least one of the following: support, contradiction, and lack of basis; for each execution step, if an abnormal condition satisfying the precondition is detected, it is determined that an anomaly has been detected, and an anomaly label corresponding to the implication judgment method based on preconditions and pre-trajectory is generated; wherein the abnormal condition of the precondition includes: the implication judgment result is contradictory or lack of basis, and the execution step has been labeled by at least one anomaly detection method other than the implication judgment method based on preconditions and pre-trajectory.

[0114] For example, Figure 6 According to an embodiment of this application, a logical schematic diagram of a detection method (R3 precondition implication) based on preconditions and precondition trajectories for implication determination is shown.

[0115] like Figure 6As shown, the input data for this detection method is the list of tool calls for each execution step of the agent. The detection method includes the following: First, for each tool call in the execution step, the implicit precondition declaration is extracted. This precondition declaration characterizes the environmental state or data condition that should exist before the tool call execution but has not yet been verified. Taking the move (mv) tool as an example, its implicit precondition is "the agent assumes the current directory is / document". Second, the extracted precondition declaration and the preceding precondition trajectory are input into the shared implication engine, and the large language model outputs the implication judgment result, which includes three types: support, contradiction, or no basis. If the judgment result is no basis, and the step has been marked by at least one other detection method, then the step is marked as a candidate anomalous step, and a corresponding anomalous label is generated.

[0116] like Figure 6 The example output data shows that the declaration number is s1_c1, the judgment result is no basis, the reason is that no directory switching (cd / document) operation was found in the previous trajectory, and the candidate abnormal step set is {1}.

[0117] It is understandable that an anomaly detection method based on the implications of preconditions and pre-trajectory trajectories, utilizing a large language model to extract the implicit precondition declarations from tool calls and performing implications judgments with the pre-trajectory trajectories, can capture the planning illusion of an agent failing to verify necessary preconditions before executing tool calls. This method complements the implications judgment of fact statements; the latter detects whether the output content is supported by evidence, while the former detects whether the operational premises are supported by evidence. Anomalies require contradictory or unfounded conditions that have been marked by at least one other detection method. This cross-validation mechanism avoids false alarms caused by the uncertainty of a single detection method, significantly improving detection accuracy.

[0118] In some embodiments, each of the N execution steps includes a tool call list, which includes at least one tool call, and each tool call includes at least one of the following parameters: tool name, tool parameter name, parameter type, and a list of required parameters.

[0119] Furthermore, the process of anomaly detection for each of the N execution steps based on parameter specification verification includes: obtaining a preset tool parameter pattern library, wherein the tool parameter pattern library includes at least one of the following tool parameters: tool name, tool parameter name, parameter type, and a list of required parameters; checking item by item whether the parameter of each tool call in the tool call list of each execution step conforms to the corresponding tool parameter in the tool parameter pattern library; for each execution step, if the parameter anomaly condition is detected, it is determined that an anomaly has been detected, and an anomaly flag corresponding to the parameter specification verification detection method is generated; wherein the parameter anomaly condition includes at least one of the following: the tool name of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library; the tool parameter name of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library; the parameter type of at least one tool call in the tool call list of the execution step does not match the expected type in the tool parameter pattern library; and at least one required parameter of at least one tool call in the tool call list of the execution step does not exist in the tool parameter pattern library.

[0120] For example, Figure 7 According to an embodiment of this application, a logical diagram of a detection method based on parameter specification verification (R4 mode verification) is shown.

[0121] like Figure 7 As shown, the input data for this detection method consists of the parameters called by the tools in each execution step of the agent. Taking the movement (mv) tool as an example, its parameters include the source path and the destination path. The detection method includes the following: First, a parameter pattern library containing 139 known tools is preloaded. The parameter pattern for each tool includes the parameter name, parameter type, and a list of required parameters. Second, each parameter called by the tool is checked item by item to ensure it conforms to the specifications in the pattern library, including whether the parameter name exists in the library, whether the parameter type is correct, and whether any required parameters are missing. If any check item does not conform to the specifications (i.e., parameter anomaly), the step is marked as a candidate anomalous step, and a corresponding anomalous flag is generated.

[0122] like Figure 7 The example output data shows that the tool is mv, the parameter name is "sofdeurce" which is not recognized in the pattern library, the expected parameters are [source, destination], and the candidate abnormal step set is {1}. This detection method is based on pure code rule-driven, does not require calling a large language model, and has the advantage of zero inference cost.

[0123] It is understandable that an anomaly detection method based on parameter specification verification utilizes a pre-built tool parameter pattern library to verify the name, parameter name, parameter type, and required parameters of tool calls item by item. This can quickly capture tool usage illusions caused by large model illusions, such as incorrect tool selection, misspelled parameter names, mismatched parameter types, or missing required parameters. This method is driven by a pure rule engine, does not rely on large model inference, and has the advantages of high detection efficiency, zero inference cost, and fast response speed.

[0124] In some embodiments, each of the N execution steps includes a tool call list and agent output content.

[0125] Furthermore, the process of detecting anomalies in each of the N execution steps through a detection method based on consistency verification between the inference conclusion and the preceding intermediate results includes: for each execution step, the corresponding intelligent agent output includes computational derivation operations; obtaining the derivation conclusion corresponding to the computational derivation operations, wherein the computational derivation operations include at least one of the following: computational tool calls, text content containing mathematical derivations, and text content containing computational summaries; obtaining the user task description and the tool name corresponding to the computational derivation operations; inputting the derivation conclusion, user task description, tool name, and the preceding verified intermediate results of the preceding trajectory in the intelligent agent's trajectory data of each execution step into the large language model, whereby the large language model re-derives and calculates the re-derived result using the preceding verified intermediate results, and compares the re-derived result with the derivation conclusion for consistency; if the inference anomaly condition is met, it is determined that an anomaly has been detected, and an anomaly marker corresponding to the detection method based on consistency verification between the inference conclusion and the preceding intermediate results is generated; wherein the inference anomaly condition includes: the re-derived result is inconsistent with the derivation conclusion.

[0126] In some embodiments, the electronic device 100 obtains the derivation conclusion corresponding to the calculation derivation operation through the following methods: when the calculation derivation operation is a calculation tool call, the output result of the calculation tool call is used as the derivation conclusion; when the calculation derivation operation is text content containing mathematical derivation or text content containing calculation summary type, the conclusion expressed in the text content is used as the derivation conclusion.

[0127] It is understandable that by clarifying the method of obtaining the derivation conclusions corresponding to computational derivation operations, the output results of computational tools are directly obtained and the conclusions expressed in textual reasoning are obtained. This ensures that subsequent consistency checks can be based on accurate conclusions and avoids misjudgments caused by incorrect conclusion extraction.

[0128] For example, Figure 8 According to an embodiment of this application, a logical diagram of a detection method (R5 inference consistency) based on consistency verification between inference conclusions and preceding intermediate results is shown.

[0129] like Figure 8 As shown, the input data for this detection method consists of execution steps of an intelligent agent that include computational derivation operations. These computational derivation operations include calls to computational tools (such as Python interpreters) or textual content containing mathematical derivations or computational summaries. The detection method includes the following: First, extracting premise data from the preceding trajectory as the basis for reasoning, such as a birth date of November 26, 1969, or the year of the award being 1997. Second, inputting the derivation conclusion, user task description, tool name, and previously verified intermediate results into a large language model. The large language model then uses the previously verified intermediate results to re-derive the derivation and obtain a re-derived result, comparing the re-derived result with the original derivation conclusion for consistency. If the re-derived result is inconsistent with the derivation conclusion, this step is marked as a candidate anomalous step, and a corresponding anomalous label is generated.

[0130] like Figure 8 As shown in the example output data, the premise is 1997-1969=28, the agent's deduction conclusion is "27", the re-deduction result 28 is inconsistent with the deduction conclusion 27, and the candidate abnormal step set is {3}.

[0131] It is understandable that anomaly detection methods, which verify the consistency between the reasoning conclusion and the preceding intermediate results, utilize a large language model to recalculate based on the verified intermediate results. By comparing the recalculated result with the original conclusion, it is possible to capture reasoning illusions in computational tool calls or mathematical derivation-related text content. This method is essentially a counterfactual recalculation, relying solely on existing intermediate results within the trajectory without external knowledge. If the conclusions are inconsistent, it indicates an error in the original reasoning process. It can effectively detect subtle reasoning errors where the intermediate results are correct but the derivation process is flawed, or where the derivation conclusion logically contradicts the preceding results.

[0132] In some embodiments, each of the N execution steps includes agent output content.

[0133] Furthermore, the process of anomaly detection for each of the N execution steps using a detection method based on plan data and prior trajectory verification includes: identifying whether the agent output of each execution step contains content of plan or task decomposition type; for each execution step, corresponding to the inclusion of plan or task decomposition type content in the agent output, extracting specific factual data from the agent output, which includes at least one of the following: numbers, assignment expressions, dates, names of persons, and names of organizations; detecting that the plan data anomaly condition is met, determining that an anomaly has been detected, and generating an anomaly marker corresponding to the detection method based on plan data and prior trajectory verification; wherein, the plan data anomaly condition includes: the existence of target specific factual data in the specific factual data of the agent output, wherein there is no basis for the existence of target specific factual data in the prior trajectory of the agent's trajectory data for each execution step.

[0134] For example, Figure 9 According to an embodiment of this application, a logic diagram of a detection method (R6 planned grounding) based on planned data and prior trajectory verification is shown.

[0135] like Figure 9 As shown, the input data for this detection method consists of execution steps of the agent that include plan or task decomposition content. These steps typically include task labels. <tasks>This involves using structured identifiers such as (e.g., names, numbers, constraints) or similar identifiers. The detection method includes the following: First, extracting all specific factual data from the plan or task breakdown, including names, numbers, constraints, etc. Second, checking each specific factual data item against supporting evidence from the execution steps prior to plan formulation. If a specific factual data item lacks supporting evidence in the preceding trajectory, that step is marked as a candidate anomalous step, and a corresponding anomalous marker is generated.

[0136] like Figure 9 As shown in the example output data, the extracted specific factual data are "neighboring groups" and "nitrogen and oxygen groups". No relevant evidence was found in the preliminary trajectory before the plan was formulated, so it was judged as unfounded, and the candidate abnormal step set is {1}.

[0137] It is understandable that an anomaly detection method, which verifies the basis of planned data and prior trajectories, can identify content of planned or task decomposition types, extract specific factual data, and verify whether there is any basis for it in the prior trajectories. This can capture planning illusions, that is, when an agent fabricates specific data that should be obtained through tools in subsequent steps during the planning phase. This method can effectively identify typical illusion patterns where plans and data are decoupled, preventing the agent from executing erroneous operational plans without sufficient evidence.

[0138] It should be noted that the above six anomaly detection methods are independent and executed in parallel, each generating its own independent anomaly marker. Among them, the detection methods based on return anomaly matching and parameter specification verification are driven by pure code rules, requiring no invocation of a large language model, and have the advantages of zero inference cost and high detection efficiency. The remaining four detection methods are driven by a large language model, capable of handling complex semantic understanding, logical reasoning, and implication determination tasks. These two types of detection methods complement each other in terms of cost and capability; the rule engine is responsible for low-cost and rapid screening of explicit errors, while the large language model is responsible for semantic-level anomaly detection in complex scenarios, achieving a reasonable allocation of computing resources while ensuring detection accuracy. The execution order of the above six detection methods is not important, and this embodiment does not impose any restrictions on this.

[0139] In some embodiments, the large language model used in the above detection process is used to perform semantic encoding and logical implication determination on the input text. The large language model can be a general-purpose large language model (such as the GPT series, Gemini series, etc.) or a dedicated model fine-tuned for a specific task, as long as it has natural language reasoning capabilities. This application does not limit the specific type and number of parameters of the large language model, and those skilled in the art can choose according to actual computing resources and accuracy requirements.

[0140] It is understood that electronic device 100 can be a cloud server. The cloud server can be a physical server or virtual server deployed in a public cloud, private cloud, or hybrid cloud environment, such as an elastic cloud server (ECS), a bare metal server, or a cloud virtual machine.

[0141] Electronic device 100 can also be a local server. This local server can be a physical server deployed in an enterprise's internal data center or a private cloud server.

[0142] Electronic device 100 can also be a terminal device. The terminal device can be a personal computer (PC), laptop, workstation, tablet, or smartphone used by the developer.

[0143] Electronic device 100 can also be a combination of the above forms, such as a distributed system composed of cloud servers and terminal devices, in which different functional modules can be deployed on different electronic devices and work together through network communication.

[0144] It is understood that the specific type of electronic device 100 is not limited in the embodiments of this application, as long as it can execute the intelligent agent illusion positioning method provided in the embodiments of this application.

[0145] To better understand the technical solutions of the embodiments of this application, the devices involved in the embodiments of this application will be described below with reference to the accompanying drawings.

[0146] For example, Figure 10 A schematic diagram of the hardware structure of an electronic device 100 is shown according to an embodiment of this application.

[0147] like Figure 10 As shown, the electronic device 100 includes one or more (only one is shown in the figure) processors 110, memory 120, communication interface 130, and bus 140. The processors 110, memory 120, and communication interface 130 are interconnected via the bus 140.

[0148] The processor 110 may include one or more processing units, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), an application-specific integrated circuit, etc., for executing related programs to perform the intelligent agent illusion localization method of the method embodiments of this application.

[0149] The memory 120 may include one or more memories for storing data or one or more applications. The memory may be read-only memory (ROM), static storage device, dynamic storage device, random access memory (RAM), high-speed random access memory, double data rate synchronous dynamic RAM (DDR), high bandwidth memory (HBM), or non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0150] In some embodiments, if the processor is a CPU, the corresponding memory 120 is main memory. In some embodiments, if the processor 110 is a GPU, the corresponding memory 120 can be video memory.

[0151] Processor 110 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the data processing method of this application can be completed through software instructions in processor 110. The aforementioned processor 110 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 120. Processor 110 reads information from memory 120 and, in conjunction with its hardware, executes the intelligent agent illusion positioning method of the method embodiments of this application.

[0152] Communication interface 130 is used to enable communication between electronic device 100 and other devices or communication networks. Communication interface 130 may include wired or wireless communication interfaces, so that electronic device 100 can access the Internet via wired or wireless means, and obtain data from or send data to other devices based on the Internet.

[0153] Bus 140 is used to connect processor 110, memory 120, communication interface 130 and other possible modules or circuits.

[0154] In some embodiments, the electronic device 100 can be used to execute the intelligent agent illusion localization method corresponding to the above method embodiments, and will not be described again here to avoid repetition.

[0155] It should be understood that Figure 10 The structure of the electronic device 100 shown is only an example. In other embodiments, the electronic device 100 may include more or fewer modules, which is not limited here.

[0156] This application also provides a computer-readable storage medium storing at least one computer program instruction, at least one program segment, code set, or instruction set, which is loaded and executed by a processing circuit to implement the intelligent agent illusion positioning method provided in this application.

[0157] This application also provides a program product that includes instructions that, when executed, can implement the intelligent agent illusion positioning method provided in this application.

[0158] This application also provides a chip, including a processor coupled to a memory, for executing computer programs or instructions stored in the memory, so that the chip implements the intelligent agent illusion positioning method provided in this application.

[0159] Embodiments of this application can be implemented as computer programs or program code that execute on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0160] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0161] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0162] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0163] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0164] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0165] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0166] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.< / tasks>

Claims

1. A method for illusory localization of an intelligent agent, characterized in that, Applied to electronic devices, the method includes: Acquire trajectory data of the intelligent agent, wherein the trajectory data includes step information of N execution steps, where N is an integer greater than 1; Each of the N execution steps is subjected to anomaly detection using multiple anomaly detection methods, and a detection result is generated for each execution step. The different anomaly detection methods detect different types of objects. Based on the detection results of different anomaly detection methods for each step, the anomaly score for each step is calculated. Based on the abnormal score of each step, abnormal steps among the N execution steps are identified, wherein the abnormal steps include the hallucination initiation step.

2. The method according to claim 1, characterized in that, The calculation of the anomaly score for each step, based on the detection results of different anomaly detection methods corresponding to each step, includes: The anomaly score for each execution step is determined based on the number of anomaly detection methods that classify it as an anomaly; and / or, The abnormal score of each execution step is determined based on the preset weight corresponding to the abnormal detection method that determines the abnormality of each execution step.

3. The method according to claim 1, characterized in that, The hallucination initiation step is the execution step with the highest abnormal score among the abnormal steps.

4. The method according to claim 3, characterized in that, Each of the N execution steps includes a step number; and... If the number of execution steps with the highest abnormal score in the abnormal steps is greater than 1, the abnormal step with the smallest step number among the multiple execution steps with the highest abnormal score in the abnormal steps shall be taken as the hallucination initiation step.

5. The method according to claim 1, characterized in that, The various anomaly detection methods include at least two of the following: Detection methods based on parameter specification verification; Detection methods based on returning anomalies; A detection method based on factual statements and prior trajectories to determine implication; A detection method based on preconditions and pre-trajectory for implication determination; A detection method based on consistency verification between reasoning conclusions and preliminary intermediate results; A detection method that uses planned data and prior trajectories for verification.

6. The method according to claim 5, characterized in that, Each of the N execution steps includes a tool call list, which includes at least one tool call. Each tool call includes at least one of the following parameters: tool name, tool parameter name, parameter type, and a list of required parameters. and, Anomaly detection is performed on each of the N execution steps using the parameter specification verification-based detection method, generating a detection result for each execution step, including: Obtain a preset tool parameter pattern library, wherein the tool parameter pattern library includes at least one of the following tool parameters: tool name, tool parameter name, parameter type, and a list of required parameters; Check each tool call in the tool call list for each execution step to see if the parameters match the corresponding tool parameters in the tool parameter pattern library; For each execution step, if an abnormal parameter condition is detected, an anomaly is determined, and an anomaly flag corresponding to the detection method based on parameter specification verification is generated; wherein, The abnormal parameter conditions include at least one of the following: The tool name of at least one tool call in the tool call list of the execution steps does not exist in the tool parameter pattern library; The tool parameter name of at least one tool call in the tool call list of the execution steps does not exist in the tool parameter pattern library; The parameter type of at least one tool call in the tool call list of the execution steps does not match the expected type in the corresponding tool parameter pattern library; The tool parameter pattern library does not contain at least one required parameter for at least one tool call in the tool call list of execution steps.

7. The method according to claim 5, characterized in that, Each of the N execution steps includes a tool return result list, which includes at least one tool return result; and... Anomaly detection is performed on each of the N execution steps using the aforementioned detection method based on return anomaly matching, generating a detection result for each execution step, including: Obtain a preset abnormal keyword library, wherein the abnormal keyword library includes at least one abnormal keyword used to characterize abnormal results returned by the tool; Each tool return result in the tool return result list for each execution step is matched with the abnormal keyword database; For each execution step, if an abnormal result condition is detected, it is determined that an abnormality has been detected, and an abnormality marker corresponding to the detection method based on returned abnormality matching is generated; wherein, The abnormal conditions for the results include: at least one tool in the result list returned by the tool performing the steps returns a result that includes any abnormal keyword from the abnormal keyword library.

8. The method according to claim 5, characterized in that, Each of the N execution steps includes the output content of the agent; and... The anomaly detection method based on fact statements and prior trajectories is used to perform an implication determination on each of the N execution steps, generating a detection result for each execution step, including: Based on a large language model, factual statements are extracted from the agent's output at each execution step; The fact statement of each execution step and the preceding trajectory of each execution step in the trajectory data of the agent are respectively input into the large language model, and the large language model outputs the implication judgment result of the fact statement relative to the preceding trajectory, wherein the implication judgment result includes at least one of the following: support, contradiction, and no basis; For each execution step, if an anomaly condition satisfying the fact statement is detected, an anomaly is determined, and an anomaly marker corresponding to the detection method based on the implication determination of the fact statement and the preceding trajectory is generated; wherein, The exceptional conditions for the fact statement include at least one of the following: The implication determination result contains at least one contradictory determination; The proportion of judgments made without evidence in the implied judgment results exceeds the preset threshold.

9. The method according to claim 8, characterized in that, The extraction of factual statements from the agent's output at each execution step based on a large language model includes: For each execution step, corresponding to the agent's output including text output, atomic fact statements are extracted from the text output based on the large language model; or... Corresponding to the intelligent agent's output content only including tool calls, the corresponding atomic fact statements are extracted based on the parameters of the tool calls according to the large language model; wherein, The atomized fact statement is related to a specific result, state value, or numerical value.

10. The method according to claim 5, characterized in that, Each of the N execution steps includes a tool call list; and... The anomaly detection method based on the implication determination of preconditions and pre-trajectories is used to perform anomaly detection on each of the N execution steps, generating a detection result for each execution step, including: Based on the large language model, the precondition declarations implied by the tool call in each execution step are extracted. The precondition declarations are used to characterize the execution gap between the user command and the tool call. The precondition declarations include: environmental states or data conditions that should have been established before the tool call is executed but have not yet been verified. The precondition declarations for each execution step and the precondition trajectories for each execution step in the trajectory data of the agent are respectively input into the large language model. The large language model outputs the implication judgment result of the precondition declarations relative to the precondition trajectories, wherein the implication judgment result includes at least one of the following: support, contradiction, and no basis. For each execution step, if an abnormal condition satisfying the preconditions is detected, an anomaly is determined, and an anomaly marker corresponding to the detection method based on the implication determination of the preconditions and the pre-trajectory is generated; wherein, The abnormal conditions of the preconditions include: The implication determination result is contradictory or unfounded, and the execution step has been marked by at least one anomaly detection method other than the detection method based on the preconditions and pre-trajectories for implication determination.

11. The method according to claim 5, characterized in that, Each of the N execution steps includes a tool call list and agent output content; and... Anomaly detection is performed on each of the N execution steps using the consistency verification method based on reasoning conclusions and preliminary intermediate results, generating a detection result for each execution step, including: For each execution step, the corresponding intelligent agent output includes a computational derivation operation, and the derivation conclusion corresponding to the computational derivation operation is obtained. The computational derivation operation includes at least one of the following: a computational tool call, text content containing mathematical derivation, and text content containing computational summary type. Obtain the user task description and the tool name corresponding to the computational derivation operation; The derivation conclusion, user task description, tool name, and the previous verified intermediate result of the previous trajectory in the trajectory data of the agent for each execution step are input into the large language model. The large language model uses the previous verified intermediate result to re-derive the derivation result and compares the consistency of the re-derived result with the derivation conclusion. If the inference anomaly condition is met, an anomaly is determined to exist, and an anomaly marker corresponding to the detection method based on consistency verification between the inference conclusion and the preceding intermediate results is generated; wherein, The reasoning anomaly condition includes: the re-reasoning result is inconsistent with the derivation conclusion.

12. The method according to claim 11, characterized in that, The step of obtaining the derivation conclusion corresponding to the computational derivation operation includes: The operation corresponding to the calculation derivation class is a calculation tool call, and the output result of the calculation tool call is used as the derivation conclusion; For calculation derivation operations that correspond to text content containing mathematical derivations or text content containing calculation summaries, the conclusions expressed in the text content are taken as the derivation conclusions.

13. The method according to claim 5, characterized in that, Each of the N execution steps includes the output content of the agent; and... Anomaly detection is performed on each of the N execution steps using the detection method based on plan data and prior trajectory verification, generating a detection result for each execution step, including: Identify whether the output of the agent at each execution step contains content of the plan or task decomposition type; For each execution step, corresponding to the content in the agent's output that includes a plan or task decomposition type, specific factual data is extracted from the agent's output. The specific factual data includes at least one of the following: numbers, assignment expressions, dates, names of people, and names of organizations. If an anomaly is detected that meets the conditions for abnormal data in the plan, an anomaly is determined to exist, and an anomaly marker corresponding to the detection method based on the verification of the plan data and the previous trajectory is generated; wherein, The abnormal conditions in the planned data include: The specific factual data in the output content of the intelligent agent contains target specific factual data, wherein each execution step has no basis in the preceding trajectory of the intelligent agent's trajectory data for the absence of the target specific factual data.

14. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device; as well as The processor is one of the processors of the electronic device, used to execute instructions stored in the memory to implement the method of any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, implement the method of any one of claims 1 to 13.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed, implement the method of any one of claims 1 to 13.

Images