Intelligent agent strengthening method

Abstract

The invention discloses an agent strengthening method. An Actor generates text reasoning or environment interaction actions; the Alfword environment interaction system generates a feedback reward or a feedback action of the reasoning problem; the Evaluator uses two modes of repeated action detection and step efficiency check to realize trajectory evaluation of the intelligent agent; after the track is judged to be failed, starting a Self-Reflection module to generate an reflection text; after the reflection text is generated, a verifier is started, the verifier processes input information through a prompt prompt word project and then submits the input information to the large language model to return binary judgment, and if the input information is returned as False, the reflection text is regenerated; and if the text is True, storing the reflection text to an experience module. The intelligent agent strengthening method is realized on the basis of a Reflection architecture, can accommodate a large amount of reflection data, greatly shortens the retrieval time, and meets the high-efficiency requirement required by a large model.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agents in computer systems, and specifically relates to a method for enhancing intelligent agents. Background Technology

[0002] Recent research has made significant progress in autonomous decision-making agents driven by large language models. However, generating satisfactory responses in a single instance remains challenging, especially for some rare or difficult queries. Therefore, a better approach is needed. Although traditional reinforcement learning techniques perform well in sequential decision-making, their reliance on large amounts of interactive data and costly gradient updates makes them unsuitable for the efficiency requirements of agents based on large language models.

[0003] Against this backdrop, lightweight frameworks (such as ReAct and Toolformer) achieve interaction with the environment by generating executable operations (such as API calls), thus avoiding the overhead of model fine-tuning. However, these methods still suffer from limited efficiency in experience reuse and a lack of explicit error correction mechanisms. The Reflexion framework employs a language feedback reinforcement mechanism, generating experience summaries through self-reflection to optimize the agent's behavior in a non-parametric manner. Its core innovation is to convert environmental feedback into natural language reflections stored in a local memory bank for use in subsequent tasks, thereby promoting a human-like "trial-and-error-reflection-improvement" learning cycle. However, Reflexion's original design still has two significant drawbacks: 1) the storage mechanism is inefficient and does not support large-scale knowledge accumulation and semantic retrieval; secondly, the quality of reflection is uncontrollable because the generated content is stored without verification, which may affect downstream decisions due to incorrect experiences. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an agent reinforcement method based on the Reflexion architecture, which can accommodate a large amount of reflective data and significantly shorten the retrieval time, meeting the high efficiency requirements of large models.

[0005] The objective of this invention is achieved through the following technical solution: an agent reinforcement method, with the following architecture:

[0006] Actor is responsible for generating textual reasoning or environmental interaction actions;

[0007] After receiving text reasoning or environmental interaction actions generated by the Actor, the Alfworld environment interaction system generates a reward for correct or incorrect reasoning, or generates an obs feedback action.

[0008] The Evaluator uses two methods—repeated action detection and step efficiency check—to evaluate the agent's trajectory. Repeated action detection means that if the agent performs the same operation for three consecutive steps and receives the same response, it is considered to be trapped in a loop, and the trajectory is considered to have failed. Step efficiency check means that the agent has not completed the task after 30 steps. If so, it is considered to be inefficient planning and is considered to have failed. Otherwise, it is considered to have succeeded.

[0009] After the Evaluator determines the trajectory as failed, it activates the Self-Reflection module. The Self-Reflection module, based on the question provided by the user and the interaction information between the Actor, the Alfworld environment interaction system, and the Evaluator, passes it to the large language model through prompt word engineering to generate reflection text.

[0010] After generating the reflection text, the validator is activated. The input information includes the user-provided question, the feedback reward or feedback action generated by the Alfworld environment interaction system, the reflection text, and the correct answer. The validator processes the input information through prompt word engineering and then passes it to the large language model to return a binary judgment. If the return is False, a regeneration instruction is sent to the Self-Reeflextion module to regenerate the reflection text. If the return is True, the reflection text is stored in the experience module.

[0011] The validator employs a hierarchical reflective quality assessment; it uses a two-layer quality assessment pipeline: Tier-1 performs lightweight semantic filtering, and Tier-2 performs LLM-based deep verification; the entire process is as follows:

[0012] Tier-1 calculates the semantic similarity between the generated reflection text and the basic fact answer. If the semantic similarity is greater than the preset threshold θ, the reflection text is judged as high-quality reflection text and stored in the reflection library; otherwise, it is regarded as low-quality reflection text and Tier-2 is executed.

[0013] Tier-2 calls LLM to evaluate the reflections of Tier-1's failure, creates prompt text, proposes evaluation criteria with three evaluation dimensions, packages task and contextual information, and together with the correct answer and the reflection text from the previous failure step, independently submits it to the large language model. The large language model then independently judges whether the given reflection information meets the evaluation criteria. If the large language model determines that it passes the evaluation, it returns True and stores the reflection text; if it determines that it fails, it returns False and proceeds to the step of regenerating the reflection text.

[0014] The beneficial effects of this invention are:

[0015] 1. At the storage architecture level, vector-based databases replace local text storage, enabling sublinear time similarity search, cross-task semantic-aware retrieval, and efficient reuse of large-scale reflection based on source-based metadata preservation.

[0016] 2. At the quality assurance level, the hierarchical evaluation mechanism combines rapid semantic screening with analysis based on large language models (accuracy / feasibility / logical consistency verification), allowing only high-quality reflections while limiting token and latency costs.

[0017] 3. At the generation optimization level, a multi-round reflection generation loop is introduced. The prompts are dynamically adjusted and the number of attempts is limited through verification feedback, thereby improving the depth and stability of reflection without requiring parameter updates.

[0018] The method of this invention is implemented based on the Reflexion architecture, which can accommodate a large amount of reflexive data and significantly shorten retrieval time, meeting the high efficiency requirements of large models. At the same time, this method provides better control over the quality of reflexes, avoiding the pollution of the reflex repository by low-quality reflexes and thus preventing negative impacts on the agent. Using the method of this invention, a higher success rate can be achieved in decision-making and reasoning tasks compared to older frameworks. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system structure for enhancing the intelligent agent according to the present invention;

[0020] Figure 2 A diagram illustrating the generation of multiple rounds of reflection. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, the present invention provides an agent enhancement method in which the system retains the core module of Refelxion, and the specific architecture is as follows:

[0023] The Actor is responsible for generating text-based reasoning or environmental interaction actions. Using a large language model limited by prompt words, the Actor generates text-based reasoning or environmental interaction actions based on the questions provided by the user (including environmental or reasoning information). The generated text-based reasoning or environmental interaction actions are represented as Actions, and the specific content of the Actions is generated by the large language model based on the questions provided by the user.

[0024] After receiving text reasoning or environmental interaction actions generated by the Actor, the Alfworld environment interaction system generates a reward for correct or incorrect reasoning, or generates an obs feedback action.

[0025] The evaluator categorizes an agent's trajectory into success / failure (or provides a reward for a specific task); a trajectory is a set of actions that constitute multiple actions required to complete a user's task.

[0026] The Evaluator uses two methods—repeated action detection and step efficiency checking—to evaluate the agent's trajectory. Repeated action detection determines the trajectory as a failure if the agent performs the same operation and receives the same response for three consecutive steps. Performing the same operation means the actor generates the same text reasoning or environmental interaction action, and the same response means the Alfworld environmental interaction system generates the same feedback reward or feedback action. Step efficiency checking checks whether the agent fails to complete the task after 30 steps; if so, it is considered inefficient planning and deemed a failure; otherwise, it is considered a success.

[0027] After the Evaluator determines a trajectory as failed, it activates the Self-Reflection module. The Self-Reflection module, based on the user-provided question and the interaction information between the Actor, the Alfworld environment interaction system, and the Evaluator (because reflection requires seeing all previous actions to identify errors, it packages the Actor's textual reasoning or environmental interaction actions, the Alfworld environment interaction system's feedback rewards or feedback actions, and the Evaluator's judgment result together as external feedback and sends it to the Self-Reflection module to assist in generating reflection), uses Prompt prompts to power a large language model, generating reflective text to diagnose errors and guide improvements.

[0028] After generating the reflection text, enable the validator. The validator is used to evaluate the quality of the reflection before it is loaded into memory. The input information includes the questions provided by the user, the feedback rewards or feedback actions generated by the Alfworld environment interaction system (as auxiliary information), the reflection text, and the correct answers (generally, the correct answers are attached to the dataset, which includes the question content and the answers. Both closed-ended answers (that is, questions where the answer is a choice of several options) and open-ended answers (the answers do not provide any options) are supported).

[0029] The validator processes the input information using prompt word engineering and then passes it to the large language model for a binary judgment. If the return is False, the validator sends a Regenerate instruction to the Self-Reeflextion module, instructing it to regenerate the reflection text. If the return is True, the reflection text is stored in the Experience module. All generated reflections are reviewed by the validator; low-quality instances are rejected and trigger controlled regeneration, while only high-quality reflections are submitted to the Experience module.

[0030] The validator employs a hierarchical reflective quality assessment. To mitigate the risk of incorporating low-quality reflectives into the knowledge base while controlling API-related costs, a two-tier quality assessment pipeline is used: Tier-1 performs lightweight semantic filtering, and Tier-2 performs LLM-based deep validation. The cascaded design uses fast, non-API operations to filter out most irrelevant content and retains expensive model calls for difficult cases where semantic proximity alone is insufficient. The entire process is as follows:

[0031] Tier-1 calculates the semantic similarity between the generated reflective text and the basic fact answer. If the semantic similarity is greater than a preset threshold θ (set to θ = 0.7 in this embodiment), the reflective text is classified as high-quality reflective text and stored in the reflective library; otherwise, it is considered low-quality reflective text, and Tier-2 is executed for further evaluation. This stage does not require external API calls and relies only on vector operations. It provides a low-cost gate to remove semantically irrelevant content; however, it operates on the surface of similarity and may misjudge high-quality paraphrases or fail to assess whether the reflection truly solves the underlying problem.

[0032] Tier-2 calls LLM to evaluate the reflections on the failure of Tier-1. LLM uses task and environmental context to evaluate the reflections on the failure of Tier-1; it creates prompt text, proposes evaluation criteria with three evaluation dimensions, packages the text information of task and environmental context, and together with the correct answer and the reflection text of the previous failure, it independently submits it to a large language model (e.g., DeepSeek model), allowing the large language model to independently judge whether the given reflection information meets the evaluation criteria based on the provided evaluation criteria. If the large language model judges it as passing the evaluation, it returns True and stores the reflection text; if it judges it as failing, it returns False and enters the step of regenerating the reflection text. The three evaluation dimensions are: (1) accuracy - reflecting whether the main cause of the failure is accurately identified; (2) feasibility - whether the proposed improvement strategy is specific and executable; (3) consistency - whether the reasoning is consistent with the correct answer.

[0033] To improve robustness and reduce cost efficiency, the threshold θ can be calibrated on a small development set, and the marginal band (close to the threshold) can be forwarded to Layer 2 to reduce false negatives. When the basic fact answer is unavailable (e.g., in an interactive environment), Layer 1 can be bypassed or replaced with similarity to successful trajectories or target specifications, while Layer 2 uses rule-based checking operations conditioned on environmental feedback. The binary decisions and brief principles generated by Layer 2 are fed back into a multi-round generation loop to guide timely updates and targeted regeneration, thereby improving reflective utility through continuous trials.

[0034] This invention employs a multi-round reflection approach: it introduces an iterative optimization loop, extending single-round reflection generation into a bounded, feedback-driven process comprising four stages: 1) Reflection generation is triggered and initiated when the trajectory is determined to be a failure; 2) Generation—an initial reflection is generated based on the current environmental state and historical trajectories; 3) Evaluation—hierarchical quality is applied as the evaluation standard; 4) Decision—if the assessment is passed, reflection continues; otherwise, if the number of attempts N < Nmax (default Nmax = 3), regeneration is performed based on structured quality feedback. This process is as follows: Figure 2 As shown.

[0035] This optimization relies on a dynamic, adaptive cueing mechanism. When a reflection is rejected due to "insufficient feasibility," subsequent cueing automatically includes operational constraints, such as "specific operational instructions are required (e.g., search / find)," and requires executable steps, tool selection, and verifiable success criteria. When rejection is due to accuracy or consistency issues, cueing adds error-locating cueing (e.g., highlighting conflicting evidence) and counterfactual cueing to guide revision. Furthermore, the decoding parameters (temperature / top-p) are appropriately adjusted during trials to balance exploration and determinism. The loop terminates early upon acceptance or returns the highest-scoring candidate when Nmax is reached, thereby improving the quality of continuous iterations of reflection while enforcing cost constraints.

[0036] In addition, the experience module uses vector storage to store the reflection text. This vectorized reflection storage system uses a similarity search library to build a high-performance approximate nearest neighbor (ANN) index and uses the Facebook AI Similarity Search Library (FAISS) to build high-performance ANN retrieval. It supports k-nearest neighbor search (k-NN) with sublinear query complexity and can be scaled to more than 106 reflection entries.

[0037] The text embedding pipeline integrates the DeepSeek model through Alibaba Cloud's Bailian API to encode reflective text into 1024-dimensional dense vectors in real time, thereby capturing significant semantic features suitable for similarity retrieval.

[0038] The lightweight metadata layer binds each vector to its source (including the original question, reflection generation strategy, timestamp, and other contextual fields), thereby enabling auditable reuse and downstream analysis.

[0039] The storage subsystem follows the singleton pattern to ensure globally unique instances and employs a hybrid mode (raw text + vector representation) to balance retrieval latency, accuracy, and storage footprint. It exposes a write interface and a retrieval interface; the latter returns the top k historical reflections that are semantically most similar to the current task state (i.e., a simple similarity calculation is performed on the vectorized reflection text content stored in the preceding vector database, and then the top k reflection texts are selected through Top-K filtering).

[0040] This design enables precise cross-task reuse in downstream scenarios such as AlfWorld project search, replacing the coarse heuristic approach of the original framework that only retained the last three reflections. Furthermore, the index supports batch collection and online incremental updates for continuous accumulation and can be configured with standard index types and distance metrics to meet application-level trade-offs between recall and throughput, thereby promoting stable operation at large scale.

[0041] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for enhancing an intelligent agent, characterized in that, The architecture is as follows: Actor is responsible for generating textual reasoning or environmental interaction actions; After receiving text reasoning or environmental interaction actions generated by the Actor, the Alfworld environment interaction system generates a feedback reward for the reasoning problem or an obs feedback action. The Evaluator uses two methods—repeated action detection and step efficiency check—to evaluate the trajectory of the agent. Repeated action detection means that if the agent performs the same operation and receives the same response for three consecutive steps, it is considered to be trapped in a loop, and the trajectory is considered to have failed. Step efficiency check means that the agent has not completed the task after 30 steps. If so, it is considered to be inefficient planning and is considered to have failed. Otherwise, it is considered to have succeeded. After the Evaluator determines the trajectory as failed, it activates the Self-Reflection module. The Self-Reflection module, based on the question provided by the user and the interaction information between the Actor, the Alfworld environment interaction system, and the Evaluator, passes it to the large language model through prompt word engineering to generate reflection text. After generating the reflection text, the validator is activated. The input information includes the user-provided question, the feedback reward or feedback action generated by the Alfworld environment interaction system, the reflection text, and the correct answer. The validator processes the input information through prompt word engineering and then passes it to the large language model to return a binary judgment. If the return is False, a regeneration instruction is sent to the Self-Reeflextion module to regenerate the reflection text. If the return is True, the reflection text is stored in the experience module.

2. The agent reinforcement method according to claim 1, characterized in that, The validator employs a hierarchical reflective quality assessment; it uses a two-layer quality assessment pipeline: Tier-1 performs lightweight semantic filtering, and Tier-2 performs LLM-based deep verification; the entire process is as follows: Tier-1 calculates the semantic similarity between the generated reflection text and the basic fact answer. If the semantic similarity is greater than the preset threshold θ, the reflection text is judged as high-quality reflection text and stored in the reflection library; otherwise, it is regarded as low-quality reflection text and Tier-2 is executed. Tier-2 calls LLM to evaluate the reflections of Tier-1's failure, creates prompt text, proposes evaluation criteria with three evaluation dimensions, packages task and contextual information, and together with the correct answer and the reflection text from the previous failure step, independently submits it to the large language model. The large language model then independently judges whether the given reflection information meets the evaluation criteria. If the large language model determines that it passes the evaluation, it returns True and stores the reflection text; if it determines that it fails, it returns False and proceeds to the step of regenerating the reflection text.

Images