Autonomous agent-oriented large language model element cognitive correction self-updating reward method

Abstract

The invention relates to an autonomous agent-oriented large language model element cognitive correction self-updating reward method, which comprises the following steps of: firstly, automatically generating an initial reward function by utilizing a large language model (LLM) according to high-level task description by using the AURA; then, meta-cognition on calculation is realized through a double-loop architecture: a rapid internal loop carries out collaborative evolution of strategies and rewards through a layered attention reward synthesizer; a slower outer cycle serves as a meta-cognitive layer, and a multi-dimensional diagnosis vector composed of strategy dynamics, value-authenticity alignment and reward architecture adaptability is continuously monitored so as to recognize anomalies in the learning process. The closed-loop feedback mechanism endows the autonomous agent with the capability of carrying out introspection and self-correction on the award function of the autonomous agent, and normal form transformation from static target specification to dynamic target correction is formed. And the robustness, the sample efficiency and the final performance of the intelligent agent in the face of sparse rewards, protocol vulnerabilities and environment uncertainty are enhanced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous intelligent agents, and more specifically, to a dynamic reward architecture management method for reinforcement learning intelligent agents, which utilizes a large language model to achieve online diagnosis and structural self-updating of the agent's objective function. Background Technology

[0002] As autonomous agents are increasingly applied in robotics, automated processes, and other fields, their success hinges on their ability to accurately understand and execute high-level goals without continuous human supervision. However, traditional reinforcement learning methods heavily rely on manually designed, static reward functions. In complex tasks, this not only easily leads to the problem of sparse rewards, making it difficult for the agent to obtain effective learning signals, but also makes it highly susceptible to goal misalignment when faced with incomplete task specifications or dynamically changing environments, thus limiting its autonomy and reliability. Autonomous agents learn optimal policies through trial-and-error interaction with the environment using the reinforcement learning (RL) paradigm, which is a core technology for automating complex tasks. The success of RL heavily depends on a predefined scalar reward function that transforms high-level task goals into optimizable signals. However, the process of precisely formalizing abstract task intentions into reward functions—reward engineering—is a recognized bottleneck in the field. It is not only time-consuming and labor-intensive, but also often leads to low learning efficiency for agents facing real-world tasks with sparse rewards. The reward shaping technique introduced to address this problem, while providing dense guidance, is highly susceptible to design flaws, leading to "specification gaming"—where agents exploit shortcuts in reward rules, maximizing reward scores but violating the designer's true intentions.

[0003] To automate the generation of reward functions, researchers have proposed various approaches. Early methods, such as inverse reinforcement learning (IRL), infer rewards from expert demonstrations, but their reliance on high-quality expert data limited their application. In recent years, the emergence of large language models (LLMs) has opened a new frontier: leveraging their powerful commonsense reasoning and code generation capabilities, they directly translate natural language task descriptions into executable reward functions. This approach significantly lowers the barrier to reward design and represents a major technological advancement in the field.

[0004] However, all current reward generation methods based on Large Language Models (LLMs) still suffer from fundamental limitations in their underlying paradigms. Specifically, these methods either rely on online intervention through external human feedback, on post-hoc attribution analysis of macroscopic failure trajectories, employ offline batch generation and selection, or only use LLM for secondary tasks such as auxiliary state representation learning, without addressing the structural correction of the core external reward function. These models share the commonality of forming an "open-loop" or "externally driven" workflow, implicitly assuming a highly unstable "first-time correctness" assumption in complex dynamic environments, or completely entrusting the triggering and judgment of corrections to external entities. This paradigm completely lacks real-time feedback on the agent's internal learning dynamics during subsequent real-world interactions with the environment when generating initial rewards. Therefore, any minute semantic deviations, logical fallacies, or incorrect presuppositions about the physical world in the initial reward function will be amplified during the learning process, irreversibly misleading the agent into local optima or causing pathological behaviors such as policy oscillations. Importantly, all the aforementioned technical frameworks lack a truly autonomous introspective machine, failing to enable agents to diagnose online whether the root cause of learning difficulties lies in the structural flaws of the reward function itself, and thus fundamentally and structurally self-correct their value system. Therefore, there is an urgent need in this field for a novel technical framework that transforms the reward function from a static, externally designed "instruction set" into an "adaptive system component" capable of co-evolving with the agent's strategy, self-examining, and dynamically correcting itself. This would fundamentally overcome the fragility, inefficiency, and risk of goal deviation inherent in existing methods due to the static rigidity of the reward function or reliance on external intervention.

[0005] AURA integrates LLM into a persistent, autonomous cognitive and reasoning engine, constructing a computational metacognitive closed loop. Its core advantage lies in its ability to proactively identify structural risks before pathological behavior becomes entrenched, without relying on delayed failure signals or external intervention. Instead, it uses its MCM module to monitor and predictively diagnose multidimensional internal learning dynamic vectors, such as value-truth alignment, in real time, proactively identifying these risks. Furthermore, it provides LLM not with qualitative natural language opinions, but with quantified adaptive instructions containing "abnormal gradient vectors," transforming the correction task from open-ended semantic guessing into a targeted mathematical structural reshaping. This autonomous correction mechanism, driven by internal diagnosis, guided by quantified instructions, and ensured by a dual-loop architecture for efficient collaboration, enables AURA to address deep-seated value misalignment problems that dynamically emerge during the learning lifecycle, which other solutions cannot handle. This represents a key technological leap towards robust and adaptive intelligent agents. Summary of the Invention

[0006] This invention presents a metacognitive correction and self-updating reward method for large language models of autonomous intelligent agents.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a metacognitive correction and self-updating reward method for large language models of autonomous intelligent agents, comprising the following steps:

[0008] Step 1, Initialize the reward architecture and learning environment: For a high-level natural language task description assigned to the autonomous agent, an initial reward architecture consisting of multiple executable reward factors is generated using a Large Language Model (LLM) with structured prompts. Simultaneously, the policy network and Hierarchical Attention Reward Synthesizer (HARS) network of the reinforcement learning agent are initialized, and an experience replay pool is established to store training data. Step 2, Perform inner-loop co-evolutionary training: Under the given current reward architecture, the agent is driven to interact with the environment, and based on the collected interaction trajectories, the parameters of the policy network and the HARS network are iteratively and co-evolved by minimizing the joint optimization objective.

[0009] Step 3, perform metacognitive monitoring in the outer loop: In parallel with the inner loop, continuously extract and compute multidimensional diagnostic vectors from the co-evolutionary training process. These vectors are used to quantify the health status of the current learning process from three dimensions: policy dynamics, value-truth alignment, and reward architecture adaptability.

[0010] Step 4, perform learning state anomaly detection: based on the historical data stream of the diagnostic vector, a data-driven, adaptive triggering mechanism is used to identify and locate pathological regions in the learning process; if not entered, return to step 2 to continue the current inner loop training; if entered, proceed to step 5.

[0011] Step 5, Perform diagnostic-driven reward architecture reconstruction: Activate the semantic reward reconstruction module, which calls a large language model and provides it with adaptive instructions containing the current reward architecture and quantified diagnostic data to structurally modify the factor set of the current reward architecture and generate a new reward architecture.

[0012] Step 6, Integrate the evolved new reward architecture: Integrate the new reward architecture into the learning system using a systematic integration process that includes multi-stage verification and selective network parameter retention, to replace the original reward architecture.

[0013] Step 7: Determine if training should terminate: Determine if the preset maximum training period has been reached or if the convergence condition has been met; if not, return to step 2 and start the next round of inner loop training under the new reward framework; if the condition is met, end the training.

[0014] Further, in step 1, the reward architecture and learning environment are initialized: for a high-level natural language task description assigned to the autonomous agent, an initial reward architecture consisting of multiple executable reward factors is generated using a large language model (LLM) through structured prompts, specifically including:

[0015] Structured Chain-of-Thought Prompting and Reward Factor Generation: To guide the large language model in high-quality task decomposition and reward factor generation, this invention designs a "meta-prompt" template. This template is not a simple single-round question-and-answer session, but rather forces the large language model to perform structured chain-of-thought reasoning by setting a series of logical stages, such as [System Role Setting] -> [Task Goal Decomposition] -> [Reward Factor Identification] -> [Code Generation and Verification]. In the [Reward Factor Identification] stage, guiding questions (e.g., "To achieve the final goal, what intermediate sub-goals should the agent pursue? Which states are desirable? Which behaviors should be encouraged or punished? Please express them as functions of state(s) and action(a).") stimulate the generation of a semantically rich and task-relevant set of reward factors.

[0016] Factor Differentiability and Annealing Mechanism: To ensure that the reward factor code generated by a large language model can support gradient-based optimization, this invention includes an automated code rewriter. This rewriter is responsible for identifying and replacing non-discontinuous or non-differentiable operations in the code, such as pointer functions. (It is 1 when condition c is true, and 0 otherwise) Replace it with its differentiable approximation, namely the Sigmoid function: Here, g(s,a) is a function that transforms condition c into a scalar, while β is a hyperparameter that controls the slope. To balance exploration and exploitation, the value of β is gradually increased using an annealing strategy as training progresses, from a small value (smooth function curve, conducive to gradient propagation and exploration) to a large value (function curve close to a step function, conducive to accurate convergence in the later stages of learning).

[0017] Furthermore, in step 2, the co-evolutionary training of the inner loop is performed. The core of this process lies in employing the Hierarchical Attention Reward Synthesizer (HARS) network. This network dynamically synthesizes reward signals through a mechanism that decouples evaluation in the factor space and time series, but deeply shares parameters. Specifically, this includes: hierarchical context-aware value synthesis in the factor space. The HARS network introduces the concept of hierarchical context, meaning that the importance of the reward factor depends not only on the local state s at the current time step t. t It should also be subject to the global context vector c, which represents the global strategic intent of the entire interaction trajectory τ. τ Modulation. To this end, a context encoder based on a self-attention mechanism is first used to modulate a complete state sequence (s0,...,s...). T-1 The encoding is the global context vector c. τ Subsequently, at each time step t, a multi-head attention mechanism is employed to process the local state s. t and the global context c τ Together as a query, it refers to the factor vector f(s) instantiated at time step t, provided by the current reward architecture. t ,a t Dynamic weighting is performed to generate a high-dimensional instantaneous reward vector r′ rich in contextual information. φ (s t ,a t The multi-head attention mechanism's insensitivity to the number and order of input factor vectors is a key technical guarantee that ensures the HARS network can seamlessly adapt to dynamic structural changes in the reward architecture caused by the outer loop.

[0018] Parameter-sharing temporal credit attribution in time series: To accurately allocate sparse true rewards provided by the environment to key time steps that contribute to the results, the HARS network employs an efficient parameter-sharing mechanism to avoid introducing a separate, overfitting-prone temporal attention network. Specifically, the context encoder is directly reused in generating the global context vector c. τ The normalized internal attention weight distribution generated at that time is α(ψ)={α0,...,α T-1 The weight distribution α(ψ) is directly used as the temporal credit coefficient to measure the contribution of each time step to the final task result. Finally, the temporal credit coefficient is element-wise multiplied with the instantaneous reward vector (mapped back to a scalar via a multilayer perceptron), and scaled using the trajectory length T to synthesize the final scalar reward signal r that simultaneously contains information on "where it matters" (factor space attention) and "when it matters" (temporal series attention).φ,ψ (s t ,a t ).

[0019] Parameter Co-update Based on Joint Optimization Objective: The updates of the policy network parameters θ and the HARS network parameters φ and ψ are minimized by using gradient descent to minimize the joint optimization objective. Driven by this, the joint optimization objective consists of two linearly weighted parts:

[0020]

[0021] (1) Strategy learning loss This term represents the policy loss function of any standard reinforcement learning algorithm (e.g., the Actor loss of PPO or SAC). Its advantage function, or Q-value, is calculated entirely based on our dynamically generated reward signal r. φ,ψ This makes the AURA framework algorithm-independent.

[0022] (2) Reward Alignment Loss It penalizes the deviation between the cumulative dense reward predicted by the HARS network and the sparse true reward R(τ) provided by the environment through a mean squared error loss function, the mathematical form of which is: This ensures that the learning of the reward architecture always aligns with the objective goals of the real world.

[0023] Furthermore, in step 3, the core of performing metacognitive monitoring in the outer loop lies in conducting multi-dimensional and systematic quantitative diagnosis of the learning process to generate a D-dimensional diagnostic vector that can comprehensively characterize the current health status of the learning process. The components of this vector are extracted and formally defined from the following three mutually orthogonal dimensions, specifically including:

[0024] (1) Policy Dynamics Dimension: This aims to quantify the policy network π θ The evolutionary behavior of the policy is used to identify convergence and stability issues. Indicators for this dimension include the policy entropy gradient (d). PEG ), which is used to quantify policy exploration (in terms of policy entropy). The rate of change of policy entropy over time is used to diagnose whether exploratory power is being prematurely dissipated. Its mathematical formalization is defined as the rate of change of policy entropy over a single evaluation period ΔT:

[0025]

[0026] This dimension also includes time-series policy divergence (d TPDThis function is used to quantify the update magnitude and stability of a policy in the parameter space. It is measured by calculating the KL divergence between consecutive policy snapshots to diagnose whether the learning process is in a high-energy oscillation state. Its mathematical formal definition is:

[0027]

[0028] (2) Value-Veracity Alignment Dimension: As a core component in diagnosing the "reward hacking" phenomenon, this dimension aims to measure the consistency between learned values ​​and objective reality. Indicators for this dimension include value-veracity relevance (d...). VVC Its direct quantification is the value function V guided by the learned dense rewards. φ The linear correlation between the predicted value and the actual sparse environmental return R(τ) is measured by the Pearson correlation coefficient ρ between the predicted value and the actual Monte Carlo return G(s), which is mathematically formalized as:

[0029]

[0030] in, It is the expected return of the real reward starting from state s.

[0031] (3) Reward Architecture Adaptability Dimension: This dimension directly examines the internal dynamics of the HARS modules to diagnose their ability to continuously learn and adapt. This dimension includes the following three metrics:

[0032] Factor attention plasticity (d) FAP ), which is used to measure the factor space attention distribution ω t Whether (φ) has become fixed is quantified by calculating the Jensen-Shannon divergence of the attention distribution between consecutive evaluation cycles. Its mathematical formalization is defined as:

[0033]

[0034] Temporal attention plasticity (d TAP ), which is used to measure the time-series credit allocation mechanism α t Whether (ψ) is stuck in a fixed state is quantified by calculating the Jensen-Shannon divergence of the temporal attention distribution between consecutive evaluation cycles. Its mathematical formalization is defined as:

[0035]

[0036] Residual prediction error (d) RPE), which is used to diagnose the current factor set. Is this sufficient to explain the true returns, aligning losses by quantifying rewards? It is defined by the lower bound of convergence within an evaluation window. Its mathematical formal definition is:

[0037]

[0038] Furthermore, in step 4, abnormal learning state detection is performed. Its core lies in employing a data-driven, adaptive triggering mechanism to identify and locate pathological regions during the learning process. Specifically, this includes:

[0039] This mechanism first models the probabilistic health learning state through online kernel density estimation. It then uses the historically generated diagnostic vector sequence {d} (1) ,d (2) ,...,d (k) The data is treated as a high-dimensional data stream, and an online kernel density estimation method is used to dynamically construct and continuously update a probability density function p for the "healthy" learning state. healthy (d) The update mechanism of this health status model is related to the external performance of the agent. That is, only when the average real reward obtained by the agent in the most recent evaluation period is higher than a dynamically updated historical performance baseline, is its corresponding diagnostic vector determined as a "healthy sample" and added to a prototype buffer for density estimation. The probability density function p of the health state is... healthy The log-likelihood form of (d) is given by the following equation:

[0040]

[0041] Subsequently, the mechanism performs anomaly score calculation based on negative log-likelihood. In each outer loop cycle k, the current diagnostic vector d is calculated based on the aforementioned health state probability model. (k) abnormal score S(d) (k) This score is defined as the negative log-likelihood of the current diagnostic vector under the health state model, which directly quantifies the degree of deviation of the current learning state from all known health states. Its mathematical formalization is:

[0042] S(d (k) ) = -logp healthy (d (k) )

[0043] Finally, the mechanism employs adaptive triggering logic based on dynamic statistical quantiles. To avoid using fixed trigger thresholds that require manual adjustment, this invention uses a dynamic thresholding method based on historical data distribution. The trigger threshold τ is... triggerDefined as the score of all historically observed anomalies {S(d)} (1) ),...,S(d (k) A certain high quantile q of}, this threshold is dynamically adjusted as the number of observed data increases. The outlier score S(d) of the current period is [value missing]. (k) For N consecutive cycles (where N≥1 is a preset integer used to increase robustness), the value remains above the dynamic trigger threshold τ. trigger Only then is it confirmed that the learning process has entered the pathological region requiring intervention, and step 5 is executed. The mathematical formalization of the dynamic trigger threshold is defined as follows:

[0044] τ trigger =Percentile({S(d (1) ),...,S(d (k) )},q)

[0045] Further, in step 5, a diagnostic-driven reward architecture reconstruction is performed: the semantic reward reconstruction module is activated. This module calls a large language model and provides it with adaptive instructions containing the current reward architecture and quantified diagnostic data to structurally modify the factor set of the current reward architecture and generate a new reward architecture, specifically including:

[0046] The process begins with the dynamic construction of diagnostic-driven adaptive instructions. When the outer loop is triggered, the semantic reward reconstruction module dynamically constructs a structured adaptive instruction provided to the LLM. This instruction is not a set of hard-coded rules or templates, but a dynamically aggregated data structure designed to provide sufficient, evidence-based context for the LLM's reasoning, and systematically includes the following four key information components:

[0047] (1) Task Axioms: Restating the fundamental, high-level task objectives defined in step 1. And key environmental constraints, serving as the logical foundation for all subsequent inferences in LLM.

[0048] (2) Current Reward Architecture Specification: A detailed description of the currently used and soon-to-be-modified set of reward factors. This includes the mathematical expression for each factor, the parameters used, and the expected semantic function interpretation.

[0049] (3) Quantitative, Multi-Level Diagnostic Data: This directly presents the pure numerical diagnostic information generated in steps 3 and 4 without natural language "translation," including the anomaly score S(d) that triggers the alarm. (k) ), the complete diagnostic vector d that leads to the abnormality (k) And the abnormal gradient vector that indicates the direction of correction. This gradient vector geometrically indicates the steepest ascent direction from the current ill-conditioned point back to the high-density region of the healthy state, providing a quantitative and actionable directional guide for LLM correction.

[0050] (4) Structured Adaptive Tasking: An open-ended but authorizing instruction that requires the LLM to follow a specific evolutionary logic. This instruction requires the LLM to act as a systems engineer and respond sequentially: First, diagnose structural defects based on diagnostic data; second, propose an adaptive solution to correct the defects; and finally, generate a completely new and complete structural specification.

[0051] Subsequently, LLM-mediated structural reshaping and interpretable inference chain generation are performed. Upon receiving the adaptive instruction, the large language model is instructed to initiate a systematic response process that generates its complete inference chain. This process maps the input quantified diagnostic data to a semantic judgment of the current reward architecture and performs structural modifications based on this judgment. The structural modification operations that LLM can perform are diverse and flexible, enabling deep reshaping of the reward architecture. These operations include parameter tuning, factor reconstruction, factor addition, and factor pruning. Finally, the LLM outputs a completely new and complete reward architecture specification. This specification is a concrete implementation of its adaptive strategy and structural modification operations.

[0052] Furthermore, in step 6, the evolved new reward architecture is integrated: the new reward architecture is integrated into the learning system using a systematic integration process that includes multi-stage verification and selective network parameter retention to replace the original reward architecture, specifically including:

[0053] The process begins with multi-stage automated verification of candidate architectures. The new reward architecture specification is generated in step 5 and output by the large language model. Before being formally adopted, a rigorous, automated candidate architecture validation phase must first be completed to filter out invalid, insecure, or contradictory structural mutations. This validation phase consists of three consecutive validation steps:

[0054] (1) Structural Integrity and Syntactic Validation: Automatically parse the new specification to ensure that it is well-formed in terms of syntax and data structure, and that all specified mathematical expressions, function calls and parameter definitions are valid and unambiguous.

[0055] (2) Functional feasibility and numerical stability test: Instantiate each independent reward factor in the new specification and call it with boundary values ​​and random values ​​to confirm that it is computationally feasible and will not produce undefined behavior or numerical errors.

[0056] (3) Preliminary Performance and Objective Relevance Analysis: Using a static dataset containing historical interaction trajectories, the cumulative reward sequence generated by the new candidate architecture is calculated offline, and the correlation between this sequence and the actual task reward R(τ) recorded in the dataset is evaluated to filter out candidate schemes that are clearly inconsistent with the fundamental task objective. If an anomaly occurs, the anomaly case is returned and the reward architecture update is re-entered. After verification, this process finally employs a selective parameter retention strategy based on knowledge transfer. Only candidate architectures that pass all verification steps will be formally adopted as the new reward factor set. And trigger the parameter integration phase of the system. In this phase, in order to maximize the transfer of existing knowledge while adapting to the new value landscape, a selective parameter retention strategy is adopted, which differentiates the parameters of different networks: (1) Reinitialize the parameters related to the factor space: due to the semantic input space of the reward architecture (i.e., the factor set) The factor space attention parameter φ in the HARS network, which directly processes reward factors, has been fundamentally changed and will be reinitialized.

[0057] (2) Retain parameters related to temporal structure and policy behavior: The temporal credit attribution parameter ψ, which is responsible for the intrinsic temporal dynamic structure of the learning task, and the policy network parameter θ, which encapsulates the basic behavioral skills already learned by the agent, will be completely retained from the previous outer loop cycle. By retaining ψ and θ, the agent can inherit its learned prior knowledge of credit allocation based on "when" and behavioral priors on "how" to act, thereby enabling it to continue learning and adapting efficiently on the new value landscape, rather than starting from scratch.

[0058] Compared to existing technologies, the advantages of this invention are as follows: This invention proposes AURA (Auto-Updating Reward Architecture), a novel framework that reconstructs the reward function from a static design product into a dynamic system component that co-evolves with the policy. AURA first automatically generates an initial reward function based on a high-level task description using Large Language Models (LLMs), and then achieves computational metacognition through a dual-loop architecture: a fast inner loop performs co-evolution of policy and reward through a Hierarchical Attention Reward Synthesizer (HARS); while a slower outer loop acts as a metacognition layer, continuously monitoring a multi-dimensional diagnostic vector composed of policy dynamics, value-truth alignment, and reward architecture adaptability to identify anomalies in the learning process.

[0059] AURA's core innovation lies in its outer loop triggering and correction mechanism. Instead of relying on fixed heuristic thresholds, it dynamically models "healthy" learning states through online kernel density estimation, thereby statistically identifying anomalous learning events. Once an anomaly is identified, the system constructs an adaptive instruction containing quantified diagnostic data and key "abnormal gradient vectors," and invokes a large language model (LLM) as a structural evolution engine to perform structural reshaping such as reconstruction, supplementation, or pruning on the semantic foundation of the reward factor set.

[0060] This closed-loop feedback mechanism endows autonomous agents with the ability to introspect and self-correct their own reward functions, constituting a paradigm shift from static goal reduction to dynamic goal correction. This fundamentally enhances the agent's robustness, sample efficiency, and final performance in the face of sparse rewards, reduction loopholes, and environmental uncertainties. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a metacognitive correction and self-updating reward method for a large language model oriented towards autonomous intelligent agents, as proposed in this invention. Detailed Implementation

[0062] The technical solutions in the implementation of the present invention will be clearly and completely described below with reference to the accompanying drawings in the examples of the present invention.

[0063] Example: Combining Figure 1This invention provides a large language model metacognitive correction and self-updating reward method for autonomous intelligent agents. For ease of understanding, the implementation of this invention will be illustrated using a classic robot control task—driving a humanoid robot to stand up completely from the ground in a MuJoCo physics simulation environment—as an example. The complete process of this method begins by using a large language model to parse a high-level task description ("get the robot to stand up and maintain balance") into an executable initial reward architecture (initially including reward factors for dimensions such as torso height, posture balance, and energy consumption). Subsequently, the system enters a parallel inner and outer double-loop co-evolution mode: (1) the inner loop acts as a fast execution engine at the parameter level, driving the humanoid robot agent to learn through trial and error, and dynamically combining the above reward factors through a hierarchical attention reward synthesizer (HARS) to generate immediate rewards, co-optimizing with the agent's strategy; (2) the outer loop acts as a slow metacognitive monitoring and correction center at the structural level, continuously abstracting the training dynamics of the inner loop into multi-dimensional diagnostic vectors, and performing online anomaly detection based on a data-driven health status probability model. Once an unhealthy deviation in the learning process is identified, the outer loop activates the LLM and provides it with an adaptive instruction containing quantified diagnostic data and key anomalous gradient vectors to drive a structural reshaping of the reward architecture. The validated new architecture is seamlessly integrated back into the inner loop, forming a logically complete adaptive closed loop from initial architecture generation to the co-evolution of parameters and structure, until the global termination condition is met.

[0064] Step 1, Initialize the reward architecture and learning environment: For a high-level natural language task description given to the autonomous agent, an initial reward architecture consisting of multiple executable reward factors is generated using a large language model (LLM) through structured prompts (for the Humanoid standing task, an initial factor set including trunk height, control cost, survival reward, etc. is generated); at the same time, the policy network and hierarchical attention reward synthesizer (HARS) network of the reinforcement learning agent are initialized, and an experience replay pool for storing training data is established.

[0065] Step 2, Perform inner loop co-evolutionary training: Under the given current reward architecture, drive the agent to repeatedly interact with the environment through trial and error to learn the standing action, and based on the collected interaction trajectory, iteratively and co-promote the parameters of its motion policy network and the HARS network by minimizing the joint optimization objective.

[0066] Step 3: Perform metacognitive monitoring in the outer loop: In parallel with the inner loop, continuously extract and calculate multidimensional diagnostic vectors from the co-evolutionary training process. These vectors are used to quantify the health status of the current learning process from three dimensions: policy dynamics (whether the humanoid robot's policy has fallen into meaningless jitter), value-truth alignment (whether its internal value prediction is consistent with the final result of whether it can stand up), and reward architecture adaptability. Step 4: Perform learning state anomaly detection: Based on the historical data stream of the diagnostic vectors, a data-driven, adaptive triggering mechanism is used to identify and locate pathological regions in the learning process (e.g., when the diagnostic vector shows a continuously decreasing "value-truth correlation," indicating that the agent is "cheating" by maximizing a reward factor decoupled from the true goal). If this step does not occur, return to Step 2 to continue the current inner loop training; if it does occur, proceed to Step 5.

[0067] Step 5, Perform diagnostic-driven reward architecture reconstruction: Activate the semantic reward reconstruction module, which calls the large language model and provides it with adaptive instructions containing the current reward architecture and quantified diagnostic data (e.g., the instructions explicitly state that the low "value-authenticity correlation" is the triggering reason and ask the LLM to propose a correction scheme) to structurally modify the factor set of the current reward architecture (e.g., the LLM inference decides to add a new reward factor that penalizes excessive trunk tilt) to generate a new reward architecture.

[0068] Step 6, Integrate the evolved new reward architecture: Integrate the new reward architecture into the learning system using a systematic integration process that includes multi-stage verification and selective network parameter retention, to replace the original reward architecture.

[0069] Step 7: Determine if training should terminate: Determine if the preset maximum training period has been reached or if the convergence condition has been met; if not, return to step 2 and start the next round of inner loop training under the new reward framework; if the condition is met, end the training.

[0070] Further, in step 1, the reward architecture and learning environment are initialized: based on a high-level natural language task description, a large language model (LLM) is used to generate an initial reward architecture consisting of multiple executable reward factors through structured prompts, specifically including:

[0071] Initializing the learning environment: The learning environment is instantiated as a robot simulation environment based on the MuJoCo physics engine, and adopts the industry-recognized "Humanoid Standing Up" task as the benchmark. This environment provides the autonomous agent (i.e., the humanoid robot) with a high-dimensional state space containing information such as its joint angles, velocities, and trunk posture, as well as an action space corresponding to the torques of each joint, thus providing the basis for interaction in the subsequent reinforcement learning process.

[0072] Structured Chain-of-Thought Prompting and Reward Factor Generation: To guide the large language model in high-quality task decomposition and reward factor generation, this invention designs a "meta-prompt" template. This template forces the large language model to perform structured chain-of-thought reasoning by setting a series of logical stages, such as [System Role Setting] -> [Task Goal Decomposition] -> [Reward Factor Identification] -> [Code Generation and Verification]. In the Humanoid task, this template guides the LLM to first decompose the macro-goal of "standing and maintaining balance" into sub-goals such as "lifting the torso," "extending the legs," and "minimizing swaying." Then, through guiding questions (e.g., "How should we quantify the sub-goal of 'lifting the torso'?"), these sub-goals are transformed into specific, executable reward factor codes, thereby generating an initial reward factor set.

[0073] Factor Differentiability and Annealing Mechanism: To ensure that the reward factor code generated by a large language model can support gradient-based optimization, this invention includes an automated code rewriter. This rewriter is responsible for identifying and replacing non-discontinuous or non-differentiable operations in the code, such as pointer functions. (It is 1 when condition c is true, and 0 otherwise) Replace it with its differentiable approximation, namely the Sigmoid function: Here, g(s,a) is a function that transforms condition c into a scalar, while β is a hyperparameter that controls the slope. To balance exploration and exploitation, the value of β is gradually increased using an annealing strategy as training progresses, from a small value (smooth function curve, conducive to gradient propagation and exploration) to a large value (function curve close to a step function, conducive to accurate convergence in the later stages of learning).

[0074] Furthermore, in step 2, the inner loop co-evolutionary training is performed. Its core lies in employing the Hierarchical Attention Reward Synthesizer (HARS) network. This network dynamically synthesizes reward signals through a mechanism that decouples evaluation in factor space and time series but deeply shares parameters. Specifically, it includes:

[0075] Hierarchical context-aware value synthesis in factor space: The HARS network introduces the concept of hierarchical context, meaning that the importance of the reward factor depends not only on the local state s at the current time step t. t It should also be subject to the global context vector c, which represents the global strategic intent of the entire interaction trajectory τ. τ Modulation. In the Humanoid task, this mechanism manifests as follows: when the agent is in the early stages of the trajectory (lying on the ground), the global context c representing "attempting to stand up" is modulated. τ The HARS attention mechanism is guided to allocate the highest weight to "torso height reward"; when the agent approaches upright, the context changes to "attempting balance," and the attention weights dynamically shift to factors such as "control cost" or "postural stability." This multi-head attention mechanism's insensitivity to the number and order of input factor vectors is a key technical guarantee ensuring that the HARS network can seamlessly adapt to the dynamic structural changes in the reward architecture caused by the outer loop.

[0076] Parameter-sharing temporal credit attribution is performed in the time series: To accurately allocate sparse true rewards provided by the environment (e.g., a large positive reward only given after a successful Humanoid stand) to key time steps that contribute to the outcome, the HARS network employs an efficient parameter-sharing mechanism. Specifically, the context encoder is directly reused in generating the global context vector c. τ The normalized internal attention weight distribution α(ψ) generated at that time is used as the temporal credit coefficient. In the Humanoid task, this means that in a successful standing trajectory, the key moments of "pushing off with both legs" and "fine-tuning after standing" will receive higher credit weights due to their significance in the attention calculation, thus letting the policy network understand that these actions are key to the final success.

[0077] Parameter Co-update Based on Joint Optimization Objective: The updates of the policy network parameters θ and the HARS network parameters φ and ψ are minimized by using gradient descent to minimize the joint optimization objective. Driven by this, the joint optimization objective consists of two linearly weighted parts:

[0078]

[0079] (1) Strategy learning loss This term represents the policy loss function of any standard reinforcement learning algorithm (e.g., the Actor loss of PPO or SAC). Its advantage function, or Q-value, is calculated entirely based on our dynamically generated reward signal r. φ,ψ It directly guides the Humanoid to learn how to control its joints to complete movements.

[0080] (2) Reward Alignment Loss It penalizes the deviation between the cumulative dense reward predicted by the HARS network and the sparse true reward R(τ) provided by the environment through a mean squared error loss function, the mathematical form of which is: This loss term is a "reality checker": if HARS rewards a behavior that causes the Humanoid to roll around in place at high energy consumption, but this behavior does not lead to eventually standing up (i.e., the real reward R(τ) is zero), the loss term will increase, forcing HARS to adjust its reward composition to align with real-world goals, thereby suppressing reward hacking.

[0081] Furthermore, in step 3, the core of performing metacognitive monitoring in the outer loop lies in conducting multi-dimensional and systematic quantitative diagnosis of the learning process to generate a D-dimensional diagnostic vector that can comprehensively characterize the current health status of the learning process. This vector is like a "health check report" of the learning process of the Humanoid agent. The components of this vector are extracted and formally defined from the following three mutually orthogonal dimensions, specifically including: (1) Policy Dynamics Dimension: which aims to quantify the policy network π θ The evolutionary behavior of the policy is used to identify convergence and stability issues. Indicators for this dimension include the policy entropy gradient (d). PEG ), which is used to quantify policy exploration (in terms of policy entropy). The rate of change of policy entropy over time is used to diagnose whether exploratory behavior has been prematurely dissipated. In the Humanoid task, if this value remains negative, it may indicate that the agent has stopped trying new ways of standing, possibly falling into some repetitive local optimum behavior (e.g., only raising its head without raising its entire torso), meaning exploratory behavior has been prematurely dissipated. Its mathematical formalization is defined as the rate of change of policy entropy over time within an evaluation period ΔT:

[0082]

[0083] This dimension also includes time-series policy divergence (d TPDThe KL divergence (or KL divergence) is used to quantify the update magnitude and stability of a policy in the parameter space. It is measured by calculating the KL divergence between consecutive policy snapshots to diagnose whether the learning process is in a high-energy oscillation state. If this value is too high, it indicates that the Humanoid learning process is in a high-energy oscillation state, with its behavioral policy changing drastically between different update cycles (e.g., suddenly changing from attempting a forward roll to an upright position to attempting a backward roll), lacking a stable learning direction. Its mathematical formalization is defined as:

[0084]

[0085] (2) Value-Veracity Alignment Dimension: As a core component in diagnosing the "reward hacking" phenomenon, this dimension aims to measure the consistency between learned values ​​and objective reality. Indicators for this dimension include value-veracity relevance (d...). VVC Its direct quantification is the value function V guided by the learned dense rewards. φ The linear correlation between (s) and the real sparse reward R(τ) of the environment. This is a key indicator for diagnosing whether the Humanoid is "cheating". For example, an agent might discover that "standing on its head" maximizes the dense reward of "torso height", thus maximizing its internal value function V. φ (s) will give high praise to the "inverted" behavior. However, "inverted" is not a successful standing, and its real reward R(τ) is zero. At this time, d VVC This will significantly reduce, revealing a disconnect between intrinsic value and the true objective. It is measured by the Pearson correlation coefficient ρ between the predicted value and the actual Monte Carlo return G(s), which is mathematically formalized as:

[0086]

[0087] in, It is the expected return of the real reward starting from state s.

[0088] (3) Reward Architecture Adaptability Dimension: This dimension directly examines the internal dynamics of the HARS modules to diagnose their ability to continuously learn and adapt. This dimension includes the following three metrics:

[0089] Factor attention plasticity (d) FAP ), which is used to measure the factor space attention distribution ω tWhether (φ) has become rigid is quantified by calculating the Jensen-Shannon divergence of the attention distribution between consecutive evaluation cycles. If this value is too low, it indicates that the HARS network has become "rigid." For example, in the Humanoid task, it may only focus on "torso height" and completely ignore "control costs," causing the robot to stand in a highly uneconomical and exaggerated manner. Its mathematical formalization is defined as:

[0090]

[0091] Temporal attention plasticity (d TAP ), which is used to measure the time-series credit allocation mechanism α t Whether (ψ) is stuck in a fixed state is quantified by calculating the Jensen-Shannon divergence of the temporal attention distribution between consecutive evaluation cycles. Its mathematical formalization is defined as:

[0092]

[0093] Residual prediction error (d) RPE ), which is used to diagnose the current factor set. Is this sufficient to explain the true returns, aligning losses by quantifying rewards? The lower bound of convergence is defined within an evaluation window. If this value remains high later in training, it indicates that the current reward factors (e.g., only height and energy consumption) are insufficient to fully describe the essence of "successful standing," potentially lacking characterization of key concepts such as "body balance" or "posture uprightness." Its mathematical formal definition is:

[0094]

[0095] Furthermore, in step 4, abnormal learning state detection is performed. Its core lies in employing a data-driven, adaptive triggering mechanism to identify and locate pathological regions during the learning process. Specifically, this includes:

[0096] This mechanism first models the probabilistic health learning state through online kernel density estimation. It then uses the historically generated diagnostic vector sequence {d} (1) ,d (2) ,...,d (k) The data is treated as a high-dimensional data stream, and an online kernel density estimation method is used to dynamically construct and continuously update a probability density function p for the "healthy" learning state. healthy(d) The update mechanism of this health status model is related to the external performance of the agent. That is, only when the average real reward obtained by the agent in the most recent evaluation period is higher than a dynamically updated historical performance baseline, is its corresponding diagnostic vector determined as a "healthy sample" and added to a prototype buffer for density estimation. In the Humanomed task, only the diagnostic vectors *d* corresponding to training rounds in which the user successfully stands are considered "healthy samples" and used to construct the health state model. This is equivalent to teaching the system "what the physiological indicators of an agent learning to stand healthily should be like." The probability density function p of the health state is... healthy The log-likelihood form of (d) is given by the following equation:

[0097]

[0098] Subsequently, the mechanism performs anomaly score calculation based on negative log-likelihood. In each outer loop cycle k, the current diagnostic vector d is calculated based on the aforementioned health state probability model. (k) abnormal score S(d) (k) This score is defined as the negative log-likelihood of the current diagnostic vector under the health state model, which directly quantifies the degree of deviation of the current learning state from all known health states. When the Humanoid begins to employ a "head-up" cheating strategy, its corresponding diagnostic vector (e.g., d) VVC (Very low) in the health state model p healthy The probability of occurrence under (d) is extremely low, resulting in an abnormal score S(d). (k) This is extremely high, like an indicator in a medical examination report that deviates significantly from the normal range. Its mathematical formal definition is:

[0099] S(d (k) ) = -logp healthy (d (k) )

[0100] Finally, the mechanism employs adaptive triggering logic based on dynamic statistical quantiles. To avoid using fixed trigger thresholds that require manual adjustment, this invention uses a dynamic thresholding method based on historical data distribution. The trigger threshold τ is... trigger Defined as the score of all historically observed anomalies {S(d)} (1) ),...,S(d (k) A certain high quantile q of}, this threshold is dynamically adjusted as the number of observed data increases. The outlier score S(d) of the current period is [value missing]. (k) For N consecutive cycles (where N≥1 is a preset integer used to increase robustness), the value remains above the dynamic trigger threshold τ. triggerOnly then is it confirmed that the learning process has entered the pathological region requiring intervention, and step 5 is executed. The mathematical formalization of the dynamic trigger threshold is defined as follows:

[0101] τ trigger =Percentile({S(d (1) ),...,S(d (k) )},q)

[0102] Further, in step 5, a diagnostic-driven reward architecture reconstruction is performed: the semantic reward reconstruction module is activated. This module calls a large language model and provides it with adaptive instructions containing the current reward architecture and quantified diagnostic data to structurally modify the factor set of the current reward architecture and generate a new reward architecture, specifically including:

[0103] The process begins with the dynamic construction of diagnostic-driven adaptive instructions. When the outer loop is triggered, the semantic reward reconstruction module dynamically constructs a structured adaptive instruction provided to the LLM. This instruction is not a set of hard-coded rules or templates, but a dynamically aggregated data structure designed to provide sufficient, evidence-based context for the LLM's reasoning, and systematically includes the following four key information components:

[0104] (1) Task Axioms: Restating the fundamental, high-level task objectives defined in step 1. And key environmental constraints, serving as the logical foundation for all subsequent inferences in LLM.

[0105] (2) Current Reward Architecture Specification: A detailed description of the currently used and soon-to-be-modified set of reward factors. This includes the mathematical expression for each factor, the parameters used, and the expected semantic function interpretation.

[0106] (3) Quantitative, Multi-Level Diagnostic Data: This directly presents the pure numerical diagnostic information generated in steps 3 and 4 without natural language "translation," including the anomaly score S(d) that triggers the alarm. (k) ), the complete diagnostic vector d that leads to the abnormality (k) And the abnormal gradient vector that indicates the direction of correction. This gradient vector geometrically indicates the steepest ascent direction from the current ill-conditioned point back to the high-density region of the healthy state, providing a quantitative and actionable directional guide for LLM correction.

[0107] (4) Structured Adaptive Tasking: An open-ended but authorizing instruction that requires the LLM to follow a specific evolutionary logic. This instruction requires the LLM to act as a systems engineer and respond sequentially: First, diagnose structural defects based on diagnostic data; second, propose an adaptive solution to correct the defects; and finally, generate a completely new and complete structural specification.

[0108] Subsequently, LLM-mediated structural reshaping and interpretable inference chain generation are performed. Upon receiving the adaptive instruction, the large language model is instructed to initiate a systematic response process that generates its complete inference chain. This process maps the input quantified diagnostic data to a semantic judgment of the current reward architecture and performs structural modifications based on this judgment. The structural modification operations that LLM can perform are diverse and flexible, enabling deep reshaping of the reward architecture. These operations include parameter tuning, factor reconstruction, factor addition, and factor pruning. Finally, the LLM outputs a completely new and complete reward architecture specification. This specification is a concrete implementation of its adaptive policy and structural modification operations. For example, the inference chain of LLM might be: "Diagnostic data shows that the agent is taking advantage of height rewards by standing upside down. This is because the current architecture only rewards height and does not constrain posture. Therefore, I need to add a new reward factor to penalize abnormal postures." Subsequently, LLM performs the factor addition operation, outputting a completely new reward architecture specification. In addition to the original factors, a new "posture penalty factor" is added to this specification: orientation_penalty = -|torso_up_vector.z-1.0|. This factor penalizes users when their torso is not perpendicular to the ground. Further, in step 6, the evolved new reward architecture is integrated: the new reward architecture is integrated into the learning system using a systematic integration process that includes multi-stage verification and selective network parameter retention to replace the original reward architecture. Specifically, this includes:

[0109] The process begins with multi-stage automated verification of candidate architectures. The new reward architecture specification is generated in step 5 and output by the large language model. Before being formally adopted, a rigorous, automated candidate architecture validation phase must first be completed to filter out invalid, insecure, or contradictory structural mutations. This validation phase consists of three consecutive validation steps:

[0110] (1) Structural Integrity and Syntactic Validation: Automatically parse the new specification to ensure that it is well-formed in terms of syntax and data structure, and that all specified mathematical expressions, function calls and parameter definitions are valid and unambiguous.

[0111] (2) Functional feasibility and numerical stability test: Instantiate each independent reward factor in the new specification and call it with boundary values ​​and random values ​​to confirm that it is computationally feasible and will not produce undefined behavior or numerical errors.

[0112] (3) Preliminary performance and objective relevance analysis: Using a static dataset containing historical interaction trajectories, the cumulative reward sequence generated by the new candidate architecture is calculated offline, and the correlation between this sequence and the real task reward R(τ) recorded in the dataset is evaluated to filter out candidate schemes that are clearly inconsistent with the fundamental task objective. If an anomaly occurs, the anomaly case is returned and the reward architecture update is resumed.

[0113] After verification, the process concludes with a selective parameter retention strategy based on knowledge transfer. Only candidate architectures that pass all verification steps will be formally adopted as the new set of reward factors. And trigger the parameter integration phase of the system. In this phase, in order to maximize the transfer of existing knowledge while adapting to the new value landscape, a selective parameter retention strategy is adopted, which differentiates the parameters of different networks: (1) Reinitialize the parameters related to the factor space: due to the semantic input space of the reward architecture (i.e., the factor set) The factor space attention parameter φ in the HARS network, which directly processes reward factors, has been fundamentally changed and will be reinitialized.

[0114] (2) Preservation of parameters related to temporal structure and policy behavior: The temporal credit attribution parameter ψ, which is responsible for the intrinsic temporal dynamic structure of the learning task, and the policy network parameter θ, which encapsulates the basic motor skills already learned by the Humanoid (such as how to extend the leg and how to swing the arm), will be fully preserved. By preserving ψ and θ, the agent can inherit its learned prior knowledge of credit allocation based on "when" and behavioral priors on "how" to act, thereby enabling it to continue learning and adapting efficiently on new value landscapes, rather than starting from scratch.

[0115] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A metacognitive self-updating reward method for large language models of autonomous intelligent agents, characterized in that, The method includes the following steps: Step 1: Initialize the reward architecture and learning environment: For a high-level natural language task description that empowers an autonomous agent, use a large language model (LLM) to generate an initial reward architecture consisting of multiple executable reward factors through structured prompts; at the same time, initialize the policy network and the hierarchical attention reward synthesizer (HARS) network of the reinforcement learning agent, and establish an experience replay pool for storing training data. Step 2: Perform co-evolutionary training within the inner loop: Under the given current reward architecture, drive the agent to interact with the environment, and based on the collected interaction trajectories, iteratively and co-operatively update the parameters of the policy network and HARS network by minimizing the joint optimization objective; Step 3: Perform metacognitive monitoring of the outer loop: In parallel with the inner loop, continuously extract and compute multidimensional diagnostic vectors from the co-evolutionary training process. These vectors are used to quantify the health status of the current learning process from three dimensions: policy dynamics, value-truth alignment, and reward architecture adaptability. Step 4: Perform learning state anomaly detection: Based on the historical data stream of the diagnostic vector, a data-driven, adaptive triggering mechanism is used to identify and locate pathological regions in the learning process; if not entered, return to step 2 to continue the current inner loop training; if entered, proceed to step 5. Step 5: Perform diagnostic-driven reward architecture reconstruction: Activate the semantic reward reconstruction module, which calls a large language model and provides it with adaptive instructions containing the current reward architecture and quantified diagnostic data to structurally modify the factor set of the current reward architecture and generate a new reward architecture. Step 6: Integrate the evolved new reward architecture: Integrate the new reward architecture into the learning system using a systematic integration process that includes multi-stage verification and selective network parameter retention, in order to replace the original reward architecture; Step 7: Determine if training should terminate: Determine if the preset maximum training period has been reached or if the convergence condition has been met; if not, return to step 2 and start the next round of inner loop training under the new reward framework; if the condition is met, end the training.

2. The metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents according to claim 1, characterized in that, In step 1, for a high-level natural language task description that endows an autonomous intelligent agent, a large language model (LLM) is used to generate an initial reward architecture consisting of multiple executable reward factors through structured prompts, specifically including: (1) Structured Chain-of-Thought Prompting and Reward Factor Generation: A "meta-prompt" template is constructed and adopted. This process specifically includes the following sequential steps: 1) System Preamble and Context Injection: First, an expert role with deep domain knowledge is set for the large language model, including a "senior robot reward engineer" or a "game theory strategy analyst," and key environmental information is injected into it, including formal descriptions of the state space S and action space A, physical constraints of the task, and the final success criteria; 2) Hierarchical Task Decomposition: Next, the model is guided to decompose the macroscopic, possibly vaguely defined high-level task objective into a set of logically orthogonal or dependent subtasks, as well as intermediate constraints that must be satisfied; 3) Semantic Factor Identification (and Formalization): Based on the aforementioned hierarchical decomposition results, the model is required to identify the positive utilities that must be pursued to achieve each sub-task and milestone, the negative costs that must be avoided, and the key system states that need to be maintained; the instruction model formally expresses these abstract semantic concepts as mathematical functions or logical expressions with state s and action a as input; 4) Code Synthesis, Annotation, and Self-Consistency Verification: Finally, the instruction model synthesizes these formalized function concepts into reward factor codes that can be executed by a computer and have detailed natural language annotations; at the same time, the model is required to perform a preliminary logical self-consistency check on the generated code, including verifying whether there are contradictory reward signals or domain errors; (2) Factor Differentiability and Smoothing Treatment: This includes an automated post-processing module activated after code generation. This module aims to ensure the end-to-end differentiability of the entire reward computation graph. Its specific functions are: 1) Scanning and Identification: Using static code analysis technology, it automatically scans all generated reward factor codes to accurately identify discontinuous or non-differentiable operations that may block or distort gradient flow during backpropagation, including if / else conditional branches, sign() sign function, round() rounding function, or discrete selection operations such as argmax; 2) Differentiable Proxy Substitution: It systematically replaces the identified non-differentiable operations with their corresponding mathematically approximate and well-properted differentiable surrogate functions, specifically, an indicator function: I(g(s,a)>0) It will be replaced with a Sigmoid function with a controllable temperature parameter β, in the form: σ(β·g(s,a))=(1+exp(-β·g(s,a))) -1 3) Adaptive Annealing Mechanism: The temperature parameter β is configured to follow a preset, non-linear annealing scheduling strategy during subsequent training; this strategy causes the value of β to gradually increase from a small initial value to a larger value.

3. The metacognitive correction and self-updating reward method for a large language model oriented towards autonomous intelligent agents according to claim 1, characterized in that, Step 2 involves performing inner-loop co-evolutionary training: Under the given current reward architecture, the agent is driven to interact with the environment, and based on the collected interaction trajectories, the parameters of the policy network and the HARS network are iteratively and collaboratively updated by minimizing the joint optimization objective. Specifically, this includes: (1) Value Synthesis in FactorSpace with Hierarchical Context-Awareness: The HARS network introduces the concept of hierarchical context at each time step t, that is, the importance of the reward factor depends not only on the local state s of the current time step. t It should also be subject to the global context vector c, which represents the global strategic intent of the entire interaction trajectory τ. τ Modulation; to this end, a context encoder based on a self-attention mechanism is first used to modulate a complete state sequence (s0,...,s...). T-1 The encoding is the global context vector c. τ Subsequently, at each time step t, a multi-head attention mechanism is employed to manage the local state s. t and global context vector c τ Together as a query, it refers to the factor vector f(s) instantiated at time step t, provided by the current reward architecture. t ,a t Dynamic weighting is performed to generate a high-dimensional instantaneous reward vector r′ rich in contextual information. φ (s t ,a t The multi-head attention mechanism's insensitivity to the number and order of input factor vectors ensures that the HARS network can seamlessly adapt to dynamic structural changes in the reward architecture caused by the outer loop. (2) Temporal Credit Attribution via Parameter Sharing in Time Series: To accurately allocate the sparse real rewards provided by the environment to the key time steps that contribute to the results, the HARS network employs a parameter sharing mechanism to avoid introducing a separate, easily overfitted temporal attention network; specifically, the context encoder is directly reused in generating the global context vector c. τ The normalized internal attention weight distribution generated at that time is α(ψ)={α0,...,α T-1 The weight distribution α(ψ) is directly used as the temporal credit coefficient to measure the contribution of each time step to the final task result. Finally, the temporal credit coefficient is multiplied element-wise with the instantaneous reward vector (after being mapped back to a scalar by a multilayer perceptron) and scaled using the trajectory length T to synthesize the final scalar reward signal r that simultaneously contains information on "where it is important" (factor space attention) and "when it is important" (temporal series attention). φ,ψ (s t ,a t ); (3) Cooperative Parameter Update based on a Joint Optimization Objective: The update of policy network parameters θ and HARS network parameters φ and ψ is achieved by minimizing the joint optimization objective through gradient descent. Driven by this, the joint optimization objective consists of two linearly weighted parts: 1) Strategy learning loss This term represents the policy loss function of any standard reinforcement learning algorithm; its advantage function or Q-value calculation depends entirely on the dynamically generated reward signal r. φ,ψ This makes the AURA framework algorithmically independent; 2) Reward alignment loss It penalizes the deviation between the cumulative dense reward predicted by the HARS network and the sparse true reward R(τ) provided by the environment through a mean squared error loss function, which is of the form: This ensures that the learning of the reward architecture always aligns with the objective goals of the real world.

4. The metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents according to claim 1, characterized in that, Step 3, in parallel with the inner loop, continuously extracts and computes a multidimensional diagnostic vector from the co-evolutionary training process. This vector is used to quantify the health status of the current learning process from three dimensions: policy dynamics, value-truth alignment, and reward architecture adaptability. Specifically, it includes: (1) Policy Dynamics Dimension: This aims to quantify the policy network π θ The evolutionary behavior is used to identify convergence and stability problems, and its indicators include: policy entropy gradient (d PEG ): Quantitative strategy exploration (using strategy entropy) The rate of change of policy entropy over time, used to diagnose whether exploratory power is prematurely dissipated, is mathematically formalized as the rate of change of policy entropy over an evaluation period ΔT: Temporal policy divergence (d) TPD ): The update magnitude and stability of the quantization strategy in the parameter space are measured by calculating the KL divergence between consecutive policy snapshots. This is used to diagnose whether the learning process is in a high-energy oscillation state. Its mathematical formalization is defined as: (2) Value-Veracity Alignment Dimension: As the core of diagnosing the "reward hacking" phenomenon, it aims to measure the consistency between learned values ​​and objective reality. Its indicators include: value-veracity correlation (d VVC ): Directly quantify the value function V guided by the learned dense rewards. φ The linear correlation between the predicted value and the actual sparse environmental return R(τ) is measured by the Pearson correlation coefficient ρ between the predicted value and the actual Monte Carlo return G(s), which is mathematically formalized as follows: in, It is the expected value of the real return starting from state s; (3) Reward Architecture Adaptability Dimension: Directly explores the internal dynamics of the HARS modules to diagnose their ability to continuously learn and adapt. Indicators include: Factor Attention Plasticity (d) FAP ): Measure of the spatial attention distribution of factors ω t Whether (φ) is stuck is quantified by calculating the Jensen-Shannon divergence (a symmetric and smooth KL divergence) of the attention distribution between consecutive evaluation cycles, which is mathematically formalized as: Temporal attention plasticity (d TAP ): Measuring the time-series credit allocation mechanism α t Whether (ψ) is stuck in a fixed state is quantified by calculating the Jensen-Shannon divergence of the temporal attention distribution between consecutive evaluation cycles, and its mathematical formalization is defined as: Residual prediction error (d) RPE Diagnostic current factor set Is this sufficient to explain the true returns, with its direct quantification of rewards aligned with losses? The lower bound of convergence within an evaluation window is mathematically formally defined as:

5. The metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents according to claim 1, characterized in that, Step 4, learning state anomaly detection, is based on the historical data stream of diagnostic vectors and employs a data-driven, adaptive triggering mechanism to identify and locate pathological regions during the learning process. Specifically, it includes: (1) Probabilistic Modeling of Healthy Learning States via Online Kernel Density Estimation: Constructing a diagnostic vector sequence {d} from diagnostic indicators. (1) ,d (2) ,...,d (k) This is treated as a high-dimensional data stream, and an online kernel density estimation method is used to dynamically construct and continuously update a probability density function p for the "healthy" learning state. healthy (d); The update mechanism of this health status model is related to the external performance of the agent. That is, the corresponding diagnostic vector d is updated only when the average real reward R(τ) obtained by the agent in the most recent evaluation period is higher than a dynamically updated historical performance baseline. (k) Only then were they classified as "healthy samples" and added to a prototype buffer for density estimation. This ensures that the probability density function can continuously and dynamically track the evolving and truly effective health learning process; the probability density function p of the health state healthy The log-likelihood form of (d) is given by the following equation: Among them, K h Let h be a Gaussian kernel function with bandwidth parameter h. For prototype buffer; (2) Anomaly Score Calculation based on Negative Log-Likelihood: In each outer loop cycle k, the current diagnostic vector d is calculated based on the health status probability model. (k) abnormal score S(d) (k) This score is defined as the negative logarithmic probability of the current diagnostic vector under the health state model. It directly quantifies the degree of deviation of the current learning state from all known health states, and its mathematical formalization is: S(d (k) )=-logp healthy (d (k) ) A high anomaly score indicates that the current diagnostic vector d (k) Located in a low-density region of the known health status distribution, indicating a statistically significant anomaly in the learning process; (3) Adaptive Triggering Logic based on Dynamic Statistical Quantile: To avoid using fixed trigger thresholds that require manual adjustment, a dynamic thresholding method based on historical data distribution is adopted; the trigger threshold τ is... trigger Defined as the set of all historically observed outlier scores {S(d (1) ),...,S(d (k) A certain high quantile q of )}, this threshold is dynamically adjusted as the amount of observed data increases, and its mathematical formal definition is: τ trigger =Percentile({S(d (1) ),...,S(d (k) )},q) If and only if the abnormal score S(d) of the current period is... (k) For N consecutive cycles (where N≥1 is a preset integer used to increase robustness), the value remains above the dynamic trigger threshold τ. trigger Only then is it finally confirmed that the learning process has entered a pathological area requiring intervention, and step 5 is executed.

6. The metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents according to claim 1, characterized in that, In step 5, the semantic reward reconstruction module is activated. This module calls a large language model and provides it with adaptive instructions containing the current reward architecture and quantitative diagnostic data to structurally modify the factor set of the current reward architecture and generate a new reward architecture, specifically including: (1) Dynamic Construction and Informational Completeness of the Diagnosis-Driven Adaptive Directive: When the outer loop is triggered, the semantic reward reconstruction module first dynamically constructs a structured adaptive directive for the Large Language Model (LLM). This directive is not a set of hard-coded rules or templates, but a dynamically aggregated data structure designed to provide sufficient evidence-based context for LLM reasoning and systematically includes the following four key information components: 1) Task Axioms: Restating the fundamental high-level task objectives defined in step 1. 1) Key environmental constraints; 2) Current Reward Architecture Specification: A detailed description of the currently used and soon-to-be-modified set of reward factors. This includes the mathematical expression for each factor, the parameters used, and their expected semantic function interpretation; 3) Quantitative, Multi-Level Diagnostic Data: Directly presenting the generated, untranslated, purely numerical diagnostic information, including: the anomaly score S(d) that triggers the alarm. (k) ) and its quantiles in the historical distribution; the complete diagnostic vector d leading to the anomaly (k) It contains specific values ​​across all dimensions; the anomaly gradient vector indicates the direction of correction. 4) Structured Adaptive Tasking: An open-ended but authorizing instruction that requires the LLM to follow a specific evolutionary logic. This instruction requires the LLM to act as a systems engineer and respond in the following sequential steps: First, diagnose structural defects based on diagnostic data; second, propose an adaptive solution to correct the defect; and finally, generate a completely new and complete structural specification. (2) LLM-Mediated Structural Refactoring with Interpretable Chain-of-Thought Generation: After receiving the adaptive instruction, the large language model is required to initiate a systematic response process to generate its complete inference chain. This process maps the input quantitative diagnostic data to a semantic judgment of the current reward architecture and performs structural modifications based on this judgment. The structural modification operations that LLM can perform are diverse and flexible to achieve deep refactoring of the reward architecture. The operation types include the following four categories: 1) Parameter Tuning: Fine-tuning the constants or hyperparameters in the existing factor mathematical expressions; 2) Factor Reconstruction Refactoring: While preserving the core semantics of factors, rewrite their mathematical expressions to change their behavior, fix potential vulnerabilities, or improve their numerical stability; 3) Factor Augmentation: Based on the understanding of diagnosed information bottlenecks, design and add one or more new reward factors with new semantics to compensate for the deficiencies in the expressive power of the current factor set or introduce new task constraints; 4) Factor Pruning: Based on the analysis of diagnosed redundancy or negative impacts, identify and remove factors that have been proven to be ineffective, highly correlated with other factors, or have a continuous negative impact on the learning process; Finally, the LLM outputs a new and complete reward architecture specification. This specification is a concrete implementation of its adaptive strategy and structural modification operations.

7. The metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents according to claim 1, characterized in that, In step 6, the new reward architecture is integrated into the learning system. This is achieved through a systematic integration process that includes multi-stage validation and selective network parameter retention, replacing the original reward architecture. Specifically, this includes: (1) Multi-Stage Automated Validation of the Candidate Architecture: The new reward architecture specification generated by the large language model in claim 6. Before being formally adopted, a rigorous and automated candidate architecture validation phase must first be conducted to filter out invalid, unsafe, or contradictory structural mutations. This validation phase includes the following three consecutive validation steps: 1) Structural Integrity and Syntactic Validation: The new specification is automatically parsed to ensure that it is well-formed in terms of syntax and data structure, and that all specified mathematical expressions, function calls, and parameter definitions are valid and unambiguous; 2) Functional Realizability and Numerical Stability Test: Each independent reward factor in the new specification is instantiated and called in a simulation environment using boundary values ​​and random values ​​sampled from the state-action space to confirm that it is computationally realizable and will not produce undefined behavior, computational errors, or numerical overflow issues; 3) Preliminary Efficacy and Goal-Relevance Analysis. Analysis: Using a static dataset containing historical interaction trajectories, the cumulative reward sequence generated by the new candidate architecture is computed offline, and the correlation or mutual information between the sequence and the real task reward R(τ) recorded in the dataset is evaluated. This analysis provides a fast, low-cost preliminary signal about the potential value of the new architecture, which can be used to filter out bad candidate schemes that are obviously inconsistent with the fundamental task objectives or generate negatively correlated guiding signals. If anomalies are found, the reward architecture is updated again. (2) Selective Parameter Retention Strategy based on Maximizing Knowledge Transfer: Only candidate architectures that pass all verification steps will be formally adopted as the new set of reward factors. This triggers the system's parameter integration phase. During this phase, to adapt to the new value landscape while leveraging the transfer of existing knowledge to achieve learning continuity, a selective parameter retention strategy is adopted. This strategy differentiates the parameters of different networks: 1) Reinitializing factor space-related parameters: Due to the semantic input space of the reward architecture (i.e., the factor set) 1) The factor space attention parameter φ in the HARS network, which directly processes reward factors, has been fundamentally changed and will be reinitialized; 2) Parameters related to temporal structure and policy behavior are retained: the temporal credit attribution parameter ψ, which is responsible for learning the intrinsic temporal dynamic structure of the task, and the policy network parameter θ, which encapsulates the basic behavioral skills learned by the agent, will be completely retained from the previous outer loop cycle; This design decision is based on a core assumption: the intrinsic temporal structure of the task and the basic behavioral skills learned by the agent remain highly correlated after the reward factors are reconstructed. By retaining ψ and θ, the agent can inherit its learned credit allocation ability that is important "when" and behavioral priors about "how" to act, so as to continue to learn and adapt efficiently on the new value landscape, rather than starting from scratch.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement a metacognitive correction and self-updating reward method for a large language model for autonomous intelligent agents as described in any one of claims 1-7.

Images