Training method of reinforcement learning intelligent agent and electronic device

Abstract

The present disclosure provides a training method of a reinforcement learning agent and an electronic device. In each training time step, the current state of the reinforcement learning environment is input into a large language model guide and a policy network respectively to obtain an action proposal distribution and a policy distribution, the fusion policy is obtained by weighting and fusing the two distributions according to a guide weight, and an action is performed. In response to the passage of a first predetermined time window, the real-time performance indicator is determined based on the reward sequence within the window, and the guide weight is updated. In response to the passage of a second predetermined time window, the policy network parameters are updated based on the experience data generated by the fusion policy using a reinforcement learning algorithm. The above steps are iterated until the training termination condition is met. Through the dual-path decision and performance feedback type dynamic weight adjustment, the present disclosure realizes the smooth transition from external knowledge guidance to autonomous reinforcement learning, and improves the training efficiency, sample utilization rate and policy generalization ability.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the integration technology of large language models and reinforcement learning, and particularly to a training method and corresponding electronic device for a reinforcement learning agent that dynamically adjusts the guiding weights of a large language model by introducing a performance feedback mechanism during reinforcement learning training. Background Technology

[0002] In recent years, artificial intelligence has made significant progress in the field of complex decision-making, especially reinforcement learning (RL) technology. RL, which can autonomously learn optimal strategies through trial and error in interaction with the environment, has been widely applied in scenarios such as robot control, game agents, autonomous driving, and resource scheduling. However, traditional reinforcement learning methods still face many challenges when dealing with high-dimensional state spaces, sparse rewards, long-term planning requirements, and dynamically changing environments.

[0003] First, the sample efficiency is low. Reinforcement learning agents typically require massive amounts of interaction data to learn effective strategies, especially in the early stages of exploration. The agent's behavior is highly random, making it difficult to quickly accumulate useful experience, resulting in long training cycles and high computational costs.

[0004] Second, the challenge of balancing exploration and exploitation. In complex tasks, agents need to strike a balance between exploring unknown states and utilizing known strategies. Traditional exploration strategies (such as ε-greedy approaches and noise injection) lack an understanding of the task structure and are prone to getting trapped in local optima or repeating ineffective explorations.

[0005] Third, credit allocation is difficult under sparse rewards. The reward signals for many practical tasks are sparse and delayed, making it difficult for agents to attribute final success or failure to specific actions taken in the early stages, resulting in slow policy updates and difficulties in convergence.

[0006] Fourth, they lack prior knowledge and common-sense reasoning abilities. Reinforcement learning agents typically learn from scratch and cannot effectively utilize existing human knowledge, task structure information, or cross-task experience, resulting in a repetitive and inefficient learning process, especially performing poorly in the task initialization phase.

[0007] To address the aforementioned challenges, researchers have attempted to incorporate external knowledge sources into reinforcement learning frameworks. Among these, Large Language Models (LLMs), due to their superior capabilities in natural language understanding, logical reasoning, multi-step planning, and cross-domain knowledge acquisition, are increasingly recognized as powerful tools for enhancing reinforcement learning. Existing technologies primarily employ the following methods to combine LLMs with reinforcement learning.

[0008] Firstly, it can be used as a policy initialization or pre-training module. Before training begins, a large language model is used to generate an initial policy or value function to accelerate early learning. However, in such methods, the knowledge of the large language model is statically injected and cannot be dynamically adjusted based on the agent's performance during training, potentially leading to difficulties in correcting initial biases.

[0009] Secondly, it can serve as a fixed auxiliary decision-making module. During reinforcement learning training, the large language model acts as a parallel policy network, fusing with the reinforcement learning policy output with fixed weights to provide action suggestions or target cues. While this approach can improve exploration efficiency to some extent, it ignores the changing dependence of the agent on the large language model at different training stages. In the later stages of training, the agent may have developed strong autonomous decision-making capabilities, and continued reliance on the large language model may limit policy optimization and adaptive improvement.

[0010] Thirdly, it can be used as a reward function design or curriculum learning tool. Large language models are used to understand task objectives and automatically generate reward functions or design progressive training courses. These methods mostly focus on early-stage training setup and lack the ability to respond to real-time status and performance fluctuations during training.

[0011] It is evident that a significant bottleneck in research combining large language models with reinforcement learning lies in the lack of a dynamic coordination mechanism. Most methods employ static or predefined combinations, failing to adaptively adjust the participation and guidance of the large language model based on the agent's real-time learning state, changes in environmental complexity, and task stage characteristics. This leads to two extremes: first, the large language model excessively dominates the entire training process, inhibiting the autonomous exploration and adaptation capabilities of the reinforcement learning agent; second, the large language model's participation is insufficient or inappropriate, failing to provide effective guidance at crucial stages (such as the initial exploration phase or the strategy bottleneck period), thus impacting training efficiency.

[0012] Furthermore, existing methods typically lack a clear performance feedback trigger mechanism and fail to dynamically determine when to enhance or weaken the intervention of the large language model based on quantitative indicators such as the agent's learning progress, policy stability, and reward enhancement rate. This deficiency makes the combination of large language models and reinforcement learning lack a "teaching rhythm," making it difficult to achieve a smooth transition from "guided to independent" learning, which is not conducive to the agent ultimately forming robust, independent, and generalizable decision-making strategies.

[0013] Therefore, there is an urgent need for an interactive training method that can dynamically adjust the collaborative relationship between the large language model and reinforcement learning based on the real-time performance of the agent. This method would enable the large language model to provide strong support when the agent needs guidance and gradually fade away as the agent's capabilities improve, thereby achieving efficient, stable, and adaptive collaborative training and improving the overall system's learning efficiency, policy quality, and task generalization ability. Summary of the Invention

[0014] To address the issues of rigid collaborative mechanisms and insufficient dynamic adaptability in the combination of large language models and reinforcement learning in existing technologies, this disclosure provides a training method for reinforcement learning agents and corresponding electronic devices to achieve a smooth transition from external knowledge guidance to autonomous reinforcement learning.

[0015] This disclosure provides a training method for a reinforcement learning agent. The method trains a policy network within the reinforcement learning agent through interaction between the agent and a reinforcement learning environment. The training method includes: at each training time step, obtaining the current state of the reinforcement learning environment; inputting the current state into a large language model guide including a pre-trained large language model to obtain an action proposal distribution for the action space of the reinforcement learning environment output by the large language model guide; inputting the current state into the policy network to obtain a policy distribution output by the policy network; weighting and fusing the action proposal distribution and the policy distribution based on guide weights to obtain a fused policy; and then, based on the fused policy distribution... The policy execution action transitions the reinforcement learning environment from its current state to a subsequent state, and acquires the reward feedback from the reinforcement learning environment. In response to the expiration of a first predetermined time window, a reward sequence is determined based on the rewards received from the reinforcement learning environment within the first predetermined time window. A real-time performance index for the reinforcement learning agent is determined based on the reward sequence, and the guiding weights are updated based on the real-time performance index. In response to the expiration of a second predetermined time window, a reinforcement learning algorithm is used to update the parameters of the policy network based on the experience data generated from the interaction between the fusion policy and the reinforcement learning environment. In response to the determination that a predetermined training termination condition has not been met based on the training time step and / or the real-time performance index, the process proceeds to the next training time step.

[0016] In embodiments of this disclosure, the training method further includes: performing an initialization step before the first training time step, the initialization step including: initializing the policy network; initializing the pre-trained large language model of the large language model guide, enabling the pre-trained large language model to output the action suggestion distribution for the action space of the reinforcement learning environment; and setting the guide weights to initial values, wherein the initial values ​​are in the range of [0.8, 1.0].

[0017] In embodiments of this disclosure, the predetermined training termination condition includes at least one of the following: the training time step reaches a predetermined number of training steps; the real-time performance indicator reaches a predetermined performance target; the rate of change of the real-time performance indicator within a predetermined performance time window is less than a convergence threshold, and the current training time step is greater than a predetermined minimum number of training steps.

[0018] In embodiments of this disclosure, the large language model facilitator outputs the action suggestion distribution through the following steps: generating a state description based on the current state; constructing prompt information based on the state description, historical trajectory, task objective description, and constraints, wherein the historical trajectory includes historical states, historical actions, and historical rewards; inputting the prompt information into the large language model facilitator to obtain a confidence score for each action in the action space; and obtaining the action suggestion distribution based on the confidence score.

[0019] In embodiments of this disclosure, the weighted fusion includes action-level fusion, wherein the action-level fusion determines the fusion strategy according to the following formula:

[0020] in, The fusion strategy, For action, This represents the current state of the reinforcement learning environment. The proposed distribution for the actions is as follows: For the policy distribution, The guiding weight, .

[0021] In embodiments of this disclosure, the weighted fusion includes attention modulation fusion, wherein the attention modulation fusion determines the fusion strategy according to the following formula:

[0022] in, The fusion strategy, For action, This represents the current state of the reinforcement learning environment. The guiding weight, , This is the query vector for the large language model. To reinforce the query vector, For action The key vector, For vector dimensions.

[0023] In embodiments of this disclosure, the experience data includes experience tuples, which include the current state, the action performed, the reward, and the subsequent state; the experience data is stored in an experience replay pool or a trajectory buffer for the reinforcement learning algorithm to read in response to the passage of a second predetermined time window.

[0024] In the embodiments of this disclosure, the real-time performance metric is a performance health score used to characterize the interaction between the reinforcement learning agent and the reinforcement learning environment under the guidance of the fusion strategy. The performance health score is obtained by weighting multiple performance sub-metrics, including reward score, policy stability score, task progress score, and exploration score.

[0025] In embodiments of this disclosure, the reinforcement learning agent further includes a value network, which is used to output a value estimate based on the state of the reinforcement learning environment; updating the parameters of the policy network includes: using an actor-critic class reinforcement learning algorithm to update the parameters of the policy network and the parameters of the value network based on the same batch of experience data.

[0026] In the embodiments of this disclosure, the actor-critic reinforcement learning algorithm is a near-end policy optimization algorithm; the step of using the actor-critic reinforcement learning algorithm to update the parameters of the policy network and the value network based on the same batch of empirical data includes: obtaining a composite loss based on the policy loss term, the value loss term, and the policy entropy regularization term, and updating the parameters of the policy network and the value network based on the composite loss, wherein the composite loss is: in, For the policy network, These are the parameters of the policy network; The parameters of the value network are... For the strategy loss term, For the aforementioned value loss item, For policy entropy regularization, and This is the loss weighting coefficient.

[0027] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program; wherein, when the processor executes the computer program, it implements the training method for a reinforcement learning agent as described above.

[0028] This disclosure structurally comprises a dual-path decision engine (i.e., a large language model-guided path and a reinforcement learning-independent path) and a performance-driven closed-loop control layer (i.e., a performance monitoring and weight adjustment loop). The training method provided in this disclosure achieves a smooth transition from external knowledge guidance to autonomous reinforcement learning through dual-path decision-making and performance-feedback-based dynamic weight adjustment. This method not only significantly improves training efficiency and sample utilization but also enhances the generalization ability and robustness of the agent's policies, providing an efficient and interpretable solution for agent training in complex decision-making tasks. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a schematic flowchart of the training method for reinforcement learning agents provided in the embodiments of this disclosure; Figure 2 This is a schematic diagram of the performance feedback mechanism framework provided in the embodiments of this disclosure; and Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such order can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0034] Various embodiments of this disclosure will now be described in detail. Numerous specific details are set forth in the following description in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without some of these specific details.

[0035] Figure 1This is a flowchart illustrating the training method for a reinforcement learning agent provided in an embodiment of this disclosure. Figure 2 This is a schematic diagram of the performance feedback mechanism framework provided in the embodiments of this disclosure.

[0036] like Figure 1 and Figure 2 As shown in the embodiments of this disclosure, the training method for reinforcement learning agents is structurally composed of a dual-path decision engine and a performance-driven closed-loop control layer. The dual-path decision engine includes a large language model-guided path and a reinforcement learning-specific path, which process the current state from the reinforcement learning environment in parallel and are then fused and output by a policy fusion unit according to the guiding weights. The performance-driven closed-loop control layer consists of performance monitoring, a performance feedback mechanism, and an adaptive weight controller. The adaptive weight controller (also called a dynamic weight controller) updates the guiding weights based on real-time performance metrics, thereby scheduling the relative participation of the two paths in the dual-path decision engine in the final decision.

[0037] In this embodiment of the disclosure, the reinforcement learning agent typically includes two types of neural networks: for example, in an embodiment employing an actor-critic reinforcement learning algorithm, it includes a policy network and a value network. The policy network is used to answer the question "in state..." The problem of "what action should be chosen" has the state as input and the action distribution as output; the policy network can be denoted as... subscript The parameters of the policy network (including weights, biases, etc.), that is, the policy network in the state Next action The probability is The value network is used to answer the question, "How high is the expected long-term return of the current situation (or the combination of the current situation and the action)?" Its input is the state (or the combination of the state and the action), and its output is a scalar value estimate. The value network can be denoted as... (State Value Network) or (Action Value Network), where the subscript These are the parameters of the value network.

[0038] In this embodiment, the interaction between the reinforcement learning environment and the reinforcement learning agent follows the standard paradigm of a Markov Decision Process. Specifically, the reinforcement learning environment includes a state space. Action space Transition probability Reward function Elements such as these. At each training time step, the reinforcement learning agent observes the current state. And based on its own strategy, it selects and executes actions. The environment is determined by the state transition probability. Give the successor state And according to the reward function Give a reward Successor state This is typically used as the starting point of the current state for the next training time step. This interaction method, as well as the specific construction of the state space, action space, transition probabilities, and reward function involved, are all standard procedures in the field, and this disclosure does not limit the specific construction method of the reinforcement learning environment.

[0039] In the embodiments of this disclosure, the specific structure (such as multilayer perceptron, convolutional neural network, Transformer, etc.) of the policy network and (optionally) value network contained within the reinforcement learning agent, as well as the parameter initialization method (such as Xavier initialization, Kaiming initialization, zero initialization, etc.), are all known technologies in the art, and this disclosure does not limit them.

[0040] like Figure 1 As shown, the training method includes the following iteratively executed steps S100 to S400 until a predetermined training termination condition is met.

[0041] Before the iterative execution steps S100 to S400, the training method further includes an initialization step. The initialization step includes: initializing the policy network 100 contained in the reinforcement learning agent; initializing the pre-trained large language model of the large language model guide 200, so that the pre-trained large language model can output an action proposal distribution for the action space of the reinforcement learning environment; and setting the guide weights to initial values.

[0042] Specifically, the initialization steps include the following sub-steps: Sub-step 1: Initialize policy network 100, denoted as ,in These are the parameters of the policy network 100. In an embodiment employing an actor-critic class reinforcement learning algorithm, the reinforcement learning agent further includes a value network 110, denoted as... (or action value network) ),in These are the parameters for Value Network 110.

[0043] Sub-step two: Initialize the large language model bootloader 200, enabling the large language model bootloader 200 to adapt to the current environment state. Output the suggested probability distribution for the action space. (It could also be a motion preference rating).

[0044] Sub-step three involves setting the guiding weight to an initial value, wherein the initial value is within the range of [0.8, 1.0]. In this embodiment, this sub-step initializes the adaptive weight controller (also known as the dynamic weight controller) 310, which maintains a real-time guiding weight. The guiding weight This represents the proportion of the guidance path in the fusion strategy of the large language model. During initialization, the guidance weights are set to their initial values. In some embodiments, the initial value lie in Within the range. In a typical embodiment, it can be set In this embodiment, it can be agreed that... With the maximum guiding weight mentioned below Equal, that is, in the training time step stage, Equal to .

[0045] Sub-step four: Initialize performance evaluation buffer 320 to record performance metrics from recent training cycles.

[0046] After completing the above initialization steps, proceed to iteration steps S100 to S400.

[0047] In step S100, at each training time step This step completes the generation of a dual-path strategy for the current state, weighted fusion, action sampling and execution, and acquisition of environmental feedback. Specifically, step S100 includes the following sub-steps S101 to S105.

[0048] In step S101, at each training time step, the current state of the reinforcement learning environment is obtained. The reinforcement learning agent observes the current training time step. The current state of the reinforcement learning environment described below and the current state Input the large language model facilitator 200 and the policy network 100 respectively.

[0049] In step S102, the current state is... The input includes a pre-trained large language model and a large language model guide 200, and the output of the large language model guide 200 is the action proposal distribution for the action space of the reinforcement learning environment. .

[0050] In an embodiment, the large language model guide 200 outputs the action suggestion distribution through the following steps.

[0051] First, based on the current state Generate state descriptions. Specifically, this involves defining the reinforcement learning states. Convert to natural language description:

[0052] in, It describes the generating function. It is a mapping rule from state to text. In one embodiment, the mapping rule... This can be achieved through template filling, for example, by setting the state. The various dimensions are filled with templates such as "position={x}, velocity={v}, target={g}". In another embodiment, the mapping rule... This is achieved through a lightweight pre-trained encoder, for example, by encoding the state vector using a Transformer encoder and then decoding it into natural language text using a caption model. This disclosure relates to the mapping rules described herein. The specific implementation method is not limited.

[0053] Second, a prompt is constructed based on the state description, historical trajectory, task objective description, and constraints. The historical trajectory includes historical states, historical actions, and historical rewards. Specifically, the prompt, which includes task context and historical information, is constructed as follows:

[0054] in, For the historical trajectory, For the length of the historical trajectory, Description of the task objectives, As a constraint. In some embodiments, when constructing the prompt message, a confidence evaluation of the policy network, such as a normalized value based on policy entropy, can also be input. (about (See the definition in the Performance Health Score Calculation section below). It is used to provide the large language model guide 200 with additional signals of the agent's current policy uncertainty, so that the large language model guide 200 can adjust its output suggestions based on the current confidence level of the policy.

[0055] Third, the prompt message Input the pre-trained large language model to obtain confidence scores for each action in the action space. ,in The large language model guide represents 200 pairs of actions. The confidence score.

[0056] Fourth, the action suggestion distribution is obtained based on the confidence score.

[0057] In one embodiment, the action suggestion distribution is obtained by normalization based on the confidence score and temperature parameter. In one embodiment, the action suggestion distribution is as follows:

[0058] in, For the large language model guide 200 pairs of actions Confidence score, To enhance the current state of the learning environment, This refers to the temperature parameter. The temperature parameter... Used to control the sharpness of the motion suggestion distribution: when When the confidence level is low, the action suggestion distribution tends to be sharper, concentrating more probability quality on the actions with the highest confidence scores; when... When the value is large, the distribution of the suggested actions tends to be flat, making the probabilities of each action more similar.

[0059] In step S103, the current state is... Input policy network 100, and obtain the policy distribution output by policy network 100. .in, Indicates that policy network 100 is in state Select action The probability of.

[0060] In step S104, based on the guiding weight Distribution of action suggestions With policy distribution We perform weighted fusion to obtain the fusion strategy. .

[0061] like Figure 2 As shown, the policy fusion unit 300 is configured to execute step S104.

[0062] In this embodiment, the weighted fusion includes action-level fusion, and the action-level fusion determines the fusion strategy according to the following formula:

[0063] in, As a fusion strategy, For action, To enhance the current state of the learning environment, For action suggestion distribution, For policy distribution, To guide weights, .

[0064] In another embodiment, the weighted fusion includes attention modulation fusion. Specifically, the attention modulation fusion includes: obtaining the query vector from the large language model side based on the action suggestion distribution. The query vector for the reinforcement learning side is obtained based on the policy distribution or the hidden layer of the policy network. and obtain the action corresponding key vector According to the aforementioned guiding weight The correlation between the query vector and the key vector corresponding to the action on the large language model side, and the correlation between the query vector and the key vector corresponding to the action on the reinforcement learning side, are weighted to obtain the fusion strategy. The attention modulation fusion is determined according to the following formula:

[0065] in, As a fusion strategy, For action, To enhance the current state of the learning environment, To guide weights, , This is the query vector for the large language model. To reinforce the query vector, For action The key vector, The dimension is vector. In one embodiment, the query vector of the large language model... The query vector is obtained by linear projection from the hidden state of the large language model; The action is obtained by linear projection from the penultimate hidden state of the policy network 100; corresponding key vector It is obtained from the embedding representation of actions; For the dimension of attention head.

[0066] In another embodiment, the weighted fusion may further include target-level hierarchical fusion. Specifically, the target-level hierarchical fusion includes: [the following is a partial translation of the original text: "the current state..."] At least one of the following should be input into the large language model guide 200: task objective description, constraints, and historical trajectory, to obtain objective guidance information. The target guidance information The target objectives used to characterize the reinforcement learning agent's phased task objectives after the current training time step may include, for example, at least one of the following: high-level sub-objectives, target states, key states, task phase identifiers, action constraints, reward preferences, or natural language task descriptions; and target guidance information. Converted into target condition information that the policy network 100 can process (e.g., mapped into a vector by the encoder). ); the current state With the target condition information Input the policy network 100, and cause the policy network 100 to output a policy distribution based on the target condition information. .

[0067] In step S105, an action is performed according to the fusion strategy to transfer the reinforcement learning environment from the current state to the subsequent state and obtain the reward feedback from the reinforcement learning environment.

[0068] In one embodiment, sampling and executing an action according to the fusion strategy includes: determining an action from the action space according to the action probability distribution indicated by the fusion strategy; and applying the determined action to the reinforcement learning environment so that the reinforcement learning environment provides feedback on the reward and subsequent state.

[0069] Specifically, reinforcement learning agents rely on fusion strategies Sampling to obtain action :

[0070] and the action This is applied to the reinforcement learning environment, causing the reinforcement learning environment to change from its current state. Transition to successor state and feedback on the action. Corresponding rewards In this embodiment, the subsequent state It is relative to the current state. and current action The next state in terms of training time step is... Reinforcement learning agents to perform actions Then, the reinforcement learning environment adjusts according to the current state. With action Based on its state transition mechanism (such as transition probability) The subsequent state is given by [the relevant factor]. At different training time steps, due to the randomness of the state, action, and environment, the resulting subsequent state may differ; if the reinforcement learning environment is random, then even the same [condition / condition] will produce different subsequent states. With the same Multiple executions may yield different subsequent states. The subsequent states... Usually used as the next training time step The current starting point. It should be noted that if the subsequent state... If the current phase is terminated, the current episode ends, the environment is reset, and the new phase begins from the initial state. In implementations using parallel environments or batch trajectory training, the subsequent state... It is the successor state of the corresponding environment instance, not a globally unique next state.

[0071] In this embodiment, the experience data includes experience tuples, which include the current state, the action performed, the reward, and the subsequent state. Specifically, the reinforcement learning agent uses experience tuples... The data is stored in an experience replay pool or a trajectory buffer for the reinforcement learning algorithm to retrieve after a second predetermined time window has elapsed. In embodiments employing off-policy algorithms (such as DQN, SAC, etc.), the experience data is stored in the experience replay pool; in embodiments employing on-policy algorithms (such as PPO, A2C, etc.), the experience data may be stored in the trajectory buffer.

[0072] In step S200, in response to the expiration of a first predetermined time window, a reward sequence is determined based on the rewards from the reinforcement learning environment feedback within the first predetermined time window, a real-time performance index of the reinforcement learning agent is determined based on the reward sequence, and the guiding weights are updated based on the real-time performance index.

[0073] In some embodiments, the first predetermined time window is a time window corresponding to a first number of training time steps, where the first number is 10 to 100; that is, the guiding weight is updated according to a predetermined weight update frequency (e.g., once every 10 to 100 training time steps).

[0074] In some embodiments, the real-time performance metric is a performance health score characterizing the interaction between the reinforcement learning agent and the reinforcement learning environment under the guidance of the fusion policy. (Performance Health Score). This refers to the performance health score. It is obtained by weighting multiple performance sub-indicators, including RewardScore, StabilityScore, ProgressScore, and ExplorationScore.

[0075] The performance health score is:

[0076] in, To score performance health, The weighting coefficients are and satisfy the following conditions: In a typical configuration, the following is acceptable: , , , In embodiments oriented towards sparse reward tasks, it can be adjusted to... (To place greater emphasis on the contribution of task progress to performance health score). This disclosure addresses... The specific value is not limited, and those skilled in the art can adjust it according to the characteristics of the specific task.

[0077] The specific calculation methods for each performance sub-indicator are explained in detail below.

[0078] Reward points The short-term and long-term reward performance of the reinforcement learning agent is reflected by the short-term average reward and long-term average reward in the reward sequence, and is determined by the following formula:

[0079] in: For short-term average rewards, The length of the short-term reward window (typically 10 to 50 steps); For long-term average rewards, The length of the long-term reward window (typically 100 to 1000 steps). This is a normalization function used to map the original mean reward to... interval; To reinforce the rewards for feedback from the learning environment; The weighting of short-term average reward and long-term average reward. It should be noted that here... The mixing coefficients within the reward score, along with the guiding weights used below to characterize the large language model. different.

[0080] Strategy stability score The convergence and volatility of the strategy are measured using the following formula, which is based on the volatility of the reward sequence and the entropy of the strategy distribution:

[0081] in: To reward standard deviation, This represents the long-term average reward. This is the ratio of policy entropy to maximum entropy; The stability sensitivity coefficient; Let be the policy entropy.

[0082] Task progress score Used to quantify task completion status, determined based on the number of completed sub-goals and the number of critical states visited, using the following formula:

[0083] in, This represents the number of sub-objectives that have been completed. The total number of sub-objectives This represents the number of key states that have been visited. This represents the total number of critical states. For critical states, access weight coefficients are used. In some embodiments, the completion of a sub-goal can be achieved through a success condition function in the task definition (such as...). The detection method for sub-targets is automatically determined by either the reward signal threshold or the reward signal threshold; this disclosure does not limit the specific detection method for sub-targets.

[0084] Explore Score The efficiency of exploration is evaluated based on the number of new states discovered and the number of states repeatedly visited within a predetermined exploration time window, determined by the following formula:

[0085] in, In order to explore within the scheduled time window The number of new states discovered during the step; In order to explore within the scheduled time window The number of states visited repeatedly in the step. The determination of the "new state" can be made using any of the following existing technologies: (i) when the cosine similarity between the state vector and the historical state library is lower than a predetermined threshold, it is determined to be a new state; (ii) when the state is mapped to discrete buckets by hash bucketing, it is determined to be a new state when the state first falls into a bucket; (iii) when the novelty of the state is evaluated by methods such as random network distillation (RND).

[0086] like Figure 2 As shown, the performance health score calculator 330 is configured to perform the performance health score calculation step in step S200, and redundant descriptions are omitted here.

[0087] The guiding weight According to training time steps Performance health score Dynamic adjustment is employed, using a piecewise adaptive function. The guiding weight is determined by the following formula:

[0088] in, This is the maximum guiding weight (typically 0.8 to 1.0, used in the early stages of training). The minimum guiding weight (typically 0.0 to 0.2, used in the later stages of training); The basic guiding weight (typically 0.3 to 0.5); This refers to the number of steps taken during the warm-up phase (typically accounting for 10% of the total training steps). This is the number of steps to begin the transition phase (typically accounting for 30% of the total training steps). The attenuation coefficient; This is the steepness coefficient of the Sigmoid function; This represents the performance threshold.

[0089] The guiding weight is obtained through the above piecewise adaptive function. It has the following four-stage characteristics: High weights in the early stages of training. When training time... hour, (Typical close to 1). At this stage, the reinforcement learning agent is in the early stages of training and the strategy has not yet been formed; by assigning a high weight to the large language model guide, its prior knowledge can be used to guide the agent to quickly enter the effective exploration area and avoid blind trial and error.

[0090] Mid-training decay transition. When... hour, According to the exponential function Smooth decay; performance health score The higher the value, the faster the decay. At this stage, as the performance of the reinforcement learning agent improves, the guiding weights gradually and smoothly decay, thereby increasing the proportion of autonomous decision-making in the policy network and encouraging the agent to internalize policies and adapt to environmental details.

[0091] Low weights in the later stages of training. hour, Depend on Given that the weights converge to In the vicinity. At this stage, the reinforcement learning agent reaches a stable learning phase, and the weights decrease to a lower level, allowing the agent to mainly rely on the policy network for decision-making, thereby ensuring the autonomy and robustness of the policy.

[0092] Resilient callback during performance fluctuations. This occurs when a performance degradation or bottleneck is detected in the reinforcement learning agent (i.e.,...). When the weight is significantly reduced, the guiding weight can temporarily rebound, allowing the large language model guide to re-intervene and provide strategy correction or new exploration directions.

[0093] It should be noted that the guided weight scheduling method used in this disclosure is a "dynamic adjustment". Although the "segment boundaries" of the piecewise adaptive function are determined by the training time step... The decision is made, but the weight values ​​within each segment are determined by the performance health score. Modulation is performed, and this dual-variable scheduling, combined with the elastic callback mechanism described below, differs from existing technologies that rely solely on training time steps. There are substantial differences between open-loop schedules (such as Cosine schedule, Polynomial decay, and other learning rate schedules).

[0094] The process of updating the guiding weights based on real-time performance metrics also includes performance degradation detection and weight recovery strategies. Specifically, it includes the following two steps.

[0095] The first step is performance degradation detection. Based on the mean of the performance health score within the current sliding window and the mean within the previous sliding window, it is determined whether the reinforcement learning agent meets the performance degradation condition. Specifically, the performance degradation detection algorithm is as follows:

[0096] in, To detect window size, This is the threshold for the decrease.

[0097] The second step is the weight restoration strategy. If the performance degradation condition is met, the guiding weights used for subsequent training time steps are increased. Specifically, the increased guiding weights are:

[0098] in, For recovery rate, To score performance health, The historical peak performance health score is used. The weight recovery strategy adopts a proportional callback method—the increase in weight is proportional to the degree of decline in the current performance health score relative to the historical peak, which is in line with the design intent of "adaptive smoothing strategy, where the rate of change of weight is inversely proportional to the rate of change of agent performance".

[0099] In another embodiment, the weight recovery strategy can adopt a single-step fixed-increment approach:

[0100] in, For weight increment, This is the threshold for single-step performance degradation. It determines whether performance has degraded based on single-step performance comparisons and increases the guiding weight by a fixed increment when a degradation occurs. The weight can be temporarily increased when a performance degradation is detected.

[0101] In another embodiment, the weight recovery strategy can also employ a bidirectional scheduling method based on performance improvement rate. Specifically, let the nearest... The performance improvement rate at each evaluation point is ,but:

[0102] in, The improvement rate threshold is set (e.g., 5%). When performance is determined to be in a plateau phase, the bootstrapping weight is increased as follows:

[0103] in, To increase the step size, The bootstrap weight is capped (usually 1.0); when performance is determined to be in a period of rapid growth, the bootstrap weight is reduced in the following way:

[0104] in, To reduce the step size, To set the lower bound of the guiding weight (which can be set to 0, or a very small value such as 0.1 to ensure minimal guidance).

[0105] In step S300, in response to the expiration of the second predetermined time window, a reinforcement learning algorithm is used to update the parameters of the policy network 100 based on the empirical data generated by the interaction between the fusion policy and the reinforcement learning environment.

[0106] In some embodiments, the second predetermined time window is a time window corresponding to a second number of training time steps, where the second number is 512 to 2048; that is, the parameters of the policy network 100 are updated according to a predetermined parameter update frequency (e.g., updated once after collecting empirical data corresponding to 512 to 2048 training time steps).

[0107] In some embodiments, the reinforcement learning agent further includes a value network 110, which is used to estimate the state output value based on the reinforcement learning environment. Updating the parameters of the policy network 100 includes: using an actor-critic class reinforcement learning algorithm to update the parameters of the policy network 100 and the parameters of the value network 110 based on the same batch of empirical data.

[0108] It should be noted that in embodiments employing actor-critic algorithms (such as Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), A2C / A3C, etc.), the policy network... With value network It is jointly trained—taking PPO as an example, the policy network parameters are updated simultaneously in a single gradient step. With value network parameters It is driven by the composite loss function.

[0109] In a typical embodiment, the actor-critic reinforcement learning algorithm is a proximal policy optimization (PPO) algorithm. Specifically, the step of using the actor-critic reinforcement learning algorithm to update the parameters of the policy network 100 and the value network 110 based on the same batch of empirical data includes: obtaining a composite loss based on the policy loss term, the value loss term, and the policy entropy regularization term, and updating the parameters of the policy network 100 and the value network 110 based on the composite loss. The policy network 100 is denoted as... ,in The parameters of the policy network 100 are denoted as ; the value network 110 is denoted as . ,in These are the parameters of the value network 110. The composite loss is:

[0110] in, For the parameters of policy network 100, For the parameters of value network 110, For the strategy loss term, For value loss items, For policy entropy regularization, and This is the loss weighting coefficient.

[0111] Specifically, the strategy loss term The form is:

[0112] in, For the estimation of the advantage function, It can be calculated by the Generalized Advantage Estimation (GAE); These are the trimming parameters; This is the policy network before the update. In a typical embodiment, it can be... , , .

[0113] It should be further explained that, in calculating the policy gradient, the embodiments of this disclosure adopt the following design: using a policy network The output policy distribution is used as the policy to be updated, and the policy is fused. Used for action sampling and the generation of experience data. That is, the reinforcement learning agent integrates strategies. It interacts with the reinforcement learning environment under guidance and collects empirical data, but the calculation of the policy gradient is performed on the policy network. It itself (rather than the fusion strategy) ), thereby ensuring the policy network What it learns is its own policy distribution. This decoupled design means that the guidance role of the large language model guide 200 is only reflected in the data collection stage, rather than directly becoming the update target of the policy network, which is conducive to the reinforcement learning agent ultimately forming an independent, robust, and generalizable decision-making policy.

[0114] In another embodiment, the reinforcement learning algorithm is a soft actor-critic (SAC) algorithm; the reinforcement learning agent also includes two action-value networks. , The step of updating the parameters of the policy network 100 using a reinforcement learning algorithm includes: constructing a soft policy gradient based on the minimum estimation of the double Q network, and updating the parameters of the policy network 100. and the parameters of the two action value networks. Updates are performed separately within each training step. In another embodiment, the reinforcement learning algorithm can be other reinforcement learning algorithms such as DQN, A2C, A3C, DDPG, TD3, etc.

[0115] In step S400, in response to the determination that a predetermined training termination condition has not been met based on the training time step and / or the real-time performance metric, the process proceeds to the next training time step, repeating steps S100 to S300. The index of the training time step... Increasing.

[0116] In some embodiments, the predetermined training termination condition includes at least one of the following.

[0117] The first requirement is that the training time step reaches the predetermined number of training steps. That is, when... Exceeding the predetermined number of training steps When the training termination condition is met, it is determined that the predetermined number of training steps is satisfied. In some embodiments, the predetermined number of training steps... The value can be selected based on the task complexity; a typical value is... to step.

[0118] The second point is that the real-time performance indicators reach the predetermined performance targets. That is, when the performance health score... When the predetermined threshold is reached, the training termination condition is determined to be met.

[0119] Thirdly, the rate of change of the real-time performance metric within a predetermined performance time window is less than a convergence threshold, and the current training time step is greater than a predetermined minimum training step. Specifically:

[0120] in, This is the convergence threshold (typically 0.01). To minimize the number of training steps, This refers to the length of the long-term reward window (i.e., the long-term reward window mentioned in the performance health score calculation above).

[0121] Training ends when the training termination condition is met.

[0122] In some embodiments, the training method may further include: fine-tuning the pre-trained large language model online based on the state-action pairs in the successful trajectory of the reinforcement learning agent, the difference between the action proposal distribution in the failed trajectory and the target action, and the reward from the reinforcement learning environment.

[0123] Specifically, in the iterative loop of the training method, the pre-trained large language model contained in the large language model facilitator 200 can be fine-tuned in parallel using lightweight online methods. The fine-tuning signals are derived from: (i) state-action pairs in the agent's successful trajectory. (ii) Large language model suggestions in the agent's failure trajectory, as positive samples; With the actual optimal action Inconsistent items are used as negative samples; (iii) the reward signals from environmental feedback and the task completion status.

[0124] In one embodiment, the objective function for online fine-tuning employs a negative log-likelihood loss based on positive and negative samples:

[0125] in, For the set of state-action pairs in the success trajectory, This is the set of inconsistencies between the large language model's suggestions and the actual optimal actions in the failure trajectory. For negative sample weights, The parameters to be fine-tuned in the large language model (a subset corresponding to efficient parameter fine-tuning methods such as LoRA or Adapter).

[0126] In some embodiments, the fine-tuning frequency is triggered once every predetermined number of steps (e.g., 10,000 steps); the fine-tuning batch size can be 64 to 256; the preferred fine-tuning method is a parameter-efficient fine-tuning method such as LoRA, Adapter, or Prefix Tuning to avoid catastrophic forgetting caused by full-scale fine-tuning. In some embodiments, the fine-tuning learning rate of the large language model... Desirable Meanwhile, the learning rate used in the reinforcement learning algorithm Desirable .

[0127] Through the online fine-tuning described above, the large language model can gradually adapt to specific task environments, providing more accurate and timely guidance.

[0128] In one embodiment of this disclosure, training can be performed using the following typical hyperparameter configuration: total number of training steps. : Selected based on task complexity, typical value is to Step; Initial large language model weights Minimum Large Language Model Weights Performance evaluation window Steps; Weight update frequency: updated every 10 to 100 steps (i.e., the first predetermined time window corresponds to 10 to 100 training time steps); Reinforcement learning network update frequency: updated after collecting empirical data corresponding to every 512 to 2048 training time steps (i.e., the second predetermined time window corresponds to 512 to 2048 training time steps); Reinforcement learning network learning rate Fine-tuning the learning rate of a large language model .

[0129] It should be understood that the above hyperparameters are merely illustrative examples, and this disclosure does not limit their specific values. Those skilled in the art can make corresponding adjustments according to the characteristics of specific tasks.

[0130] The training method for reinforcement learning agents provided in this disclosure achieves a smooth transition from external knowledge guidance to autonomous reinforcement learning through dual-path decision-making and performance feedback-based dynamic weight adjustment. Compared with the prior art, this disclosure has the following beneficial effects.

[0131] Significantly improves training efficiency and sample utilization. By assigning a high guidance weight to the large language model guide in the early stages of training, its prior knowledge can be used to guide the reinforcement learning agent to quickly enter the effective exploration region, avoiding blind trial and error; thus significantly improving training efficiency and sample utilization. Enhances the generalization ability and robustness of the agent's policy. By reducing the guidance weight of the large language model in the later stages of training, the reinforcement learning agent gradually transitions to autonomous decision-making; thus avoiding performance inhibition and adaptive degradation caused by a fixed guidance mode, enhancing the generalization ability and robustness of the agent's policy. Has dynamic weight adjustment capability for agent performance fluctuations. Through a performance feedback mechanism, when a decline in agent performance or a bottleneck is detected, the guidance weight can be dynamically increased, allowing the large language model guide to re-intervene and provide policy correction or new exploration directions; thus avoiding long-term entrapment in local optima. Applicable to various task types and various reinforcement learning algorithms. This disclosure does not limit the specific construction method of the reinforcement learning environment or the specific reinforcement learning algorithm used, so it can be widely applied to scenarios such as robot control, game agents, autonomous driving and resource scheduling, and is compatible with various reinforcement learning algorithms such as PPO, SAC, and DQN.

[0132] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0133] Reference Figure 3 An electronic device 3000 according to embodiments of the present disclosure may include a processor 3100 and a memory 3200. The processor 3100 may include (but is not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a microprocessor, an application-specific integrated circuit (ASIC), etc. The memory 3200 may store computer programs to be executed by the processor 3100. The memory 3200 includes high-speed random access memory and / or a non-volatile computer-readable storage medium. When the processor 3100 executes the computer program stored in the memory 3200, the training method for the reinforcement learning agent as described above can be implemented.

[0134] Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and to provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. In one example, the computer programs and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0135] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for training a reinforcement learning agent, characterized in that, The training method is used to train the policy network contained in the reinforcement learning agent through the interaction between the reinforcement learning agent and the reinforcement learning environment. The training method includes: At each training time step, obtain the current state of the reinforcement learning environment; The current state is input into a large language model guide that includes a pre-trained large language model to obtain the action suggestion distribution for the action space of the reinforcement learning environment output by the large language model guide. The current state is input into the policy network to obtain the policy distribution output by the policy network; The action suggestion distribution and the policy distribution are weighted and fused based on guiding weights to obtain a fused policy; and The action is executed according to the fusion strategy to transition the reinforcement learning environment from the current state to the subsequent state, and the reward fed back by the reinforcement learning environment is obtained. In response to the expiration of a first predetermined time window, a reward sequence is determined based on the rewards fed back by the reinforcement learning environment within the first predetermined time window, a real-time performance index of the reinforcement learning agent is determined based on the reward sequence, and the guiding weight is updated based on the real-time performance index. In response to the expiration of the second predetermined time window, the parameters of the policy network are updated based on the empirical data generated by the interaction between the fusion strategy and the reinforcement learning environment; In response to the determination that the predetermined training termination condition has not been met based on the training time step and / or the real-time performance metric, proceed to the next training time step.

2. The training method according to claim 1, characterized in that, The training method further includes: performing an initialization step before the first training time step, the initialization step including: Initialize the policy network; Initialize the pre-trained large language model of the large language model facilitator, enabling the pre-trained large language model to output the action suggestion distribution for the action space of the reinforcement learning environment; and The guiding weight is set to an initial value, wherein the initial value is in the range of [0.8, 1.0].

3. The training method according to claim 1, characterized in that, The predetermined training termination conditions include at least one of the following: The training time reaches the predetermined number of training steps; The real-time performance indicators have reached the predetermined performance targets; The rate of change of the real-time performance metric within the predetermined performance time window is less than the convergence threshold, and the current training time step is greater than the predetermined minimum training step.

4. The training method according to claim 1, characterized in that, The large language model facilitator outputs the action suggestion distribution through the following steps: Generate a state description based on the current state; Based on the state description, historical trajectory, task objective description, and constraints, prompt information is constructed, wherein the historical trajectory includes historical states, historical actions, and historical rewards. The prompt information is input into the large language model guide to obtain the confidence score for each action in the action space; The action suggestion distribution is obtained based on the confidence score.

5. The training method according to claim 1, characterized in that, The weighted fusion includes action-level fusion, and the action-level fusion determines the fusion strategy according to the following formula: in, The fusion strategy, For action, This represents the current state of the reinforcement learning environment. The proposed distribution for the actions is as follows: For the policy distribution, The guiding weight, .

6. The training method according to claim 1, characterized in that, The weighted fusion includes attention modulation fusion, and the attention modulation fusion determines the fusion strategy according to the following formula: in, The fusion strategy, For action, This represents the current state of the reinforcement learning environment. The guiding weight, , This is the query vector for the large language model. To reinforce the query vector, For action The key vector, For vector dimensions.

7. The training method according to claim 1, characterized in that, The experience data includes experience tuples, which include the current state, the action performed, the reward, and the subsequent state. The experience data is stored in an experience replay pool or a trajectory buffer for the reinforcement learning algorithm to read after a second predetermined time window has passed.

8. The training method according to claim 1, characterized in that, The real-time performance metric is a performance health score used to characterize the interaction between the reinforcement learning agent and the reinforcement learning environment under the guidance of the fusion strategy. The performance health score is obtained by weighting multiple performance sub-metrics, including reward score, policy stability score, task progress score, and exploration score.

9. The training method according to claim 1, characterized in that, The reinforcement learning agent further includes a value network, which is used to output a value estimate based on the state of the reinforcement learning environment; updating the parameters of the policy network includes: An actor-critic class reinforcement learning algorithm is used to update the parameters of the policy network and the value network based on the same batch of experience data.

10. The training method according to claim 9, characterized in that, The actor-critic class reinforcement learning algorithm is a near-end policy optimization algorithm; the step of using the actor-critic class reinforcement learning algorithm to update the parameters of the policy network and the value network based on the same batch of empirical data includes: A composite loss is obtained based on the policy loss term, the value loss term, and the policy entropy regularization term. The parameters of the policy network and the value network are then updated based on this composite loss. The composite loss is: in, For the policy network, These are the parameters of the policy network; The parameters of the value network are... For the strategy loss term, For the aforementioned value loss item, For policy entropy regularization, and This is the loss weighting coefficient.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program; wherein, when the processor executes the computer program, it implements the training method as described in any one of claims 1 to 10.

Images