A Method and System for Assisted Hierarchical Reinforcement Learning Based on Large Language Models

Abstract

This invention proposes an assisted hierarchical reinforcement learning training method and system based on a large language model, comprising: a teacher agent and a student agent receiving the same environmental observation information; the teacher agent using a large language model to provide a guidance strategy based on the received environmental observation information, and the guidance strategy being introduced into a training module; and the student agent providing a high-level strategy based on the environmental observation information, wherein the high-level strategy is passed to the corresponding low-level strategy option π. ω At that time, π ω The system will generate a series of actions based on the high-level policy objectives and the environment until the termination condition is met or the maximum number of task steps is exceeded. At the same time, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment. The training module will use a loss function to update the high-level policy network of the student agent.

Description

Technical Field

[0001] This invention belongs to the field of large language model-assisted hierarchical reinforcement training technology, and particularly relates to a method and system for assisted hierarchical reinforcement learning training based on large language models. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] One of the goals of modern optimal control is to develop policies capable of completing long sequences of tasks, which requires an accurate understanding of the environment and strong reasoning abilities. Reinforcement learning has achieved great success in solving sequential decision-making problems. Optimal policies are learned through the interaction between the agent and the environment, thus reducing the need for prior knowledge.

[0004] With the continuous development of reinforcement learning technology, solving complex long-sequence decision-making tasks has become an important research topic in the field of artificial intelligence. However, the performance of traditional reinforcement learning deteriorates significantly when encountering long-sequence decision-making tasks. This is because the state and action spaces of long-sequence decision-making tasks are very large, leading to a significant performance drop in traditional reinforcement learning when solving such problems. Most research attempts to compensate for this limitation by designing reward functions. However, designing high-performance reward functions is difficult because the desired behavior is hard to quantify, or because dense and accurate rewards need to be decomposed into sub-goals or achieve asymptotic goals.

[0005] Therefore, in traditional reinforcement learning methods, due to the large state space and action space, the exploration efficiency of long sequence tasks is low, especially when environmental rewards are sparse, traditional methods are difficult to learn the optimal policy quickly.

[0006] Hierarchical reinforcement learning (HRL) uses a divide-and-conquer approach to solve problems. High-level policies operate on an abstract state space and are guided by external rewards. In contrast, low-level policies follow independent policies and receive separate rewards upon successful execution, which are hidden within the global reward. The superior performance of HRL can be attributed to more efficient reward credit allocation and more efficient exploration. By abstracting action sequences, HRL can learn the value function faster because rewards can be accurately propagated further within the sequence. According to Driess et al., HRL explores in a semantically meaningful and temporally extended manner, while traditional reinforcement learning leads to overexploration of nearby states.

[0007] Large Language Models (LLMs), with their powerful reasoning capabilities and extensive common sense base, are increasingly being introduced into the field of reinforcement learning. The use of large language models is a method to guide reinforcement learning. The reasoning power and extensive common sense of large language models can significantly improve the performance of reinforcement learning. While there is no unified approach to combining large language models with reinforcement learning, various strategies have been explored. Large language models are used for preprocessing states, shaping rewards, making direct decisions to improve sampling efficiency, generating world dynamics, or acting as expert guidance. For example, LLMs are used to translate historical trajectories or natural language instructions into task-specific language, thereby improving training efficiency. Large language models can act as implicit or explicit rewards. Huang et al. used small-scale large language models for fine-tuning to accelerate learning. Hao et al. encoded observations along with the objective as serial inputs to enhance policy generalization. In current hierarchical reinforcement learning methods, low-level policies can be predefined by experts or based on prior knowledge. However, high-level policies in hierarchical reinforcement learning are typically learned from scratch, which limits their performance.

[0008] However, while existing large language models perform excellently in fields such as natural language processing, they still suffer from problems when solving specific tasks, such as sorting items or using a key to open a door—common household tasks with long decision chains. These issues include a misalignment between input / output information and the dynamics of the environment, preventing the generation of executable strategies. This problem arises because large language models struggle to fully understand their surroundings solely based on textual information. Current solutions include repeatedly accessing the large language model to obtain feasible task plans, or introducing feedback mechanisms to inform the model of task planning failures and to revise the plan. Summary of the Invention

[0009] To overcome the shortcomings of the existing technology, this invention provides an assisted hierarchical reinforcement learning training method based on a large language model. By utilizing the prior knowledge of the large language model, it outputs suggested high-level strategies based on environmental information to assist the high-level strategies of hierarchical reinforcement learning, which can effectively reduce the action space and improve learning efficiency and task completion rate.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] Firstly, a method for training assisted hierarchical reinforcement learning based on a large language model is disclosed, including:

[0012] The teacher agent and the student agent receive the same environmental observation information;

[0013] The teacher agent uses a large language model to provide guidance strategies based on received environmental observation information. The guidance strategies include the probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategies are introduced into the training module.

[0014] The student agent provides a high-level policy based on environmental observations. When the high-level policy is passed to the corresponding low-level policy option π, ω At that time, π ω A series of actions will be taken based on the high-level strategic objectives and the environment until the termination conditions are met or the maximum number of task steps is exceeded.

[0015] Meanwhile, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0016] The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the cumulative sum of environmental rewards.

[0017] As a further technical solution, for the teacher intelligent agent:

[0018] First, define the high-level option set K. s ,K t K s K represents the task skills that the agent can currently choose. t This represents the task objectives that the agent can currently choose.

[0019] The large language model obtains the prompt text, where the prompt text p∈P consists of two parts: p m and p o , where p m p represents the description of the current task. o It is a textual description of the current environmental observation o;

[0020] The large language model will then output a high-level policy a to guide the student agent based on p. L ;

[0021] The action decomposition module will a L Decomposed into and The distribution vector representing the skill options chosen by the large language model. The distribution vector representing the target options selected by the large language model;

[0022] Query the large language model n times with the same prompt text, and count the results. and Option probability distribution

[0023] As a further technical solution, for student agents, including high-level policies and low-level policies, a hierarchical reinforcement learning method based on options is adopted for learning;

[0024] Among them, for high-level strategies, in K s Select the skill option in K t The final high-level decision is a combination of these two options: selecting the target for the current skill and choosing the target.

[0025] The lower-level strategy is responsible for generating specific lower-level actions and interactions with the environment based on the higher-level strategy.

[0026] As a further technical solution, the student agent uses two independent fully connected networks that share the same convolutional network for sensing the environment;

[0027] Two fully connected layers with a softmax function as the network output layer yield the student's higher-level strategy. and

[0028] Through student high-level strategies and Get the option index vector Combine them into a high-level action and pass it into the low-level strategy.

[0029] As a further technical solution, high-level strategies for student agents and Training is performed by minimizing the following loss function:

[0030]

[0031] Among them, L RL (s) and L RL (t) represents and Both use the standard reinforcement learning loss function. This represents the difference between the high-level strategies recommended by teachers and those given by students. This difference can be calculated using KL divergence or cosine similarity, where λ is the annealing parameter.

[0032] As a further technical solution, the annealing parameter λ is used to control the degree of guidance provided by the teacher agent to the student agent, and the annealing parameter λ is set according to the following formula:

[0033]

[0034] At the start of training, λ is set to a large value λ0 to ensure that the student agent receives more teacher guidance in the early stages of interaction with the environment. As the number of training steps i increases, the value of λ will decrease linearly with a decay rate k and become 0 after training to the i1th step.

[0035] Secondly, an assisted hierarchical reinforcement learning training system based on a large language model is disclosed, including: a teacher agent and a student agent, wherein the teacher agent includes a large language model, the student agent includes a high-level policy and a low-level policy, and the high-level policy includes a training module.

[0036] The teacher agent uses a large language model to provide guidance strategies based on received environmental observation information. The guidance strategies include the probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategies are introduced into the training module.

[0037] The student agent provides a high-level policy based on environmental observation information. When the high-level policy is passed to the corresponding low-level policy option π, ω At that time, π ω A series of actions will be taken based on the high-level strategic objectives and the environment until the termination conditions are met or the maximum number of task steps is exceeded.

[0038] Meanwhile, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0039] The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the cumulative sum of environmental rewards.

[0040] As a further technical solution, the teacher intelligent agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module;

[0041] The acquisition module is used to acquire environmental observation information and task text; the task text mainly includes task-related information.

[0042] The translation module receives environmental observation information and translates it into environmental information text.

[0043] The task text and environmental information text will be combined to form a prompt text, which will be sent to the large language model to obtain text actions and then sent to the action decomposition module.

[0044] The action decomposition module transforms text actions into skill option distribution vectors and target option distribution vectors. The index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model.

[0045] The option probability distribution acquisition module is configured to: repeatedly decompose the same prompt text into action to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector.

[0046] As a further technical solution, the student intelligent agent also includes a skill module and a target module. The skill module adopts a first fully connected network, and the target module adopts a second fully connected network.

[0047] The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting that skill.

[0048] The second fully connected network outputs multi-dimensional data, which constitutes the target option strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting that target.

[0049] Based on the skill selection strategy and the target selection strategy, select the skill and target with the highest probability, treat the index value of the skill and target as the high-level action, and pass it into the low-level strategy;

[0050] The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The corresponding strategy skill will be activated by the given skill index, and the target of the skill interaction will be determined by the given target index.

[0051] The low-level strategy controls the student agent to perform a series of actions until a termination condition is met or the maximum number of task steps is exceeded. Simultaneously, the student agent receives environmental rewards for each action performed. From r 1, arrive The sum of This is seen as a reward for the actions taken by the current top leadership. It will be sent to the training module for the student agent to learn from the environment.

[0052] The above one or more technical solutions have the following beneficial effects:

[0053] The technical solution of this invention is a hierarchical reinforcement learning framework assisted by a large language model. In this framework, the large language model acts as a teacher agent, guiding the exploration of high-level strategies in hierarchical reinforcement learning. In the high-level strategies, actions are decomposed into skills and objectives, thereby effectively reducing the action space and improving learning efficiency and task completion rate.

[0054] Because large language models have limited text input length, the technical solution of this invention implements a higher level of state abstraction at the input layer. Specifically, a state-based language encoder was developed. Figure 1The translation module (e.g., "the agent is holding the key") enables the large language model to output high-level actions that are consistent with expectations and have a certain degree of interpretability. The low-level policy must meet the following conditions: (1) the action input can be summarized in one sentence (e.g., "the agent picks up the key"). (2) the action of the low-level policy must conform to the high-level action and have a high success rate. During training, the large language model does not interact directly with the environment, but rather serves as an external reward to guide hierarchical reinforcement learning. The guidance weight of the large language model gradually decreases during training. Hierarchical reinforcement learning shifts from learning from the large language model to learning through interaction with the environment, thereby surpassing the performance of the large language model.

[0055] This invention proposes a novel method for embedding high-level policy actions. The actions of the high-level policy are decomposed into skills and objectives. The objective represents the interaction object selected by the high-level policy (e.g., a door, a key, etc.), and this interaction object changes with the environment. The skill represents the interaction method with the objective (e.g., basic actions such as picking up or opening). This interaction method remains consistent across different environments.

[0056] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0057] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0058] Figure 1 This is a schematic diagram of an auxiliary hierarchical reinforcement learning framework based on a large language model, according to an embodiment of the present invention.

[0059] Figure 2 These are example diagrams of four tasks used for experimental evaluation in this invention;

[0060] Figure 3 This is a schematic diagram illustrating a task prompt example of the present invention;

[0061] Figure 4 A diagram illustrating the success rate during the training process for four different tasks;

[0062] Figure 5 A diagram illustrating the average reward during the four task processes;

[0063] Figure 6 This is a diagram illustrating the training success rates for four tasks in the ablation experiment.

[0064] Figure 7 A schematic diagram illustrating the average training reward for four tasks in the ablation experiment;

[0065] Figure 8 This is a schematic diagram of the overall process of a sub-method in an embodiment of the present invention;

[0066] Figure 9 This is a schematic diagram of the overall process of the data processing method for the sub-teacher intelligent agent according to an embodiment of the present invention;

[0067] Figure 10 This is a schematic diagram of the overall process of the student intelligent agent data processing method according to an embodiment of the present invention. Detailed Implementation

[0068] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0069] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0070] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0071] In handling long-sequence decision-making tasks, frequent queries to large language models can lead to excessively high costs (if using commercial models) or longer inference times. In Yu's approach, the agent must consult the large language model for each action, and the large language model takes several seconds to respond. This approach cannot provide real-time responses. Many methods have been developed to address this issue. Bang developed GPTcache to query semantic answers, improving response time by 2-10 times. Others have proposed methods to restrict the large language model to use only during training, using reinforcement learning to train optimal policies, such as reward design and enhanced reasoning capabilities. In some schemes, the large language model is only used to output a rough plan, reducing consultation latency by selecting low-level policies required by the environment. Miao proposed a hierarchical knowledge representation framework to explore the integration of large language models in robot manipulation, which enhances task understanding through multi-source knowledge updates and embedding-based reasoning techniques.

[0072] The reasoning capabilities of large language models need improvement to directly solve long sequence tasks. As Valmeekam and Lou pointed out, large language models instinctively output reasoning trajectories in an autoregressive manner, serving as semantic logical chains of hypotheses. However, humans possess an internal world model, iteratively exploring various options and evaluating possible outcomes. In other words, large language models require an appropriate reward mechanism, a world model, and a balance between exploration and exploitation. Huang proposed a dialogue-like inner monologue to thoroughly observe the environment and utilize feedback from it. Chains of thought and adding examples are widely used methods for deconstructing complex problems and provide an interpretable window. ReAct has been proposed to improve performance by integrating reasoning and action generation in an interleaved manner. In ReAct, simply adding "Let's think it through step by step" can also improve the zero-shot reasoning performance of large language models. Another widely used approach is reinforcement learning with human feedback, such as MaxMin-RLHF with human preference data and InstructGPT with human labelers.

[0073] Aligning inputs and outputs with the environment is a challenge because large language models (LLMs) need to accurately perceive environmental information with limited inputs. Specifically, large LLMs may generate erroneous actions that the environment cannot perform, or they may fail to accurately understand the dynamics of the environment, particularly implicit interdependencies. Reinforcement learning is used at the output layer to overcome the alignment shortcomings of LLMs, as this algorithm can surpass the limited success rate of LLMs. In another approach, actors and commentators share the same LLM, updated via PPO using a low-rank adapter. Another approach outputs extensive high-level guidelines and fine-tunes them using PPOs, limiting misalignment. Some approaches use availability functions to verify the executableness of LLM output actions. Some approaches use code generation as a controlled output for robot control. Some approaches introduce MLP layers during generation and separate actions from values ​​for better accuracy. It also introduces embedded task instructions and visual observations through an encoder.

[0074] However, reinforcement learning struggles to learn optimal policies when completing a task requires excessive time and steps, and when environmental rewards are sparse. Utilizing prior knowledge can improve sample efficiency and training speed. Option-based hierarchical reinforcement learning categorizes actions into high-level and low-level actions. Each high-level action, or option, contains a series of low-level actions given by existing knowledge or policies. The agent learns high-level action policies to complete the entire task.

[0075] This paper presents a novel approach to address the challenges of long-sequence decision-making tasks in sparse reward environments by leveraging the reasoning capabilities and prior knowledge of large language models. However, applying large language models to long-sequence decision-making tasks still faces numerous challenges, such as their real-time response, their reasoning capabilities in complex tasks, and their understanding of the environment.

[0076] This embodiment describes the problem as option-based hierarchical reinforcement learning, which can perform temporary extension actions across multiple time steps. Option W is a triple (L w ,π w ,β w Let π represent the start condition, policy, and termination condition of the option, respectively. Given an environmental observation o, the higher-level policy selects the option policy π. w Where o∈L w π w This will involve multiple steps. Once the termination condition β is met... w The option strategy will stop and a new option strategy will be selected and executed by a higher-level strategy.

[0077] Example 1

[0078] This embodiment discloses an assisted hierarchical reinforcement learning training method based on a large language model, including:

[0079] like Figure 1 , 8 As shown, the framework consists of a teacher agent and a student agent. For each time t... h Teachers and students will receive the same observations. Students' intellectual understanding is based on observation A high-level decision is given, and the lower-level strategies, guided by the high-level decision, generate a series of specific actions. Wait until the lower-level strategy reaches the termination condition β w Then, the last observation This will be the next observation of high-level strategy. Make decisions. Each time the high-level strategy of the student agent is observed... At the same time, teachers' intelligent experiences receive the same observations. It also provides guidance strategies, namely recommended task skills and goals. These guidance strategies and environmental rewards will be incorporated into the training module to train students' high-level policy networks.

[0080] See appendix again Figure 1As shown, the teacher agent provides recommended task skills and objectives as high-level decision suggestions based on textual information from the environment and current task cues. The student agent operates within an option-based hierarchical reinforcement learning framework. It selects appropriate task skills and objectives as high-level policies based on environmental information. Low-level policies control the agent to perform specific actions and interact with the environment based on the high-level policies provided by the student agent. In the training module, the high-level decisions provided by the teacher and the student are incorporated into the loss function by calculating the KL divergence, enabling the teacher to guide the student.

[0081] The specific formula for calculating KL divergence is as follows:

[0082]

[0083] Regarding teacher intelligent agents:

[0084] First, define the high-level option set K. s ,K t K s K represents the task skills that the agent can currently choose. t This represents the task objective that the agent can choose at present. The prompt text p∈P obtained by LLM consists of two parts: p m and p o , where p m p represents the description of the current task. o This is a textual description of the current environmental observation, o. The LLM will then output a high-level strategy, a, to guide the student, based on p. L A hard-coded action decomposition module will a L Decomposed into and The vector representing the distribution of skill options chosen by LLM. This represents the distribution vector of the target options selected by the LLM. Considering that the LLM does not directly interact with the environment, the LLM cannot confirm the resulting action 'a'. L This represents the optimal high-level decision based on the current environmental observation. This embodiment's sub-technical solution queries the LLM n times using the same prompt text, and then statistically analyzes the results. and Option probability distribution

[0085] Regarding student intelligent agents:

[0086] High-level policies: The student agent architecture employs an option-based hierarchical reinforcement learning approach. For high-level policies, a skill module is used to manage K... s Select the skill option; a target module is responsible for K. tThe student agent selects the target that the current skill is applying. The final high-level decision is a combination of these two options, such as choosing the blue key. This combination reduces the action space of the high-level decision options, especially when the student agent possesses a large number of skills and there are many objects in the environment that the agent can interact with. Specifically, two independent fully connected networks are used, sharing the same convolutional network for environmental perception. The two fully connected layers use a softmax function as the network output layer to obtain the student's high-level policy. and Ultimately, high-level strategies are implemented through strategies. and Get the option index vector Combine them into a high-level action and pass it into the low-level strategy.

[0087] Low-level policies: Low-level policies are responsible for generating specific low-level actions and interactions with the environment based on high-level policies. This example uses multiple hard-coded policy options based on human expert knowledge. These policy options can also be replaced using pre-trained reinforcement learning networks. When the high-level policy is passed the corresponding low-level policy option π ω At that time, π ω A series of actions will be taken based on the strategic objectives of the higher authorities and the environment. Until the termination condition β is reached w Or exceeding the maximum number of task steps T low At the same time, each action output by the low-level policy will interact with the environment and receive environmental rewards. From r 1, arrive The sum of This is seen as a reward for the actions taken by the current top leadership. It will be sent to the training module for the student agent to learn from the environment.

[0088] Training module:

[0089] High-level strategies of student agents and Training is performed by minimizing the loss function.

[0090] This embodiment introduces an annealing parameter λ to control the degree to which the teacher agent guides the student agent.

[0091] The entire learning process is summarized in Algorithm 1.

[0092]

[0093]

[0094] For more specific implementation examples, see Appendix Figure 9 As shown, the teacher agent data flow is as follows:

[0095] 1. The teacher agent obtains environmental observation information o from the environment, where o is an RGB image.

[0096] 2. The environmental observation information o is fed into the translation module, which is implemented by a pre-trained visual language model such as the clip model or a hard-coded program. The translation module translates the environmental observation information o into text information p. o For example, "The agent observes that <red key>, <blue key>, <blue door> holds <nothing> facing <nothing>".

[0097] 3. Task text p m This mainly includes task-related information such as: "In a closed two-dimensional grid room, the agent needs to open the door."

[0098] The rules are as follows:

[0099] (1) An agent can only open a door if it has a key that matches the color of the door and is facing the door.

[0100] (2) An agent can only pick up a key when it is facing the key.

[0101] (3) The agent faces the object by walking up to it.

[0102] (4) An intelligent agent can only hold one object.

[0103] (5) An agent can perform the following operations: explore, go to <object>, pick up <object>, put down <object> or open <object>.

[0104] (6) Based on the information provided, clarify the objects that the agent sees and holds. Then infer what the agent can do in the current state.

[0105] (7) Finally, please select the correct action.

[0106] Example:

[0107] The intelligent agent sees <blue key>, <red key>, and <blue door>, holds <nothing>, and faces <nothing>.

[0108] Objects seen and held by the agent: The agent sees the blue key, the red key, and the blue door, holds nothing, and faces nothing.

[0109] What the agent should do: The agent should go to the blue key.

[0110] Action: {Go to <Blue Key>}.

[0111] 4. Task text p m and environmental information text po The two parts are combined to form the prompt text p. The prompt text p is then fed into the large language model (gpt4), and the output a of the large language model is obtained. L For example, "Go to the Blue Key".

[0112] 5. Place a L The text is sent to the action decomposition module, which is responsible for decomposing the text action 'a'. L Transform into a skill option distribution vector and target option distribution vector The index of each skill option vector represents the skill selected by the large language model, such as "go to". The first position represents explore, the second position represents pick up, the third position represents go to, the fourth position represents put down, and the fifth position represents open. The index of each target option distribution vector represents the interaction target selected by the large language model, such as "blue key". The first digit represents none, the second digit represents a red key, the third digit represents a blue key, the fourth digit represents a red door, and the fifth digit represents a blue door.

[0113] 6. Repeat steps 4 and 5 n times using the same p, to obtain... and Option probability distribution Right now:

[0114]

[0115] in, This represents the distribution vector of skill options obtained from the i-th query of LLM. This represents the distribution vector of the target options obtained from the i-th query LLM. Guidance will be provided during students' online training; this part will be introduced in the student section.

[0116] 7. The data is fed into the training module to assist students in training their intelligent agents.

[0117] See appendix Figure 10 As shown, the student agent data flow is as follows:

[0118] 1. The student agent structure adopts an option-based hierarchical reinforcement learning approach. The student agent is decomposed into high-level policies and low-level policies.

[0119] 2. The environmental observation (image) is fed into the convolutional network to sense the environment. There are three convolutional layers: the first layer outputs 16 feature maps using a 3x3 convolutional kernel with 1-pixel padding at the edges; the second layer outputs 32 feature maps using a 3x3 convolutional kernel with 1-pixel padding at the edges; and the third layer outputs 64 feature maps using a 3x3 convolutional kernel with 1-pixel padding at the edges. Finally, the feature maps are flattened into a one-dimensional tensor using the ReLU activation function.

[0120] 3. The output of the convolutional network is connected to two fully connected networks to form the skill selection network and the target selection network. The skill selection network consists of a single 64-node fully connected network and a softmax activation function, outputting 5-dimensional data to constitute the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of choosing that skill. The target option network consists of a single 64-node fully connected network and a softmax activation function, outputting multi-dimensional data (determined by the number of environmental targets) that constitute the target option policy output by the student agent. Each dimension represents a goal, and the data in each dimension represents the probability of selecting that goal.

[0121] 4. According to and Select the skill and target with the highest probability, treat their index values ​​as high-level actions, and pass them into the low-level strategy.

[0122] 5. The low-level strategy possesses multiple hard-coded policy skills based on human expert knowledge, such as controlling the agent to explore, move towards, pick up, put down, and open targets in the environment. The skill index given in step 4 will activate the corresponding policy skill, and the target index given in step 4 will determine the target of the skill interaction.

[0123] 6. The low-level policy module controls the agent to produce a series of actions. Until the termination condition β is reached w Or exceeding the maximum number of task steps T low At the same time, the agent receives environmental rewards for each action it performs. From r1, to The sum of This is seen as a reward for the actions taken by the current top leadership. It will be sent to the training module for the student agent to learn from the environment.

[0124] Training module data flow:

[0125] 1. Environmental rewards The data is fed into the training module, and the high-level network loss function L of the student agent is obtained through the Proximal Policy Optimization (PPO) reinforcement learning algorithm.RL (s)+L RL (t);

[0126] 2. The data are fed into the training module for calculation. and The distribution differences and The distributional differences are measured using the KL divergence, i.e. Add it to the loss function.

[0127] The two losses obtained from steps 1 and 2 are summed together, and an annealing parameter λ is introduced to control the degree to which the teacher agent guides the student agent.

[0128] 3. At the start of training, λ is set to a relatively large value, λ0, to ensure that the student agent receives more teacher guidance in the early stages of interaction with the environment. As the number of training steps i increases, the value of λ will decrease linearly with a decay rate k and become 0 after the i1th training step. At this point, the student agent will learn entirely through interaction with the environment, further improving its robustness to specific tasks.

[0129]

[0130] 4. The final loss function is as follows:

[0131]

[0132] The loss function will be used to update the high-level policy network of the student agent.

[0133] Among them, L RL (s) and L RL (t) represents and Both methods employ standard reinforcement learning loss functions. This paper uses the Proximal Policy Optimization (PPO) algorithm as the reinforcement learning algorithm. The agent learns from the environment by maximizing the agent's expected reward. This indicates the difference between the high-level strategies recommended by teachers and those given by students. and Let $\mathbf{a}$ and $\mathbf{b}$ represent the expected values ​​of teacher and student skill strategies and target strategies under environmental observation $\mathbf{o}$, respectively. This strategy difference can be calculated using KL divergence or cosine similarity.

[0134] During the training process, the training module participates in the training of the high-level policy network, while during the testing phase, only the skill module and the target module are used for testing.

[0135] The LLMHRL was evaluated through various experiments, demonstrating its significant advantages over other baseline methods.

[0136] This example uses the MiniGrid environment to evaluate LLMHRL. MiniGrid provides a grid-based visual environment where users can freely set the grid size and the interactive objects for the agent, such as walls, doors, and keys. By interacting with objects, the state of the objects can be changed, for example, turning a closed door into an open one. The agent navigates to specific objects and completes a series of interactions to accomplish a task, such as "put all the balls into the box." The agent's action space is discrete, including {move forward, turn left, turn right, open, close, stop}. The observation space is set to a 3x3 locally observable environment with built-in memory to record discovered blocks or targets.

[0137] This article uses the MiniGrid framework to construct four different tasks, with increasing difficulty, such as... Figure 2 As shown. Specifically, more challenging tasks require longer decision chains and have a lower tolerance for incorrect behavior in order to obtain rewards and achieve goals.

[0138] Two-color door key: After generating two keys and a door, the agent needs to find and retrieve the correct color key for the door and use it to open the door.

[0139] Three-color door key: After generating three keys and a door, the agent needs to find and retrieve the correct color key for the door and use it to open the door.

[0140] Three-Ball Box: Generate three balls and three boxes. The agent needs to find and put the balls into the boxes of the correct colors.

[0141] Four Doors to the Goal: A three-level environment is generated. The agent needs to explore and find the correct key for each door, retrieve the key, open the door, and place the key in the first and second levels. The environment also generates a fake door that cannot be opened with a key as a distraction. In the final level, the agent must find and reach the goal.

[0142] The above tasks use the same reward function:

[0143]

[0144] step finish This represents the number of low-level policy steps taken by the agent to complete the current task. max This represents the maximum number of low-level policy steps the agent can execute. Such a reward function design encourages the agent to complete the task in as few steps as possible.

[0145] Experimental setup:

[0146] Baseline: In the comparative experiments, this paper uses four baseline methods to evaluate LLMHRL.

[0147] LLM4Teach is an end-to-end reinforcement learning method guided by a large language model, where the model learns task policies based on behavioral suggestions provided by the LLM. Similar to LLMHRL, this method also uses a large language model for policy guidance; therefore, this example uses LLM4Teach as a comparative experiment.

[0148] LLM uses a large language model for high-level decision-making, providing the high-level actions the agent should take. The low-level policy module provides the specific actions the agent needs to execute. Using this approach, the large language model must remain online to make real-time decisions. In comparison, it's possible to evaluate whether LLMHRL effectively learns high-level policies from the large language model in the early stages of training and gradually surpasses it in later stages.

[0149] Hierarchical Reinforcement Learning (HLRL) employs an option-based hierarchical reinforcement learning approach. Each option represents a low-level policy consistent with LLMHRL. This paper compares hierarchical reinforcement learning with LLMHRL to evaluate whether LLMHRL effectively learns high-level policies from LLMs.

[0150] RL (PPO) is an end-to-end reinforcement learning baseline approach that uses a proximal policy optimization algorithm. This approach will demonstrate the performance of typical reinforcement learning algorithms in sparse reward environments.

[0151] Large Language Model: In this example, Chat-GPT4 is used as a large language model to guide the high-level policy of LLMHRL. Its powerful reasoning ability can provide practical guidance policies for LLMHRL. The large language model method in the comparative experiment also uses this model to ensure fairness.

[0152] To enhance the reasoning ability of the large language model, this embodiment employs a chain-like text prompting system, including task text and environment text. The task text provides the rules and specific examples of the current task, helping the large language model understand the meaning of each high-level decision and the format of the output text. The environment text is provided based on the agent's observations of the environment; the large language model needs to provide appropriate high-level policy guidance based on different environment observation texts. The high-level decisions output by the large language model include an explanation of the current environment, the high-level decisions the agent should subsequently make, and the keywords of the corresponding decisions. Specific examples are as follows... Figure 3 As shown.

[0153] Low-level policies: The LLMHRL method constructs a low-level policy library containing skills and objectives, with five different skills and twenty-four different objectives. The execution of the low-level policy is jointly determined by the skills and objectives.

[0154] The "Explore" skill automatically detects undiscovered blocks and moves towards them. It's important to note that for the "Four Doors to the Target" task, the "Explore" skill can only discover the target within its own level. If the maximum number of steps is exceeded, this skill will automatically terminate. The "Go To" skill retrieves target information from historical observations and performs target localization. If the target has been discovered, the agent will use the A* algorithm to move towards it. The "Pick Up" skill determines whether the target in front of the agent matches the input target. If correct, the target will be picked up. The "Put Down" skill will only put down the target if there is nothing in front of the agent and the agent is holding the same target as the input target. The "Open" skill will change the state of the input target when it is within the agent's range. For example, a locked purple door requires an agent carrying a purple key to change its state to open.

[0155] Experimental results:

[0156] Table I

[0157] Success rates of four tasks

[0158]

[0159] Table II

[0160] Average reward of four tasks

[0161]

[0162]

[0163] Table I and Figure 4 The success rate performance of five methods deployed in four different environmental tasks is shown. LLMHRL achieved the highest success rate in three of the four tasks. This indicates that LLMHRL can effectively adapt to different tasks. Furthermore, the convergence speed of LLMHRL's success rate training curve is significantly faster than the other methods. This suggests that LLMHRL receives sufficient guidance from the large language model in the early stages of training, thus rapidly improving task performance. Table II and... Figure 5 The average return of the five methods deployed in four different environmental tasks is shown. A higher average return means that the agent completes the task in fewer steps.

[0164] LLMHRL also achieved the highest average reward in three out of the four tasks. Notably, LLMHRL's average reward surpassed that of large language models in all four tasks. This indicates that LLMHRL's performance further improves during interaction with the environment. Furthermore, LLMHRL maintains a high success rate and average reward even as the task difficulty increases. The performance of the other methods, however, declines significantly.

[0165] LLM4Teach performs well quickly on most tasks and converges to a good policy after 10,000 steps. However, it performs poorly on difficult tasks. This is because the student agent learns low-level actions. This means the agent needs to execute the correct sequence while mastering different skills. As the task decision chain becomes longer, the student agent requires more time steps to learn the correct policy under the guidance of a large language model.

[0166] Traditional reinforcement learning, regardless of whether it employs a hierarchical structure, fails to achieve satisfactory results. Without the aid of a large language model, reinforcement learning cannot learn effective policies, regardless of the hierarchical structure. Due to the sparse rewards, the agent needs assistance to understand which target to interact with to obtain the desired outcome, thus failing to comprehend the environment. Even with a hierarchical structure to reduce the length of the decision chain, this is still insufficient.

[0167] Based on large language models, agents can interact with their environment relatively successfully. However, the success rate is limited because language models lack task-specific knowledge and their output is inconsistent. This paper outlines several key issues with the output of large language models:

[0168] Unreadable strategy: The large language model is asked to output one of the specified actions, but instead of outputting "explore", it outputs "exploit". The low-level strategy cannot understand this output.

[0169] Limited reasoning ability: When seeing "red key, blue door", the large language model often outputs "pick up the red key" because it prefers to pick up any key regardless of its color. Conversely, when expecting to find the correct "blue key" to open the door, it should output "explore".

[0170] Lack of task-specific knowledge: In MiniGrid, the agent can only pick up a target if it is not holding any items. However, LLM keeps outputting "Pick up the blue key" even when it sees a "blue key" and is holding a "red key". Instead, it should suggest "Put down the red key".

[0171] Ablation experiment:

[0172] Table III

[0173] Success rate and average return of ablation experiments

[0174]

[0175] LLMHRL decomposes high-level policies into combinations of skills and objectives. Whether this decomposition-combination approach is effective needs to be verified experimentally. Therefore, this paper conducts a set of ablation experiments to compare LLMHRL with a method that does not decompose but directly gives all high-level policy options, which is called LLMHRL(NS).

[0176] As shown in Table III, Figure 6 and Figure 7 As shown, decomposing the action space into skills and objectives can improve performance on complex tasks. Specifically, except for the two-color door key task, the average training speed is faster and the success rate is higher than LLMHRL(NS). The reason behind this is that challenging tasks expand the action space, leading to a decrease in exploration efficiency. By dividing the action space into two parts, LLMHRL can effectively reduce the action space. This allows higher-level policies to better learn the guided policies of large language models.

[0177] This example combines option-based hierarchical reinforcement learning with a large language model to address the challenges of long-sequence decision-making tasks in sparse reward environments. LLMHRL guides reinforcement learning by calculating the difference in high-level policy action distribution between the large language model and the reinforcement learning process, and incorporates this difference into the loss function. High-level actions are divided into skill and objective components to reduce the number of selectable actions. Low-level actions consist of a skill set to perform different high-level decisions. Experiments show that LLMHRL can learn policies from large language models faster and gradually learn better policies through interaction with the environment. However, LLMHRL also has some limitations, such as the fixed options for high-level actions, which makes handling unknown objectives challenging.

[0178] The main contributions of LLMHRL are:

[0179] 1) In order to improve the performance of hierarchical reinforcement learning in long sequence tasks, a method is proposed to use a large language model to assist in training hierarchical reinforcement learning.

[0180] 2) To address the challenges of high-level strategies in long-sequence tasks with large action spaces, a method is introduced to decompose high-level actions into combinations of skills and objectives.

[0181] Example 2

[0182] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0183] Example 3

[0184] The purpose of this embodiment is to provide a computer-readable storage medium.

[0185] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0186] Example 4

[0187] See appendix again Figure 1 The purpose of this embodiment is to provide an assisted hierarchical reinforcement learning training system based on a large language model, including: a teacher agent and a student agent, wherein the teacher agent includes a large language model, the student agent includes a high-level policy and a low-level policy, and the high-level policy includes a training module;

[0188] The teacher agent uses a large language model to provide guidance strategies based on received environmental observation information. The guidance strategies include the probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategies are introduced into the training module.

[0189] The student agent provides a high-level policy based on environmental observation information. When the high-level policy is passed to the corresponding low-level policy option π, ω At that time, π ω A series of actions will be taken based on the high-level strategic objectives and the environment until the termination conditions are met or the maximum number of task steps is exceeded.

[0190] Meanwhile, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0191] The training module will use a loss function to update the high-level policy network of the student agent.

[0192] The teacher agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module;

[0193] The acquisition module is used to acquire environmental observation information and task text; the task text mainly includes task-related information.

[0194] The translation module receives environmental observation information and translates it into environmental information text.

[0195] The task text and environmental information text will be combined to form a prompt text, which will be sent to the large language model to obtain text actions and then sent to the action decomposition module.

[0196] The action decomposition module transforms text actions into skill option distribution vectors and target option distribution vectors. The index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model.

[0197] The option probability distribution acquisition module is configured to: repeatedly decompose the same prompt text into action to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector.

[0198] The student intelligent agent also includes a skill module and a goal module. The skill module uses a first fully connected network, and the goal module uses a second fully connected network.

[0199] The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting that skill.

[0200] The second fully connected network outputs multi-dimensional data, which constitutes the target option strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting that target.

[0201] Based on the skill selection strategy and the target selection strategy, select the skill and target with the highest probability, treat the index value of the skill and target as the high-level action, and pass it into the low-level strategy;

[0202] The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The corresponding strategy skill will be activated by the given skill index, and the target of the skill interaction will be determined by the given target index.

[0203] The low-level strategy controls the student agent to perform a series of actions until a termination condition is met or the maximum number of task steps is exceeded. Simultaneously, the student agent receives environmental rewards for each action performed. From r 1, arrive The sum of This is seen as a reward for the actions taken by the current top leadership. It will be sent to the training module for the student agent to learn from the environment.

[0204] Example 5

[0205] The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments.

[0206] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0207] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0208] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A hierarchical reinforcement learning training method based on a large language model, characterized by: include: The teacher agent and the student agent receive the same environmental observation image information; For the teacher agent: First, define the high-level option set. , , This represents the task skills that the agent can currently choose. This represents the task objectives that the agent can currently choose. The large language model obtains the prompt text, prompt text p P consists of two parts: and ,in A description representing the current task. It is a textual description of the current environmental observation o; The large language model will then output p to guide the high-level strategy of the student agent. This strategy is a text action; The motion decomposition module will Decomposed into and , The distribution vector representing the skill options chosen by the large language model. The distribution vector representing the target options selected by the large language model; Query the large language model n times with the same prompt text, and count the results. and Option probability distribution , ; The teacher agent uses a large language model to provide guidance strategies based on the received environmental observation image information. The guidance strategies include the option probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategies are introduced into the training module. The student agent provides a high-level policy based on environmental observation image information. When the high-level policy is passed on, the corresponding low-level policy option is selected. hour, A series of actions will be taken based on the high-level strategic objectives and the environment until the termination conditions are met or the maximum number of task steps is exceeded. The student agent feeds the environmental observation image into a convolutional network to perceive the environment. The output of the convolutional network is connected to two fully connected networks as a skill selection network and a target selection network, respectively. Two independent fully connected networks are used, while sharing the same convolutional network for perceiving the environment. Two fully connected layers with a softmax function as the network output layer yield the student's higher-level skill selection strategy. and high-level target options strategy ; Through student high-level skills options strategy and high-level target options strategy Get the option index vector , This is combined into a high-level action and passed into the low-level strategy. Meanwhile, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment. The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the cumulative sum of environmental rewards.

2. The assisted hierarchical reinforcement learning training method based on a large language model as described in claim 1, characterized in that, For student agents, including high-level and low-level policies, a choice-based hierarchical reinforcement learning approach is used for learning; Among them, for high-level strategies, in Select the skill option, in The final high-level decision is a combination of these two options: selecting the target for the current skill and choosing the target. The lower-level strategy is responsible for generating specific lower-level actions and interactions with the environment based on the higher-level strategy.

3. The assisted hierarchical reinforcement learning training method based on a large language model as described in claim 1, characterized in that, students High-level strategies of intelligent agents and Training is performed by minimizing the following loss function: in, and express and Both use the standard reinforcement learning loss function. This represents the difference between the high-level strategies recommended by teachers and those proposed by students. This difference can be calculated using KL divergence or cosine similarity. These are the annealing parameters.

4. The assisted hierarchical reinforcement learning training method based on a large language model as described in claim 3, characterized in that, The annealing parameters Annealing parameters are used to control the degree to which the teacher agent guides the student agent. The setting is based on the following formula: At the start of training, Set as To ensure that the student's intelligent agent receives more teacher guidance in the early stages of interaction with the environment, as the number of training steps i increases, The value will decrease linearly with a decay rate k and will continue to decrease until the training reaches the th . After the step, it becomes 0.

5. A large language model-based assisted hierarchical reinforcement learning training system employing the method described in claim 1, characterized in that, include: The system includes a teacher agent and a student agent, wherein the teacher agent includes a large language model, and the student agent includes a high-level policy and a low-level policy, wherein the high-level policy includes a training module. The teacher agent uses a large language model to provide guidance strategies based on the received environmental observation image information. The guidance strategies include the option probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategies are introduced into the training module. The student agent provides a high-level strategy based on environmental observation image information, and when the high-level strategy is passed in the corresponding low-level strategy option... hour, A series of actions will be taken based on the high-level strategic objectives and the environment until the termination conditions are met or the maximum number of task steps is exceeded. Meanwhile, each action output by the low-level policy will interact with the environment and receive environmental rewards. The sum of environmental rewards is regarded as the reward obtained by the current high-level action. The sum of environmental rewards will be sent to the training module for the student agent to learn from the environment. The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the cumulative sum of environmental rewards.

6. The assisted hierarchical reinforcement learning training system based on a large language model as described in claim 5, characterized in that, The teacher intelligent agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module; The acquisition module is used to acquire environmental observation image information and task text; the task text mainly includes task-related information. The translation module receives environmental observation image information and translates the environmental observation image information into environmental information text. The task text and environmental information text will be combined to form a prompt text, which will be sent to the large language model to obtain text actions and then sent to the action decomposition module. The action decomposition module transforms text actions into skill option distribution vectors and target option distribution vectors. The index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model. The option probability distribution acquisition module is configured to: repeatedly decompose the action using the same prompt text to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector; The student intelligent agent also includes a skill module and a target module. The skill module adopts a first fully connected network, and the target module adopts a second fully connected network. The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting that skill. The second fully connected network outputs multi-dimensional data, which constitutes the target option strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting that target. Based on the skill selection strategy and the target selection strategy, select the skill and target with the highest probability, treat the index value of the skill and target as the high-level action, and pass it into the low-level strategy; The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The corresponding strategy skill will be activated by the given skill index, and the target of the skill interaction will be determined by the given target index. The low-level strategy controls the student agent to perform a series of actions until a termination condition is met or the maximum number of task steps is exceeded. Simultaneously, the student agent receives environmental rewards for each action performed. ,from arrive The sum of This is seen as a reward for the actions taken by the current top leadership. It will be sent to the training module for the student agent to learn from the environment.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-4 above.

Images