Auxiliary hierarchical reinforcement learning training method and system based on large language model

Abstract

The invention provides an auxiliary hierarchical reinforcement learning training method and system based on a large language model. The method comprises the steps that a teacher agent and a student agent receive the same environment observation information; the teacher intelligent agent adopts a large language model and gives a guidance strategy according to the received environment observation information, and the guidance strategy is introduced into the training module; the student agent gives a high-level strategy according to environment observation information, and when the high-level strategy is transmitted into a corresponding low-level strategy option pi omega, the pi omega generates a series of actions according to a high-level strategy target and the environment until a termination condition is reached or the maximum task step number is exceeded; meanwhile, each action output by the low-level strategy interacts with the environment and obtains an environment reward, and the environment reward accumulation sum is regarded as a reward obtained by the current high-level action, and the environment reward accumulation sum is sent to the training module for the student intelligent agent to learn towards the environment; and the training module is used for updating the high-level strategy network of the student intelligent agent by using the loss function.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language model-assisted hierarchical reinforcement training, and in particular to a large language model-based assisted hierarchical reinforcement learning training method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] One of the goals of modern optimal control is to develop strategies that can complete long sequences of tasks, which requires accurate understanding of the environment and strong reasoning capabilities. Reinforcement learning has achieved great success in solving sequential decision-making problems. The optimal strategy is learned through the interaction between the agent and the environment, thus reducing the need for prior knowledge.

[0004] With the continuous development of reinforcement learning technology, solving complex long-sequence decision-making tasks has become an important research topic in the field of artificial intelligence. However, the performance of traditional reinforcement learning will drop significantly when encountering long-sequence decision-making tasks. Due to the large state and action space of long-sequence decision-making tasks, the performance of traditional reinforcement learning will drop significantly when solving such problems. Most studies compensate for this lack of exploration by designing reward functions. The design of high-performance reward functions is difficult to achieve because the expected behavior is difficult to quantify, or dense and accurate rewards need to be decomposed into sub-goals or achieve progressive goals.

[0005] Therefore, in traditional reinforcement learning methods, due to the huge state space and action space, the exploration efficiency of long sequence tasks is low, especially when the environmental rewards are sparse, it is difficult for traditional methods to quickly learn the optimal strategy.

[0006] Hierarchical reinforcement learning (HRL) uses a divide-and-conquer approach to solve problems. High-level policies operate on an abstract state space and are guided by external rewards. In contrast, low-level policies follow independent policies and receive separate rewards upon successful execution, which are hidden in the global reward. The superior performance of hierarchical reinforcement learning can be attributed to more efficient reward credit assignment and more efficient exploration. By abstracting action sequences, hierarchical reinforcement learning can learn value functions faster. This is because rewards can be accurately propagated further in the sequence. According to Driess et al., hierarchical reinforcement learning explores in a semantically meaningful and temporally extended way, while traditional reinforcement learning leads to over-exploration of nearby states.

[0007] Large language models (LLMs) have been gradually introduced into the field of reinforcement learning due to their powerful reasoning ability and extensive common sense base. The method of guiding reinforcement learning is to use large language models. The reasoning ability and extensive common sense of large language models can significantly improve the performance of reinforcement learning. Although there is no unified method to combine large language models with reinforcement learning, various strategies have been explored. Large language models are used to preprocess states, shape rewards, make direct decisions to improve sampling efficiency, generate world dynamics, or act as expert guidance. For example, LLMs are used to convert historical trajectories or natural language instructions into task-specific language to improve training efficiency. Large language models can act as implicit rewards or explicit rewards. Huang et al. used small-scale large language models for fine-tuning to speed up learning. Hao et al. encoded observations together with targets as serial inputs to enhance policy generalization. In current hierarchical reinforcement learning methods, low-level policies can be pre-defined by experts or based on prior knowledge. However, high-level policies in hierarchical reinforcement learning are usually learned from scratch, which limits their performance.

[0008] However, although the existing large language models perform well in areas such as natural language processing, they still have problems such as misalignment between input and output information and the dynamics of the environment, and inability to generate executable strategies when solving specific tasks such as classifying items, using keys to open doors, and other common household long decision-making chain tasks. This problem is caused by the difficulty of large language models to fully understand the surrounding environment based solely on text information. Current processing methods include multiple visits to the large language model to obtain feasible task plans, or introducing a feedback mechanism to inform the large language model that the task plan has failed and to reformulate the task plan. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an auxiliary hierarchical reinforcement learning training method based on a large language model, which utilizes the prior knowledge of the large language model and outputs recommended high-level strategies based on environmental information to assist the high-level strategies of hierarchical reinforcement learning, which can effectively reduce the action space and improve learning efficiency and task completion rate.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] In the first aspect, an auxiliary hierarchical reinforcement learning training method based on a large language model is disclosed, including:

[0012] The teacher agent and the student agent receive the same observations of the environment;

[0013] The teacher agent uses a large language model to give a guidance strategy based on the received environmental observation information. The guidance strategy includes the option probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategy is introduced into the training module.

[0014] The student agent gives a high-level strategy based on the environmental observation information. When the high-level strategy is passed to the corresponding low-level strategy option π ω When ω A series of actions will be generated according to the high-level strategy goals and the environment until the termination condition is reached or the maximum number of task steps is exceeded;

[0015] At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards. The accumulated sum of environmental rewards is regarded as the reward obtained by the current high-level action. The accumulated sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0016] The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the accumulated environmental rewards.

[0017] As a further technical solution, for the teacher agent:

[0018] First, define the high-level option set K s ,K t , K s represents the task skills that the current agent can choose, K t Represents the task objectives that the current agent can choose;

[0019] The large language model obtains the prompt text. The prompt text p∈P consists of two parts: p m and p o , where p m Represents the description of the current task, p o is the text description of the current environmental observation o;

[0020] The large language model will then output a high-level strategy a to guide the student agent based on p. L ;

[0021] The action decomposition module converts a L Decompose into and Represents the skill option distribution vector selected by the large language model, The target option distribution vector representing the selection of the large language model;

[0022] Use the same prompt text to ask the large language model n times, and count them separately and The probability distribution of options

[0023] As a further technical solution, for the student agent, including high-level strategies and low-level strategies, an option-based hierarchical reinforcement learning method is adopted for learning;

[0024] Among them, for high-level strategies, in K s Select the Skills option in K t The target of the current skill is selected in the , and the final high-level decision is a combination of the options in these two parts;

[0025] The low-level strategy is responsible for generating specific low-level actions and interacting with the environment based on the high-level strategy.

[0026] As a further technical solution, the student agent uses two independent fully connected networks and shares a same convolutional network for sensing the environment;

[0027] The two fully connected layers use the softmax function as the network output layer to obtain the student's high-level strategy and

[0028] Senior strategies through students and Get option index vector Combine them into high-level actions and pass them into low-level strategies.

[0029] As a further technical solution, the high-level strategy of the student agent and Training is performed by minimizing the following loss function:

[0030]

[0031] Among them, L RL (s) and L RL (t) means and The loss function of standard reinforcement learning is used. It represents the difference between the high-level strategy recommended by the teacher and the high-level strategy given by the student. This strategy difference can be calculated by KL divergence or cosine similarity. λ is the annealing parameter.

[0032] As a further technical solution, the annealing parameter λ is used to control the degree to which the teacher agent guides the student agent. The annealing parameter λ is set according to the following formula:

[0033]

[0034] At the beginning of training, λ is set to a larger value λ0 to ensure that the student agent gets more teacher guidance in the early stage of interaction with the environment. As the number of training steps i increases, the value of λ will decrease linearly at the decay rate k and become 0 after training to the i1th step.

[0035] In a second aspect, an auxiliary hierarchical reinforcement learning training system based on a large language model is disclosed, comprising: a teacher agent and a student agent, wherein the teacher agent comprises a large language model, the student agent comprises a high-level strategy and a low-level strategy, and the high-level strategy comprises a training module;

[0036] The teacher agent uses a large language model to give a guidance strategy based on the received environmental observation information, wherein the guidance strategy includes the option probability distribution of the skill option distribution vector and the target option distribution vector, and the guidance strategy is introduced into the training module;

[0037] The student agent gives a high-level strategy based on the environmental observation information. When the high-level strategy is passed to the corresponding low-level strategy option π ω When ω A series of actions will be generated according to the high-level strategy goals and the environment until the termination condition is reached or the maximum number of task steps is exceeded;

[0038] At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards. The accumulated sum of environmental rewards is regarded as the reward obtained by the current high-level action. The accumulated sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0039] The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the accumulated environmental rewards.

[0040] As a further technical solution, the teacher agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module;

[0041] The acquisition module is used to acquire environmental observation information and task text; the task text mainly includes task related information;

[0042] The translation module receives the environmental observation information and translates the environmental observation information into environmental information text;

[0043] The task text and the environment information text are combined into a prompt text, which is fed into a large language model to obtain a text action and fed into an action decomposition module;

[0044] The action decomposition module converts the text action into a skill option distribution vector and a target option distribution vector, wherein the index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model;

[0045] The option probability distribution acquisition module is configured to: repeat the action decomposition with the same prompt text to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector.

[0046] As a further technical solution, the student agent further includes a skill module and a target module, the skill module adopts a first fully connected network, and the target module adopts a second fully connected network;

[0047] The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting the skill.

[0048] The second fully connected network outputs multi-dimensional data, which constitutes the target selection strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting the target.

[0049] Select the skill and target with the highest probability according to the skill option strategy and target option strategy, regard the index value of the skill and target as a high-level action, and pass it to the low-level strategy;

[0050] The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The given skill index will activate the corresponding strategy skill and the given target index will determine the target of the skill interaction.

[0051] The low-level strategy will control the student agent to take a series of actions until the termination condition is reached or the maximum number of task steps is exceeded. At the same time, the student agent will receive environmental rewards for each action it performs. From r 1, arrive The cumulative sum It is regarded as a reward for the current high-level action. It will be fed into the training module for the student agent to learn from the environment.

[0052] One or more of the above technical solutions have the following beneficial effects:

[0053] The technical solution of the present invention is a hierarchical reinforcement learning framework assisted by a large language model. In this framework, the large language model acts as a teacher agent to guide the exploration of high-level strategies in hierarchical reinforcement learning. In the high-level strategy, actions are decomposed into skills and goals, thereby effectively reducing the action space and improving learning efficiency and task completion rate.

[0054] Since the text input length of a large language model is limited, the technical solution of the present invention implements a higher level of state abstraction for the input layer. Specifically, a state language encoder is developed, namely Figure 1The large language model is used to translate the input action into a translation module (e.g., “the agent picks up the key”), so that the large language model can output high-level actions that are consistent with expectations and have a certain degree of interpretability. The low-level policy must meet the following conditions: (1) The action input can be summarized in one sentence (e.g., “the agent picks up the key”). (2) The actions of the low-level policy must be consistent with the high-level actions and have a high success rate. During training, the large language model does not interact directly with the environment, but serves as an external reward to guide hierarchical reinforcement learning. The guidance weight of the large language model will gradually decrease during training. Hierarchical reinforcement learning shifts from learning from the large language model to learning through interaction with the environment, thereby surpassing the performance of the large language model.

[0055] The technical solution of the present invention proposes a novel high-level policy action embedding method. The high-level policy action is decomposed into skills and goals. The goal represents the interaction object (for example, door, key, etc.) selected by the high-level policy, and the interaction object will change with the change of the environment. The skill represents the interaction method with the target (for example, basic actions such as picking up and opening). This interaction method will remain consistent in different environments.

[0056] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0058] Figure 1 A schematic diagram of an auxiliary hierarchical reinforcement learning framework based on a large language model according to an embodiment of the present invention;

[0059] Figure 2 Example diagrams of four tasks used for experimental evaluation of the present invention;

[0060] Figure 3 This is a schematic diagram of an example of a task prompt of the present invention;

[0061] Figure 4 Schematic diagram of the success rate during the training of four tasks;

[0062] Figure 5 Schematic diagram of average rewards during the four tasks;

[0063] Figure 6 Schematic diagram of the training success rate of four tasks in the ablation experiment;

[0064] Figure 7 Schematic diagram of the average return of four task training in ablation experiments;

[0065] Figure 8 This is a schematic diagram of the overall process of implementing the example method of the present invention;

[0066] Fig. 9 This is a schematic diagram of the overall process of the method for processing data of a sample teacher agent according to an embodiment of the present invention;

[0067] Fig.10 The figure is a schematic diagram of the overall flow of the method for processing student agent data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0069] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0070] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0071] When dealing with long sequence decision tasks, frequently querying a large language model may result in excessive costs (if a commercial model is used) or longer reasoning times. In Yu's method, the agent must consult the large language model for every action, and the large language model takes several seconds to respond. This approach cannot respond in real time. Many approaches have been taken to solve this problem. Bang developed GPTcache to query semantic answers, which improved the response time by 2-10 times. Some methods have also been proposed to limit the use of large language models only in training to train optimal strategies using reinforcement learning, such as reward design and enhanced reasoning capabilities. In some schemes, the large language model is only used to output a rough plan, and the consultation delay can be reduced by selecting the low strategy required by the environment. Miao proposed a hierarchical knowledge representation framework that explores the integration of large language models in robotic operations, which enhances task understanding through multi-source knowledge updates and embedding-based reasoning techniques.

[0072] The reasoning ability of large language models needs to be improved in order to directly solve long sequence tasks. As pointed out by Valmeekam and Lou, large language models instinctively output reasoning trajectories in an autoregressive manner as a semantic logic chain of hypotheses. However, humans have an internal world model that iteratively explores various options and evaluates possible outcomes. In other words, large language models require a proper reward mechanism, a world model, and a balance between exploration and exploitation. Huang proposed a dialogue-like inner monologue to thoroughly observe the environment and exploit feedback from the environment. Chaining of ideas and adding examples is a widely used method to deconstruct complex problems and provide an interpretable window. ReAct was proposed to improve performance by integrating reasoning and action generation in an interleaved manner. In, simply adding "Let's think step by step" can also improve the zero-shot reasoning performance of large language models. Another widely used method is reinforcement learning with human feedback, such as MaxMin-RLHF with human preference data and InstructGPT with human labelers.

[0073] Aligning inputs and outputs with the environment is challenging because large language models need to accurately perceive environmental information given limited inputs. Specifically, large language models may generate incorrect actions that the environment cannot perform, or large language models may not accurately understand the dynamics of the environment, especially implicit interdependencies. The dynamics of the environment, especially implicit interdependencies. Reinforcement learning is used at the output layer to overcome the alignment defects of LLM, as the algorithm can surpass the limited success rate of LLM. In another scheme, the actor and critic share the same LLM, which is updated through PPO using a low-rank adapter. Another scheme outputs extensive high-level guidelines and uses PPO for fine-tuning, limiting misalignment. Some schemes use affordance functions to verify the executableness of LLM output actions. Some schemes use code generation as controlled outputs for robot control. Some schemes introduce MLP layers in the generation process and separate actions from values ​​for better accuracy. It also introduces embedded task instructions and visual observations through the encoder.

[0074] However, when it takes too long to complete the task and the environment rewards are sparse, reinforcement learning has difficulty learning the optimal strategy. Using prior knowledge can improve sample efficiency and training speed. Option-based hierarchical reinforcement learning divides actions into high-level actions and low-level actions. Each high-level action, or option, contains a series of low-level actions given by existing knowledge or strategies. The agent learns high-level action strategies to complete the entire task.

[0075] A new approach that leverages the reasoning power and prior knowledge of large language models to address the challenges of long-sequence decision tasks in sparse reward environments. There are still many problems in applying large language models to long-sequence decision tasks, such as the real-time response of large language models, the reasoning power of large language models in complex tasks, and the understanding of the environment by large language models.

[0076] This embodiment describes this problem as a hierarchical reinforcement learning based on options, which can perform temporary extended actions across multiple time steps. Option W is a triple (L w ,π w ,β w ), which represent the starting state condition of the option, the option strategy and the termination condition of the option respectively. Given the environment observation o, the high-level strategy selects the option strategy π w where o∈L w π w Multiple steps will be executed. Once the termination condition β is achieved w , the option strategy will stop and the high-level strategy will continue to select a new option strategy to execute.

[0077] Embodiment 1

[0078] This embodiment discloses an auxiliary hierarchical reinforcement learning training method based on a large language model, including:

[0079] like Figure 1 , 8 As shown in the figure, the framework consists of a teacher agent and a student agent. h , teachers and students will receive the same observations Students' intelligence experience based on observation Given a high-level decision, the low-level strategy generates a series of specific actions based on the guidance of the high-level decision. Wait until the lower-level strategy reaches the termination condition β w After that, the last observation Will be the next observation as a high-level strategy Make decisions. Each time the student agent's high-level strategy receives an observation At the same time, the teacher agent receives the same observations The guidance strategy, i.e. the recommended task skills and goals, will be given. The guidance strategy and environmental rewards will be introduced into the training module to realize the training of students' high-level strategy network.

[0080] See again the attached Figure 1As shown in the figure, the teacher agent provides recommended task skills and goals as high-level decision suggestions based on the text information from the environment and the current task prompt. The student agent operates within the option-based hierarchical reinforcement learning framework. It selects appropriate task skills and goals as high-level policies based on the environmental information. The low-level policies control the agent to perform specific actions and interact with the environment according to the high-level policies given by the student agent. In the training module, the high-level decisions provided by the teacher and the high-level decisions of the student are incorporated into the loss function by calculating the KL divergence, allowing the teacher to guide the student.

[0081] When calculating KL divergence, the specific calculation formula is:

[0082]

[0083] About the Teacher Agent:

[0084] First, define the high-level option set K s ,K t . K s represents the task skills that the current agent can choose, K t Represents the task goal that the current agent can choose. The prompt text p∈P obtained by LLM consists of two parts: p m and p o , where p m Represents the description of the current task, p o is a textual description of the current environmental observation o. Then LLM will output a high-level strategy a to guide students based on p L A hard-coded action decomposition module converts a L Decompose into and represents the skill option distribution vector selected by LLM, represents the distribution vector of target options selected by LLM. Considering that LLM does not interact directly with the environment, LLM cannot confirm the action a generated L is the best high-level decision for the current environment observation o. The sub-technical solution of this embodiment uses the same prompt text to query LLM n times, and counts and The probability distribution of options

[0085] About the Student Agent:

[0086] High-level strategy: The student agent structure adopts an option-based hierarchical reinforcement learning method. For the high-level strategy, a skill module is used to be responsible for s Select the skill option in K tThe final high-level decision is a combination of the options in these two parts, such as the blue key. This combination can reduce the action space of high-level decision options, especially when the student agent has a large number of skills and there are many objects in the environment that can be interacted with by the agent. Specifically, two independent fully connected networks are used, which share the same convolutional network for sensing the environment. The two fully connected layers use the softmax function as the network output layer to obtain the student high-level strategy and The final high-level strategy is passed through the strategy and Get option index vector Combine them into high-level actions and pass them into low-level strategies.

[0087] Low-level strategy: The low-level strategy is responsible for generating specific low-level actions and interacting with the environment based on the high-level strategy. This example uses multiple hard-coded strategy options based on human expert knowledge. These strategy options can also be replaced by pre-trained reinforcement learning networks. When the high-level strategy is passed to the corresponding low-level strategy option π ω When ω A series of actions will be generated based on high-level strategic goals and the environment Until the termination condition β is reached w Or exceeds the maximum number of task steps T low At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards From r 1, arrive The cumulative sum Considered as a reward for the current high-level action. It will be fed into the training module for the student agent to learn from the environment.

[0088] Training Modules:

[0089] High-Level Strategies of the Student Agent and Training is performed by minimizing the loss function.

[0090] The sub-technical solution of this embodiment introduces an annealing parameter λ to control the degree of guidance of the teacher agent to the student agent.

[0091] The whole learning process is summarized in Algorithm 1.

[0092]

[0093]

[0094] For more specific implementation examples, see the attached Fig. 9 As shown, the teacher agent data flow:

[0095] 1. The teacher agent obtains environmental observation information o from the environment, where o is an RGB image.

[0096] 2. The environmental observation information o is sent to the translation module, which is implemented by a pre-trained visual language model such as a clip model or a hard-coded program. The translation module translates the environmental observation information o into text information p o , such as "the agent observes that <red key><blue key><blue door> holds <nothing> and faces <nothing>"

[0097] 3. Task text p m It mainly includes task-related information such as: "In a closed two-dimensional grid room, the agent needs to open the door.

[0098] The rules are as follows:

[0099] (1) The agent can open a door only if it holds a key that matches the door’s color and is facing the door.

[0100] (2) The agent can pick up the key only when it is facing the key.

[0101] (3) The agent faces the object by walking towards it.

[0102] (4) The agent can only hold one object.

[0103] (5) The agent can perform the following operations: explore, go to <object>, pick up <object>, put down <object>, or open <object>.

[0104] (6) Based on the information provided, clarify what objects the agent sees and holds. Then infer what the agent can do in the current state.

[0105] (7)Finally, please choose the right action.

[0106] Example:

[0107] The agent sees <Blue Key><Red Key><Blue Door>, holds <None>, and faces <None>.

[0108] Objects seen and held by the agent: The agent sees the blue key, the red key, and the blue door, holds nothing, and faces nothing.

[0109] What the agent should do: The agent should go to the blue key.

[0110] Action: {Go to <Blue Key>}."

[0111] 4. Task text p m and environmental information text po The prompt text p will be combined together to form the prompt text p. The prompt text p will be sent to the large language model (gpt4), and then the output result a of the large language model will be obtained. L As in "Go to Blue Key".

[0112] 5. Place a L The action decomposition module is responsible for converting text action a L Converted into skill option distribution vector and the target option distribution vector The index of each skill option vector represents the skill selected by the large language model, such as "go to" The first position represents exploration, the second position represents pick up, the third position represents going to, the fourth position represents putting down, and the fifth position represents opening. The index of each target option distribution vector represents the interaction target selected by the large language model, such as "blue key" The first digit represents none, the second digit represents the red key, the third digit represents the blue key, the fourth digit represents the red door, and the fifth digit represents the blue door.

[0113] 6. Repeat steps 4 and 5 n times with the same p, and get and The probability distribution of options Right now:

[0114]

[0115] in, represents the skill option distribution vector obtained by querying LLM for the i-th time, Represents the target option distribution vector obtained by querying LLM for the i-th time. Guidance will be provided during the student network training, which will be covered in the student section.

[0116] 7. Send to the training module to assist the student agent training.

[0117] See attached Fig.10 As shown, the student agent data flow is:

[0118] 1. The student agent structure adopts an option-based hierarchical reinforcement learning method, which decomposes the student agent into high-level strategies and low-level strategies.

[0119] 2. Send the environmental observation o (image) to the convolutional network to sense the environment. There are three convolutional layers. The first layer outputs 16 feature maps using a 3x3 convolution kernel and padding 1 pixel on the edge. The second layer outputs 32 feature maps using a 3x3 convolution kernel and padding 1 pixel on the edge. The third layer outputs 64 feature maps using a 3x3 convolution kernel and padding 1 pixel on the edge. Finally, the feature map is flattened into a one-dimensional tensor. Use the Relu activation function.

[0120] 3. The output of the convolutional network is connected to two fully connected networks as the skill option network and the target option network. The skill option network consists of a 64-node fully connected network and a softmax activation function, outputting 5-dimensional data to form the skill option strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting that skill. The target option network consists of a 64-node fully connected network and a softmax activation function, which outputs multi-dimensional data (determined by the number of environmental targets) to form the target option strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting the target.

[0121] 4. According to and Select the skill and target with the highest probability, treat the index value of the skill and target as the high-level action, and pass it to the low-level strategy.

[0122] 5. The low-level policy has multiple hard-coded policy skills based on human expert knowledge, such as controlling the agent to explore the target in the environment, go to the target, pick up the target, put down the target, and open the target. The skill index given in step 4 will activate the corresponding policy skill and the target index given in step 4 will determine the target of the skill interaction.

[0123] 6. The low-level policy module will control the agent to produce a series of actions Until the termination condition β is reached w Or exceeds the maximum number of task steps T low At the same time, when the agent performs each action, it will receive environmental rewards From r1, to The cumulative sum Considered as a reward for the current high-level action. It will be fed into the training module for the student agent to learn from the environment.

[0124] Training module data flow:

[0125] 1. Make the environment a reward The training module is sent to the training module, and the student agent high-level network loss function L is obtained through the proximal policy optimization reinforcement learning algorithm (PPO).RL (s)+L RL (t);

[0126] 2. Send to the training module and calculate and The distribution difference of and The distribution difference here is measured using KL divergence, that is Put it into the loss function.

[0127] The two losses obtained in step 1 and step 2 are added together, and an annealing parameter λ is introduced to control the degree to which the teacher agent guides the student agent.

[0128] 3. At the beginning of training, set λ to a larger value λ0 to ensure that the student agent gets more teacher guidance in the early stage of interaction with the environment. As the number of training steps i increases, the value of λ will decrease linearly at a decay rate k and become 0 after training to step i1. At this point, the student agent will learn completely through interaction with the environment, further improving its robustness to specific tasks.

[0129]

[0130] 4. The final loss function is as follows:

[0131]

[0132] This loss function will be used to update the high-level policy network of the student agent.

[0133] Among them, L RL (s) and L RL (t) means and The loss function of standard reinforcement learning is used in both. This paper uses the proximal policy optimization (PPO) algorithm as the reinforcement learning algorithm. The agent learns from the environment by maximizing the expected return of the agent. Indicates the difference between the high-level strategies recommended by teachers and those given by students. and They represent the expectations of the teacher and student skill strategies and the target strategy under the environment observation o, respectively. This strategy difference can be calculated by KL divergence or cosine similarity.

[0134] During the training process, the training module will participate in the training of the high-level policy network, and in the testing phase, only the skill module and the target module will be used for testing.

[0135] We evaluate LLMHRL through various experiments and demonstrate that LLMHRL has significant advantages over other baseline methods.

[0136] This example evaluates LLMHRL using the MiniGrid environment. MiniGrid provides a grid-based visual environment where users can freely set the grid size and the agent's interactive objects, such as walls, doors, keys, etc. By interacting with objects, the state of the objects can be changed, such as turning a closed door into an open door. The agent navigates to a specific object and completes a series of interactions to complete a task, such as "put all the balls into the box". The agent's action space is discrete, including {move forward, turn left, turn right, open, close, stop}. The observation space is set to a local observable environment of size 3x3, and has a built-in memory function to record the blocks or goals that have been discovered.

[0137] This paper uses the MiniGrid framework to build four different tasks, with increasing difficulty, such as Figure 2 Specifically, more challenging tasks require longer decision chains with less tolerance for incorrect behavior in order to obtain rewards and reach goals.

[0138] Two-color door key: After generating two keys and a door, the agent needs to find and retrieve the correct color key for the door and use it to open the door.

[0139] Three-color door key: After generating three keys and a door, the agent needs to find and retrieve the correct color key for the door and use it to open the door.

[0140] Three-ball box: Three balls and three boxes are generated, and the agent needs to find and put the balls into the boxes of the correct color.

[0141] Four Doors to Goal: Generates a three-level environment where the agent needs to explore and find the correct key for the door, retrieve the key, open the door, and drop the key in the first and second levels. The environment also generates a fake door that opens without a key as a distractor. In the last level, the agent must find and reach the goal.

[0142] The above tasks use the same reward function:

[0143]

[0144] step finish Indicates the number of low-level strategy steps taken by the agent to complete the current task. max Represents the maximum number of low-level policy steps that the agent can execute. Such a reward function design can encourage the agent to complete the task in as few steps as possible.

[0145] Experimental setup:

[0146] Baselines: In comparative experiments, we use four baseline methods to evaluate LLMHRL.

[0147] LLM4Teach is an end-to-end reinforcement learning method guided by a large language model. The model learns task strategies based on the behavior suggestions given by the LLM. Similar to LLMHRL, this method also uses a large language model for strategy guidance. Therefore, this example uses LLM4Teach as a comparative experiment.

[0148] LLM uses the large language model to make high-level decisions and gives the high-level actions that the agent should take. The low-level policy module gives the specific actions that the agent needs to perform. Using this approach, the large language model must be kept online to make real-time decisions. Compared with this method, it is possible to evaluate whether LLMHRL effectively learns high-level policies from the large language model in the early stages of training and gradually surpasses the large language model in the later stages of training.

[0149] Hierarchial RL adopts an option-based hierarchical reinforcement learning approach. Each option represents a low-level policy consistent with LLMHRL. This paper compares hierarchical reinforcement learning with LLMHRL to evaluate whether LLMHRL effectively learns high-level policies from LLM.

[0150] RL(PPO) is an end-to-end reinforcement learning baseline method using a proximal policy optimization algorithm. This method will demonstrate the performance of typical reinforcement learning algorithms in sparse reward environments.

[0151] Large language model: This example uses Chat-GPT4 as the large language model to guide the high-level strategy of LLMHRL. Its powerful reasoning ability can provide LLMHRL with practical guidance strategies. The large language model method in the comparative experiment also uses this model to ensure fairness.

[0152] In order to improve the reasoning ability of the large language model, the sub-technical solution of this embodiment adopts a chain of text prompts, including task text and environment text. The task text provides the rules and specific examples of the current task to help the large language model understand the meaning of each high-level decision and the format of the output text. The environment text is provided based on the agent's observation of the environment. The large language model needs to provide appropriate high-level strategy guidance based on different environmental observation texts. The high-level decisions output by the large language model include an explanation of the current environment, the high-level decisions that the agent should take in the future, and the keywords of the corresponding decisions. Specific examples are as follows: Figure 3 shown.

[0153] Low-level strategy: The LLMHRL method constructs a low-level strategy library that includes skills and goals, including five different skills and twenty-four different goals. The execution of the low-level strategy is determined by the skills and goals.

[0154] The "Explore" skill automatically detects undiscovered blocks and walks towards them. It is worth noting that for the "Four Doors to a Goal" task, the "Explore" skill can only discover goals within its level. If the maximum number of steps is exceeded, this skill will automatically end. The "Go To" skill retrieves goal information from historical observations and performs goal positioning. If the goal has been discovered, the agent will walk towards the goal using the A* algorithm. The "Pick Up" skill determines whether the goal in front of the agent is consistent with the input goal. If correct, the goal will be picked up. The "Put Down" skill will only put down the goal if there is nothing in front of the agent and the agent is holding the same goal as the input goal. The "Open" skill will change the state of the input goal when it is within the agent's range. For example, a locked purple door requires an agent carrying a purple key to change its state to open.

[0155] Experimental results:

[0156] Table I

[0157] The success rate of the four tasks

[0158]

[0159] Table II

[0160] Average returns of four tasks

[0161]

[0162]

[0163] Table I and Figure 4 The success rate performance of the five methods deployed in four environmental tasks is shown. LLMHRL achieved the highest success rate in three of the four tasks. This shows that LLMHRL can effectively adapt to different tasks. At the same time, the convergence speed of the success rate training curve of LLMHRL is significantly faster than that of the other methods. This shows that LLMHRL fully receives the guidance of the large language model in the early stage of training, thereby quickly improving the task performance. Table II and Figure 5 The average return performance of the five methods deployed in four environment tasks is shown. The higher the average return, the fewer steps the agent uses to complete the task.

[0164] LLMHRL also achieved the highest average return in three of the four tasks. It is worth noting that the average return of LLMHRL exceeded that of the large language model in all four tasks. This shows that the performance of LLMHRL further improves during the interaction with the environment. In addition, as the difficulty of the task increases, LLMHRL can still maintain a high success rate and average return. The performance of the other methods decreases significantly.

[0165] LLM4Teach can quickly achieve good performance on most tasks and converge to a good policy after 10,000 steps. However, for difficult tasks, LLM4Teach performs poorly. This is because the actions learned by the student agent are low-level actions. This means that the agent needs to execute the correct sequence while mastering different skills. As the task decision chain becomes longer, it takes longer time steps for the student agent to learn the correct policy under the guidance of the large language model.

[0166] Traditional reinforcement learning cannot achieve good results regardless of whether it adopts a hierarchical structure. Without the help of a large language model, reinforcement learning cannot learn effective strategies regardless of whether it adopts a hierarchical structure. Since the rewards are sparse, the agent needs help to understand which target to interact with to get the desired result, so it cannot understand the environment. Even if a hierarchical structure is adopted to reduce the length of the decision chain, it is still not enough.

[0167] Based on a large language model, the agent can interact with the environment relatively successfully. However, the success rate is limited because the language model does not have task-specific knowledge and the output is unstable. This paper outlines several key issues with the output of large language models:

[0168] Unreadable policy: The large language model was asked to output one of the specified actions, but instead of outputting “explore”, it output “exploit”, which the low-level policy cannot understand.

[0169] Limited reasoning ability: When seeing “red key, blue door”, a large language model often outputs “pick up the red key” because it prefers to pick up any key regardless of its color. In contrast, when it expects to find the correct “blue key” to open the door, it should output “explore”.

[0170] Lack of task-specific knowledge: In MiniGrid, the agent can only pick up a target when it is not holding anything. But LLM keeps outputting “pick up the blue key” when it sees a “blue key” and is holding a “red key”. Instead, it should suggest “drop the red key”.

[0171] Ablation experiment:

[0172] Table III

[0173] Success rate and average return of ablation experiments

[0174]

[0175] LLMHRL decomposes high-level strategies into a combination of skills and goals. Whether this decomposition-combination method is effective requires experimental verification. Therefore, this paper conducts a set of ablation experiments to compare LLMHRL with a method that does not decompose but directly gives all high-level strategy options, called LLMHRL (NS).

[0176] As shown in Table III, Figure 6 and Figure 7 As shown in Figure 2, decomposing the action space into skills and goals can improve performance in complex tasks. Specifically, except for the two-color door key task, the average speed during training is faster and the success rate is higher than that of LLMHRL (NS). The reason behind this is that challenging tasks will expand the action space, resulting in a decrease in exploration efficiency. By dividing the action space into two parts, LLMHRL can effectively reduce the action space. This enables the high-level policy to better learn the guidance policy of the large language model.

[0177] This example combines option-based hierarchical reinforcement learning with a large language model to address the challenges of long sequence decision-making tasks in sparse reward environments. LLMHRL guides reinforcement learning by calculating the difference in the distribution of high-level policy actions between the large language model and reinforcement learning, and adds the difference to the loss function. High-level actions are divided into skill and goal components to reduce the number of selectable actions. Low-level actions consist of a skill library to complete different high-level decisions. Experiments show that LLMHRL can learn the policies of large language models faster and gradually learn better policies through interactions with the environment. However, LLMHRL also has some limitations, such as the options for high-level actions are fixed, which makes it challenging to deal with unknown goals.

[0178] The main contributions of LLMHRL are:

[0179] 1) In order to improve the performance of hierarchical reinforcement learning in long sequence tasks, a method of using a large language model to assist in training hierarchical reinforcement learning is proposed.

[0180] 2) To address the challenges of high-level policies in long sequence tasks with large action spaces, we introduce a method to decompose high-level actions into a combination of skills and goals.

[0181] Embodiment 2

[0182] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0183] Embodiment 3

[0184] The purpose of this embodiment is to provide a computer-readable storage medium.

[0185] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0186] Embodiment 4

[0187] See again the attached Figure 1 The purpose of the embodiment shown is to provide an auxiliary hierarchical reinforcement learning training system based on a large language model, including: a teacher agent and a student agent, the teacher agent includes a large language model, the student agent includes a high-level strategy and a low-level strategy, and the high-level strategy includes a training module;

[0188] The teacher agent uses a large language model to give a guidance strategy based on the received environmental observation information, wherein the guidance strategy includes the option probability distribution of the skill option distribution vector and the target option distribution vector, and the guidance strategy is introduced into the training module;

[0189] The student agent gives a high-level strategy based on the environmental observation information. When the high-level strategy is passed to the corresponding low-level strategy option π ω When ω A series of actions will be generated according to the high-level strategy goals and the environment until the termination condition is reached or the maximum number of task steps is exceeded;

[0190] At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards. The accumulated sum of environmental rewards is regarded as the reward obtained by the current high-level action. The accumulated sum of environmental rewards will be sent to the training module for the student agent to learn from the environment.

[0191] The training module will use the loss function to update the high-level policy network of the student agent.

[0192] The teacher agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module;

[0193] The acquisition module is used to acquire environmental observation information and task text; the task text mainly includes task related information;

[0194] The translation module receives the environmental observation information and translates the environmental observation information into environmental information text;

[0195] The task text and the environment information text are combined into a prompt text, which is fed into a large language model to obtain a text action and fed into an action decomposition module;

[0196] The action decomposition module converts the text action into a skill option distribution vector and a target option distribution vector, wherein the index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model;

[0197] The option probability distribution acquisition module is configured to: repeat the action decomposition with the same prompt text to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector.

[0198] The student agent also includes a skill module and a target module. The skill module uses a first fully connected network, and the target module uses a second fully connected network.

[0199] The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting the skill.

[0200] The second fully connected network outputs multi-dimensional data, which constitutes the target selection strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting the target.

[0201] Select the skill and target with the highest probability according to the skill option strategy and target option strategy, regard the index value of the skill and target as a high-level action, and pass it to the low-level strategy;

[0202] The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The given skill index will activate the corresponding strategy skill and the given target index will determine the target of the skill interaction.

[0203] The low-level strategy will control the student agent to take a series of actions until the termination condition is reached or the maximum number of task steps is exceeded. At the same time, the student agent will receive environmental rewards for each action it performs. From r 1, arrive The cumulative sum It is regarded as a reward for the current high-level action. It will be fed into the training module for the student agent to learn from the environment.

[0204] Embodiment 5

[0205] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0206] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0207] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0208] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. Auxiliary hierarchical reinforcement learning training method based on a large language model, characterized by: include: The teacher agent and the student agent receive the same observations of the environment; The teacher agent uses a large language model to give a guidance strategy based on the received environmental observation information. The guidance strategy includes the option probability distribution of the skill option distribution vector and the target option distribution vector. The guidance strategy is introduced into the training module. The student agent gives a high-level strategy based on the environmental observation information. When the high-level strategy is passed to the corresponding low-level strategy option hour, A series of actions will be generated according to the high-level strategy goals and the environment until the termination condition is reached or the maximum number of task steps is exceeded; At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards. The accumulated sum of environmental rewards is regarded as the reward obtained by the current high-level action. The accumulated sum of environmental rewards will be sent to the training module for the student agent to learn from the environment. The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the accumulated environmental rewards.

2. The auxiliary hierarchical reinforcement learning training method based on a large language model as claimed in claim 1, characterized in that: For the teacher agent: First define the high-level options set , , Represents the task skills that the current agent can choose, Represents the task objectives that the current agent can choose; The large language model obtains the prompt text, the prompt text p P consists of two parts: and ,in Represents the description of the current task. is the text description of the current environmental observation o; The large language model will then output a high-level strategy to guide the student agent based on p. ; The action decomposition module will Decompose into and , Represents the skill option distribution vector selected by the large language model, The target option distribution vector representing the selection of the large language model; Use the same prompt text to ask the large language model n times, and count them separately and The probability distribution of options , .

3. The auxiliary hierarchical reinforcement learning training method based on a large language model as claimed in claim 1, characterized in that: For the student agent, including high-level strategies and low-level strategies, the option-based hierarchical reinforcement learning method is adopted for learning; Among them, for high-level strategies, Select the Skills option in The target of the current skill is selected in the , and the final high-level decision is a combination of the options in these two parts; The low-level strategy is responsible for generating specific low-level actions and interacting with the environment based on the high-level strategy.

4. The auxiliary hierarchical reinforcement learning training method based on a large language model as claimed in claim 1, characterized in that: The student agent uses two independent fully connected networks and shares a same convolutional network for sensing the environment; The two fully connected layers use the softmax function as the network output layer to obtain the student's high-level strategy and ; Senior strategies through students and Get option index vector , , and combine them into high-level actions and pass them into low-level strategies.

5. The auxiliary hierarchical reinforcement learning training method based on a large language model as claimed in claim 1, characterized in that the student The agent's high-level strategy and Training is performed by minimizing the following loss function: in, and express and The loss function of standard reinforcement learning is used. It represents the difference between the high-level strategy recommended by the teacher and the high-level strategy given by the student. This strategy difference can be calculated by KL divergence or cosine similarity. is the annealing parameter.

6. The auxiliary hierarchical reinforcement learning training method based on a large language model as claimed in claim 5, characterized in that: The annealing parameters Used to control the degree to which the teacher agent guides the student agent, annealing parameters The setting is based on the following formula: At the beginning of training, set For a larger value To ensure that the student agent gets more teacher guidance in the early stage of interaction with the environment, as the number of training steps i increases, The value of will decrease linearly with the decay rate k and will be After the step it becomes 0.

7. Auxiliary hierarchical reinforcement learning training system based on large language model, characterized by: include: A teacher agent and a student agent, wherein the teacher agent includes a large language model, and the student agent includes a high-level strategy and a low-level strategy, and the high-level strategy includes a training module; The teacher agent uses a large language model to give a guidance strategy based on the received environmental observation information, wherein the guidance strategy includes the option probability distribution of the skill option distribution vector and the target option distribution vector, and the guidance strategy is introduced into the training module; The student agent gives a high-level strategy based on the environmental observation information. When the high-level strategy is passed to the corresponding low-level strategy option hour, A series of actions will be generated according to the high-level strategy goals and the environment until the termination condition is reached or the maximum number of task steps is exceeded; At the same time, each action output by the low-level strategy will interact with the environment and obtain environmental rewards. The accumulated sum of environmental rewards is regarded as the reward obtained by the current high-level action. The accumulated sum of environmental rewards will be sent to the training module for the student agent to learn from the environment. The training module updates the high-level policy network of the student agent using a loss function based on the received guidance policy and the accumulated environmental rewards.

8. The auxiliary hierarchical reinforcement learning training system based on a large language model as claimed in claim 7, characterized in that: The teacher agent includes an acquisition module, a translation module, an action decomposition module, and an option probability distribution acquisition module; The acquisition module is used to acquire environmental observation information and task text; the task text mainly includes task related information; The translation module receives the environmental observation information and translates the environmental observation information into environmental information text; The task text and the environment information text are combined into a prompt text, which is fed into a large language model to obtain a text action and fed into an action decomposition module; The action decomposition module converts the text action into a skill option distribution vector and a target option distribution vector, wherein the index of each skill option vector represents the skill selected by the large language model, and the index of each target option distribution vector represents the interaction target selected by the large language model; The option probability distribution acquisition module is configured to: repeat the action decomposition with the same prompt text to obtain the option probability distribution of the skill option distribution vector and the target option distribution vector; Preferably, the student agent further includes a skill module and a target module, the skill module adopts a first fully connected network, and the target module adopts a second fully connected network; The first fully connected network outputs multi-dimensional data, which constitutes the skill selection strategy output by the student agent. Each dimension represents a skill, and the data in each dimension represents the probability of selecting the skill. The second fully connected network outputs multi-dimensional data, which constitutes the target selection strategy output by the student agent. Each dimension represents a target, and the data in each dimension represents the probability of selecting the target. Select the skill and target with the highest probability according to the skill option strategy and target option strategy, regard the index value of the skill and target as a high-level action, and pass it to the low-level strategy; The low-level strategy has multiple hard-coded strategy skills based on human expert knowledge. The given skill index will activate the corresponding strategy skill and the given target index will determine the target of the skill interaction. The low-level strategy will control the student agent to take a series of actions until the termination condition is reached or the maximum number of task steps is exceeded. At the same time, the student agent will receive environmental rewards for each action it performs. ,from arrive The cumulative sum It is regarded as a reward for the current high-level action. It will be fed into the training module for the student agent to learn from the environment.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

Images