Robot operation skill learning method for generating rewards based on large language model

Generating reward functions through large language models solves the problem of difficulty in designing reward functions in complex operating environments, realizes fast and efficient skill learning of robots, reduces the dependence on human expertise and the complexity of manual programming.

CN119067105APending Publication Date: 2024-12-03NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411253617.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In complex operating environments, it is difficult to design appropriate reward functions to guide robots to quickly learn tasks, especially due to the need for human expertise and high frequency of trials and errors, resulting in inefficiency and poor quality of learning.

Method used

Large language models such as GPT-4 are used to generate reward functions, and by designing prompt paradigms, generating reward functions, integrating them into reinforcement learning algorithms, and optimizing them based on human feedback, reducing the dependence on human expertise and manual programming complexity.

Benefits of technology

It improves the efficiency and quality of robot learning tasks, shortens training cycles, reduces error rates, and enables robots to master skills faster and more accurately in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067105A_ABST
    Figure CN119067105A_ABST
Patent Text Reader

Abstract

The robot operation skill learning method for generating rewards based on the large language model comprises the steps of designing a prompt normal form of a large language model generation reward function, and obtaining prompt content; generating a reward function for the prompt content by using a large language model; integrating the reward function into a reinforcement learning algorithm for training, and detecting and correcting grammar errors and / or errors during operation; human user feedback is obtained, codes of the reward function are improved through the human user feedback by the large language model, and the improved reward function is used for achieving skill learning of the robot. According to the method, the accurate reward function is automatically generated by using the large language model, so that the robot is guided to optimize the behavior through self-learning; according to the method, the dependence on human professional knowledge is reduced, and the complexity and error rate of manual programming are reduced, so that the robot can realize higher learning speed and higher operation precision in a wider application scene, and the task learning efficiency and quality of the robot are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of robotics and artificial intelligence technology, and specifically relates to a robot operation skill learning method based on generating rewards based on a large language model. Background Art

[0002] Skill learning has always been a major challenge in the development of robotics. Especially in tasks involving complex operations or action patterns, traditional learning methods are often limited by low learning efficiency and poor quality. To this end, researchers and engineers have developed a variety of methods to accelerate learning, such as directly transferring human knowledge and strategies to robots through human-computer interaction, enabling them to quickly master core skills. Although these methods are effective, they still face the problem of requiring intensive human participation, which is particularly prominent in commercial applications that need to be deployed quickly.

[0003] Reinforcement learning is an algorithm that learns the optimal strategy through the interaction between the agent and the environment, thereby obtaining the optimal solution in a complex environment. In reinforcement learning, the goal of the agent is formally represented as a reward signal. At each moment, the reward signal is a real-valued signal. The goal of the agent is to maximize the accumulated total reward. Using the total reward as a formal description of the learning goal is one of the most unique features of reinforcement learning that distinguishes it from other types of artificial intelligence learning. The reward function plays a vital role in reinforcement learning. It is used to evaluate the performance of the agent in a specific action. Therefore, the reasonable design and optimization of the reward function can directly affect the performance and convergence speed of the reinforcement learning algorithm. In fact, when using reinforcement learning to train agents, most of the time the agent cannot get rewards. It is very difficult to train agents without rewards. If the rewards in the environment are very sparse, the problem of reinforcement learning becomes very difficult, but humans can learn on very sparse rewards. People usually do not get any rewards or penalties. However, people can still take a variety of actions. Therefore, a high-performance artificial intelligence agent should be able to learn how to interact with the environment even with sparse rewards.

[0004] Reinforcement learning usually relies on a reward function to guide the learning process, which is usually gradually adjusted during the training process. However, in many cases, designing a suitable reward function is very difficult because many different factors need to be taken into account, such as the complexity of the task, the dynamics of the environment, the feasibility of the strategy, etc. For complex robotic skills, designing a suitable reward function to guide the robot to learn the task quickly requires rich expertise, which may be difficult in practice. Therefore, how to make reinforcement learning methods solve high-dimensional problems with fewer trial and error is the key to their widespread application in the field of robotics.

[0005] Providing intermediate rewards that encourage progress toward a goal can reduce unnecessary exploration, a concept known as reward shaping. Reward shaping is a method to address sparse rewards. Its purpose is to improve the performance of reinforcement learning by modifying the original reward function to make it more consistent with domain knowledge. Specifically, reward shaping adds an additional reward component that is designed based on domain knowledge, which can guide the agent to take actions that are more effective in specific tasks. Reward shaping addresses the problem of how to effectively incorporate domain knowledge into the reward function in reinforcement learning, so that the agent can better adapt to specific tasks and environments. However, designing intermediate rewards is challenging, especially for non-experts, because it involves many factors such as task complexity, environment dynamics, and policy feasibility. Designing an appropriate reward function to guide a robot to quickly master a task requires extensive expertise and is challenging in practice. Therefore, how to shape rewards with less trial and error is the key to accelerating robot skill learning. Summary of the invention

[0006] In order to overcome the above problems, this application proposes a robot operation skill learning method based on reward generation by a large language model, which uses the large language model GPT-4 to automatically generate an accurate reward function, thereby guiding the robot to optimize its behavior through self-learning. It not only reduces the dependence on human expertise, but also reduces the complexity and error rate of manual programming, enabling robots to achieve faster learning speeds and higher operation accuracy in a wider range of application scenarios, especially in complex and changeable operating environments, effectively improving the efficiency and quality of robot learning tasks.

[0007] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0008] A robot operation skill learning method based on generating rewards with a large language model, comprising:

[0009] Design a prompt model for the large language model to generate reward functions and obtain prompt content;

[0010] Use a large language model to generate a reward function based on the prompt content;

[0011] Integrate reward functions into reinforcement learning algorithm training to detect and correct syntax errors and / or runtime errors;

[0012] Obtain human user feedback, and use the large language model to improve the reward function code using human user feedback. The improved reward function is used to implement the robot's skill learning.

[0013] Design a prompt model for the large language model to generate a reward function and obtain prompt content, including:

[0014] Describe the task in short phrases or sentences;

[0015] Based on the task description, define the abstract variables related to the task;

[0016] Set reward standards to guide the large language model to decompose the complex main task into several subtasks.

[0017] The large language model formulates reward functions for each subtask separately, allocates rewards or penalties according to the degree of successful completion of each subtask, integrates the reward functions of all subtasks into a comprehensive reward mechanism, and obtains the reward function of the main task.

[0018] The GPT-4 model is used as the large language model.

[0019] Integrate reward functions into reinforcement learning algorithm training to detect and correct syntax errors and / or runtime errors, including:

[0020] The reward function is embedded and integrated into the SAC algorithm for verification. During the verification process, if errors are detected, the code of the reward function is adjusted to enhance the accuracy and functionality of the reward function.

[0021] After each training cycle of the reinforcement learning algorithm is completed, suggestions for improving the code of the reward function proposed by human users based on strategy visualization are obtained and provided to the large language model for improving the code of the reward function.

[0022] A robot operation skill learning method based on generating rewards with a large language model, comprising:

[0023] Instruction acquisition module: Design a prompt model for the large language model to generate a reward function and obtain the prompt content;

[0024] Reward function generation module: uses a large language model to generate a reward function for the prompt content;

[0025] Reward function optimization module: Integrate the reward function into the reinforcement learning algorithm for training, detect and correct syntax errors and / or runtime errors; obtain human user feedback, and use the human user feedback to improve the reward function code by the large language model. The improved reward function is used to implement the robot's skill learning.

[0026] An electronic device, comprising:

[0027] at least one processor;

[0028] and, a memory communicatively coupled to the at least one processor;

[0029] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a robot operation skill learning method based on generating rewards based on a large language model.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a robot operation skill learning method based on generating rewards based on a large language model.

[0031] Compared with the prior art, the beneficial effects of this application are:

[0032] This application uses a combination of large language models and reinforcement learning to reduce reliance on professionals and reduce subjective bias and error rates in the design process. Due to the automatic generation and optimization of the reward function, the robot's training cycle is shortened, making the deployment of robot skills faster, which is particularly important for industries that require rapid response and skill updates, such as manufacturing and logistics.

[0033] This application integrates advanced natural language processing capabilities to interpret complex task descriptions and autonomously formulate reward strategies to guide robot behavior, aiming to improve the efficiency and effectiveness of robot training procedures, thereby reducing reliance on extensive manual programming and predefined data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart of the method of this application;

[0035] Figure 2 For this application, Metaworld door-open-v2 scene;

[0036] Figure 3 This is a comparison chart of the training reward value curve for this application;

[0037] Figure 4 This is a comparison chart of the training success rate curve for this application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clear, the present application is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0039] Large language models (LLMs) such as GPT-3 and GPT-4 models not only understand the nuances of language, but also acquire rich semantic knowledge through pre-training on large text datasets. This enables the model to transform complex task descriptions into detailed, text-based sequences of operations through natural language processing. In particular, this approach can automatically generate reward functions, greatly simplifying the reward design process, reducing the possibility of human error, and speeding up the response of the entire system. Large language models, through extensive knowledge and data learning, are able to provide more objective and comprehensive reward design solutions and reduce the impact of human errors.

[0040] Large language models (LLMs) allow for effective output using a small amount of human prompts and are remarkably adaptable to new contexts. This capability has also been extended to the field of robotics, but due to limited scene-related training data, existing large language models are difficult to directly output low-level robot commands or require domain-specific training. The reward function can serve as an interface to bridge the gap between natural language and low-level robot actions, making it easier to leverage the capabilities of general large language models.

[0041] The robot operation skill learning method based on the large language model to generate rewards in this application is divided into two parts. The method block diagram is as follows Figure 1 As shown:

[0042] 1. Design of a prompting model for generating reward functions

[0043] In order to generate reward functions using LLM, this application uses the Chain-of-Thought (CoT) technique to enhance LLM's ability to solve complex reasoning tasks. This technique involves constructing prompts to include a series of intermediate reasoning steps, thereby promoting more comprehensive analysis and better solution generation. The design process includes four different stages:

[0044] (1) Task description: Use concise phrases or short sentences to describe the task.

[0045] (2) Define abstract variables related to the task: Based on the task description, accurately define the key variables that generate rewards in a specific environment.

[0046] (3) Setting reward criteria: First, decompose the complex main task into simpler and more specific subtasks. This process is based on the description of the task and the specific goals to be achieved. The purpose of decomposing tasks is to simplify a large, complex problem into several small, manageable tasks, each of which focuses on a specific aspect or step of the original task.

[0047] (4) Verification of the reward function: After the initial reward code is generated, it is integrated into the SAC algorithm robot door opening environment for verification to detect and correct any syntax or runtime errors.

[0048] 2. Improve reward code based on human feedback

[0049] In actual applications, human operators can monitor the robot’s learning progress and performance and provide feedback based on actual operation conditions. These feedbacks are processed again by LLM to optimize and adjust the reward function to make it more accurately meet the task requirements and operating environment.

[0050] Example 1

[0051] This application discloses a method for automatically generating reward functions using large language models (LLMs) to teach robots to perform tasks. The method aims to improve the efficiency and quality of robot learning in complex and dynamic environments. The implementation steps are as follows:

[0052] Step 1: Statement of task description.

[0053] When defining a task, use precise and concise phrases or sentences to set a clear context for the large language model to work in. For example, using "To make the manipulator open the door" directly states the goal of the task. However, using a statement like "Ensure the door is opened by the manipulator" may cause the LLM to focus on other potential aspects of the task. This example highlights the importance of choosing the wording when designing prompts, because subtle differences in wording not only affect the model's understanding of the task, but may also change the way it generates the reward function, thereby affecting the effectiveness and speed of learning. Therefore, using accurate and consistent language to describe the task is crucial to effectively using large language models for task learning.

[0054] Step 2: Based on the first step, define abstract variables related to the task.

[0055] Based on the task description, the key variables related to reward generation in a specific environment need to be precisely defined. This covers the state of the task objects, such as the relative position of the robot arm and the door handle, the door opening angle, and the specific criteria for successful door opening. In addition, the programming libraries required (such as NumPy, etc.) should be clearly specified to support data processing and model calculations. The accurate definition of these variables is crucial because they enable the large language model (LLM) to accurately evaluate the environment and understand the dynamic interactions required to complete the task. The accuracy of the variable definition directly affects the effectiveness of the reward function. Inaccurate definition may lead to the generation of a reward function that does not conform to the actual situation, thereby misleading the learning process of the agent. Therefore, ensuring that these variables are accurately identified and defined is essential to optimizing the learning and performance of the agent.

[0056] Step 3: Based on the first two steps, set reward standards.

[0057] Decomposing a complex main task into simpler, more specific subtasks is performed based on a detailed description of the task and the specific goals it aims to achieve. The purpose of task decomposition is to simplify a large and complex problem into several small, manageable subtasks, each of which focuses on handling a specific aspect or step of the main task. By analyzing the natural language description of the task, the large language model is able to grasp the core goals and key requirements of the task, including identifying the main purpose of the task, the key objects involved (e.g., the objects the robot manipulates), and the specific actions that must be performed. Based on a deep analysis of the task, the LLM breaks the main task into a series of subtasks. For example, in the task of the robot opening a door, the subtasks may include locating the door handle, grasping the door handle, turning the door handle, and pulling the door open. For each subtask, the LLM develops a dedicated reward function that assigns rewards or penalties based on the degree of successful completion of the subtask. For example, successfully grasping the door handle may result in a positive reward, while failure to perform it correctly may result in a penalty. Finally, the LLM combines the reward functions of all subtasks into a comprehensive reward mechanism that guides the agent at every step from the start of the task to the completion.

[0058] The prompt content is:

[0059] Q:

[0060] 1.Python function for reward shaping in a reinforcement learning task

[0061] 2.The task involves a robot with an articulated arm tasked withopening a hinged door in a simulated environment.

[0062] 3.The observation is a numerical array where:

[0063] -The first three values represent the robot’s hand position.

[0064] -The next three values represent the door’s position,which requiresadding a fixed offset to be calculated.

[0065] -’theta’is the angle of the door in radians,where 0 is closed and pi / 2 is fully open,provided in an’info’dictionary.

[0066] 4.The reward function,named’reward_shaping’,should take an’info’dictionary containing’obs’and’theta’as arguments.

[0067] 5.The’obs’includes the full observation array and’theta’is thecurrent angle of the door.

[0068] 6.Criteria for the reward:

[0069] -A higher reward for when the robot’s hand is close to the door,witha specified’distance_threshold’for this proximity reward.

[0070] -Incremental positive rewards as the door opens wider, with a bonuswhen the angle is less than a'door_open_angle'threshold.

[0071] -Efficiency is encouraged with a small reward for each timesteptaken, promoting the robot to open the door promptly.

[0072] 7.Please provide the implementation of the'calculate_reward'function based on the above description, using numpy for any mathematical operations.

[0073] The code for the reward function is:

[0074]

[0075]

[0076] Step 4: Based on the first three steps, verify the reward function.

[0077] Once the initial reward function code is generated, the code undergoes an exhaustive validation process to identify and correct any potential syntax errors or runtime anomalies. This validation phase is critical to ensure that the developed reward function is not only theoretically correct, but also performs effectively in real-world applications. During the validation process, once errors are detected, code adjustments are made to enhance the accuracy and functionality of the reward function. Through this iterative improvement process, it is ensured that the final reward function can accurately meet the requirements of the specific task while maintaining perfect consistency with the environmental dynamics and the operating capabilities of the robotic system. This step not only improves the reliability of the system, but also ensures efficiency and safety during operation.

[0078] Step 5: Building on the previous four steps, improve the reward code based on human feedback.

[0079] Humans can help refine and adjust the learning process by providing nuanced assessments that automated systems may not fully capture. Subjective assessments provided by humans can help adjust reward mechanisms rather than relying solely on traditional, objective performance metrics. In initial interactions, humans often provide instructions that may lack precise details, which can make it challenging to create optimal reward mechanisms. For example, in a task instructing a robot to open a door, the user may not specify whether to pull the handle or push the door itself. While both actions can open the door, pulling the handle is often more ideal because it reduces potential damage to furniture and the robot itself.

[0080] In order to solve the problem of suboptimal reward functions caused by imprecise instructions, this application seeks direct feedback from users to enhance the generated rewards. After each reinforcement learning training cycle, users can provide feedback by watching videos showing how the current policy performs the task. Based on these observations, users can make specific adjustment suggestions, emphasize the advantages and disadvantages of task execution, and then feed back the code fine-tuning suggestions to the large language model. For example, if the robot arm falls into a local optimal state when approaching a door handle, an appropriate increase in the penalty can guide the robot arm to approach the target more accurately; conversely, if the robot arm gets too close and collides with the door handle, an appropriate increase in the safety reward can ensure the safe completion of the task. These feedbacks will be integrated into subsequent reward functions to generate a more accurate and effective reward mechanism. This iterative process ensures that the reward system can be continuously improved and optimized based on direct evaluations by users, and ultimately forms a reward system that not only meets actual operational needs but also effectively promotes robot learning and performance improvement.

[0081] Example 2

[0082] In order to prove the effectiveness of the robot operation task learning method based on large language model generation reward proposed in this application, the method proposed in this application is simulated and verified in the door-open-v2 environment of Metaworld. The reinforcement learning algorithm selected is Soft Actor-Critic (SAC). During the training process, the learning rate of the actor and the critic is set to 0.0003, the total number of training steps is set to 1,000,000 steps, and the batch size is 512. The double Q-evaluator (double_q_critic) is configured with a hidden layer dimension of 256 and a hidden depth of 3 layers, and the diagonal Gaussian actor (diag_gaussian_actor) is also set with the same hidden layer dimension and depth. This series of operations will ensure that each seed value is executed repeatedly to test and verify the effect of the algorithm. Figure 2 For the simulation environment scene.

[0083] The experimental results are compared with the environmental benchmark reward value. Figure 3 and Figure 4 As shown, the environment baseline reward refers to a dense, expert-written reward function explicitly provided in the environment, from Figure 3 It can be seen from the figure that the reward function generated by the method of the present application and the environmental benchmark reward function have similar convergence speeds in the training of the door opening task. Figure 4 As can be seen, there is no significant difference in the training success rate of the two methods, indicating that the reward function generated by the large language model LLM is very close to the version written by experts in terms of the effectiveness of training the agent.

Claims

1. A robot operation skill learning method based on large language model to generate rewards, It is characterized in that include: Design a prompt model for the large language model to generate reward functions and obtain prompt content; Use a large language model to generate a reward function based on the prompt content; Integrate reward functions into reinforcement learning algorithm training to detect and correct syntax errors and / or runtime errors; Obtain human user feedback, and use the large language model to improve the reward function code using human user feedback. The improved reward function is used to implement the robot's skill learning.

2. The robot operation skill learning method based on generating rewards based on a large language model according to claim 1, It is characterized in that Design a prompt model for the large language model to generate a reward function and obtain prompt content, including: Describe the task in short phrases or sentences; Based on the task description, define the abstract variables related to the task; Set reward standards to guide the large language model to decompose the complex main task into several subtasks.

3. The robot operation skill learning method based on generating rewards based on a large language model according to claim 2, It is characterized in that The large language model formulates reward functions for each subtask separately, allocates rewards or penalties according to the degree of successful completion of each subtask, integrates the reward functions of all subtasks into a comprehensive reward mechanism, and obtains the reward function of the main task.

4. The robot operation skill learning method based on generating rewards based on a large language model according to claim 1, It is characterized in that The GPT-4 model is used as the large language model.

5. The robot operation skill learning method based on generating rewards based on a large language model according to claim 1, It is characterized in that Integrate reward functions into reinforcement learning algorithm training to detect and correct syntax errors and / or runtime errors, including: The reward function is embedded into the SAC algorithm for verification. During the verification process, if errors are detected, the code of the reward function is adjusted to enhance the accuracy and functionality of the reward function.

6. The robot operation skill learning method based on generating rewards based on a large language model according to claim 1, It is characterized in that After each training cycle of the reinforcement learning algorithm is completed, suggestions for improving the code of the reward function proposed by human users based on strategy visualization are obtained and provided to the large language model for improving the code of the reward function.

7. A robot operation skill learning method based on large language model to generate rewards, It is characterized in that include: Instruction acquisition module: Design a prompt pattern for the large language model to generate a reward function and obtain the prompt content; Reward function generation module: uses a large language model to generate a reward function for the prompt content; Reward function optimization module: Integrate the reward function into the reinforcement learning algorithm for training, detect and correct syntax errors and / or runtime errors; obtain human user feedback, and use the human user feedback to improve the reward function code by the large language model. The improved reward function is used to implement the robot's skill learning.

8. An electronic device, It is characterized in that include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robot operation skill learning method based on generating rewards based on a large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the robot operation skill learning method based on generating rewards based on a large language model as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Multi-view video reward mechanism learning framework and construction method thereof

    CN119830993A

  • Digital twin intelligent manufacturing production line optimization system, method and computer program product

    CN121635177A

  • Control learning device and control learning method

    JP7940783B1