Agent reinforcement learning reward function optimization method based on vlm closed-loop feedback

Abstract

The application discloses an agent reinforcement learning reward function optimization method based on VLM closed-loop feedback. First, an initial reward function code and a parameter configuration file are generated according to natural language task instructions by using a large language model; then, the reward function is used to drive a robot arm to perform reinforcement learning strategy training, and a video stream of task execution is recorded and the task success rate is calculated. When the preset failure trigger condition is met, the parameter update is paused. An algorithm based on perceptual hashing and time sequence hybrid weight is used to extract a key frame sequence from the video, a visual language model is used for semantic diagnosis of the key frame, and improvement suggestion is fed back to the large language model, and the reward function is optimized and hot reloaded into the training process. By introducing the visual language model to construct an automatic feedback correction closed loop, the problem that the large language model generates a reward function lacking physical verification and leading to failure is solved, and the training efficiency and success rate of the agent for complex operation tasks are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of automation control technology, and relates to intelligent control and artificial intelligence technology, specifically to an agent reinforcement learning reward function optimization method based on VLM closed-loop feedback. Background Technology

[0002] Deep Reinforcement Learning (DRL) has become a mainstream method for solving complex robotic tasks such as grasping, contact assembly, and object manipulation. The core mechanism of reinforcement learning lies in the agent maximizing accumulated rewards through interaction with the environment. Therefore, the design of the reward function determines whether the agent can converge quickly and the performance of the final policy.

[0003] Traditional reward function design relies solely on binary signals of task success or failure, resulting in sparse rewards. This leads to an excessively large exploration space in long-term tasks, making it difficult for the agent to obtain effective feedback and resulting in extremely low training efficiency. Designing dense rewards typically requires experts to manually adjust the weights of various reward components, such as distance penalties, pose alignment, and contact forces, based on experience. This process is time-consuming, laborious, and difficult to generalize. Some methods attempt to use Large Language Models (LLMs) to directly generate reward function code based on natural language instructions. However, LLMs lack the ability to perceive the physical world, and the generated code often contains linguistic illusions or logical flaws, causing the trained policy to completely fail in physical simulations, and lacking an automatic correction mechanism.

[0004] Therefore, it is urgent to optimize the existing reinforcement learning reward function for robotic arms and automatically close the loop based on the observation results. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an agent reinforcement learning reward function optimization method based on VLM closed-loop feedback. It introduces a Vision-Language Model (VLM) to perform visual semantic diagnosis of failed cases during training and feeds physical facts back to the LLM for code-level correction. This effectively suppresses the code generation illusion of the LLM, ensuring that the reward function possesses both logical validity and physical realism, thereby achieving fully automated policy learning and improving the stability of model training and task success rate.

[0006] The method for optimizing the reward function of agent reinforcement learning based on VLM closed-loop feedback includes the following steps:

[0007] Step 1: Obtain the natural language task instructions and environmental abstraction description input by the user; use the Large Language Model (LLM) to infer and generate the initial reward function code and its corresponding parameter configuration file, and then load it into the reinforcement learning environment.

[0008] Step 2: Use the reward function generated in Step 1 to drive the agent to train a reinforcement learning policy, while simultaneously recording a video stream of the task execution and calculating the task success rate during training. If the task success rate meets the preset failure trigger condition, pause parameter updates.

[0009] Step 3: Process the video stream that failed in the last task, calculate the perceptual hash (pHash) Hamming distance between frames as a visual change feature, combine it with the temporal offset, calculate the cumulative mixed score for each frame, sample based on the cumulative mixed score, and extract several video frames to form a keyframe image sequence.

[0010] Step 4: Input the keyframe image sequence obtained in Step 3 into the Visual Language Model (VLM), use VLM to analyze the robot arm's motion logic and the reasons for task failure, and generate natural language feedback suggestions.

[0011] Step 5: Feed the suggestions generated in Step 4 back to the large language model to optimize the reward function, generate an updated reward function and hot reload it into the reinforcement learning training process. Repeat Steps 2 to 5 until the preset training rounds are reached.

[0012] The present invention has the following beneficial effects:

[0013] 1. Achieved true closed-loop automation: It establishes a closed loop of "code generation - environment execution - visual feedback - code correction", solving the problem of lack of physical verification in traditional "blind writing" code;

[0014] 2. Improved success rate of complex tasks: By introducing VLM as a feedback source, visual semantic analysis of the extracted execution video key frames is performed to directly evaluate the real interaction state between the robotic arm and the target object. This can discover logical errors that are difficult to be reflected on numerical curves, making the design of the reward function more in line with physical laws.

[0015] 3. Efficient keyframe extraction algorithm: The proposed extraction method based on perceptual hashing and temporal hybrid weights calculates the visual changes of adjacent video frames and scores and filters images by combining temporal bias and cumulative hybrid scoring. This effectively solves the problem of a large number of redundant frames generated by the robotic arm when it is stationary or performing repetitive actions, significantly reduces the inference overhead of VLM, and ensures accurate capture of fault scenes. Attached Figure Description

[0016] Figure 1The flowchart shows the method for optimizing the reward function of agent reinforcement learning based on VLM closed-loop feedback.

[0017] Figure 2 This is a flowchart of a keyframe extraction method based on a hybrid weighting of perceptual hashing and temporal sequence.

[0018] Figure 3 The diagram illustrates the interaction principle for optimizing the reward function based on VLM closed-loop feedback. Detailed Implementation

[0019] The present invention will be further explained below with reference to the accompanying drawings;

[0020] This embodiment uses robotic arm control as an example to illustrate the agent reinforcement learning reward function optimization method based on VLM closed-loop feedback, such as... Figure 1 As shown, the specific steps include the following detailed steps:

[0021] Step 1: The robotic arm control system receives user input. Including task instructions and code-level environment abstraction description .

[0022] In this embodiment, task instructions To open the drawer, a code-level environment abstraction description Specifically, it encapsulates the physical kinematic attributes of the robot body and the interactive target entity. The physical kinematic attributes of the robot body include end-effector pose, joint state, and grasping detection interface, while the physical kinematic attributes of the interactive target entity include spatial pose, motion velocity, 3D point cloud, and local symbolic distance field interface.

[0023] The Qwen-2.5-Max large language model is adopted, based on the environment state space. and action space Map the task objective to a reward function Output a Python code file containing the reward logic.

[0024] The reward function Modeled as a linear weighted combination of multiple feature components:

[0025]

[0026] in, This is the current state. This is the current action. These are the corresponding weighting coefficients; For task completion indication function, This is the sparse success reward constant. The number of basic reward items; For the first One basic reward item.

[0027] In this embodiment, the basic reward items include: , , .in , , These represent the three-dimensional coordinates characterizing the end effector, drawer handle, and target position, respectively. This is the motion vector of the robotic arm; This represents the Euclidean distance.

[0028] Step 2: Use the reward function obtained in Step 1 to drive the robotic arm to train a reinforcement learning policy in a simulation environment. In this embodiment, a flexible motion evaluation algorithm is used to update the policy network, maximizing the maximum entropy objective function. While ensuring the exploratory nature of the actions, optimize the control strategy:

[0029]

[0030] in, To reinforce the maximum number of time steps in a single round of a learning task; for The physical environment at any given moment for The robotic arm operates according to the strategy. The control actions performed Indicating in strategy Guided state-action edge distribution Find the expected value; Indicates the state Next action The value of the single-step reward function at that time. For temperature coefficient, Let be the policy entropy.

[0031] During strategy training, a course learning callback monitor is set up to record RGB video streams of the robotic arm during task execution. The average success rate of the robotic arm within the sliding window is calculated. :

[0032]

[0033] in, The length of the sliding window. For the first The task completion indicator function value for each training round is set to 1 if the robotic arm successfully completes the task within the maximum time step of that round, and 0 otherwise.

[0034] Set an average task success rate threshold =20%, failure trigger condition is: That is, recently Average success rate over training rounds Below the preset When the system detects that the task success rate meets the preset failure trigger conditions, it determines the reward function code generated by the current LLM. There may be a logical flaw or "language illusion". Pause network parameter updates and trigger diagnostic mode.

[0035] Step 3: To reduce the computational load of VLM and accurately locate faults, a keyframe extraction method based on a hybrid weighted approach of perceptual hashing and temporal sequence is proposed, such as... Figure 2 As shown, each frame of the video stream is first converted to RGB format, and a perceptual hash (pHash) fingerprint is extracted from each frame. Then, the Hamming distance between adjacent frames is calculated as the single-step visual difference. :

[0036]

[0037]

[0038] in, Indicates the first Single-step visual differences between frames This indicates the total number of frames in the video stream. This is the function for calculating Hamming distance.

[0039] Secondly, to prevent the omission of long-cycle static holding actions such as the stabilization process after the robotic arm grasps the object, a time offset is introduced into the single-step visual difference to calculate the single-step mixed difference. Calculate the cumulative mixed score starting from the first frame. :

[0040]

[0041]

[0042] in, For the first Single-step blending differences of frames; , which is the time bias term; For the first The cumulative mixed score of the frames.

[0043] Then, keyframe sampling is performed, with a preset total number of keyframes to be extracted. =10, in the interval Endogenous generation A uniformly distributed target score :

[0044]

[0045] For each target score , sequentially in the cumulative mixed score sequence { Find the frame index with the smallest absolute difference in} :

[0046]

[0047] in, For the first The expected score for each target sampling point The index of the video frame that best matches the expected score.

[0048] If two consecutive frame indices are found to be the same, the current frame index is forced to be shifted one frame forward to remove duplicates. Then, the first and last frames are forcibly included in the sampling set, and the finally selected frame is... The image sequence corresponding to each index is output as the keyframe set fed into the visual language model. .

[0049] Step 4: Using the Qwen-VL-Max visual language model, set the prompt word templates as shown in Table 1:

[0050] Table 1

[0051]

[0052] The keyframe set obtained in step 3 The input is processed by a visual language model, which analyzes the image content and generates text feedback. .

[0053] Step 5: The Large Language Model (LLM) receives the text feedback generated in Step 4. Through mapping function Use the current reward code Update to the new reward function code. :

[0054]

[0055] like Figure 3 As shown, LLM selects one of the following update modes based on the severity of the feedback:

[0056] ① Parameter fine-tuning: Maintain code structure, only update weight vectors ;

[0057] ② Code refactoring: Refactoring the reward function The calculation logic, such as modifying the basic reward items.

[0058] The newly generated file overwrites the old file through the file system, automatically reloads the module, and continues training in step 2 until the task reaches the preset training rounds.

[0059] To evaluate the improvement of our method over existing technologies, experiments were conducted on 10 tasks in the Metaworld dataset. After 1 million training steps, the average success rate of our method on the ten tasks was 89%; the average success rate of the Text2Reward method on the ten tasks was 80%; and the average success rate of the manually defined reward function method on the ten tasks was 71%. It can be seen that our method represents a significant improvement over existing methods.

[0060] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for optimizing the reward function of an agent in reinforcement learning based on VLM closed-loop feedback, which utilizes a large language model to infer user input, generates initial reward function code and its corresponding parameter configuration file, and then loads it into the reinforcement learning environment. The reward function is then optimized based on the training results. Its features include: Record video streams of the agent performing tasks and calculate the task success rate; when the task success rate meets the preset failure trigger conditions, pause parameter updates. Select the video stream from the last failed task, calculate the perceptual hash Hamming distance between frames, and then combine it with the temporal bias to calculate the cumulative mixed score for each frame. Extract several video frames to form a keyframe image sequence. Use a visual language model to analyze the agent's action logic and the reasons for task failure in the keyframe image sequence, and generate natural language feedback suggestions. Feed the generated suggestions back to the large language model to optimize the reward function.

2. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 1, characterized in that: The user input includes natural language task instructions and an abstract description of the environment.

3. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 2, characterized in that: The environmental abstraction description includes the end pose, joint state, and grasping detection interface of the intelligent agent, as well as the spatial pose, motion speed, 3D point cloud, and local symbolic distance field interface of the interactive target entity.

4. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 1, characterized in that: The large language model uses Qwen-2.5-Max, and the visual language model uses Qwen-VL-Max.

5. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 4, characterized in that: Using a visual language model, the actions of an agent in a keyframe image sequence are subjected to contact detection, operation detection, and stability and quality assessment. The most critical physical phenomena are objectively described, along with the root causes of these phenomena and suggestions for improvement.

6. The method for optimizing the reward function of agent reinforcement learning based on VLM closed-loop feedback as described in claim 1, characterized in that: Set the failure trigger condition to recent. The average success rate over the training rounds was lower than the preset average task success rate threshold.

7. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 1, characterized in that: Extract perceptual hash fingerprints from each frame of the video stream. Then, the Hamming distance between adjacent frames is calculated as the single-step visual difference. Introducing a time bias term Calculate the first Single-step blending difference of frames And calculate the cumulative mixed score starting from the first frame. : Set the total number of keyframes to be extracted. In the interval Intrinsic generation A uniformly distributed target score : For each target score , sequentially in the cumulative mixed score sequence { Find the frame index with the smallest absolute difference in} : Where Z represents the total number of frames in the video stream; For the first The expected score for each target sampling point The index of the video frame that best matches the expected score; Selected The video frame image sequence corresponding to each video frame index is output as the keyframe set fed into the visual language model. .

8. The agent reinforcement learning reward function optimization method based on VLM closed-loop feedback as described in claim 7, characterized in that: If two consecutive frame indices are found to be the same, the current frame index will be forced to be shifted one frame to the right to remove duplicates.

9. The method for optimizing the reward function of agent reinforcement learning based on VLM closed-loop feedback as described in claim 1, characterized in that: The large language model fine-tunes the weight vector of the reward function or reconstructs the calculation logic of the reward function based on the severity of the feedback suggestions output by the perceptual language model.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 9.

Images