Robot reinforcement learning training method and device based on general process reward modeling and electronic equipment

Abstract

The invention provides a robot reinforcement learning training method and device based on general process reward modeling and electronic equipment, and relates to the technical field of artificial intelligence and robot control, and the method comprises the steps: obtaining a robot operation demonstration video related to a reinforcement learning training task, performing fine tuning on a pre-trained general process reward model based on the robot operation demonstration video to obtain a target reward model; and executing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed. According to the robot reinforcement learning training method and device based on general process reward modeling and the electronic equipment provided by the invention, a multi-view fusion reward model based on relative progress jump prediction is adopted, and a strategy invariant reward shaping mechanism based on potential energy is combined, so that the progress can still be accurately evaluated under strong shielding; and the robot is guided to quickly learn the task under the condition of few samples and does not deviate from the target.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot reinforcement learning training method, apparatus and electronic device based on general process reward modeling. Background Technology

[0002] Reinforcement learning (RL) is a machine learning method that allows an agent to learn the optimal policy through trial and error in its interaction with the environment. It has become an important means of realizing complex robot operation tasks.

[0003] In reinforcement learning, the design of the reward function is crucial. Related techniques primarily employ sparse rewards, which reward only upon successful task completion (e.g., +1 for success, 0 for failure). However, this approach leads to an excessively large exploration space and low training efficiency in long-range, fine-touch tasks. Summary of the Invention

[0004] The purpose of this application is to provide a robot reinforcement learning training method, device, and electronic device based on general process reward modeling. It adopts a multi-view fusion reward model based on relative progress jump prediction and combines a policy-invariant reward shaping mechanism based on potential energy. It achieves the remarkable effect of accurately assessing progress under strong occlusion and guiding the robot to quickly learn the task without deviating from the goal with very few samples.

[0005] This application provides a robot reinforcement learning training method based on general process reward modeling, including: Obtain robot operation demonstration videos related to the reinforcement learning training task, and fine-tune a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; execute the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein, the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step action.

[0006] Optionally, the general process reward model is trained based on the following steps: acquiring a multi-source dataset, dividing each operation video in the multi-source dataset into multiple sub-task segments, and determining the initial state and task target state of each sub-task segment; the multi-source dataset contains operation videos from different sources; the operation videos from different sources include at least one of the following: robot operation videos, simulation environment operation videos, and real human operation videos; for each sub-task segment containing multiple image frames, calculating the relative change of each image pair as the label of the image pair, and constructing a training sample set based on the label of each image pair to obtain a target sample set; each image pair includes: an image frame before the operation and an image frame after the operation; training a visual language model using the target sample set to obtain the general process reward model; wherein, the input of the visual language model includes: text instructions for describing the current task, and multi-view images; the multi-view images include: a task initial state map set, a task target state map set, a state map set before the operation at the current time, and a state map set after the operation at the current time; each state map set contains images from multiple perspectives.

[0007] Optionally, calculating the relative change of each image pair as a label for the image pair includes: if the change magnitude of the two image frames contained in the target image pair is greater than or equal to a preset change threshold, determining the change type of the two image frames, and calculating the relative change of the two image frames based on the change type; if the change magnitude of the two image frames contained in the target image pair is less than the preset change threshold, setting the relative change of the two image frames to 0; wherein, the target image pair is any image pair contained in the plurality of subtask segments.

[0008] Optionally, determining the change type of the two image frames and calculating the relative change amount of the two image frames based on the change type includes: when the task completion progress increases after the operation is completed, determining the ratio of the progress increment to the remaining task amount as the relative change amount of the two image frames; when the task completion progress decreases after the operation is completed, determining the ratio of the progress reduction to the completed task amount as the relative change amount of the two image frames.

[0009] Optionally, the step of executing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed includes: determining a first state before the action is executed and a second state after the action is executed based on the action executed by the robot based on the current policy network; predicting the global progress at the current moment using the target reward model based on the first state and the second state, and calculating the global progress value based on the prediction result; calculating the final reward at the current moment based on the global progress value, and iteratively updating the parameters of the policy network based on the final reward; wherein the final reward is calculated based on sparse success reward and potential energy shaping reward; the potential energy shaping reward is calculated by using the potential energy-based difference formula to calculate the task completion progress.

[0010] Optionally, the step of predicting the global progress at the current moment based on the first state and the second state using the target reward model includes: inputting multi-view images corresponding to the first state, the second state, the initial state of the task, and the target state of the task, as well as the task instruction text, into the target reward model to predict the global progress at the current moment and obtain the relative jump value at the current moment predicted by the target reward model.

[0011] Optionally, the step of calculating the global progress value based on the prediction result includes: adding the global progress of the previous moment to the relative jump value of the target reward model to obtain a first progress value; inputting the second state and the initial state of the task into the target reward model to predict a second progress value; and inputting the target state of the task and the second state into the target reward model to predict a third progress value; and using the average of the first progress value, the second progress value, and the third progress value as the global progress value.

[0012] Optionally, calculating the final reward at the current moment based on the global progress value includes: if the global progress value exceeds a preset progress threshold, then determining a first reward value as a sparse reward; otherwise, determining a second reward value as a sparse reward; the first reward value is used to characterize the completion of the robot task, and the second reward value is used to characterize the incomplete execution of the robot task; calculating the target product of a discount factor and the estimated global progress value at the next moment, and subtracting the target product from the global progress value to obtain a target difference; adding the target difference to the sparse reward to obtain the final reward at the current moment.

[0013] This application also provides a robot reinforcement learning training device based on general process reward modeling, including: The data acquisition module is used to acquire robot operation demonstration videos related to the reinforcement learning training task; the model adaptation module is used to fine-tune a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; the task execution module is used to execute the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein, the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step action.

[0014] Optionally, the device further includes: a label generation module and a training module; the data acquisition module is further configured to acquire a multi-source dataset, divide each operation video in the multi-source dataset into multiple sub-task segments, and determine the initial state and task target state of each sub-task segment; the multi-source dataset contains operation videos from different sources; the operation videos from different sources include at least one of the following: robot operation videos, simulation environment operation videos, and real human operation videos; the label generation module is configured to calculate the relative change of each image pair as an image pair for the multiple image frames contained in each sub-task segment. The system assigns labels to each image pair and constructs a training sample set based on the labels of each image pair to obtain the target sample set. Each image pair includes an image frame before the operation and an image frame after the operation. The training module is used to train the visual language model using the target sample set to obtain the general process reward model. The input of the visual language model includes text instructions describing the current task and multi-view images. The multi-view images include: a task initial state graph set, a task target state graph set, a state graph set before the operation at the current time, and a state graph set after the operation at the current time. Each state graph set contains images from multiple perspectives.

[0015] Optionally, the tag generation module is specifically used to determine the change type of the two image frames when the change amplitude of the two image frames contained in the target image pair is greater than or equal to a preset change threshold, and to calculate the relative change amount of the two image frames based on the change type; the tag generation module is further used to set the relative change amount of the two image frames to 0 when the change amplitude of the two image frames contained in the target image pair is less than the preset change threshold; wherein, the target image pair is any image pair contained in the plurality of subtask segments.

[0016] Optionally, the tag generation module is specifically used to determine the ratio of the progress increment to the remaining amount of the task as the relative change between the two image frames when the task completion progress increases after the operation is completed; the tag generation module is also specifically used to determine the ratio of the progress reduction to the amount of task completion as the relative change between the two image frames when the task completion progress decreases after the operation is completed.

[0017] Optionally, the task execution module is specifically used to determine a first state before the action is executed and a second state after the action is executed based on the action executed by the robot based on the current policy network; the task execution module is further used to predict the global progress at the current moment using the target reward model based on the first state and the second state, and calculate the global progress value based on the prediction result; the task execution module is further used to calculate the final reward at the current moment based on the global progress value, and iteratively update the parameters of the policy network based on the final reward; wherein, the final reward is calculated based on sparse success reward and potential energy shaping reward; the potential energy shaping reward is calculated by using the potential energy-based difference formula to calculate the task completion progress.

[0018] Optionally, the task execution module is specifically used to input the multi-view images corresponding to the first state, the second state, the task initial state, and the task target state, as well as the task instruction text, into the target reward model to predict the global progress at the current moment and obtain the relative jump value at the current moment predicted by the target reward model.

[0019] Optionally, the task execution module is specifically configured to add the global progress of the previous moment to the relative jump value of the target reward model to obtain a first progress value, input the second state and the initial state of the task into the target reward model to predict a second progress value, and input the target state of the task and the second state into the target reward model to predict a third progress value; the task execution module is further configured to use the average of the first progress value, the second progress value, and the third progress value as the global progress value.

[0020] Optionally, the task execution module is specifically configured to determine a first reward value as a sparse reward if the global progress value exceeds a preset progress threshold, and otherwise determine a second reward value as a sparse reward; the first reward value is used to represent the completion of the robot task, and the second reward value is used to represent the incomplete execution of the robot task; the task execution module is further configured to calculate the target product of the discount factor and the estimated global progress value at the next time step, and subtract the target product from the global progress value to obtain a target difference; the task execution module is further configured to add the target difference to the sparse reward to obtain the final reward at the current time step.

[0021] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the robot reinforcement learning training method based on general process reward modeling as described above.

[0022] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the robot reinforcement learning training method based on general process reward modeling as described above.

[0023] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot reinforcement learning training method based on general process reward modeling as described above.

[0024] The robot reinforcement learning training method, apparatus, and electronic device based on general process reward modeling provided in this application first acquire robot operation demonstration videos related to the reinforcement learning training task, and then fine-tune a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; then, the reinforcement learning training task is executed based on the target reward model until the reinforcement learning training task is completed; wherein, the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step. Thus, by employing a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, significant results are achieved in accurately assessing progress even under strong occlusion, and guiding the robot to quickly learn the task with minimal samples without deviating from the target. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is one of the flowcharts of the robot reinforcement learning training method based on general process reward modeling provided in this application; Figure 2 This is the second flowchart of the robot reinforcement learning training method based on general process reward modeling provided in this application; Figure 3 This is a schematic diagram of the robot reinforcement learning training device based on general process reward modeling provided in this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. All actions involving the acquisition of signal information or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the relevant device.

[0029] The following is a detailed description of the technical terms used in the embodiments of this application: Reinforcement Learning (RL): A machine learning method in which an agent optimizes its behavioral strategies by interacting with its environment and receiving "reward" or "punishment" signals.

[0030] Process Reward Model (PRM): This refers to a model that provides feedback for each intermediate step in the task execution process, unlike models that only provide success / failure feedback at the end of the task.

[0031] General Reward Model (GRM): A core model proposed in this solution, referring to a reward evaluation model trained on large-scale data that can be applied to different tasks and scenarios.

[0032] Hop-based Mechanism: This scheme proposes a progress assessment method that does not directly predict absolute progress values, but calculates progress by comparing the relative changes between two states (i.e., "hops").

[0033] Policy-Invariant Reward Shaping: A method for designing reward functions that ensures that artificially introduced auxiliary reward signals do not change the original optimal solution of the task, preventing the agent from making erroneous behaviors in order to obtain rewards.

[0034] Semantic trap: In reinforcement learning, if the reward is not set properly (e.g., simply accumulating progress points), the agent may find that "maintaining a progress state without completing the task" can obtain a higher total score, thus stagnating.

[0035] To address the aforementioned technical problems in related technologies, this application provides a robot reinforcement learning training method based on general process reward modeling, which uses a visual language model as the process reward model. However, using a visual language model also faces many technical challenges. For example, in complex robotic arm operations, occlusion is easily caused between the robotic arm itself or other objects. When key parts are occluded, the model cannot obtain effective information, leading to ineffective progress evaluation or illusions, and failing to provide accurate reward signals. Directly using progress increments as rewards mathematically alters the policy optimization objective, resulting in a "semantic trap" phenomenon. That is, to maximize cumulative rewards, the agent tends to quickly reach a high progress state and stagnate or oscillate there, rather than actually completing the task, which seriously misleads the learning direction of the policy.

[0036] Based on this, the robot reinforcement learning training method based on general process reward modeling provided in this application constructs a general process reward model (GRM) based on a "jump mechanism" and "multi-perspective fusion" for accurately perceiving task progress. Furthermore, a "policy-invariant reward shaping" method is proposed to transform progress evaluation into dense reward signals that do not change the optimal policy, thereby guiding the robot to efficiently and correctly learn complex operational skills. Figure 1 As shown, this method first trains a general process reward model. Then, it fine-tunes the model specifically for different reinforcement learning training tasks. Finally, the fine-tuned model is used in the reinforcement learning training task.

[0037] The robot reinforcement learning training method based on general process reward modeling provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0038] like Figure 2 As shown in the embodiment of this application, a robot reinforcement learning training method based on general process reward modeling is provided. This method may include the following steps 201 and 202: Step 201: Obtain robot operation demonstration videos related to the reinforcement learning training task, and fine-tune the pre-trained general process reward model based on the robot operation demonstration videos to obtain the target reward model.

[0039] The general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step.

[0040] For example, in order for a general "referee" model (i.e., the aforementioned general process reward model) to quickly become familiar with a specific new task (such as a particular workpiece or environment), it is necessary to fine-tune the model's parameters. This mainly includes the following two steps: 1. Obtain demo data When faced with a new robot operation task, a successful and complete demonstration video can be recorded by a human simply by remotely operating the robot or by dragging and dropping to teach it. This is the robot operation demonstration video mentioned above.

[0041] 2. Rapid model fine-tuning Using the acquired demonstration video, a small number of training samples are generated following the same method used to generate training samples during the training of the general process reward model. Then, keeping most of the parameters of the general reward model unchanged, supervised fine-tuning (SFT) is performed using these small number of samples.

[0042] It should be noted that the parameters of the general process reward model are fine-tuned to enable the model to quickly adapt to the characteristics of the new environment, such as lighting and object texture, and to calibrate its judgment criteria for the progress of this specific task. After adaptation, the model becomes the dedicated reward model for this task, namely the target reward model mentioned above.

[0043] Step 202: Execute the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed.

[0044] For example, after fine-tuning the general process reward model and obtaining the target reward model, the target reward model can be used to participate in reinforcement learning training tasks, that is, the robot interacts with the environment and obtains feedback through the adapted reward model to optimize the strategy.

[0045] Specifically, step 202 above may also include the following steps: 202a, 202b, and 202c: Step 202a: Based on the actions performed by the robot according to the current policy network, determine the first state before the action is performed and the second state after the action is performed.

[0046] Step 202b: Based on the first state and the second state, the global progress at the current moment is predicted using the target reward model, and the global progress value is calculated based on the prediction result.

[0047] Step 202c: Based on the global progress value, calculate the final reward at the current moment, and iteratively update the parameters of the policy network based on the final reward.

[0048] The final reward is calculated based on the sparse success reward and the potential energy-based shaping reward; the potential energy-based shaping reward is calculated by using the potential energy-based difference formula to calculate the task completion progress.

[0049] For example, the robot performs an action according to the current policy network, and the environment changes accordingly. The system records a first state "before performing the action" and a second state "after performing the action." Then, the target reward model is used to predict the global progress at the current moment.

[0050] Specifically, step 202b above, which involves using the target reward model to predict the global progress at the current moment, may further include the following step 202b1: Step 202b1: Input the multi-view images corresponding to the first state, the second state, the initial state of the task, and the target state of the task, as well as the task instruction text, into the target reward model to predict the global progress at the current moment and obtain the relative jump value predicted by the target reward model at the current moment.

[0051] For example, the input to the target reward model includes: multi-view images corresponding to the four states, and task instruction text describing the robot's current task (e.g., "put the red square into the plate"). After network inference, the target reward model outputs a discretized token, representing the relative jump value (i.e., the hop value) between "after the operation" and "before the operation".

[0052] For example, the above multi-view images may include: frontal view image, left wrist view image, right wrist view image, etc.

[0053] Specifically, step 202b above, which calculates the global progress value based on the prediction result, may further include steps 202b2 and 202b3: Step 202b2: Add the global progress of the previous moment to the relative jump value of the target reward model to obtain the first progress value; input the second state and the initial state of the task into the target reward model to predict the second progress value; and input the target state of the task and the second state into the target reward model to predict the third progress value.

[0054] Step 202b3: Take the average of the first progress value, the second progress value, and the third progress value as the global progress value.

[0055] For example, the adapted reward model is used to estimate the global progress of the current state. To ensure accuracy, the system executes the following three calculation logics simultaneously and takes the average: 1. An incremental perspective that focuses on capturing action details: Based on the progress of the previous moment, plus the "jump value" of the current step predicted by the model, which is the first progress value mentioned above.

[0056] 2. Forward anchoring perspective with emphasis on global reference: Input the "current state" and "initial state" into the model to directly predict how much has been completed, i.e. the second progress value mentioned above.

[0057] 3. Backward anchoring perspective focusing on endpoint sensitivity: Input the "current state" and "target state" into the model to directly predict how much is left to be completed, i.e. the third progress value mentioned above.

[0058] For example, the final global progress value is obtained by averaging the three progress values ​​mentioned above. By integrating information from three dimensions, even if a camera at one perspective is blocked by the robotic arm, information from other perspectives can be supplemented to ensure that progress assessment is uninterrupted and does not drift.

[0059] For example, in order to prevent the robot from deliberately not completing the task in order to earn progress points (i.e., semantic trap), the system does not send the progress value directly to the robot, but calculates the final reward through a specific method.

[0060] It should be noted that step 202b2 above includes three calculation steps. These three calculation steps are not distinguished by their order and can be executed simultaneously or sequentially, all of which fall within the protection scope of the embodiments of this application.

[0061] Specifically, step 202 above may also include steps 202c1 to 202c3: Step 202c1: If the global progress value exceeds the preset progress threshold, the first reward value is determined as a sparse reward; otherwise, the second reward value is determined as a sparse reward.

[0062] The first reward value is used to indicate that the robot task has been completed, and the second reward value is used to indicate that the robot task has not been completed.

[0063] Step 202c2: Calculate the target product of the discount factor and the global progress estimate at the next time step, and subtract the target product from the global progress value to obtain the target difference.

[0064] Step 202c3: Add the target difference to the sparse reward to obtain the final reward at the current moment.

[0065] For example, the method for calculating the final reward r provided in this application embodiment includes the following: 1. Determine if it was successful: If the calculated global progress If the success rate exceeds a set threshold (e.g., 95%), the task is considered successful and a large sparse reward (e.g., 1 point) is given; otherwise, 0 points are given.

[0066] 2. Calculate the shaping reward using a difference formula based on potential energy: The final reward r = sparse reward + discount factor * next time step global progress estimate - current time step global progress estimate. Here, the discount factor is a constant less than 1, for example, 0.99.

[0067] 3. Feedback to the strategy: The calculated final reward Send it to the reinforcement learning algorithm (such as policy gradient).

[0068] For example, the reinforcement learning algorithm updates the robot's policy network parameters based on the received reward signal, repeating the above process until the entire reinforcement learning training task is completed.

[0069] It should be noted that, due to the special formula design of the final reward, the cumulative rewards in the intermediate process can be mathematically offset. This means that in order to maximize the total benefit, the robot must pursue the final sparse reward (task success), while the dense progress rewards in the middle only serve as a "compass" to accelerate the search for the path to success, thus perfectly solving the "semantic trap" problem.

[0070] Optionally, in the embodiments of this application, before steps 201 and 202 above, it is necessary to train an "experienced" referee model (i.e. the above general process reward model) so that it can be applied to various different robots and task scenarios.

[0071] For example, prior to step 201 above, the robot reinforcement learning training method based on general process reward modeling provided in this application embodiment may further include the following steps 203 to 205: Step 203: Obtain a multi-source dataset, divide each operation video in the multi-source dataset into multiple sub-task segments, and determine the initial state and task target state of each sub-task segment.

[0072] The multi-source dataset contains operation videos from different sources; the operation videos from different sources include at least one of the following: robot operation videos, simulation environment operation videos, and real human operation videos.

[0073] For example, the data sources used to train the visual language model may include a dataset containing mixed video data (e.g., approximately 35 million frames in total) that includes real robot actions, simulated environments, and human first-person perspective actions. This ensures that the model not only recognizes robot movements but also understands the operational logic of human hands (such as grasping, folding, etc.).

[0074] For example, after obtaining the above data, the continuous video trajectory can be divided into several sub-task segments according to keyframes, and the initial state (i.e. the state when the task starts) and the task target state (i.e. the state when the task is completed) of each segment can be defined.

[0075] Step 204: For each subtask segment containing multiple image frames, calculate the relative change of each image pair as the label of the image pair, and construct the training sample set based on the label of each image pair to obtain the target sample set.

[0076] Each image pair includes: an image frame before the operation and an image frame after the operation.

[0077] For example, for each subtask segment containing multiple image frames, layered sampling and label creation are also required.

[0078] Specifically, step 204 above, which calculates the relative change of each image pair as the label of the image pair, may further include the following steps 204a and 204b: Step 204a: If the change magnitude of the two image frames contained in the target image pair is greater than or equal to a preset change threshold, determine the change type of the two image frames and calculate the relative change amount of the two image frames based on the change type.

[0079] For example, in calculating the relative hop label, for any pair of sampled "before" and "after" images in the subtask segment, instead of directly labeling their absolute progress (e.g., 50%), their relative change is calculated for the image frames contained in the subtask segment.

[0080] Specifically, step 204a above may also include the following steps 204a1 and 204a2: Step 204a1: After the operation is completed and the task completion progress increases, the ratio of the progress increment to the remaining task amount is determined as the relative change between the two image frames.

[0081] Step 204a2: If the task completion progress decreases after the operation is completed, the ratio of the progress reduction to the amount of task completion is determined as the relative change between the two image frames.

[0082] For example, if progress increases (advances) after an action, the label = (progress increment) / (how far to the finish line). This means the closer to the finish line, the higher the score for the same amount of progress. If progress decreases (regresses) after an action, the label = (progress decrement) / (how far to the starting point). This means the closer to the beginning, the heavier the penalty for regression.

[0083] The calculated values ​​are then mapped to the [-1, 1] interval, serving as the ground truth for model training, i.e., the aforementioned labels. When training the visual language model, a stratified sampling method can be used to select training samples, ensuring that samples from each label interval can be utilized.

[0084] Step 204b: If the change magnitude of the two image frames contained in the target image pair is less than the preset change threshold, the relative change of the two image frames is set to 0.

[0085] The target image pair is any one of the image pairs contained in the plurality of subtask segments.

[0086] For example, in an embodiment of this application, in order to teach the model not to generate the illusion of "progress" when the robot is dazed or stuck, zero-hop data augmentation was performed on the training samples. That is, a batch of samples with almost no change before and after the scene were specifically selected, and their labels were forcibly set to 0.

[0087] Step 205: Train the visual language model using the target sample set to obtain the general process reward model.

[0088] The input to the visual language model includes: text instructions describing the current task and multi-view images; the multi-view images include: task initial state map set, task target state map set, state map set before the operation at the current time, and state map set after the operation at the current time; each state map set contains images from multiple perspectives.

[0089] For example, after constructing the training samples, the visual language model can be trained. The training process mainly includes: 1. Model Input: Construct a Visual Language Model (VLM). Input includes: Text instructions describing the current task (e.g., "Put the red square into the plate"). Four sets of multi-view images: initial task state image, task target state image, state image before the current operation, and state image after the current operation. Each set of images contains multiple perspectives (e.g., frontal view, left wrist view, right wrist view).

[0090] 2. Prediction Target: After network inference, the model outputs a discrete token, representing the relative jump value (i.e., Hop value) between "after operation" and "before operation".

[0091] In this way, the training of the visual language model can be completed, and the aforementioned general process reward model can be obtained.

[0092] The robot reinforcement learning training method based on general process reward modeling provided in this application has the following advantages compared with the technical solutions in related technologies: 1. Extremely high progress perception accuracy and robustness against occlusion. Advantages: The general reward model (GRM) constructed in this scheme achieves a video frame ordering correlation (VOC) score of 0.953 and a task completion accuracy of 92.8% on eight benchmark datasets including real-world and simulation environments, which is significantly better than existing single-view reward models (such as GVL, VLAC) and general vision large models (such as GPT-4V).

[0093] Cause: A multi-view progress fusion technique was employed, combining image information from both the frontal and wrist perspectives. When one perspective (such as the wrist camera) is obstructed by the robotic arm's movement, other perspectives (such as the frontal camera) can supplement key information, thus maintaining the continuity and accuracy of the evaluation in complex operations. A hop mechanism was used instead of directly regressing absolute progress. This relative prediction method effectively avoids the error accumulation problem in long sequence tasks, making the model more sensitive to subtle changes in movement.

[0094] 2. Excellent sample efficiency and extremely fast training convergence speed Advantages: Compared to traditional reinforcement learning methods that often require tens of thousands of interactions, this solution only requires one human demonstration for adaptation. Within approximately 150 online interactions (about 1 hour of real-device operation time), the success rate of complex operation tasks can be increased from nearly 0% to 95%.

[0095] Reasons for this: One-shot Adaptation: Utilizing fine-tuning techniques to quickly transfer general physical prior knowledge to a specific task, significantly reducing startup costs. Dense Reward Feedback: Compared to sparse rewards (which only award points at the finish line), this solution provides high-quality feedback signals at every step, greatly reducing the space for policy search.

[0096] 3. It fundamentally eliminates the "semantic trap" and ensures policy optimality. Advantages: It effectively solves the "score-farming" phenomenon (i.e., the agent lingers in place or deliberately fails to complete the task in order to gain process points) that is easily caused by traditional intensive rewards. Experiments show that removing this mechanism causes the success rate to plummet by 43.7%.

[0097] Cause: The policy-invariant reward shaping method was employed. This method, based on rigorous mathematical derivation (potential-based reward shaping theory), transforms progress evaluation into a special form of differential reward. This form is mathematically equivalent to only awarding a reward for final success, thus ensuring that the agent's optimal policy always points to "completing the task" rather than "accumulating process points."

[0098] 4. Strong generalization ability and environmental adaptability Advantages: When faced with out-of-distribution (OOD) situations such as unseen object layouts, cluttered backgrounds, or changes in lighting, the performance degradation of this solution is only 15-24%, which is far better than traditional behavioral cloning methods (performance degradation of 50-60%).

[0099] Reason for this: The model was pre-trained on large-scale (35 million samples) and multi-source (real / simulated / human video) data, and learned general rules of object state change, rather than memorizing the robot's specific trajectory. Therefore, it has strong generalization ability across scenes and forms.

[0100] The robot reinforcement learning training method based on general process reward modeling provided in this application first acquires robot operation demonstration videos related to the reinforcement learning training task, and then fine-tunes a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model. Next, the reinforcement learning training task is executed based on the target reward model until the task is completed. The general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value characterizes the relative change in the robot's task completion progress before and after a single step. Thus, by employing a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, the method achieves significant results: it can accurately assess progress even under strong occlusion, and guides the robot to quickly learn the task with minimal samples without deviating from the target.

[0101] It should be noted that the robot reinforcement learning training method based on general process reward modeling provided in this application embodiment can be executed by a robot reinforcement learning training device based on general process reward modeling, or by a control module within that device for executing the method. This application embodiment uses the execution of the robot reinforcement learning training method based on general process reward modeling by a robot reinforcement learning training device based on general process reward modeling as an example to illustrate the robot reinforcement learning training device based on general process reward modeling provided in this application embodiment.

[0102] It should be noted that, in the embodiments of this application, the robot reinforcement learning training methods based on general process reward modeling shown in the accompanying drawings are all illustrated using one accompanying drawing from one of the embodiments of this application as an example. In specific implementation, the robot reinforcement learning training methods based on general process reward modeling shown in the accompanying drawings of the above methods can also be implemented in conjunction with any other accompanying drawings shown in the above embodiments, which will not be elaborated here.

[0103] The robot reinforcement learning training device based on general process reward modeling provided in this application is described below. The robot reinforcement learning training method based on general process reward modeling described above can be referred to in the following text.

[0104] Figure 3A schematic diagram of the structure of the robot reinforcement learning training device based on general process reward modeling provided in the embodiments of this application is shown below. Figure 3 As shown, it specifically includes: The data acquisition module 301 is used to acquire robot operation demonstration videos related to the reinforcement learning training task; the model adaptation module 302 is used to fine-tune a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; the task execution module 303 is used to execute the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein, the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step action.

[0105] Optionally, the device further includes: a label generation module and a training module; the data acquisition module 301 is further configured to acquire a multi-source dataset, divide each operation video in the multi-source dataset into multiple sub-task segments, and determine the initial state and task target state of each sub-task segment; the multi-source dataset contains operation videos from different sources; the operation videos from different sources include at least one of the following: robot operation videos, simulation environment operation videos, and real human operation videos; the label generation module is configured to calculate the relative change of each image pair as an image for each sub-task segment containing multiple image frames. The system assigns labels to each image pair and constructs a training sample set based on the labels of each image pair to obtain the target sample set. Each image pair includes an image frame before the operation and an image frame after the operation. The training module is used to train the visual language model using the target sample set to obtain the general process reward model. The input of the visual language model includes text instructions describing the current task and multi-view images. The multi-view images include: a task initial state graph set, a task target state graph set, a state graph set before the operation at the current time, and a state graph set after the operation at the current time. Each state graph set contains images from multiple perspectives.

[0106] Optionally, the tag generation module is specifically used to determine the change type of the two image frames when the change amplitude of the two image frames contained in the target image pair is greater than or equal to a preset change threshold, and to calculate the relative change amount of the two image frames based on the change type; the tag generation module is further used to set the relative change amount of the two image frames to 0 when the change amplitude of the two image frames contained in the target image pair is less than the preset change threshold; wherein, the target image pair is any image pair contained in the plurality of subtask segments.

[0107] Optionally, the tag generation module is specifically used to determine the ratio of the progress increment to the remaining amount of the task as the relative change between the two image frames when the task completion progress increases after the operation is completed; the tag generation module is also specifically used to determine the ratio of the progress reduction to the amount of task completion as the relative change between the two image frames when the task completion progress decreases after the operation is completed.

[0108] Optionally, the task execution module 303 is specifically used to determine a first state before the action is executed and a second state after the action is executed based on the action executed by the robot based on the current policy network; the task execution module 303 is further used to predict the global progress at the current moment based on the first state and the second state using the target reward model, and calculate the global progress value based on the prediction result; the task execution module 303 is further used to calculate the final reward at the current moment based on the global progress value, and iteratively update the parameters of the policy network based on the final reward; wherein, the final reward is calculated based on sparse success reward and potential energy shaping reward; the potential energy shaping reward is calculated by using the potential energy-based difference formula to calculate the task completion progress.

[0109] Optionally, the task execution module 303 is specifically used to input the multi-view images corresponding to the first state, the second state, the task initial state, and the task target state, as well as the task instruction text, into the target reward model to predict the global progress at the current moment and obtain the relative jump value at the current moment predicted by the target reward model.

[0110] Optionally, the task execution module 303 is specifically used to add the global progress of the previous moment to the relative jump value of the target reward model to obtain a first progress value, input the second state and the initial state of the task into the target reward model to predict a second progress value, and input the target state of the task and the second state into the target reward model to predict a third progress value; the task execution module 303 is also specifically used to take the average of the first progress value, the second progress value and the third progress value as the global progress value.

[0111] Optionally, the task execution module 303 is specifically configured to determine a first reward value as a sparse reward if the global progress value exceeds a preset progress threshold, and otherwise determine a second reward value as a sparse reward; the first reward value is used to represent the completion of the robot task, and the second reward value is used to represent the incomplete execution of the robot task; the task execution module 303 is further configured to calculate the target product of the discount factor and the estimated global progress value at the next time step, and subtract the target product from the global progress value to obtain a target difference; the task execution module 303 is further configured to add the target difference to the sparse reward to obtain the final reward at the current time step.

[0112] The robot reinforcement learning training device based on general process reward modeling provided in this application first acquires robot operation demonstration videos related to the reinforcement learning training task, and then fine-tunes a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model. Next, the reinforcement learning training task is executed based on the target reward model until the task is completed. The general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value characterizes the relative change in the robot's task completion progress before and after a single step. Thus, by employing a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, it achieves the significant effect of accurately assessing progress even under strong occlusion and guiding the robot to quickly learn the task with minimal samples and without deviating from the target.

[0113] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a robot reinforcement learning training method based on a general process reward model. This method includes: first, acquiring robot operation demonstration videos related to the reinforcement learning training task, and fine-tuning a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; then, executing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step. Thus, by adopting a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, we can achieve the remarkable effect of accurately assessing progress even under strong occlusion and guiding the robot to quickly learn the task without deviating from the goal with very few samples.

[0114] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, enable the computer to execute the robot reinforcement learning training method based on general process reward modeling provided by the above methods. This method includes: first, acquiring robot operation demonstration videos related to the reinforcement learning training task, and fine-tuning a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; then, executing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein, the general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step. Thus, by employing a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, a significant effect is achieved: accurate progress assessment can still be performed under strong occlusion, and the robot can be guided to quickly learn the task without deviating from the target with very few samples.

[0116] Furthermore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned robot reinforcement learning training methods based on general process reward modeling. This method includes: first, acquiring robot operation demonstration videos related to the reinforcement learning training task, and fine-tuning a pre-trained general process reward model based on the robot operation demonstration videos to obtain a target reward model; then, executing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed; wherein the general process reward model is a visual language model used to predict relative jump values ​​between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step. Thus, by employing a multi-view fusion reward model based on relative progress jump prediction, combined with a policy-invariant reward shaping mechanism based on potential energy, significant results are achieved: accurate progress assessment is still possible under strong occlusion, and the robot is guided to quickly learn the task with minimal samples without deviating from the target.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A robot reinforcement learning training method based on a general process reward model, characterized in that, include: Obtain robot operation demonstration videos related to reinforcement learning training tasks, and fine-tune the pre-trained general process reward model based on the robot operation demonstration videos to obtain the target reward model; The reinforcement learning training task is performed based on the target reward model until the reinforcement learning training task is completed. The general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step.

2. The method according to claim 1, characterized in that, The general process reward model is trained based on the following steps: A multi-source dataset is acquired, and each operation video in the multi-source dataset is divided into multiple sub-task segments. The initial state and task target state of each sub-task segment are determined. The multi-source dataset contains operation videos from different sources. The operation videos from different sources include at least one of the following: robot operation videos, simulation environment operation videos, and real human operation videos. For each subtask segment containing multiple image frames, the relative change of each image pair is calculated as the label of the image pair, and the training sample set is constructed based on the label of each image pair to obtain the target sample set; Each image pair includes: the image frame before the operation and the image frame after the operation; The visual language model is trained using the target sample set to obtain the general process reward model; The input to the visual language model includes: text instructions describing the current task and multi-view images; the multi-view images include: task initial state map set, task target state map set, state map set before the operation at the current time, and state map set after the operation at the current time; each state map set contains images from multiple perspectives.

3. The method according to claim 2, characterized in that, The calculation of the relative change of each image pair as the label of the image pair includes: If the change magnitude of the two image frames contained in the target image is greater than or equal to a preset change threshold, determine the change type of the two image frames and calculate the relative change amount of the two image frames based on the change type. If the change in the two image frames contained in the target image pair is less than the preset change threshold, the relative change between the two image frames is set to 0. The target image pair is any one of the image pairs contained in the plurality of subtask segments.

4. The method according to claim 3, characterized in that, The step of determining the change type of two image frames and calculating the relative change amount of the two image frames based on the change type includes: When the task completion progress increases after the operation is completed, the ratio of the progress increment to the remaining task amount is determined as the relative change between the two image frames. When the task completion progress decreases after the operation is completed, the ratio of the decrease in progress to the amount of task completion is determined as the relative change between the two image frames.

5. The method according to claim 1, characterized in that, The step of performing the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed includes: Based on the actions performed by the robot according to the current policy network, determine the first state before the action is performed and the second state after the action is performed. Based on the first state and the second state, the global progress at the current moment is predicted using the target reward model, and the global progress value is calculated based on the prediction result. Based on the global progress value, calculate the final reward at the current moment, and iteratively update the parameters of the policy network based on the final reward; The final reward is calculated based on the sparse success reward and the potential energy-based shaping reward; the potential energy-based shaping reward is calculated by using the potential energy-based difference formula to calculate the task completion progress.

6. The method according to claim 5, characterized in that, The step of predicting the global progress at the current moment using the target reward model based on the first state and the second state includes: The multi-view images corresponding to the first state, the second state, the initial state of the task, and the target state of the task, as well as the task instruction text, are input into the target reward model to predict the global progress at the current moment, thereby obtaining the relative jump value at the current moment predicted by the target reward model.

7. The method according to claim 5 or 6, characterized in that, The calculation of the global progress value based on the prediction results includes: The global progress at the previous moment is added to the relative jump value of the target reward model to obtain the first progress value. The second state and the initial state of the task are input into the target reward model to predict the second progress value. The target state of the task and the second state are input into the target reward model to predict the third progress value. The average of the first progress value, the second progress value, and the third progress value is used as the global progress value.

8. The method according to claim 5 or 6, characterized in that, The calculation of the final reward at the current moment based on the global progress value includes: If the global progress value exceeds the preset progress threshold, the first reward value is determined as a sparse reward; otherwise, the second reward value is determined as a sparse reward. The first reward value is used to indicate that the robot task has been completed, and the second reward value is used to indicate that the robot task has not been completed. Calculate the target product of the discount factor and the global progress estimate at the next time step, and subtract the target product from the global progress value to obtain the target difference; The final reward at the current moment is obtained by adding the target difference to the sparse reward.

9. A robot reinforcement learning training device based on general process reward modeling, characterized in that, The device includes: The data acquisition module is used to acquire robot operation demonstration videos related to reinforcement learning training tasks; The model adaptation module is used to fine-tune the pre-trained general process reward model based on the robot operation demonstration video to obtain the target reward model; The task execution module is used to execute the reinforcement learning training task based on the target reward model until the reinforcement learning training task is completed. The general process reward model is a visual language model used to predict the relative jump value between videos; the relative jump value is used to characterize the relative change in the robot's task completion progress before and after a single step.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the robot reinforcement learning training method based on general process reward modeling as described in any one of claims 1 to 8.

Images