Embodied intelligent reinforcement learning method based on visual feature semantic clustering and value guidance
Patent Information
- Application Number
- CN202610743497.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-27
- Publication Date
- 2026-09-04
AI Technical Summary
[0007]为了克服上述现有技术的不足，本发明提供一种基于视觉特征语义聚类与价值引导的具身智能强化学习方法，以解决具身智能模型在长时序机器人操作任务中存在的奖励信号稀疏、任务进度难以度量、人工奖励设计成本高以及策略泛化能力不足等问题
[0032]This invention provides an embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance. It automatically extracts visual features from task trajectories through a pre-trained visual model and obtains semantically structured task stage divisions based on the changing trends of these visual features, avoiding the complex process of manually designing dense reward functions. This method can transform intermediate progress during task execution into continuous, fine-grained value signals, enabling even failed trajectories to provide effective training information based on their actual completion level, thereby alleviating the difficulty of credit allocation under sparse reward conditions. This method can be applied to embodied intelligence tasks such as robot grasping, handling, placement, object sorting, container manipulation, and dual-arm collaboration. It can also be combined with reinforcement learning algorithms such as proximal policy optimization and group relative policy optimization to improve the training efficiency and generalization ability of embodied intelligence models in complex operational tasks. This invention has strong versatility and scalability, applicable to different robot operation tasks and different reinforcement learning frameworks, and can effectively improve the policy learning efficiency, task success rate, and robustness of embodied intelligence models in complex environments.
Smart Images
Figure CN122693756A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control technology and relates to a method for training embodied intelligence models and optimizing policies using computer vision, deep learning, visual language action models and reinforcement learning techniques. In particular, it relates to an embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance, which can be used to solve problems such as sparse rewards, difficulty in credit allocation and insufficient policy generalization ability in long-term operation tasks of robots. Background Technology
[0002] In recent years, artificial intelligence, robot control, and deep learning technologies have developed rapidly. With the advancement of large-scale visual language models and robot learning techniques, visual language action models have gradually become an important research direction in the field of embodied intelligence. These models can unify visual perception, language instruction understanding, and action generation into a single policy model, enabling robots to complete tasks such as grasping, handling, placing, and assembling based on the external visual environment and natural language task instructions. They have shown great application potential in scenarios such as industrial manufacturing, warehousing and logistics, and home service robots.
[0003] Existing embodied intelligence models typically rely on supervised fine-tuning or imitation learning using large-scale robot demonstration data. By collecting human operation trajectories, the model can learn the mapping relationship from visual observations and verbal commands to robot actions. However, collecting robot demonstration data is costly, and large amounts of data often need to be collected again when facing new environments, new objects, new tasks, or different robot forms. Therefore, relying solely on human demonstration data for model training has certain limitations in terms of scalability, training cost, and generalization ability.
[0004] To address these issues, reinforcement learning has been gradually introduced into the training process of embodied intelligence models. Reinforcement learning can further optimize policies through the interaction between the robot and its environment, freeing the model from complete reliance on human demonstration data, thereby enhancing its autonomous learning and generalization abilities in complex tasks. However, in long-term robotic tasks, reinforcement learning typically only provides binary terminal rewards of success or failure. For example, a reward of 1 for task completion and 0 for failure. This sparse reward fails to reflect the true progress of task execution, making it impossible to distinguish between completely failed trajectories and those that are close to success but not yet completed, leading to severe difficulties in credit allocation during model training.
[0005] To alleviate the sparse reward problem, existing methods typically attempt to design dense reward functions or introduce progress estimation methods based on metrics such as time distance and remaining steps. However, manually designing dense rewards often requires formulating complex rules for specific tasks, object shapes, and robot motion processes, resulting in high design costs and poor transferability and scalability. On the other hand, progress estimation based solely on time steps is difficult to accurately represent the actual contribution of different actions to the degree of task completion. For example, in robot operation, the stages of approaching the target object, successfully grasping the target object, and moving the target object to the designated position have different meanings for task completion, and simple time distance cannot fully express these semantic differences.
[0006] Therefore, existing embodied intelligence reinforcement learning methods still suffer from problems such as sparse reward signals, coarse task progress representation, difficulty in designing artificial rewards, and insufficient policy generalization ability, which limit the practical application effect of embodied intelligence models in complex long-term robot operation tasks. Summary of the Invention
[0007] To overcome the shortcomings of the prior art, this invention provides an embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance, in order to solve the problems of embodied intelligence models in long-term robot operation tasks, such as sparse reward signals, difficulty in measuring task progress, high cost of manual reward design, and insufficient policy generalization ability.
[0008] The technical solution of the present invention is as follows:
[0009] An embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance is characterized by: extracting visual features during robot task execution using a pre-trained visual model; performing semantic stage clustering of the task trajectory based on changes in visual feature similarity to generate structured value labels representing task completion progress; further training an embodied value model to predict the task progress at the current state; and using the predicted value for reward reshaping in the reinforcement learning of the embodied intelligence model, thereby guiding the embodied intelligence model to optimize its strategy; specifically including the following steps:
[0010] S1. Given a robot operation task and corresponding language instructions, collect trajectory data generated when the embodied intelligent model performs the robot operation task. The trajectory data includes visual observation sequence, task language instructions, action information and task termination status.
[0011] S2. Use a pre-trained visual model to encode the visual observation sequence in the trajectory data to obtain the visual feature representation of each moment in the trajectory.
[0012] S3. Calculate the degree of state proximity during task execution based on the similarity between the visual features of each moment in the trajectory and the visual features of the termination state or the local sub-target state.
[0013] S4. Based on the changing trend of visual feature similarity, perform semantic clustering on the task trajectory, divide the complete task trajectory into multiple stages with operational semantics, and generate structured value labels based on stage progress and changes in similarity within stages.
[0014] S5. Train an embodied value model using structured value labels. The embodied value model is used to predict the task completion progress corresponding to the current state based on the current visual observation and task language instructions.
[0015] S6. During the embodied intelligence reinforcement learning training process, the task completion progress predicted by the embodied value model is used to reshape the original sparse terminal reward, and the reshaped reward signal is used for policy optimization of the embodied intelligence model; after multiple rounds of reinforcement learning training, the optimized embodied intelligence model is output.
[0016] Furthermore, the pre-trained visual model in step S2 is a visual world model, a visual Transformer model, a video self-supervised model, or a visual language pre-trained model.
[0017] Furthermore, step S4 specifically includes:
[0018] S4.1 Semantic clustering based on the trend of visual feature similarity changes;
[0019] Starting from the trajectory termination state or the termination state of the currently undivided portion, candidate segmentation points are searched in reverse order within the trajectory. A linear fit is then performed on the trend of visual feature similarity changes within a sliding window. When the slope of the fitted trend exceeds a preset threshold, the corresponding frame is determined as the semantic stage boundary. This method divides the complete task trajectory into multiple stages with operational semantics.
[0020]
[0021] Where N represents the number of semantic stages, C n This represents the nth semantic stage; different semantic stages correspond to the processes of approaching the target object, grasping the target object, moving the target object, aligning with the target area, and releasing or placing the target object in the robot task.
[0022] S4.2, Generate structured value tags;
[0023] Let state t belong to the nth semantic stage C. n The local sub-objective state at this stage is K. n Then the structured value corresponding to state t is represented as:
[0024]
[0025] Among them, Z t Features of the current state For local sub-target state features, This represents the similarity of normalized visual features. Indicates the progress of a phased task. This indicates the degree of proximity of the current state within that stage; thus, a structured value label is generated for each state in the trajectory.
[0026] Furthermore, the embodied value model includes a visual language base model and a value prediction head. The training data inputs of the model include the current visual observation, task language instructions, and prompts; the model output is the predicted value corresponding to the current state.
[0027] Furthermore, in step S6, embodied intelligence reinforcement learning training, when the task is successful, the terminal reward is 1; when the task fails, the termination state of the reinforcement learning interaction trajectory and the task instruction are input to the trained embodied value model. The embodied value model predicts the completion progress of the termination state, and the task completion progress predicted by the embodied value model is used as the progress reward for the failed trajectory. The reshaped reward signal is used to calculate the trajectory reward or advantage function, and the policy parameters of the embodied intelligence model are updated accordingly. The reshaped terminal reward is expressed as follows:
[0028]
[0029] Among them, s T This indicates a terminated state; 'g' represents the task language instruction. The predicted progress of the task's final state is given by the embodied value model. This represents the set of termination states indicating successful task completion. This is an indicator function.
[0030] Furthermore, in step S6, embodied intelligent reinforcement learning training, the reinforcement learning algorithm is a proximal policy optimization algorithm, a group relative policy optimization algorithm, an actor-critic algorithm, or a policy optimization algorithm based on a dominance function.
[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0032] This invention provides an embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance. It automatically extracts visual features from task trajectories through a pre-trained visual model and obtains semantically structured task stage divisions based on the changing trends of these visual features, avoiding the complex process of manually designing dense reward functions. This method can transform intermediate progress during task execution into continuous, fine-grained value signals, enabling even failed trajectories to provide effective training information based on their actual completion level, thereby alleviating the difficulty of credit allocation under sparse reward conditions. This method can be applied to embodied intelligence tasks such as robot grasping, handling, placement, object sorting, container manipulation, and dual-arm collaboration. It can also be combined with reinforcement learning algorithms such as proximal policy optimization and group relative policy optimization to improve the training efficiency and generalization ability of embodied intelligence models in complex operational tasks. This invention has strong versatility and scalability, applicable to different robot operation tasks and different reinforcement learning frameworks, and can effectively improve the policy learning efficiency, task success rate, and robustness of embodied intelligence models in complex environments. Attached Figure Description
[0033] Figure 1 This is a flowchart illustrating the overall process of the embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance of this invention.
[0034] Figure 2 This is a schematic diagram illustrating the process of embodied value model training, progress-aware reward reshaping, and strategy optimization in this invention. Detailed Implementation
[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.
[0036] This invention proposes an embodied intelligence reinforcement learning method based on visual feature semantic clustering and value guidance. The overall flowchart is as follows: Figure 1 As shown, the specific implementation method includes the following steps:
[0037] S1. Collect robot task execution trajectory data.
[0038] Given a robot operation task and corresponding language instructions, trajectory data generated by the embodied intelligence model during task execution is collected. The trajectory data includes visual observation sequences, action sequences, task language instructions, and task termination states. The visual observation sequence can be represented as:
[0039]
[0040] Where T represents the trajectory length, v tThis represents the image or video frame at time t. The task may include, but is not limited to, grasping, handling, placing, sorting, container manipulation, single-arm robot operation, and dual-arm robot collaborative operation.
[0041] S2. Extract visual features using a pre-trained visual model.
[0042] The visual observation sequence obtained in step S1 is input into the pre-trained visual model to extract the visual features corresponding to each time step in the trajectory. Let the encoder of the pre-trained visual model be... Then the visual features at time t are represented as:
[0043]
[0044] Among them, Z t This represents the image features at time t. The pre-trained visual model can be, but is not limited to, a visual world model, a visual Transformer model, a video self-supervised model, or a visual language pre-trained model. In one embodiment, V-JEPA 2 can be used as the pre-trained visual model to perform patch-level visual feature extraction on the trajectory image.
[0045] S3. Calculate the visual feature similarity between the current state and the target state.
[0046] After obtaining the visual features at each time step in the trajectory, the visual feature similarity between the current state and the final state or the local sub-target state is calculated to obtain a visual feature similarity sequence. Let the feature of the current state be Z. t The target state or local sub-target state feature is Z. K Then, the visual feature similarity between the two can be expressed as:
[0047]
[0048] Where P represents the number of visual patches, z t,p z K,p Let represent the feature vectors corresponding to the p-th visual patch and the current local alphabet visual patch, respectively. exp() represents an exponential function with the natural constant e as the base, and β is the scaling parameter. Represents the Manhattan norm. This represents the Euclidean norm. This similarity measure considers both the differences in feature direction and feature magnitude, and is used to measure the degree of closeness of the current state to the target state or a local sub-target state.
[0049] S4. Semantic stage clustering and generation of structured value tags.
[0050] S4.1 Semantic clustering based on the trend of visual feature similarity changes.
[0051] Based on the visual feature similarity sequence obtained in step S3, the robot task trajectory is semantically segmented. Specifically, starting from the termination state of the task trajectory or the termination state of the currently unsegmented portion, candidate segmentation points in the trajectory are searched in reverse, and the trend of visual feature similarity changes is linearly fitted within a sliding window. When the slope of the fitted trend exceeds a preset threshold, the corresponding frame is determined as the semantic stage boundary.
[0052] Using the above method, the complete task trajectory is divided into multiple stages with operational semantics:
[0053]
[0054] Where N represents the number of semantic stages, C n This represents the nth semantic stage. Different semantic stages correspond to processes in robotic tasks such as approaching the target object, grasping the target object, moving the target object, aligning with the target area, and releasing or placing the target object. This semantic clustering process does not rely on manually labeled stages but is automatically completed based on changes in visual features.
[0055] S4.2 Generate structured value tags.
[0056] After semantically segmenting the robot's task trajectory, a structured value label is generated for each state in the trajectory. This value label considers both macroscopic stage progress and fine-grained progress within each stage. Let state t belong to the nth semantic stage C. n The local sub-objective state at this stage is K. n Then the structured value corresponding to state t can be expressed as:
[0057]
[0058] Among them, Z t Features of the current state For local sub-target state features, This represents the similarity of normalized visual features. Indicates the progress of a phased task. This indicates the degree of proximity of the current state within that stage. This allows for the generation of continuous, fine-grained value labels with semantic structure for each state in the trajectory.
[0059] S5, Training the Embodied Value Model.
[0060] like Figure 2 As shown, the embodied value model is trained using the structured value labels generated in step S5. The embodied value model is used to predict the task completion progress corresponding to the current state based on the current visual observation and task language instructions.
[0061] In one embodiment, the embodied value model includes a visual language base model and a value prediction head. The model's training data input includes the current visual observation, task language instructions, and prompts; the model output is the predicted value corresponding to the current state. Optionally, the continuous value interval [0,1] can be discretized into several value levels, transforming the value prediction task into a classification task to improve training stability. The visual language base model can employ, but is not limited to, Qwen-VL, LLaVA, InternVL, CLIP visual language model, or other multimodal models capable of fusing image and language inputs.
[0062] S6. Reward reshaping and strategy optimization for embodied intelligence reinforcement learning.
[0063] S6.1 Re-rewards using embodied value models.
[0064] During reinforcement learning training, the trained embodied value model is used to reshape the original sparse terminal rewards. Traditional methods typically only assign binary terminal rewards based on task success (1 for success, 0 for failure). This approach cannot distinguish the degree of task completion for different failure trajectories.
[0065] like Figure 2 As shown, when a task fails, the termination state of the reinforcement learning interaction trajectory and the task instructions are input into the trained embodied value model. The embodied value model predicts the completion progress of the termination state, and the task completion progress predicted by the embodied value model is used as the progress reward for the failed trajectory. Let the termination state be s. T The task language instruction is 'g', and the embodied value model predicts the task's termination state and completion progress as 'g'. The redesigned terminal reward is:
[0066]
[0067] in, This represents the set of termination states indicating successful task completion. The indicator function is denoted as . When the task succeeds, the terminal reward is 1; when the task fails, the terminal reward is given by the embodied value model based on the task completion progress at the termination state. Thus, the reinforcement learning process can distinguish between completely failed trajectories and near-successful trajectories, thereby utilizing the effective progress information in the failed trajectories.
[0068] S6.2 Optimize strategies based on reshaping rewards.
[0069] The reshaped terminal reward obtained in step S6.1 is used for reinforcement learning training of the embodied intelligence model. The reinforcement learning algorithm may be, but is not limited to, proximal policy optimization, group relative policy optimization, actor-critic algorithm, or other policy optimization algorithms based on the advantage function. During training, the reshaped reward signal is used to calculate the trajectory reward or advantage function, and the policy parameters of the embodied intelligence model are updated accordingly.
[0070] After multiple rounds of reinforcement learning training, an optimized embodied intelligence model is output. This model can generate robot actions based on current visual observations and task language instructions, and is used to complete robot operation tasks such as grasping, carrying, placing, sorting, and dual-arm collaboration.
[0071] In one specific embodiment, the pre-trained visual model uses V-JEPA 2, and the embodied value model consists of a visual language model and a multilayer perceptron value head. Visual feature similarity is used for trajectory semantic stage segmentation, structured value labels are used to train the embodied value model, and the prediction results of the embodied value model are used for reinforcement learning reward reshaping. The above model structure, visual encoder type, clustering threshold, sliding window length, number of discrete value levels, and reinforcement learning algorithm can all be adjusted according to the specific task and robot platform.
[0072] The above methods can be widely applied to embodied intelligence model training, robot operation strategy optimization, visual language action model reinforcement learning, single-arm robot operation, and dual-arm robot collaborative tasks. By transforming the visual structural information in the pre-trained visual model into a value supervision signal that can be utilized in reinforcement learning, this invention can alleviate the difficulty of credit allocation caused by sparse rewards without relying on manually designed dense reward functions, thereby improving the training stability, task success rate, and generalization ability of embodied intelligence models in complex long-term tasks.
[0073] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.
Claims
1. A reinforcement learning method for embodied intelligence based on visual feature semantic clustering and value guidance, characterized in that, Visual features are extracted during robot task execution using a pre-trained visual model. Semantic stage clustering of the task trajectory is then performed based on changes in visual feature similarity to generate structured value labels representing task progress. These structured value labels are used to train an embodied value model to predict the current task progress level. The predicted value is then used for reward reshaping in the reinforcement learning of the embodied intelligence model, guiding policy optimization. The specific steps include: S1. Given a robot operation task and corresponding language instructions, collect trajectory data generated when the embodied intelligent model performs the robot operation task. The trajectory data includes visual observation sequence, task language instructions, action information and task termination status. S2. Use a pre-trained visual model to encode the visual observation sequence in the trajectory data to obtain the visual feature representation of each moment in the trajectory. S3. Calculate the degree of state proximity during task execution based on the similarity between the visual features of each moment in the trajectory and the visual features of the termination state or the local sub-target state. S4. Based on the changing trend of visual feature similarity, perform semantic clustering on the task trajectory, divide the complete task trajectory into multiple stages with operational semantics, and generate structured value labels based on stage progress and changes in similarity within stages. S5. Train an embodied value model using structured value labels. The embodied value model is used to predict the task completion progress corresponding to the current state based on the current visual observation and task language instructions. S6. During the embodied intelligence reinforcement learning training process, the task completion progress predicted by the embodied value model is used to reshape the original sparse terminal reward, and the reshaped reward signal is used for policy optimization of the embodied intelligence model; after multiple rounds of reinforcement learning training, the optimized embodied intelligence model is output.
2. The method as described in claim 1, characterized in that, The pre-trained visual model in step S2 is a visual world model, a visual Transformer model, a video self-supervised model, or a visual language pre-trained model.
3. The method as described in claim 1, characterized in that, Step S4 specifically involves: S4.1 Semantic clustering based on the trend of visual feature similarity changes; Starting from the trajectory termination state or the termination state of the currently undivided portion, candidate segmentation points are searched in reverse order within the trajectory. A linear fit is then performed on the trend of visual feature similarity changes within a sliding window. When the slope of the fitted trend exceeds a preset threshold, the corresponding frame is determined as the semantic stage boundary. This method divides the complete task trajectory into multiple stages with operational semantics. Where N represents the number of semantic stages, C n This represents the nth semantic stage; different semantic stages correspond to the processes of approaching the target object, grasping the target object, moving the target object, aligning with the target area, and releasing or placing the target object in the robot task. S4.2, Generate structured value tags; Let state t belong to the nth semantic stage C. n The local sub-objective state at this stage is K. n Then the structured value corresponding to state t is represented as: Among them, Z t Features of the current state For local sub-target state features, This represents the similarity of normalized visual features. Indicates the progress of a phased task. This indicates the degree of proximity of the current state within that stage; thus, a structured value label is generated for each state in the trajectory.
4. The method as described in claim 1, characterized in that, The embodied value model includes a visual language base model and a value prediction head. The training data inputs of the model include the current visual observation, task language instructions, and prompts. The model output is the predicted value corresponding to the current state.
5. The method as described in claim 1, characterized in that, In step S6, embodied intelligence reinforcement learning training, when the task is successful, the terminal reward is 1; when the task fails, the termination state of the reinforcement learning interaction trajectory and the task instructions are input to the trained embodied value model. The embodied value model predicts the completion progress of the termination state, and the task completion progress predicted by the embodied value model is used as the progress reward for the failed trajectory. The reshaped reward signal is used to calculate the trajectory reward or advantage function, and the policy parameters of the embodied intelligence model are updated accordingly. The reshaped terminal reward is expressed as follows: Among them, s T This indicates a terminated state; 'g' represents the task language instruction. The predicted progress of the task's final state is given by the embodied value model. This represents the set of termination states indicating successful task completion. This is an indicator function.
6. The method as described in claim 1, characterized in that, In step S6, embodied intelligence reinforcement learning training, the reinforcement learning algorithm is a proximal policy optimization algorithm, a group relative policy optimization algorithm, an actor-critic algorithm, or a policy optimization algorithm based on the advantage function.