A world model-based embodied robot vla model strategy optimization method

Abstract

This invention provides a method for policy optimization of embodied robots based on a world model's vision-language-action (VLA) model, belonging to the field of embodied robot technology. The method of this invention integrates multimodal feature preprocessing that combines spatiotemporal context and motion process perception; predicts and generates control trajectory sequences based on a multimodal vision-language-action model; generates video streams of candidate action trajectories based on a generative world model; and performs closed-loop optimization and policy self-evolution based on the evaluation of the predicted video stream. Online policy optimization is implemented in a virtual environment constructed by the generative world model, significantly improving the performance of embodied intelligent robots in complex tasks. Compared with existing behavior cloning methods or traditional reinforcement learning schemes, this invention achieves breakthrough technical effects in terms of safety, sample efficiency, long-term stability, and behavioral emergentness.

Description

Technical Field

[0001] This invention belongs to the field of embodied robot technology, specifically relating to a method for optimizing the vision-language-action (VLA) model of embodied robots based on a world model. Background Technology

[0002] Currently, representative research institutions in this field include Google DeepMind (which proposed the RT-2 series of models), Stanford University (which proposed the OpenVLA model), the University of California, Berkeley (which has in-depth research in the field of robot imitation learning), and ByteDance's Seed team. While the mainstream Vision-Language-Action (VLA) models proposed by these institutions perform well in semantic understanding and cross-task generalization, they still face the following key challenges in practical physical interactions:

[0003] First, existing VLA models rely heavily on high-quality expert demonstration data. Because collecting robot motion data in the real world is extremely costly and includes few failure cases and correction actions, these models often lack the ability to self-correct when faced with unseen disturbances or environmental changes.

[0004] Secondly, strategy optimization based on pure behavior cloning suffers from a serious compound error problem. Because the model lacks explicit modeling of the laws of the physical world, in long-distance, high-precision task execution, small prediction deviations accumulate rapidly over time, eventually causing the robot's execution trajectory to deviate from the target, or even causing safety accidents such as collisions.

[0005] Furthermore, while existing reinforcement learning (RL) optimization schemes aim to address the self-improvement problem, training them directly on real robots faces extremely high sample complexity and hardware overhead. Although some latent space-based world model schemes have been proposed (such as the Dreamer series), these models often suffer from information loss when processing high-resolution pixel information, making it difficult to perfectly align with the visual features learned from large-scale Web data in pre-trained VLA models. Therefore, how to construct a system that can understand physical laws, achieve pixel-level accurate prediction, and optimize VLA policies online without relying on real-world interaction is a pressing technical challenge in the field of embodied intelligence. Summary of the Invention

[0006] The purpose of this invention is to address the key technical problems existing in the execution of embodied intelligence tasks using current vision-language-action models. Addressing the shortcomings mentioned in the background art, such as excessive reliance on massive amounts of expert demonstration data, rapid accumulation of compound errors due to the lack of physical law modeling in long-term tasks, and high sample complexity and high hardware consumption encountered when performing reinforcement learning optimization in real-world environments, this invention aims to provide a policy optimization system and method based on a world model.

[0007] The specific technical problems to be solved by this invention include: First, how to address the insufficient utilization of historical temporal information and lack of motion process perception in existing models. Existing models often focus only on the current single-frame observation when processing dynamic tasks, leading to difficulty in understanding the causal relationship between actions and environmental evolution. This invention aims to construct a world model with spatiotemporal correlation modeling capabilities, achieving deep aggregation of historical operational context and accurate representation of future motion trends, thereby compensating for the model's insufficient understanding of motion continuity in complex physical interactions. Second, how to utilize the alignment mechanism between the "imagined trajectory" and the pre-trained feature space to achieve online policy optimization without real-environment interaction, eliminating compound errors and endowing the model with self-correction capabilities. Third, how to significantly improve the model's generalization performance and sample utilization in complex manipulation tasks using efficient policy improvement algorithms on sparse demonstration data, ultimately achieving high safety and robustness in robot task execution.

[0008] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0009] A method for policy optimization of embodied robot VLA model based on world model includes the following steps:

[0010] Step 1: First, feature extraction is performed on the acquired image data and point cloud data respectively. Then, the image features and point cloud features are fused to obtain an enhanced visual representation. Based on the pre-trained natural language encoder, feature mapping is performed on the acquired natural language instructions. Based on the multimodal encoder, the acquired robot dynamic parameters are mapped into high-dimensional state embedding vectors. The enhanced visual representation, natural language instruction encoding, and state embedding vector are subjected to unified dimensional mapping and spatiotemporal alignment processing to obtain a multimodal spatiotemporal representation vector.

[0011] Step 2: Input the multimodal spatiotemporal representation vector as a state condition signal into the multimodal VLA policy model to obtain a set of candidate action trajectories;

[0012] Step 3: Input the candidate motion trajectory set and the multimodal spatiotemporal representation vector into the generative world model to obtain the predicted video stream;

[0013] Step 4: Based on the multi-dimensional strategy evaluator, the predicted video stream is comprehensively scored, the candidate motion trajectory with the highest comprehensive score is selected as the trajectory to be executed, and it is sent to the robot controller for execution;

[0014] If the overall score of all candidate action trajectories is lower than a preset threshold, a closed-loop optimization process is triggered: the relative advantage value of the group is calculated based on the overall score of each candidate action trajectory, and the network parameters of the multimodal VLA policy model are updated using the group relative policy optimization algorithm.

[0015] Furthermore, step 1 is detailed as follows:

[0016] Step 1.1: Image and Spatial Geometric Coding: Using a pre-trained visual backbone network, image data is transformed into visual feature maps; the spatial geometric coding branch is used to model the local geometric structure of the point cloud, and then the data is fused to obtain an enhanced visual representation.

[0017] Step 1.2: Natural Language Instruction Encoding: The input natural language instruction is feature-mapped using a pre-trained natural language encoder. Then, based on the decoupling mechanism of contrastive learning, the semantic gap between action verbs and target nouns is expanded in the feature space.

[0018] Step 1.3: Robot self-state encoding: Obtain the robot's own dynamic parameters through the internal state monitoring interface, including the current angle of each joint of the robotic arm, the real-time pose of the end effector, the velocity vector of the mobile chassis, and the force feedback information of the end sensor; then map the dynamic parameters into a high-dimensional state embedding vector through the state alignment layer of the multimodal encoder.

[0019] Step 1.4: Construction of multimodal spatiotemporal representation vector: Features obtained from image and spatial geometry encoding, natural language instruction encoding, and robot's own state encoding are respectively input into a learnable linear mapping layer, so that each modal feature is projected to the same latent feature space. Then, temporal position encoding and cross-modal attention mechanism are introduced to align and fuse modal information with different sampling frequencies and different data dimensions to obtain multimodal spatiotemporal representation vector.

[0020] Furthermore, in step 2, during the output stage of the multimodal VLA strategy model, a timing encoder is used to predict the future control trajectory sequence of multiple future time step control commands as candidate action trajectories.

[0021] Furthermore, step 3 is detailed as follows:

[0022] Each candidate action trajectory Represented as a sequence of n consecutive motion control variables:

[0023]

[0024] in Indicates the first The candidate action trajectory in the first The motion control quantity corresponding to each moment, k=1,2,…,n-1; the motion control quantity includes the end-effector pose increment, joint angular velocity, gripper opening / closing amount or movement path parameters;

[0025] Generative world models use multimodal spatiotemporal representation vectors and candidate action trajectories As a conditional input, a sequence of future potential states is generated; this process can be represented as:

[0026]

[0027] in Indicates the first The candidate action trajectory in the first The potential state corresponding to each moment. This represents the latent space state transition function in a generative world model. Represents the parameters of the generative world model;

[0028] After obtaining the sequence of future potential states, corresponding pixel-level prediction frames are generated based on the potential states at each time step; for the The candidate action trajectories are represented as the predicted video stream. ,in, Indicates execution of the first The first candidate action trajectory Visual frames for each predicted time.

[0029] Furthermore, step 4 is detailed as follows:

[0030] For the The comprehensive score of the candidate action trajectories can be expressed as:

[0031]

[0032] in This indicates the score for task completion. Indicates visual semantic alignment score Indicates the physical consistency score. Indicates the smoothness score of motion. This represents the safety constraint score, which is obtained in advance by experts. These are the corresponding weighting coefficients;

[0033] If the highest overall score among all candidate motion trajectories is lower than a preset threshold The system determines that the candidate action trajectories generated by the current multimodal VLA strategy model cannot reliably complete the task and triggers a closed-loop optimization process; if the score of a candidate action trajectory is not lower than a preset threshold... If the system selects the candidate motion trajectory with the highest score as the trajectory to be executed, it will send it to the robot controller for execution.

[0034] Furthermore, the closed-loop optimization process is as follows:

[0035] The system calculates the group relative advantage value based on the comprehensive score of each candidate action trajectory. For the first... The candidate action trajectory and its dominance value Represented as:

[0036]

[0037] in Indicates the same batch The average score of the candidate motion trajectories, Indicates the standard deviation of the rating. To prevent the stability coefficient from being zero;

[0038] Subsequently, the system updates the VLA policy model network parameters using the group relative policy optimization algorithm; let the VLA policy model network parameters be... , No. The probability of generating each candidate action trajectory under the current policy is: The strategy optimization loss is then expressed as:

[0039] when At that time, parameter updates will improve the candidate action trajectory. The generation probability when; At that time, parameter updates will reduce the candidate action trajectory. The generation probability;

[0040] The parameter update process is represented as follows ,in This represents the learning rate.

[0041] This invention proposes a VLA policy optimization method based on a world model. By implementing online policy optimization in a virtual environment constructed using a generative world model, it significantly improves the performance of embodied intelligent robots in complex tasks. Compared to existing behavior cloning methods or traditional reinforcement learning schemes, this invention achieves groundbreaking technical results in terms of security, sample efficiency, long-term stability, and behavioral emergentness.

[0042] 1. Completely solves the problem of compound error accumulation in long-range embodied tasks: In existing VLA models based on behavior cloning, robots often exhibit "frame-by-frame decision-making" characteristics, lacking long-term expectations of the consequences of actions. Since the model has only seen perfect expert data during training, once a small prediction deviation occurs during execution, the deviation will rapidly amplify over time, causing the trajectory to deviate completely from the target (i.e., the compound error problem).

[0043] The technical solution of this invention introduces forward dynamics modeling, enabling the model to "predict the future." Before executing an action, the world model predicts the visual state for several steps ahead. This prediction mechanism acts as a "correction device" for the policy: when the prediction results indicate that the current action sequence will lead to the robotic arm colliding with or deviating from an object, the policy optimizer adjusts the action probability distribution in real time using the GRPO algorithm. Experimental data shows that in complex manipulation tasks of more than 50 steps, the trajectory preservation rate of this invention is improved by approximately 65% ​​compared to the benchmark OpenVLA model. This suppression of compound errors ensures that the robot exhibits extremely high robustness when handling long-range tasks such as "placing objects across multiple obstacles."

[0044] 2. Significantly reduces the reliance on and wear and tear of embodied intelligence training to real hardware interaction: Traditional reinforcement learning methods typically require thousands of trial-and-error interactions in real physical environments. For robot hardware, this training method means extremely high wear and tear risks, high power consumption, and long time costs. In addition, collision errors in the real environment can cause permanent damage to expensive sensors or actuators.

[0045] The core contribution of this invention lies in constructing a VLA model based on world model predictions, thereby creating a perfect "offline simulation environment" in the digital space. By synthesizing virtual trajectories and calculating the relative advantage of the group in the world model, the VLA policy can complete policy iteration in a purely virtual environment. According to tests, this invention, using only a small amount of expert demonstration data (Few-shot), achieves the success rate that traditional RL methods require tens of thousands of hours of real data collection through the "virtual self-game" of the world model. This technical effect not only greatly shortens the development cycle of embodied intelligence, but also fundamentally eliminates the constraint of hardware consumption on algorithm evolution.

[0046] 3. Achieves a deep understanding of physical common sense and automatic emergence of behavioral error correction: While existing VLA models understand instructions at the semantic level, they often lack "physical intuition," such as not knowing that gravity causes suspended objects to fall, or that hard collisions cause objects to displace. This invention internalizes these fundamental physical laws into the model's predictive priors by pre-training a generative world model on massive amounts of data.

[0047] This physical understanding leads to an unexpected and highly valuable technological effect: the automatic emergence of behavioral correction. When a robot performs a task (such as grasping an apple), if the initial attempt fails due to external interference (such as human intervention), traditional business logic (BC) models typically freeze or report errors because they are stuck in an unseen state. However, the framework of this invention, having learned the transition logic from the failure state to the success state in virtual rehearsals, can automatically generate corrective actions—for example, after missing a target, the model spontaneously adjusts its posture and makes a second attempt based on the prediction feedback from the world model. This "self-correction after failure" is not manually labeled by experts, but rather a high-order intelligent performance spontaneously learned by the model through policy optimization after understanding the "gap between the current state and the target state" through the world model.

[0048] 4. Achieves a high degree of consistency between the visual feature space and task semantics: Unlike previous schemes that performed abstract predictions in the latent space, this invention insists on using pixel-level prediction combined with VLA feature alignment. This design ensures a dual technical effect: First, visual realism. The high-definition visual frames recovered through the diffusion model architecture include the texture, lighting, and shadows of objects, allowing the evaluator to extract features using pre-trained large visual models (such as SigLIP), ensuring the richness of the reward signal. Second, semantic relevance. Since the prediction results are pixel-level, they can be directly compared with the massive amount of Web visual-language knowledge learned by the pre-trained VLA model. This means that the model does not need to relearn what a "cup" or "basket" is, but can directly use existing semantic knowledge to determine whether the predicted future state conforms to the task instructions. This alignment mechanism makes this invention extremely generalizable in open scenes, achieving a success rate 40% higher than pure latent space prediction schemes when facing unseen object categories.

[0049] 5. High-efficiency convergence and stability brought by the GRPO algorithm: In the policy optimization stage, the swarm relative policy optimization algorithm introduced in this invention shows significantly better technical stability than the traditional PPO algorithm. In embodied intelligence scenarios, traditional RL often leads to training failure due to excessively sparse reward functions or inaccurate estimation of value functions.

[0050] The GRPO algorithm of this invention calculates relative advantage by comparing action candidate groups generated under the same observation state. This relative evaluation mechanism naturally offsets the fluctuation of baseline reward under different task backgrounds. Its technical effects are manifested in: an extremely smooth loss function curve during training and a clearer gradient update direction. Simultaneously, by eliminating the complex value network training, the system's memory overhead is reduced by approximately 30%, making it feasible to deploy complex VLA policy optimization on a single mainstream server. This improvement in algorithm efficiency directly translates into the model's efficient learning ability for complex instructions (such as multi-step inference actions).

[0051] 6. Enhanced robustness of embodied intelligence under long-term operation: In practical applications, robots need to work continuously for extended periods, and subtle shifts can occur due to environmental lighting, object placement, and wear on the robotic arm. Traditional static strategies struggle to adapt to such dynamic changes. This invention utilizes real-time feedback provided by a world model, enabling the robot to perform minute "hypotheses and verifications" before each frame of action. This online optimization characteristic endows the system with extremely strong adaptability to dynamic environments. Even under sudden changes in lighting or environmental background interference, the system can still identify physically stable action sequences through the predictive feedback of the world model, thereby maintaining a very high success rate for long-term operation.

[0052] In summary, this invention, through the implementation of the embodied framework, not only significantly outperforms existing benchmarks in academic metrics but also solves the data bottlenecks, security risks, and robustness challenges in the industrial application of embodied intelligence. Its physical understanding, self-correcting capabilities, and extremely high sample utilization provide crucial technical support for realizing truly universal service robots. Attached Figure Description

[0053] Figure 1 This is a flowchart of the method of the present invention.

[0054] Figure 2 This is a schematic diagram of a six-axis robotic arm. Detailed Implementation

[0055] This invention provides a method for policy optimization of a vision-language-action model based on a world model. This technical solution solves the core problems of compound error accumulation and low sample efficiency in existing VLA models for long-range embodied tasks by constructing a pixel-level world model with physical common sense and performing online policy optimization in its virtual environment.

[0056] The core scientific principle of this invention lies in decoupling the ability to "imagine the future" into an independent world model, providing physically consistent feedback signals for end-to-end policies. The system mainly consists of three parts: a vision-language-action policy network, a generative world model, and a policy optimizer. Its basic logical chain is as follows: the policy network generates candidate actions based on current observations; the world model predicts the future state after executing the action in virtual space; and the optimizer iteratively improves the policy by evaluating the alignment between the predicted state and the target task.

[0057] This architecture draws inspiration from the human brain's "visuokinetic" decision-making mechanism, which involves mentally rehearsing the potential consequences of an action before taking it. In the realm of embodied intelligence, this means that the model no longer mechanically imitates expert actions, but rather understands the physical intentions behind the actions and the environmental feedback.

[0058] In the technical solution provided by this invention, the core task of the world model is to construct a highly fitting physical evolution simulator, enabling it to deeply understand and learn the essential evolutionary laws of the physical world. To achieve this goal, this invention abandons the traditional rule-driven modeling approach and instead adopts a generative architecture based on self-supervised learning as the underlying logic. This embodiment provides a method for optimizing the policy of an embodied robot VLA model based on a world model, as follows: Figure 1 As shown, the specific steps include:

[0059] Step 1: Preprocessing of multimodal features that integrate spatiotemporal context and motion process perception.

[0060] The primary step of this invention lies in constructing a high-dimensional spatiotemporal representation system capable of deeply analyzing the causality of the physical environment, the continuity of motion, and the intent of the task. At the core of this system is a multimodal encoder architecture, designed to transform diverse heterogeneous information from the physical world into a unified feature space that the robot can understand and process. In real-world scenarios of embodied intelligence task execution, the system uses high-precision visual sensors, force feedback sensors, and a natural language input interface to collect real-time environmental observation information, the robot's own state, and human user commands.

[0061] Step 1.1: Image and spatial geometry coding.

[0062] In the visual perception dimension, the multimodal encoder first extracts features from the real-time acquired high-resolution color observation images. The system utilizes a pre-trained high-performance visual backbone network, employing a deep convolutional neural network or visual transformer architecture, to transform pixel-level raw information into visual feature maps with high-order semantics. Simultaneously, the system acquires point cloud data of the environment and uses a spatial geometry coding branch to model the local geometric structure of the point cloud. This process accurately captures the edge constraints of target objects, the spatial poses of obstacles, and the topological relationships of the scene. Through an RNN encoder and a Pointnet-MSE encoder, the two-dimensional image features are fused with the three-dimensional point cloud data at multiple scales, thereby constructing an enhanced visual representation containing depth information and object attributes, providing the perceptual foundation for the robot.

[0063] Step 1.2: Natural language instruction encoding.

[0064] To achieve a deep understanding of complex task objectives, the multimodal encoder integrates a high-performance instruction processing branch based on a pre-trained large language model. This branch is responsible for receiving and parsing natural language instructions from human users. During the instruction processing of the multimodal encoder, the system employs a combination of deep semantic parsing and cross-modal alignment. First, the natural language processing unit uses the pre-trained natural language encoder to perform feature mapping on the input raw instructions, completing not only basic analyses such as word segmentation and part-of-speech tagging, but also introducing a decoupling mechanism based on contrastive learning. This mechanism expands the semantic gap between action verbs and target nouns in the feature space by constructing positive and negative sample pairs, achieving fine-grained decomposition of task intent. For example, the dynamic operation of "picking" is decoupled from the static anatomical structure of "apple," ensuring that the model can accurately extract the core physical logic in the instruction. In subsequent step 1.4, these decoupled language embedding vectors are projected onto a joint latent space consistent with the visual feature dimension through a learnable linear mapping layer. Within this space, the system deploys a multi-layered cross-attention mechanism, using linguistic features as guiding signals to globally scan the visual features of image information. In this way, the system can automatically identify and lock onto specific regions or target poses related to instructions within the visual scene, achieving pixel-level alignment at the "language-visual" level. This alignment process, by calculating the attention weight distribution, strengthens spatial features related to the task objective, thus providing clear semantic guidance for subsequently generating candidate actions that better align with expert operational logic. This means the encoder can automatically locate the subject and object mentioned in the instruction within the visual scene, ensuring that subsequent decisions fully conform to human task expectations, achieving a precise mapping from abstract instructions to specific visual targets.

[0065] Step 1.3: Robot's own state encoding.

[0066] In terms of ontological perception, the encoder acquires the robot's own dynamic parameters through an internal state monitoring interface. This includes the current angles of each joint of the robotic arm, the real-time pose of the end effector, the velocity vector of the mobile chassis, and force feedback information from the end sensor. The multimodal encoder utilizes a state alignment layer to map these low-dimensional numerical vectors into high-dimensional state embedding vectors, thereby ensuring feature consistency. By cascading its own state features with external environment features, the system can assess the relative orientation of the robotic arm to environmental targets and the current motion margin in real time. This "body-environment" fusion encoding method enables the model to understand its own motion capability boundaries, thereby generating safer and more reasonable candidate actions. In this process, the encoder introduces a temporal aggregation operator to extract features from the state evolution at historical moments, thereby not only capturing the robot's current static configuration but also resolving its own acceleration, momentum changes, and historical operational inertia. The key to this step is that it deeply integrates the robot's spatiotemporal "self-awareness" into the environmental perception system. By cascading these self-state characteristics, which incorporate historical trends, with external environmental features, the system can assess the dynamic relative position between the robotic arm and environmental targets in real time and accurately calculate the current motion margin and dynamic boundaries. This "body-environment" fusion encoding method, which combines historical temporal information, enables the model to deeply understand its own motion capability boundaries and their evolution in physical space. This allows it to effectively avoid execution deviations caused by neglecting motion inertia in complex dynamic interactive tasks, generating candidate action sequences with higher physical consistency, safety, and operational rationality. Furthermore, this temporal state encoding provides a necessary reference frame for subsequent identification of potential physical conflicts, ensuring that the robot maintains posture smoothness and trajectory robustness when performing long-range tasks.

[0067] Step 1.4: Construction of multimodal spatiotemporal representation vectors.

[0068] After completing image and spatial geometry encoding, natural language command encoding, and robot state encoding, the system performs unified dimensional mapping and spatiotemporal alignment processing on different modal features. Specifically, image features, point cloud spatial geometry features, natural language command features, and robot state features are input into a learnable linear mapping layer, so that each modal feature is projected onto the same latent feature space. Subsequently, temporal position encoding and cross-modal attention mechanisms are introduced to align and fuse modal information with different sampling frequencies and different data dimensions, so that the visual environment state, task semantic intent, and robot motion state form a correspondence at the same time.

[0069] After the above processing, the system obtains a multimodal spatiotemporal representation vector. This vector simultaneously contains current environmental observation information, task instruction semantic information, robot body state information, and historical motion trend information, which are used as unified input parameters for subsequent VLA policy networks and generative world models.

[0070] Step 2: Control trajectory sequence prediction and generation based on multimodal vision-language-action model

[0071] The multimodal spatiotemporal representation vector output in step 1 is obtained. Subsequently, the system inputs these as state condition signals into the multimodal VLA policy model to perform long-range motion planning and trajectory sequence prediction. This multimodal policy model is based on a transformer self-attention mechanism architecture. Its core task is to perform cross-modal multidimensional alignment and spatiotemporal correlation modeling by fusing high-dimensional spatial geometric features, sequential image features, robot historical dynamic state features, and human natural language command features through a backbone network. In the specific forward inference process, the multimodal policy model analyzes the environmental evolution trend and task semantic association implied by the aforementioned multimodal spatiotemporal representation vectors, and autoregressively predicts the multi-step control parameters that best fit the expert teaching paradigm in the current state.

[0072] The key technical approach in this step lies in the fact that, during the control output phase, the multimodal policy model abandons the traditional single-frame real-time motion mapping mode and instead utilizes a temporal encoder to directly predict and output a sequence of future control trajectories containing control commands for multiple consecutive future time steps. This sequence of future control trajectories is formally represented as a high-dimensional temporal motion tensor, which implicitly encodes the robot's end effector's three-dimensional spatial pose change vector, joint angle rotation matrix, and motion rate parameters within consecutive future time windows. Through this serialized trajectory prediction mechanism based on multimodal input, the system can construct a virtual control path that guides the long-term development of the task before issuing actual physical commands to the hardware actuator. This provides high-confidence baseline trajectory samples for subsequent evolutionary simulation, safety verification, and policy adaptive error correction using a world model, significantly improving the robot's motion smoothness and path planning robustness in unstructured interactive environments.

[0073] Step 3: Generate video stream based on candidate motion trajectories from a generative world model.

[0074] After obtaining the candidate action trajectory set output in step 2, the system combines the candidate action trajectory set with the multimodal spatiotemporal representation vector obtained in step 1.4. Both are input into the generative world model. The set of candidate action trajectories is denoted as... ,in This represents the number of candidate action trajectories generated in parallel under the same task state. Indicates the first 10 candidate motion trajectories. Each candidate motion trajectory... It can be represented as a sequence of multiple continuous motion control variables:

[0075]

[0076] in Indicates the first The candidate action trajectory in the first The motion control quantity corresponding to each moment, k=1,2,…,n-1; the motion control quantity includes the end-effector pose increment, joint angular velocity, gripper opening / closing amount, or movement path parameters.

[0077] Generative world models are used to simulate the visual evolution of a robot after it executes candidate action trajectories in a real physical environment. Specifically, generative world models use multimodal spatiotemporal representation vectors. and candidate action trajectory As a conditional input, a sequence of future potential states is first generated through a latent space dynamics prediction unit. This process can be represented as follows:

[0078]

[0079] in Indicates the first The candidate action trajectory in the first The potential state corresponding to each moment. This represents the latent space state transition function in a generative world model. Represents the parameters of the world model.

[0080] After obtaining the sequence of future potential states, the video frame generation unit generates corresponding pixel-level predicted frames based on the potential states at each time step. For the ... The candidate action trajectories and their predicted video stream can be represented as:

[0081]

[0082] in , This represents a video frame generation unit. Indicates video generation parameters, Indicates execution of the first The first candidate action trajectory Visual frames for each predicted time.

[0083] Through the above processing, the system can identify each candidate motion trajectory. Generate the corresponding predicted video stream This allows for the prediction of visual execution results that might result from different candidate action trajectories before the actual action is executed. The predicted video stream... It includes not only the movement process of the robotic arm itself, but also the pose changes of the target object, the relative position changes of obstacles, the contact process, the collision risk, and the trend of task objective completion, providing input basis for evaluation, threshold judgment, closed-loop optimization and strategy self-evolution in step 4.

[0084] Step 4: Closed-loop optimization and strategy self-evolution based on predicted video stream evaluation

[0085] The system will generate predicted video streams corresponding to each candidate motion trajectory in step 3. The input is fed into a multi-dimensional strategy evaluator, which comprehensively scores the execution effect of each candidate action trajectory. The multi-dimensional strategy evaluator includes a task completion evaluation unit, a visual-semantic alignment evaluation unit, a physical consistency evaluation unit, a motion smoothness evaluation unit, and a safety constraint evaluation unit. For the first... The comprehensive score of the candidate action trajectories can be expressed as:

[0086]

[0087] in This indicates the score for task completion. $S_i^{phys}$ represents the visual semantic alignment score, and $S_i^{phys}$ represents the physical consistency score. Indicates the smoothness score of motion. This represents the safety constraint score, which is obtained in advance by experts. These are the corresponding weighting coefficients.

[0088] The system scores the candidate motion trajectories based on a comprehensive evaluation. Determine if the current strategy output meets the task execution requirements. If the highest score among all candidate action trajectories is lower than a preset threshold. That is, satisfying If the system determines that the candidate action trajectory generated by the current VLA policy network cannot reliably complete the task, it will trigger a closed-loop optimization process; if the score of a candidate action trajectory is not lower than a preset threshold, the system will proceed accordingly. If the system selects the candidate motion trajectory with the highest score as the trajectory to be executed, it will send it to the robot controller for execution.

[0089] During the closed-loop optimization process, the system calculates the group's relative advantage value based on the comprehensive score of each candidate action trajectory. For the first... The dominance value of a candidate action trajectory can be expressed as:

[0090]

[0091] in Indicates the same batch The average score of the candidate motion trajectories, Indicates the standard deviation of the rating. To prevent the stability coefficient from being zero, the system determines the strategy update direction based on the relative merits of candidate action trajectories within the same batch, rather than relying on absolute reward values.

[0092] Subsequently, the system updates the VLA policy network parameters using the group relative policy optimization algorithm. Let the VLA policy network parameters be... , No. If the generation probability of a candidate action trajectory under the current policy is $\pi_\omega(a_i|H_t)$, then the policy optimization loss can be expressed as:

[0093]

[0094] when At that time, parameter updates will improve the candidate action trajectory. The generation probability when; At that time, parameter updates will reduce the candidate action trajectory. The generation probability. The parameter update process can be represented as... ,in This represents the learning rate.

[0095] Through the aforementioned iterative process of "candidate action trajectory generation - predicted video stream generation - video stream evaluation - threshold judgment - policy parameter update," the VLA policy network can continuously adjust its parameters based on the virtual execution feedback provided by the generative world model. This gradually increases the probability of generating high-scoring action trajectories while gradually decreasing the probability of generating low-scoring action trajectories or those with risks of collision, deviation, or task failure. Thus, the system achieves closed-loop optimization and policy self-evolution based on predicted video stream feedback without requiring extensive trial-and-error in real-world environments. Specific Implementation

[0097] Figure 2The six-axis robotic arm structure shown is the physical core for performing compliant scanning and strategy verification in this embodiment of the invention. Its overall architecture consists of a base, six rotary joints, and an end effector. The bottom of the robotic arm is firmly mounted on the base, establishing a global reference coordinate system to provide a benchmark for the kinematic calculations of the entire system. Extending upwards from the base, the first joint is responsible for overall rotation in the horizontal plane, while the subsequent two joints determine the probe's extension radius and working height in three-dimensional space through the pitch movements of the upper and lower arms. The three-axis wrist joint assembly at the end of the robotic arm forms a spherical motion space, enabling precise attitude adjustment of the end tool in three dimensions, ensuring that the ultrasound probe conforms to the human body surface at the optimal beam angle. In the specific control logic, each joint of the robotic arm integrates a high-precision encoder to provide real-time feedback of joint angle information, while a six-dimensional force sensor installed between the end flange and the actuator is responsible for capturing minute fluctuations in contact force. This hardware data is uniformly transmitted in real time to the central control system, where the motion commands output by the multimodal VLA strategy model are converted into precise angular displacements of the joint motors. Specifically, the tool coordinate system in this structure is defined at the scanning center of the ultrasound probe. Through the "pre-play" function of the world model, the system can simulate the movement trajectory of the structure in physical space in advance before the action is issued, thereby effectively avoiding the risk of mechanical limitation and ensuring compliance and safety when interacting with the human body.

[0098] In the core operation flow of the system, this embodiment details the online pre-simulation and policy update mechanism based on the world model for the real-time execution and policy iteration of embodied intelligent tasks. Upon receiving a task instruction (such as "grab the black tweezers on the table and place them in a sterile box"), the system does not directly drive the six-axis robotic arm shown in the diagram to perform the action. Instead, it first initiates a "conception" program within the digital twin space. At this time, the multimodal VLA policy model generates G sets of candidate action sequences based on the initial visual observation s(t) captured by the current RGB camera, using its internal probability distribution sampling. These candidate actions have a certain diversity in the parameter space, representing different path attempts to bypass obstacles or different grasping angles. Subsequently, these action groups are injected in parallel into the generative world model. The world model uses its pre-trained physical prior knowledge to forward extrapolate the pixel-level visual frames for the next n steps for each action sequence. This extrapolation process not only simulates the motion trajectory of each joint of the robotic arm in space but also accurately simulates the physical feedback when the end effector contacts the tweezers, such as whether the tweezers slip due to uneven force or whether displacement occurs due to collision. Through this pixel-level simulation, the system can intuitively "see" the potential consequences of each candidate decision in the future.

[0099] After acquiring multiple sets of predicted video streams generated in step 3, the system enters the crucial value assessment, threshold judgment, and closed-loop optimization stages. This embodiment employs a multi-dimensional reward evaluation system to comprehensively evaluate the predicted video stream corresponding to each candidate motion trajectory. First, a pre-trained visual feature encoder is used to extract semantic features of key predicted frames in the predicted video stream, and these features are compared with the semantic features of the target success state to assess the likelihood of the candidate motion trajectory achieving the task objective. Second, the system analyzes the pixel flow smoothness, target object pose continuity, robotic arm motion continuity, and contact process stability of the predicted video stream over time to determine whether the candidate motion trajectory exhibits any abrupt changes, clipping, collisions, or abnormal displacements that violate physical principles. Third, the system, in conjunction with safety constraints, identifies and deducts points for potential mechanical limitations, abnormal contact forces, target object slippage, and obstacle interference in the predicted video stream.

[0100] After completing the above evaluation, the system obtains a comprehensive score for each candidate action trajectory and compares each comprehensive score with a preset threshold. If a candidate action trajectory has a score that meets the threshold requirement, the system selects the candidate action trajectory with the highest score as the trajectory to be executed. If the scores of all candidate action trajectories are lower than the preset threshold, it indicates that the candidate action trajectories generated by the current VLA policy network cannot reliably complete the task, and the system triggers a closed-loop optimization and policy self-evolution process. During the closed-loop optimization process, the system applies a population relative policy optimization algorithm to perform a horizontal comparison of multiple candidate action trajectories within the same batch, identify action features that are more likely to produce successful execution results, and feed the evaluation results back to the VLA policy network through a backpropagation algorithm, thereby increasing the generation probability of high-scoring candidate action trajectories and decreasing the generation probability of low-scoring or potentially dangerous candidate action trajectories.

[0101] Through multiple rounds of iterations involving "candidate motion trajectory generation, predicted video stream generation, video stream evaluation, threshold judgment, and strategy parameter update," the policy network can adaptively optimize based on virtual execution feedback before the actual action is issued, thereby learning a better manipulation strategy. Once closed-loop optimization is complete and candidate motion trajectories with scores meeting the threshold requirements exist, the system selects the candidate motion trajectory with the highest overall score and sends it to the robotic arm controller. During the physical execution phase, the servo motors of each joint of the six-axis robotic arm are precisely synchronized according to the control commands in the candidate motion trajectories, driving the end effector to approach the target point.

[0102] Meanwhile, the system maintains a continuous perception loop, with force sensors monitoring contact force in real time and vision sensors continuously acquiring environmental observation images. If the actual execution state deviates from the expected execution result represented by the predicted video stream beyond the allowable range—for example, the contact force exceeds a safety threshold, the target object slides unexpectedly, the robotic arm trajectory deviates significantly from the predicted path, or new obstacles appear in the scene—the system immediately triggers a replanning mechanism, restarting the candidate action trajectory generation, predicted video stream generation, evaluation, and closed-loop optimization process based on the generative world model.

[0103] This dynamic implementation path enables the robot to not only perform virtual rehearsals before issuing actions ("think before acting"), but also to continuously adjust its strategies based on actual perception feedback during action execution. In long-cycle autonomous operation tasks, this closed-loop control architecture based on generative world models and predictive video stream evaluation can effectively suppress the accumulation of compound errors. By combining pixel-level predictive video stream evaluation with group relative policy optimization, the robot can improve task success rate, action smoothness, and environmental adaptability in unseen open scene tasks, ultimately achieving adaptive interactive control with the ability to understand physical common sense.

[0104] Through the iterative process of the above steps, this invention achieves the self-evolution of embodied intelligent strategies without requiring massive amounts of negative samples from the real environment. In practical deployment, those skilled in the art can flexibly adjust the sampling step size or population size of the world model according to the limitations of different computing platforms to achieve the best balance between real-time response speed and strategy optimization depth. This highly scalable implementation provides a solid technical guarantee for general-purpose service robots to perform complex operations in unstructured environments.

[0105] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A strategy optimization method for a VLA model of an embodied robot based on a world model, characterized in that, Includes the following steps: Step 1: First, feature extraction is performed on the acquired image data and point cloud data respectively. Then, the image features and point cloud features are fused to obtain an enhanced visual representation. Based on the pre-trained natural language encoder, feature mapping is performed on the acquired natural language instructions. Based on the multimodal encoder, the acquired robot dynamic parameters are mapped into high-dimensional state embedding vectors. The enhanced visual representation, natural language instruction encoding, and state embedding vector are subjected to unified dimensional mapping and spatiotemporal alignment processing to obtain a multimodal spatiotemporal representation vector. Step 2: Input the multimodal spatiotemporal representation vector as a state condition signal into the multimodal VLA policy model to obtain a set of candidate action trajectories; Step 3: Input the candidate motion trajectory set and the multimodal spatiotemporal representation vector into the generative world model to obtain the predicted video stream; Step 4: Based on the multi-dimensional strategy evaluator, the predicted video stream is comprehensively scored, the candidate motion trajectory with the highest comprehensive score is selected as the trajectory to be executed, and it is sent to the robot controller for execution; If the overall score of all candidate action trajectories is lower than a preset threshold, a closed-loop optimization process is triggered: the relative advantage value of the group is calculated based on the overall score of each candidate action trajectory, and the network parameters of the multimodal VLA policy model are updated using the group relative policy optimization algorithm.

2. The method for optimizing the VLA model strategy of an embodied robot based on a world model according to claim 1, characterized in that, Step 1 is described in detail as follows: Step 1.1: Image and Spatial Geometric Coding: Using a pre-trained visual backbone network, image data is transformed into visual feature maps; the spatial geometric coding branch is used to model the local geometric structure of the point cloud, and then the data is fused to obtain an enhanced visual representation. Step 1.2: Natural Language Instruction Encoding: The input natural language instruction is feature-mapped using a pre-trained natural language encoder. Then, based on the decoupling mechanism of contrastive learning, the semantic gap between action verbs and target nouns is expanded in the feature space. Step 1.3: Robot self-state encoding: Obtain the robot's own dynamic parameters through the internal state monitoring interface, including the current angle of each joint of the robotic arm, the real-time pose of the end effector, the velocity vector of the mobile chassis, and the force feedback information of the end sensor; then map the dynamic parameters into a high-dimensional state embedding vector through the state alignment layer of the multimodal encoder. Step 1.4: Construction of multimodal spatiotemporal representation vector: Features obtained from image and spatial geometry encoding, natural language instruction encoding, and robot's own state encoding are respectively input into a learnable linear mapping layer, so that each modal feature is projected to the same latent feature space. Then, temporal position encoding and cross-modal attention mechanism are introduced to align and fuse modal information with different sampling frequencies and different data dimensions to obtain multimodal spatiotemporal representation vector.

3. The method for optimizing the strategy of a VLA model of an embodied robot based on a world model according to claim 2, characterized in that, In step 2, during the output stage of the multimodal VLA strategy model, a timing encoder is used to predict the future control trajectory sequence of multiple future time step control commands as candidate action trajectories.

4. The method for optimizing the VLA model strategy of an embodied robot based on a world model according to claim 3, characterized in that, Step 3 is as follows: Each candidate action trajectory Represented as a sequence of n consecutive motion control variables: in Indicates the first The candidate action trajectory in the first The motion control quantity corresponding to each moment, k=1,2,…,n-1; the motion control quantity includes the end-effector pose increment, joint angular velocity, gripper opening / closing amount or movement path parameters; Generative world models use multimodal spatiotemporal representation vectors and candidate action trajectories As a conditional input, a sequence of future potential states is generated; this process can be represented as: in Indicates the first The candidate action trajectory in the first The potential state corresponding to each moment. This represents the latent space state transition function in a generative world model. Represents the parameters of the generative world model; After obtaining the sequence of future potential states, corresponding pixel-level prediction frames are generated based on the potential states at each time step; for the The candidate action trajectories are represented as the predicted video stream. ,in, Indicates execution of the first The first candidate action trajectory Visual frames for each predicted time.

5. The method for optimizing the strategy of a VLA model of an embodied robot based on a world model according to claim 4, characterized in that, Step 4 is as follows: For the The comprehensive score of the candidate action trajectories can be expressed as: in This indicates the score for task completion. Indicates visual semantic alignment score Indicates the physical consistency score. Indicates the smoothness score of motion. This represents the safety constraint score, which is obtained in advance by experts. These are the corresponding weighting coefficients; If the highest overall score among all candidate motion trajectories is lower than a preset threshold The system determines that the candidate action trajectories generated by the current multimodal VLA strategy model cannot reliably complete the task and triggers a closed-loop optimization process; if the score of a candidate action trajectory is not lower than a preset threshold... If the system selects the candidate motion trajectory with the highest score as the trajectory to be executed, it will send it to the robot controller for execution.

6. The method for optimizing the strategy of a VLA model of an embodied robot based on a world model according to claim 5, characterized in that, The closed-loop optimization process is as follows: The system calculates the group relative advantage value based on the comprehensive score of each candidate action trajectory. For the first... The candidate action trajectory and its dominance value Represented as: in Indicates the same batch The average score of the candidate motion trajectories, Indicates the standard deviation of the rating. To prevent the stability coefficient from being zero; Subsequently, the system updates the VLA policy model network parameters using the group relative policy optimization algorithm; let the VLA policy model network parameters be... , No. The probability of generating each candidate action trajectory under the current policy is: The strategy optimization loss is then expressed as: when At that time, parameter updates will improve the candidate action trajectory. The generation probability when; At that time, parameter updates will reduce the candidate action trajectory. The generation probability; The parameter update process is represented as follows ,in This represents the learning rate.

Images