Prediction system and method based on a world action language model aligned with the size of the brain collaboration

Abstract

This invention discloses a prediction system and method based on a world action language model with cerebellar and cerebellar co-alignment. The system acquires observed images of the physical world using machine vision, obtains the robot's current state parameters and received task text instructions, and extracts features from the images, state, and text. The extracted features are fused to obtain global features. High-level semantic planning reasoning is performed on the global features to generate high-level semantic planning text that conforms to physical laws and includes task breakdown steps, action sequence logic, physical constraints, and task target achievement standards. Based on the global features and the corresponding high-level semantic planning text, lexical features of the robot's subsequent low-level action instructions are generated, and the decoding actions are guided by these lexical features, converting the high-level semantic planning text into low-level action decisions. After the low-level action decisions are executed, observation feedback is obtained based on the observed image at the next moment of the task to adjust the high-level semantic planning until the task is completed.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and large model technology, and in particular to a prediction system and method based on a world action language model with cerebellar and cerebellar coordination. Background Technology

[0002] World models are a core technology in the fields of embodied intelligence and robot control. By learning the dynamic evolution of the physical world, they can predict and model the future state of the world, providing robots with "internal rehearsal" capabilities. Early world models mostly focused on future predictions based on pure vision or state space, only learning how the world changes without deep integration with robot motion generation, making them difficult to use directly for real-time robot control.

[0003] Building upon this foundation, World Action Models (WAMs) further integrate world state prediction and action generation into a joint model. Representative works such as DreamZero use pre-trained video diffusion as a backbone, simultaneously outputting future videos and robot actions given observations and verbal commands. This forces the model to learn physical laws and improves zero-shot generalization ability. These WAMs transform robot control into a joint task of video and action generation, significantly outperforming traditional vision-language-action (VLA) models. They can learn from heterogeneous, non-repetitive data, achieving cross-scene, cross-task, and cross-embodiment transfer.

[0004] However, this type of world action model still has fundamental flaws: lacking a high-level macroscopic understanding module centered on language, existing world action models primarily rely on video-action generation for prediction. While they can effectively simulate and control the robot's end-effector process, they lack the "brain" thinking required for high-level semantic planning and physical law reasoning, making it difficult to structurally decompose complex tasks, logically reason, and explicitly express physical constraints. In existing technologies, some solutions employ an additional large language model as the "brain" to achieve high-level planning, combining it with the world action model. However, the large language model and the world action model are separate, independent modules lacking a unified architectural support. Information exchange between these modules is hampered, preventing end-to-end collaborative optimization and hindering the establishment of a close causal relationship between "high-level planning → low-level actions → future prediction." Summary of the Invention

[0005] To address the shortcomings of existing technologies, achieve end-to-end collaborative optimization, improve the ability to dynamically and accurately model the physical world, reduce model training complexity, and enhance model generalization ability, this invention adopts the following technical solution:

[0006] The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment includes the following steps:

[0007] Based on machine vision, observe images of the physical world to obtain the robot's current state parameters and received task text instructions;

[0008] Feature extraction is performed on the observed image, state parameters, and task text instructions respectively to obtain image features. State characteristics and text features ;

[0009] Multimodal fusion is performed on the image features, state features, and text features to extract global features;

[0010] High-level semantic planning reasoning is performed on the global features to generate high-level semantic planning text that conforms to physical laws and includes task decomposition steps, action sequence logic, physical constraints, and task objective achievement criteria. This serves as the causal foundation for subsequent underlying action decisions and future video predictions.

[0011] Based on the global features and the corresponding high-level semantic planning text, lexical features are generated for the robot's subsequent low-level action commands. And guide the decoding action through lexical features. This transforms high-level semantic planning text into continuous or discrete, executable low-level action decisions.

[0012] After the underlying action decision is executed, the observation feedback is obtained based on the observation image at the next moment of the task to adjust the high-level semantic planning until the task is completed.

[0013] Based on a "brain + cerebellum" collaborative mechanism, the model can observe images or videos, task text instructions, and robot body state as joint inputs. Through hierarchical processing and fusion output of the "brain + cerebellum" collaborative mechanism, it can achieve accurate modeling of the dynamics of the physical world. The model outputs at least two types of results: high-level semantic planning and low-level action decisions. The "brain" module receives the joint inputs and, based on its built-in learning and analysis of physical laws, generates high-level semantic planning containing physical laws using text generation as a carrier. The high-level semantic planning is used to clarify the task objectives, action logic, and execution principles that conform to physical constraints. The output of this module is used as the basis for low-level action decisions. The "cerebellum" module takes over the high-level semantic planning output by the "brain" module, parses and transforms high-dimensional abstract instructions, generates future actions, and outputs low-level action decisions that can be directly used by the robot for execution. The low-level action decisions include the action parameters, timing logic, and execution priorities of each robot execution component.

[0014] The “brain + cerebellum” collaborative world action language model also follows a closed-loop feedback mechanism in actual operation, that is, it cyclically executes the work cycle of “external observation input - brain semantic planning - cerebellum action decision - underlying hardware execution” or “external observation input - brain semantic planning - cerebellum action decision - underlying hardware execution - future video prediction” until the task is completed.

[0015] Furthermore, based on the image features, state features, text features, and the corresponding high-level semantic planning text and action decisions, lexical features are generated to represent the changes in visual observation after the robot executes the corresponding low-level decisions. This guides video decoding to generate future prediction videos with coherent action logic, accurately presenting the robot's actions and the dynamic evolution of the physical world.

[0016] The model may further include a video prediction module, which receives high-level semantic planning from the "brain" module and low-level action decisions from the "cerebellum" module, performs cross-modal fusion processing on the two, and generates future video predictions that conform to physical laws based on the fusion results, so as to present the dynamic evolution process of the physical world and robot actions.

[0017] Furthermore, based on a video diffusion strategy, lexical features of visually observed changes are used to guide the decoding of the video; using these lexical features as guiding conditions, a multi-step video denoising process with conditional input is carried out: from pure noisy video... Starting at noise intensity t=T, the noise intensity is gradually reduced during multi-step iterative denoising. The current step's video information retention coefficient is combined with the proportion of the cumulative video information retention coefficient to generate the decoded video for the next iteration. This process continues until the noise intensity drops to t=0, at which point a clean future prediction video is obtained. The iterative formula for each step is as follows:

[0018]

[0019] in, This indicates the decoded video for the next iteration. This represents the decoded video at the current iteration step, where t represents the current iteration step. This represents the video information retention coefficient for the current step. This represents the cumulative video information retention coefficient. This represents a video denoising neural network with conditional input. Lexical features representing changes in visual observation, iterated to t=0, This is the final decoded output action sequence.

[0020] Furthermore, a joint loss including video training loss is constructed. For the future prediction video, the denoised video with conditional input is compared with the real video. A video diffusion generation loss function is constructed. By minimizing the video diffusion generation loss, the future prediction video is trained for prediction.

[0021]

[0022] in, Represents the real videos in the dataset. Indicates the steps Noisy video at the time, This represents a video denoising neural network.

[0023] Furthermore, the decoding action sequence is guided by the lexical features of the underlying action instructions based on a diffusion strategy; using the lexical features of the underlying action instructions as guiding conditions, a multi-step process of denoising the action sequence with conditional input is carried out: from the action sequence of pure noise... Starting at noise intensity t=T, the noise intensity is gradually reduced during multi-step iterative denoising. The decoding action sequence for the next iteration is generated by combining the proportion of the current step's action information retention coefficient with the cumulative action information retention coefficient. This process continues until the noise intensity drops to t=0, at which point a clean decoding action sequence is obtained. The iterative formula for each step is as follows:

[0024]

[0025] in, This represents the sequence of decoding actions for the next iteration step. This represents the decoding action sequence of the current iteration step, where t represents the current iteration step. This represents the coefficient for retaining action information at the current step. This represents the cumulative action information retention coefficient. This represents an action denoising neural network with conditional input. The lexical features representing the underlying action instructions, iterated to t=0, This is the final decoded output action sequence.

[0026] Furthermore, a joint loss is constructed, which includes text prediction loss and underlying decision loss, corresponding to the high-level semantic planning text and the underlying action decision, respectively. The model parameters are updated through backpropagation to ensure the generation accuracy of each output.

[0027] The text prediction loss is based on the comparison between the word units and the ground truth values ​​of the word units in the high-level semantic planning text predicted by the global features. An autoregressive generation loss function is constructed, and the high-level semantic planning text is trained for prediction by minimizing the autoregressive generation loss function.

[0028]

[0029] in, Indicates the truth length of high-level semantic programming. Indicates the first of the annotations The truth value of each word element. Indicates a prefix of a word, Lexical units indicating prediction;

[0030] The underlying decision loss is obtained by comparing the denoised action with the conditional input with the true action decision, constructing a diffusion strategy loss function, and then using the diffusion strategy loss function to predict and train the underlying action decision.

[0031]

[0032] in, This represents the sequence of labeled actions in the dataset. Indicates the steps Noisy action sequences at the time This represents a neural network for action denoising.

[0033] Furthermore, the high-level semantic planning reasoning and / or the decoding action are trained through reinforcement learning;

[0034] Based on the current image features, state features, and text features, a reinforcement learning state is constructed, and high-level semantic planning text and continuous or discrete action policies are sampled. The text and action space are explored; the execution effect of the text and action strategy is evaluated through any one or more combinations of actual simulation environment, pre-trained reward model, and predefined evaluation indicators, and the corresponding reward signal is obtained. Based on the reward signal, the weight parameters are updated through a policy optimization algorithm to complete reinforcement learning training and iterative optimization.

[0035] The formula for the Proximity Policy Optimization (PPO) algorithm is as follows:

[0036]

[0037] in, This represents the network with the policy to be optimized. This represents the old policy network before the update. Represents the dominance function. This represents the pruning threshold, used to limit the magnitude of policy updates and ensure training stability. Other reinforcement learning algorithms can also be applied to this model.

[0038] Furthermore, the observed image or video is a physical world scene image or video acquired by the robot's vision acquisition module, including the position, shape, motion state of objects in the scene and environmental parameters;

[0039] The robot's physical state includes the current position, posture, motion parameters, and energy consumption status of each of the robot's execution components.

[0040] The high-level semantic planning includes a description of the robot's own state and its interaction with the environment, task decomposition steps, action sequence logic, physical constraints and target achievement standards. The physical constraints are generated based on the physical law knowledge base built into the model, including the basic physical laws of gravity, friction and inertia. The generation methods used include, but are not limited to, autoregressive generation and discrete diffusion language text generation methods.

[0041] In the future motion generation process, a sequence generation algorithm will be used, combined with robot body state constraints, to convert high-level semantic planning into continuous or discrete, executable low-level motion instructions, ensuring that the motion execution process conforms to physical laws and robot motion limits. The generation methods used include, but are not limited to, diffusion strategies, autoregressive strategies, and control signal generation methods.

[0042] The prediction system based on a world action language model with cerebellar-cerebral co-alignment includes an image encoder, a robot state encoder, a multimodal backbone model, and a robot action decoder. Employing the aforementioned prediction method based on the world action language model with cerebellar-cerebral co-alignment, the image encoder and robot state encoder extract image and state features, combine them with the acquired robot command text features, and perform feature fusion through the multimodal backbone model. Based on the fused global features, high-level semantic planning and reasoning are performed to obtain high-level semantic planning text. The robot action decoder, based on the global features and the high-level semantic planning text, generates lexical features for subsequent low-level robot action commands to guide the generation of low-level action decisions. The image encoder acquires the observation image at the next moment after the execution of the low-level action decisions, using this observation feedback to adjust the high-level semantic planning until the task is completed.

[0043] Furthermore, the system also includes a visual decoder that generates lexical features of visual observation changes based on the global features, high-level semantic planning text, and action decisions, in order to guide video decoding and generate future prediction videos with coherent action logic.

[0044] The advantages and beneficial effects of this invention are as follows:

[0045] This invention introduces a language-driven high-level "brain" module on top of a world model, forming the smallest collaborative unit of "language semantic planning + action execution decision-making." By constructing a hierarchical collaborative mechanism in a unified architecture, it achieves end-to-end semantic planning, action decision-making, and future video prediction. The "language brain" performs top-level understanding, task decomposition, and causal planning, while the "action cerebellum" generates low-level action decisions based on high-level semantic planning and current multimodal inputs. Combined with the above information, it predicts the video images that the robot will receive in the future. Thus, based on the existing world action model, it fills the gaps in macroscopic understanding and interpretable planning, improves the robot's task execution capabilities in open and complex physical environments, and can further integrate future video prediction capabilities. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the system structure in an embodiment of the present invention.

[0047] Figure 2 This is a diagram illustrating the process of synthesizing high-level semantic planning text training data in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0050] To overcome the shortcomings of existing World Action Language Models (WALMs), such as the lack of a language-driven high-level macro-level understanding module, the disconnect between high-level planning and low-level actions, and insufficient modeling accuracy, this invention proposes a prediction system based on the World Action Language Model (WALM) with cerebellar-cerebral co-alignment. Figure 1 As shown, the system's overall architecture is centered on a "brain + cerebellum" collaborative mechanism, processing data such as images and videos, text commands, and low-level execution control signals. The core outputs are high-level semantic planning and low-level action decisions, and can further output future video prediction content. The model includes a visual encoder, a robot state encoder, a multimodal backbone model, a robot action decoder, and a visual decoder. Task text commands are directly encoded into text features. The observed image is input into the visual encoder, and the visual features are obtained after encoding. The robot's physical state is input into the robot state encoder, and the encoded state characteristics are obtained. Task text instruction encoding Visual features Robot state characteristics A single feature encoding sequence is formed by concatenation and then input into a multimodal backbone model to generate high-level semantic planning text. and lexical features representing robot actions Lexical features The underlying control signals are decoded by the robot motion decoder. If it is necessary to predict future videos, the multimodal backbone model additionally outputs lexical features representing future video frames. Lexical features The predicted video image is obtained by decoding with a visual decoder. The entire process can achieve end-to-end control and reasoning from the "brain" to the "cerebellum". Among them, , , The generation order and causal dependency between them can be freely adjusted, and any generation order / causal dependency is within the protection scope of this invention.

[0051] Visual encoder: Performs feature extraction and encoding on the observed images or videos acquired by the robot's vision acquisition module. After the observed images or videos are input into the visual encoder, they undergo feature extraction and encoding processing to output visual features. , which serves as the input to the multimodal backbone model.

[0052] In this embodiment of the invention, a pre-trained CLIP model is used as the initial weights for feature extraction and encoding of the observation images acquired by the robot's vision acquisition module. The observation images are acquired by the robot's vision acquisition module (such as an industrial camera or a depth camera).

[0053] Robot State Encoder: The input consists of robot body state parameters, including the current position, posture, motion parameters (such as velocity and acceleration), and energy consumption status of each robot actuator (such as the robotic arm and mobile chassis). After encoding, the output is a robot state feature vector. This ensures that the robot's state information can be effectively integrated into the multimodal backbone model processing flow. The model structure can adopt a multi-layer fully connected model or other models.

[0054] In this embodiment of the invention, the robot state encoder adopts a multi-layer fully connected module.

[0055] The multimodal backbone model is the core module for data processing. It is primarily responsible for the fusion and processing of multimodal features, the generation of high-level semantic planning, and the output of feature signals representing low-level control commands and future video prediction results. It is the core carrier of the "brain" module, enabling language-driven macro-reasoning and planning functions. Its structure can be implemented using a Transformer model with multiple self-attention layers and fully connected layers connected in series, but other backbone model structures are also within the scope of this invention.

[0056] In this embodiment of the invention, a block-based autoregressive self-attention decoder structure (Transformer) is adopted, consisting of multiple self-attention layers and fully connected layers. The input is a single sequence formed by integrating task text instructions, visual feature vectors, and robot state feature vectors, which is fed into the self-attention layer of the Transformer for fusion and encoding processing. At the output end, the Transformer first directly outputs high-level semantic planning text, and then outputs lexical features representing robot actions and future video image prediction results, providing inputs for the robot action decoder and the visual decoder, respectively.

[0057] Robot motion decoder: A set of low-level control commands representing the robot output from the multimodal backbone model. Guided by the lexical features, it generates low-level action decisions that can be directly used by the robot. It is the core decision-making unit of the "cerebellum" module. It can adopt a lightweight self-attention module to output discrete or continuous action sequences based on a diffusion strategy, or it can adopt other model structures or generation algorithms that can achieve the same function.

[0058] In this embodiment of the invention, the robot action decoder is a small, multi-layer self-attention model. A set of lexical features output by the Transformer are mapped through a linear layer and injected as conditional signals into the diffusion strategy module. Mutual attention calculation provides guidance for the underlying action decisions. Through a complete diffusion denoising process, the feature vector output by the Transformer is received, the features are decoded and transformed to generate underlying action decisions that can be directly used for robot execution, and the output is a continuous sequence of actions.

[0059] Taking the diffusion strategy as an example, the decoding action The process can be achieved through diffusion, from pure Gaussian noise Starting with lexical features To guide the process, a clean action sequence is obtained through multi-step denoising iterations. :

[0060]

[0061] Where T is the total iteration step size of the diffusion strategy, and t is the current iteration step. To preserve coefficients for the current step information, The cumulative information retention coefficient. This is a neural network for action denoising with conditional input. It should be noted that this explanation only uses a diffusion strategy for generation as an example; other action decoding algorithms, calculated according to their respective decoding formulas, also fall within the scope of this invention.

[0062] Visual Decoder: If it is necessary to predict future videos, the visual decoder will output a set of lexical features representing the prediction results of future videos from the multimodal backbone model. Guided by specific conditions, the system generates a video recording of the dynamic evolution of the robot and the physical world after the robot adopts corresponding action strategies. This can be achieved, but is not limited to, using pre-trained text-based or image-based video models as the video decoder.

[0063] In this embodiment of the invention, a pre-trained text-based video model is used as the structure. The lexical features output by the Transformer are mapped through a linear layer and then injected as conditional signals into the diffusion process of video generation. A mutual attention module provides guidance for the video generation process. Through a complete denoising process, a future video prediction that conforms to physical laws is generated.

[0064] Taking video diffusion generation algorithm as an example, decoding video The process can be achieved through multi-step noise reduction, transforming pure Gaussian noise. Starting with lexical features To guide the process, a clean video is obtained through multi-step denoising iterations. :

[0065]

[0066] in, Let t be the total iteration step size for video diffusion generation, and t be the current iteration step. To preserve coefficients for the current step information, The cumulative information retention coefficient. This is a video denoising neural network with conditional input. It should be noted that this explanation only uses video diffusion generation as an example; other video decoding algorithms calculated according to their respective decoding formulas also fall within the scope of this invention.

[0067] This invention employs multiple task losses to jointly train the system, achieving collaborative optimization of each module and ensuring system inference accuracy, causal link coherence, and output consistency. Figure 2 As shown, the specific training process is as follows:

[0068] 1. Training Preparation. Collect physical world scene images, robot task text instructions, robot body state data, high-level semantic planning text corresponding to action sequences, low-level action decision data, and future video data to construct a training dataset. Preprocess the data, including image normalization, state parameter standardization, text segmentation and word embedding, and video frame extraction and alignment, and divide the dataset into training, validation, and test sets. Specifically, the above data is collected in the following ways:

[0069] (1) Based on the observed video data and various sensor signals, expand the multi-dimensional observation data as the basic data source. This includes: robot vision observation data, body state sensor data, and environmental parameter sensor data; among which, robot vision observation data corresponds to the data collected by equipment such as industrial cameras, depth cameras, and infrared cameras; body state sensor data corresponds to the data collected by equipment such as encoders, gyroscopes, force sensors, and energy consumption sensors; and environmental parameter sensor data corresponds to the data collected by equipment such as temperature sensors, humidity sensors, pressure sensors, and light sensors.

[0070] (2) From the above observation data, using artificial intelligence tools (such as visual language models) or manual semantic labeling, extract action descriptions of the interaction between robot-related parts and the environment and objects in the environment. Specifically, this includes the motion trends, states and postures of robot-related parts, the positional relationship between robot parts and the environment, and the interaction relationship between robot parts and objects in the environment. The focus is on core information such as the physical processes and laws related to robot operation, while extracting other auxiliary information related to robot interaction. Further refine the high-level semantic action descriptions of robot interaction with environmental objects, and use them as high-level semantic planning text in the training data. The high-level semantic action descriptions will be used as core data for model training, providing accurate training samples for the high-level semantic planning reasoning of the model's "brain" module and the low-level execution decision generation of the "cerebellum" module. High-level semantic data achieves cross-scene, cross-task, and cross-ontology abstraction and generalization capabilities by abstracting the physical processes of interaction between the machine body and the environment and objects at a high-level semantic level.

[0071] (3) Match and verify the extracted high-level semantic action description with the real robot action sequence to ensure that the output semantic description is consistent with the actual robot action. Select high-quality sample data that meet the matching conditions and include them in the training dataset. Or use artificial intelligence tools (such as visual language models) or manual semantic methods to discover and correct erroneous high-level semantic action descriptions to ensure that the output high-level semantic action description is highly consistent with the actual robot action, thereby ensuring the accuracy and effectiveness of the training data.

[0072] The process of synthesizing training data is not limited to the core steps mentioned above. It may include only some of the core steps, or add other auxiliary processing steps (including but not limited to data deduplication, abnormal data preprocessing, data augmentation, etc.) on the basis of the core process mentioned above. As long as the training data can be effectively synthesized, the accuracy of the training data can be ensured, and the core logic is consistent with the present invention, it falls within the protection scope of the present invention.

[0073] 2. At least two types of loss functions are used for joint optimization. The core loss functions include text prediction loss and underlying decision loss, with video training loss potentially added as well. These correspond to high-level semantic planning, underlying action decision-making, and the optional future video prediction process, respectively. Model parameters are updated through backpropagation of the loss functions to ensure the accuracy of each output generation. The three types of losses are as follows:

[0074] (1) Text prediction loss: To optimize the high-level semantic planning ability of the "brain", autoregressive generation loss or other losses can be used. Taking autoregressive generation loss as an example, its loss function is:

[0075]

[0076] in The truth length of the high-level semantic planning. The first one marked Each word element, It is a prefix for word elements.

[0077] (2) Strategy training loss: Used to optimize the underlying action decision generation process of the "cerebellum" module, it can be diffusion strategy loss or other losses. Taking diffusion strategy loss as an example, its formula is:

[0078]

[0079] In the formula For the labeled action sequences in the dataset, In the steps Noisy action sequences at the time The neural network used for action denoising is consistent with the definition in the previous part of the patent.

[0080] (3) Video training loss: Used to optimize the future video prediction process of the visual decoder, it can be diffusion loss or other losses. Taking video diffusion generation loss as an example, its formula is:

[0081]

[0082] In the formula For real videos in the dataset, In the steps Noisy video at the time, The neural network for video denoising is consistent with the definition above in the patent.

[0083] Based on the above losses, one or more modules of the global action language model can be trained independently or jointly. These losses will be... After weighting and balancing the three coefficients, the total loss function is:

[0084]

[0085] 3. This model can be further optimized through reinforcement learning. By iteratively optimizing the model's capabilities through execution feedback in real / simulation environments, it addresses issues such as distribution bias, insufficient physical compliance, and limited generalization ability present in supervised training, thereby enhancing the model's performance in open and complex scenarios. The specific process is as follows:

[0086] State input and action sampling: The model receives multimodal inputs at the current moment to form the reinforcement learning state. Sampling yields continuous or discrete action strategies Complete the exploration of the action space.

[0087] Execution effect evaluation and reward calculation: the sampled action strategies The input evaluation module can obtain reward signals through any one or a combination of the following evaluation methods. :

[0088] (1) Evaluation of actual simulation environment: Input the action strategy into the robot simulation environment (such as Isaac Sim, Gazebo, MuJoCo), obtain the state and task completion status of the next moment of the environment feedback after the action is executed, and calculate the reward based on the task execution result;

[0089] (2) Evaluation of pre-trained reward model: Based on the current observed image, high-level semantic planning, and future video prediction after action execution, the pre-trained visual language reward model outputs a comprehensive score representing the task completion and compliance, which is mapped to the corresponding reward.

[0090] (3) Evaluation based on predefined evaluation indicators: Rewards are calculated based on predefined quantitative indicators. Core indicators include, but are not limited to, task completion rate, error between motion trajectory and target trajectory, compliance rate of robot motion limits, compliance with physical interaction rules, and motion smoothness.

[0091] Weight Update and Model Optimization: The expected total reward of the policy is calculated based on the single-step reward signal. The expected total reward is maximized through a policy optimization algorithm, and the model's weight parameters are updated via backpropagation. Repeat the above sampling-evaluation-update process until the model converges and the task completion rate reaches the preset threshold, thus completing the reinforcement learning training.

[0092] Taking the Proximal Policy Optimization (PPO) algorithm, which is the most commonly used and most stable algorithm in the field of robotics, as an example, the core optimization objective formula is:

[0093]

[0094] In the formula, For the network of strategies to be optimized, For the old policy network before the update, For the dominant function, The pruning threshold is used to limit the magnitude of policy updates and ensure training stability. Other reinforcement learning algorithms can also be applied to this model.

[0095] by Taking this generation sequence as an example, this invention proposes a prediction method based on a world action language model with cerebellar-cerebellopontine co-alignment, such as... Figure 3 As shown, the specific steps include the following:

[0096] Step 1: Multimodal Input Acquisition and Encoding. The robot's vision acquisition module acquires observed images of the physical world scene, the robot's state acquisition module acquires its own body state parameters, and simultaneously receives externally input task text commands; the observed images are input into the visual encoder, and visual features are obtained through encoding processing. The robot's body state parameters are input into the robot state encoder, and the robot's state features are obtained through encoding processing. The task text instructions are encoded and converted into text features. .

[0097] In this embodiment of the invention, the robot's vision acquisition module can be an industrial camera or a depth camera. The observed images include the position, shape, motion state, and environmental parameters of objects in the scene. The robot's own body state parameters include the current position, posture, motion parameters, and energy consumption status of each actuator.

[0098] Step 2: Multimodal Feature Integration and Input. Integrate the above text features... Visual features Robot state characteristics By splicing together, a length of [length missing] is formed. The single input sequence is fed into the multimodal backbone model for fusion, and global feature information is extracted to provide a foundation for subsequent "brain" and "cerebellum" reasoning.

[0099] In this embodiment of the invention, the multimodal backbone model can use the self-attention layer of Transformer to fuse and encode multimodal features.

[0100] Step 3: High-Level Semantic Planning Reasoning – “Brain” Module Reasoning. Based on its learned knowledge of physical laws and its multimodal data understanding capabilities, the multimodal backbone model analyzes and reasons about the global features of the aforementioned input, generating high-level semantic planning text. Depending on the language model architecture used, complete sentences are generated through methods including, but not limited to, autoregressive processes and discrete diffusion processes. Finally, the “brain” reasoning generates a high-level semantic plan that conforms to physical laws, includes task decomposition steps, action sequence logic, physical constraints, and goal achievement standards. This serves as the foundation for subsequent underlying action decisions and future video predictions.

[0101] In this embodiment of the invention, the "brain" module relies on a unified attention coding language model. Based on the model's learned knowledge of physical laws and its multimodal data understanding capabilities, it analyzes and infers the global features of the input to generate high-level semantic planning text. Depending on the language model architecture used, it can generate individual words sequentially through an autoregressive model, or gradually generate complete sentences through a discrete diffusion language model with T-step denoising iterations. Finally, it generates high-level semantic planning, which serves as the causal pre-foundation for subsequent underlying action decisions and future video predictions. For example, when the task text instruction is "grab the water cup on the table and place it in the designated location," the high-level semantic planning will be broken down into "move the robotic arm above the water cup → adjust the robotic arm posture → grab the water cup → move it to the designated location → release the water cup."

[0102] Step 4: Low-level action decision-making reasoning – “cerebellum” module reasoning. The multimodal backbone model uses… The sequence is taken as input, and the output is lexical features representing the robot's subsequent low-level action commands. Then, the robot action decoder completes the generation and reasoning of future actions, transforming high-level abstract semantic planning into continuous or discrete, executable low-level action decisions. Specifically, the lexical features output by the multimodal backbone model, after mapping transformation, serve as conditional signals to guide the low-level action decisions. It is generated through diffusion strategies or other algorithms.

[0103] In this embodiment of the invention, after the "brain" module outputs the high-level semantic plan, the Transformer, based on the previous input, adds the high-level semantic plan as input and outputs lexical features representing the robot's subsequent low-level action commands. These lexical features, after being mapped through a linear layer, are injected as conditional signals into the robot's action decoder. Guiding signals are obtained through mutual attention calculation, and then, after a complete diffusion denoising process, the high-level abstract semantic plan is transformed into continuous, executable low-level action decisions. For example, for the high-level semantic plan "move the robotic arm above the water cup," the low-level action decision will output specific parameters such as the angle changes, movement speed, and acceleration of each joint of the robotic arm.

[0104] Step 5 (optional): Future video prediction inference. The multimodal backbone model uses... The sequence is used as input, and the output represents the lexical features of the observed video after the robot executes the corresponding low-level decisions. Then, the visual decoder uses this lexical feature as a conditional signal to guide it in predicting video frames, generating future video predictions with continuous frame sequences and coherent action logic. This accurately presents the robot's actions and the dynamic evolution of the physical world, completing the entire reasoning process.

[0105] In this embodiment of the invention, the Transformer adds a low-level action decision as an additional input to the previous input, and outputs lexical features that characterize the predicted changes in the observed video after the robot executes the low-level decision. These lexical features are mapped by a linear layer and injected into the visual decoder as a conditional signal. Guided by the mutual attention module, and after a complete diffusion denoising process, a future video prediction with a continuous frame sequence and coherent action logic is generated, accurately presenting the dynamic evolution process of the robot's actions and the physical world, thus completing a single inference process.

[0106] Step 6: The robot repeats the above loop of "external observation input - brain semantic planning - cerebellum action decision - low-level hardware execution" or "external observation input - brain semantic planning - cerebellum action decision - low-level hardware execution - future video prediction" until the task is completed. The "brain" module adjusts its high-level semantic planning based on feedback from the next loop, forming a closed-loop collaboration that ensures the output of high-level semantic planning, low-level action decisions, and future video predictions all conform to physical laws and ultimately complete the task.

[0107] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A prediction method based on a world action language model with cerebellar-cerebellopontine co-alignment, characterized by: Based on machine vision, observe images of the physical world to obtain the robot's current state parameters and received task text instructions; Feature extraction is performed on the observed image, state parameters, and task text instructions respectively to obtain image features, state features, and text features; Multimodal fusion is performed on the image features, state features, and text features to extract global features; High-level semantic planning reasoning is performed on the global features to generate high-level semantic planning text required for the robot to perform its tasks; Based on the global features and the corresponding high-level semantic planning text, lexical features of the robot's subsequent low-level action instructions are generated, and the decoding action is guided by the lexical features to convert the high-level semantic planning text into low-level action decisions. After the underlying action decision is executed, the observation feedback is obtained based on the observation image at the next moment of the task to adjust the high-level semantic planning until the task is completed.

2. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment as described in claim 1, characterized in that: Based on the image features, state features, text features, and the corresponding high-level semantic planning text and action decisions, lexical features are generated to represent the changes in visual observation after the robot executes the corresponding low-level decisions. This guides video decoding to generate future prediction videos with coherent action logic.

3. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment as described in claim 2, characterized in that: Based on a video diffusion strategy, the lexical features of visual observation changes are used to guide the decoding of the video. Using the lexical features of visual observation changes as guiding conditions, a multi-step video denoising process with conditional input is carried out. Specifically, starting from a purely noisy video, the noise intensity is gradually reduced in the multi-step iterative denoising process. The decoding video for the next iteration step is generated by combining the proportion of the current step's video information retention coefficient in the cumulative video information retention coefficient. This process continues until the noise intensity drops to a first threshold, resulting in a clean future prediction video.

4. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment as described in claim 3, characterized in that: A joint loss function, including video training loss, is constructed for the future prediction video. The denoised video with conditional input is compared with the real video. A video diffusion generation loss function is constructed, and the prediction training is performed on the future prediction video by minimizing the video diffusion generation loss.

5. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment according to claim 1, characterized in that: The decoding action sequence is guided by the lexical features of the underlying action instructions based on the diffusion strategy. Using the lexical features of the underlying action instructions as the guiding condition, a multi-step process of denoising the action sequence with conditional input is carried out. Specifically, starting from a purely noisy action sequence, the noise intensity is gradually reduced in the multi-step iterative denoising process. The decoding action sequence for the next iteration step is generated by combining the proportion of the current step action information retention coefficient in the cumulative action information retention coefficient. This process is repeated until the noise intensity drops to a second threshold and a clean decoding action sequence is obtained.

6. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment as described in claim 5, characterized in that: Construct a joint loss consisting of text prediction loss and underlying decision loss, corresponding to the high-level semantic planning text and the underlying action decision, respectively; The text prediction loss is based on the comparison between the word units and the ground truth values ​​of the word units in the high-level semantic planning text predicted by the global features. An autoregressive generation loss function is constructed, and the high-level semantic planning text is trained for prediction by minimizing the autoregressive generation loss function. The underlying decision loss is constructed by comparing the denoised action with the conditional input with the true action decision, constructing a diffusion policy loss function, and then using the diffusion policy loss function to predict and train the underlying action decision.

7. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment according to claim 1, characterized in that: The high-level semantic planning and reasoning and / or the decoding action are trained through reinforcement learning; Based on the current image features, state features, and text features, a reinforcement learning state is constructed. High-level semantic planning text and continuous or discrete action policies are sampled to explore the text and action space. The execution effect of the text and action policies is evaluated through any one or more combinations of actual simulation environment, pre-trained reward model, and predefined evaluation indicators to obtain the corresponding reward signal. Based on the reward signal, the weight parameters are updated through policy optimization algorithm to complete reinforcement learning training and iterative optimization.

8. The prediction method based on the world action language model of cerebellar-cerebellopontine co-alignment according to claim 1, characterized in that: The observed images or videos are physical world scene images or videos acquired by the robot's vision acquisition module, including the position, shape, motion state of objects in the scene and environmental parameters; The robot's physical state includes the current position, posture, motion parameters, and energy consumption status of each of the robot's execution components. The high-level semantic planning text includes, but is not limited to, robot state, interaction with the environment, task decomposition steps, action sequence logic, physical laws and constraints, task target point achievement standards, execution priority, safety rules, exception handling strategies, and multi-component collaborative logic. The physical constraints are generated based on the physical law knowledge base built into the model, including the basic physical laws of gravity, friction, and inertia. The generation methods used include, but are not limited to, autoregressive generation and discrete diffusion language text generation methods. In the future motion generation process, a sequence generation algorithm will be used, combined with robot body state constraints, to convert high-level semantic planning into continuous or discrete, executable low-level motion instructions. The generation methods used include, but are not limited to, diffusion strategies, autoregressive strategies, and control signal generation methods.

9. A prediction system based on a world action language model with cerebellar and cerebellar co-alignment, comprising an image encoder, a robot state encoder, a multimodal backbone model, and a robot action decoder, characterized in that: The prediction method based on the world action language model with cerebellar-cerebellopontine co-alignment as described in any one of claims 1 to 8 is adopted. Image features and state features are extracted by the image encoder and robot state encoder. Combined with the acquired robot instruction text features, feature fusion is performed by the multimodal backbone model. Based on the fused global features, high-level semantic planning reasoning is performed to obtain high-level semantic planning text. The robot action decoder generates lexical features of subsequent low-level action instructions of the robot based on the global features and the high-level semantic planning text to guide the decoding action to generate low-level action decisions. The image encoder obtains the observation image at the next moment after the execution of the low-level action decision as observation feedback to adjust the high-level semantic planning until the task is completed.

10. The prediction system based on the cerebellar-cerebrum co-alignment world action language model according to claim 9, characterized in that: The system also includes a visual decoder, which generates lexical features of visual observation changes based on the global features, high-level semantic planning text, and action decisions, to guide video decoding and generate future prediction videos with coherent action logic.

Images