Robot motion generation method and device with dual vlm and hierarchical routing coordination

Abstract

This application belongs to the field of embodied intelligent robot technology, specifically relating to a method and apparatus for robot motion generation through dual VLM and hierarchical routing collaboration. The method includes: first, acquiring the robot's current visual observations and high-level natural language commands; second, inputting the commands and observations into a large-scale VLM, which asynchronously outputs current subtask commands and features at each layer and caches them; simultaneously, inputting the subtask commands and real-time observations into a small-scale VLM, which frequently outputs features at each layer; furthermore, a policy network, through learnable routing weights, performs cross-model and cross-layer adaptive selection and weighted aggregation of features from all layers of the two VLMs to obtain fused features; finally, based on the fused features, outputting a sequence of actions to control the robot. This application effectively improves the planning capability, feature utilization efficiency, and environmental robustness of robot motion generation, providing reliable support for the autonomous control of embodied intelligent robots.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligent robot technology, specifically relating to a method and apparatus for generating robot motion in collaboration with dual VLM and hierarchical routing. Background Technology

[0002] In the field of embodied intelligent robots, the core challenge lies in enabling robots to accurately understand human natural language commands through visual observation and autonomously generate corresponding action sequences (i.e., "vision-language-action" interaction). The Vision-Language-Action (VLA) model is the mainstream technical approach to address this challenge. Its core lies in utilizing the powerful visual and language joint understanding capabilities of a pre-trained Visual Language Model (VLM) to drive the robot's policy network to generate control commands.

[0003] Currently, the mainstream approach is to use an independent policy network (PLN) bridged with a Virtual Model (VLM). For example, the VLA-Adapter scheme employs a fixed hierarchical mapping mechanism, connecting each layer of the policy network to the corresponding layer of the VLM. However, this approach has inherent drawbacks: First, the forced binding of the number of layers between the policy network and the VLM leads to rigid network depth design and a lack of flexibility; second, the fixed mapping limits the policy network's ability to acquire information across layers, while research shows that the information encoded by different layers of the VLM (such as spatial details and high-level semantics) differs significantly, and the fixed mapping cannot adaptively acquire optimal features.

[0004] Furthermore, VLA-Adapter is essentially a single VLM architecture, making it difficult to reconcile the contradiction between model capability and computational efficiency: large-scale VLMs have strong understanding capabilities but slow inference, failing to meet the demands of high-frequency real-time control. To address this issue, another dual-system approach (such as DuoCore-FS) has been proposed, employing asynchronous dual-channel architecture: a slow VLM performs inference, while a fast action generation module (fast module) executes control. However, this approach introduces new drawbacks: its fast module is not a complete visual-language model and lacks independent visual-language understanding capabilities. When the environment changes dynamically, the fast module cannot independently understand based on real-time observations and can only rely on potentially outdated cached features, resulting in insufficient control robustness.

[0005] In summary, existing technologies suffer from several shortcomings, including design rigidity and suboptimal feature utilization due to the binding of VLM and policy network layers, the contradiction between the capabilities and efficiency of a single model, and insufficient control robustness due to the lack of understanding of fast modules in dual systems. These shortcomings urgently need to be addressed. Summary of the Invention

[0006] This application proposes a robot motion generation scheme based on dual visual language models and hierarchical routing collaboration. It aims to solve the problems in the prior art, such as design rigidity and suboptimal feature utilization caused by the binding of VLM and policy network layers, the contradiction between capability and efficiency under a single VLM architecture, and insufficient control robustness caused by the lack of fast module understanding capability in the existing dual-system scheme.

[0007] The first aspect of this application provides a robot motion generation method based on dual-vision language model and hierarchical routing collaboration, including: Acquire the robot's current visual observations and high-level natural language commands; The high-level natural language instruction and the current visual observation are input into a large-scale visual language model, which processes them and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model. The current subtask instruction and the current visual observation are input into the small-scale visual language model, processed by the small-scale visual language model, and the second features of each layer of the small-scale visual language model are output. The small-scale visual language model runs more frequently than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models. The policy network adaptively selects and aggregates features from the first features of multiple layers of the large-scale visual language model and the second features of multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features. Based on the fusion features, the robot is output with a sequence of actions to execute the current sub-task.

[0008] In some embodiments of this application, the step of inputting the high-level natural language instructions and visual observations into a large-scale visual language model, processing them, and outputting the current subtask instruction includes: The large-scale visual language model analyzes the task execution phase based on the high-level natural language instructions and the current visual observation, and outputs the current subtask instructions in text form corresponding to the task execution phase.

[0009] In some embodiments of this application, before the policy network adaptively selects and aggregates features from multiple layers of the large-scale visual language model and multiple layers of the small-scale visual language model using learnable routing weights to obtain fused features, the method further includes: The first features of each layer output by the large-scale visual language model are cached; Based on the difference between the current time and the feature cache time, time encoding is applied to the cached first feature to obtain the time-encoded first feature, which is then used as the input for the policy network to select and aggregate.

[0010] In some embodiments of this application, the time encoding employs a sinusoidal position encoding function for the difference. The coded components of the 2kth and 2k+1th dimensions are defined by the following formula: ,

[0011] in, Representing feature dimension, Integer index, value .

[0012] In some embodiments of this application, the learnable routing weights are obtained by jointly normalizing the learnable parameters corresponding to all layers of the large-scale visual language model and all layers of the small-scale visual language model.

[0013] In some embodiments of this application, the policy network is... The layer of the small-scale visual language model Second routing weights of layer features and the first large-scale visual language model First routing weight of layer features They are defined by the following formulas respectively: , , in, and These are the policy networks of the 1st, 2nd, and 3rd generations. Layer-by-layer small-scale visual language model Layers and large-scale visual language models The learnable routing parameters of each layer, and each layer of the policy network maintains its own independent learnable routing parameters. and These represent the total number of layers in the small-scale visual language model and the large-scale visual language model, respectively. The policy network is the Layer fusion features It is calculated using the following formula: , in, The small-scale visual language model represents the first... The second characteristic of the layer, This represents the large-scale visual language model after time-encoded processing. The first characteristic of the layer, It is a linear projection layer used to map the hidden layer dimensions of the large-scale visual language model to a dimension space consistent with that of the small-scale visual language model; and This forms a joint softmax normalized distribution across all layers of the large-scale visual language model and all layers of the small-scale visual language model.

[0014] In some embodiments of this application, the first feature and the second feature each comprise multiple types of feature streams; For each type of feature flow, maintain independent learnable routing parameters. and The first route weight and the second route weight are calculated respectively to perform independent adaptive selection and aggregation.

[0015] In some embodiments of this application, the method is performed using a trained neural network model, which is trained through the following steps: In the first stage, the main parameters of the large-scale visual language model and the small-scale visual language model are frozen, and the learnable routing weights and the parameters of the policy network are trained. In the second stage, the main parameters of the large-scale visual language model are unfrozen and jointly fine-tuned with the learnable routing weights, the policy network, and the small-scale visual language model.

[0016] In some embodiments of this application, the large-scale visual language model runs continuously in a control thread independent of the small-scale visual language model and the policy network; After each run is completed, the current subtask instructions and the first features of each layer are updated to the cache, and the next run begins immediately.

[0017] A second aspect of this application provides a robot motion generation device based on dual-vision language model and hierarchical routing collaboration, comprising: The acquisition module is used to acquire the robot's current visual observations and high-level natural language commands; The first feature extraction module is used to input the high-level natural language instruction and the current visual observation into the large-scale visual language model, which processes the instruction and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model. The second feature extraction module is used to input the current subtask instruction and the current visual observation into the small-scale visual language model, which processes the input and outputs the second features of each layer of the small-scale visual language model. The small-scale visual language model has a higher running frequency than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models. The fusion module is used by the policy network to adaptively select and aggregate first features from multiple layers of the large-scale visual language model and second features from multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features. The action generation module is used to output an action sequence for controlling the robot to perform the current sub-task based on the fused features.

[0018] In summary, the robot motion generation method and apparatus based on dual-visual language model and hierarchical routing collaboration provided in the embodiments of this application firstly, by constructing a "planning-execution" collaborative architecture composed of a large-scale and a complete small-scale VLM, the high-frequency requirements of real-time control are met while ensuring the accuracy of high-level planning, thus resolving the contradiction between capability and efficiency. The small-scale VLM, as a complete model with independent visual-language understanding capabilities, ensures the autonomy of the execution end in dynamic environments, thereby overcoming the insufficient control robustness caused by the lack of understanding capabilities of fast modules in existing dual-system schemes. Secondly, by introducing a learnable hierarchical routing mechanism, the fixed layer binding between the policy network and the VLM is broken, achieving cross-model and cross-layer adaptive feature fusion, solving the problems of design rigidity and suboptimal feature utilization caused by fixed mapping. Furthermore, by applying time encoding to cached features, the system can perceive feature freshness, enhancing the temporal stability of asynchronous collaboration. In this application, the dual-VLM architecture and hierarchical routing mechanism are interdependent and work in close collaboration, jointly achieving higher-level planning capabilities, better feature utilization efficiency, and stronger environmental robustness. Attached Figure Description

[0019] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a comparative diagram of existing VLA model architectures, where: Figure 1a is a schematic diagram of the OpenVLA architecture; Figure 1b is a schematic diagram of the π0 architecture; Figure 1c shows VLA. Adapter architecture diagram; Figure 2 This is a schematic diagram of the Probing experiment results, where: Figure 2 'a' is a schematic diagram of the scene classification experiment results; Figure 2 b is a schematic diagram illustrating the experimental results of command interpretation; Figure 2 c is a schematic diagram of the attention distribution experiment in the policy network; Figure 2 d is a schematic diagram comparing attention distribution at different checkpoints; Figure 2 e is a schematic diagram of the target localization experiment results; Figure 3 is a schematic diagram of the overall architecture of the robot motion generation method based on dual-vision language model and hierarchical routing collaboration according to some embodiments of this application; Figure 4 This is a schematic diagram illustrating the workflow of a hierarchical routing mechanism according to some embodiments of this application; Figure 5 The following is a schematic diagram of the feature caching and time encoding mechanism and timing flow according to some embodiments of this application, wherein: Figure 5a is a schematic diagram of the feature caching and time coding mechanism; Figure 5b is a schematic diagram of the timing process of feature caching and time encoding; Figure 6 This is an exemplary computer system architecture diagram to which the robot motion generation method based on dual-vision language model and hierarchical routing collaboration of this application applies; Figure 7 This is a flowchart illustrating a robot motion generation method based on dual-vision language model and hierarchical routing collaboration, according to some embodiments of this application; Figure 8 This is a schematic diagram of a robot motion generation device based on dual-vision language model and hierarchical routing collaboration, according to some embodiments of this application. Detailed Implementation

[0020] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0021] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0022] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0023] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0024] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0025] In the field of embodied intelligent robots, achieving precise "vision-language-action" interaction—that is, enabling robots to understand human commands and generate corresponding actions based on real-time visual observation—is the core of promoting their autonomy. Currently, the mainstream technical approach relies on the Vision-Language-Action (VLA) model, which drives the robot's policy network through the understanding capabilities of a pre-trained Visual Language Model (VLM).

[0026] However, existing VLA models have significant limitations in the key “bridging” mechanism connecting VLMs and policy networks.

[0027] Specifically, Figure 1a c represents a schematic diagram of the existing technical architecture. The diagram uses English labels, the meaning of which will be explained in the following text. For example... Figure 1As shown in ac, existing mainstream solutions, whether OpenVLA (which does not have an independent policy network), Figure 1 a) Shared Attention mechanism, leading to π0 (which forces the VLM and the motion expert layer to be consistent) Figure 1 b), or still using a fixed-level mapping VLA-Adapter ( Figure 1 c) The policy network (or action expert) and the number of layers in the VLM are forcibly bound. This structure leads to a lack of flexibility in the design of the policy network, and more importantly, the fixed mapping relationship restricts its freedom in acquiring information. To further reveal this deficiency, the inventors conducted a probing experiment to diagnose and analyze the information encoding patterns of each layer of the VLM and the attention distribution characteristics of the policy network. The experimental results and analysis are detailed below. Figure 2 a e. Experiments show that the types of information encoded by different layers of VLM differ significantly: shallow features are rich in spatial details, while deep features focus on high-level semantics. The fixed mapping in existing schemes forces each layer of the policy network to receive only a single type of feature from its corresponding layer, preventing it from adaptively acquiring optimal information across layers. Therefore, the binding of VLM to the number of policy network layers in existing technologies leads to design rigidity and suboptimal feature utilization.

[0028] Looking further, the above solutions all belong to a single VLM architecture, making it difficult to balance model capabilities and inference efficiency: large-scale VLMs have strong understanding and planning capabilities, but slow inference speeds, failing to meet real-time control requirements; small-scale VLMs have fast inference speeds, but limited semantic understanding and task decomposition capabilities. To attempt to resolve this contradiction, the industry has proposed a dual-system solution (such as DuoCore-FS), which uses asynchronous dual channels, with a slow VLM performing inference and a fast action generation module performing control. However, the fast module in this solution is not a complete visual-language model and lacks independent visual-language understanding capabilities. When the environment changes dynamically, the fast module cannot independently understand based on real-time observations and can only rely on potentially outdated cached features, resulting in insufficient control robustness due to the lack of understanding capabilities of the fast module.

[0029] To address the aforementioned shortcomings, this application proposes a robot motion generation method based on hierarchical routing and a dual-vision language model. For example... Figure 3 As shown, the core of this application lies in constructing a dual-complete VLM collaborative system with "planning-execution" separation, and using an innovative hierarchical routing mechanism as an intelligent "scheduling hub".

[0030] The technical solution of this application is specifically implemented as follows: The solution adopts a collaborative architecture consisting of a large-scale VLM and a small-scale VLM. The large-scale VLM, as a complete visual language model, runs asynchronously and at low frequency in an independent thread. It is responsible for semantic understanding and task decomposition of high-level language instructions based on visual observations, outputting sub-task instructions in text form and their features at each layer. The small-scale VLM, also as a complete visual language model, runs at a high frequency in sync with the control loop, providing real-time visual-language features based on current visual observations and the latest sub-task instructions. The two VLMs can have different numbers of layers. The key hierarchical routing mechanism (its workflow is detailed in [link to documentation]) is also described. Figure 4 It is integrated into each layer of the policy network, and through learnable routing weights, it processes large-scale VLMs (after feature caching and time encoding, see...) Figure 5 The outputs of all layers of the policy network and the small-scale VLM are jointly softmax normalized, and the weighted sum is dynamically calculated. This allows each layer of the policy network to adaptively select and aggregate the most valuable features from any layer of the two VLMs, completely breaking the layer binding. Simultaneously, feature caching and temporal encoding mechanisms (such as...) are implemented. Figure 5 (As shown) This ensures the stability of the asynchronous collaborative operation of the two VLMs. Finally, based on the fused features after intelligent aggregation, the policy network outputs the action sequence for controlling the robot to execute the current subtask.

[0031] To reveal the information encoding patterns of each layer of the VLM and the working characteristics of the policy network, the inventors conducted a systematic probing experiment analysis on the VLA-Adapter architecture. Figure 2a e is a schematic diagram of the Probing experiment results. The labels in the diagram are in English, and their specific meanings will be explained in the following text.

[0032] Experiments show that the information encoded by different layers of VLM exhibits significant differences and clear transformation patterns: as shown in Figure 2a, shallow VLM features achieve higher accuracy in scene classification tasks, primarily encoding visual spatial information; while... Figure 2 As shown in b, its deep features significantly improve accuracy in instruction comprehension tasks, shifting towards encoding high-level semantic information. This confirms that shallow VLM focuses on spatial details, while deep VLM focuses on semantic understanding.

[0033] Further experimental results revealed the problems with the fixed-level mapping architecture: Figure 2 c shows that the attention distribution of the policy network is highly consistent from the shallow to the middle layers, with a large amount of attention focused on visual features, and only significant attention is paid to semantic features in the deepest layers, indicating that there is obvious computational redundancy in its shallow and middle layers. Figure 2 The comparison results at different training stages confirm that this redundant pattern exists stably and is caused by fixed-level mapping, rather than by insufficient training. Figure 2The target localization experiment further verified that the spatial location information of shallow features in VLM is the most accurate (with the lowest localization error). This information degrades as the network deepens, which once again proves that the feature value distribution of each layer is uneven.

[0034] In summary, the Probing experiment fully demonstrates that the existing VLA architecture, due to the forced binding of the VLM and the number of policy network layers and the fixed and rigid layer mapping, makes it impossible for the policy network to adaptively select the optimal features across layers, which in turn leads to problems such as low feature utilization efficiency and network computational redundancy.

[0035] Figure 6 This is an exemplary computer system architecture diagram applicable to the robot motion generation method based on dual-vision language model and hierarchical routing collaboration in this application. Figure 6 As shown, the system mainly includes an execution unit (robot body), a sensing unit, and a computing unit (core computing unit). Each unit interacts with other units via a communication link. Specifically: The execution unit (robot body) is the final execution carrier of actions, such as a robotic arm or mobile robot platform. It receives action sequence instructions from the computing unit and drives mechanical parts to complete physical operations through its underlying controller (such as joint motor controller), realizing real-time closed-loop control of the robot.

[0036] Sensing Unit: Integrated into the robot platform, it is used to collect real-time status data of the working environment. The core includes vision sensors (such as cameras), which are responsible for providing the current visual observation data required for robot motion generation and sending the perceived data to the computing unit.

[0037] The computing unit (core computing unit) is the core computing device and execution entity that loads and runs the "robot motion generation method based on hierarchical routing and dual VLM collaboration" described in this application. This unit calls a large-scale VLM to perform task planning, calls a small-scale VLM to perform real-time feature extraction, fuses the features of the two VLMs through a hierarchical routing mechanism, and generates the final motion sequence by a policy network, which is then sent to the execution unit.

[0038] The computing unit can be flexibly deployed physically on the robot body, nearby edge servers, or cloud servers, depending on computing resources and latency requirements. The sensing unit and the execution unit (robot body) are typically deployed as a single unit. The entire system constitutes a complete intelligent robot control hardware and software solution.

[0039] Figure 7 This is a flowchart illustrating a robot motion generation method based on dual-vision language model and hierarchical routing collaboration, according to some embodiments of this application. In some embodiments, the robot motion generation method based on dual-vision language model and hierarchical routing collaboration can be... Figure 6The core computing unit in the system shown executes the following steps: S710 acquires the robot's current visual observations and high-level natural language commands.

[0040] The execution of this method begins with an initialization phase.

[0041] Initialization phase: The system receives high-level language instructions. and initial visual observation .

[0042] After initialization, the system enters a continuous main control loop.

[0043] In each control step The specific steps for this process are: obtaining the current time step. Visual observation High-level language instructions It remains unchanged throughout the mission.

[0044] S720, the high-level natural language instruction and the current visual observation are input into the large-scale visual language model, which processes them and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model.

[0045] In this application, large-scale VLM Responsible for high-level mission planning. It receives raw high-level language instructions. (e.g., "pick up the red cup and place it on the plate") and visual observation, outputting two types of information: (1) decomposed subtask instructions (Text form, such as "approach the red cup"); (2) Hidden state features of each layer are used for hierarchical routing.

[0046] Large-scale VLMs utilize their pre-trained language reasoning capabilities, combined with the current visual observation and analysis task execution phase, to autonomously determine and output sub-task instructions corresponding to the current scene. Large-scale VLMs continuously cycle according to their own reasoning speed, and their timing relationship with the high-frequency control loop is as follows: Figure 5 As shown in b. After each inference iteration, the current subtask instruction and feature cache are updated. Assume a large-scale VLM has... Layer, its first The output features of the layer are denoted as: , in This represents the visual observation time step corresponding to the most recent completion of inference in a large-scale VLM. For the first Hidden state characteristics of the layer The current subtask instruction output by VLM.

[0047] Based on the above design, the execution of this step includes the following specific processes: During the initialization phase, following the description in S710, large-scale VLMs are based on high-level language instructions. and initial visual observation Perform the first inference and output the initial subtask instruction. and features of each layer Write the features to the cache and save the difference between the current time and the feature cache time. Initialize to 0. Then, the large-scale VLM enters a continuous loop mode.

[0048] Within the main control loop, large-scale VLMs operate as independent, asynchronous, continuous loops: Large-scale VLMs continue to run at their own inference speed. Each time an inference round is completed, the latest subtask instructions are sent. Write the features of each layer to the cache and reset. The value was 0, and then immediately updated with the latest visual observations and high-level instructions. Begin the next round of inference. During each inference iteration, the large-scale VLM combines current visual observations with higher-level instructions, utilizes its language reasoning capabilities to analyze the task execution phase, and outputs corresponding subtask instructions (such as switching from "approach" to "grasp"). Its execution frequency is naturally determined by its own inference time.

[0049] S730, the current subtask instruction and the current visual observation are input into the small-scale visual language model, processed by the small-scale visual language model, and the second features of each layer of the small-scale visual language model are output. The small-scale visual language model runs more frequently than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models.

[0050] In this application, small-scale VLM (denoted as ) This is also a complete visual language model, providing real-time feature support for subtask execution. It receives subtask instructions from the output of a large-scale VLM. and current visual observation It operates at a high frequency (synchronized with the policy network) to provide real-time visual-language features to the policy network. Assume a small-scale VLM has... Layer, its first The output features of the layer are denoted as: , Number of layers in two VLMs and They are usually different.

[0051] Based on the above design, the specific actions of this step in the main control loop are as follows: Small-scale VLM Current visual observation and current subtask instructions (from large-scale VLM) Perform complete forward inference using the most recent output, and output the features of each layer. .

[0052] S740, the policy network adaptively selects and aggregates features from the first features of multiple layers of the large-scale visual language model and the second features of multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features.

[0053] This step is a crucial step in the main control loop, and its implementation relies on two core mechanisms: feature caching and time encoding, and hierarchical routing. The following section first explains these two mechanisms, and then illustrates their specific application in this step.

[0054] Feature caching and time encoding mechanisms: Since large-scale VLMs operate at low frequency, the policy network uses cached historical features between inferences in large-scale VLMs. To enable the policy network to be aware of the timeliness of cached features, this application introduces a feature caching and time-coding mechanism, the principle of which is as follows: Figure 5 As shown in a.

[0055] Suppose that the most recent inference of a large-scale VLM occurs at time step The current time step is The cache latency is then... Define the time encoding function. It adopts the form of sinusoidal position encoding: , , in, Representing feature dimension, Integer index, value .

[0056] Apply time encoding to the caching characteristics of large-scale VLMs: , This time is encoded for each delay value. Generate unique encoding vectors, enabling the policy network to distinguish cached features of different "freshness". With... As the cache size increases, the encoding vector undergoes a systematic change, allowing the policy network to learn different utilization strategies for cache features with varying latency levels.

[0057] Hierarchical routing mechanism: The core innovation of this application is the introduction of a hierarchical routing mechanism, breaking the layer binding between policy networks and VLMs. The principle is as follows: Figure 4 As shown.

[0058] Assume the policy network has a total of layer( Can be with , (All are different), the first of the policy network The layer adaptively selects and aggregates features from all layers of the two VLMs through a routing mechanism.

[0059] For the policy network, the first The layer is defined with the following routing weights: , , in, and These are the policy networks of the 1st, 2nd, and 3rd generations. Layer-by-layer small-scale visual language model Layers and large-scale visual language models The learnable routing parameters of each layer, and each layer of the policy network maintains its own independent learnable routing parameters. and These represent the total number of layers in the small-scale visual language model and the large-scale visual language model, respectively. and This forms a joint softmax normalized distribution across all layers of the two VLMs.

[0060] Policy Network The aggregation features received by the layer are:

[0061] in, The small-scale visual language model represents the first... The second characteristic of the layer, This represents the large-scale visual language model after time-encoded processing. The first characteristic of the layer, It is a linear projection layer used to map the hidden dimensions of the large-scale visual language model to a dimensional space consistent with that of the small-scale visual language model.

[0062] The aggregation feature Subsequently, the policy is injected into the Cross-Attention network. In the calculation of the layer.

[0063] The above formula provides a general form using a single feature stream as an example. In actual implementation, when each layer of the VLM output contains multiple types of features (such as visual patch features and ActionQuery features), each type of feature stream maintains independent routing parameters and performs independent route aggregation.

[0064] The key advantages of hierarchical routing mechanisms are: (1) the number of layers in the policy network. With the number of layers of two VLMs , (1) Completely decoupled and can be set independently; (2) Each policy network can obtain information from any layer of the two VLMs without being restricted by fixed mapping; (3) The routing weights automatically learn the optimal feature selection strategy through end-to-end training.

[0065] Based on the above two mechanisms, the specific execution actions of this step (S740) in the main control loop are as follows: Read the features from each layer of the large-scale VLM in the feature cache and apply temporal encoding to obtain... ,in This represents the difference in steps between the current time step and the most recent inference completed by the large-scale VLM.

[0066] For each layer of the policy network : Calculate route weights: Based on the formula defined in the hierarchical routing mechanism, learnable parameters are used. and The routing weights for each layer of a small-scale VLM are calculated. and routing weights for each layer of a large-scale VLM .

[0067] Aggregation features: According to the aggregation formula defined in the hierarchical routing mechanism The fusion features of this layer are calculated. .

[0068] Injection calculation: Aggregation features The policy is injected into the network via Cross-Attention. In the computation of the layer, and based on this, perform the forward computation of the layer itself.

[0069] S750, based on the fusion features, outputs the action sequence for controlling the robot to execute the current sub-task.

[0070] This step is the final step in the main control loop.

[0071] In this step, the policy network outputs an action sequence. ,in The time step for action prediction (Action Chunk).

[0072] The following detailed explanation of this application will be based on a specific example.

[0073] 1. Model Configuration In this embodiment, Qwen2.5-3B-Instruct is selected as the large-scale VLM. ( (Layer), Qwen2.5-0.5B-Instruct as a small-scale VLM ( (Layer). Both VLMs are paired with SigLIP and DINOv2 as visual encoders, receiving RGB images from a stereo camera as visual input. The policy network adopts an MLPResNet structure, setting... Layers (less than two VLMs) are used to verify the effectiveness of layer decoupling.

[0074] 2. Input and Feature Extraction The system receives high-level language instructions. (e.g., "pick up the red cup and place it on the plate"). In each control step, the system receives images from the binocular camera. (Resolution 224×224) and the robot's current joint status (7-dimensional vector).

[0075] Large-scale VLMs continuously perform inference in a separate thread, receiving instructions from higher levels. And visual observation. After each inference is completed, output the current subtask instruction. (e.g., "approach the red cup", "grasp the cup", "lift the cup", etc.), and extract two types of feature sequences from its 36 layers. and The data is written to the feature cache, and then the next round of inference begins immediately. Large-scale VLMs automatically determine task progress and update subtask instructions based on changes in the visual scene.

[0076] A small-scale VLM receives the current subtask instruction at each step. and visual observation Perform complete forward inference to extract two types of feature sequences from its 24 layers: visual patch features. (512 tokens per layer) and ActionQuery features (64 learnable query tokens per layer). Since the time taken for a single inference in the 3B model is several times that of the 0.5B model, the actual update frequency of a large-scale VLM is naturally lower than that of a small-scale VLM.

[0077] 3. Hierarchical routing implementation In the 16 layers of the policy network, each layer maintains a set of routing parameters. Route aggregation is performed independently for the visual Patch feature stream and the ActionQuery feature stream. Taking the ActionQuery feature stream as an example, the policy network... The routing process of the layer is as follows: 1. Calculate route weights (For each layer of a small-scale VLM) and (For each layer of a large-scale VLM), a total of Each weight is normalized using softmax.

[0078] 2. Apply time encoding to large-scale VLM features: .

[0079] 3. Aggregation characteristics: ,in Project the hidden layer dimensions of the 3B model to be consistent with the 0.5B model.

[0080] 4. Along with robot status Together, we inject policy into the Cross-Attention network. layer.

[0081] The visual patch feature stream is processed in a similar way, by aggregating it through another set of independent routing parameters and injecting it into a policy network with a gated Cross-Attention mechanism.

[0082] 4. Training Strategies The training is divided into two phases: Phase 1: Freeze the main parameters of both VLMs, apply LoRA fine-tuning only to the smaller VLM, and train the policy network and routing parameters. The training data is a robot operation demonstration dataset (such as the LIBERO benchmark), and the loss function is the L1 loss between the predicted action and the demonstrated action.

[0083] Phase 2 (optional): Unfreeze the LoRA parameters of large-scale VLMs and perform joint fine-tuning to further align the feature spaces of large-scale VLMs with those of small-scale VLMs.

[0084] 5. Reasoning Deployment Upon inference startup, the large-scale VLM first performs a full inference on the initial observations and initializes the feature cache, then enters a continuous loop mode. The small-scale VLM and policy network run synchronously in the main thread at a controlled frequency (e.g., 10Hz), reading the cached features of the large-scale VLM at each step. The large-scale VLM continuously loops inference at its own speed in a separate thread, updating the cache and resetting the time-encoding delay after each iteration. The two do not need to be synchronized; the update frequency of a large-scale VLM is naturally determined by its inference time.

[0085] Figure 8 This is a schematic diagram of a robot motion generation device based on dual-vision language model and hierarchical routing collaboration, according to some embodiments of this application. Figure 8 As shown, the robot motion generation device 800 based on dual-vision language model and hierarchical routing collaboration includes an acquisition module 810, a first feature extraction module 820, a second feature extraction module 830, a fusion module 840, and a motion generation module 850. The robot motion generation function based on dual-vision language model and hierarchical routing collaboration can be performed by... Figure 6 The core computing unit in the system shown executes the commands. Wherein: The acquisition module 810 is used to acquire the robot's current visual observations and high-level natural language commands; The first feature extraction module 820 is used to input the high-level natural language instruction and the current visual observation into the large-scale visual language model, which processes the instruction and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model. The second feature extraction module 830 is used to input the current subtask instruction and the current visual observation into a small-scale visual language model, which processes the data and outputs the second features of each layer of the small-scale visual language model. The small-scale visual language model has a higher running frequency than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models. The fusion module 840 is used by the policy network to adaptively select and aggregate first features from multiple layers of the large-scale visual language model and second features from multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features. The motion generation module 850 is used to output a sequence of actions for controlling the robot to perform the current sub-task based on the fused features.

[0086] In summary, the robot motion generation method and apparatus based on dual-visual language model and hierarchical routing collaboration provided in the embodiments of this application firstly, by constructing a "planning-execution" collaborative architecture composed of a large-scale and a complete small-scale VLM, the high-frequency requirements of real-time control are met while ensuring the accuracy of high-level planning, thus resolving the contradiction between capability and efficiency. The small-scale VLM, as a complete model with independent visual-language understanding capabilities, ensures the autonomy of the execution end in dynamic environments, thereby overcoming the insufficient control robustness caused by the lack of understanding capabilities of fast modules in existing dual-system schemes. Secondly, by introducing a learnable hierarchical routing mechanism, the fixed layer binding between the policy network and the VLM is broken, achieving cross-model and cross-layer adaptive feature fusion, solving the problems of design rigidity and suboptimal feature utilization caused by fixed mapping. Furthermore, by applying time encoding to cached features, the system can perceive feature freshness, enhancing the temporal stability of asynchronous collaboration. In this application, the dual-VLM architecture and hierarchical routing mechanism are interdependent and work in close collaboration, jointly achieving higher-level planning capabilities, better feature utilization efficiency, and stronger environmental robustness.

[0087] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0088] Although the foregoing embodiments are described within the general context of computer systems running in conjunction with operating systems and applications, those skilled in the art will recognize that other implementations can also be performed by incorporating other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0089] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0090] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A robot motion generation method based on dual-vision language model and hierarchical routing collaboration, characterized in that, include: Acquire the robot's current visual observations and high-level natural language commands; The high-level natural language instruction and the current visual observation are input into a large-scale visual language model, which processes them and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model. The current subtask instruction and the current visual observation are input into the small-scale visual language model, processed by the small-scale visual language model, and the second features of each layer of the small-scale visual language model are output. The small-scale visual language model runs more frequently than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models. The policy network adaptively selects and aggregates features from the first features of multiple layers of the large-scale visual language model and the second features of multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features. Based on the fusion features, the robot is output with a sequence of actions to execute the current sub-task.

2. The method according to claim 1, characterized in that, The step of inputting the high-level natural language instructions and visual observations into a large-scale visual language model, processing them, and outputting the current subtask instruction includes: The large-scale visual language model analyzes the task execution phase based on the high-level natural language instructions and the current visual observation, and outputs the current subtask instructions in text form corresponding to the task execution phase.

3. The method according to claim 1, characterized in that, Before obtaining the fused features, the policy network adaptively selects and aggregates features from multiple layers of the large-scale visual language model and multiple layers of the small-scale visual language model using learnable routing weights, and further includes: The first features of each layer output by the large-scale visual language model are cached; Based on the difference between the current time and the feature cache time, time encoding is applied to the cached first feature to obtain the time-encoded first feature, which is then used as the input for the policy network to select and aggregate.

4. The method according to claim 3, characterized in that, The time encoding uses a sinusoidal position encoding function for the difference. The coded components of the 2kth and 2k+1th dimensions are defined by the following formula: ， in, Representing feature dimension, Integer index, with a value of .

5. The method according to claim 1, characterized in that: The learnable routing weights are obtained by jointly normalizing the learnable parameters corresponding to all layers of the large-scale visual language model and all layers of the small-scale visual language model.

6. The method according to claim 5, characterized in that: The policy network is the The layer of the small-scale visual language model Second routing weights of layer features and the first large-scale visual language model First routing weight of layer features They are defined by the following formulas respectively: ， ， in, and These are the policy networks of the 1st, 2nd, and 3rd generations. Layer-by-layer small-scale visual language model Layers and large-scale visual language models The learnable routing parameters of each layer, and each layer of the policy network maintains its own independent learnable routing parameters. and These represent the total number of layers in the small-scale visual language model and the large-scale visual language model, respectively. The policy network is the Layer fusion features It is calculated using the following formula: ， in, The small-scale visual language model represents the first... The second characteristic of the layer, This represents the large-scale visual language model after time-encoded processing. The first characteristic of the layer, It is a linear projection layer used to map the hidden layer dimensions of the large-scale visual language model to a dimension space consistent with that of the small-scale visual language model; and This forms a joint softmax normalized distribution across all layers of the large-scale visual language model and all layers of the small-scale visual language model.

7. The method according to claim 6, characterized in that: The first feature and the second feature each contain multiple types of feature streams; For each type of feature flow, maintain independent learnable routing parameters. and The first route weight and the second route weight are calculated respectively to perform independent adaptive selection and aggregation.

8. The method according to claim 1, characterized in that, The method is executed using a trained neural network model, which is trained through the following steps: In the first stage, the main parameters of the large-scale visual language model and the small-scale visual language model are frozen, and the learnable routing weights and the parameters of the policy network are trained. In the second stage, the main parameters of the large-scale visual language model are unfrozen and jointly fine-tuned with the learnable routing weights, the policy network, and the small-scale visual language model.

9. The method according to claim 1, characterized in that, include: The large-scale visual language model runs continuously in a control thread independent of the small-scale visual language model and the policy network. After each run is completed, the current subtask instructions and the first features of each layer are updated to the cache, and the next run begins immediately.

10. A robot motion generation device based on dual-vision language model and hierarchical routing collaboration, characterized in that, include: The acquisition module is used to acquire the robot's current visual observations and high-level natural language commands; The first feature extraction module is used to input the high-level natural language instruction and the current visual observation into the large-scale visual language model, which processes the instruction and outputs the current subtask instruction and the first features of each layer of the large-scale visual language model. The second feature extraction module is used to input the current subtask instruction and the current visual observation into the small-scale visual language model, which processes the input and outputs the second features of each layer of the small-scale visual language model. The small-scale visual language model has a higher running frequency than the large-scale visual language model, and both the large-scale visual language model and the small-scale visual language model are complete visual language models. The fusion module is used by the policy network to adaptively select and aggregate first features from multiple layers of the large-scale visual language model and second features from multiple layers of the small-scale visual language model through learnable routing weights to obtain fused features. The action generation module is used to output an action sequence for controlling the robot to perform the current sub-task based on the fused features.

Images