Robot Control Method and System Based on VLM Dynamic Routing and Sparse Search

By using ontology-state driven dynamic routing and sparse retrieval mechanisms, the problems of fixed layer selection and information redundancy in VLA technology are solved, achieving adaptive VLM hierarchical attention and efficient robot motion prediction, which is applicable to a variety of robot platforms.

CN123103633APending Publication Date: 2026-11-17GUOCHUANG DECISION INTELLIGENCE TECHNOLOGY RESEARCH INSTITUTE (CHANGZHOU) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202611252059.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing VLA technologies suffer from problems such as fixed layer selection, redundant information transmission, and full token participation in the utilization of VLM multi-layer representations. They lack state-driven dynamic layer selection, global semantic pre-concentration, and query-guided Top-K token retrieval mechanisms.

Method used

Design a robot control method based on VLM dynamic routing and sparse retrieval. Drive the independent router of the action-by-action expert layer through the ontology state, construct a learnable Active Query Token, perform causal self-attention processing, and perform global retrieval query before cross attention. Select the Top-K Token to participate in the calculation and dynamically adjust the hierarchical attention.

Benefits of technology

It achieves adaptive selection of VLM hidden layer based on robot operation state, reduces redundant calculation, improves the concentration of effective attention resources, is suitable for rolling time domain closed-loop control, reduces computational overhead, and is adaptable to different robot platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103633A_ABST
    Figure CN123103633A_ABST
Patent Text Reader

Abstract

This invention relates to a robot control method and system based on VLM dynamic routing and sparse retrieval, belonging to the field of embodied intelligence and robot control technology. The method includes: acquiring multimodal input and extracting all hidden layer states of the VLM; constructing ontology tokens and learnable action queries; aggregating the last layer of the VLM and ontology states under causal self-attention using Active Query Tokens to form a global retrieval query; each action expert layer generating layer weights driven by the ontology state and weighted fusion; calculating the relevance of the fused candidate token pool using the global retrieval query, selecting the Top-K Tokens as the Key / Value for cross-attention; and outputting action blocks after parallel decoding following multi-layer action expert processing. This invention achieves state adaptive utilization of multi-layer VLM representations and token-level sparse retrieval, reducing computational overhead and improving action prediction accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and robot control technology, and in particular to a method and system for predicting robot motion trajectories based on VLM (Vision-Language Model) dynamic feature routing and sparse attention retrieval. Background Technology

[0002] Vision-Language-Action (VLA) models enable robots to autonomously generate continuous sequences of control actions based on visual perception input and natural language task instructions, and are one of the core technologies in the field of embodied intelligence. Current mainstream VLA technologies employ an architecture combining a fine-tuned pre-trained Vision-Language Model (VLM) with a lightweight action decoding head: a large-scale pre-trained VLM (such as the Qwen-VL series, LLaVA series, etc.) serves as the backbone network for visual and language perception, with an external action decoding network (Action Expert / Action Head) attached to its output. Through imitation learning, the multimodal representations of the VLM are mapped to robot actions from human teleoperation demonstration data. This paradigm leverages the general visual and language understanding capabilities of VLMs pre-trained on large-scale internet image and text data, significantly reducing the need for expensive robot teleoperation demonstration data.

[0003] However, this architecture faces the following three problems in actual deployment: 1. Fixed approach to utilizing VLM multi-layer representations. VLM Transformer encoders typically contain dozens of layers. The first few layers retain rich spatial texture and local geometric information, the middle layers encode object parts and spatial relationships, and the deeper layers focus on high-level semantic understanding. Different operational stages (such as needing fine spatial localization to approach the target, and needing deep semantics to understand the task semantics) have different requirements for VLM representation levels, but existing solutions generally adopt a fixed level selection strategy.

[0004] 2. Redundancy problem of VLM token sequence. The number of tokens output by VLM after processing multi-view high-resolution images can reach thousands. Among them, a large number of tokens correspond to background, irrelevant objects and other redundant information that do not contribute to the current action decision. Using all of them in the cross-attention calculation not only wastes computing resources, but also introduces noise interference.

[0005] 3. Lack of pre-condensation before multi-source information fusion. Before the visual-language token, ontology perception token, and action query token enter the cross-attention interaction, there is a lack of a step to pre-condense the global VLM semantic information into a compact guiding vector, resulting in a lengthy information transmission path and signal dilution.

[0006] In response to the above issues, the academic and industrial communities have conducted various explorations.

[0007] Direction 1: Robot motion control based on the VLA paradigm. The general paradigm of VLA has been established by Google DeepMind's RT-2 and Stanford / UC Berkeley's OpenVLA, among others: using a frozen VLM as the perception backbone, connecting an action decoding module to its output, and learning the mapping from visual language representation to action end-to-end from teleoperation teaching data through behavior cloning. RT-2 only uses the single token representation or the pooled overall representation of the final output of the VLM, discarding the spatial details of the intermediate layers; OpenVLA concatenates the ontological state on the VLM representation before the fully connected layer and then decodes it, but does not make differentiated use of the representations of different VLM layers; Octo uses a Transformer decoder as the action head and performs cross-attention interaction on the single-layer output of the VLM, but the selection of the VLM layer is fixed; π0 is a VLA scheme based on the flow matching model, using the VLM output as a conditional signal for the flow matching process, but it still uses fixed layer selection. The above schemes utilize VLM multi-layer representations in a fixed manner, with only one or a few layers, and do not have the ability to adaptively adjust the representation level attention according to the robot's current operating state.

[0008] Chinese patent CN122210661A discloses a robot control method and system based on a visual language action agent. It extracts and fuses multimodal representation vectors from the first K layers of a cloud-native multimodal visual language backbone network. These vectors are then concatenated with the robot's ontological perception vectors to construct joint conditional features. Using these joint conditional features as conditions, a continuous control action sequence is iteratively generated through a flow-matched driven pure cross-attention diffusion Transformer. Its limitations include a fixed selection of the first K VLM layers, making it impossible to adaptively adjust the inter-layer weight distribution based on the robot's current operating state; and the lack of a filtering mechanism for redundant tokens, as all visual language tokens participate in the cross-attention diffusion calculation.

[0009] Chinese patent CN122045828A discloses a method for training and applying a motion prediction model for air and ground robots. It involves inputting human operation samples into a visual language action model, extracting the hidden state sequence of the last layer output from the Transformer, and obtaining a fixed-length multimodal representation of the samples after preprocessing. Its limitation lies in using only the single last layer hidden state of the VLM Transformer, completely discarding information crucial for fine-grained operations, such as texture, spatial location, and local geometry contained in the shallow layers.

[0010] Chinese patent CN122210632A discloses a robot control method and device, focusing on the switching strategy between VLA head control mode and function call head control mode. This is an optimization at the high-level decision-making level and does not involve the utilization and fusion optimization of VLM multi-layer representation.

[0011] Direction Two: Motion Experts and Hierarchical Control Strategies. Chinese patent CN122185213A discloses a robot long-sequence control method and device based on graph thinking chains and dual-channel architecture. It maintains the explicit task state through graph actuators, driving the VLM (Virtual Machine Library) to generate text-channel intermediate products and heatmap-style spatial intermediate products. Its limitation lies in the dual-channel design's emphasis on joint text and spatial guidance at the task level, rather than the dynamic selection and fusion of the VLM representation layer. Chinese patent CN121821344A discloses a robot object manipulation method based on the collaboration of multi-motion networks and decision networks. It constructs a hierarchical control strategy combining multi-motion expert modules and decision expert modules, dynamically selecting operational skills rather than the VLM representation layer.

[0012] Direction Three: Academic Exploration of VLM Feature Utilization. Besides the patented solutions mentioned above, academic research has also explored the utilization of multi-layer features in VLM: First, there's the layer-wise feature fusion approach: simply concatenating or average pooling the outputs of multiple VLM layers, but the weights are fixed and lack state adaptability; second, there's the cross-attention over multiple layers approach: allowing the action decoder to simultaneously cross-attention multiple VLM layers, but the weights of each layer are not learnable and do not change with the input, significantly increasing computational overhead. However, these solutions still haven't solved the core problem of dynamically allocating inter-layer weights with state adaptability.

[0013] In summary, existing VLA schemes have three core defects in the utilization of VLM multi-layer representations: fixed layer selection, redundant information transmission, and full quantification of token participation. Moreover, these three problems exist independently in existing schemes, lacking an end-to-end architecture that forms a complete closed loop by integrating state-driven dynamic layer selection, global semantic pre-concentration, and query-guided Top-K token retrieval. Summary of the Invention

[0014] The technical problem to be solved by this invention is to provide a robot control method and system based on VLM dynamic routing and sparse retrieval. The technical problem to be solved includes: (1) How to design an adaptive VLM representation layer dynamic selection mechanism based on robot ontology state so that the weight distribution between layers changes with the operation state; (2) How to design a cross-modal global information pre-concentration mechanism to condense VLM semantics and ontology state into a compact guiding vector before entering cross-attention interaction; (3) How to design a query-driven Top-K Token-level sparse retrieval mechanism to select only the most relevant Token from thousands of VLM Tokens to participate in cross-attention calculation; (4) How to organically integrate the above three into an end-to-end trainable, training stable, cross-embodied robot action prediction network architecture.

[0015] The technical solution adopted by this invention to solve its technical problem is: a robot control method based on VLM dynamic routing and sparse retrieval, comprising the following steps: Acquire multimodal input, which includes images captured by at least one camera, natural language task instructions, and robot body perception data; Input the image and task instructions into the visual language model to obtain the hidden layer states of the visual language model; Construct an ontology token based on the ontology perception data; Construct a learnable Active Query Token, perform causal self-attention processing on at least some tokens of the visual language model, the ontology token, the Active Query Token and multiple learnable action query tokens, and take the output at the corresponding position of the Active Query Token as the global retrieval query; For each of the multiple action expert layers, a weight covering the multiple hidden layers is generated based on the ontology token, and the states of the multiple hidden layers are fused according to the weight to obtain the candidate token pool for that action expert layer. The global search query is correlated with each token in the candidate token pool. At least one token is selected from the candidate token pool based on the correlation. A selected token sequence is obtained based on the selected token. Using the ontology token and the action query token as queries, and the selected token sequence as the key, the action expert layer is executed, and after processing by the multiple action expert layers, a continuous action block is decoded and output.

[0016] Furthermore, the present invention describes causal self-attention processing of at least a portion of the tokens of the visual language model, the ontology token, the Active Query Token, and multiple learnable action query tokens, including: The last layer token of the visual language model, the ontology token, the Active Query token, and the action query token are concatenated in sequence to form a fusion sequence; Causal self-attention with a lower triangular causal mask is applied to the fused sequence, so that the output of the Active QueryToken depends only on the visual language model token and ontology token preceding it, and not on the action query token following it.

[0017] Furthermore, the relevance calculation of the global search query and each token in the candidate token pool described in this invention includes: projecting the global search query and each token in the candidate token pool onto the search space respectively, and calculating the scaled dot product as the relevance.

[0018] Furthermore, for each of the multiple action expert layers, the step of generating weights covering the multiple hidden layers based on the ontology token includes: inputting the ontology token into the router corresponding to the action expert layer, mapping it through the router, and outputting the weight distribution covering all hidden layers through the Softmax function, with different router parameters corresponding to different action expert layers.

[0019] Furthermore, before fusing the states of the multiple hidden layers according to the weights, the present invention further includes: retaining a preset number of hidden layers with the highest weights from the weights, renormalizing the retained weights, and using only the preset number of hidden layers for fusion.

[0020] Furthermore, the processing of the action expert layer described in this invention includes sequential execution of cross-attention, causal self-attention, and feedforward networks. Each processing branch is configured with LayerScale parameters and a gated residual structure, and the gated linear layer weights of the gated residual structure are initialized with bias zero.

[0021] Furthermore, the decoding output of the continuous action block in this invention includes: after processing by the multiple action expert layers, inputting the hidden state corresponding to the action query token into the action head and mapping it into a continuous action block; and pruning the continuous action block according to the effective action dimension mask to obtain the action sequence of the target robot's action space.

[0022] Furthermore, the present invention also includes a cross-embodied unified representation step: arranging, filling, and normalizing the ontological states and supervisory actions of different robot platforms according to a preset maximum dimension, and generating an effective dimension mask; unifying the number of different cameras to a preset maximum number of cameras, and generating a camera mask.

[0023] Meanwhile, this invention also provides a robot control system based on VLM dynamic routing and sparse retrieval, including: A multimodal input module is used to acquire images captured by at least one camera, natural language task instructions, and robot body perception data; The visual language model feature extraction module is used to input the image and the task instruction into the visual language model to obtain multiple hidden layer states of the visual language model; The ontology token construction module is used to construct an ontology token based on the ontology perception data. The Active Query fusion module is used to construct a learnable Active Query Token, which performs causal self-attention processing on at least some tokens of the visual language model, the ontology token, the Active Query Token and multiple learnable action query tokens, and takes the output at the corresponding position of the Active Query Token as the global retrieval query. The dynamic layer routing module is used to generate weights covering the multiple hidden layers based on the ontology token for each of the multiple action expert layers, and to fuse the states of the multiple hidden layers based on the weights to obtain the candidate token pool for that action expert layer. The Token sparse retrieval module is used to calculate the relevance between the global retrieval query and each Token in the candidate Token pool, select at least one Token based on the relevance, and obtain a selected Token sequence based on the selected Token. The motion expert module is used to perform the processing of the motion expert layer by using the ontology token and the motion query token as queries and the selected token sequence as key values; The action decoding module is used to decode and output continuous action blocks after processing by the multiple action expert layers.

[0024] In addition, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0025] The beneficial effect of this invention is that it solves the defects existing in the prior art. 1. According to the present invention, the action-specific expert layer independent router driven by the ontology state enables each action expert layer to adaptively generate a weight distribution covering all hidden layers of the VLM according to the current operation state of the robot (such as the arm extension angle, the distance between the end effector and the target object, etc.), instead of adopting a fixed layer strategy or a binary layer activation / skipping strategy as in the prior art (CN122210661A fixes the first K layers, CN122045828A takes the last layer, MoLe-VLA skips layers). The present invention performs weighted fusion on all layers, which can dynamically adjust the information proportion of different levels according to the operation phase: it leans towards the semantic of middle and deep layers during long-distance movement, and leans towards the spatial details of shallow layers during fine alignment.

[0026] 2. Each action expert layer has independent router parameters, so that the shallow action expert and the deep action expert can form different VLM layer attention modes, while the STAR router of MoLe-VLA makes a global layer activation decision at the LLM input stage, and cannot achieve differentiated layer selection according to the decoding depth.

[0027] 3. According to the present invention, the semantics of the last layer of the VLM and the ontology state are aggregated under causal constraints through the Active Query Token to form a unified retrieval query q , while the sparsification basis of the existing CogVLA is only instruction relevance, lacking the integration of the ontology state. The present invention enables Token retrieval to be driven by both task semantics and the current physical state of the robot, reducing the blindness of screening Tokens only by a single action query or static saliency.

[0028] 4. The present invention reduces the length of the Key / Value sequence used for cross-attention from N to K. The main Token-related complexity of full cross-attention is approximately O(M·C·N·D), while the main Token-related complexity of the present invention is O(M·N·d_r+M·C·K·D). When K<<N and d_r<<D, the main calculation overhead of cross-attention can be significantly reduced.

[0029] 5. Tokens of background, irrelevant objects and template text no longer necessarily enter all action expert layers, which is beneficial to improving the concentration of effective attention resources.

[0030] 6. The learnable action query predicts actions at multiple future moments in parallel, without the need for multiple rounds of denoising iteration like a diffusion model, and is suitable for rolling time-domain closed-loop control.

[0031] 7. Different robot platforms can enter a unified training process through maximum dimension filling, camera mask and effective action dimension mask.

[0032] 8. LayerScale, gated residuals, pre-normalization, and gradient clipping limit residual perturbations in deep action-side networks during the early stages of training. Attached Figure Description

[0033] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the present invention; Figure 3 This is a schematic diagram illustrating the causal fusion of Active Query Tokens. Figure 4 This is a schematic diagram of dynamic layer routing and sparse token retrieval; Figure 5 This is a diagram illustrating the training and deployment process. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0035] like Figures 1-5 The robot control method and system based on VLM dynamic routing and sparse retrieval shown includes the following steps: S1: Multimodal input acquisition. Acquire images, natural language task commands, and robot body perception data captured by one or more cameras at the current moment or within a time window; the body perception data includes at least one of the following: joint position, end effector pose, and gripper state.

[0036] S2: VLM Multilayer Feature Extraction. Valid camera images and task instructions are encoded according to the input template of the visual language model. The visual language model performs forward computation and outputs L hidden layer states H¹…H. L ,in B represents the batch size, N represents the number of tokens per layer, and D represents the hidden dimension. The visual language model can be frozen or fine-tuned jointly at specific stages.

[0037] S3: Ontology Token and Action Query Construction. Mapping the current ontology state to the ontology through two layers of MLP. Construct C learnable action queries and their learnable position embeddings as parallel queries for the next C action moments.

[0038] S4: Active Query Information Pre-condensation. Constructing a learnable Active Query Token. Take the last layer token of VLM Construct the fusion sequence in the following order: S0 =

[0039] in to Let C be learnable actions that query tokens. Apply multi-head causal self-attention with a lower triangular causal mask to S0, making... The output depends only on the last layer token of the preceding VLM and the main token, and does not depend on the subsequent action query token. The output at the location of the Active Query Token is taken as the global search query. .

[0040] S5: Dynamic routing per action expert layer. For the e-th layer (e=1,…,M) among M action expert layers, the ontology token is input into the router Re independent of that layer to obtain the weights covering all L VLM hidden layers. The candidate Key / Value Token pool for the action expert layer is formed by weighted summation of multiple levels of hidden states based on Token position. :

[0041] S6: Optional layer sparsity. When layer Top-K is enabled, from... The one with the highest weight is retained. Each VLM layer re-normalizes the retained weights before performing layer-weighted fusion; when not enabled, the Softmax weights of all layers are used.

[0042] S7: Query-driven Token sparse retrieval. and Projection to a low-dimensional retrieval space:

[0043]

[0044] Select the K tokens with the highest relevance:

[0045] Softmax is applied to the selected score. This forms a weighted key / value sequence:

[0046] S8: Motion Expert Update. Using the Ontology Token and motion query as the Query, For Key / Value pairs, pre-normalized cross-attention, causal self-attention, and feedforward networks are executed sequentially; each residual branch is configured with LayerScale and gated residual structure, and the weights and biases of the gated linear layers are initialized to zero.

[0047] S9: Action Block Decoding. After passing through M layers of motion experts and the final LayerNorm, the ontology token position is removed, and the C hidden motion states are mapped to a continuous action block of C×Pmax through the motion head; it is then trimmed into the target robot motion space based on the effective motion dimension mask.

[0048] S10: Model Training and Closed-Loop Execution. The future action blocks in the teaching data are used as supervision signals, and continuous regression loss is used for training. During deployment, the predicted actions can be denormalized and issued to the robot after passing safety constraints. The robot can execute the complete action block or only execute the first h steps and then re-observe and roll the prediction.

[0049] The corresponding system includes: a multimodal input module, a VLM multi-layer feature extraction module, an ontology token construction module, an Active Query fusion module, a dynamic layer routing module, a token sparse retrieval module, a motion expert module, a motion decoding module, a training module, and a robot control interface module. These modules can be implemented by one or more processors executing programs stored in memory, or they can be deployed on the robot, at an edge computing device, or on a server.

[0050] The following detailed description is provided through examples.

[0051] I. Terminology, Symbols and Optimal Parameters

[0052] Let the batch size be The hidden dimension of VLM is The number of hidden layers that VLM can access is The number of tokens per layer is The length of the action block is The maximum dimension of a unified action / state is The action expert level is Search dimensions are The number of tokens retrieved is K.

[0053] As shown in the table below:

[0054] II. Training Samples and Cross-Embodiment Preprocessing

[0055] Each training sample includes at least: multi-view images Text task Current ontology state Future Action Block Camera effective mask and optional action effective dimension mask and action time effective mask .

[0056] Image processing: Scaling or cropping images from each camera to the target resolution; using a camera mask when the number of robot cameras is less than the preset number, and not writing the fill image to the VLM message template.

[0057] Body / Motion Unification: Mapping fields such as joints, end effector poses, and grippers from different robot platforms to predefined slots. (Insufficient...) Fill in zeros for the part; the training loss only calculates the effective slots.

[0058] Normalization: Calculate the mean / standard deviation, quantiles, or upper and lower bounds for each data source or unified dataset, and perform consistent normalization on the ontology and actions; the inference output is denormalized according to the target robot statistics.

[0059] Action block construction: capturing from the current moment A future move; insufficient tail. You can discard, repeat the last frame, or use a time mask.

[0060] Multi-source sampling: Uniform, scale- or temperature-weighted sampling of 33 or more heterogeneous data sources; when fine-tuning the target domain, only complete and valid action blocks can be retained.

[0061] III. Extraction of Multi-Layer Hidden States in VLM

[0062] Write the valid camera image and the text "Task: {instruction}Predict the next robotaction." into the VLM dialogue template, perform forward computation, and obtain:

[0063] in, , , and These represent the data batch size for the model's forward computation, the number of decoder layers in the VLM model, the length of the token sequence, and the latent dimension of the model, respectively; while This represents the operation of accumulating and concatenating tensors layer by layer. In frozen mode, VLM parameters do not participate in gradient updates and remain in evaluation mode; the ontology encoder, action expert, and action head remain in training state. In alternative embodiments, only the visual adapter, the last few layers, or the low-rank adaptation parameters of the VLM can be unfrozen.

[0064] IV. Ontology Token and Action Query

[0065] At the present moment The robot acquires its own perception state and arranges, normalizes, and fills it to the maximum state dimension according to a predefined cross-body state slot order. The ontology state tensor is obtained. .in, Indicates batch size. This represents the maximum dimension of the body state used uniformly across different robot platforms. Any insufficient dimensions are filled with preset values, with zero values ​​being preferred.

[0066] The ontology state tensor Input a multilayer perceptron consisting of two fully connected layers to obtain ontology state tokens consistent with the hidden space dimensions of the visual language model:

[0067] in, This indicates the hidden feature dimension used by VLM and motion expert networks; This represents the trainable weight matrix of the first fully connected layer; This represents the trainable bias vector of the first fully connected layer; This represents the trainable weight matrix of the second fully connected layer; This represents the trainable bias vector of the second fully connected layer; This represents the activation function of the Gaussian error linear unit. The ontology state token... Used to characterize the robot's joint state, end effector state, gripper state, or a combination thereof at the current moment.

[0068] For parallel prediction of future continuous The action sequence at each action moment, setting learnable action query parameters. and learnable action position embedding .in, This indicates the number of action moments contained in the action block to be predicted. This represents the hidden feature dimension for each action query; Used to provide an initial query representation for each action moment. Used to distinguish different time positions within an action block. The two are added element-wise, and then copied or broadcast along the batch dimension. From a sample, we obtain the initial action query tensor:

[0069] in, This means keeping the action-time dimension and hidden feature dimension unchanged, and expanding the input tensor along the batch dimension to... This operation does not introduce new trainable parameters. The resulting first... The action query is used to characterize and predict the first action block. A future action moment, among which .

[0070] V. Active Query Token Pre-concentration

[0071] Construct a learnable, proactive query token: ,in This indicates that parameters are actively queried and updated through backpropagation during model training. This represents the hidden feature dimension used by the VLM and Action Expert Network. The active query token is shared among different samples and expanded along the batch dimension as follows:

[0072] in, Indicates batch size. This indicates that the input tensor is copied or broadcast along the batch dimension. This operation does not introduce new trainable parameters.

[0073] Let VLM include There are 1 accessible hidden layer; output the last one of them.

[0074] in, This indicates the layer number of the last hidden layer in the selected VLM; This indicates the length of the token sequence obtained after VLM encoding; Indicates the last hidden layer A visual language token, The visual language tokens may include image tokens, text tokens, and special tokens in the model input template.

[0075] Let the ontology state token be... The initial action query tensor is

[0076] in, Indicates the current moment of robot control; This represents the ontology state token obtained by mapping the current robot ontology perception state; This indicates the number of future action moments contained in the action block to be predicted; Indicates the first Query the initial action corresponding to each future action moment. Following the order of "visual language tokens, ontology state tokens, active query tokens, and action query tokens," the tensors mentioned above are concatenated along the sequence dimension to obtain the initial fusion sequence:

[0077] in, This represents a tensor concatenation operation performed along the token sequence dimension.

[0078] The initial fusion sequence Input a multi-head causal self-attention module with rotational position encoding to obtain the fused output:

[0079]

[0080] in, This indicates a multi-head causal self-attention operation using Rotated Position Encoding (RoPE); RoPE applies a rotational position transformation to the query vector and key vector of each attention head based on the position of each token in the fusion sequence. This represents the lower triangular causal attention mask, its first... line, number Column elements can be represented as:

[0081] Therefore, the output at the active query token position can only depend on all the visual language tokens preceding it, the ontology state token, and the active query token itself, and cannot depend on the action query tokens following it. This prevents future action queries from participating in the token retrieval conditions used to generate themselves. If the visual language sequence contains padding tokens, a valid token mask can be superimposed on the causal attention mask to prevent valid tokens from focusing on the padding position.

[0082] To clarify the meaning of each position in the fused output, Written as:

[0083] The horizontal line above the symbol represents the updated representation of the corresponding input token after causal self-attention fusion.

[0084] The fusion output corresponding to the actively queried token is used as the retrieval query for the subsequent Token Retrieval module:

[0085] in, This represents a global retrieval query determined by both the visual language context and the current robot ontology state. It is used to calculate relevance and filter task-related tokens from a dynamically fused visual language token pool. Simultaneously, it compares the updated ontology state token with the updated... Each action query token is concatenated along the sequence dimension to form the initial query sequence of the action expert network:

[0086] in, The first token is the ontology state token after visual-language context fusion, and subsequent tokens... Each token is located in the action block. The query sequence corresponds to an update action at each future action moment. This query sequence is then used as the Query input for the cross-attention and causal self-attention modules of the action expert network.

[0087] VI. Dynamic Layer Routing for Each Action Expert Layer

[0088] For the A level of motion experts, Using the entity state token single token vector As routing input. Among them, Indicates batch size. Indicates the hidden feature dimension. Indicates the level of the motion expert.

[0089] No. Each action expert layer is configured with an independent router. The preferred dimension is Two-layer MLP:

[0090]

[0091] in Indicates the number of VLM hidden layers participating in the fusion; , They represent the first The trainable weight matrix of a router's two fully connected layers. , This represents the corresponding trainable bias vector; Indicates the first Normalized weights assigned to each VLM hidden layer by each action expert layer.

[0092] If dynamic layer Top- is enabled Then keep The largest value One component is used, and the remaining components are set to zero and renormalized. Let the first... The output of each VLM hidden layer is

[0093] in, This indicates the number of tokens at each level. The dynamic fusion token sequence corresponding to each action expert layer is as follows:

[0094] in, Indicates that it is provided to the first A sequence of candidate Key / Value Tokens for each action expert layer.

[0095] Because different action expert levels are configured with independent routers The same sample can obtain different VLM level combinations in different motion expert layers. In alternative embodiments, the router backbone can be shared and layer-specific output headers can be set, or the routing input can be extended to a combination of ontology state, task text features, historical actions, or force information.

[0096] VII. Query-Driven Top-K Token Retrieval

[0097] For the Each action expert layer will retrieve vectors using Active Query. And the dynamically merged candidate token sequence Project to Dimensional search space:

[0098] in, Indicates batch size. Indicates the number of candidate tokens. Indicates the hidden feature dimension. Indicates the spatial dimension of the search; These are the trainable weight matrices for query projection and key projection, respectively. This is the corresponding trainable bias vector.

[0099] Calculate the first The correlation between each candidate token and the scaling dot product of the retrieval query:

[0100] Select the highest value from all relevance scores. Tokens:

[0101] in, This indicates the number of tokens retained by each action expert layer. This represents the corresponding set of token indexes. Softmax is applied to the relevance of the selected token to obtain the normalized search weight. And form a weighted key / value sequence:

[0102] in, Serves as the key / value input for subsequent cross-attention.

[0103] In the specific implementation, first according to Extract the selected token, multiply it by the corresponding Softmax weight, and then perform standard cross-attention Query, Key, and Value projection. If the candidate sequence contains a Padding Token, then select it in the Top-... Set its relevance to negative infinity before selection to prevent invalid tokens from being selected.

[0104] 8. Motion Expert Transformer Layer

[0105] Each motion expert layer executes the following three pre-normalized residual sub-modules in sequence: Cross attention: with As a query, As key and value, and RoPE position encoding can be applied to the query and key; Causal self-attention: The ontology token is at the beginning of the sequence, and action queries are arranged in chronological order. Each action query can only focus on the ontology token, the query for itself, and the actions preceding it. Feedforward network: adopts It uses a two-layer MLP and the GELU activation function.

[0106] For any of the above residual submodules The update process is represented as follows:

[0107]

[0108]

[0109] in, Representation layer normalization; This represents any branch in a cross-attention, causal self-attention, or feedforward network. Represents a gated linear mapping; This represents a gate vector whose values ​​range from 0 to 1; Represents the LayerScale learnable vector, preferably initialized as follows: ; This indicates element-wise multiplication. The weights and biases of the gated linear layer are preferably initialized to zero.

[0110] IX. Action Decoding and Loss Function

[0111] go through After the layer action expert, the output query sequence is processed. The execution layer normalizes and removes the first and last tokens of the sequence to obtain the hidden action state:

[0112] The input dimension of the hidden action state is... Action head:

[0113] in, Indicates batch size. Indicates the length of the action block. Indicates the hidden feature dimension. Represents the maximum dimension of the uniform action space; , and , These represent the trainable weight matrix and bias vector of the action head, respectively.

[0114] The preferred approach is to use L1 loss with an effective dimension mask.

[0115] in, Represents a real action block. This indicates whether the corresponding action dimension is valid. To prevent the default positive number with a denominator of zero, the loss function can also be replaced with Smooth L1, MSE, or other continuous motion regression losses, and velocity smoothing, acceleration smoothing, gripper classification, or collision penalty terms can be added.

[0116] 10. Training Process Implementation Examples

[0117] (1) Read train_activevla.yaml or equivalent configuration: freeze VLM, , , , Dynamic layer Top-K=16, Token K=64 .

[0118] (2) Construct a multi-source data loader to sample multiple robot datasets. The width and height of the target image are both 224 pixels, and a maximum of 3 cameras are used.

[0119] (3) Initialize the ontology MLP, action head, attention projection, router, and retrieval projection; use Xavier Uniform for linear layers; set the gated layers to zero; set LayerScale to [value missing]. .

[0120] (4) Use AdamW or an equivalent optimizer, with a learning rate of Weight decay of 0.01; training 200,000 steps, BF16 mixed precision, maximum gradient norm of 1.0, cosine annealing with 1% warm-up.

[0121] (5) Each forward pass returns predicted_actions and VLM hidden_states; calculate L1 loss based on future action blocks, and only update the action-side parameters with requires_grad=True.

[0122] (6) Save checkpoints at fixed steps and record training loss, routing entropy, weights of each layer, retrieval token distribution, throughput and memory usage.

[0123] XI. Target Domain Fine-tuning Example

[0124] (1) Load multi-source pre-training checkpoints. The target task example is ALOHA simulation insertion or chemical lab bench insertion task.

[0125] (2) Keep the VLM frozen, and the action block length can be changed to The target domain learning rate was reduced to The training involved approximately 40,000 steps.

[0126] (3) Only sample target domain samples with complete action blocks, using the same state / action field order and normalization rules as pre-training.

[0127] (4) You can choose to freeze the dynamic router and fine-tune only a few layers after the action expert; or you can choose to fine-tune only the router and retrieve the projection to adapt to the hierarchical preferences and token relevance of the target scenario.

[0128] XII. Examples of Reasoning and Robotic Execution

[0129] (1) Receive task instructions and real-time camera frames, and read the current robot body status.

[0130] (2) Generate VLM input template based on camera mask and extract all hidden layers forward.

[0131] (3) Construct ontology token, active query, and action query; complete dynamic layer routing, token retrieval, and... Layer motion experts calculate.

[0132] (4) First forward output The step block is denormalized to the target robot unit.

[0133] (5) Perform safety procedures, including joint upper and lower limits, speed / acceleration limits, gripper range, workspace constraints, collision detection, and abnormal stop.

[0134] (6) Choose to execute the complete action block or only the preceding action block. Step; in After each step, the observation is repeated and the prediction is repeated to form a rolling time-domain closed loop.

[0135] 13. Complete Application Example: Inserting test tubes into the centrifuge tray

[0136] The robot is configured with a 7-DOF robotic arm and a motorized gripper, equipped with a fixed scene camera and a wrist camera. The task instruction is: remove the blue-labeled test tube and insert it into the third hole on the centrifuge tray. The current robot state includes 7-dimensional joint angles, 7-dimensional joint velocities, 7-dimensional end-effector pose representation, and 1-dimensional gripper opening / closing distance, which are filled into a predefined sequence. .

[0137] The VLM encodes the two images and text and returns L=29 hidden states. The Active Query Token observes all the final layer VLM Tokens and the ontology Token in the causal self-attention process, obtaining... . 1st~ Each motion expert layer generates based on the current ontology state. When the robotic arm approaches the hole, different You can choose a combination of layers that include more spatial detail. Each layer is... right Retrieve K=64 tokens and bring tokens related to the tube edge, gripper contact point, and target hole location into cross-attention.

[0138] The motion head outputs incremental joint or end-effector movements for the next 50 steps. The controller executes only the first 5-10 steps each time and then re-predicts; when the visual detection or force safety module detects a tube collision, slippage, or abnormal insertion force, it stops the current motion block and triggers a repositioning or backtracking strategy.

[0139] The following changes can be made according to other implementation methods: Routing signal changes: The ontology state can be replaced or extended into a fusion vector of ontology state and task token, visual pooling token, force, touch, historical actions, and stage encoding.

[0140] Routing granularity variation: VLM layer weights can be generated separately for each action expert layer, each attention head, each action time step, or each group of tokens; a shared routing backbone and multiple output heads can be used.

[0141] Routing sparsity variations: Hard Top-K, sparse normalization, Entmax, Gumbel-Softmax, temperature annealing, or load balancing regularization can be used.

[0142] The number of Active Queries can vary: a single global Active Querie can be used, or multiple queries can be used to represent the target object, contact area, obstacle, left and right robotic arms, or motion phase, respectively.

[0143] Active Query integration variations: can employ causal self-attention, restricted bidirectional attention, PerceiverResampler, pooled attention, or gated recursive units; the key lies in forming a query representation that combines tasks and states before Token Retrieval.

[0144] Token retrieval variations: relevance can be achieved using dot product, cosine, bilinear function, MLP, locality-sensitive hashing, or learnable clustering; K can be fixed, adaptive based on routing entropy, or dynamically adjusted based on computational budget.

[0145] Search scope changes: It can be searched independently in each action expert layer, or it can be reused across layers after the first search; it can retain special text tokens, global image tokens, or nearest neighbor space tokens as a mandatory retention set.

[0146] Action generator changes: Action experts can be deterministic Transformers, diffusion Transformers, flow matching networks, discrete action token decoders, or hybrid action heads; dynamic routing and retrieval remain as conditional representation selection modules.

[0147] Time-based modeling changes: Action queries can be generated in parallel or autoregressively; time location encoding, continuous time encoding, or action frequency conditions can be added.

[0148] Cross-embodied variations: In addition to padding, robot morphology encoding, joint name tokens, URDF encoding, kinematic graph networks, or motion space adapters can be used.

[0149] Sensor variations: Images can be expanded to include depth, point clouds, event cameras, haptics, torque, sound, and IMU; these tokens can be fused with VLM tokens in Active Query or retrieved independently.

[0150] Deployment changes: It can run on robot-side GPUs, edge servers, or the cloud; VLM and motion experts can be deployed separately and cache static language tokens or visual tokens.

[0151] Training variations: Two-stage training can be adopted, first disabling hard Top-K and using soft retrieval, and then gradually shrinking K; routing entropy regularization, retrieval consistency loss, token coverage loss and cross-view consistency loss can be added.

[0152] The above description is only a specific embodiment of the present invention. Various examples and illustrations do not constitute a limitation on the substantive content of the present invention. Those skilled in the art can make modifications or variations to the above-described specific embodiments after reading the specification without departing from the substance and scope of the invention.

Claims

1. A robot control method based on VLM dynamic routing and sparse retrieval, characterized in that: Includes the following steps: Acquire multimodal input, which includes images captured by at least one camera, natural language task instructions, and robot body perception data; Input the image and task instructions into the visual language model to obtain the hidden layer states of the visual language model; Construct an ontology token based on the ontology perception data; Construct a learnable Active Query Token, perform causal self-attention processing on at least some tokens of the visual language model, the ontology token, the Active Query Token and multiple learnable action query tokens, and take the output at the corresponding position of the Active Query Token as the global retrieval query; For each of the multiple action expert layers, a weight covering the multiple hidden layers is generated based on the ontology token, and the states of the multiple hidden layers are fused according to the weight to obtain the candidate token pool for that action expert layer. The global search query is correlated with each token in the candidate token pool. At least one token is selected from the candidate token pool based on the correlation. A selected token sequence is obtained based on the selected token. Using the ontology token and the action query token as queries, and the selected token sequence as the key, the action expert layer is executed, and after processing by the multiple action expert layers, a continuous action block is decoded and output.

2. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: The step of performing causal self-attention processing on at least a portion of the tokens of the visual language model, the ontology token, the Active Query Token, and multiple learnable action query tokens includes: The last layer token of the visual language model, the ontology token, the active query token, and the action query token are concatenated in sequence to form a fusion sequence; Causal self-attention with a lower triangular causal mask is applied to the fused sequence, so that the output of the Active Query Token depends only on the visual language model Token and ontology Token preceding it, and not on the action query Token following it.

3. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: The step of calculating the relevance between the global search query and each token in the candidate token pool includes: projecting the global search query and each token in the candidate token pool onto the search space respectively, and calculating the scaled dot product as the relevance.

4. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: For each of the multiple motion expert layers, generating weights covering the multiple hidden layers based on the ontology token includes: inputting the ontology token into the router corresponding to the motion expert layer, mapping it through the router, and outputting the weight distribution covering all hidden layers through the Softmax function, with different router parameters corresponding to different motion expert layers.

5. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: Before fusing the states of the multiple hidden layers according to the weights, the method further includes: retaining a preset number of hidden layers with the highest weights from the weights, renormalizing the retained weights, and using only the preset number of hidden layers for fusion.

6. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: The processing of the action expert layer includes cross-attention, causal self-attention and feedforward network executed sequentially. Each processing branch is configured with LayerScale parameters and gated residual structure. The gated linear layer weights of the gated residual structure are initialized with bias zero.

7. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: The decoded output continuous action block includes: after processing by the multiple action expert layers, inputting the hidden state corresponding to the action query token into the action head and mapping it into a continuous action block; and pruning the continuous action block according to the effective action dimension mask to obtain the action sequence of the target robot's action space.

8. The robot control method based on VLM dynamic routing and sparse retrieval as described in claim 1, characterized in that: It also includes a cross-embodied unified representation step: arranging, filling, and normalizing the ontological states and supervised actions of different robot platforms according to a preset maximum dimension, and generating an effective dimension mask; unifying the number of different cameras to a preset maximum number of cameras, and generating a camera mask.

9. A robot control system based on VLM dynamic routing and sparse retrieval, characterized in that: include: A multimodal input module is used to acquire images captured by at least one camera, natural language task instructions, and robot body perception data; A visual language model feature extraction module is used to input the image and the task instruction into the visual language model to obtain multiple hidden layer states of the visual language model; The ontology token construction module is used to construct an ontology token based on the ontology perception data. The Active Query fusion module is used to construct a learnable Active Query Token, which performs causal self-attention processing on at least some tokens of the visual language model, the ontology token, the Active Query Token and multiple learnable action query tokens, and takes the output at the corresponding position of the Active Query Token as the global retrieval query. The dynamic layer routing module is used to generate weights covering the multiple hidden layers based on the ontology token for each of the multiple action expert layers, and to fuse the states of the multiple hidden layers based on the weights to obtain the candidate token pool for that action expert layer. The Token sparse retrieval module is used to calculate the relevance between the global retrieval query and each Token in the candidate Token pool, select at least one Token based on the relevance, and obtain a selected Token sequence based on the selected Token. The action expert module is used to perform the processing of the action expert layer by using the ontology token and the action query token as queries and the selected token sequence as key values; The action decoding module is used to decode and output continuous action blocks after processing by the multiple action expert layers.

10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Robot object operation method based on cooperation of multi-action network and decision network

    CN121821344A

  • Air-ground robot action prediction model training and application method, equipment and medium

    CN122045828A

  • Robot long sequence control method and device based on graph thinking chain and double channels

    CN122185213A

  • Robot control method and device

    CN122210632A

  • Robot control method and system based on visual language action intelligent agent

    CN122210661A