Robot control method, apparatus, device, and medium based on sparse visual tokens

Abstract

A robot control method, device, equipment and medium based on sparse visual tokens, relating to the field of robot control, comprising segmenting and remodeling a target image corresponding to a target robot to obtain dense visual tokens; performing global coordinate prediction based on the dense visual tokens to obtain a set of predicted coordinates; for each predicted coordinate, performing weighted processing on the predicted coordinate and its corresponding neighbor coordinate token to obtain a sampling token, and obtaining a sampling visual feature by aggregating all the sampling tokens; performing position embedding mapping processing on the set of predicted coordinates to obtain a coordinate embedding vector; generating sparse visual tokens based on the sampling visual feature and the coordinate embedding vector; generating target action tokens through the sparse visual tokens and target language tokens and transmitting the target action tokens to the target robot, so that the target robot implements action control based on the target action tokens. The present application can effectively ensure the control accuracy of the robot while reducing the cost of robot control.

Description

Technical Field

[0001] This application relates to the field of robot control technology, specifically to a robot control method, apparatus, device, and medium based on sparse visual tokens. Background Technology

[0002] Visual-Language-Motion Models (VLAs) integrate semantic reasoning and low-level control into a unified architecture, enabling them to exhibit strong generalization capabilities in robot manipulation tasks. However, due to the large number of tokens generated by visual encoders, VLAs face high computational costs, resulting in relatively high costs for robot control based on VLAs. Consequently, VLAs cannot better serve consumer robots and can only serve industrial robots.

[0003] To alleviate this bottleneck, researchers have begun exploring visual token pruning. For example, they have used training-independent cross-modal attention to prune tokens. However, this semantically driven selection inherently biases the model towards salient objects, neglecting geometrically critical but semantically subtle regions. Another approach utilizes temporal redundancy by caching visual features across time steps, but this relies on discrete update thresholds on a fixed grid, often failing to capture fine-grained geometric deformations during dynamic interactions. Thus, while these traditional methods reduce computational cost through pruning, effectively controlling robot control costs, their pruning relies on discrete selections on a fixed grid, introducing unavoidable quantization errors. This sacrifices the essential sub-block spatial fidelity required for precise control, thereby reducing the robot's control accuracy.

[0004] Therefore, how to reduce the cost of robot control while ensuring the control accuracy of the robot is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a robot control method, apparatus, device, and medium based on sparse visual tokens, which can effectively ensure the control accuracy of the robot while reducing the robot control cost.

[0006] In a first aspect, embodiments of this application provide a robot control method based on sparse visual tokens, the robot control method based on sparse visual tokens comprising: The target image corresponding to the target robot is segmented and reconstructed to obtain dense visual tokens; Global coordinate prediction is performed based on dense visual tokens to obtain a set of predicted coordinates; For each predicted coordinate, a sampling token is obtained by weighting the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate. The sampled visual features are obtained by aggregating all the sampled tokens. A set of predicted coordinates is processed by position embedding mapping to obtain a coordinate embedding vector; A sparse visual token is generated based on the sampled visual features and the coordinate embedding vector; A target action token is generated using the sparse visual token and the target language token, and then transmitted to the target robot so that the target robot can perform motion control based on the target action token.

[0007] In conjunction with the first aspect, in one implementation, the step of weighting the predicted coordinates and the neighbor coordinate tokens corresponding to the predicted coordinates to obtain the sampling token includes: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

[0008] In conjunction with the first aspect, in one implementation, the neighbor coordinate tokens include a top-left neighbor coordinate token, a top-right neighbor coordinate token, a bottom-left neighbor coordinate token, and a bottom-right neighbor coordinate token corresponding to the predicted coordinates.

[0009] In conjunction with the first aspect, in one implementation, the calculation expression for the interpolation weights is:

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] In the formula, This represents the interpolation weight of the top-left neighbor's coordinate token. and These represent the x and y coordinates of the predicted coordinates, respectively. and These represent the x and y coordinates of the top-left neighbor token, respectively. and These represent the x and y coordinates of the top-right neighbor's token, respectively. and These represent the x and y coordinates of the lower left neighbor's token, respectively. and These represent the x and y coordinates of the lower right neighbor token, respectively.

[0018] In conjunction with the first aspect, in one implementation, the step of weighting and solving the interpolation weights and the neighbor coordinate tokens to obtain the sampling token corresponding to the predicted coordinates includes: Substitute each neighbor coordinate token and its corresponding interpolation weight into the following calculation expression for weighted summation to obtain the sampled token corresponding to the predicted coordinates:

[0019] In the formula, This represents the sampling token corresponding to the predicted coordinates. This represents the interpolation weight of the k-th neighbor's coordinate token. This represents the coordinate token of the k-th neighbor.

[0020] In conjunction with the first aspect, in one implementation, the global coordinate prediction based on dense visual tokens to obtain a set of predicted coordinates includes: The dense visual tokens are subjected to global pooling to obtain a global feature vector; The global feature vector is mapped and transformed dimensionally to establish the correlation between visual semantic features and key region coordinates. The transformation results are normalized to obtain a set of predicted coordinates.

[0021] Secondly, embodiments of this application provide a robot control device based on sparse visual tokens, characterized in that the robot control device based on sparse visual tokens includes: The generation unit is used to segment and reshape the target image corresponding to the target robot to obtain dense visual tokens; The pruning unit is used to predict global coordinates based on dense visual tokens to obtain a set of predicted coordinates; for each predicted coordinate, a weighted processing is performed on the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate to obtain a sampled token, and a sampled visual feature is obtained by aggregating all sampled tokens; a position embedding mapping processing is performed on the set of predicted coordinates to obtain a coordinate embedding vector; and sparse visual tokens are generated based on the sampled visual features and the coordinate embedding vector. The control unit is used to generate a target action token using the sparse visual token and the target language token, and transmit the target action token to the target robot so that the target robot can perform action control based on the target action token.

[0022] In conjunction with the second aspect, in one embodiment, the pruning unit is specifically used for: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

[0023] In conjunction with the second aspect, in one implementation, the neighbor coordinate tokens include top-left neighbor coordinate tokens, top-right neighbor coordinate tokens, bottom-left neighbor coordinate tokens, and bottom-right neighbor coordinate tokens corresponding to the predicted coordinates.

[0024] In conjunction with the second aspect, in one implementation, the calculation expression for the interpolation weights is:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] In the formula, This represents the interpolation weight of the top-left neighbor's coordinate token. and These represent the x and y coordinates of the predicted coordinates, respectively. and These represent the x and y coordinates of the top-left neighbor token, respectively. and These represent the x and y coordinates of the top-right neighbor's token, respectively. and These represent the x and y coordinates of the lower left neighbor's token, respectively. and These represent the x and y coordinates of the lower right neighbor token, respectively.

[0033] In conjunction with the second aspect, in one embodiment, the pruning unit is further configured to: Substitute each neighbor coordinate token and its corresponding interpolation weight into the following calculation expression for weighted summation to obtain the sampled token corresponding to the predicted coordinates:

[0034] In the formula, This represents the sampling token corresponding to the predicted coordinates. This represents the interpolation weight of the k-th neighbor's coordinate token. This represents the coordinate token of the k-th neighbor.

[0035] In conjunction with the second aspect, in one embodiment, the pruning unit is further configured to: The dense visual tokens are subjected to global pooling to obtain a global feature vector; The global feature vector is mapped and transformed dimensionally to establish the correlation between visual semantic features and key region coordinates. The transformation results are normalized to obtain a set of predicted coordinates.

[0036] Thirdly, embodiments of this application provide a robot control device based on sparse visual tokens. The robot control device based on sparse visual tokens includes a processor, a memory, and a robot control program based on sparse visual tokens stored in the memory and executable by the processor. When the robot control program based on sparse visual tokens is executed by the processor, it implements the steps of the robot control method based on sparse visual tokens as described above.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing a robot control program based on sparse visual tokens, wherein when the robot control program based on sparse visual tokens is executed by a processor, it implements the steps of the robot control method based on sparse visual tokens as described above.

[0038] The beneficial effects of the technical solutions provided in this application include: By segmenting and reconstructing the target image corresponding to the target robot, dense visual tokens containing complete environmental information are obtained, providing basic feature representations for subsequent tasks such as target detection and image generation. Global coordinate prediction based on the dense visual tokens yields a set of predicted coordinates, indicating several key regions in the target image most relevant to the task (such as object grasping points or obstacle locations), thus identifying key points in the image worthy of feature extraction and providing a data foundation for sparse representation. For each predicted coordinate, precise boundaries are sampled by assigning balanced weights to neighbors; that is, sampling tokens are obtained by weighting the predicted coordinates and their corresponding neighbor coordinate tokens, and then aggregating all sampled tokens to obtain sampled visual features. Then, a set of predicted coordinates undergoes position embedding mapping to obtain coordinate embedding vectors, increasing expressive power while preserving spatial relationships. Sparse visual tokens are then generated based on the sampled visual features and coordinate embedding vectors, significantly reducing the number of visual tokens to be processed. Finally, target action tokens are generated using the sparse visual tokens and target language tokens, and transmitted to the target robot for action control. As can be seen, this application reformulates token compression as a continuous resampling problem rather than a discrete choice problem. That is, it is no longer limited to the fixed output grid of the visual encoder, but learns continuous feature maps at the actively querying task-driven coordinates. This can significantly compress the length of the visual sequence passed to the encoder, thereby eliminating the computational bottleneck. At the same time, it retains the basic sub-block spatial fidelity required for operation. Therefore, it can effectively ensure the control accuracy of the robot while reducing the robot control cost. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating an embodiment of the robot control method based on sparse visual tokens in this application; Figure 2 This is a schematic diagram of the token pruning model framework involved in the embodiments of this application; Figure 3 This is a schematic diagram of the neighbor coordinate tokens corresponding to the predicted coordinates involved in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the experimental results of the task success rate involved in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the experimental results of the average execution time involved in the embodiments of this application; Figure 6 This is a schematic diagram of the hardware structure of a robot control device based on sparse visual tokens involved in the embodiments of this application. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] In a first aspect, embodiments of this application provide a robot control method based on sparse visual tokens.

[0043] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the robot control method based on sparse visual tokens according to this application. Figure 1 As shown, the robot control method based on sparse visual tokens includes: Step S10: Segment and reconstruct the target image corresponding to the target robot to obtain dense visual tokens.

[0044] Exemplary, in this embodiment, the target robot refers to a robot that needs to perform an operational task, on which a VLA model is mounted; the target image refers to a first image captured by the target robot itself through its own camera device and / or a second image captured by cameras mounted in the workspace around the target robot (as above) while performing the operational task (which provides an overall understanding of the scene layout and the object's position relative to the robot's base); based on this, see... Figure 2 As shown, the target image The input image is fed into the visual encoder of the Standard Dense Representation module. This visual encoder can preferably be a Siglip model or a DINOv2 model, where H and W represent the height and width of the target image, respectively. The visual encoder segments the target image into H×W patch features and reshapes these features by flattening each patch into a one-dimensional vector, resulting in H×W feature vectors. These flattened features are then reorganized into a three-dimensional tensor of H×W×C, generating dense visual tokens (denoted as H×W×C) containing complete environmental information. ),and The dimensions are H×W×C, meaning the reshaped shape is H×W×C, where C represents the number of channels for each patch after linear transformation (e.g., C=512), representing the high-level semantic dimension of the feature. It should be noted that the specific working principle of reshaping is common knowledge in this field, and for the sake of brevity, it will not be elaborated upon here.

[0045] Furthermore, it is worth noting that complete environmental information includes, but is not limited to, spatial layout information, semantic category information, dynamic state information, geometric attribute information, and task-related context. Among these, spatial layout information describes the spatial distribution and geometric relationships of objects or scene elements in the environment, such as tile location encoding, object boundaries and shapes, and scene hierarchy. Semantic category information identifies the category labels of objects or regions in the environment, such as "chair," "ground," and "water stains." Dynamic state information describes the dynamic attributes of objects or scenes in the environment, such as motion, deformation, and changes in lighting. Geometric attribute information quantifies the physical dimensions, distances, angles, and other geometric parameters of objects. Task-related context refers to environmental information directly related to the robot's current task. For example, regarding target priority, in a search task, tokens may highlight objects similar to the target (e.g., when "finding the red cup," the feature weight of the red area is higher).

[0046] Step S20: Perform global coordinate prediction based on dense visual tokens to obtain a set of predicted coordinates.

[0047] As is evident, effective active mesh sampling requires an understanding of the scene's global semantic structure; therefore, this embodiment does not process local blocks in isolation, but rather uses dense visual tokens... Send in Figure 2 The GridS (Differentiable Grid Sampler) token pruning module in the GridS token pruning module is used to prune dense visual tokens. Global coordinate prediction is performed so that the network does not output discrete grid indices, but predicts K consecutive normalized coordinates, that is, outputs a set of predicted coordinates (P). This set of predicted coordinates P indicates the K key regions in the image that are most relevant to the task (such as object grab points or obstacle positions), thereby effectively discarding irrelevant regions while retaining the basic fine-grained details required for robot operation; where K is a positive integer, and its specific value can be determined according to actual needs, and is not limited here.

[0048] Furthermore, in one embodiment, the global coordinate prediction based on dense visual tokens to obtain a set of predicted coordinates includes: The dense visual tokens are subjected to global pooling to obtain a global feature vector; The global feature vector is mapped and transformed dimensionally to establish the correlation between visual semantic features and key region coordinates. The transformation results are normalized to obtain a set of predicted coordinates.

[0049] Exemplary, see Figure 2 As shown, the entire dense visual token is processed through a global pooling layer. Perform global average pooling aggregation to... Compressed into a global feature vector z, i.e., a global feature vector. and This operation provides a summary of the current observations, enabling the model to make informed decisions about areas of interest; subsequently, the global feature vector is processed through a lightweight multilayer perceptron (MLP layer). Dimensional mapping and nonlinear transformation are performed to establish the correlation between visual semantic features and key region coordinates, in order to predict the location of the most discriminative region. The transformation results are then normalized using the Sigmoid activation function, resulting in a set of predicted coordinates P. and ,in, It is the Sigmoid function that ensures the coordinates remain within the image boundaries. It should be understood that K... (H×W) represents the target number of activity tokens. It should be noted that the specific workflow and principles of the global pooling layer and multilayer perceptron are common knowledge in this field, and will not be elaborated upon here for the sake of brevity.

[0050] Step S30: For each predicted coordinate, perform weighted processing based on the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate to obtain a sampling token, and obtain the sampled visual feature by aggregating all the sampling tokens.

[0051] Exemplary, and noteworthy, the original dense visual token The text describes the characteristics of adjacent location coordinates, thus allowing for the use of dense visual tokens. To determine the neighbor coordinate tokens corresponding to each predicted coordinate, for example, through dense visual tokens. The specific coordinates of the top-left neighbor corresponding to the predicted coordinates can be obtained; based on this, see... Figure 2 As shown, for each predicted coordinate, the predicted coordinate and its corresponding neighbor coordinate tokens can be used to perform weighted feature extraction through a differentiable grid sampler (GridS) to obtain the sampling token corresponding to the predicted coordinate. After determining the sampling token corresponding to each predicted coordinate... Then, the sampled visual features (denoted as ) can be obtained. Its dimension is K×C, that is, sampling visual features. It is all sampling tokens A set of.

[0052] Further, in one embodiment, the step of weighting the predicted coordinates and the neighbor coordinate tokens corresponding to the predicted coordinates to obtain the sampling token includes: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

[0053] Exemplary and noteworthy, unlike traditional discrete selection strategies that suffer from quantization errors, this embodiment employs differentiable bilinear sampling to extract features from the predicted coordinates. This enables the model to extract sub-block features; for example, if a grasping point happens to be located on an edge shared by two blocks, the model can accurately extract that grasping point. Based on this, when using dense visual tokens... After determining the neighbor coordinate tokens corresponding to each predicted coordinate, feature extraction is performed on each predicted coordinate using differentiable bilinear sampling to obtain its corresponding sampling token. Since the feature extraction method and principle are the same for each predicted coordinate, for the sake of simplicity, this embodiment will use one of the predicted coordinates. For example: Regarding the predicted coordinates For each corresponding neighbor coordinate token, calculate the relationship between that neighbor coordinate token and the predicted coordinates. The horizontal and vertical offsets are used to calculate the interpolation weights of the neighbor coordinate tokens; the interpolation weights are then calculated based on these offsets and the predicted coordinates. After calculating the interpolation weights of all neighbor coordinate tokens, a weighted sum is calculated for all neighbor coordinate tokens and their corresponding interpolation weights to obtain the predicted coordinates. Corresponding sampling token That is, predict coordinates Corresponding sampling token The value is defined as the weighted interpolation of the nearest neighbor on the original grid.

[0054] Furthermore, in one embodiment, the neighbor coordinate tokens include top-left neighbor coordinate tokens, top-right neighbor coordinate tokens, bottom-left neighbor coordinate tokens, and bottom-right neighbor coordinate tokens corresponding to the predicted coordinates.

[0055] As an example, in this embodiment, in order to enable the model to perform data-driven adaptive pruning, sampling tokens will be computed using a bilinear interpolation method. This ensures that the interpolation results are continuous and first-order differentiable, thereby achieving an optimal balance between computational efficiency and accuracy; based on this, see... Figure 3 As shown, in this embodiment, the number of neighbor coordinate tokens for the predicted coordinates is preferably four, and they are the top-left neighbor coordinate tokens corresponding to the predicted coordinates. Top right neighbor coordinate token Left-bottom neighbor coordinate token and the coordinate token of the bottom right neighbor and through neighbor coordinate tokens to This is used to perform bilinear interpolation on the predicted coordinates to smooth out the coordinate prediction error and thus improve the overall reliability.

[0056] Furthermore, in one embodiment, the calculation expression for the interpolation weights is:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] In the formula, This represents the interpolation weight of the top-left neighbor's coordinate token. and These represent the x and y coordinates of the predicted coordinates, respectively. and These represent the x and y coordinates of the top-left neighbor token, respectively. and These represent the x and y coordinates of the top-right neighbor's token, respectively. and These represent the x and y coordinates of the lower left neighbor's token, respectively. and These represent the x and y coordinates of the lower right neighbor token, respectively.

[0065] As an example, in this embodiment, it is set that Let be a certain predicted coordinate in the feature map space. In this case, the assumption is that the predicted coordinates... The corresponding top-left neighbor coordinate token is Then its predicted coordinates The offset is Similarly, the top-right neighbor coordinate token To the predicted coordinates The offset is Left-bottom neighbor coordinate token To the predicted coordinates The offset is and the bottom right neighbor coordinate token To the predicted coordinates The offset is Then the four neighbor coordinate tokens { , , , Interpolation weights The calculation formulas are as follows:

[0066]

[0067]

[0068]

[0069] It is worth noting that a key characteristic of the above bilinear calculation formula is its dependence on coordinates. Fully differentiable, this allows the model to perform data-driven adaptive pruning. Furthermore, due to the weights... yes and It is a linear function, therefore the gradient can be derived from the task loss. Flow back coordinate prediction section:

[0070] This means that the MLP layer used for coordinate generation does not use a heuristic method, but is actively trained through downstream tasks to move sampling points to the region that minimizes action prediction error, so as to keep in line with specific application requirements and thus effectively improve the task success rate.

[0071] Further, in one embodiment, the step of weighting and solving the interpolation weights and the neighbor coordinate tokens to obtain the sampling token corresponding to the predicted coordinates includes: Substitute each neighbor coordinate token and its corresponding interpolation weight into the following calculation expression for weighted summation to obtain the sampled token corresponding to the predicted coordinates:

[0072] In the formula, This represents the sampling token corresponding to the predicted coordinates. This represents the interpolation weight of the k-th neighbor's coordinate token. This represents the coordinate token of the k-th neighbor.

[0073] As an example, in this embodiment, since the sampling points are typically not aligned with the center of the integer grid, this embodiment will be compared with the predicted coordinates to achieve sub-block level accuracy. Corresponding sampling token The value is defined as the weighted interpolation of the four nearest neighbors on the original grid, that is, the sampled feature is a weighted sum of the neighbor features, then in calculating the predicted coordinates When sampling tokens, substitute the four neighboring coordinate tokens and their corresponding interpolation weights into the following calculation formula to obtain the result. :

[0074] It should be understood that when k=1, This indicates the coordinates of the top-left neighbor token. This represents the interpolation weight of the top-left neighbor's coordinate token, and so on. When k=4, This indicates the coordinates of the bottom right neighbor token. This represents the interpolation weight of the bottom-right neighbor's coordinate token.

[0075] As can be seen, GridS in this embodiment samples precise boundaries by assigning balanced weights to neighbors, while discrete selection is forced to choose only one side. GridS facilitates end-to-end training with VLAs strategies without relying on auxiliary supervision, such as additional loss functions or true attention maps. This design ensures seamless compatibility with various paradigms, including autoregressive and flow matching models. In summary, the GridS provided in this embodiment can accurately locate and sample the most discriminative tokens, retaining only 6.25% (or less) of the original number. This allows the model to filter out a few task-relevant regions from dense visual tokens, thus becoming an effective pruning strategy.

[0076] Step S40: Perform position embedding mapping on a set of predicted coordinates to obtain coordinate embedding vectors.

[0077] Exemplary and understandable, this involves sampling predicted grid points from dense visual tokens to generate sampled visual features. At this time, the spatial structure of the original mesh is destroyed. Therefore, in order to restore geometry perception, this embodiment will perform mesh sampling with geometry injection, that is, as follows: Figure 2The diagram illustrates how a coordinate encoder maps a set of predicted coordinates P to location embeddings, resulting in coordinate embedding vectors (denoted as...). This allows for increased expressive power while preserving spatial relationships. It should be noted that the methods and working principles of how coordinate encoders achieve position embedding are common knowledge in this field, and will not be elaborated here for the sake of brevity.

[0078] Step S50: Generate sparse visual tokens based on the sampled visual features and the coordinate embedding vector.

[0079] Exemplary, see Figure 2 As shown, this embodiment will sample visual features. With coordinate embedding vector Element-wise addition is performed to add the coordinate embedding vector to the sampled visual features, thereby generating the final Visual Sparse Tokens (denoted as ). The dimension is K×C, which significantly reduces the number of visual tokens that need to be processed. Understandably, aggressive pruning inevitably discards key geometric details (such as contact points), leading to a severe performance degradation. This forces researchers to compromise, limiting the achievable compression ratio and thus limiting the potential acceleration effect. However, this embodiment breaks this trade-off by rethinking compression as geometrically aware, continuous token resampling in the visual encoder. Specifically, this embodiment proposes the differentiable grid sampler GridS, a plug-and-play module that performs task-aware, continuous visual token resampling in VLAs. By adaptively predicting a minimum set of salient coordinates and extracting features through differentiable interpolation, GridS can achieve significant token compression (e.g., less than 10% of the original visual tokens) while preserving essential spatial information, ensuring that the success rate of robot maneuvering tasks does not decrease.

[0080] Step S60: Generate a target action token using the sparse visual token and the target language token, and transmit the target action token to the target robot so that the target robot can perform action control based on the target action token.

[0081] Exemplary, see Figure 2 As shown, in this embodiment, sparse visual tokens are used. The target language tokens (i.e., text commands to the robot) output by VLAs are input together into the downstream Transformer model (preferably Paligemma or SmolVLM) so that the downstream Transformer model can be based on... Visual and linguistic information is fused with Language Tokens, and the initial action tokens output by VLAs are subjected to autoregressive inference based on the fusion results, thereby outputting a series of target action tokens. Finally, the target action tokens are transmitted to the target robot, so that the target robot's controller decodes these target action tokens into physical control signals to drive the target robot's actuators to complete the specified tasks, thereby achieving efficient robot control.

[0082] As can be seen, this embodiment reformulates token compression as a continuous resampling problem rather than a discrete selection problem. That is, it is no longer limited to the fixed output grid of the visual encoder, but learns continuous feature maps at the actively querying task-driven coordinates. This can significantly compress the length of the visual sequence passed to the encoder, thereby eliminating the computational bottleneck. At the same time, it retains the basic sub-block spatial fidelity required for operation. Therefore, it can effectively ensure the control accuracy of the robot while reducing the robot control cost.

[0083] To further demonstrate the advantages of the sparse visual token-based robot control method, this embodiment was tested on a standard Linux workstation equipped with an NVIDIA RTX 3090 (24GB) GPU. The experiment was based on the SmolVLA architecture and trained 50,000 optimization steps with a batch size of 16. Three real-world operation tasks were performed on the SO100 robot. The dataset was constructed as follows: (1) Pick & Place: 83 episodes in which the robot picks up a green cube from two different initial positions and places it into a green bowl; (2) Transfer Pen: 74 episodes in which the robot picks up a pen and places it into a container; (3) Stack Cubes: 75 episodes in which green cubes are stacked on top of blue cubes, including 50 trajectories for the blue cube on the right side of the SO100 and 25 trajectories for the left side.

[0084] Based on this, the experimental results are as follows: Figure 4As shown, GridS consistently outperforms the baseline across all tasks, with this performance gap being particularly pronounced in the most challenging Stack Cubes task, where GridS achieves a 60.0% success rate, compared to the baseline's mere 7.6%, resulting in a significant improvement of +52.4%. Crucially, GridS demonstrates superior robustness in Out-of-Distribution (OOD) scenarios; while the baseline fails completely in the Transfer Pen and Stack Cubes OOD settings, GridS maintains good performance. Furthermore, the sparse token method in this embodiment delivers tangible efficiency gains; see [link to relevant documentation]. Figure 5 As shown, compared to the baseline, GridS reduces the average execution time across all tasks, particularly on the Stack Cubes task, where it reduces the average execution time by 3.3 seconds. These results confirm that GridS effectively eliminates redundancy and improves policy robustness and inference speed. In summary, by replacing fixed grid redundancy with high-fidelity sparse sampling, GridS not only reduces computational costs but also achieves a significant improvement in out-of-distribution robustness. Understandably, the effectiveness of GridS stems from its grid sampler with bilinear interpolation, which uses a learnable MLP to adaptively sample using global contextual cues, ensuring that sparse tokens successfully cover most of the original semantic content. In conclusion, the goal of this embodiment is to enhance the computational efficiency of vision-language-action models, i.e., to reduce energy consumption by significantly lowering inference costs, thereby contributing to the "green AI" initiative and enabling the widespread adoption of advanced robotic functions in consumer hardware.

[0085] Secondly, embodiments of this application also provide a robot control device based on sparse visual tokens.

[0086] In one embodiment, the robot control device based on sparse visual tokens includes: The generation unit is used to segment and reshape the target image corresponding to the target robot to obtain dense visual tokens; The pruning unit is used to predict global coordinates based on dense visual tokens to obtain a set of predicted coordinates; for each predicted coordinate, a weighted processing is performed on the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate to obtain a sampled token, and a sampled visual feature is obtained by aggregating all sampled tokens; a position embedding mapping processing is performed on the set of predicted coordinates to obtain a coordinate embedding vector; and sparse visual tokens are generated based on the sampled visual features and the coordinate embedding vector. The control unit is used to generate a target action token using the sparse visual token and the target language token, and transmit the target action token to the target robot so that the target robot can perform action control based on the target action token.

[0087] Furthermore, in one embodiment, the pruning unit is specifically used for: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

[0088] Furthermore, in one embodiment, the neighbor coordinate tokens include top-left neighbor coordinate tokens, top-right neighbor coordinate tokens, bottom-left neighbor coordinate tokens, and bottom-right neighbor coordinate tokens corresponding to the predicted coordinates.

[0089] Furthermore, in one embodiment, the calculation expression for the interpolation weights is:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] In the formula, This represents the interpolation weight of the top-left neighbor's coordinate token. and These represent the x and y coordinates of the predicted coordinates, respectively. and These represent the x and y coordinates of the top-left neighbor token, respectively. and These represent the x and y coordinates of the top-right neighbor's token, respectively. and These represent the x and y coordinates of the lower left neighbor's token, respectively. and These represent the x and y coordinates of the lower right neighbor token, respectively.

[0098] Furthermore, in one embodiment, the pruning unit is specifically used for: Substitute each neighbor coordinate token and its corresponding interpolation weight into the following calculation expression for weighted summation to obtain the sampled token corresponding to the predicted coordinates:

[0099] In the formula, This represents the sampling token corresponding to the predicted coordinates. This represents the interpolation weight of the k-th neighbor's coordinate token. This represents the coordinate token of the k-th neighbor.

[0100] Furthermore, in one embodiment, the pruning unit is specifically used for: The dense visual tokens are subjected to global pooling to obtain a global feature vector; The global feature vector is mapped and transformed dimensionally to establish the correlation between visual semantic features and key region coordinates. The transformation results are normalized to obtain a set of predicted coordinates.

[0101] The functions of each unit in the robot control device based on sparse visual tokens correspond to the steps in the robot control method embodiment based on sparse visual tokens. Their functions and implementation processes will not be described in detail here.

[0102] Thirdly, embodiments of this application provide a robot control device based on sparse visual tokens. The robot control device based on sparse visual tokens can be a personal computer (PC), a laptop computer, a server, or other devices with data processing capabilities.

[0103] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of a robot control device based on sparse visual tokens, as described in an embodiment of this application. In this embodiment, the robot control device based on sparse visual tokens may include a processor, a memory, a communication interface, and a communication bus.

[0104] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0105] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the sparse vision token-based robot control device, as well as interfaces used for interconnecting the sparse vision token-based robot control device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user devices can be displays, keyboards, etc.

[0106] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0107] The processor can be a general-purpose processor, which can call the robot control program based on sparse visual tokens stored in memory and execute the robot control method based on sparse visual tokens provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the robot control program based on sparse visual tokens is called can be referred to in the various embodiments of the robot control method based on sparse visual tokens in this application, and will not be repeated here.

[0108] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0109] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0110] The present application has a readable storage medium storing a robot control program based on sparse visual tokens, wherein when the robot control program based on sparse visual tokens is executed by a processor, it implements the steps of the robot control method based on sparse visual tokens as described above.

[0111] The method implemented when the robot control program based on sparse visual tokens is executed can be referred to in various embodiments of the robot control method based on sparse visual tokens in this application, and will not be repeated here.

[0112] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0113] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0114] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0115] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0116] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0118] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A robot control method based on sparse visual tokens, characterized in that, The robot control method based on sparse visual tokens includes: The target image corresponding to the target robot is segmented and reconstructed to obtain dense visual tokens; Global coordinate prediction is performed based on dense visual tokens to obtain a set of predicted coordinates; For each predicted coordinate, a sampling token is obtained by weighting the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate. The sampled visual features are obtained by aggregating all the sampled tokens. A set of predicted coordinates is processed by position embedding mapping to obtain a coordinate embedding vector; A sparse visual token is generated based on the sampled visual features and the coordinate embedding vector; A target action token is generated using the sparse visual token and the target language token, and then transmitted to the target robot so that the target robot can perform motion control based on the target action token.

2. The robot control method based on sparse visual tokens as described in claim 1, characterized in that, The step of weighting the predicted coordinates and their corresponding neighbor coordinate tokens to obtain the sampling token includes: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

3. The robot control method based on sparse visual tokens as described in claim 2, characterized in that: The neighbor coordinate tokens include the top-left neighbor coordinate token, the top-right neighbor coordinate token, the bottom-left neighbor coordinate token, and the bottom-right neighbor coordinate token, which correspond to the predicted coordinates.

4. The robot control method based on sparse visual tokens as described in claim 3, characterized in that, The calculation expression for the interpolation weights is as follows: In the formula, This represents the interpolation weight of the top-left neighbor's coordinate token. and These represent the x and y coordinates of the predicted coordinates, respectively. and These represent the x and y coordinates of the top-left neighbor token, respectively. and These represent the x and y coordinates of the top-right neighbor's token, respectively. and These represent the x and y coordinates of the lower left neighbor's token, respectively. and These represent the x and y coordinates of the lower right neighbor token, respectively.

5. The robot control method based on sparse visual tokens as described in claim 4, characterized in that, The step of weighting and solving the interpolation weights and the neighbor coordinate tokens to obtain the sampling token corresponding to the predicted coordinates includes: Substitute each neighbor coordinate token and its corresponding interpolation weight into the following calculation expression for weighted summation to obtain the sampled token corresponding to the predicted coordinates: In the formula, This represents the sampling token corresponding to the predicted coordinates. This represents the interpolation weight of the k-th neighbor's coordinate token. This represents the coordinate token of the k-th neighbor.

6. The robot control method based on sparse visual tokens as described in claim 1, characterized in that, The global coordinate prediction based on dense visual tokens yields a set of predicted coordinates, including: The dense visual tokens are subjected to global pooling to obtain a global feature vector; The global feature vector is mapped and transformed dimensionally to establish the correlation between visual semantic features and key region coordinates. The transformation results are normalized to obtain a set of predicted coordinates.

7. A robot control device based on sparse visual tokens, characterized in that, The robot control device based on sparse visual tokens includes: The generation unit is used to segment and reshape the target image corresponding to the target robot to obtain dense visual tokens; The pruning unit is used to predict global coordinates based on dense visual tokens to obtain a set of predicted coordinates; for each predicted coordinate, a weighted processing is performed on the predicted coordinate and the neighbor coordinate tokens corresponding to the predicted coordinate to obtain a sampled token, and a sampled visual feature is obtained by aggregating all sampled tokens; a position embedding mapping processing is performed on the set of predicted coordinates to obtain a coordinate embedding vector; and sparse visual tokens are generated based on the sampled visual features and the coordinate embedding vector. The control unit is used to generate a target action token using the sparse visual token and the target language token, and transmit the target action token to the target robot so that the target robot can perform action control based on the target action token.

8. The robot control device based on sparse visual tokens as described in claim 7, characterized in that, The pruning unit is specifically used for: For each neighbor coordinate token, the interpolation weight is determined based on the offset between the neighbor coordinate token and the predicted coordinate; The interpolation weights and the neighbor coordinate tokens are weighted and summed to obtain the sampling token corresponding to the predicted coordinates.

9. A robot control device based on sparse visual tokens, characterized in that, The sparse visual token-based robot control device includes a processor, a memory, and a sparse visual token-based robot control program stored in the memory and executable by the processor, wherein when the sparse visual token-based robot control program is executed by the processor, it implements the steps of the sparse visual token-based robot control method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a robot control program based on sparse visual tokens, wherein when the robot control program based on sparse visual tokens is executed by a processor, it implements the steps of the robot control method based on sparse visual tokens as described in any one of claims 1 to 6.

Images