Robot control methods, robots and media
By dividing the robot model into a core module for perception and understanding and a motion expert module for motion generation, and optimizing each module separately, the problem of insufficient computing power in embedded devices is solved, and low-latency real-time robot control is achieved.
Patent Information
- Application Number
- CN202611613070.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
The limited computing power of embedded devices results in high prediction latency for embodied basic models, making it difficult to meet the real-time control requirements of robots.
The robot model is divided into a core module for perception and understanding and a motion expert module for action generation, and each module is optimized separately. The core module is used as an inference engine for efficient feature extraction and fusion, while the motion expert module retains its original structure for caching and reuse, reducing redundant computation.
It significantly reduces inference latency on embedded devices, meeting the requirements of real-time robot control while maintaining motion generation quality.
Smart Images

Figure CN123103704A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics, and in particular to a robot control method, a robot, and a medium. Background Technology
[0002] With the continuous development of artificial intelligence and robotics technologies, embodied models are increasingly being applied to scenarios such as humanoid robot control, robotic arm operation, and mobile robot control. Embodied models can include, for example, Vision-Language-Action Models (VLA) and World-Action Models (WAM). To improve the model's ability to understand complex environments and robotic tasks, these models often have a large number of parameters and high computational demands.
[0003] In the actual operation of humanoid robots, to reduce cloud communication latency and improve the system's independent operation capability, it is usually necessary to deploy the embodied basic model on embedded devices. However, embedded devices have relatively limited computing power, video memory capacity, and memory bandwidth. When directly running large-scale embodied basic models, there is a problem of high model prediction latency, which makes it difficult to meet the real-time control requirements of robots. Currently, there is a need for a solution for robot control to reduce the single-prediction latency of the model. Summary of the Invention
[0004] This application provides a robot control method, a robot, and a medium.
[0005] According to a first aspect, a robot control method is provided, wherein the robot has an embedded edge computing device, in which a target robot model is deployed, the target robot model including a backbone module and a motion expert module; the backbone module includes an inference engine; the inference engine is constructed based on a first part of a pre-trained original robot model used for perception and understanding; the method involves a model deployment phase and an inference application phase; the method is executed by the robot; in the inference application phase, the method includes: acquiring input data; the input data including visual image information, language command information, robot state information, and robot historical action information; using the inference engine in the backbone module to infer the input data, outputting multiple conditional features during the inference process, and fusing the multiple conditional features into a fused conditional feature; inputting the fused conditional feature into the motion expert module, and caching the multiple conditional features in the motion expert module; iteratively performing multiple denoising operations based on the fused conditional feature and the multiple conditional features to generate an action prediction result; in the partial denoising operation, caching and reusing the conditional features and intermediate features generated from the conditional features; and outputting control commands for the robot based on the action prediction result.
[0006] According to a second aspect, a robot is provided, the robot including a processor; the robot has an embedded edge computing device built in it, the embedded edge computing device deploying a target robot model, the target robot model including a backbone module and a motion expert module; the backbone module includes an inference engine; the inference engine is constructed based on a first part of a pre-trained original robot model used for perception and understanding; in the inference application stage of the target robot model: the processor is used to acquire input data; the input data includes visual image information, language command information, robot state information, and robot historical action information; the embedded edge computing device is used to infer the input data using the inference engine in the backbone module, outputting multiple conditional features during the inference process, and fusing the multiple conditional features into a fused conditional feature; inputting the fused conditional feature into the motion expert module, and caching the multiple conditional features in the motion expert module; performing multiple denoising operations iteratively based on the fused conditional feature and the multiple conditional features to generate a motion prediction result; in the partial denoising operation, caching and reusing the conditional features and intermediate features generated from the conditional features; the processor is also used to output control commands for the robot based on the motion prediction result.
[0007] According to a third aspect, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the first aspects above.
[0008] The technical solution provided in this application may include the following beneficial effects:
[0009] This application provides a robot control scheme that accelerates feature extraction and fusion through an inference engine, reducing computational latency in the perception stage. By caching conditional features in the action expert module, redundant computations in multi-step denoising are avoided. Furthermore, by reusing intermediate features in some denoising steps, the total computational load of diffusion iterations is reduced. These mechanisms collectively accelerate the entire process from perception to action generation, significantly reducing inference latency on embedded devices while ensuring the quality of action generation, thus meeting the requirements for real-time robot control.
[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This application illustrates a scenario of a robot model deployment scheme according to an exemplary embodiment. Figure 2 This application is a schematic diagram illustrating a robot control scheme scenario according to an exemplary embodiment. Figure 3 This is a flowchart illustrating a method for deploying a robot model according to an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating a robot control method according to an exemplary embodiment of this application. Detailed Implementation
[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.
[0014] Figure 1 This is a schematic diagram illustrating a deployment scenario of a robot model according to an exemplary embodiment. This embodiment demonstrates the conversion process from the original robot model to the target robot model and the deployment architecture of the target robot model on an embedded edge computing device.
[0015] like Figure 1 As shown, firstly, a pre-trained original robot model can be obtained. This original robot model can be an embodied base model, which may include a visual backbone network, a visual encoder, a language encoder, a state encoder, and a motion expert module. The visual backbone network is used to perform preliminary feature extraction on the input visual image. The visual encoder is used to further encode the extracted visual features into a high-dimensional visual representation. The language encoder is used to parse language commands into semantic features. The state encoder is used to encode the robot's own body perception information, such as joint angles and end effector poses, into state features. The motion expert module adopts a diffuse Transformer structure, which can generate continuous motion commands based on multimodal fusion features. This original robot model can be a VLA model or a WAM model.
[0016] During the deployment preparation phase, the original robot model can be divided into two parts. The first part is used to implement perception and understanding functions, including a visual backbone network, visual encoder, language encoder, and state encoder. The second part is used to implement motion generation and control functions, including a motion expert module. The first part can be converted into an inference engine to achieve efficient feature extraction and fusion, while the second part will be retained in its original form to accelerate its iterative generation process with a dedicated caching and reuse mechanism. Through this division, the two originally coupled functions in the original robot model are decoupled, thus providing the prerequisite for subsequently adopting different acceleration strategies.
[0017] Next, an inference engine can be constructed based on the first part, thereby obtaining the backbone modules of the target robot model. Specifically, the first part can be converted into an intermediate computation graph, and optimization operations can be performed on the intermediate computation graph to obtain an optimized computation graph. Based on the optimized computation graph, the inference engine is generated. During the construction of the inference engine, the computational precision of each network layer can be configured according to the sensitivity of different network layers to numerical precision in the first part, and the configured precision strategy is solidified into the optimized computation graph. Through the above construction process, the computationally intensive perception and understanding network in the first part of the original robot model is transformed into an inference engine that can run efficiently on an embedded GPU.
[0018] Finally, the backbone module and motion expert module are deployed to the embedded edge computing device. During deployment, the motion expert module remains unchanged; it is deployed directly to the embedded edge computing device with the original robot model weights and network structure, without undergoing transformation or optimization by the inference engine. This "divide and conquer" deployment strategy allows the computationally intensive perception tasks to be handled by the deeply optimized inference engine, while the motion generation tasks with multi-step iterative characteristics are completed by the motion expert module, which retains the original dynamic structure, in conjunction with a cache reuse mechanism. Thus, in this embedded edge computing device, the backbone module and motion expert module together constitute the target robot model. The target robot model is functionally equivalent to the original robot model, both capable of achieving a complete mapping from multimodal input to motion output, but with a significant improvement in computational efficiency.
[0019] Figure 2 This is a schematic diagram illustrating a robot control scheme scenario according to an exemplary embodiment. This embodiment demonstrates the inference application process of a target robot model on an embedded edge computing device. The target robot model includes a backbone module and a motion expert module, wherein the backbone module deploys an inference engine, and the motion expert module is deployed in the embedded edge computing device in its original network form.
[0020] like Figure 1As shown, in the reasoning application phase, the robot first acquires input data. This input data can include visual images, voice commands, robot state information, and historical action information. Visual images can be captured by the robot's onboard camera, providing visual information about the current scene. Voice commands can be issued by the user and converted into text-based language instructions by the speech recognition module, describing the task objectives that the robot needs to perform. Robot state information can include proprioceptive data such as the angles of the robot's joints, the position and orientation of the end effector, and the opening and closing degree of the gripper, characterizing the robot's current physical state. Historical action information can include the sequence of actions performed by the robot in the previous moment or multiple previous moments, providing a reference for the temporal smoothness of action generation. This input data forms the original information foundation for the subsequent reasoning process.
[0021] Then, preprocessing operations can be performed on the input data. These preprocessing operations may include scaling and pixel normalization of the visual image, adjusting the image to the input size required by the inference engine, and normalizing pixel values to a preset range. Speech commands are segmented and encoded, converting them into semantic vector sequences that can be processed by the language encoder. Robot state information and historical action information are normalized, mapping their numerical ranges to preset intervals. After preprocessing, the input data is converted into a standard tensor format that the inference engine can accept and then input into the inference engine.
[0022] The inference engine receives preprocessed input data and performs inference on it to output conditional features and fused conditional features. Specifically, the inference engine performs visual feature extraction on the preprocessed visual image, semantic feature extraction on the preprocessed semantic vector sequence, and state feature extraction on the preprocessed robot state information and historical action information, obtaining visual features, linguistic features, and state features. These visual features, linguistic features, and state features constitute the conditional features. The inference engine also performs multimodal fusion of the visual features, linguistic features, and state features in the conditional features to obtain fused conditional features. This fused conditional feature is a unified representation after jointly encoding the current visual scene, user task instructions, and robot body state, used to guide the subsequent action generation process. The inference engine outputs both the conditional features and the fused conditional features to the motion expert module.
[0023] The action expert module receives and caches fused conditional features. Based on these fused and conditional features, the module iteratively performs multiple denoising operations to generate action prediction results. In the initial denoising operation, guided by the fused conditional features, the module performs a complete forward computation starting from random noise, recording the key and value tensors corresponding to the conditional features, as well as the intermediate states of each network layer. In subsequent denoising operations, the module determines whether the cache reuse condition is met based on the degree of change in intermediate features between the current and previous diffusion time steps. When the cache reuse condition is met, the module skips redundant computations of the current network layer and directly reads the cached intermediate results; when the condition is not met, the module performs computations of the current network layer and updates the cached content. Through this cache reuse mechanism, the module significantly reduces redundant computations in multi-step denoising processes while maintaining action generation quality, thereby reducing inference latency.
[0024] Next, post-processing operations are performed on the motion prediction results. These operations may include: denormalizing the motion prediction results, mapping their numerical ranges from preset intervals back to the joint angle ranges and end effector position ranges in real physical space; truncating the motion prediction results by limiting joint angles or position values that exceed the robot's physical limits to a safe range; and smoothing the motion prediction results to ensure a smooth transition between the current motion output and the historical motion sequence, preventing robot jitter or loss of control due to abrupt changes in motion. After post-processing, the motion prediction results are converted into control commands that the robot can execute.
[0025] Finally, control commands can be output to the robot's actuators. These control commands include joint control commands, end effector control commands, and gripper control commands. Joint control commands drive the robot's joints to move to the target angle, achieving overall pose control of the robotic arm. End effector control commands drive the robot's end effector to move to the target position, enabling fine manipulation. Gripper control commands drive the gripper or dexterous hand to perform grasping or releasing actions.
[0026] Thus, the robot completes the entire reasoning cycle from acquiring input data to executing robot control, achieving low-latency real-time robot control.
[0027] The solutions provided in this specification will be described in detail below with reference to specific embodiments.
[0028] like Figure 3 As shown, Figure 3This is a flowchart illustrating a method for deploying a robot model according to an exemplary embodiment. This method can be applied to an execution device, which can be implemented as any device, platform, server, or device cluster with computing and processing capabilities. The method includes the following steps:
[0029] In step 301, the original robot model is divided into a first part and a second part.
[0030] In this embodiment, the original robot model can be a pre-trained embodied base model, such as a visual-language-action model or a world-action model. The original robot model may include a visual backbone network, a visual encoder, a language encoder, a state encoder, a feature fusion module, and a motion expert module. The visual backbone network performs preliminary feature extraction on the input visual image; the visual encoder further encodes the extracted visual features into high-dimensional visual representations; the language encoder parses language commands into semantic features; the state encoder encodes the robot's own joint angles, end effector poses, and other ontological perception information into state features; and the feature fusion module performs multimodal fusion of the aforementioned visual features, language features, and state features to generate fused conditional features. The motion expert module, for example, can employ a diffused Transformer structure, capable of generating continuous motion commands based on the multimodal fused features.
[0031] In addressing the problem of accelerating an original robot model, the inventors discovered that the network modules within the original robot model can be divided into parts with perception and understanding functions and parts with motion generation and control functions. In this embodiment, the original robot model can be divided into a first part for implementing perception and understanding functions and a second part for implementing motion generation and control functions. The first part may include a visual backbone network, a visual encoder, a language encoder, a state encoder, and a feature fusion module; the second part may include a motion expert module.
[0032] Dividing the original robot model into two parts clarifies the boundaries of subsequent differentiated processing. The first part will be converted into an inference engine for efficient feature extraction and fusion, while the second part will be preserved in its original form to facilitate its iterative generation process using a dedicated caching and reuse mechanism. Through this division, the two originally coupled functionalities in the original robot model are decoupled, thus providing the prerequisite for subsequently employing different acceleration strategies.
[0033] In step 302, an inference engine is constructed based on the first part, resulting in a backbone module including the inference engine.
[0034] In this embodiment, the first part can first be converted into an intermediate computation graph. During the conversion process, the model operators, constant computation nodes, redundant tensor operations, matrix multiplication and convolution operators, and intermediate tensors included in the first part are parsed into nodes or edges of the intermediate computation graph.
[0035] In this context, model operators refer to the basic operational units that constitute a neural network (such as convolution, matrix multiplication, and activation function operations). Model operators are resolved into nodes performing specific computational functions in the intermediate computation graph. Constant computation nodes are computation nodes whose inputs are fixed constants, such as weight parameters and bias parameters in a neural network. Constant computation nodes are resolved into constant nodes in the intermediate computation graph and can be pre-computed in subsequent optimizations. Redundant tensor operations are redundant operations that do not contribute to the inference result or can be merged, such as consecutive transpose operations or shape transformation operations. Redundant tensor operations are identified and marked as removable redundant nodes in the intermediate computation graph. Matrix multiplication and convolution operators are the two most computationally intensive operations in deep learning that have the most significant impact on inference latency. They are resolved into specific types of operator nodes in the intermediate computation graph. Intermediate tensors refer to the data flowing between the network layers of the model, i.e., the data output by each layer before the final output. These tensors are resolved into edges connecting nodes in the intermediate computation graph.
[0036] Based on this, optimization operations can be performed on the intermediate computation graph to obtain an optimized computation graph. These optimization operations can include fusing nodes corresponding to model operators, merging multiple consecutive computation nodes into a single composite computation node, such as merging convolutional layers, batch normalization layers, and activation function layers into one node, to reduce GPU kernel startup times and memory read / write overhead. The optimization operation can also include folding nodes corresponding to constant computation nodes, i.e., pre-computing the results of constant nodes during the construction phase and replacing the constant nodes with these results, thereby avoiding redundant computations during inference. The optimization operation can also include eliminating nodes corresponding to redundant tensor operations, i.e., removing nodes identified as redundant from the intermediate computation graph to simplify its structure. Finally, the optimization operation can also include selecting an appropriate computation kernel for the nodes corresponding to the matrix multiplication and convolution operators based on the architecture characteristics of the GPU in the embedded edge computing device, to fully utilize the GPU's computing resources. The optimization operation may also include reusing the memory of the edges corresponding to the intermediate tensor. That is, by analyzing the life cycle of each intermediate tensor, it is planned to share the memory space of the GPU in the embedded edge computing device, thereby reducing the memory usage during inference.
[0037] Finally, an inference engine can be built based on the optimized computation graph. This inference engine could be, for example, the TensorRT inference engine. Once built, the inference engine exists as a static file. This static file contains the necessary inference components and can be understood as a highly optimized executable package. Furthermore, during the inference engine construction process, the computational precision of each network layer can be configured according to its sensitivity to numerical precision as described in Part 1. Specifically, the sensitivity of each network layer to numerical precision in Part 1 can be obtained, and layers with lower sensitivity can be configured with a first precision, while layers with higher sensitivity can be configured with a second precision, where the first precision is lower than the second precision.
[0038] For example, FP16 or BF16 precision is configured for the convolutional layers, linear layers, and attention matrix computation layers in the visual backbone network to reduce computational and memory access requirements. FP32 precision is configured for the normalization layers and exponential operation layers in the visual backbone network to ensure numerical stability and model accuracy. The configured precision strategy is then embedded into the optimized computation graph. Through the above construction process, the computationally intensive perception and understanding network in the first part of the original robot model can be transformed into an inference engine that can run efficiently on an embedded GPU.
[0039] In step 303, the backbone module and motion expert module are deployed to the embedded edge computing device to obtain the target robot model.
[0040] In this embodiment, the static file-based inference engine can be loaded into the embedded edge computing device to complete the deployment of the backbone module. The motion expert module is also deployed into the embedded edge computing device, thus forming the target robot model within the device. It should be noted that the motion expert module remains unchanged during deployment; that is, it is deployed directly to the embedded edge computing device with its original model weights and network structure, without undergoing conversion or optimization by the inference engine. This divide-and-conquer deployment strategy allows the computationally intensive perception tasks to be handled by the deeply optimized inference engine, while the motion generation tasks, which involve multi-step iterations, are completed by the motion expert module, which retains the original dynamic structure, in conjunction with a cache reuse mechanism.
[0041] The target robot model is functionally equivalent to the original robot model, both capable of achieving a complete mapping from multimodal input to action output, but with significantly improved computational efficiency. The inference engine efficiently extracts features and fuses multimodal data from the input data, outputting fused conditional features. The motion expert module generates motion prediction results based on the fused conditional features through multi-step denoising iterations using a diffusion Transformer. During the denoising iteration process, conditional features and intermediate features generated from them are cached and reused to skip redundant calculations between adjacent denoising steps, thereby further reducing inference latency. Thus, the embedded edge computing device can output control commands for the robot in real time based on the input data, achieving low-latency robot control.
[0042] This application proposes a prediction acceleration scheme that optimizes different modules of a robot model separately. By dividing the original robot model into a first part and a second part, and building an inference engine for the first part, it achieves deep optimization of the computationally intensive perception and understanding parts of the model, enabling it to run efficiently on embedded edge computing devices, thereby reducing the inference latency of feature extraction and multimodal fusion. Simultaneously, this scheme deploys the motion expert module in its original form without converting the inference engine, thus preserving the original network structure and dynamic computational characteristics of this module. This allows it to cooperate with subsequent cache reuse mechanisms for multi-step denoising iterations, avoiding accuracy loss or structural damage caused by over-optimization, and achieving acceleration while ensuring the quality of motion generation. Furthermore, by deploying the backbone module and the motion expert module on the same embedded edge computing device, the perception and understanding tasks and the motion generation tasks are physically coexisting on the same device, avoiding additional latency caused by cross-device data transmission. This also allows the two modules to be updated and replaced independently, reducing the complexity of system maintenance and upgrades. Based on the above deployment method, the inference engine and motion expert module in the backbone module work together in the same embedded edge computing device, which makes the target robot model functionally equivalent to the original model, while significantly improving the overall inference efficiency and meeting the low latency requirements of robot real-time control.
[0043] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a robot control method according to an exemplary embodiment. The robot has an embedded edge computing device built-in, in which a target robot model is deployed. The target robot model includes a backbone module and a motion expert module. The backbone module includes an inference engine, which is built based on a first part of a pre-trained original robot model used for perception and understanding. The method involves a model deployment phase and an inference application phase, which is executed by the robot in the inference application phase. The method includes the following steps:
[0044] In step 401, input data is obtained.
[0045] In this embodiment, the robot can acquire input data, which may include visual image information, language command information, robot state information, and robot historical action information. The visual image information can be captured by the robot's onboard camera, providing visual information about the current scene, such as the position of the target object, the layout of the surrounding environment, and the distribution of obstacles. The language command information can be issued by the user and converted into text form by a speech recognition module. This language command describes the task objective that the robot needs to perform, such as "grab the apple on the table" or "move the object to the designated location." The robot state information may include proprioceptive data such as the angles of the robot's joints, the position and posture of the end effector, and the opening and closing degree of the gripper. This data characterizes the robot's current physical state, providing an accurate starting state reference for subsequent action generation. The historical action information may include the sequence of actions performed by the robot in the previous moment or multiple previous moments. This data provides a reference for the temporal smoothness of action generation, preventing abrupt changes in actions over time. The above input data collectively constitute the original information foundation for the reasoning process.
[0046] In step 402, the inference engine in the backbone module is used to infer the input data. During the inference process, multiple conditional features are output and fused into a fused conditional feature. In step 403, the fused conditional feature is input into the action expert module and multiple conditional features are cached in the action expert module.
[0047] In this embodiment, the inference engine in the backbone module performs inference calculations on the preprocessed input data. This inference engine is an offline-built and optimized inference engine, internally containing operator fusion strategies, computational kernel selection, and memory reuse plans for embedded GPUs. The inference engine extracts features from visual images to obtain visual features; performs semantic encoding on language instructions to obtain language features; and performs state encoding on robot state information and historical action information to obtain state features. The aforementioned visual features, language features, and state features constitute multiple conditional features.
[0048] The inference engine can also fuse multiple conditional features using a cross-modal attention mechanism through its internal feature fusion module to generate fused conditional features. These fused conditional features are a unified representation of the current visual scene, user task instructions, and robot body state after joint encoding. The fused conditional features align and compress information from different modalities, resulting in a dimensionality lower than the sum of the dimensions of the multiple conditional features, thus serving as efficient conditional guidance signals for subsequent action generation. The inference engine can transmit the multiple conditional features and the fused conditional features to the action expert module through different output interfaces. The multiple conditional features are used for subsequent cache reuse, while the fused conditional features guide the denoising process.
[0049] In this embodiment, the inference engine inputs the fused conditional features into the action expert module and caches multiple conditional features in the action expert module. When caching conditional features, the action expert module can save the above multiple conditional features into its internal cache space before the first denoising operation, so that each subsequent denoising operation can directly read these conditional features from the cache space without repeated calculation.
[0050] In step 404, multiple denoising operations are performed iteratively based on the fused conditional features and multiple conditional features to generate action prediction results. In some denoising operations, the conditional features and intermediate features generated from the conditional features are cached and reused.
[0051] In this embodiment, the action expert module can perform multiple denoising operations based on fused conditional features and multiple conditional features to generate action prediction results. The action expert module can adopt a diffusion Transformer structure, using fused conditional features as guidance, starting from random noise, and gradually generating a continuous action sequence through multiple denoising iterations. In the multi-step denoising process of the diffusion model, the fused conditional features serve as fixed guidance throughout the entire denoising process. Multiple conditional features are stored separately in key-value pairs, providing efficient query support for subsequent cross-attention calculations.
[0052] In the initial denoising operation, the action expert module can perform a complete forward computation starting from random noise, recording the key and value tensors corresponding to the conditional features, as well as the intermediate states of each network layer. In subsequent denoising operations, the action expert module can determine whether the cache reuse condition is met based on the degree of change in intermediate features between the current and previous diffusion time steps. Specifically, it can calculate the feature similarity of the intermediate features between the current and previous diffusion time steps. When the feature similarity is greater than or equal to a preset similarity, the cache reuse condition is met. When the feature similarity is less than the preset similarity, the cache reuse condition is not met. When the cache reuse condition is met, the action expert module skips the computation of the current network layer and directly reads the cached result; when the cache reuse condition is not met, the action expert module performs the computation of the current network layer and updates the cached content.
[0053] The intermediate features include the key and value tensors corresponding to the conditional features, the hidden states of some network layers in the action expert module, the output features of the attention module in the action expert module, the output features of the feedforward network in the action expert module, and intermediate features whose changes between adjacent diffusion time steps satisfy the reuse condition. Through the above-mentioned cache reuse mechanism, the action expert module significantly reduces the amount of repeated computation in the multi-step denoising process while maintaining the quality of action generation, thereby reducing inference latency.
[0054] In step 405, control commands for the robot are output based on the motion prediction results.
[0055] In this embodiment, motion prediction results can be mapped from the model output space to the robot's physical control space. This mapping process includes restoring normalized motion values to joint angle values, end effector position values, and gripper opening / closing values. Physical constraint verification can also be performed on the mapped control values, adjusting those exceeding robot joint limits or motion speed thresholds to within allowable ranges. Time-domain smoothing is applied to the control values to maintain the rate of change between the current and previous control values within a preset range, preventing robot motion instability due to sudden changes in control values. After the above processing, the control values are encoded into an instruction format that the robot's actuators can parse and sent to the corresponding execution units of the robot, achieving complete closed-loop control from perception to action.
[0056] This application provides a robot control method that accelerates feature extraction and fusion through an inference engine, reducing computational latency in the perception stage. By caching conditional features in the action expert module, redundant computations in multi-step denoising are avoided. Furthermore, by reusing intermediate features in some denoising steps, the total computational load of diffusion iterations is reduced. These mechanisms collectively accelerate the entire process from perception to action generation, significantly reducing inference latency on embedded devices while ensuring the quality of action generation, thus meeting the requirements for real-time robot control.
[0057] In related technologies, methods for accelerating inference in embodied base models typically employ a uniform optimization strategy for the entire model. This strategy ignores the fundamental differences in computational characteristics between different modules within the model. The inventors of this application, through analysis, discovered that the inference process of embodied base models exhibits a two-stage load characteristic. The first half consists of a visual backbone network, a multimodal encoder, and a feature fusion module. Its core operations are convolution, matrix multiplication, and attention calculations, which are computationally intensive tasks suitable for acceleration using a statically compiled inference engine based on graph optimization and kernel tuning. The action expert module in the second half adopts a diffusion Transformer structure. Its inference process involves multi-step iterative denoising. The intermediate feature changes between adjacent denoising steps are small, resulting in a large amount of redundant computation. Therefore, a feature caching and reuse mechanism is suitable to skip redundant computations.
[0058] Based on the above findings, this application proposes a prediction acceleration method that optimizes different modules of the robot model separately. The original robot model is split into a first part for perception and understanding and a second part for action generation. The first part is used to construct an inference engine to accelerate feature extraction and fusion. The second part retains its original network structure, and reusable intermediate features are cached and reused during its denoising iteration to reduce redundant computation. Through this divide-and-conquer differentiated acceleration strategy, the single prediction latency on embedded devices is significantly reduced while maintaining model prediction accuracy.
[0059] Therefore, although there are general methods for accelerating inference of deep learning models in the prior art, the effectiveness of single acceleration methods in scenarios where embodied base models are deployed on embedded edge computing devices to achieve real-time robot control is limited, and it is difficult to simultaneously consider the computational efficiency of the perception and generation modules. Those skilled in the art have not fully recognized the root cause of the problem: the inference process of embodied base models has a significant two-stage load characteristic. The technical solution of this application, by discovering and utilizing the aforementioned two-stage load characteristic, splits the model into a first part for perception and understanding and a second part for action generation, and employs two differentiated acceleration strategies—inference engine construction and intermediate feature cache reuse—to avoid the incompatibility problem of a single optimization method for different modules of the model. Therefore, by discovering the problem, it also solves the problem that the inference latency of embedded devices is too high in the aforementioned scenario, failing to meet the real-time control requirements of robots.
[0060] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0061] This application also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figures 3 to 4 The robot control method provided in any embodiment.
[0062] It should be noted that the computer-readable medium described in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.
[0063] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0064] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.
[0065] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0066] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above descriptions are merely specific implementations of the embodiments of this application and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this application should be included within the scope of protection of this invention.
Claims
1. A control method of a robot, the robot being built-in with an embedded edge computing device, a target robot model being deployed in the embedded edge computing device, the target robot model comprising a backbone module and action specialist modules; the backbone module comprising an inference engine; the inference engine being built based on a first part for perception and understanding in a pre-trained original robot model. The method involves a model deployment phase and an inference application phase; The method is performed by a robot; In the reasoning application phase, the method includes: Acquire input data; the input data includes visual image information, language command information, robot state information, and robot historical action information; The inference engine in the main module is used to infer the input data, and multiple conditional features are output during the inference process, and the multiple conditional features are fused into a fused conditional feature; The fusion conditional features are input into the motion expert module, and the multiple conditional features are cached in the motion expert module; Based on the fused conditional features and the multiple conditional features, multiple denoising operations are performed iteratively to generate action prediction results; in some denoising operations, the conditional features and the intermediate features generated from the conditional features are cached and reused; Based on the motion prediction results, control commands are output for the robot.
2. The method of claim 1, wherein, During the model deployment phase, the target robot model is deployed to the embedded edge computing device in the following manner: The original robot model is divided into a first part for perception and understanding and a second part for control execution; wherein, the first part includes a visual backbone network, a visual encoder, a language encoder, a state encoder and a feature fusion module, and the second part includes the motion expert module; Based on the first part, the inference engine is constructed to obtain the backbone module including the inference engine; The backbone module and the motion expert module are deployed to the embedded edge computing device to obtain the target robot model.
3. The method of claim 2, wherein, The construction of the inference engine based on the first part includes: The first part is converted into an intermediate computation graph, such that the model operators, constant computation nodes, redundant tensor operations, matrix multiplication and convolution operators included in the first part, as well as the intermediate tensor, are parsed into nodes or edges of the intermediate computation graph. An optimization operation is performed on the intermediate computation graph to obtain an optimized computation graph. The optimization operation includes: fusing the nodes corresponding to the model operators, folding the nodes corresponding to the constant computation nodes, eliminating the nodes corresponding to the redundant tensor operations, selecting a computation kernel adapted to the embedded GPU for the nodes corresponding to the matrix multiplication and convolution operators, and reusing the video memory for the edges corresponding to the intermediate tensors. The inference engine is generated based on the optimized computation graph.
4. The method according to claim 3, wherein constructing the inference engine based on the first part further comprises: Obtain the sensitivity of different network layers in the first part to numerical precision; Based on the sensitivity, a corresponding computational precision is configured for each network layer in the first part; wherein, a first precision is configured for a first network layer whose sensitivity is less than a preset threshold, and a second precision is configured for a second network layer whose sensitivity is greater than or equal to the preset threshold, wherein the first precision is lower than the second precision; The first accuracy and the second accuracy are embedded into the optimized calculation graph.
5. The method according to claim 4, wherein, The first network layer includes a convolutional layer, a linear layer, and an attention matrix calculation layer in the visual backbone network; the second network layer includes a normalization layer and an exponential operation layer in the visual backbone network.
6. The method according to claim 1, wherein, The multiple conditional features include at least visual conditional features, linguistic conditional features, and robot state conditional features.
7. The method according to claim 1, wherein, The intermediate features include one or more of the following: The key tensor and value tensor corresponding to the conditional features; The motion expert module includes the hidden states of some network layers; The motion expert module includes the output features of the attention module; The action expert module includes the output features of the feedforward network; Intermediate characteristics that satisfy the reuse condition for changes in the amount of change between adjacent diffusion time steps.
8. The method according to claim 1, wherein, In the partial denoising operation, the conditional features and the intermediate features generated from the conditional features are cached and reused, including: In non-first denoising operations, the intermediate features between the current diffusion time step and the previous diffusion time step are used to determine whether the cache reuse condition is met. When the cache reuse condition is met, skip the repeated calculation of the current network layer and directly read the cache result; when the cache reuse condition is not met, perform the calculation of the current network layer and update the cache content.
9. The method according to claim 8, wherein, The step of determining whether the cache reuse condition is met based on the intermediate features between the current diffusion time step and the previous diffusion time step includes: Calculate the feature similarity of the intermediate features between the current diffusion time step and the previous diffusion time step; When the feature similarity is greater than or equal to the preset similarity, the cache reuse condition is determined to be met.
10. A robot, the robot including a processor; the robot having an embedded edge computing device built-in, the embedded edge computing device deploying a target robot model, the target robot model including a backbone module and a motion expert module; the backbone module including an inference engine; the inference engine being built based on a first part of a pre-trained original robot model used for perception and understanding; in the inference application phase of the target robot model: The processor is used to acquire input data; the input data includes visual image information, language command information, robot state information, and robot historical action information. The embedded edge computing device is used to infer the input data using the inference engine in the backbone module, output multiple conditional features during the inference process, and fuse the multiple conditional features into a fused conditional feature; input the fused conditional feature into the action expert module, and cache the multiple conditional features in the action expert module; perform multiple denoising operations iteratively based on the fused conditional feature and the multiple conditional features to generate an action prediction result; in some denoising operations, the conditional features and the intermediate features generated from the conditional features are cached and reused; The processor is also configured to output control commands for the robot based on the motion prediction results.
11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.
