Method for controlling execution action of robot and device for controlling execution action

Abstract

The embodiment of the application discloses a robot execution action control method and an execution action control device. The method comprises the following steps: in response to the execution thread controlling the robot to complete or about to complete a first output action, acquiring current motion state information of the robot, and acquiring first observation data through the observation thread; determining a predicted action block of the robot through a trajectory predictor; determining a first action block of the robot through an inference thread; after the first time node and before the second time node, controlling the robot to execute a predicted output action corresponding to the predicted action block through the execution thread; after the second time node, controlling the robot to execute a second output action corresponding to the first action block through the execution thread. Through the scheme, motion jam in the synchronous architecture is eliminated, continuous and smooth robot control is realized, and operation continuity is improved.

Description

Technical Field

[0001] This application relates to the field of robot control, and more particularly to a method and device for controlling the execution of robot actions. Background Technology

[0002] With the rapid development of embodied intelligence technology, the vision-language-action (VLA) model has become a core technological path connecting high-level natural language reasoning with low-level robot joint control. Specifically, the VLA model, through an end-to-end approach, directly generates robotic arm joint speeds or end effector commands from visual observations and natural language instructions, enabling a leap from the traditional separate architecture (perception-planning-control) to a unified neural network architecture for robot intelligent control.

[0003] However, existing VLA model deployment schemes generally adopt synchronous control pipelines, meaning that after the robot completes the execution of the previous action chunk, it must completely stop moving and wait for the VLA model to complete the forward propagation inference of the current visual image before it can acquire the next action chunk and resume movement. This serial mode of inference-wait-execution causes obvious periodic stuttering and pauses in the robotic arm during movement, seriously affecting the continuity of operation and the success rate of tasks. Summary of the Invention

[0004] This application addresses the shortcomings by providing a robot motion control method and device. This approach completely decouples observation and acquisition, VLA model inference, and robot physical execution into three independent asynchronous threads, eliminating the robot motion stuttering problem caused by inference waiting in traditional synchronous architectures and significantly improving operational continuity. Simultaneously, a trajectory predictor generates transitional trajectories that conform to physical constraints during inference intervals, achieving full overlap between inference and execution times.

[0005] Firstly, a method for controlling the execution of actions in a robot is provided. This method is applied to a robot that acquires data through an observation thread, performs decision calculations through an inference thread, and performs execution control through an execution thread. The method includes: in response to the execution thread controlling the robot to complete or be about to complete a first output action, acquiring the robot's current motion state information; and acquiring first observation data through the observation thread, the first observation data including first body state information of the robot after completing the first output action and acquired first visual image data; determining the robot's predicted action block based on the robot's current motion state information using a trajectory predictor and the inference thread; and determining the predicted action block of the robot based on the first observation data and vision-language-action data using the inference thread. The robot uses a VLA (Virtual Area Model) to determine its first action block, which includes a sequence of first joint spatial motion points. The trajectory predictor determines a first time point of the predicted action block before the inference thread determines a second time point. After the first time point and before the second time point, the execution thread, based on the predicted action block and the underlying motion controller, controls the robot to execute the predicted output action corresponding to the predicted action block. After the second time point, the execution thread, based on the first action block, the underlying motion controller, and the robot's output action at the second time point, controls the robot to execute a second output action corresponding to the first action block.

[0006] This scheme utilizes a trajectory predictor to fill in transitional trajectories during inference intervals and generates virtual observation feedback to the inference thread, enabling the robot's movement to proceed without interruption during VLA model inference. Furthermore, it achieves complete overlap between inference time and physical execution time, eliminating the periodic stuttering inherent in traditional synchronous architectures. This allows the robot to complete complex tasks in a continuous and smooth manner, significantly improving operational coherence and task success rate.

[0007] In conjunction with the first aspect, in a possible implementation of the first aspect, the current motion state information includes the robot's joint motion state information, which includes at least one of joint position, velocity, and acceleration. Determining the robot's predicted action block based on the robot's current motion state information includes: determining the predicted observation based on the current motion state information and dynamic constraint information, whereby the dynamic constraint information includes DH parameters defining the geometric relationships of the robot's joints; and sending the predicted observation to an inference thread, so that the inference thread determines the predicted action block.

[0008] This scheme enables the trajectory predictor to generate predicted trajectory points based on the robot's kinematic model and DH parameters, and strictly adheres to physical constraints such as joint limits, velocity limits, and acceleration limits to ensure the physical rationality of the predicted trajectory and avoid safety risks caused by the predicted trajectory violating dynamic constraints.

[0009] In conjunction with the first aspect, in a possible implementation of the first aspect, controlling the robot to execute the second output action corresponding to the first action block includes: controlling the robot to transition from the output action at the second time node to the second output action through a smooth blending function.

[0010] With this approach, when a new VLA inference result arrives, the system achieves a seamless switch from the predicted trajectory to the inferred trajectory through a smooth mixing function (such as polynomial interpolation), ensuring that there are no speed or acceleration jumps during the switch and making the robot's motion transition smooth and natural.

[0011] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: obtaining first key-value cache data in the VLA model, the first key-value cache data including a key vector and a value vector; applying an orthogonal rotation transformation to the key vector and the value vector based on a vector rotation quantization algorithm, and performing low-bit quantization on the rotated key vector and the value vector to obtain optimized second key-value cache data; and projecting the second key-value cache data to a low-dimensional space based on a random projection matrix to obtain dimensionality-reduced key-value cache data.

[0012] This scheme employs vector rotation quantization and dimensionality reduction projection. First, orthogonal rotation transformation is used to eliminate the impact of outliers on quantization accuracy. Then, random projection dimensionality reduction is used to maintain the distance relationship between vectors in a probabilistic sense, which can compress the memory usage of the KV cache.

[0013] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: generating a sliding window based on the time dimension of the dimensionality-reduced key-value cache data, wherein the window size of the sliding window is determined according to the memory usage of the dimensionality-reduced key-value cache data; deleting data in the dimensionality-reduced key-value cache data that exceeds the range of the sliding window, and retaining cached data corresponding to the key frames of the robot in the dimensionality-reduced key-value cache data, wherein the key frames are determined according to the attention weights of the VLA model.

[0014] This scheme maintains a sliding window to evict expired cache entries along the time dimension, while automatically identifying and retaining key-value (KV) caches of key frames (such as task start frames and subtask switching frames) based on the attention weight distribution of the VLA model. Furthermore, it ensures that critical spatiotemporal context information is not lost while further controlling GPU memory usage, guaranteeing the quality of the VLA model's action output in long-duration tasks.

[0015] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the robot's first action block based on the first observation data and the vision-language VLA model includes: determining a visual feature vector based on the first visual image data; acquiring the current task instruction text and determining a language feature vector based on the current task instruction text; determining a first discrete action token sequence based on the visual feature vector, the language feature vector, and the action output head of the VLA model; performing speculative decoding on the first discrete action token sequence based on the VLA model to determine a second discrete action token sequence; and determining the first action block based on the second discrete action token sequence and a codebook, wherein the codebook is used to perform cluster analysis on the robot's action dataset and map continuous joint space motion trajectories to a finite number of discrete action tokens.

[0016] This approach transforms the action output space of the VLA model from continuous floating-point regression to discrete token prediction. Combined with speculative decoding techniques, it improves the generation speed of action sequences while effectively avoiding the problem of decreased action regression output accuracy caused by coarse-grained quantization.

[0017] In conjunction with the first aspect, in a possible implementation of the first aspect, in the codebook, the number of codewords corresponding to actions in the action dataset is related to the frequency of occurrence and / or the amplitude of movement of the action, wherein the higher the frequency of occurrence and / or the smaller the amplitude of movement of the action, the more codewords are correspondingly generated.

[0018] This scheme employs a non-uniform quantization strategy in the codebook design, allocating more codewords to high-frequency, fine-grained operations to ensure accuracy, and allocating fewer codewords to low-frequency, large-amplitude movements, thus achieving optimal allocation of action accuracy within a limited codebook capacity.

[0019] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: acquiring the action dataset and extracting continuous joint trajectory data of various operation tasks from the action dataset; dividing the continuous joint trajectory data into a fine operation action region and a coarse operation action region according to the motion amplitude and motion frequency; assigning a first number of codewords to the fine operation action region and a second number of codewords to the coarse operation action region, wherein the first number is greater than the second number; and performing a clustering algorithm on the fine operation action region and the coarse operation action region respectively to obtain the codebook vector corresponding to the codebook.

[0020] This scheme constructs a codebook that conforms to the distribution characteristics of actual operations by performing frequency-adaptive clustering on a large-scale robot action dataset, providing a high-quality codebook foundation for discretized action word segmentation.

[0021] In conjunction with the first aspect or any of the above implementations, in a possible implementation of the first aspect, the VLA model includes a main VLA model and a draft model. Determining a second discrete action token sequence based on the VLA model and the first discrete action token sequence includes: determining a candidate discrete action token sequence based on the first discrete action token sequence and the draft model; verifying the candidate discrete action token sequence through the main VLA model to accept matching tokens and regenerate non-matching tokens to obtain the second discrete action token sequence.

[0022] This scheme uses a draft model to quickly predict candidate token sequences, while the main VLA model verifies all candidate tokens in parallel, only regenerating mismatched tokens. This transforms the serial process of autoregressive generation into a parallel verification process, significantly improving the generation speed of action sequences without sacrificing output accuracy.

[0023] Secondly, an action control device is provided, comprising: an acquisition module, configured to acquire the current motion state information of the robot in response to the execution thread controlling the robot to complete or be about to complete a first output action, and to acquire first observation data through the observation thread, the first observation data including first body state information of the robot after completing the first output action and acquired first visual image data; a processing module, configured to determine the predicted action block of the robot based on the current motion state information of the robot using a trajectory predictor; specifically, the processing module is configured to determine the first action block of the robot based on the first observation data and a VLA model through the inference thread, the first action block being... The action block includes a sequence of spatial motion points of the robot's first joints, wherein the trajectory predictor determines the first time node of the predicted action block before the inference thread determines the second time node of the first action block; the processing module is specifically used, after the first time node and before the second time node, to control the robot to execute the predicted output action corresponding to the predicted action block through the execution thread, based on the predicted action block and the underlying motion controller; the processing module is specifically used, after the second time node, to control the robot to execute the second output action corresponding to the first action block through the execution thread, based on the first action block, the underlying motion controller, and the robot's output action at the second time node.

[0024] Thirdly, a robot is provided, comprising: at least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; and when the computer program code / instructions are executed by the processor, causing the robot to implement the action control method as described in any one of the first aspects or any possible implementations of the first aspect.

[0025] Fourthly, a readable storage medium is provided, on which instructions are stored, which, when executed on an electronic device, cause the electronic device to perform the action control method as described in any one of the first aspects or any possible implementations of the first aspect.

[0026] Fifthly, a computer program product is provided, comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform an action control method as described in any one of the first aspects or any possible implementations of the first aspect. Attached Figure Description

[0027] Figure 1 A schematic flowchart of a robot motion control method provided in an embodiment of this application is shown;

[0028] Figure 2 A schematic diagram of the core module of the execution action control device 200 provided in an embodiment of this application is shown;

[0029] Figure 3 This illustration shows a schematic diagram of an action control device provided in an embodiment of this application deployed in a humanoid robot scenario;

[0030] Figure 4 This illustration shows a schematic diagram of the module composition of an action control device according to an embodiment of this application;

[0031] Figure 5 A block diagram of the robot provided in an embodiment of this application is shown;

[0032] Figure 6 This illustration shows a structural schematic diagram of a system-on-chip (SoC) provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] First, the prior art and technical problems involved in the embodiments of this application are introduced.

[0035] With the rapid development of embodied intelligence technology, VLA models have become a core technological path connecting high-level natural language reasoning with low-level robot joint control. VLA models generate robotic arm joint speeds or end effector commands directly from visual observations and natural language instructions in an end-to-end manner, representing an important direction for robot intelligent control to leap from the traditional separate architecture (perception-planning-control) to a unified neural network architecture.

[0036] Currently, advanced VLA models typically have billions or even tens of billions of model parameters, and their inference process involves multiple computationally intensive steps such as multimodal feature encoding, cross-modal attention computation, and autoregressive action sequence generation. When run on high-performance GPU clusters in the cloud or data center, these models can demonstrate excellent task planning and motion control capabilities.

[0037] The core application scenarios of embodied intelligence require that VLA models be deployed on edge computing platforms, such as embedded motherboards, of the robot itself to meet rigid requirements such as decentralized autonomous operation, low communication latency, and privacy and security. Edge computing platforms typically have only 1 / 10 to 1 / 50 the computing power of cloud graphics processing units (GPUs), with only 8-16GB of unified memory and power consumption limited to 15-75W. Therefore, there are strict constraints in several aspects, which sharply contradicts the massive computational demands of large VLA models.

[0038] Furthermore, in the current fields of industrial and service robots, the real-time requirements for motion control are extremely high. Typical robotic arm servo control frequencies range from 100 to 1000 Hz, meaning a new joint command needs to be received every 1 to 10 milliseconds. However, the single forward inference latency of VLA models on edge devices is typically on the order of 100 to 500 milliseconds, far from meeting the requirements for real-time continuous control.

[0039] Furthermore, as mentioned earlier, existing VLA model deployment schemes generally employ synchronous control pipelines. This means that after the robot completes the execution of the previous action chunk, it must completely stop moving and wait for the VLA model to complete forward propagation inference of the current visual image before it can acquire the next action chunk and resume movement. This serial mode of inference-wait-execution causes obvious periodic stuttering and pauses in the robotic arm during movement, severely affecting the continuity of operation and the success rate of tasks.

[0040] To address the aforementioned issues, this application proposes a robot motion control method 100. Figure 1 A schematic flowchart of a robot motion control method 100 provided in an embodiment of this application is shown. Figure 1As shown, the method 100 includes steps S110 to S160.

[0041] This method is applied to robots, where the robot operates through three independent threads: an observation thread responsible for collecting data from sensors, an inference thread responsible for running a VLA model for decision calculations, and an execution thread responsible for sending motion commands to the underlying motion controller and driving the robot's movement. For example, the execution thread can refer to the software module in the robot's underlying control loop responsible for continuously issuing motion commands. The observation thread can refer to the module that reads data from sensors such as joint encoders and cameras at a fixed frequency (e.g., 30Hz).

[0042] In this method 100, by completely decoupling observation acquisition, VLA model inference, and robot physical execution into three independent asynchronous threads, the robot motion stuttering problem caused by inference waiting in traditional synchronous architectures is eliminated, significantly improving operational continuity. Simultaneously, a trajectory predictor generates transitional trajectories that conform to physical constraints during inference intervals, achieving full overlap between inference and execution times.

[0043] Step S110: In response to the execution thread controlling the robot to complete or be about to complete the first output action, obtain the current motion state information of the robot, and obtain the first observation data through the observation thread.

[0044] The first output action can refer to a block of actions or a sequence of action points that the robot has already completed. Then, when the execution thread controls the robot to complete the current first output action, or is about to complete it, it triggers the control flow for the robot's next round of output actions.

[0045] Optionally, in embodiments of this application, the current motion state information may include the robot's joint motion state information, which includes at least one of joint position, velocity, and acceleration. The first observation data may include the robot's first body state information after performing the first output action and the acquired first visual image data. For example, the first observation data includes the robot's first body state information after performing the first output action, such as joint angles, joint angular velocities, and end-effector pose, as well as first visual image data acquired by an RGB camera or depth camera, such as color images or multimodal image data of the scene.

[0046] Step S120: Based on the robot's current motion state information, determine the robot's predicted action block using the trajectory predictor and inference thread.

[0047] The trajectory predictor can be a module that predicts trajectories based on a kinematic model. It can be automatically activated when the execution thread has consumed all trajectory points of the current action block and a new VLA inference result has not yet been generated. Furthermore, the trajectory predictor can read the current motion state information (e.g., joint position, velocity, and acceleration) obtained in step S110, combine it with the robot's kinematic model (e.g., a model built using DH parameters), generate predictive observations, and send these observations to the inference thread, enabling the inference thread to determine the predicted action block. For example, DH parameters describe the coordinate transformation relationship between adjacent links, thus allowing the robot's kinematic model to be constructed based on DH parameters. By introducing DH parameters, the trajectory predictor can accurately calculate the robot's end effector's expected position and orientation in Cartesian space, while simultaneously adhering to physical constraints such as joint limits, velocity limits, and acceleration limits.

[0048] Optionally, the predicted action block can contain a series of predicted joint positions or end effector poses, strictly satisfying dynamic constraints such as joint limits, velocity limits, and acceleration limits. Simultaneously, the trajectory predictor feeds the state of the last predicted point as a virtual observation back to the inference thread, enabling the VLA model to begin the next round of inference calculations based on the predicted future state, thus achieving an overlap between inference time and physical execution time. Here, virtual observation can refer to simulated future sensor data, which can replace real observation packets and be sent to the inference thread, allowing the VLA model to begin planning subsequent actions before the physical world reaches that state.

[0049] Step S130: Determine the robot's first action block using the inference thread, based on the first observation data and the VLA model.

[0050] The first action block includes a sequence of spatial motion points of the robot's first joints. Furthermore, the trajectory predictor determines the first time point of the predicted action block before the inference thread determines the second time point. In other words, the determination time of the predicted action block is earlier than the determination time of the first action block. Therefore, since the trajectory predictor generates the predicted action block first at the first time point, while the inference thread completes the calculation of the first action block at the slightly later second time point, it can be guaranteed that when the execution thread needs new instructions, even if the VLA inference is not yet complete, the gap can be filled with the predicted action block.

[0051] Optionally, in step S130, the generation of the action sequence can be accelerated by speculative decoding technology. Specifically, step S130 may include: determining a visual feature vector based on the first visual image data; obtaining the current task instruction text and determining a language feature vector based on the current task instruction text; determining a first discrete action token sequence based on the visual feature vector, the language feature vector, and the action output head of the VLA model; performing speculative decoding on the first discrete action token sequence based on the VLA model to determine a second discrete action token sequence; and determining the first action block based on the second discrete action token sequence and the codebook.

[0052] In one embodiment of this application, speculative decoding can use a draft model with only 1 / 10 to 1 / 20 the number of parameters of the VLA main model to quickly predict the candidate token sequence. Then, the main model verifies all candidate tokens in parallel, only regenerating mismatched tokens, thereby transforming the serial autoregressive process into a parallel verification process, significantly improving the generation speed. Furthermore, in another embodiment of this application, the second discrete action token sequence can refer to the final token sequence after verification and correction by the main model.

[0053] It should be understood that this codebook is used to perform cluster analysis on the robot's motion dataset and map continuous joint space motion trajectories to a finite number of discrete motion tokens. In this codebook, the number of codewords corresponding to an action in the motion dataset is related to the frequency of occurrence and / or the amplitude of the action. Specifically, the higher the frequency of occurrence and / or the smaller the amplitude of the action, the more codewords are assigned. For example, more codewords are allocated to frequently occurring small displacement adjustments to ensure reconstruction accuracy; fewer codewords are allocated to less frequent large movements. This achieves optimal allocation of motion accuracy within a limited codebook capacity.

[0054] Optionally, in embodiments of this application, the method 100 may further include an offline codebook construction step. Specifically, embodiments of this application may acquire an action dataset and extract continuous joint trajectory data for various operational tasks from the action dataset. Then, the continuous joint trajectory data is divided into a fine-grained operational action region and a coarse-grained operational action region according to the amplitude and frequency of movement. Next, a first number of codewords is assigned to the fine-grained operational action region and a second number of codewords is assigned to the coarse-grained operational action region, where the first number is greater than the second number. Finally, a clustering algorithm (e.g., k-means) is performed on the fine-grained operational action region and the coarse-grained operational action region respectively to obtain the codebook vector corresponding to the codebook.

[0055] Step S140: After the first time node and before the second time node, the execution thread controls the robot to execute the predicted output action corresponding to the predicted action block according to the predicted action block and the underlying motion controller.

[0056] Since the trajectory predictor has already generated the predicted action block at the first time point, and the inference result of the VLA model (the first action block) can only be completed at the second time point, the execution thread will not be idle during the time window from the first time point to the second time point. Instead, it will read trajectory points from the predicted action block point by point. After that, after physical safety verification, it will be sent to the underlying motion controller to drive the robot's joint motors to move according to the predicted trajectory. In the embodiments of this application, the predicted action block strictly follows kinematic and dynamic constraints, which can ensure the continuity of robot motion and physical safety. At the same time, the trajectory points in the predicted action block can be marked with decreasing confidence weights for subsequent smooth mixing with the actual inference results.

[0057] Step S150: After the second time node, the execution thread controls the robot to execute the second output action corresponding to the first action block based on the output actions of the first action block, the underlying motion controller, and the robot at the second time node.

[0058] Specifically, after the inference thread completes the calculation of the first action block at the second time point, the execution thread immediately retrieves the first action block from the action output queue. Since the robot is in an intermediate state of executing the predicted output action at the second time point (i.e., the output action at that second time point), the execution thread can use a smooth blending function to gradually transition the robot's current output action to the trajectory points of the first action block. This ensures that there are no abrupt changes in velocity or acceleration during the transition. After the transition is complete, the execution thread continues to control the robot's movement according to the remaining trajectory points in the first action block, thus achieving a seamless transition from the predicted trajectory to the actual trajectory output by the VLA model.

[0059] It should be understood that, in the embodiments of this application, the second output action can refer to the continuous motion sequence corresponding to the first action block planned by the VLA model. The smoothing blending function can refer to an interpolation algorithm, such as multiple polynomial interpolation, whose boundary conditions are the starting position, velocity, acceleration, and the ending position, velocity, acceleration, which can guarantee the continuity of the first and second derivatives of the trajectory.

[0060] Optionally, to address the technical problem of uncontrollable memory usage and high susceptibility to memory overflow in the key-value cache when the VLA model handles long-running tasks, the embodiments of this application can further compress and maintain the KV cache. Specifically, the method further includes: obtaining first key-value cache data in the VLA model, the first key-value cache data including key vectors and value vectors; applying an orthogonal rotation transformation to the key vectors and value vectors based on a vector rotation quantization algorithm, and performing low-bit quantization on the rotated key vectors and value vectors to obtain optimized second key-value cache data; and projecting the second key-value cache data to a low-dimensional space based on a random projection matrix to obtain dimensionality-reduced key-value cache data.

[0061] It should be understood that, in the embodiments of this application, the vector rotation quantization algorithm can refer to the PolarQuant mechanism, which can make the vector components more uniformly distributed and eliminate outliers through orthogonal rotation. Low-bit quantization can refer to converting floating-point representation to INT4 or INT8 representation. The random projection matrix can refer to the sparse random matrix used in dimensionality reduction, which is used to reduce dimensionality while preserving distance relationships.

[0062] Optionally, to further control memory usage and retain key information, in embodiments of this application, a sliding window and keyframe retention mechanism can be used to eliminate old caches. Specifically, the method further includes: generating a sliding window based on the time dimension of the reduced-dimensional key-value cache data, the window size of which is determined according to the memory usage of the reduced-dimensional key-value cache data; deleting data in the reduced-dimensional key-value cache data that exceeds the range of the sliding window; and retaining cache data corresponding to the robot's keyframes in the reduced-dimensional key-value cache data, the keyframes being determined according to the attention weights of the VLA model.

[0063] In the embodiments of this application, the sliding window can refer to a fixed-length cache interval maintained in chronological order, and the oldest entries exceeding the window are evicted. Keyframes can refer to historical frames with high cumulative attention weight values ​​(e.g., top 5%), such as task start frames or subtask switching frames. The KV cache of these frames is not evicted to ensure task continuity.

[0064] This embodiment of the application completely decouples observation acquisition, VLA model inference, and robot physical execution into three independent asynchronous threads. It utilizes a trajectory predictor to generate transitional trajectories conforming to physical constraints and virtual observation feedback during inference intervals, achieving complete overlap between inference time and physical execution time. This eliminates the periodic stuttering inherent in traditional synchronous architectures, enabling the robot to complete complex tasks in a continuous and smooth manner, significantly improving operational coherence and task success rate. Furthermore, this embodiment of the application also uses vector rotation quantization and dimensionality reduction projection cascaded compression of KV cache memory, allowing edge devices to handle long-running tasks without overflow.

[0065] The following describes detailed embodiments of method 100 with reference to the accompanying drawings. Embodiment 1 illustrates the core modules of the action control device according to this application; Embodiment 2 illustrates the method 100 for action control; and Embodiment 3 describes the detailed process of method 100 in a desktop object grasping and placing task scenario.

[0066] Example 1:

[0067] Figure 2 A schematic diagram of the core module of the action control device 200 provided in an embodiment of this application is shown. Figure 2 As shown, the action control device 200 includes a multimodal observation and acquisition module, an asynchronous inference scheduler, a physical kinematic trajectory predictor, a KV cache compression acceleration module, a discretized action segmenter, and a physical security verification module.

[0068] For example, the multimodal observation and acquisition module may include an RGB camera, a depth camera, and a proprioceptive sensor (such as a joint encoder, an inertial measurement unit (IMU)) mounted on the robot body. It can be responsible for continuously acquiring visual images and body state information in step S110 at a fixed frequency (such as 30fps) and writing the acquired multimodal data into a circular buffer for subsequent modules to read asynchronously.

[0069] For another example, the asynchronous inference scheduler can be responsible for coordinating the fully asynchronous operation of the three stages: the visual observation thread, the VLA model inference, and the robot's physical execution. For instance, the asynchronous inference scheduler can maintain three independent execution threads: an observation thread, an inference thread, and an execution thread. These three threads can exchange data through a lock-free queue. Furthermore, the three threads are completely decoupled and do not block each other. Based on steps S140 and S150 of method 100, when the inference thread is performing a time-consuming VLA forward propagation, the execution thread will not pause and wait, but will continue to consume the remaining action points in the action output queue.

[0070] For another example, when the execution thread has consumed all trajectory points of the current action block and a new inference result is not yet ready, the physical kinematic trajectory predictor is activated to execute the aforementioned step S120. This physical kinematic predictor can predict the robot's expected end-effector state over several future control cycles based on the robot's kinematic model (e.g., including DH parameters) and current joint states (e.g., including joint position, velocity, and acceleration), and using forward kinematics deduction of physical constraints. Furthermore, the predicted end-effector state can be fed back to the inference thread as a virtual observation, allowing the VLA model to start the next round of inference calculations ahead of time, achieving an overlap between inference time and physical execution time. In addition, the transition trajectory points output by the predictor are marked with a low confidence level, and when a new VLA inference result arrives, a smoothing blending function can be used to achieve a seamless switch from the predicted trajectory to the inferred trajectory.

[0071] For another example, the KV cache compression acceleration module can manage the key-value cache in the VLA model architecture, preventing memory overflow during long-running continuous tasks. For instance, as mentioned earlier, this embodiment can employ a vector rotation quantization algorithm based on the PolarQuant mechanism to apply orthogonal rotation transformations to the key and value vectors in the KV cache, making the numerical distribution of vector components more uniform and eliminating the impact of outliers on quantization accuracy. Then, low-bit quantization (e.g., from FP16 to INT4) is performed on the rotated vectors. Next, combined with quantization Johnson-Lindenstrauss dimensionality reduction projection, a random projection matrix is ​​used to project the high-dimensional KV vectors into a low-dimensional space, preserving the distance relationship between vectors in a probabilistic sense, further compressing the dimension of the KV cache. The above compression process can be executed asynchronously after each inference, without consuming the computational resources of the main inference thread.

[0072] For another example, a discretized action tokenizer can transform the action output space of a VLA model from traditional continuous floating-point regression to discrete token prediction. For instance, the discretized action tokenizer can pre-build a frequency-adaptive codebook. This codebook can map continuous joint space motion trajectories to a finite number of discrete action tokens through cluster analysis of a large-scale robot motion dataset. Furthermore, the codebook design can employ a non-uniform quantization strategy. For example, more codewords are allocated to high-frequency, fine-grained actions to ensure accuracy, while fewer codewords are allocated to low-frequency, large-amplitude movements. The VLA model's action output head is replaced with a classification head, outputting the token probability distribution. Discretized output allows the VLA model to be accelerated using speculative decoding techniques.

[0073] For another example, the physical safety verification module can perform real-time safety checks on all trajectory commands to be sent to the underlying motion controller, including joint position limit verification, joint velocity limit verification, joint acceleration limit verification, self-collision detection, and singularity proximity detection. Furthermore, when the predicted trajectory point or the motion point output by VLA inference does not meet safety constraints, the safety verification module performs progressive limiting processing, correcting the violating command to the nearest safety boundary value. In the embodiments of this application, the safety verification module operates with hardware priority, independent of VLA inference and asynchronous scheduling.

[0074] Example 2:

[0075] Example 2, based on the aforementioned method 100, provides a process for VLA asynchronous inference execution and video memory compression, including steps one through four. The method of Example 2 can be executed by the multiple modules shown in Example 1.

[0076] In step one, system initialization and model loading can be performed, i.e., the initialization step. Specifically, Figure 2 After the device 100 is powered on, the asynchronous inference scheduler can read the hardware specifications of the edge computing platform and automatically select the optimal model quantization precision and KV cache compression ratio configuration. Then, the VLA model weights can be loaded into the GPU memory. If the original model weights exceed the available memory, mixed-precision quantization is performed to ensure the model is fully loaded. Next, the codebook file of the FAST action segmenter, including the codebook vector table and decoding mapping table, can be loaded. Then, a draft model, with 1 / 10 to 1 / 20 the number of parameters of the main VLA model, is loaded for inference decoding acceleration.

[0077] Subsequently, this embodiment can initialize the physical kinematic trajectory predictor, read the robot's model description file, construct the kinematic chain, and obtain the initial joint states from the joint encoder. Furthermore, it creates and starts three asynchronous execution threads: an observation thread, an inference thread, and an execution thread, and initializes a lock-free queue and a circular buffer between the threads. Finally, it initializes the KV cache compression module, pre-allocates compressed KV cache memory space, generates a dimension-reduced projection matrix, and caches it in GPU memory.

[0078] In step two, corresponding to steps S110 to S150, the embodiments of this application can execute the operation of the asynchronous inference-execution pipeline.

[0079] Corresponding to step S110, the observation thread can acquire data from the multimodal sensor at a fixed frequency (e.g., 30Hz). For example, it can acquire the current visual frame, the current joint state vector, and the current task instruction text, encapsulate them into an observation packet, and write them to the observation circular buffer.

[0080] Corresponding to step S130, the inference thread can obtain the latest observation packet from the observation buffer and perform forward inference of the VLA model. For example, the inference thread can extract visual feature vectors from images through a visual encoder and generate language feature vectors from task instructions through a language encoder. Then, the visual features and language features are fused through a cross-attention layer to generate a unified multimodal representation.

[0081] Furthermore, the action output head of the VLA model can autoregressively generate a discrete action token sequence (i.e., the aforementioned first discrete action token sequence) based on this multimodal representation. Then, based on the accelerated speculative decoding process, a draft model can be used to quickly predict candidate token sequences, while the main VLA model verifies all candidate tokens in parallel, accepting matching tokens and only regenerating unmatched tokens, ultimately obtaining the aforementioned second discrete action token sequence. Finally, the second discrete token sequence is decoded into a continuous joint space action point sequence through a codebook lookup table, where each sequence point represents a target joint position vector. Finally, the decoded action block (i.e., the aforementioned first action block) is pushed into the action output queue, and the inference completion timestamp is updated simultaneously.

[0082] Corresponding to step S150, the execution thread runs continuously at a high frequency (e.g., 100Hz), reads the latest available action block (i.e., the aforementioned first action block) from the action output queue, and can consume action points one by one at a frequency of one action point every 10ms. Each action point is sent to the underlying motion controller after being checked in real time by the physical security verification module.

[0083] However, as mentioned earlier, the inference time for the aforementioned action blocks is relatively long. Therefore, corresponding to step S120, the physical kinematic trajectory predictor can fill the inference gap. That is, when the execution thread has consumed all trajectory points of the current action block, but the inference thread has not yet output a new action block, the physical kinematic trajectory predictor is automatically activated. At this time, the physical kinematic trajectory predictor can read the robot's current joint state, joint velocity, and joint acceleration (i.e., the aforementioned current motion state information), and extrapolate based on the kinematic model and motion trend to generate transition trajectory points for the next M control cycles. For example, the process of generating the predicted trajectory can be shown in the following formula:

[0084]

[0085] In the formula, This indicates the current joint state. For joint velocity, The acceleration is the joint acceleration. The predicted trajectory strictly adheres to joint limits, velocity limits, and acceleration limits. Furthermore, transition trajectory points can be labeled with decreasing confidence weights.

[0086] The physical kinematic trajectory predictor can simultaneously feed the state of the last predicted point as a virtual observation back to the inference thread, allowing the VLA model to begin inference ahead of time based on the predicted future state. When a new VLA inference result arrives, the system switches trajectories using a smoothing mixture function. For example, the smoothing mixture function can be represented by the following formula:

[0087]

[0088] In the formula, The joint states predicted by the physical kinematic trajectory predictor. For VLA reasoning, the joint states. For a smooth transition function from 0 to 1, a fifth-order polynomial interpolation can be used, as shown in the following equation:

[0089]

[0090] The transition duration is 2-5 control cycles to ensure that there are no speed or acceleration jumps during the switching process.

[0091] In step three, this embodiment of the application can perform key-value cache compression and maintenance. Specifically, after each VLA forward inference is completed, the key-value cache compression module is executed asynchronously in the background and detects the current total video memory usage M. If M exceeds a preset threshold (usually set to 60% of available video memory), the compression process is triggered.

[0092] First, PolarQuant vector rotation quantization is performed. Specifically, for each layer of the key-value cache, the key vector matrix K∈R... T×d Sum value vector matrix V∈R T×d (T is the sequence length, d is the feature dimension), apply a pre-computed orthogonal rotation matrix R∈R d×d As shown in the following formula:

[0093]

[0094] The design of the orthogonal rotation matrix R can make the variance of the rotated vector components tend to be consistent, eliminating the loss of quantization accuracy caused by a small number of abnormally large components in the original space.

[0095] Then, low-bit asymmetric quantization is performed on the rotated K′ and V′, as shown in the following equation:

[0096]

[0097] In the formula, the quantization step size , and zero point , Calculated per channel and stored with FP16 precision.

[0098] Next, Johnson-Lindenstrauss dimension reduction projection is performed, which means applying random projection to the quantized K and V vectors for further dimension reduction. A sparse random projection matrix P∈R is generated beforehand. d×d’ Where d' < d, and the typical compression ratio is d' / d = 0.5. The dimensionality-reduced KV vector is:

[0099]

[0100] According to the Johnson-Lindenstrauss lemma, for any two vectors The inner product after projection satisfies the following formula:

[0101]

[0102] That is, the calculation error of attention score in low-dimensional space is controlled within an acceptable range with a high probability.

[0103] Next, old cache entries are evicted. In this embodiment, a sliding window is maintained for the compressed key-value cache along the time dimension, and the window size is dynamically adjusted based on available video memory. The oldest cache entries exceeding the window range are evicted, but the system can retain key-value cache entries for key frames (such as task start frames and subtask switching frames). Key frame identification can be determined by the attention weight distribution of the VLA model; for example, historical frames with the top 5% of cumulative attention weight values ​​are marked as key frames.

[0104] In step four, this embodiment of the application can construct the codebook for the action segmenter (e.g., in an offline stage). First, continuous joint trajectory data for various operational tasks can be collected from a large-scale robot action dataset, with the data volume reaching millions of trajectory segments. Then, time-frequency domain analysis is performed on all trajectory segments, dividing the action space into a fine-grained operation region (e.g., joint displacement <0.01 rad / step) and a coarse-grained motion region (e.g., joint displacement ≥0.01 rad / step) according to motion amplitude and frequency characteristics. Furthermore, 70% of the codewords can be allocated to the fine-grained operation region, and 30% to the coarse-grained motion region. K-means clustering is then performed in both regions to generate the codebook vector {c1, c2, …, c…}. N}, where N is the codebook size, such as N=512. Finally, codebook optimization is performed. In this embodiment, end-to-end fine-tuning can be used to jointly optimize the action output header and codebook vector of the VLA model, minimizing the reconstruction error of discretized encoding and decoding.

[0105] Through the solutions described in the above embodiments, this application completely eliminates the robot motion stuttering problem caused by inference waiting in traditional synchronous architectures by completely decoupling observation acquisition, VLA model inference, and robot physical execution into three independent asynchronous threads, thereby increasing the effective motion ratio to 100%. Furthermore, this application utilizes the physical priors of the robot's kinematic model to generate transitional trajectories that conform to physical constraints during inference intervals through a trajectory predictor, and achieves temporal overlap between inference and execution through a "virtual observation" mechanism, reducing the equivalent inference latency to 1 / 5 to 1 / 10 of the original latency.

[0106] Furthermore, this embodiment of the application, through the cascaded use of PolarQuant vector rotation quantization and Johnson-Lindenstrauss dimensionality reduction projection, compresses the key-value cache memory usage to 1 / 8 to 1 / 16 of its original size, enabling the VLA model to handle continuous tasks exceeding 10 minutes even on edge devices with only 16GB of video memory. Additionally, by employing a frequency-adaptive codebook to discretize the continuous action space into tokens, combined with speculative decoding techniques, the generation speed of action sequences is increased by 3-5 times, while a non-uniform quantization strategy ensures the accuracy of fine-grained operations.

[0107] Finally, the physical safety verification module in this embodiment operates independently with the highest hardware priority, ensuring that all motion commands sent to the underlying layer are rigorously verified, thus guaranteeing the robot's physical motion safety. Furthermore, the asynchronous inference scheduler, KV cache compression module, and action segmenter are all designed as general-purpose modules decoupled from specific VLA model architectures, making them applicable to VLA models with different parameter scales and different robot platforms.

[0108] Example 3:

[0109] Figure 3 This diagram illustrates a deployment scenario of an action control device provided in this application for a humanoid robot. Wherein, as... Figure 2 The illustrated motion control device 200 can be deployed in Figure 3 On the humanoid robot shown.

[0110] In the embodiments of this application, the edge computing platform may adopt the following hardware configuration: the processor may be an NVIDIA Jetson AGX Orin or an equivalent computing platform; the GPU may be a 2048 CUDA core with the Ampere architecture; the video memory may be 32GB of unified memory; the power consumption may be limited to within 60W; the VLA model construction may be based on the OpenVLA architecture (7B parameters), and after INT8 mixed precision quantization, the video memory usage is approximately 8GB; the inference latency (without acceleration) for a single forward propagation is approximately 180ms.

[0111] like Figure 3 As shown, the action control device can be deployed with observation threads, inference threads, and execution threads. For example, the observation thread can be bound to central processing unit (CPU) core 0, running at a frequency of 30Hz, with a circular buffer capacity of 10 frames; the inference thread can exclusively use the GPU, requesting the next frame observation immediately after inference is completed; the execution thread can be bound to CPU core 1, running at a frequency of 100Hz. These three threads can communicate with each other based on a single producer single consumer (SPSC) lock-free queue, resulting in a single enqueue / dequeue latency of less than 1μs.

[0112] Furthermore, in the embodiments of this application, the codebook size N can be set to 512. The number of fine-grained operation codewords is set to 358 (covering fine movements with joint displacements less than 0.01 rad), and the number of coarse-grained motion codewords is set to 154 (covering large movements with joint displacements greater than or equal to 0.01 rad). The dimension of a single codeword is consistent with the number of degrees of freedom of the robot joint, such as a 7DoF robotic arm corresponding to a 7-dimensional vector. The number of parameters in the aforementioned draft model can be 700M (1 / 10 of the main VLA model), the inference latency of the draft model is 12ms, and the prediction length gamma is 8, that is, predicting 8 candidate tokens at a time.

[0113] based on Figure 3 The humanoid robot shown, taking a tabletop object grasping and placement task as an example, allows the operator to give the voice command, "Please put the red cup on the blue plate." The natural language interface of the motion control device then encodes the command into task text and writes it to the observation buffer. The observation thread continuously acquires RGB images of the tabletop scene and the joint states of the robotic arm at 30Hz. The inference thread acquires the first observation packet, and the VLA model performs forward inference (taking approximately 56ms, accelerated by speculative decoding), generating an action block A1 containing 16 trajectory points, which is then pushed into the action output queue. The execution thread begins consuming the trajectory points of A1 at 100Hz, controlling the robotic arm to move towards the red cup.

[0114] Then, when the 14th trajectory point of A1 is consumed, the inference thread has completed the inference for A2 based on the new observation, and A2 is pushed into the queue. The execution thread seamlessly switches from A1 to A2 without any pause. The robotic arm continues to move smoothly.

[0115] Subsequently, during the execution of the 5th action block, the task duration reached a cumulative 30 seconds. The key-value cache compression module was triggered in the background, compressing the key-value cache from 412MB to 52MB. The robotic arm completed the grasping action (gripper closing) in the 8th action block and the placement action (gripper releasing) in the 12th action block, taking approximately 4.8 seconds in total. In addition, during the execution, the physical safety verification module intercepted and corrected 3 trajectory points that exceeded the speed limit (all caused by extrapolation errors from the trajectory predictor), all of which were safely resolved through speed limiting.

[0116] Furthermore, the motion control device of this application embodiment can be adapted to different robot platforms without retraining the VLA model by loading the unified robot description format (URDF) description file of the target robot, and the trajectory predictor automatically parses the kinematic chain, joint limit range, and dynamic parameters. Then, a pre-trained FAST motion codebook file for the joint configuration of the target robot is loaded. For new robot models, only about 10 hours of operational data needs to be collected on the platform to train the new codebook. Finally, the parameter configuration file (joint limit values, velocity limit values, collision detection model) of the safety verification module is modified. The asynchronous inference scheduler and KV cache compression module can run directly on the new platform without any modification.

[0117] It is understood that the specific functions and corresponding technical effects of the execution action control device provided in this embodiment can be referred to the explanation of the above embodiment. Figure 4This diagram illustrates the module composition of an execution action control device 400 provided in an embodiment of this application. Specifically, the acquisition module 410 is used to acquire the current motion state information of the robot in response to the execution thread controlling the robot to complete or be about to complete a first output action, and to acquire first observation data through the observation thread. The first observation data includes first body state information of the robot after completing the first output action and first visual image data collected. The processing module 420 is used to determine the predicted action block of the robot based on the current motion state information of the robot through a trajectory predictor. Specifically, the processing module 420 is used to determine the first action block of the robot through the inference thread based on the first observation data and the visual-language VLA model. The first action block includes the robot's... A sequence of spatial motion points of a human's first joint, wherein the trajectory predictor determines the first time node of the predicted motion block before the inference thread determines the second time node of the first motion block; the processing module 420 is specifically used, after the first time node and before the second time node, through the execution thread, to control the robot to execute the predicted output action corresponding to the predicted motion block according to the predicted motion block and the underlying motion controller; the processing module 420 is specifically used, after the second time node, through the execution thread, to control the robot to execute the second output action corresponding to the first motion block according to the first motion block, the underlying motion controller and the robot's output action at the second time node.

[0118] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0119] This application also provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0120] This application also provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0121] The following describes the hardware module of the robot of this application, which can be used to implement the aforementioned method 100 and Figures 2 to 3 The execution process is shown.

[0122] Now for reference Figure 5The diagram shows a block diagram of a robot 500 according to one embodiment of this application. The robot 500 may include one or more processors 501 coupled to a controller hub 503. In at least one embodiment, the controller hub 503 communicates with the processor 501 via a multi-branch bus such as a front side bus (FSB), a point-to-point interface such as a quick path interconnect (QPI), or a similar connection 510. The processor 501 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 503 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0123] Robot 500 may also include a first coprocessor 502 coupled to a controller hub 503 and a memory 504. Alternatively, one or both of the memory and the GMCH may be integrated within the processor, with the memory 504 and the first coprocessor 502 directly coupled to the processor 501 and the controller hub 503, which resides on a single chip with the IOH. The memory 504 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, the first coprocessor 502 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of the first coprocessor 502 are indicated by dashed lines. Figure 5 middle.

[0124] Memory 504, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 504 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0125] In one embodiment, robot 500 may further include a network interface controller (NIC) 506. NIC 506 may include a transceiver for providing a radio interface to robot 500, enabling communication with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, NIC 506 may be integrated with other components of robot 500. NIC 506 can implement the functions of the communication unit in the above embodiments.

[0126] Robot 500 may further include input / output (I / O) devices 505. I / O 505 may include: a user interface designed to allow a user to interact with robot 500; an interface designed to allow peripheral components to also interact with robot 500; and / or sensors designed to determine environmental conditions and / or position information related to robot 500.

[0127] It is worth noting that, Figure 5 This is merely an example. That is, although... Figure 5 The diagram shows that the robot 500 includes multiple devices such as a processor 501, a controller hub 503, and a memory 504. However, in practical applications, devices using the methods of this application may include only a portion of the devices in the robot 500. For example, it may include only the processor 501 and the NIC 506. Figure 5 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 504, which is a computer-readable storage medium, stores instructions that, when executed on a computer, cause the robot 500 to perform the methods according to the above embodiments. Specific details can be found in the methods of the above embodiments, and will not be repeated here.

[0128] Now for reference Figure 6 The diagram shown is a block diagram of a SoC 600 according to an embodiment of this application. Figure 6 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 6 In this SoC 600, the following are included: an interconnect unit 650 coupled to an application processor 610; a system proxy unit 680; a bus controller unit 690; an integrated memory controller unit 640; one or more second coprocessors 620, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 630; and a direct memory access (DMA) unit 660. In one embodiment, the second coprocessor 620 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0129] The static random-access memory (SRAM) cell 630 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 600 to perform the execution action control method according to the above embodiments; specific details can be found in the methods of the above embodiments, and will not be repeated here.

[0130] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0131] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0132] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0133] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0134] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0135] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0136] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for controlling the execution of actions of a robot, characterized in that, Applied to robots, the robots acquire data through an observation thread, perform decision calculations through an inference thread, and perform execution control through an execution thread, including: In response to the execution thread controlling the robot to complete or be about to complete the first output action, the current motion state information of the robot is obtained, and the first observation data is obtained through the observation thread. The first observation data includes the first body state information of the robot after completing the first output action and the first visual image data collected. Based on the robot's current motion state information, the predicted action block of the robot is determined by the trajectory predictor and the inference thread. The inference thread determines the robot's first action block based on the first observation data and the visual-language-action (VLA) model. The first action block includes the robot's first joint spatial action point sequence. The trajectory predictor determines the first time node of the predicted action block before the inference thread determines the second time node of the first action block. After the first time node and before the second time node, the execution thread controls the robot to execute the predicted output action corresponding to the predicted action block according to the predicted action block and the underlying motion controller. After the second time node, the execution thread controls the robot to execute the second output action corresponding to the first action block based on the output actions of the first action block, the underlying motion controller, and the robot at the second time node.

2. The method according to claim 1, characterized in that, The current motion state information includes the robot's joint motion state information, which includes at least one of joint position, velocity, and acceleration. Determining the robot's predicted action block based on the robot's current motion state information includes: Based on the current motion state information and dynamic constraint information, predictive observations are determined, wherein the dynamic constraint information includes DH parameters that define the geometric relationships of the robot's joints; The predicted observation is sent to the inference thread, which then determines the predicted action block.

3. The method according to claim 1 or 2, characterized in that, The control of the robot to execute the second output action corresponding to the first action block includes: The robot is controlled to transition from the output action at the second time point to the second output action by using a smooth blending function.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the first key-value cache data in the VLA model, the first key-value cache data including key vectors and value vectors; Based on the vector rotation quantization algorithm, an orthogonal rotation transformation is applied to the key vector and the value vector, and low-bit quantization is performed on the rotated key vector and the value vector to obtain the optimized second key-value cache data; Based on the random projection matrix, the second key-value cache data is projected to a low-dimensional space to obtain dimensionality-reduced key-value cache data.

5. The method according to claim 4, characterized in that, The method further includes: Based on the dimensionality-reduced key-value cache data, a sliding window is generated according to the time dimension. The window size of the sliding window is determined according to the memory usage of the dimensionality-reduced key-value cache data. Delete data in the dimensionality-reduced key-value cache that exceeds the range of the sliding window, and retain cached data in the dimensionality-reduced key-value cache corresponding to the robot's key frames, wherein the key frames are determined according to the attention weights of the VLA model.

6. The method according to claim 1 or 2, characterized in that, The step of determining the robot's first action block based on the first observation data and the visual-language-action (VLA) model includes: Based on the first visual image data, determine the visual feature vector; Obtain the current task instruction text, and determine the language feature vector based on the current task instruction text; The first discrete action token sequence is determined based on the visual feature vector, the language feature vector, and the action output head of the VLA model; Based on the VLA model, the first discrete action token sequence is inferred and decoded to determine the second discrete action token sequence; The first action block is determined based on the second discrete action token sequence and the codebook, wherein the codebook is used to perform cluster analysis on the robot's action dataset and map continuous joint space action trajectories to a finite number of discrete action tokens.

7. The method according to claim 6, characterized in that, In the codebook, the number of codewords corresponding to actions in the action dataset is related to the frequency of occurrence and / or the amplitude of movement of the action. The higher the frequency of occurrence and / or the smaller the amplitude of movement of the action, the more codewords are corresponding to it.

8. The method according to claim 6, characterized in that, The method further includes: Acquire the motion dataset and extract continuous joint trajectory data for various operation tasks from the motion dataset; The continuous joint trajectory data is divided into a fine operation motion region and a coarse operation motion region according to the motion amplitude and motion frequency. A first number of codewords is assigned to the fine operation movement area and a second number of codewords is assigned to the coarse operation movement area, wherein the first number is greater than the second number; Clustering algorithms are performed on the fine-grained operation region and the coarse-grained operation region respectively to obtain the codebook vector corresponding to the codebook.

9. The method according to claim 6, characterized in that, The VLA model includes a main VLA model and a draft model. The step of determining the second discrete action token sequence based on the VLA model and the first discrete action token sequence includes: Based on the first discrete action token sequence and the draft model, a candidate discrete action token sequence is determined; The candidate discrete action token sequence is verified by the main VLA model to accept matching tokens and regenerate non-matching tokens to obtain the second discrete action token sequence.

10. An action control device, characterized in that, include: The acquisition module is used to acquire the current motion state information of the robot in response to the execution thread controlling the robot to complete or about to complete the first output action, and to acquire the first observation data through the observation thread. The first observation data includes the first body state information of the robot after completing the first output action and the first visual image data collected. The processing module is used to determine the predicted action block of the robot based on the current motion state information of the robot through the trajectory predictor; The processing module is specifically used to determine the first action block of the robot through the inference thread based on the first observation data and the visual-language VLA model. The first action block includes the sequence of the first joint spatial action points of the robot. The trajectory predictor determines the first time node of the predicted action block before the inference thread determines the second time node of the first action block. The processing module is specifically used to control the robot to execute the predicted output action corresponding to the predicted action block through the execution thread after the first time node and before the second time node, based on the predicted action block and the underlying motion controller. The processing module is specifically used to, after the second time node, control the robot to execute the second output action corresponding to the first action block through the execution thread, based on the first action block, the underlying motion controller, and the robot's output action at the second time node.

11. A robot, characterized in that, include: At least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, causing the robot to implement the action control method as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the execution action control method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the action control method as described in any one of claims 1 to 9.

Images