Methods, systems, and computing devices for predicting actions of a robot

Abstract

A method and a computing device for predicting actions of a robot, the method comprising: obtaining a first state of the robot and a first action sequence corresponding to a first time period; determining, based on the first state and the first action sequence, position information of a changed tile of a first target frame; sending the position information to a second node; instructing the robot to perform the first action sequence in the first time period, obtaining a second target frame captured by a camera of the robot at a target time within the first time period; sending the second target frame to the second node, the second node comprising a second model; receiving, from the second node, intermediate data generated by the second model based on the second target frame, the position information, and first key data and first value data corresponding to the first target frame; and predicting, by a first model, a second action sequence of the robot in a second time period based on the intermediate data and a second state of the robot.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of robotics technology, and in particular relate to a method, system, and computing device for predicting the actions of a robot. Background Technology

[0002] Currently, with the rapid development of embodied intelligence, the Vision Language Action Model (VLA model) has been widely used in humanoid robots, mobile manipulation robots, industrial collaborative arms, and Robot as a Service (RaaS) platforms. The VLA model can directly take visual images and natural language commands as input and output robot action sequences end-to-end.

[0003] Mainstream VLA models have more than 4 bytes of parameters. If VLA models are deployed on the robot itself, there is a problem of insufficient computing power on the robot. To alleviate the computational and storage pressure on the robot, one solution attempts to introduce a cache reuse strategy on a single machine. For example, by calculating the pixel cosine similarity between adjacent frame image patches, regions where the content has not changed significantly are identified, and the key-value pair matrix of historical tokens from the previous frame (Key-Value Cache, or KV Cache) is directly reused. However, the above solution has inherent technical limitations: similarity comparison must rely on posterior calculations after the new image frame has fully arrived, and cannot be planned in advance before image acquisition or network transmission.

[0004] If the VLA model is deployed to the cloud, the robot needs to upload images to the cloud via the network. The cloud predicts the action based on the autoregressive method and transmits the predicted action to the robot. The end-to-end latency in this process is as high as 400-600ms, making it impossible to achieve real-time control of the robot.

[0005] Therefore, the industry generally adopts the Split Inference architecture as a solution that balances model accuracy and response latency. This involves deploying a computationally intensive Vision Language Model Backbone (VLM Backbone) in the cloud, and deploying a smaller but latency-sensitive Action Head on the robot body or near-edge devices. The two sides exchange intermediate inference results through the network to achieve collaborative computation.

[0006] In the Transformer-based VLA model, the self-attention mechanism continuously generates and caches key-value pair matrices of historical tokens during sequence generation. In the segmentation-inference architecture, after the cloud-based VLM Backbone completes image feature extraction and language understanding, it needs to pass the current KV Cache as core context information to the edge-side ActionHead to support the autoregressive generation of subsequent action sequences. The data size of the KV Cache is directly related to the number of model parameters and the length of the context, constituting the main load for data interaction between cloud and edge nodes.

[0007] In existing technologies, a full KV cache transmission mechanism is typically used to achieve segmentation inference for VLA models. This approach requires the cloud to serialize all generated KV cache tensors after completing the full forward inference for each frame and then distribute them completely to edge nodes over the network. However, as the size of VLA models continues to increase, the amount of KV cache data generated by a single frame of inference can reach 1GB to 4GB (in terms of conventional floating-point precision). Under actual network transmission conditions, such a large amount of data will cause severe network congestion and transmission latency. Measured data shows that the end-to-end latency under this architecture is typically as high as 400ms to 600ms. The network transmission time completely offsets the theoretical benefits of distributed computing, making it difficult to achieve real-time control of the robot through segmentation inference. Summary of the Invention

[0008] The purpose of this invention is to provide a scheme for predicting robot actions to reduce prediction delay.

[0009] This specification provides, in a first aspect, a method for predicting the actions of a robot, the robot including a camera, the method being executed by a first node, the first node including a first model, the method comprising:

[0010] The robot's first state and a first action sequence corresponding to a first time period are obtained. The first action sequence is predicted by the first model relative to a first target frame and the first state. The first target frame is captured by the camera before the first time period. The first state is the state of the robot when the first target frame is captured.

[0011] The location information of the target tile in the first target frame is determined based on the first state and the first action sequence.

[0012] The location information is sent to the second node; the robot is instructed to execute the first action sequence during the first time period to acquire the second target frame captured by the camera at the target time within the first time period.

[0013] Send the second target frame to the second node;

[0014] Intermediate data is received from the second node, the intermediate data being generated based on the second target frame and the location information;

[0015] Using the first model, based on the intermediate data and the robot's second state, a second action sequence of the robot is predicted in a second time period, where the second time period is the time period after the target time, and the second state is the state of the robot when the second target frame is captured.

[0016] A second aspect of this specification provides a method for predicting the actions of a robot, the robot including a camera, the method being executed by a second node, the second node including a second model, the second node storing first key data and first value data corresponding to a first target frame generated by the second model, the first target frame being captured by the camera before a first time period, the method comprising:

[0017] The robot receives the location information of the target tile of the first target frame from the first node. The location information is determined based on the robot's first state and a first action sequence corresponding to a first time period. The first action sequence corresponds to the first target frame, and the first state is the state of the robot when it captures the first target frame.

[0018] The second target frame is received from the first node, and the second target frame is captured by the camera at a target time within the first time period;

[0019] The second model generates intermediate data based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame;

[0020] The intermediate data is sent to the first node to predict the robot's second action sequence in a second time period, which is the period after the target time.

[0021] A third aspect of this specification provides a system for predicting the actions of a robot, comprising a first node and a second node. The robot includes a camera. The first node includes a first model, and the second node includes a second model. The second node stores first key data and first value data corresponding to a first target frame generated by the second model. The first target frame is captured by the camera before a first time period.

[0022] The first node is configured to: acquire a first state of the robot and a first action sequence corresponding to a first time period, wherein the first action sequence is predicted by the first model relative to a first target frame, the first target frame is captured by the camera before the first time period, and the first state is the state of the robot when the first target frame is captured; determine the position information of the target tile of the first target frame based on the first state and the first action sequence; send the position information to the second node; instruct the robot to execute the first action sequence during the first time period; acquire a second target frame captured by the camera at a target time within the first time period; and send the second frame to the second node.

[0023] The second node is used to: generate intermediate data based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame using the second model; and send the intermediate data to the first node;

[0024] The first node is further configured to: predict a second action sequence of the robot in a second time period based on the intermediate data and the second state of the robot using the first model, wherein the second time period is a time period after the target time, and the second state is the state of the robot when the second target frame is captured.

[0025] A fourth aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in the first or second aspect.

[0026] A fifth aspect of this specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first or second aspect.

[0027] A sixth aspect of this specification provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0028] The solution in the embodiments of this specification introduces a visual change region prediction mechanism based on forward kinematics (FK) of robots. Through the "signaling first" strategy, the edge end prioritizes sending a mask with very small changes, so that the cloud can preprocess the image data at the same time as receiving it, thereby reducing the inference latency of the cloud. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of a system architecture for predicting robot actions in an embodiment of this specification;

[0031] Figure 2 This is a schematic diagram of a system architecture for predicting robot movements according to another embodiment of this specification;

[0032] Figure 3 This is a flowchart of the method for predicting robot actions in the embodiments of this specification;

[0033] Figure 4 A schematic diagram illustrating the scenario of robot motion prediction and execution in the embodiments of this specification is shown;

[0034] Figure 5 This document illustrates a schematic diagram of the cloud-edge interaction process in one embodiment of the present specification.

[0035] Figure 6 This is a schematic diagram of the cloud-edge interaction process in another embodiment of this specification;

[0036] Figure 7 This is a VLM model architecture diagram from one embodiment of this specification;

[0037] Figure 8 This is a schematic diagram of incremental KV calculation in one embodiment of this specification;

[0038] Figure 9 This is a schematic diagram of layered differential KV transmission in one embodiment of this specification. Detailed Implementation

[0039] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0040] Figure 1This is a schematic diagram of a system architecture for predicting robot actions, as illustrated in an embodiment of this specification. The system is primarily applied to cloud-edge collaborative robot control scenarios, aiming to address the latency issue in current segmentation and inference schemes. The system may include robot end nodes, cloud nodes, and edge end nodes.

[0041] The robot end node includes the robot body equipped with sensors and motion control modules, as well as its embedded control unit. For example, the robot body may include humanoid robots, mobile manipulation robots (such as AGVs with robotic arms), industrial collaborative robotic arms, or quadruped robots. The embedded control unit may include, for example, a main control computer or an embedded development board. The sensors may include, for example, cameras, depth cameras, LiDAR, and inertial measurement units (IMUs), and the motion control modules may include, for example, joint encoders and servo motor drivers.

[0042] Edge nodes are typically deployed at the network edge close to the robot body. In specific implementations, edge nodes can be mobile edge computing (MEC) servers deployed at 5G base stations, or high-performance industrial control computers or local private cloud servers within a local area network. For example, in warehousing and logistics scenarios, edge nodes can be edge computing gateways or rack-mounted servers located locally in the warehouse; in factory scenarios, edge nodes can be edge computing boxes connecting multiple production lines. Edge nodes maintain low-latency communication with robot end nodes via wired (e.g., Ethernet, fiber optic) or wireless (e.g., 5G, Wi-Fi 6) methods. Their hardware configuration is usually higher than that of the robot body, possessing sufficient memory and computing resources to run lightweight action head models and handle high-frequency kinematic calculation tasks.

[0043] Cloud nodes are deployed in remote data centers or public cloud platforms, primarily responsible for running the computationally intensive Visual Language Learning (VLM) backbone network, which has a massive number of parameters. Specifically, cloud nodes can include servers equipped with high-performance graphics processing units (GPUs). Cloud nodes typically rely on the infrastructure of large cloud service providers, leveraging their powerful parallel computing capabilities and massive storage resources to process image data concurrently uploaded from multiple robots, perform complex inference calculations, and manage a global key-value (KV) cache pool. Cloud nodes connect to edge nodes via wide area networks or dedicated high-speed links, ensuring sufficient bandwidth and stability when transmitting incremental KV cache data. Through this layered deployment structure, the system can fully utilize the unlimited computing power of the cloud and the low-latency response advantages of the edge to achieve highly efficient intelligent robot control.

[0044] exist Figure 1In this scenario, sensors within the robot body are responsible for collecting environmental image frames and the robot's state data (such as joint angles and end-effector poses) at the current moment, and sending this data to the edge nodes. The motion control module of the robot's edge nodes receives motion sequence instructions from the edge nodes and drives the robot body to complete the corresponding physical operations. Under this architecture, the robot's edge nodes do not perform complex model inference; they only act as data sources and execution terminals, thereby minimizing the computational power required from the robot body.

[0045] The edge node internally runs a kinematics prediction module and an Action Head model. First, based on the predicted action sequence from the robot, the edge node uses a forward kinematics algorithm to predict the camera's pose change in the next moment, thus determining the location information of significantly changed tiles in the most recently uploaded image frame. Subsequently, the edge node uploads this location information along with the corresponding next frame image to the cloud node and receives intermediate data returned by the cloud node. Next, the edge node uses the Action Head model to combine the ontology state data and intermediate data to predict the action sequence for the next time period and sends it to the robot node for execution. By having the edge node predict the location information of the target tiles (also known as changing tiles) in advance based on the action sequence, the cloud can immediately perform inference based on this location information after receiving the next frame image, accelerating the cloud's processing speed. Furthermore, the cloud can only calculate the key and value data corresponding to the changing tiles, reducing the computational load and further accelerating the cloud's processing speed.

[0046] The cloud node receives image frames and location information of changing tiles from the edge node, and uses VLMBackbone to generate intermediate data based on the image frames and location information. This intermediate data includes, for example, key and value data (hereinafter referred to as △KV) of the changing tiles, latent vectors, etc. Compared to full transmission, the cloud node only sends back a small amount of intermediate data to the edge node, greatly reducing latency. In addition, through this division of labor, the cloud node focuses on high-computing-power visual semantic understanding, while offloading low-latency action decision-making and local state fusion tasks to the edge, achieving efficient utilization of computing resources and optimization of system response speed.

[0047] Figure 2 This is a schematic diagram of a system architecture for predicting robot movements according to another embodiment of this specification. Figure 1 In the illustrated embodiment, the system adopts a three-layer "cloud-edge-device" architecture. Edge nodes, deployed as independent computing entities (such as MEC servers or local industrial control computers) at the network edge, are responsible for carrying the Action Head model and kinematic prediction module. Robot end nodes are only responsible for data acquisition and action execution. Figure 2 In the illustrated embodiment, the system is simplified to a two-layer architecture. Figure 1 The functions of the edge nodes in the robot are integrated into the robot's end nodes, meaning that the main control computer inside the robot body also undertakes the task of calculating the predicted actions.

[0048] Specifically, in Figure 2 In this scenario, the robot end node not only includes sensors and motion control modules, but also incorporates embedded computing devices with high computing power (such as high-performance embedded GPU modules). These devices directly run the kinematics prediction module and the Action Head model. Internally, the robot end node is divided into a perception and control subsystem and an inference and computation subsystem. The perception and control subsystem is responsible for acquiring image frames and body state data and sending them to the inference and computation subsystem. The inference and computation subsystem uses local computing power to perform forward kinematics calculations to generate the position information of changing tiles and uploads the image frames along with this position information to the cloud node. The structure of the cloud node is similar to... Figure 1 Similarly, VLMBackbone is still deployed, which is responsible for generating intermediate data based on the received data and sending the intermediate data to the robot end node. The robot end node can then predict the robot's actions based on the intermediate data and the robot's body state data.

[0049] In one implementation, the VLM can be deployed in the cloud, with the robot end node directly connected to the cloud node via a wide area network or 5G network. In another implementation, the VLM can be deployed at the edge, with the edge node maintaining low-latency communication with the robot end node via wired (e.g., Ethernet, fiber optic) or wireless (e.g., 5G, Wi-Fi 6) methods.

[0050] Figure 3 This is a flowchart of a method for predicting robot actions in an embodiment of this specification. The method may consist of a first node and a second node. Wherein, in Figure 1 In the scenario shown, the first node can be an edge node, and the second node can be a cloud node. Figure 2 In the scenario shown, the first node can be a robot-side node, and the second node can be a cloud-side node or an edge-side node. The following text will use... Figure 1 The scenario shown is used as an example for description. It can be understood that this solution can also be used in the same way. Figure 2 In the scenario shown.

[0051] like Figure 3 As shown, in step S301, the edge device acquires the robot's first state and a first action sequence corresponding to the first time period. The first action sequence is predicted by the edge device's action head relative to a first target frame, which is captured by the robot's camera before the first time period. The first state is the robot's state at the time the first target frame was captured.

[0052] Figure 4 This diagram illustrates a scenario of robot motion prediction and execution in an embodiment of this specification. In this scenario, the user can issue a command to the robot, such as "pick up the cylinder on the table," via voice or input. For voice commands, the robot can convert the voice into natural language commands using a speech recognition model. The robot (e.g., a robotic arm) responds to the user's natural language command, "pick up the cylinder on the table," and initiates a visual perception and motion planning process. The robot's camera begins continuously capturing environmental video streams, forming a series of image frames arranged chronologically. The target frames are then sent to an edge device for motion prediction. It is understood that in this embodiment, the robot is not limited to responding to natural language commands; it can also automatically initiate operations based on preset triggering modes. For example, after initiation, the robot can capture environmental video and send multiple frames from the video to an edge device or the cloud. The VLM deployed at the edge device or cloud then determines whether to initiate an operation. In a home assistance scenario, such operations include, for example, tidying clothes or picking up scattered items.

[0053] At the initial time t1, the system acquires several frames of images I(t1) and the current robot state q(t1). Through collaborative reasoning between the cloud and the edge, it generates and executes an initial action sequence. These frames of images I(t1) can be one or more frames captured by a camera within a preset time range including time t1.

[0054] The robot state can include the following: joint angles, i.e., the current displacement of each rotational or translational joint of the robotic arm; joint velocities, i.e., the rate of motion of each joint at the current moment; end effector pose, including the position coordinates (x, y, z) of the tool center point (TCP) at the end effector in three-dimensional space and attitude information (such as quaternions or Euler angles); and gripper / tool ​​state, such as the opening and closing width of the gripper, the magnitude of the gripping force, or the on / off state of the suction cup. If the robot has a mobile base, the robot state may also include the wheel speed of the base, IMU acceleration, and odometry pose. This state data is collected in real time by the robot's sensors and serves as one of the key inputs to the motion head, ensuring that the predicted motion conforms to the robot's current physical constraints.

[0055] At the initial time t1, the system is in a cold start state, and the local cache pool of the edge nodes does not yet contain any historical context information. Due to the lack of reference information from the previous frame, the system triggers a full initialization process. The edge nodes upload the first frame image I(t1) and related language instructions (or only the first frame image I(t1)) to the cloud node. The visual language backbone network (VLM Backbone) of the cloud node performs a complete full forward inference on image I(t1), including feature extraction from the image encoder and attention calculations for each layer of the Transformer, generating complete intermediate data corresponding to all image patches and language tokens.

[0056] It should be noted that the "intermediate data" here refers to all the high-dimensional semantic representations output by the VLM Backbone, used to support the generation of subsequent actions. In one specific implementation, the aforementioned intermediate data can be a complete key-value pair cache (KV Cache); in another implementation, if the action head at the edge is a diffusion model, the intermediate data may also include latent vectors or conditional embeddings used in the diffusion denoising process. Subsequently, the cloud sends this full set of intermediate data to the edge nodes.

[0057] The edge node receives the full amount of intermediate data. If the intermediate data is a KV cache, the edge node stores it in its local memory, completing the context initialization. Based on this, the edge node's action head generates an action sequence for the initial time period T1 based on the intermediate data and the robot's state q(t1). Figure 4 Show action sequence , , , As an example, this motion sequence consists of multiple consecutive motion steps, driving the robotic arm's end effector to approach the target cylinder along a preset trajectory τ1. Trajectory τ1 reflects the smooth motion planning of the robotic arm during the approach to the object, ensuring the stability and safety of the motion.

[0058] The action head in the edge nodes can include a lightweight policy network (such as a diffusion policy head, a Transformer decoder head, or a PerceiverIO architecture). The action head obtains key-value inputs or conditional contexts for the attention mechanism based on intermediate data, and performs action inference using the robot's own state as a query or additional condition.

[0059] The above action sequence may include the following types of actions:

[0060] The first category is joint space motion, which can specifically include the target angle of each joint, as well as the target angular velocity or linear velocity of each joint.

[0061] The second category is task space actions, also known as Cartesian space actions. These specifically include the three-dimensional position increments or absolute coordinates of the end effector, used to control the translational motion of the robotic arm's end effector; and the attitude changes of the end effector, typically represented as increments of quaternions, Euler angles, or rotation matrices, used to control the orientation adjustment of the end effector. Additionally, it includes the linear and angular velocities of the end effector, used to achieve velocity-based closed-loop control.

[0062] The third category is end effector operation, which is used to control the state of tools or grippers installed at the end of a robotic arm. This state can include parameters such as the opening and closing degree of the grippers and the clamping force.

[0063] The fourth category is base motion, applicable to composite robots with mobility capabilities (such as mobile manipulation robots). These commands include the linear and angular velocities of the base, or displacement increments relative to the current position.

[0064] The motion sequence output by the motion head is a multi-dimensional vector sequence, with each time step's vector incorporating the various instructions mentioned above. Through this fine-grained, multi-degree-of-freedom motion description, the system can precisely control the robot to complete a full closed-loop operation from perception and planning to execution, ensuring high-precision and high-stability task execution in dynamic environments.

[0065] Subsequently, at time t2, the system can again acquire several frames of image I(t2) and robot state q(t2) corresponding to the next time period T2, and through collaborative reasoning between the cloud and the edge, generate and execute the action sequence for the next time period. Figure 4 The middle shows , , , As an example, this sequence of actions drives the robotic arm's end effector to approach the target cylinder along a preset trajectory τ2, and so on until the robot completes the task of "picking up the cylinder on the table".

[0066] In one implementation, the first target frame can be the initial frame after system startup, such as frame I(t1) mentioned above, which is the first frame processed by the Visual Language Backbone (VLM Backbone). During this initialization phase, due to the lack of historical context information, the system performs a full inference process: the cloud node performs complete visual encoding and Transformer layer forward propagation on the first target frame, generating a complete key-value pair cache (KV Cache) containing all image patches and language token information, and transmits it in full to the mobile edge node. The edge node receives and stores this complete KV Cache to complete the initialization of its local cache pool. Subsequently, based on this complete context and the robot's body state, it generates a first action sequence corresponding to the first time period (here, for example, time period T1) through the Action Head. This implementation will be described below as an example.

[0067] In another implementation, the first target frame can also be any non-first frame during normal system operation, such as I(t2) mentioned above. At this time, the system is in the incremental inference loop phase, and the edge nodes have already stored the KV Cache corresponding to the previous target frame. In this case, the edge nodes can receive the incremental KV Cache corresponding to the first target frame from the cloud based on the differential update mechanism described below; the edge nodes merge this incremental KV Cache with the locally stored historical KV Cache to reconstruct the complete context representation corresponding to the first target frame, and finally generate the first action sequence corresponding to the first time period (here, for example, time period T2) based on the updated KV Cache and the robot's state using the Action Head model.

[0068] Back Figure 3 In step S303, the position information of the changed patch (or target patch) of the first target frame is determined based on the first state and the first action sequence.

[0069] Figure 5 This diagram illustrates a cloud-edge interaction process in one embodiment of this specification. The edge component includes a position prediction module and an action head. As described above, the position prediction module receives real-time state data of the robot at time t1 (i.e., the first state q(t1)) collected by the robot's body sensors, and a first action sequence predicted by the action head within a first time period T1. , , , By performing forward kinematics calculations and geometric projection, the system outputs position information indicating the patch (hereinafter referred to as the changed patch) in the first target frame (image I(t1)) that will undergo significant changes.

[0070] In one implementation, the system calculates the pose change of the camera based on a first action sequence; determines the content change of each tile in the first target frame based on the pose change; and determines the position information of the changed tiles according to the ratio of the content change of each tile. Specifically, the position prediction module first reads the state q(t1) at the current time t1, and simultaneously reads the first action sequence output by the motion head for the robot to execute in the time period T1 after time t1. , , , Next, the module uses the Forward Kinematics (FK) algorithm, combining the current state q(t1) with the state change Δq(a(t1)) derived from the first action sequence, to predict the state q(t3) at the end of time interval T1, i.e., q(t3) = q(t1) + Δq(a(t1)). Subsequently, the camera pose pose_cam(t1) = FK(q(t1)) at time t1 and the camera pose pose_cam(t3) = FK(q(t3)) at time t3 are calculated respectively, thus obtaining the camera pose change Δpose between adjacent frames. Since robot vision models typically adopt the Vision Transformer (ViT) architecture, the input image is divided into N fixed-size patches, for example, a 224×224 pixel image is divided into 14×14 patches, totaling 196 16×16 pixel patches, with each patch corresponding to a token vector.

[0071] The position prediction module projects the pose change Δpose in 3D space onto the 2D image plane, generating pixel-level optical flow fields or displacement maps. Specifically, for the current first target frame... Each pixel in Assuming its corresponding depth value is (This can be obtained directly from an RGB-D camera), and the pixel is back-projected onto a 3D point in the camera coordinate system. : , where K refers to the camera's intrinsic parameter matrix, which describes the camera's internal geometric and optical properties.

[0072] Using the pose change Δpose, the 3D point is transformed to the camera coordinate system at the next moment:

[0073]

[0074] in, for The rotation matrix is ​​included. In one implementation, the next time step in the above formula can also be approximately set to t3, i.e. .

[0075] Then Projecting back onto the image plane at the next time step, we obtain the predicted pixel coordinates.

[0076] :

[0077] Therefore, the displacement vector of each pixel can be calculated. .

[0078] The location prediction module for the first For each image patch, calculate the mean or maximum magnitude of the displacement vectors of all pixels within it, denoted as . Set a threshold. ,like If a tile is identified as a "changing tile," it is marked as 1 (True) in the change mask; otherwise, it is marked as 0 (False). Finally, the location prediction module generates a Boolean tensor, the change mask, whose length is equal to the total number of tiles N. True indicates that the tile is a changing tile and the key-value pair cache needs to be recalculated, while False indicates that the tile remains stable and the cache from the previous frame can be reused. This process is entirely based on kinematic calculations and does not require waiting for the actual acquisition of a new image frame I(t3), achieving zero-latency prediction.

[0079] In another implementation, the location information of changed patches in the first target frame can be determined based on the first state and the first action sequence using a pre-trained visual change prediction model. Specifically, a deep neural network model can be built and trained offline. The input dimensions of this model include the image, the robot's ontology state vector corresponding to the image, and the predicted action sequence vector. The output dimension corresponds to the number of patches segmented from the input image, and the model output indicates whether each patch is a changed patch. By deploying this model on edge nodes, it is possible to quickly infer which regions in the image will undergo significant visual content changes based solely on the current physical state and action sequence before acquiring new image frames.

[0080] By generating location information for changed tiles, the system can guide cloud nodes to recalculate only the key and value data of the changed tiles, reducing the computational load on cloud nodes. Furthermore, by transmitting the key and value data of the changed tiles as intermediate data in the cloud, it significantly reduces network bandwidth usage and transmission latency.

[0081] In step S305, the edge node sends the location information of the determined change patch to the cloud node.

[0082] The edge nodes send the location information of the determined changed tiles to the cloud nodes. This process aims to transmit the prior index of visual changes to the cloud with extremely low communication overhead, thereby guiding the cloud to perform targeted KV cache recalculation. The edge nodes can first perform efficient encoding and compression on the generated change mask. Since the change mask is essentially a binary vector of length N (where N is the total number of tiles the image is divided into, for example, 14×14=196), directly transmitting the original Boolean array, although the data volume is small, still has room for optimization in high-frequency inference scenarios. Therefore, the edge nodes can use sparse index encoding or run-length encoding (RLE) strategies to process the change mask. If the proportion of changed tiles is low (e.g., less than 20%), the system extracts the index IDs of all tiles marked as "changed" to form a sparse index list; if the distribution of changed tiles has spatial continuity, run-length encoding is used to record the length of continuously stable or continuously changing intervals. The encoded location information data is typically only tens to hundreds of bytes in size, and its transmission delay is negligible compared to a complete image frame or KV buffer data.

[0083] Subsequently, the edge node encapsulates the encoded location information and sends it to the cloud node via a low-latency communication protocol. In this embodiment, considering the stringent real-time requirements of robot control, a data channel based on User Datagram Protocol (UDP) or Quick UDP Internet Connections (QUIC) can be established between the edge node and the cloud node.

[0084] Edge nodes prioritize sending location information over image data. This "signaling-first" transmission strategy allows cloud nodes to know the processing strategy for the next frame before receiving massive amounts of image frame data. After receiving the location information data packet, the cloud node can immediately unpack and decode it, reconstruct the set of location indices for the changed tiles, and temporarily store it in the preprocessing queue of the KV cache manager, awaiting the arrival of subsequent image data.

[0085] This achieves decoupling and parallelization of control flow and data flow in cloud-edge collaborative inference. In traditional solutions, the cloud must wait for the complete image to be uploaded and encoded before it can determine which areas need to be recalculated, leading to accumulated latency in serial processing. In this implementation, the advance transmission of location information allows cloud nodes to utilize the time window of image data transmission to pre-allocate computing resources, lock memory addresses, or prepare corresponding attention masks. When the cloud node subsequently receives the second target frame image uploaded by the edge node, it can directly update the KV values ​​of only the tokens corresponding to the changed patches based on the ready location information. Furthermore, the independent transmission of location information enhances the robustness of the system; even if the image data experiences slight jitter or out-of-order processing due to network fluctuations, the cloud can still maintain the consistency of the context state based on the accurately arriving location signaling.

[0086] In step S307, the edge node instructs the robot to execute the first action sequence and acquire the second target frame.

[0087] Edge nodes can send the first action sequence to the robot's control unit, driving the robot to move along a preset trajectory. During this process, the system can employ a pipelined parallel processing mechanism, allowing image acquisition and action execution to overlap in time, maximizing system throughput and reducing end-to-end latency. The robot can send all captured video to the edge node, which then determines the target frame; alternatively, the robot can determine the target frame and send only that frame to the edge node. Depending on the trade-offs between real-time performance and prediction accuracy, the second target frame can be selected as a frame captured at an intermediate moment during action execution (e.g., time t2) or a frame captured at the end of action execution (e.g., time t3). These two implementation methods will be described in detail below.

[0088] In the first embodiment, the second target frame is a plurality of image frames captured at time t2 during the execution of the first action sequence. (See reference...) Figure 4 The scenario shown assumes that the first action sequence contains multiple consecutive action steps, for example, from action... To action To eliminate the waiting pauses in the traditional "perception-planning-execution" serial process, the system does not wait for all actions to complete after the robot begins executing the first action sequence. Instead, it acquires a frame captured at a preset time point t2 as the second target frame. Since time t2 is earlier than the theoretical end time of the first action sequence, this "early acquisition" strategy can significantly shorten the idle waiting time of the next inference cycle, allowing cloud nodes to receive image data and begin inference earlier, thus ensuring the continuity of action generation.

[0089] However, since the robot has not yet fully reached the expected pose when the image is acquired, there is a deviation between the actual acquired second target frame and the ideal pose predicted based on the complete action sequence, i.e., there is a certain approximation error. To correct the above approximation error and ensure the accuracy of motion control, the system can use interpolation or state fusion strategies when determining the effective action actually performed by the robot at time t2. Specifically, the edge device (or robot end) can read the robot body state data q(t2) at time t2 and combine it with the action corresponding to that time in the first action sequence (e.g., action...). ) and the action corresponding to that moment in the second action sequence (e.g., action) The exact equivalent action at time t2 is calculated using linear interpolation or spline interpolation algorithms.

[0090] In the second implementation, the second target frame is an image frame captured at time t3 after the first action sequence has been fully executed. In this mode, the edge node first monitors the execution status of the robot's underlying controller. When it detects that all action steps in the first action sequence have been completed and the robot body has reached a stationary or stable state, it immediately triggers the camera to capture the second target frame. At this time, the robot's actual pose is highly consistent with the final pose predicted based on the first action sequence. Therefore, the geometric correspondence between the second target frame and the prediction model is the most accurate, with almost no pose deviation caused by incomplete motion. The advantage of this implementation is the high degree of matching between visual content and kinematic prediction, which helps improve the accuracy of cloud-based key-value (KV) cache reuse and reduces KV drift caused by pose mismatch.

[0091] However, the second implementation inevitably introduces execution waiting latency. During the entire time the robot performs its action, the visual perception link remains idle, only initiating the acquisition and transmission of the next frame after the action ends. To mitigate this latency, the system typically employs a high-precision timer synchronization mechanism to ensure image capture is completed within a millisecond window after the action ends. Furthermore, this approach is suitable for operational scenarios requiring extremely high positioning accuracy but with relatively relaxed real-time requirements, such as precision assembly or static object grasping. In practical applications, the system can dynamically switch between these two acquisition strategies based on the task type: for dynamic tracking or fast-moving tasks, an early acquisition mode at time t2 is prioritized to achieve low latency; for delicate manipulation tasks, a delayed acquisition mode at time t3 is used to achieve high accuracy. Through this flexible time alignment mechanism, this solution can achieve the optimal balance between performance and accuracy in different robot application scenarios.

[0092] In step S309, the edge node sends the second target frame to the cloud node.

[0093] After acquiring the second target frame, the edge node can first preprocess the image frame, including format conversion, resolution adjustment, and lossy compression based on JPEG or H.264 / H.265 standards, to reduce network transmission load. Subsequently, the edge node encapsulates the compressed second target frame data into the data payload of the transmission protocol and sends it to the cloud node through an established low-latency communication link (such as QUIC or a dedicated TCP channel).

[0094] Figure 6 This is a schematic diagram of the cloud-edge interaction process in another embodiment of this specification. For example... Figure 6 As shown, during this cloud-edge interaction process, the cloud can send the key-value data (hereinafter referred to as incremental KV) of the changing map tiles to the edge as intermediate data for predicting actions.

[0095] exist Figure 6 In the scenario shown, in one implementation, the edge terminal may include a refresh module. The refresh module may include the following units: a frame counter unit, which maintains an approximate frame counter to record the number of frames continuously transmitted using incremental KV differential transmission. When incremental KV transmission is determined for the current frame, the counter value is incremented by 1; when full KV transmission is determined for the current frame, the counter value is set to 0. A scene change monitoring unit, which receives change masks from the position prediction module in real time, calculates the proportion of tiles marked as changed tiles, and optionally calculates the image similarity between the current frame (second target frame) and the previous frame (first target frame). A motion error feedback unit, which monitors the execution status of the robot's underlying controller and collects indicators such as motion execution failure rate or trajectory deviation.

[0096] The refresh module can determine whether to instruct the cloud to transmit the full key-value pairs based on at least one of the following: counter value, image change ratio, and error feedback signal. When the counter is less than the threshold K, the change ratio is lower than the set upper limit, and the action error is within the allowable range, the refresh module outputs an "incremental update" command. The edge device sends this "incremental update" command along with the second target frame to the cloud to instruct the cloud to calculate and transmit the incremental key-value pairs. Conversely, if any condition triggers the refresh mechanism (such as the counter count being greater than or equal to the threshold, a drastic scene change, or an error exceeding the limit), the refresh module outputs a "full update" command. The edge device sends this "full update" command along with the second target frame to the cloud to instruct the cloud to calculate and transmit the full key-value pairs and reset the counter to zero. The edge device can, for example, indicate whether to perform an incremental update or a full update through a binary flag (or binary field) in the information sent to the cloud.

[0097] In one implementation, the edge nodes employ a periodic refresh mechanism based on a frame counter to determine the cache update strategy. The refresh module internally maintains an approximate frame counter to record the number of consecutive frames using incremental KV differential transmission. When the refresh module determines to perform incremental KV transmission on the current frame, it increments the counter by 1; when it determines to perform full KV transmission on the current frame, it sets the counter to 0. The refresh module can set a dynamically adjustable refresh threshold K, the size of which depends on the current robot motion state, the dynamic nature of the scene, and the task accuracy requirements. For example, in static operation scenarios (such as grasping a fixed object on a desktop), due to slow camera pose changes and a stable background, the KV cache reuse rate is high, and the cumulative error grows slowly; therefore, the threshold K can be set to a larger value (e.g., 15-20 frames). However, in dynamic operation scenarios (such as a robotic arm grasping on a moving chassis) or violent motion scenarios (such as bipedal walking), due to rapid changes in perspective and potential motion blur, KV drift accumulates rapidly, and the threshold K needs to be set to a smaller value (e.g., 3-8 frames). When the counter value is less than the threshold K, the edge node sends an "incremental update" instruction along with the second target frame. The cloud node then calculates only the incremental KV data corresponding to the changed tiles. When the counter value is greater than or equal to the threshold K, the edge node sends a "full refresh" instruction and resets the counter to zero. The cloud node then performs a complete Transformer forward propagation on the second target frame, generates and returns the full KV cache to eliminate the accumulated errors caused by long-term approximate reuse.

[0098] In another implementation, the edge node employs an adaptive refresh mechanism based on scene content changes. In this approach, before sending the second target frame, the edge node monitors in real-time the proportion of changed tiles in the change mask generated in step S303. If the proportion of changed tiles is lower than a preset dynamic threshold (e.g., 70%-80%), it indicates that most areas of the scene remain stable and suitable for continued incremental transmission; in this case, the edge node sends an "incremental update" instruction. However, if the proportion of changed tiles exceeds this threshold, or if calculating the global similarity between the second and first target frames (e.g., SSIM or cosine similarity) reveals a similarity below a safe lower limit, a dramatic scene change is determined, such as a robot rapidly turning, sudden removal of obstructions, or drastic changes in lighting. In this situation, the original KV cache context is severely invalidated, and continued reuse would lead to serious inference errors. Therefore, the edge node immediately triggers a forced refresh mechanism, sending the second target frame along with a "full refresh" instruction and requiring the cloud node to ignore historical caches and recalculate the KV data for all tiles. In addition, this mechanism can also be combined with action execution feedback. If an abnormally high failure rate of robot action execution is detected or an unexpected oscillation occurs in the robot's state, it can also be used as an auxiliary criterion to trigger a full refresh to ensure the robustness of the system.

[0099] In step S311, the cloud generates intermediate data based on the second target frame, location information, and key-value data of the first target frame using the second model.

[0100] Upon receiving the second target frame and its corresponding update instruction, the cloud node executes different processing logic based on the instruction type. If the received instruction is "incremental update," the cloud node utilizes the previously received changed tile location information to perform visual encoding and attention calculation only on the tiles marked as changed in the second target frame, generating incremental key-value data (ΔKV) and constructing a complete contextual representation by combining it with historical cache. If the received instruction is "full refresh," the cloud node performs complete forward inference on all tiles in the second target frame, generating entirely new full key-value data. This collaborative mechanism, with intelligent decision-making at the edge and on-demand execution by the cloud, ensures efficient low-bandwidth transmission under normal motion conditions while promptly restoring contextual accuracy at critical points of scene abrupt changes or error accumulation, effectively balancing the contradiction between transmission efficiency and inference accuracy.

[0101] like Figure 7 As shown, the second model (VLM Backbone) typically includes an image encoder (such as ViT), a speech encoder, and a multi-layer Transformer structure. Figure 7(The image is shown as the attention module). When processing input data, the cloud node first preprocesses the second target frame, dividing it into multiple fixed-size patches, and then converts each patch into a corresponding image token sequence using an image encoder. Simultaneously, if natural language instructions exist, they are converted into text token sequences using a language encoder. In one implementation, the model's input only includes the image token sequence corresponding to the second target frame, where the model primarily focuses on changes in the image context. In another implementation, the model's input also includes the text token sequence corresponding to task-related natural language instructions. The image tokens and text tokens are concatenated and used together as input to the Transformer layer to achieve joint understanding of multimodal information. Regardless of the approach, the cloud node utilizes the previously received and decoded positional information to identify "target patches" (i.e., change patches) in the second target frame that have undergone significant changes relative to the first target frame, thereby guiding subsequent differential computation.

[0102] The specific form of the generated "intermediate data" depends on the system architecture and the requirements of the edge action head. In one implementation, the intermediate data is the incremental or full portion of a key-value cache (KV Cache). The incremental portion consists of the second key data and the second value data corresponding to the change tile, used to update the local cache at the edge to support autoregressive inference. In another implementation, if the edge action generation model adopts a diffusion policy, the intermediate data may also include latent vectors output by the VLM Backbone or conditional embeddings used in the diffusion denoising process. This embodiment mainly uses KV Cache as an example for illustration.

[0103] When the intermediate data is the incremental KV (ΔKV), such as Figure 8 As shown, the cloud node first divides the second target frame into multiple tiles, and then converts them into corresponding embedding vector sequences through an image encoder (e.g., ...). (etc.). Subsequently, these embedding vectors are sequentially input into the multi-layer Transformer structure of the second model (VLM Backbone) for processing.

[0104] With the first layer (Layer1, For example, Including a learnable query weight matrix Key weight matrix Sum weight matrix For each input embedding vector First, the corresponding query vector is calculated through linear projection. .

[0105] Based on this, the cloud node uses the location information (change_mask) received from the edge to distinguish between changed and unchanged patches in the second target frame relative to the first target frame, and executes a differentiated key-value generation strategy:

[0106] For unchanged tiles: cloud nodes do not recalculate their values. Vector sum Instead of using vectors, it directly reuses the key vector of the corresponding position tile in the stored first target frame. ) and value vector ( ), where j represents the number of layers in the attention module.

[0107] For changing tiles: cloud nodes are based on the embedding vector at the current time step. Recalculate its new vector( )and vector( ).

[0108] Next, the L1 layer, based on the corresponding embedding vectors... Vectors, and a mixture of reused historical key-value pairs and newly calculated key-value pairs. The set of attention vectors is used to calculate the output attention vector of this layer using the following attention formula. .

[0109]

[0110] Subsequently, this output attention vector is used as the next layer (Layer 2, The above process is repeated for the input embedding vector. In all subsequent Transformer layers, the same principle applies: based on position information, the key-value data cached from the previous frame is reused for unchanged tiles, and the changed tiles are recalculated. Vector sum vector.

[0111] After layer-by-layer processing by multiple Transformers, the cloud node finally extracts the data that was recalculated only for the changed tiles across all levels. Vector sum Vectors. These newly generated vectors. This constitutes the incremental portion relative to the first target frame, namely the incremental KV ( () as intermediate data.

[0112] When the edge node indicates a full key-value transmission (e.g., triggering a refresh mechanism, a dramatic scene change, or a counter reaching a threshold), the cloud node will ignore the reuse logic of the historical cache and perform a complete full forward inference on the second target frame. At this point, the cloud node performs a complete attention calculation on all tiles (and optional natural language instruction tokens) of the second target frame through all Transformer layers of VLMBackbone, generating key and value data (full key-value cache) containing all tiles and tokens as intermediate data. After receiving this full data, the edge node will replace its local old key-value cache, thereby eliminating key-value drift errors that may accumulate due to long-term incremental updates and ensuring the accuracy of subsequent inference.

[0113] like Figure 8 As shown, the cloud node significantly reduces the computational load of the second model through a location-based differentiated computation strategy. In the traditional full-scale inference process, the cloud node needs to perform a complete Transformer forward propagation on the embedding vectors corresponding to all tiles in the second target frame, including linear projection calculations of the query, key, and value vectors at each layer, as well as attention mechanism operations. The computational complexity is proportional to the number of tiles and the number of model layers. In this embodiment, the cloud node uses the location information received from the edge to accurately identify the "target tiles" that have changed relative to the first target frame and the "non-target tiles" that remain stable in the second target frame. For non-target tiles, the cloud node directly reuses the stored historical key and value data corresponding to the first target frame, completely skipping the linear projection calculations of these tiles in the current frame inference process (i.e., no calculation is required). and The cloud node only performs recalculation for a small number of embedding vectors that are labeled as target tiles. Since the proportion of tiles that undergo significant visual changes between adjacent frames is typically only 15%-35% in typical robot motion scenarios, this means that the cloud node can omit 65%-85% of the key-value pair generation computation in each Transformer layer.

[0114] refer to Figure 9 To further balance computational accuracy and transmission efficiency, cloud nodes can also employ a layered differential transmission implementation. Given that different Transformer layers have different characteristics in processing visual information, cloud nodes divide the VLMBackbone's Transformer layers into shallow layers (e.g., Figure 9 In (layer) and deep (e.g.) Figure 9 In The system employs different key-value (KV) generation and transmission strategies for shallow layers (e.g., layers 1-24). For shallow layers, which primarily extract low-level image features and exhibit minimal inter-frame variation, cloud nodes can fully reuse the KV from the previous frame if the proportion of changed patches is below a certain threshold. If the proportion of changed patches exceeds this threshold, cloud nodes can recalculate the KV for the changed patches and transmit the ΔKV to the edge. For deep layers (e.g., layers 25-48), which focus on high-level semantics and are sensitive to change, cloud nodes accurately recalculate the ΔKV for changed patches and transmit the full-precision ΔKV, while reusing the KV for unchanged patches. This layered, differentiated processing reduces the effective data transmission volume by approximately 40%-50% while controlling accumulated errors.

[0115] Specifically, for shallow attention modules, when the proportion of changed patches to all patches in the second target frame is lower than a preset proportion, it can be based on... Attention is calculated for each tile, and the latent vector of each tile is output. ,in, This is obtained by multiplying the input vector corresponding to each tile with the Q matrix. The latent vector output by the previous attention module is used as the input vector for the next attention module. For example, refer to... Figure 9 shallow The latent vectors output by the layer are used as the deep layer's latent vectors. The input vector of the layer. For deep attention modules, the latent vector is the input vector corresponding to each tile. calculate For unchanged tiles, cloud nodes do not recalculate their values. Vector sum Instead of using vectors, it directly reuses the key vector of the corresponding position tile in the stored first target frame. ) and value vector ( For changing tiles, the cloud nodes use the latent vectors of the input attention module corresponding to that tile. Recalculate its new vector( )and vector( This allows for the visualization of deep change patches. Vector sum The vector (i.e., ΔKV) is generated as intermediate data and sent to the edge.

[0116] In another implementation, the cloud node includes a compression module, which may employ a Query-Former (Q-Former) architecture to further reduce the data size of the Key-Value Cache (KV Cache) through feature compression. Specifically, after the cloud node receives the second target frame and its corresponding location information, it first divides the second target frame into multiple original tiles and extracts their feature vectors using an image encoder. Subsequently, the compression module introduces a preset number of learnable query embeddings and uses a cross-attention mechanism to interact these query vectors with the feature vectors of the original tiles. During this process, the compression module aggregates key semantic information from the massive amount of original tile features, compressing the image feature sequence containing a large number of tiles (e.g., 196 or more) into a significantly reduced number of compressed tiles (i.e., bottleneck tokens, such as 32 or 64). This compression process not only preserves the core semantic content of the image but also significantly reduces the data dimensionality of subsequent processing.

[0117] Because the compression module aggregates multiple original tiles into a small number of compressed tiles, cloud nodes can identify the changed tiles in the compressed tiles based on the many-to-one mapping relationship between the original tiles and the compressed tiles, and the positional information of the changed tiles in the original tiles. Specifically, in the Q-Former architecture, if the set of original tiles that a compressed tile is interested in contains any original tile marked as changed, then the compressed tile is determined to be a changed tile.

[0118] Building upon this, the cloud node recalculates the corresponding second key and second value data for compressed tiles identified as changed tiles using the VLM Backbone. For compressed tiles not identified as changed tiles, the cloud node directly reuses the historical key and value data of the corresponding compressed tiles in the first target frame. Ultimately, the cloud node only extracts and transmits the incremental key-value pairs (ΔKV) corresponding to the "changed tiles" in the compressed tiles. This mechanism ensures that even at the feature compression level, the system can accurately recalculate and transmit only the parts that have undergone substantial semantic changes, thereby maximizing the reduction of network transmission load while maintaining inference accuracy. Since the number of compressed tiles is far less than the number of original tiles, the amount of generated second key and second value data is also significantly reduced.

[0119] After receiving the compressed incremental key-value pair, the edge node merges it with the key-value cache corresponding to the historical bottleneck tokens stored locally, thus reconstructing a complete context representation, which in turn drives the action head model to predict robot actions. This implementation method, which combines feature compression and differential transmission, minimizes data transmission bandwidth consumption and communication latency between the cloud and the edge while ensuring semantic understanding accuracy. It is particularly suitable for robot control scenarios with limited bandwidth or extremely high real-time requirements.

[0120] In step S313, the cloud node sends intermediate data to the edge node.

[0121] To further optimize transmission efficiency, for incremental KV transmission, the cloud node first performs serialization and encoding processing on the generated incremental KV. Specifically, since the incremental KV only contains key and value data corresponding to the changed tiles, it is presented as a sparse tensor structure in memory. The cloud node uses a sparse tensor encoding format to represent the incremental KV as an index-key-value triple sequence (Index-Key-Value Tuples), where the index identifies the position of the changed tile in the original image grid, and the key and value are the K vector and V vector of the corresponding tile, respectively. This encoding method avoids transmitting a large amount of zero or invalid data, significantly reducing the data payload. Subsequently, to further compress the data volume, the cloud node performs quantization processing on the encoded incremental KV, for example, quantizing high-precision floating-point numbers (such as Float16 or Float32) into low-precision integers (such as Int8), reducing the data volume by about 50% while maintaining acceptable inference accuracy. Next, the cloud node uses lossless compression algorithms (such as ZSTD or LZ4) to compress the quantized data to eliminate data redundancy. For full KV transmission, cloud nodes can similarly quantize and compress the full KV data before sending it to edge nodes.

[0122] In step S315, the edge node predicts the second action sequence based on intermediate data and the robot state corresponding to the second target frame using the first model.

[0123] After receiving the intermediate data, the edge nodes first perform decompression, inverse quantization, and decoding operations to reconstruct the original intermediate data for use in the prediction of the first model. (Reference) Figure 6 If the original intermediate data is an incremental key-value pair, the edge node replaces the old data at the corresponding position in the key-value cache with the new key data and value data, thus completing the incremental update of the key-value cache; if the original intermediate data is a full key-value pair, it directly overwrites the original local key-value cache.

[0124] The first model, as a lightweight action generation module, typically has far fewer parameters than the cloud-based visual language backbone network (e.g., less than 500M parameters), aiming to meet the stringent requirements of low-latency inference at the edge. In its implementation, the first model can adopt several architectural forms. The first implementation uses a diffusion strategy head, which models action generation as a denoising process, generating smooth and physically constrained action trajectories from a noise distribution through iterative denoising. The second implementation uses a Transformer decoder head, which employs an autoregressive approach, treating action sequences as token sequences for progressive generation. The third implementation uses a Perceiver IO architecture head, which interacts with the input context through a set of learnable latent vectors, efficiently fusing multimodal information. Regardless of the architecture used, the core function of the first model is to utilize the semantic context provided by the cloud (e.g., updated KV cache) and locally perceived ontology states to output a multi-dimensional sequence of action commands, including joint angles, end-effector poses, or gripper states.

[0125] In the first implementation, the first model employs a diffusion strategy head. In this architecture, edge nodes acquire robot state data (such as joint angles and end-effector poses) at the time of capturing the second target frame (e.g., t2) and encode it into an embedding vector of the ontology state using a multilayer perceptron (MLP). Subsequently, the diffusion strategy head initializes a random noisy action sequence and uses the reconstructed KV cache (or conditional vector) and the ontology-aware embedding vector as conditional inputs to the denoising process. In the denoising network of the first model, through multiple iterative steps, a cross-attention mechanism is used to capture the correlation between visual-linguistic semantics and ontology state, gradually reconstructing a clear action sequence from the noise. For example, the K and V vectors of each token in the KV cache can be used as the K and V data in the cross-attention calculation, respectively, and the Q data in the cross-attention calculation can be calculated based on the embedding vector of the ontology state, thereby performing cross-attention calculation. Finally, after a preset number of denoising steps, a smooth second action sequence that conforms to robot kinematic constraints is output. This generation method can effectively handle the uncertainty of multimodal inputs and generate diverse and robust action trajectories.

[0126] In the second implementation, the first model employs a Transformer decoder head. In this architecture, action generation is treated as an autoregressive sequence prediction task. Edge nodes also first update or construct a complete context representation (e.g., a complete key-value cache) based on intermediate data. Next, the edge nodes encode robot state data into ontology-aware tokens and append them to the end of the key-value cache sequence. The first model defines a set of learnable action query vectors. During inference, the first model calculates the attention weights between the action token of the current step and the historical context (including visual, linguistic, and previously generated action tokens), generates the probability distribution of the next action token, and selects the most probable action value through sampling or a greedy strategy. This process is repeated until a second action sequence of full length is generated. This autoregressive generation approach is structurally simple, easy to implement, and fully leverages the advantages of the Transformer architecture in sequence modeling.

[0127] In the third implementation, the first model employs a Perceiver IO architecture. Under this architecture, the edge nodes first define a fixed number of learnable action latent vectors (e.g., 32). Then, the first model utilizes a cross-attention mechanism, treating the "action latent vector" as a query and the keys and values ​​in the "complete KV cache" as the queried objects, performing cross-attention calculation to obtain updated action latent vectors. These updated action latent vectors include semantic information most relevant to the current robot state and task, extracted from a vast visual-linguistic context. Simultaneously, the edge nodes convert the robot state data corresponding to the second target frame into ontology state embedding vectors using an MLP encoder, and then concatenate or fuse these embedding vectors with the action latent vectors. Next, a self-attention mechanism is used to perform internal feature interaction and integration on the fused vectors to obtain latent vectors, capturing the dependencies between action dimensions. Finally, a multilayer perceptron (MLP) maps the latent vectors to a continuous second action sequence. The advantage of the PerceiverIO architecture lies in its ability to efficiently compress and process large-scale KV cache information with a small number of latent vectors, significantly reducing computational complexity while maintaining high-precision motion prediction capabilities, making it particularly suitable for edge devices with limited computing power.

[0128] refer to Figure 4The edge node can send the second target frame to the cloud before the end of the first time period T1 (t2). With incremental key-value (KV) calculation and transmission in the cloud, the cloud node can utilize a very short incremental inference and transmission window to generate intermediate data and send it back to the edge node before the end of the first time period T1. Therefore, before the end of the time period T1, the edge node can predict the robot's second action sequence in the second time period using the first model, based on the intermediate data and the robot state data corresponding to the second target frame. In this process, a pipelined approach allows the edge node's action execution to proceed in parallel with the cloud's inference and data transmission. The edge node does not need to wait for any network round-trip time (RTT) and can directly utilize the ready-made complete context (updated KV cache) and robot state to immediately infer and generate the next action sequence, ensuring the smoothness and stability of the robot's actions and achieving a real-time response speed close to local inference.

[0129] The solution in the embodiments of this specification introduces a visual change region prediction mechanism based on robot forward kinematics. Through the "signaling first" strategy, the edge end prioritizes sending a mask with very small changes, so that the cloud can preprocess the image data at the same time as receiving it, thereby reducing the inference latency of the cloud.

[0130] By transmitting incremental key-value pairs across nodes, key-value reuse, originally limited to single machines, is extended to cloud-edge collaborative scenarios. This approach directly avoids network congestion caused by full key-value transmission, eliminates transmission bottlenecks in distributed inference, and enables edge devices with limited computing power to obtain the semantic context of large cloud models in real time, thereby achieving low-latency segmentation inference.

[0131] Based on the varying sensitivities of each Transformer layer to image feature processing, a layered strategy of high reuse in shallow layers and precise recalculation in deep layers is implemented. This approach leverages the high spatiotemporal redundancy of shallow features, avoiding redundant computation and transmission of invalid data, while ensuring the accuracy of deep semantic information. This further reduces the amount of data transmitted.

[0132] In addition, to address the issue of accumulated errors (i.e., KV drift) that may arise from the near reuse of the KV cache in the Transformer global attention mechanism, the embodiments in this specification effectively ensure the stability and accuracy of long-term inference through a refresh control mechanism.

[0133] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used when writing program development code. The original code before compilation must also be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit that implements the logic method flow can be easily obtained.

[0134] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0135] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0136] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0137] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0138] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0143] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0144] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0147] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for predicting the actions of a robot, the robot including a camera, the method being executed by a first node, the first node including a first model, the method comprising: The robot's first state and a first action sequence corresponding to a first time period are obtained. The first action sequence is predicted by the first model relative to a first target frame and the first state. The first target frame is captured by the camera before the first time period. The first state is the state of the robot when the first target frame is captured. The location information of the target tile in the first target frame is determined based on the first state and the first action sequence. Send the location information to the second node; The robot is instructed to execute the first action sequence during the first time period, and to acquire the second target frame captured by the camera at the target time within the first time period. Send the second target frame to the second node; Intermediate data is received from the second node, the intermediate data being generated based on the second target frame and the location information; Using the first model, based on the intermediate data and the robot's second state, a second action sequence of the robot is predicted in a second time period, where the second time period is the time period after the target time, and the second state is the state of the robot when the second target frame is captured.

2. The method according to claim 1, wherein the intermediate data includes second key data and second value data corresponding to the target tile, and the first node stores the first key data and the first value data corresponding to the first target frame obtained from the second node; The step of predicting the robot's second action sequence in the second time period using the first model, based on the intermediate data and the robot state data corresponding to the second target frame, includes: Based on the second key data and the second value data, update the first key data and the first value data corresponding to the first target frame to the first key data and the first value data corresponding to the second target frame; Based on the first key data and first value data corresponding to the second target frame and the robot state data corresponding to the second target frame, the second action sequence of the robot in the second time period is predicted.

3. The method according to claim 2, wherein the first node records the target frame number, and sending the second target frame to the second node includes: If the target frame number is less than the threshold, the second target frame and the first instruction for indicating incremental key-value data transmission are sent to the second node together, and the target frame number is incremented by 1. The method further includes: when the number of target frames is greater than or equal to a preset threshold, sending the second target frame and a second instruction for indicating full key-value data transmission to the second node, and setting the number of target frames to 0.

4. The method according to claim 3, wherein sending the second target frame to the second node comprises: Calculate the similarity between the second target frame and the first target frame. If the similarity is higher than a preset threshold, send the second target frame and the first instruction together to the second node.

5. The method according to claim 1, wherein determining the position information of the target tile of the second target frame relative to the first target frame based on the first action sequence includes: The pose change of the camera is calculated based on the first action sequence; The content changes of each tile in the first target frame are determined based on the pose change amount. The location information of the target tile is determined based on the rate of change of the content of each tile.

6. The method according to claim 2, wherein the target time is separated from the end time of the first time period by a preset time interval. Receiving intermediate data from the second node includes receiving intermediate data from the second node before the end time; The step of predicting the robot's second action sequence in the second time period using the first model, based on the intermediate data and the robot state data corresponding to the second target frame, includes: Before the end time, the robot's second action sequence in the second time period is predicted using the first model based on the intermediate data and the robot state data corresponding to the second target frame.

7. A method for predicting the actions of a robot, the robot including a camera, the method being executed by a second node, the second node including a second model, the second node storing first key data and first value data corresponding to a first target frame generated by the second model, the first target frame being captured by the camera before a first time period, the method comprising: The robot receives the location information of the target tile of the first target frame from the first node. The location information is determined based on the robot's first state and a first action sequence corresponding to a first time period. The first action sequence corresponds to the first target frame, and the first state is the state of the robot when it captures the first target frame. The second target frame is received from the first node, and the second target frame is captured by the camera at a target time within the first time period; The second model generates intermediate data based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame; The intermediate data is sent to the first node to predict the robot's second action sequence in a second time period, which is the period after the target time.

8. The method according to claim 7, wherein the second model includes a multi-layer attention module, and the step of generating intermediate data through the second model based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame includes: The second target frame is divided into multiple sequentially arranged tiles, and the target tile among the multiple tiles is determined based on the location information; The second key data and second value data corresponding to each target tile in each attention module are recalculated as the intermediate data.

9. The method according to claim 7, wherein the second model comprises a plurality of first attention modules and a plurality of second attention modules arranged after the plurality of first attention modules. The step of generating intermediate data using the second model based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame includes: When the proportion of the target patch to all patches in the second target frame is lower than a preset proportion, the multiple first attention modules generate a first latent feature based on the second target frame and the key data and value data corresponding to the first target frame. The multiple second attention modules generate second key data and second value data corresponding to each target patch based on the first latent feature, which are used as intermediate data.

10. The method according to claim 7, wherein the second node includes a compression module, and the step of generating intermediate data based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame through the second model includes: The compression module compresses the multiple tiles included in the second target frame into a reduced number of compressed tiles. Based on the location information and the mapping relationship between the multiple map tiles and the multiple compressed map tiles, the target map tile in the multiple compressed map tiles is determined; The second model generates second key data and second value data corresponding to each of the target tiles, which are used as intermediate data.

11. A system for predicting the actions of a robot, comprising a first node and a second node, the robot including a camera, the first node including a first model, the second node including a second model, the second node storing first key data and first value data corresponding to a first target frame generated by the second model, the first target frame being captured by the camera before the first time period. The first node is used to: acquire a first state of the robot and a first action sequence corresponding to a first time period, wherein the first action sequence is predicted by the first model relative to a first target frame, the first target frame is captured by the camera before the first time period, and the first state is the state of the robot when the first target frame is captured; Based on the first state and the first action sequence, determine the location information of the target tile in the first target frame; send the location information to the second node; The robot is instructed to execute the first action sequence during the first time period, and to acquire the second target frame captured by the camera at the target time within the first time period; the second target frame is then sent to the second node. The second node is used to: generate intermediate data based on the second target frame, the location information, and the first key data and first value data corresponding to the first target frame using the second model; and send the intermediate data to the first node; The first node is further configured to: predict a second action sequence of the robot in a second time period based on the intermediate data and the second state of the robot using the first model, wherein the second time period is a time period after the target time, and the second state is the state of the robot when the second target frame is captured.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-10.

Images