Generative large model robot control method and system for indoor dynamic scene

Abstract

The invention discloses an indoor dynamic scene-oriented generative large model robot control method and system, and the method comprises the steps: predicting the future ontology state of a robot when the remaining length of an action buffer area is equal to the reasoning time delay step number; adjacent RGB images collected by a robot are obtained, and an optical flow field is calculated and mapped into an RGB-like image; respectively carrying out visual coding on the current RGB image and the RGB-like image, and splicing to form visual context features; inputting the visual context feature, the future ontology state and the input instruction into a slow flow macro planner, generating a macro action block, and writing the macro action block into an action buffer area; in each control period, popping up a head action of an action buffer area as a macro action, inputting the head action, the current RGB image and the body state into a fast flow residual error refinement device together, outputting an action residual error, and correcting and executing the macro action according to the action residual error; and circularly executing the process until the end. According to the invention, the robot can stably complete the target operation in the dynamic control scene.

Description

Generative Large Model Robot Control Method and System for Indoor Dynamic Scenes Technical Field

[0001] This invention relates to the field of robot intelligent control technology, and in particular to a generative large-scale robot control method and system for indoor dynamic scenes. Background Technology

[0002] Robot manipulation typically employs a closed-loop control structure: the controller outputs control commands in real time based on sensor observations (such as RGB / depth images, force sensing, joint position / velocity, etc.) to enable the end effector to achieve the target pose or complete tasks such as grasping and placing. In recent years, Vision-Language-Motion Models (VLAs) have learned from large-scale multimodal data, mapping natural language commands with visual observations and robot body states into action sequences, enabling the execution of various manipulation tasks in open environments.

[0003] To ensure the temporal consistency and smooth execution of action sequences, many VLAs employ "action chunking": at time t, an action block for the next H steps is predicted and executed sequentially in the subsequent K steps. To avoid execution pauses, asynchronous inference is typically used during deployment: when the remaining buffer action length is close to the inference latency T_lat, the next inference is triggered in advance, and the new action block is appended to the action buffer.

[0004] Existing representative solutions can be divided into three categories:

[0005] (1) Static input VLA action block strategy (such as PI0.5 based on Conditional Flow Matching): The model input is usually a single frame RGB image I t Language instructions c text and the robot's current body state q t Output action block A t ={a t+k Since vision only contains static semantic information and lacks motion derivative information such as velocity, it is difficult to infer the trajectory of a moving target.

[0006] (2) Improvements to asynchronous execution for inference latency (such as RTC, Training-Time RTC): By completing / patch the action prefix during inference or introducing action prefix conditions during training, the "stop-go" effect can be mitigated and the efficiency of asynchronous execution can be improved. However, this type of method still uses static RGB as input, and the execution of action blocks is still mainly open-loop.

[0007] (3) Forward rollout method for state drift (such as VLASH): When triggering inference, the robot body state is rolled forward to estimate the body state at the execution time, and then the action block is generated accordingly, thereby alleviating some of the coordinate drift caused by time delay. However, this method usually still lacks explicit motion perception (such as optical flow) and high-frequency closed-loop correction mechanism within the block, so it is still prone to failure in fast dynamic scenarios (conveyor belt grasping, mobile container delivery, dynamic stacking, etc.).

[0008] The existing solutions have the following main technical drawbacks:

[0009] 1. Perception gap: Most VLAs use a single frame of RGB as visual input, which lacks explicit expression of motion cues such as target speed, direction, and acceleration. It is difficult to predict the future position and contact timing of moving targets, resulting in large grasping / placement alignment errors.

[0010] 2. Latency Gap: Large model inference typically has significant latency (on the order of hundreds of milliseconds). In asynchronous inference, action blocks are generated based on past observations and past ontology states, but the system state has changed by the time of execution, leading to a mismatch between the action and the real state. In dynamic tasks, this can easily result in failures such as "not being able to catch up" or "missing the target".

[0011] 3. Control gap: The action blocks are executed in an open loop within K steps. During the block, the robot is "invisible" to changes in the environment and lacks the ability to make high-frequency closed-loop corrections, making it difficult to resist disturbances or sudden changes in the target's movement. Summary of the Invention

[0012] To address the shortcomings of existing technologies, this invention provides a generative large-scale robot control method and system for indoor dynamic scenarios. In dynamic control scenarios where there is inference latency and the task object / environment is in motion, it can simultaneously supplement motion perception capabilities, offset state drift caused by latency, and provide real-time closed-loop correction during action block execution, enabling the robot to stably complete the target operation.

[0013] Firstly, a generative large-scale robot manipulation method for indoor dynamic scenes is provided, including the following steps: when the remaining length of the action buffer is equal to the number of inference delay steps... Time, Read the Future The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. The process involves acquiring RGB images from the current and previous moments, calculating the optical flow field based on the RGB images from both moments, mapping it to an RGB-like image, visually encoding the current RGB image and the RGB-like image, and then concatenating them to form a unified visual context feature. ;by( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, Input instructions; in each control cycle, pop the action buffer header and the action as a macro action. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. And execute; repeat the above process until the task is completed or the termination condition is met.

[0014] Furthermore, the robot's future ontological state is obtained by constructing a kinematic or dynamic model of the robot. ; Obtain the current state of the robot and the future The sequence of actions to be executed in step The future physical state of robots Calculated using the following formula: In the formula, express The first in One action.

[0015] Furthermore, the optical flow field is calculated based on the RGB images at two time points and mapped to an RGB-like image. Specifically, this includes inputting the RGB images acquired by the robot at the current time point and the previous time point into the optical flow estimation model to obtain the optical flow field. By performing polar coordinate mapping on the two-dimensional optical flow field, mapping the direction of motion to hue and the velocity to saturation, an RGB-like image is obtained. ; corresponds to a pixel in a two-dimensional optical flow field The mapping process is represented as follows: In the formula, Indicates hue, Indicates saturation.

[0016] Furthermore, unify visual context features This is obtained through a parallel image encoder. and optical flow encoder For the RGB image at the current time respectively and RGB-like images Encoding is performed to obtain semantic features and motion features respectively; the semantic features and motion features are then concatenated along the sequence dimension to form a unified visual context feature. , means as follows: In the formula, This indicates a splicing operation.

[0017] Furthermore, when training the VLA-based slow-flow macro-planner, time-shift augmentation is employed: the vision at time t is compared with that at time t+. The entity state is taken as input, and time t+ The action blocks are used as the supervision targets, enabling the VLA-based slow-flow macro-planner to learn to generate execution-time-oriented actions when there is a time delay.

[0018] Furthermore, the specific process of the fast-flow residual refiner trained by reinforcement learning is as follows: The current RGB image is input into the visual encoder to obtain... splicing The current state of the entity. and , obtain fused input ;Will Input the MLP obtained through reinforcement learning, and output the action residuals. The process is represented as follows: ; In the formula, This represents the scaling factor, used to limit the output amplitude; This represents the Tanh activation function. This represents the operation of a multilayer perceptron.

[0019] Furthermore, the slow-flow macro planner is deployed on a GPU server or a high-computing edge, while the fast-flow residual refiner is deployed on the robot's local computing unit. The two communicate through a shared motion buffer or message queue.

[0020] Secondly, a generative large-scale robot control system for indoor dynamic scenes is provided, including: an action buffer and scheduling module for maintaining the action buffer, satisfying the condition that the remaining length of the action buffer is equal to the number of inference delay steps. At that time, slow-flow action reasoning is triggered; future state prediction is used to read the future. The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. The system comprises: a visual acquisition module for acquiring RGB images from the current and previous moments; an optical flow estimation module for calculating the optical flow field based on the RGB images from the two moments; an optical flow visualization module for mapping the optical flow field to an RGB-like image; and a dual-encoder visual fusion module for visually encoding the current RGB image and the RGB-like image, respectively, and then concatenating them to form a unified visual context feature. ; Slow-flow motion generation module, used to generate ( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, For input commands; the fast-flow residual correction module is used to pop the action buffer header as a macro action in each control cycle. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. The execution module will execute the modified actions. Send it to the robot controller for execution.

[0021] Thirdly, a generative large model robot control system for indoor dynamic scenes is provided, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to execute the generative large model robot control method for indoor dynamic scenes as described above.

[0022] Compared with the prior art, the present invention has the following beneficial effects: (1) Improve the observability of dynamic targets: By introducing optical flow and dual encoder fusion, the model simultaneously obtains information on "what it is (semantics)" and "how it moves (speed / direction)," thereby being able to predict the trajectory of the moving target and perform timing control, solving the perception gap of static VLA; (2) Counteract state drift caused by inference delay: Through future state perception (kinematic forward unfolding based on action buffer), the slow-flow macro-planner uses the ontological state at the execution moment as a condition when generating actions, significantly reducing the mismatch between actions and real states, solving the delay gap; (3) Block Real-time closed-loop correction: The fast flow residual refiner injects residual correction into the macro action according to real-time observation in each control cycle to realize high-frequency closed-loop control, solve the control gap caused by open-loop action block, and improve the robustness to disturbances and target mutations; (4) Taking into account global consistency and real-time performance: the slow flow is responsible for semantic reasoning and long-term planning, and the fast flow is responsible for rapid response, forming a complementary relationship; compared with the solution that only improves the efficiency of asynchronous execution, it can maintain stable execution in high dynamic / high latency scenarios; (5) Can be deployed in an engineering manner: the slow flow can run at low frequency on the high computing power end, and the fast flow can run lightly on the end side, which meets the computing power and latency constraints of the actual robot system. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a flowchart of the generative large model robot control method for indoor dynamic scenes provided by an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] While Visual-Language-Action (VLA) models excel in static tasks, they perform poorly in dynamic environments such as conveyor belt operations. Three fundamental limitations hinder current VLA performance in these scenarios: the perception gap (lack of temporal motion cues in static visual input); the latency gap (inference delays leading to outdated actions); and the control gap (lack of real-time adjustments during open-loop action block execution). Therefore, this invention proposes a Dynamic-VLA, a slow-fast dual-stream framework integrating motion-aware forward planning with real-time residual correction. The slow-stream macro-planner proactively offsets inference latency by enhancing VLA with optical flow for time-aware and future-state-aware mechanisms, generating globally consistent, motion-aware action blocks. Complementing this, the fast-stream residual refiner employs a lightweight RL (reinforcement learning) strategy, injecting high-frequency closed-loop corrections into planned action blocks based on real-time observations. The technical solution of this invention is further described below with reference to specific embodiments.

[0027] As shown in Figure 1, this embodiment of the invention provides a generative large-scale robot control method for indoor dynamic scenes, including the following steps: S1: When the remaining length of the action buffer is equal to the number of inference delay steps. Time, Read the Future The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. .

[0028] The number of inference delay steps is determined by the model inference time and the duration of each control time step. Therefore, whenever the remaining length of the action buffer equals the number of inference delay steps... When this happens, the next slow-flow inference needs to be triggered, and the slow-flow macro-planner generates a new action block and appends it to the action buffer.

[0029] In this embodiment, the future body state of the robot is obtained through the following method: S11: Constructing a kinematic model or dynamic model of the robot. ;Preferably, a forward kinematics model is used to unfold the robot's body state;The method for constructing the robot's kinematics model or dynamics model is existing technology and will not be described in detail here;S12: Obtain the current body state of the robot. and the future The sequence of actions to be executed in step S13: The robot's future physical state Calculated using the following formula: In the formula, express The first in One action. Using... This represents the robot's future state at the current moment. For robots in The state of the entity after each action is completed.

[0030] S2: Obtain the RGB images collected by the robot at the current moment and the previous moment, calculate the optical flow field based on the RGB images at the two moments, and map it into an RGB-like image.

[0031] In this embodiment, the optical flow field is calculated based on the RGB images at two time points and mapped to an RGB-like image. Specifically, this includes: S21: Inputting the RGB images acquired by the robot at the current time point and the previous time point into the optical flow estimation model to obtain the optical flow field. The optical flow estimation model can be the RAFT (Recurrent All-Pairs Field Transforms) model or other optical flow estimation network models.

[0032] S22: The two-dimensional optical flow field is mapped to polar coordinates, the motion direction is mapped to hue, and the velocity is mapped to saturation, resulting in a three-channel RGB-like image. ; corresponds to a pixel in a two-dimensional optical flow field The mapping process is represented as follows: In the formula, Indicates hue, Indicates saturation.

[0033] By mapping a two-dimensional optical flow field to a three-channel RGB-like image, geometric motion information is transformed into a color space representation, enabling alignment with the semantic space of a pre-trained visual encoder. It should be noted that the RGB image can be a single-view RGB image or a multi-view RGB image; if it is a multi-view RGB image, the corresponding RGB-like image is also a multi-view RGB image.

[0034] S3: Visually encode the RGB image and the RGB-like image at the current moment, and then concatenate them to form a unified visual context feature. .

[0035] In this embodiment, unified visual context features This is obtained through a parallel image encoder. and optical flow encoder For the RGB image at the current time respectively and RGB-like images Encoding is performed to obtain semantic features (e.g., object identity) and motion features (e.g., motion trajectory); the image encoder and optical flow encoder can be either ViT or CLIP encoders; the semantic features and motion features are concatenated along the sequence dimension to form a unified visual context feature. , means as follows: In the formula, This indicates a splicing operation.

[0036] It should be noted that if the RGB image is a multi-view RGB image, the RGB images from multiple views at the current moment are visually encoded separately using an image encoder, and then concatenated to obtain the fused semantic features. Similarly, the RGB-like images from multiple views at the current moment are visually encoded separately using an optical flow encoder, and then concatenated to obtain the fused motion features. Finally, the fused semantic features and the fused motion features are concatenated to obtain the unified visual context features. .

[0037] S4: with ( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, For input instructions.

[0038] The slow flow macro-planner employs a VLA-based architecture and can utilize generative strategies (such as Conditional Flow Matching, diffusion, or autoregression) to output continuous actions (control variables, such as pose changes, joint increments, or velocities). When training the VLA-based slow flow macro-planner, temporal displacement augmentation is used: time-shifting is used... Vision and Moment The ontological state is taken as input, at time... The action blocks are used as the supervision targets, enabling the VLA-based slow-flow macro-planner to learn to generate execution-time-oriented actions when there is a time delay.

[0039] S5: In each control cycle, pop the action buffer header action as a macro action. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. + And execute.

[0040] In this embodiment, the fast-flow residual refiner can employ a lightweight network (such as a ResNet feature extraction network + MLP) and an Actor-Critic structure. For example, a ResNet feature extraction network can be used as an encoder, and the encoded features are then input into an MLP to obtain an action. The encoded features and action are then input together into another MLP to obtain a critic (Q-value). The residual output is trained through reinforcement learning. The specific process of the fast-flow residual refiner trained by reinforcement learning is as follows: S51: Input the current RGB image into the visual encoder to obtain... In this embodiment, the visual encoder uses a ResNet-10 backbone network followed by a learnable spatial embedding layer. The learnable spatial embedding layer uses a set of Ns learnable spatial blocks (preferably Ns=8) to extract key geometric features, ultimately obtaining... This method captures the precise spatial relationship between the robot's end effector and the object. For example, the ResNet-10 backbone outputs a feature map of shape [B, 512, 4, 4]. The learnable spatial embedding layer has a learnable parameter kernel of shape [1, 512, 4, 4, 8]. The feature map and kernel are element-wise weighted and then summed over the spatial dimension (H, W) to obtain [B, 512, 8]. This is then flattened to [B, 4096] and fed into a subsequent Linear layer + LayerNorm (layer normalization) + Tanh (hyperbolic tangent activation function), compressing it to a 256-dimensional representation of a single image. It should be noted that if the RGB image is a multi-view RGB image, then... To obtain the image by inputting RGB images from multiple viewpoints at the current moment into a visual encoder for encoding and then stitching them together; S52: Stitching The current state of the entity. and , obtain fused input S53: will Input the MLP obtained through reinforcement learning, and output the action residuals. The process is represented as follows: ; In the formula, This represents the Tanh activation function. This represents the scaling factor, and the residual output is processed by the Tanh activation function and... Limit the output amplitude to ensure safe execution; This represents the multilayer perceptron operation. In this embodiment, the policy network is an MLP (Multilayer Perceptron) with two 256-unit hidden layers.

[0041] During training, data enhancements such as random pixel shifts can be introduced to improve robustness. Training is an asynchronous actor-learner closed loop: the actor interacts with the environment while sampling actions, initially exploring randomly, then sampling with the current policy, continuously writing (observation, action, reward, next observation, termination) into a queue and sending it to the replay buffer; the learner starts training once the replay buffer reaches a threshold, performing 3 critic-only updates + 1 actor / critic joint update per round by default. That is, first, the Bellman error of the critic network is repeatedly minimized using replay data, allowing the critic to learn to score actions, and then the value signal provided by the critic is used to update the actor, moving the policy towards actions with higher Q values. Subsequently, the learner periodically synchronizes the latest parameters back to the actor, so the actor is responsible for "collecting new data", the critic is responsible for "evaluating the quality of actions", and the actor improves according to the critic's judgment; if demonstrations are provided, the training batch will also be a mix of half demos and half online experience. On the image side, the pixel learning route of DrQ (Data-regularized Q-learning) is followed: the observation first enters the visual encoder (the example uses a pre-trained ResNet-10 by default), and during training, DrQ's random translation / random cropping enhancement is applied to the image to perform data regularization, so that Q-learning is more stable to small visual perturbations and the sample efficiency is higher.

[0042] S6: Repeat the above process until the task is completed or the termination condition is met.

[0043] In practice, a parallel deployment strategy is adopted: the slow-flow macro planner is deployed on a GPU server or a high-computing edge device, with an execution frequency of 2-3Hz; the fast-flow residual refiner is deployed on the robot's local computing unit, with an execution frequency of 30Hz or higher; the two communicate through a shared motion buffer or message queue.

[0044] The above embodiment provides a generative large model robot control method for indoor dynamic scenes, which has the following beneficial effects: (1) Improve the observability of dynamic targets: optical flow is calculated from adjacent frames and used as explicit motion cue input. By introducing optical flow and dual encoder fusion, the model simultaneously obtains information on "what it is (semantics)" and "how it moves (speed / direction)," thereby being able to predict the trajectory of the moving target and perform timing control, solving the perception gap of static VLA; (2) Offset state drift caused by inference delay: through future state perception (kinematic forward unfolding based on action buffer), the slow-flow macro-planner uses the body state at the execution time as a condition when generating actions, offsetting the state drift caused by delay, and significantly reducing the state drift caused by inference delay. (3) Real-time closed-loop correction within the block: The fast flow residual refiner injects residual correction into the macro action according to real-time observation in each control cycle to realize high-frequency closed-loop control, solve the control gap caused by open-loop action block, and limit the residual amplitude to ensure execution stability and safety, and improve the robustness to disturbances and target mutations; (4) Taking into account global consistency and real-time performance: Slow flow is responsible for semantic reasoning and long-term planning, and fast flow is responsible for rapid response, forming a complementary relationship; compared with the solution that only improves the efficiency of asynchronous execution, it can maintain stable execution in high dynamic / high latency scenarios; (5) Can be deployed in an engineering manner: Slow flow can run at low frequency on the high computing power end, and fast flow can run lightly on the end side, meeting the computing power and latency constraints of the actual robot system.

[0045] The above method is applicable to single-arm / dual-arm robots, fixed / mobile base robots, and dynamic control tasks in simulation environments. When the environment directly provides state observations, the optical flow estimation module can be omitted and replaced by state derivatives or velocity vectors.

[0046] The following set of experiments illustrates the effectiveness of the generative large-scale robot control method for indoor dynamic scenes provided in the above embodiments.

[0047] Hardware and Task Setup: The proposed Dynamic-VLA was validated on a real robot setup using a 6-DOF Agilex Piper robotic arm equipped with two Intel RealSense D435 cameras. The slow-flow macro planner was driven by a PI0.5 backbone with an action block size of H=10. The fast-flow residual refiner operated at 30Hz. Four challenging dynamic tasks were designed: (1) Conveyor belt grasping, where the robot must grasp a moving target cube on a conveyor belt; (2) Dynamic dropping, where the robot must release the grasped object into a moving container; (3) Dynamic stacking, where the robot must grasp a cube and stack it onto another moving cube; and (4) Moving pouring, a high-precision task where the robot pours water into a moving cup.

[0048] Table 1 quantifies the significant differences between the static baseline and the method of this invention across 50 trials for each task. Dynamic-VLA, by effectively addressing the three major gaps, significantly outperforms the PI0.5 baseline and competing methods (RTC, VLASH). The PI0.5 baseline exhibits a catastrophic failure mode in dynamic settings, with an average success rate of only 7.5%. Notably, in dynamic stacking tasks requiring precise relative motion planning, the baseline fails completely (0%). In contrast, Dynamic-VLA achieves an average task success rate of 48.5%, a 6.5-fold improvement over the baseline. Even compared to state-of-the-art asynchronous methods such as VLASH (26.5%), the performance of the method of this invention is nearly doubled. This is most evident in high-precision moving and tipping tasks, where Dynamic-VLA achieves a 56% success rate, while VLASH achieves 30%. Empirical evidence demonstrates that addressing latency alone is insufficient; active high-frequency residual correction is indispensable for smooth interaction with moving targets.

[0049] Table 1. Real-world success rate on dynamic operation tasks

[0050] As can be seen from the table above, Dynamic-VLA has the highest success rate across all tasks, with an average improvement of 41%.

[0051] Design selection ablation: To demonstrate the rationality of the specific architecture decision of this invention, a fine-grained ablation study was conducted on the dynamic crawling task, and the results are shown in Table 2.

[0052] Table 2. Design choice ablation results on real-world dynamic grasping tasks As can be seen from Table 2:

[0053] (1) Optical flow effect (no optical flow): Complete removal of optical flow input resulted in a catastrophic drop in success rate to 28%, confirming that standard static RGB input lacks the high-order derivative information (velocity) required to track moving objects.

[0054] (2) Effect of polar coordinate mapping (without polar coordinate mapping): The success rate of directly projecting the original optical flow to the VLA token space using the ResNet-34 encoder is only 30%, which is attributed to the misalignment of the feature space.

[0055] (3) Impact of future state perception (no future): The success rate of disabling this mechanism is 34%. Without this module, the slow-flow macro-planner generates actions based on the state at the time of observation, ignoring system delays, which leads to coordinate system mismatch.

[0056] (4) Closed-loop correction effect (without fast flow residual refiner): Removing the fast flow residual refiner reduces performance to 24%, highlighting the severity of the control gap.

[0057] (5) Impact of RL training (fast flow residual refiner trained with SFT): The performance of the fast flow residual refiner was only restored to 36% when trained with SFT instead of RL, which is much lower than the RL-trained version, indicating that the RL formula is crucial to the fast flow responsiveness.

[0058] This invention also provides a generative large-scale robot control system for indoor dynamic scenes, including: an action buffer and scheduling module for maintaining an action buffer, satisfying that when the remaining length of the action buffer is equal to the number of inference delay steps. At that time, slow-flow action reasoning is triggered; future state prediction is used to read the future. The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. The system comprises: a visual acquisition module for acquiring RGB images from the current and previous moments; an optical flow estimation module for calculating the optical flow field based on the RGB images from the two moments; an optical flow visualization module for mapping the optical flow field to an RGB-like image; and a dual-encoder visual fusion module for visually encoding the current RGB image and the RGB-like image, respectively, and then concatenating them to form a unified visual context feature. ; Slow-flow motion generation module, used to generate ( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, For input commands; the fast-flow residual correction module is used to pop the action buffer header as a macro action in each control cycle. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. The execution module will execute the modified actions. Send it to the robot controller for execution.

[0059] It should be understood that the functional unit modules in the various embodiments of the present invention can be concentrated in one processing unit, or each unit module can exist physically separately, or two or more unit modules can be integrated into one unit module, and can be implemented in hardware or software.

[0060] This invention also provides a generative large model robot control system for indoor dynamic scenes, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to execute the generative large model robot control method for indoor dynamic scenes as described above.

[0061] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0062] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0066] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A generative large-scale robot control method for indoor dynamic scenes, characterized in that, Includes the following steps: When the remaining length of the action buffer equals the number of inference delay steps Time, Read the Future The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. The process involves acquiring RGB images from the current and previous moments, calculating the optical flow field based on the RGB images from both moments, mapping it to an RGB-like image, visually encoding the current RGB image and the RGB-like image, and then concatenating them to form a unified visual context feature. ;by( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, Input instructions; in each control cycle, pop the action buffer header and the action as a macro action. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. And execute; repeat the above process until the task is completed or the termination condition is met.

2. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, The robot's future ontological state is obtained by constructing a kinematic or dynamic model of the robot. ； Obtain the current state of the robot. and the future The sequence of actions to be executed in step The future physical state of robots Calculated using the following formula: In the formula, express The first in One action.

3. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, The optical flow field is calculated based on the RGB images at two time points and mapped to an RGB-like image. Specifically, this involves inputting the RGB images acquired by the robot at the current time point and the previous time point into the optical flow estimation model to obtain the optical flow field. By performing polar coordinate mapping on the two-dimensional optical flow field, mapping the direction of motion to hue and the velocity to saturation, an RGB-like image is obtained. ; corresponds to a pixel in a two-dimensional optical flow field The mapping process is represented as follows: In the formula, Indicates hue, Indicates saturation.

4. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, Unified visual context features This is obtained through a parallel image encoder. and optical flow encoder For the RGB image at the current time respectively and RGB-like images Encoding is performed to obtain semantic features and motion features respectively; the semantic features and motion features are then concatenated along the sequence dimension to form a unified visual context feature. , means as follows: In the formula, This indicates a splicing operation.

5. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, When training a VLA-based slow-flow macro-planner, time-shift augmentation is employed: the vision at time t is compared with that at time t+. The entity state is taken as input, and time t+ The action blocks are used as the supervision targets, enabling the VLA-based slow-flow macro-planner to learn to generate execution-time-oriented actions when there is a time delay.

6. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, The specific process of the fast-flow residual refiner trained by reinforcement learning is as follows: Input the RGB image at the current time step into the visual encoder to obtain... splicing The current state of the entity. and , obtain fused input ;Will Input the MLP obtained through reinforcement learning, and output the action residuals. The process is represented as follows: ； In the formula, This represents the scaling factor, used to limit the output amplitude; This represents the Tanh activation function. This represents the operation of a multilayer perceptron.

7. The generative large-scale robot control method for indoor dynamic scenes according to claim 1, characterized in that, The slow-flow macro planner is deployed on a GPU server or a high-computing edge, while the fast-flow residual refiner is deployed on the robot's local computing unit. The two communicate through a shared motion buffer or message queue.

8. A generative large-scale robot control system for indoor dynamic scenes, characterized in that, include: The action buffer and scheduling module is used to maintain the action buffer, ensuring that the remaining length of the action buffer equals the number of inference delay steps. At that time, slow-flow action reasoning is triggered; future state prediction is used to read the future. The sequence of actions to be executed in step And use kinematic or dynamic models to predict the robot's future body state. ； The visual acquisition module is used to acquire RGB images of the robot at the current moment and the previous moment; the optical flow estimation module is used to calculate the optical flow field based on the RGB images at the two moments. The optical flow visualization module is used to map the optical flow field into an RGB-like image; The dual-encoder visual fusion module is used to visually encode the current RGB image and the RGB-like image separately, and then concatenate them to form a unified visual context feature. ； Slow-flow motion generation module, used for ( , , The input condition is used to generate a VLA-based slow-flow macro planner, which then generates macro action blocks. And write it to the action buffer; where, For input instructions; The fast-flow residual correction module is used to pop the action buffer header as a macro action in each control cycle. The current RGB image, the body state, and Input a fast-flow residual refiner trained based on reinforcement learning, output action residuals, and then refine the action residuals accordingly. Make corrections to obtain the modified action. The execution module will execute the modified actions. Send it to the robot controller for execution.

9. A generative large-scale robot control system for indoor dynamic scenes, characterized in that, include: One or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to execute the generative large model robot manipulation method for indoor dynamic scenes as described in any one of claims 1 to 7.

Images