Robot control method and device, electronic device

Abstract

The present disclosure provides a robot control method and device, and an electronic device. The method comprises: in each control cycle, reading an action block generated by an inference process from a shared memory, and determining whether a new action block arrives in the action block according to an action block identifier; determining a state of a state machine according to an arrival state of the new action block and a remaining time of a trajectory of the robot; determining a speed factor in the current state; updating a virtual time according to the speed factor, and querying a spline interpolation to obtain an end pose of the robot by using the virtual time; and controlling the robot according to the obtained end pose. The present disclosure solves the technical problem that, in a VLA inference frequency and controller execution frequency asynchronous decoupling architecture, when a new action block is delayed due to a calculation bottleneck, network fluctuation or sudden increase in scene complexity, there is a lack of a trajectory transition mechanism that can guarantee execution continuity and avoid pose out-of-control.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more specifically, to a robot control method and apparatus, and electronic equipment. Background Technology

[0002] After the robot's Vision-Language-Action (VLA) model inference frequency (e.g., 10Hz) is decoupled from the controller execution frequency (e.g., 250Hz), the controller needs to query the end effector pose to be executed in each control cycle (4ms). However, the generation cycle of a new action block is much longer than the control cycle (e.g., the generation cycle of a new action block is about 0.1s). When the inference pipeline delays the generation of a new action block due to computational bottlenecks, network fluctuations, or a sudden increase in scene complexity, if the controller simply stops and waits, it will cause sudden stop jitter. If it extrapolates the previous action block, it will cause pose divergence. Summary of the Invention

[0003] This disclosure provides a robot control method, apparatus, and electronic device to address the technical problem of lacking a trajectory transition mechanism that can both ensure execution continuity and avoid posture loss of control when the inference pipeline is delayed in producing new action blocks due to computational bottlenecks, network fluctuations, or a sudden increase in scene complexity under an asynchronous decoupled architecture of VLA inference frequency and controller execution frequency.

[0004] According to one aspect of the present disclosure, a robot control method is provided. The method includes: in each control cycle, reading action blocks generated by an inference process from shared memory, and determining whether a new action block has arrived in the action blocks based on the action block identifier; wherein the action block includes a plurality of action points arranged in chronological order and their spline interpolation coefficients; determining the state of a state machine based on the arrival state of the new action block and the remaining trajectory time of the robot; wherein the state includes: a normal tracking state, a deceleration approach state, and a freeze waiting state; the remaining trajectory time is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller; determining a velocity factor in the current state; wherein the velocity factor is the advancement ratio of the virtual time relative to the real time; updating the virtual time based on the velocity factor, and querying the spline interpolation to obtain the robot's end pose using the virtual time; and controlling the robot according to the queried end pose.

[0005] Optionally, the state of the state machine is determined based on the arrival status of the new action block and the remaining trajectory time of the robot, including: if a new action block arrives or the remaining trajectory time is greater than or equal to the deceleration window, the state of the state machine is determined to be in the normal tracking state; wherein, the deceleration window is the time threshold for entering the deceleration approach state; if the remaining trajectory time is greater than the control cycle and less than the deceleration window, and no new action block arrives, the state of the state machine is determined to be in the deceleration approach state; if the remaining trajectory time is less than or equal to the control cycle, and no new action block arrives, the state of the state machine is determined to be in the frozen waiting state.

[0006] Optionally, the deceleration window may be greater than or equal to the control period.

[0007] Optionally, the speed factor in the current state is determined as follows: if the state machine is in a normal tracking state, the speed factor is 1; if the state machine is in a deceleration approximation state, the speed factor is the ratio of the remaining trajectory time to the deceleration window; if the state machine is in a frozen waiting state, the speed factor is 0.

[0008] Optionally, updating the virtual time based on the speed factor includes: if the state machine is in a normal tracking state or a deceleration approach state, updating the virtual time using the virtual time increment, and the virtual time is less than or equal to the end time of the current action block; wherein the virtual time increment is the product of the speed factor and the control cycle; if the state machine is in a frozen waiting state, the virtual time is equal to the end time of the current action block.

[0009] Optionally, determining whether a new action block has arrived in the action block based on the action block identifier includes: recording the action block identifier of the action block read in each control cycle in a monotonically increasing order; if the action block identifier of the action block read in the current control cycle is greater than the action block identifier of the action block read in the previous control cycle, it is determined that a new action block has arrived.

[0010] Optionally, the above method further includes: determining the time to enter the freeze waiting state; if the difference between the actual time and the time to enter the freeze waiting state is greater than a preset hard timeout threshold, exiting the control loop.

[0011] Optionally, determining the velocity factor in the current state further includes: if the state of the state machine is a deceleration approximation state, the velocity factor is determined by a nonlinear curve; or ; Where s is the velocity factor, The remaining time for the trajectory. For deceleration window, This refers to the sigmoid function.

[0012] Optionally, the deceleration window can be dynamically adjusted based on historical inference delay statistics.

[0013] Optionally, the frozen waiting state includes multiple sub-states; in different sub-states, the virtual time clamp time gradually approaches the end time of the current action block.

[0014] According to another aspect of the present disclosure, a robot control device is provided, comprising: a first determining module, configured to read action blocks generated by the inference process from shared memory in each control cycle, and determine whether a new action block has arrived in the action blocks according to the action block identifier; wherein, the action block includes a plurality of action points arranged in chronological order and their spline interpolation coefficients; a second determining module, configured to determine the state of a state machine according to the arrival state of the new action block and the remaining trajectory time of the robot; wherein, the state includes: normal tracking state, deceleration approach state, and frozen waiting state; the remaining trajectory time is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller; a third determining module, configured to determine a velocity factor in the current state; wherein, the velocity factor is the advancement ratio of the virtual time to the real time; a processing module, configured to update the virtual time according to the velocity factor, and use the virtual time to query the spline interpolation to obtain the end pose of the robot; and control the robot according to the queried end pose.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described method.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method.

[0017] The beneficial effects of the technical solution provided in this disclosure are: by using a virtual time decoupling mechanism and a three-stage state machine, execution jitter caused by inference latency is eliminated, ensuring the posture accuracy of fine operations, improving the system's tolerance to inference latency, reducing computational overhead and inference power consumption, and making the robot system both efficient and robust under the asynchronous architecture of "slow inference and fast execution". Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0019] Figure 1 A flowchart illustrating a robot control method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a robot control device provided in an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.

[0021] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term, for example, “A and / or B” or “A, B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0022] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0023] First, the technical terms used in this disclosure will be introduced and explained: The vision-language-action model transfers the semantic knowledge of a pre-trained vision-language model to the model paradigm of robot operation, taking RGB images and language commands as input and outputting robot action sequences.

[0024] Action blocks, also known as trajectory blocks, are a series of robot action points (such as an end-effector pose sequence) that are output by a VLA policy in a time-sequential order. Typically, a new block is generated every 0.1 seconds (10 Hz).

[0025] As mentioned above, after the robot's VLA model inference frequency (e.g., 10Hz) is decoupled from the controller execution frequency (e.g., 250Hz), the controller needs to query the end pose that should be executed in each control cycle (4ms). However, the generation cycle of a new action block is much longer than the control cycle (e.g., the generation cycle of a new action block is about 0.1s). When the inference pipeline delays the generation of a new action block due to computational bottlenecks, network fluctuations, or a sudden increase in scene complexity, if the controller simply stops and waits, it will cause sudden stop jitter. If it extrapolates the previous action block, it will cause pose divergence.

[0026] Existing technologies mainly include the following solutions: 1) Fixed-frequency synchronous execution: The controller and inference engine operate at the same frequency (e.g., 10Hz), and each control cycle waits for inference to complete before executing the next action. This is represented by the early synchronous mode of ACT (Action Chunking with Transformers). This scheme has tight time coupling, and inference delay directly causes execution stuttering, failing to fully utilize the high-frequency capabilities of the controller.

[0027] 2) Asynchronous execution + fixed timeout hard stop: The controller and inferencer operate decoupled. The controller is set with a fixed timeout threshold (e.g., 0.5s), and stops immediately after the timeout to wait for the next chunk. This scheme causes an abrupt stop at the moment of timeout, leading to robot jitter and the risk of task interruption. This strategy is used in some Open VLA implementations.

[0028] 3) Linear extrapolation or repeated end-point: When a new chunk has not yet arrived, the controller extrapolates the end-point velocity of the previous chunk (linear extrapolation) or repeats the end-point attitude (simple repetition). The extrapolation scheme is prone to attitude divergence (especially in non-uniform speed segments such as turns and contacts), while the repetition scheme is safe but lacks smooth transition.

[0029] 4) Prediction-based look-ahead interpolation: Predict the content of the next chunk using a trajectory prediction model (such as Kalman filtering or trajectory RNN) and interpolate it in advance. This approach relies on prediction accuracy; prediction errors can lead to attitude changes when the actual chunk arrives, and also increase computational overhead.

[0030] Existing technologies have the following main drawbacks when dealing with the problem of late new chunks in asynchronous scenarios characterized by "slow inference and fast execution": Defect 1: Fixed timeout hard stop causes jitter when executing emergency stop, the speed suddenly drops from the normal value (e.g., 0.2 m / s) to 0, the acceleration jumps by more than 5 m / s², causing mechanical shock.

[0031] Defect 2: Linear extrapolation is prone to attitude divergence in non-uniform speed sections (turning, contact). After extrapolation for 0.5s, the position error can reach 5-10 cm, which exceeds the tolerance of fine operation.

[0032] Defect 3: The simple repetitive end posture lacks deceleration transition. There is a speed jump from normal execution (virtual speed v) to repetition (virtual speed 0), and there is still a slight jitter.

[0033] Defect 4: Forward interpolation based on prediction depends on prediction accuracy. When the prediction error is 5-15 cm, the arrival of a new chunk will cause a change in attitude. In addition, the prediction calculation takes 10-20 ms, which consumes inference computing power.

[0034] Defect 5: The existing solution lacks a hard timeout safety net, and the system will hang indefinitely or behave unpredictably when the inference pipeline fails.

[0035] The robot control method, apparatus, and electronic equipment disclosed herein are intended to solve at least one of the above-mentioned technical problems in the prior art.

[0036] This disclosure addresses the high-frequency execution control scenario of VLA models. In the case of asynchronous decoupled architecture with slow inference (10Hz) and fast execution (250Hz), it proposes an adaptive deceleration and freezing scheme based on virtual time decoupling to solve the problem of execution jitter and pause caused by the late arrival or absence of new action chunks.

[0037] The core concept of this scheme is: the controller internally maintains a virtual time variable. Its propulsion speed is not fixed at a 1:1 ratio and follows real time. Instead, it dynamically adjusts the propulsion rate based on the remaining amount of the current trajectory block and the arrival status of the new chunk, using a three-stage state machine (NORMAL, DECEL, FROZEN). When there is sufficient remaining trajectory and the new chunk arrives on time, virtual time advances normally at a 1:1 ratio with real time. When the remaining amount is nearly exhausted and the new chunk is late, virtual time automatically decelerates, asymptotically approaching the end without overshooting. When no new chunk arrives at the end, virtual time clamps at the end, and the controller repeatedly outputs the end attitude smoothly, waiting until the new chunk arrives or a timeout occurs. This mechanism completely decouples virtual time from real time, ensuring that trajectory interpolation is not affected by inference delays, providing a hard timeout safety net while guaranteeing smooth execution.

[0038] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0039] The virtual time decoupling mechanism disclosed herein serves as the core time-progression logic of the controller in the VLA execution system, residing between the inference pipeline and the underlying SDK. The overall system architecture comprises three main processes: Process 1: Inference Process Function: Receive observations (image + state), call VLA policy inference to generate a new action chunk.

[0040] Frequency: 10Hz (limited by model calculation time and hardware).

[0041] Output: Track block (action block), including N action points ( , ) and its spline interpolation coefficients are written to shared memory.

[0042] Key parameters: chunk id (monotonically increasing, used for deduplication), time interval of the trajectory block.

[0043] Process 2: Control Process Function: Frequently reads the trajectory block in shared memory, obtains the current end pose by querying spline interpolation through virtual time, and sends it to the underlying SDK for execution.

[0044] Frequency: 250Hz (control cycle) =4ms).

[0045] Core modules: Chunk polling module: Checks shared memory every cycle and uses chunk ID to deduplicate and determine if a new block has arrived.

[0046] Virtual Time Advancement Module: Updates virtual time based on a three-stage state machine. .

[0047] Spline interpolation query module: using The end pose is obtained by querying the cubic spline. .

[0048] Attitude distribution module: Distributes the retrieved attitude to the underlying SDK.

[0049] Process 3: Observation Process Function: Acquire camera images and joint states, and send them to the inference process.

[0050] Frequency 30-60Hz.

[0051] The observation process has no direct interaction with this disclosure. An interruption of the observation stream will cause inference to stall, triggering a hard timeout fallback in the controller.

[0052] Figure 1This is a flowchart of a robot control method provided in this embodiment of the disclosure, such as... Figure 1 As shown, the method includes the following steps: Step S101: In each control cycle, read the action block generated by the inference process from the shared memory, and determine whether a new action block has arrived in the action block according to the action block identifier; wherein, the action block includes a number of action points arranged in chronological order and their spline interpolation coefficients.

[0053] As mentioned above, the inference process writes the generated action blocks to shared memory. The control process reads the action blocks from shared memory in each control cycle. Shared memory enables zero-copy inter-process communication (IPC) with a latency of less than 0.5ms, significantly reducing communication latency.

[0054] In some embodiments, the trajectory block includes a time interval. , and action point sequence ( , ) spline interpolation coefficients.

[0055] In some alternative embodiments, the action block identifier (chunk id) can be used to determine whether a new block has arrived.

[0056] According to some optional embodiments of this disclosure, the execution step S101, which determines whether a new action block has arrived in the action block based on the action block identifier, can be implemented by the following method: recording the action block identifier of the action block read in each control cycle in a monotonically increasing order; if the action block identifier of the action block read in the current control cycle is greater than the action block identifier of the action block read in the previous control cycle, it is determined that a new action block has arrived.

[0057] In some embodiments, the chunk ID deduplication mechanism records the chunk ID of the last read operation and determines that a new block has arrived only when the newly read chunk ID is greater than the recorded value. Chunk ID deduplication prevents the controller from repeatedly reading the same chunk, which could lead to virtual time rollback or jumps.

[0058] Step S102: Determine the state of the state machine based on the arrival state of the new action block and the remaining time of the robot's trajectory; wherein, the state includes: normal tracking state, deceleration approach state and frozen waiting state; the remaining time of the trajectory is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller.

[0059] According to some optional embodiments of this disclosure, the arrival state of a new action block includes two states: a new action block has arrived and no new action block has arrived.

[0060] Remaining time for the robot's trajectory ,in, This is the end time of the current action block. (Controller internal maintenance) The query time used for trajectory interpolation is different from the actual system time. .

[0061] In embodiments of this disclosure, the three-stage state machine includes: a normal tracking state (NORMAL state), a deceleration approach state (DECEL state), and a freeze-wait state (FROZEN state).

[0062] Step S103: Determine the velocity factor in the current state; where the velocity factor is the advancement ratio of virtual time to real time.

[0063] Step S104: Update the virtual time according to the velocity factor, and use the virtual time to query the spline interpolation to obtain the robot's end-effector posture; control the robot according to the obtained end-effector posture.

[0064] Through the above technical solutions, the execution jitter caused by inference latency is eliminated by using a virtual time decoupling mechanism and a three-stage state machine, ensuring the posture accuracy of fine operations, improving the system's tolerance to inference latency, reducing computational overhead and inference power consumption, and making the robot system both efficient and robust under the asynchronous architecture of "slow inference and fast execution".

[0065] This embodiment of the disclosure provides a possible implementation method. Step S102 determines the state of the state machine based on the arrival status of the new action block and the remaining trajectory time of the robot, including: if a new action block arrives or the remaining trajectory time is greater than or equal to the deceleration window, the state of the state machine is determined to be the normal tracking state; wherein, the deceleration window is the time threshold for entering the deceleration approach state; if the remaining trajectory time is greater than the control cycle and less than the deceleration window, and no new action block arrives, the state of the state machine is determined to be the deceleration approach state; if the remaining trajectory time is less than or equal to the control cycle, and no new action block arrives, the state of the state machine is determined to be the frozen waiting state.

[0066] This disclosure designs a three-stage state machine, where the state is determined by the remaining time of the trajectory. With two thresholds (deceleration window) and control cycle The size relationship is determined. The deceleration window is the time threshold for entering DECEL state, with a default value of 0.3s. The choice of 0.3s balances the following: 1) long enough to cover normal fluctuations in the inference pipeline (3 times the margin of 0.1s for a 10Hz cycle); 2) short enough to avoid premature deceleration affecting efficiency. This design allows the system to operate in the NORMAL state without loss of performance under 99% of normal conditions, only smoothly degrading to DECEL and FROZEN when there is a real delay in inference.

[0067] In some embodiments, if a new action block arrives, the state machine is determined to be in a NORMAL state, and the virtual time... Snap to real time immediately Seamlessly switch to the new track block.

[0068] In some embodiments, if The state of the state machine is determined to be NORMAL.

[0069] In some embodiments, when If no new chunk is created, the process will switch to DECEL mode.

[0070] In some embodiments, If no new chunk is available, the process will transition to the FROZEN state.

[0071] The mathematical models for the three states are explained below: 1) Mathematical model of NORMAL state Triggering condition: A new chunk arrives or .

[0072] Equations of state: , in, The start time offset of the current chunk, such that Within the chunk time interval [ , ]Inside.

[0073] Boundary conditions: When a new chunk arrives: virtual time is immediately snapped to real time. .

[0074] Remaining time decreased The following is when: Switch to DECEL mode.

[0075] Numerical stability: the difference between virtual time and real time (Single-cycle error), the accumulated error is periodically cleared to zero through snap operations.

[0076] 2) DECEL State Mathematical Model Triggering conditions: And there are no new chunks.

[0077] Equations of state: ,in, .

[0078] Discretization (control cycle) ): , Recurrence relation: Let Let the remaining time be the time for step k. Then: , Geometric series decreasing, common ratio .by , For example, ,half life .

[0079] Convergence proof: Virtual Time (Asymptotic convergence).

[0080] Boundary conditions: Remaining time decreased The following states: Transition to FROZEN state.

[0081] When a new chunk arrives: immediately return to NORMAL state and snap.

[0082] 3) FROZEN state mathematical model Triggering conditions: And there are no new chunks.

[0083] Equations of state: ,in, The time to enter FROZEN (auxiliary variable, used for timeout judgment).

[0084] Timeout detection: ,in, This is the hard timeout threshold (constant, default 5s).

[0085] Output attitude: Spline interpolation query The return value is constant, and the controller sends the same posture to the underlying SDK every cycle.

[0086] Boundary conditions: When a new chunk arrives: immediately return to the NORMAL state, snap virtual time, and reset it to zero. .

[0087] When timeout is triggered: Set all_finish_flag=True to control the loop exit.

[0088] In some embodiments, the arrival of a new chunk has a higher priority than the exit due to timeout; the exit due to timeout has a higher priority than the internal state transition.

[0089] According to some optional embodiments of this disclosure, the deceleration window is greater than or equal to the control period.

[0090] In embodiments of this disclosure, the deceleration window Forced to meet during configuration loading This can prevent the NORMAL range from jumping directly to FROZEN when the DECEL range is empty.

[0091] In some optional embodiments of this disclosure, the following robustness-enhancing processing schemes are also proposed for boundary conditions, for example, In cases where chunk time intervals overlap or jump, the snap operation is forcibly aligned each time a new chunk arrives, ignoring the unexecuted portion of the previous chunk and using the chunk as the reference.

[0092] For control frequency jitter (actual control cycle) In the case of non-constant time intervals, the virtual time increment uses the actual measured period (read from the configuration or measured in real time), rather than an assumed constant. This approach ensures that the state machine behavior remains consistent under ±10% frequency jitter.

[0093] In case of shared memory corruption or chunk ID rollback, a monotonicity check of the chunk ID is performed. If rollback occurs or the chunk ID is not found, None is returned, and the controller maintains the previous chunk or FROZEN state. This prevents virtual time misalignment caused by abnormal chunks.

[0094] This disclosure provides an optional implementation method in which step S103 is performed to determine the speed factor in the current state, including: if the state machine is in a normal tracking state, the speed factor is 1; if the state machine is in a deceleration approximation state, the speed factor is the ratio of the remaining trajectory time to the deceleration window; if the state machine is in a frozen waiting state, the speed factor is 0.

[0095] The virtual time progression speed cannot be abrupt (which would cause acceleration jumps and control jitter), nor can it decelerate too early (which would unnecessarily prolong the task time). A segmented and smooth progression strategy is required. The core challenge of the virtual time mechanism is how to design state division and transition conditions so that the virtual time follows the real time without delay under normal circumstances, decelerates smoothly when approaching the timeout, and stabilizes and clamps after reaching the end.

[0096] In the embodiments of this disclosure, in the NORMAL state, the velocity factor s=1, and the virtual time advances with the real time at a velocity factor s=1, that is... (Deducting the initial offset of the trajectory block). There is sufficient trajectory remaining at this stage, and no deceleration is required.

[0097] In DECEL mode, speed factor Virtual time is expressed in terms of speed factor Deceleration propulsion. This factor follows... Linearly decreasing, as the remaining time decreases from Down to As s smoothly decreases from 1 to near 0, virtual time asymptotically approaches... But don't cross over.

[0098] When the remaining trajectory is nearly exhausted and a new chunk has not yet arrived, virtual time cannot simply stop advancing (this would cause the attitude output to freeze and the robot to momentarily oscillate at zero speed), nor can it continue advancing at a 1:1 ratio (this would overshoot the end point and enter extrapolation). A deceleration strategy is needed so that virtual time approaches the end point infinitely within a finite real time without overshooting it, while maintaining continuous velocity and bounded acceleration.

[0099] This disclosure employs a linear deceleration law and a velocity factor. It is proportional to the remaining time. This deceleration law has the following mathematical properties: 1) Asymptotic convergence: Virtual time increment Remaining time after each step Updated to This forms a geometric progression of decreasing numbers. , For example, the remaining amount in each step is multiplied by The half-life is approximately 52 steps (0.2 seconds), and the virtual time asymptotically approaches the time. But theoretically it will never be reached (in reality, it will only reach its limit when the remaining amount decreases). (Transfer to FROZEN).

[0100] 2) Continuous speed: When transferring from NORMAL to DECEL correspond , with NORMAL state No jump; When transferring from DECEL to FROZEN correspond Close to FROZEN The speed transition is smooth.

[0101] 3) Acceleration is bounded: Rate of change of velocity factor s (because ), Substitute have to In the DECEL range , Maximum The corresponding virtual acceleration is bounded and will not produce abrupt changes.

[0102] Mathematical basis: Let the time of entry into DECEL be . ,time left Then the differential equation is: , Solution The exponential decays to 0. Virtual time ,when hour, (Approaching gradually).

[0103] Linear deceleration law It is the simplest deceleration strategy that satisfies asymptotic convergence, continuous velocity, and bounded acceleration. Compared to fixed deceleration (which requires pre-calculation of the stopping distance and is prone to overshoot) or quadratic deceleration (which has complex parameters), linear law is simple to implement, robust, and naturally exhibits a conservative behavior of "the less residual, the slower." The measured DECEL state duration is typically less than 0.2s (the inference delay arrives within the deceleration window), and this deceleration law maintains smooth attitude output and imperceptible velocity jumps within this timescale.

[0104] In FROZEN state, the velocity factor s=0, and the virtual time clamp is in... Spline interpolation always returns the end pose. The controller repeatedly outputs this posture. The robot remains at the end effector waiting for a new chunk, without shaking or outward divergence.

[0105] When virtual time reaches the end of the trajectory When no new chunk is available, the controller cannot stop outputting the attitude (which would cause a heartbeat interruption with the underlying SDK), nor can it extrapolate the trajectory (which would lead to attitude divergence and unpredictable behavior). A "safe stop" mechanism is needed to allow the robot to remain in the end-effector's attitude while waiting for new instructions, while maintaining control loop operation and a normal heartbeat.

[0106] This disclosure, in FROZEN state, virtual time clamping is... ( (velocity factor s=0), spline interpolation query Always returns the end pose of the trajectory block. The pose instruction sequence received by the underlying SDK is as follows: , , ..., the robot remains stationary in that posture.

[0107] Mathematical foundation: The virtual time evolution in the FROZEN state is as follows: , in This is the moment of entering FROZEN. The spline interpolation output is always... End attitude The value is determined during trajectory block generation and is the interpolation of the last action point of that block. The controller output frequency remains 250Hz, but the output content does not change with time, equivalent to a steady-state point with zero velocity and zero acceleration.

[0108] The end-effector attitude clamping mechanism ensures that the robot's behavior in the FROZEN state is completely deterministic and safe: attitude output is continuous (no jumps); velocity is zero (no drift); heartbeat with the underlying SDK is maintained (control loop is uninterrupted); waiting time is bounded (hard timeout fallback). Compared to simple shutdown waiting (heartbeat interruption, SDK error) or extrapolation trajectory (divergence risk), the clamping mechanism provides the most conservative safe behavior while maintaining system operation. In practice, the duration of the FROZEN state is typically less than 1 second (inference pipeline recovery or user interruption), and the mechanism performs stably within this timescale.

[0109] In the embodiments of this disclosure, the three-stage state machine is naturally divided by two-level thresholds for the remaining trajectory time. NORMAL ensures zero latency, DECEL ensures smooth deceleration, and FROZEN ensures stable clamping, with continuous state transitions without jumps. Compared to fixed timeout hard stops or uniform speed extrapolation, this disclosure achieves a three-stage smooth transition through virtual time decoupling: in the normal state, virtual time is synchronized with real time 1:1 with no additional delay; when approaching the timeout, virtual time automatically decelerates, asymptotically approaching the end without overshooting, giving the inference pipeline more time windows; upon reaching the end, virtual time clamps and repeatedly outputs the end pose, allowing the robot to smoothly stop rather than jitter or diverge; when a new chunk arrives, the snap immediately returns to normal, seamlessly connecting. This mechanism is entirely driven by time geometry and does not rely on signal fusion or external sensing.

[0110] According to some optional embodiments of this disclosure, step S104, which updates the virtual time based on the speed factor, includes the following steps: if the state machine is in a normal tracking state or a deceleration approach state, the virtual time is updated using the virtual time increment, and the virtual time is less than or equal to the end time of the current action block; wherein, the virtual time increment is the product of the speed factor and the control cycle; if the state machine is in a frozen waiting state, the virtual time is equal to the end time of the current action block.

[0111] In each control cycle, the virtual time increment is s is the velocity factor.

[0112] In this embodiment, the method for updating the virtual time according to the velocity factor is as follows: NORMAL or DECEL status: Clamping guarantee .

[0113] FROZEN status: It remains unchanged.

[0114] Next, using virtual time The current end-effector attitude can be obtained by querying the spline interpolation value. It is then sent to the underlying robot SDK for execution.

[0115] This disclosure also provides an optional implementation method, which determines the time to enter the freeze waiting state; if the difference between the actual time and the time to enter the freeze waiting state is greater than a preset hard timeout threshold, the control loop is exited.

[0116] Virtual time decoupling mechanisms smooth out inference latency using DECEL or FROZEN, but if the inference pipeline completely fails (model crash, network disconnection, input stream interruption) resulting in no new chunks for a long time, the system cannot wait indefinitely and needs a safety fallback mechanism to detect anomalies and exit.

[0117] This disclosure introduces a hard timeout in FROZEN state. (Default 5s) Records the real time 'frozen_since' when entering FROZEN. Each control cycle checks the difference between the current real time 'real_time' and 'frozen_since'. If 'real_time - frozen_since' > 0... The system determines that the inference pipeline has failed, sets the global flag 'all_finish_flag=True', and exits the control loop via `break`. After exiting, the main loop checks this flag and prints the error message "Error: VLA timeout, no new trajectory!!!", and the system stops execution. This hard timeout is the last line of defense, ensuring that the system does not enter an uncontrollable state due to inference failure.

[0118] Mathematical foundation: set up The timeout condition for entering the real moment of FROZEN is: ; The choice of timeout needs to balance security and fault tolerance: too short (e.g., 1s) may trigger false alarms under normal but slightly slower inference fluctuations, while too long (e.g., 30s) will delay the system response when a real failure occurs. The default of 5s is based on the following considerations: 1) The normal inference cycle is 0.1s, and even if a complex scenario causes a single inference to take 1s, there is sufficient margin in the 0.3s DECEL window + FROZEN wait; 2) 5s is long enough to cover short interruptions such as network reconnection and model warmup; 3) 5s is short enough to stop losses in time when a real failure occurs, avoiding long waiting times for users.

[0119] The hard timeout mechanism is the safety baseline of the virtual time decoupling system, ensuring that the system can exit instead of suspending indefinitely when the inference pipeline fails. The 5-second timeout threshold performed well in real-world testing: 1) it was never falsely triggered during normal operation (FROZEN state typically recovers within <1 second); 2) it can promptly detect and exit when the inference pipeline truly fails (e.g., model process is killed, network connection is lost). This mechanism complements the smooth transition of DECEL / FROZEN, with the former handling short delays (<0.5s) and the latter handling long-term anomalies (>5s), together forming a complete fault-tolerant and safety system.

[0120] According to some optional embodiments of this disclosure, determining the velocity factor in the current state further includes: if the state of the state machine is a deceleration approximation state, the velocity factor is determined by a nonlinear curve; or ; Where s is the velocity factor, The remaining time for the trajectory. For deceleration window, This refers to the sigmoid function.

[0121] In some embodiments, a nonlinear deceleration curve, such as a quadratic curve, can be used. or sigmoid function .

[0122] Nonlinear deceleration curves are suitable for scenarios that require higher smoothness of acceleration (such as high-speed operation and heavy load).

[0123] The quadratic curve decelerates faster near the end, reducing FROZEN waiting time; the sigmoid curve is smoother at the transition point.

[0124] According to some optional embodiments of this disclosure, the deceleration window can be dynamically adjusted based on historical inference delay statistics.

[0125] In some embodiments, adjustments can be made dynamically based on historical inference delay statistics. ,like , where EMA is the exponential moving average.

[0126] The adaptive deceleration window is suitable for systems where the inference delay distribution varies significantly with the scenario (e.g., slow inference in complex scenarios and fast inference in simple scenarios).

[0127] The advantage of adaptive deceleration windows is that they reduce latency in aggressive scenarios (low latency). Improve efficiency and increase efficiency in conservative scenarios (high latency). Slow down in advance.

[0128] This disclosure provides an optional implementation where the frozen waiting state includes multiple sub-states; in different sub-states, the virtual time clamp time progressively approaches the end time of the current action block.

[0129] In this embodiment, a multi-level FROZEN state is designed, such as FROZEN-1 (clamped in...). , ), FROZEN-2 (clamped in) (and) gradually approach the end.

[0130] Multi-level FROZEN states are suitable for scenarios where slight movement needs to be maintained during the FROZEN waiting period (such as contact holding, force control, etc.). They can avoid sudden changes in friction or fluctuations in contact force caused by complete stillness, and maintain slight "exploratory" movement.

[0131] The virtual time decoupling mechanism and three-stage state machine proposed in this disclosure produce the following beneficial effects in practical VLA robot systems: Effect 1: Eliminates execution jitter caused by inference delay, significantly reduces mechanical shock, and makes robot movement more natural and smooth.

[0132] Effect 2: Ensures the accuracy of posture during fine operations.

[0133] Effect 3: Improves the system's tolerance to inference latency. The three-stage state machine smoothly covers the entire range of zero latency (NORMAL), short latency (0-0.3s, DECEL), and long latency (0.3-5s, FROZEN), with a 5s hard timeout providing a safety net. Occasional long-tail latency in the inference pipeline (such as in complex scenarios or network jitter) no longer causes immediate task failure.

[0134] Effect 4: Reduced computational overhead and inference computational power consumption; virtual time advancement requires only one floating-point subtraction. One conditional judgment, one multiplication No trajectory prediction model is required.

[0135] Effect 5: Plug-and-play compatibility and black-box policy compatibility. The virtual timing mechanism is implemented entirely within the controller, with zero coupling to the inference pipeline, and only reads chunks via shared memory. Existing VLA systems can be seamlessly upgraded without damaging existing training data and model checkpoints.

[0136] Effect 6: Provides deterministic failure detection and safe exit, avoiding the accumulation of task failures or user waiting caused by long-term suspension, and facilitating rapid intervention by the upper-level scheduling system.

[0137] This disclosure decouples virtual time from real time, ensuring smooth execution (effects 1 and 2) while significantly improving the system's tolerance to inference latency (effect 3), reducing computational overhead (effect 4), maintaining plug-and-play functionality (effect 5), and providing deterministic safety fallback (effect 6). These six effects work together to make the VLA robot system both efficient and robust in its asynchronous architecture characterized by slow inference and fast execution.

[0138] The method proposed in this disclosure will be described below with reference to a specific embodiment: 1. t=0: Controller starts, initializing virtual time. No track block (action=None); 2. t=0.1s: The inference process produces the first chunk (chunk id=1) and writes it to shared memory; 3. t=0.104s (control cycle is 4ms): When the controller polls for a new chunk in the next cycle, the virtual time snaps to the real time. It enters the NORMAL state; 4. t=0.104~0.2s: Virtual time progresses 1:1 with real time, and spline interpolation is performed along chunk 1; 5. t=0.2s: The inference process produces chunk 2, which the controller reads in the next cycle (t=0.204s). The virtual time is snapped to 0.204, and the controller switches to chunk 2. 6. t=0.3s: Abnormal scenario—the inference process failed to produce chunk 3 on time due to a computational bottleneck; 7. t=0.3~0.4s: The virtual time continues to advance along chunk 2, and the remaining time is at t=0.38s. Upon entering DECEL mode, virtual time begins to slow down; 8. t=0.4~0.45s: DECEL state continues, virtual time asymptotically approaches the end of chunk 2, and the rate factor decreases from 1 to 0.013; 9. t=0.45s: The virtual time reaches the end of chunk 2, enters the FROZEN state, and records frozen_since=0.45; 10. t=0.45~0.52s: FROZEN state waiting, the controller repeatedly outputs the end pose of chunk 2 (once every 4ms), and the robot remains in this pose; 11. t=0.52s: The inference process resumes, produces chunk 3 (delay 0.22s), the controller reads it in the next cycle, the virtual time is snapped to 0.524, and returns to the NORMAL state, frozen_since=None; 12. t>0.52s: Normal recovery execution.

[0139] If the inference process does not resume in step 11, FROZEN continues until t=5.45s (0.45+5.0) triggers a hard timeout, the controller sets all_finish_flag=True and exits.

[0140] The table below provides recommended ranges for the values ​​of some key parameters: Table 1

[0141] Recommended configuration (based on actual testing): Lightweight scenarios (simple crawling, contactless): , Aggressive configuration to minimize average task time.

[0142] Standard scenario (general operation): , Balanced configuration (default in the code).

[0143] Complex scenarios (contact, assembly): , Conservative configuration, tolerating long-tail delays in inference.

[0144] This disclosure provides a robot control device, such as... Figure 2 As shown, the robot's control device 20 may include: The first determining module 201 is used to read the action blocks generated by the inference process from the shared memory in each control cycle, and determine whether a new action block has arrived in the action block according to the action block identifier; wherein, the action block includes a number of action points arranged in chronological order and their spline interpolation coefficients.

[0145] The second determining module 202 is used to determine the state of the state machine based on the arrival state of the new action block and the remaining time of the robot's trajectory; wherein, the state includes: normal tracking state, deceleration approach state and frozen waiting state; the remaining time of the trajectory is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller.

[0146] The third determining module 203 is used to determine the velocity factor in the current state; wherein, the velocity factor is the advancement ratio of virtual time relative to real time.

[0147] The processing module 204 is used to update the virtual time according to the velocity factor, and use the virtual time to query the spline interpolation to obtain the robot's end pose; and control the robot according to the obtained end pose.

[0148] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this disclosure correspond to the steps in the method of each embodiment of this disclosure. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, and it will not be repeated here.

[0149] This disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this disclosure. Compared with the prior art, it achieves the following: by using a virtual time decoupling mechanism and a three-stage state machine, it eliminates execution jitter caused by inference latency, ensures the posture accuracy of fine operations, improves the system's tolerance to inference latency, reduces computational overhead and inference power consumption, and makes the robot system both efficient and robust under the asynchronous architecture of "slow inference, fast execution".

[0150] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 3000 includes a processor 3001 and a memory 3003. The processor 3001 and the memory 3003 are connected, for example, via a bus 3002. Optionally, the electronic device 3000 may further include a transceiver 3004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 3004 is not limited to one type, and the structure of the electronic device 3000 does not constitute a limitation on the embodiments of this disclosure.

[0151] Processor 3001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein. Processor 3001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0152] Bus 3002 may include a pathway for transmitting information between the aforementioned components. Bus 3002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 3002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0153] The memory 3003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0154] The memory 3003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 3001 to execute them. The processor 3001 is used to execute the computer programs stored in the memory 3003 to implement the steps shown in the foregoing method embodiments.

[0155] Electronic devices include, but are not limited to, computer equipment.

[0156] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0157] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0158] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0159] The above description is only an optional implementation method for some implementation scenarios of this disclosure. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this disclosure without departing from the technical concept of this disclosure also fall within the protection scope of the embodiments of this disclosure.

Claims

1. A method for controlling a robot, characterized in that, include: In each control cycle, action blocks generated by the inference process are read from shared memory, and it is determined whether a new action block has arrived in the action block based on the action block identifier; wherein, the action block includes a number of action points arranged in chronological order and their spline interpolation coefficients; The state of the state machine is determined based on the arrival state of the new action block and the remaining trajectory time of the robot; wherein, the state includes: normal tracking state, deceleration approach state, and frozen waiting state; the remaining trajectory time is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller; the frozen waiting state includes multiple levels of sub-states; in different levels of sub-states, the virtual time clamping time gradually approaches the end time of the current action block; Determine the velocity factor in the current state; wherein, the velocity factor is the advancement ratio of the virtual time relative to the real time; The virtual time is updated according to the velocity factor, and the end-effector pose of the robot is obtained by querying spline interpolation using the virtual time; the robot is controlled according to the obtained end-effector pose.

2. The method according to claim 1, characterized in that, Determining the state of the state machine based on the arrival state of the new action block and the remaining time of the robot's trajectory includes: If a new action block arrives or the remaining time of the trajectory is greater than or equal to the deceleration window, the state of the state machine is determined to be the normal tracking state; wherein, the deceleration window is the time threshold for entering the deceleration approximation state; If the remaining time of the trajectory is greater than the control period and less than the deceleration window, and no new action block arrives, the state of the state machine is determined to be the deceleration approximation state; If the remaining time of the trajectory is less than or equal to the control cycle, and no new action block arrives, the state machine is determined to be in the frozen waiting state.

3. The method according to claim 2, characterized in that, The deceleration window is greater than or equal to the control period.

4. The method according to claim 2, characterized in that, The determination of the velocity factor in the current state includes: If the state of the state machine is the normal tracking state, the speed factor is 1; If the state of the state machine is the deceleration approximation state, the speed factor is the ratio of the remaining trajectory time to the deceleration window; If the state machine is in the frozen waiting state, the speed factor is 0.

5. The method according to claim 2, characterized in that, Updating the virtual time according to the velocity factor includes: If the state machine is in the normal tracking state or the deceleration approximation state, the virtual time is updated using the virtual time increment, and the virtual time is less than or equal to the end time of the current action block; wherein, the virtual time increment is the product of the speed factor and the control cycle; If the state machine is in the frozen waiting state, the virtual time is equal to the end time of the current action block.

6. The method according to claim 1, characterized in that, The step of determining whether a new action block has arrived in the action block based on the action block identifier includes: Record the action block identifiers of the action blocks read in each control cycle in a monotonically increasing order; If the action block identifier of the action block read in the current control cycle is greater than the action block identifier of the action block read in the previous control cycle, it is determined that the new action block has arrived.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Determine the time to enter the frozen waiting state; If the difference between the actual time and the time of entering the frozen waiting state is greater than a preset hard timeout threshold, the control loop is exited.

8. The method according to claim 2, characterized in that, The determination of the velocity factor in the current state also includes: If the state of the state machine is the deceleration approximation state, the speed factor is determined by a nonlinear curve; or ; Where s is the velocity factor, The remaining time for the trajectory. For the deceleration window, This refers to the sigmoid function.

9. The method according to claim 2, characterized in that, The deceleration window can be dynamically adjusted based on historical inference delay statistics.

10. A control device for a robot, characterized in that, include: The first determining module is used to read the action blocks generated by the inference process from the shared memory in each control cycle, and determine whether a new action block has arrived in the action block according to the action block identifier; wherein, the action block includes a number of action points arranged in chronological order and their spline interpolation coefficients; The second determining module is used to determine the state of the state machine based on the arrival state of the new action block and the remaining trajectory time of the robot; wherein, the state includes: normal tracking state, deceleration approach state, and frozen waiting state; the remaining trajectory time is the difference between the end time of the current action block and the virtual time; the virtual time is a time variable maintained internally by the controller; the frozen waiting state includes multiple levels of sub-states; in different levels of sub-states, the virtual time clamping time gradually approaches the end time of the current action block; The third determining module is used to determine the speed factor in the current state; wherein, the speed factor is the advancement ratio of the virtual time relative to the real time; The processing module is used to update the virtual time according to the velocity factor, and use the virtual time to query spline interpolation to obtain the end-effector posture of the robot; and control the robot according to the obtained end-effector posture.

11. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the method of any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

Images