A robot VLA cooperative control method based on space-time enhancement and trajectory smoothing

Abstract

The application relates to a robot VLA cooperative control method based on space-time enhancement and trajectory smoothing. The method comprises the following steps: receiving and preprocessing RGB image data, natural language instructions and robot body state data, and generating action query features; a dynamic space-time topology atlas is constructed in the original 2D feature space, feature interaction and implicit geometric deduction are carried out through a graph neural network, and visual enhancement features are output; the action query features are used as a guide, a bridge attention mechanism is used to perform nonlinear mapping and feature fusion on the visual enhancement features; the fused feature flow is input into a preset strategy network, original discrete action blocks containing the length of the future time domain field of view are decoded and output; a continuous physical manifold space containing the physical limit of the robot is defined; a total energy loss function for balancing semantic reservation and physical smoothing is constructed, the original discrete action blocks are projected into the continuous physical manifold space for optimization, smooth action signals are generated and are executed.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and specifically to a robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing. Background Technology

[0002] With the development of multimodal artificial intelligence, Vision-Language-Motion (VLA) models have become one of the core technologies for achieving intelligent operation of robots. This technology is used to build an end-to-end control system that can directly fuse and understand visual observations from cameras, natural language commands from operators, and the robot's own state, and output the underlying motion control sequence. By pre-training on large-scale internet data and fine-tuning on robot teaching data, such models can transfer powerful scene understanding and semantic reasoning capabilities to physical control, thereby handling open and diverse tasks. However, when tasks require extremely high spatial operation accuracy or highly fluid dynamic execution, how to efficiently and accurately extract geometric information from visual input and generate physically feasible, continuous, and smooth control trajectories has become a key problem to be solved in the application of VLA technology.

[0003] In existing technologies, robot VLA control technology mainly employs the following two methods to address tasks requiring precise spatial perception: First, it utilizes a conditional policy network (CPR) based on an attention mechanism. This CPR can deeply integrate high-level semantic features extracted from a pre-trained visual-language model with language commands and directly map them into control actions, thus enabling generalization of open-ended tasks based on natural language to a certain extent. Second, it adopts a technical solution that introduces an explicit geometric understanding module, such as real-time 3D scene reconstruction or relying on a large language model to output text coordinates describing the position of objects. This provides the control system with clear spatial reference information, enabling direct spatial pointing and interaction with the target to a certain extent.

[0004] However, the aforementioned existing technology has defects, for the following reasons.

[0005] In terms of spatial perception and positioning, schemes that focus on high-level semantic fusion lack the ability to explicitly reason about the underlying geometric features, resulting in insufficient spatial positioning accuracy in precision operations. On the other hand, schemes that rely on additional 3D reconstruction or text coordinate transformation suffer from high computational complexity, introduce significant delays, and have problems with spatial description coordinate errors due to the "big model illusion," making it difficult to simultaneously meet the dual requirements of real-time control and millimeter-level operational accuracy.

[0006] In terms of action generation and execution, existing solutions typically decompose long-term tasks into discrete short-cycle action sequences for processing, which disrupts the continuity of states between actions. This results in sudden changes in velocity and acceleration at the splicing points of the generated instructions. In real physical systems, this discontinuity can cause severe intermittent mechanical jitter and may generate unsafe instructions that exceed the physical limits of the actuator. This not only affects the smoothness and success rate of dynamic tasks but may also exacerbate mechanical wear, damage hardware lifespan, and impair system robustness. Summary of the Invention

[0007] This invention can effectively solve the inherent contradiction between spatial perception accuracy and motion execution smoothness in existing robot VLA control frameworks, and has the advantages of high precision, low latency and high smoothness; it is conducive to enabling robots to reliably execute long sequences of high-precision tasks in complex dynamic environments; and it can significantly improve the deployment feasibility of robots in real-world scenarios.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention provides a robot VLA cooperative control method based on spatiotemporal enhancement and trajectory smoothing, the method comprising:

[0010] S1. Receive and preprocess RGB image data, natural language commands, and robot body state data; generate action query features based on the preprocessed natural language commands and robot body state data; construct a dynamic spatiotemporal topology map in the native 2D feature space, and perform feature interaction and implicit geometric inference through graph neural networks to output visually enhanced features with depth perception capabilities.

[0011] S2. Guided by the action query features, the visual enhancement features are nonlinearly mapped and fused using a bridging attention mechanism; the fused feature stream is input into a preset policy network, and the decoded output contains the original discrete action blocks containing the future temporal field of view length.

[0012] S3. Define a continuous physical manifold space that includes the robot's physical limits; construct a total energy loss function to balance semantic preservation and physical smoothing; project the original discrete actions into the continuous physical manifold space for optimization; generate smooth action signals and send them out for execution.

[0013] As a preferred embodiment of the present invention, in S1, the method for generating the action query features is specifically as follows:

[0014] The natural language instructions are embedded and encoded, and the robot's body state data is standardized and linearly mapped; the two are then combined and fused using a multilayer perceptron to generate the action query features.

[0015] As a preferred embodiment of the present invention, S1 includes the following sub-steps:

[0016] S11. The RGB image data is processed into a continuous T-frame RGB image sequence within a target time sliding window, and each frame image is normalized; based on a visual language model, a multi-layer original visual word sequence is extracted from the RGB image sequence; the multi-layer original visual word sequence contains multiple visual words.

[0017] S12. Guided by the action query features, the visual lexical units are defined as topological nodes, and based on the feature similarity between each topological node, spatial edges within the same time frame and temporal edges across adjacent time frames are constructed respectively.

[0018] S13. Calculate the dynamic adjacency matrix based on the spatial edge, the temporal edge, and the preset language intent weight;

[0019] S14. Based on the dynamic adjacency matrix, feature propagation and iterative updates are performed through multi-layer graph convolution to output the visual enhancement features.

[0020] As a preferred embodiment of the present invention, in S2, the policy network is constructed based on the Transformer architecture.

[0021] As a preferred embodiment of the present invention, in S3, the total energy loss function includes a first term for ensuring that the action direction does not deviate from the original discrete action block, and a second regularization penalty term for limiting abrupt changes in trajectory acceleration.

[0022] As a preferred embodiment of the present invention, the second regularization penalty term is obtained by integrating the urgency of the action command over time.

[0023] As a preferred embodiment of the present invention, in S3, the method for defining the continuous physical manifold space is specifically as follows:

[0024] Obtain the underlying physical limit parameters of the physical robot, which include the maximum joint rotation speed, the maximum joint acceleration, and the range of configuration singularities;

[0025] The set of all continuous robot motion commands that satisfy the following constraints is defined as a low-dimensional continuous physical manifold space: at any time, the corresponding joint rotation speed is less than or equal to the maximum joint rotation speed, the corresponding joint acceleration is less than or equal to the maximum joint acceleration, and the configuration is not located within the range of configuration singularities.

[0026] As a preferred embodiment of the present invention, in S3, the specific steps for optimizing the projection of the original discrete action block onto the continuous physical manifold space are as follows: taking the original discrete action block as the initial value, and under the constraint of the total energy loss function, performing forward projection calculation to generate a smooth action signal.

[0027] In a second aspect, the present invention provides an electronic device, including a processor and a memory;

[0028] The processor is connected to the memory;

[0029] The memory is used to store executable program code;

[0030] The processor reads the executable program code stored in the memory to run the program corresponding to the executable program code, in order to execute the steps of the aforementioned robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing.

[0031] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing as described above.

[0032] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0033] 1. This invention achieves accurate and efficient reasoning of underlying spatial relationships by constructing a dynamic spatiotemporal topological map and performing implicit geometric deduction in the native 2D feature space. Compared with the high latency and positioning deviation caused by relying on high-energy-consuming 3D reconstruction or error-prone text coordinate transformation in the prior art, this invention can complete the dimensional understanding at the 2D level, and has the advantages of low reasoning latency and complete avoidance of model illusion.

[0034] 2. This invention optimizes trajectory smoothing by introducing physical prior constraints, projecting the original discrete motion blocks onto a continuous physical manifold space defined by joint limits for optimization. This achieves physical feasibility correction of the motion trajectory. Compared with existing technologies that suffer from physical discontinuity and mechanical jitter in motion sequences due to long task segmentation, this invention can forcibly eliminate non-physical instantaneous jumps while preserving the high-frequency intent of the task. It has the advantage of balancing semantic fidelity and execution smoothness, which is beneficial for completing dynamic tasks with extremely high requirements for motion continuity.

[0035] 3. Through the coordinated operation of the aforementioned spatial enhancement and trajectory smoothing components, this invention achieves deep coupling between the perception and control links. Compared with the system fragility caused by the relatively fragmented perception, planning, and control modules or the use of post-event remedial smoothing in the prior art, the solution of this invention can ensure continuous spatial tracking when objects are dynamically moving or partially occluded, and ensure that the output command is always within the safe working envelope of the actuator, effectively avoiding instantaneous overload and mechanical wear of the motor, and greatly improving the success rate of the robot in real industrial and home environments. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of the method of the present invention;

[0038] Figure 2 The flowchart for VLA collaborative control in this embodiment is shown. Detailed Implementation

[0039] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0040] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0041] like Figure 1 and Figure 2As shown, this robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing includes at least the following steps:

[0042] S1. Receive and preprocess RGB image data and natural language commands. Robot body state data Based on preprocessed natural language instructions Robot body state data Generate action query features A dynamic spatiotemporal topological map is constructed within the native 2D feature space, and feature interaction and implicit geometric inference are performed through graph neural networks to output visually enhanced features with depth perception capabilities. .

[0043] Understandably, the purpose of this step is to abandon high-latency 3D reconstruction or unreliable text coordinate descriptions. Within an efficient native 2D visual feature space, a dynamic spatiotemporal topological map guided by task intent is constructed to achieve accurate reasoning about the geometric structure and motion trends of the target object. This provides enhanced visual features with depth perception capabilities for subsequent action generation. .

[0044] Furthermore, this step first receives RGB image data and natural language instructions. Robot body state data ; where RGB image data comes from the input of the vision system; natural language instructions Operator input from the interactive interface; robot body state data Data from the robot's low-level sensors (such as encoders and inertial measurement units) contains information such as joint angles and end-effector pose. RGB image data has continuity and temporal sequence, containing spatiotemporal information about the robot's operational target, and natural language commands. The high-level mission objectives and ontology state data are specified. This reflects the robot's real-time configuration. Subsequently, unified preprocessing was performed to ensure the consistency of scale across different modalities, laying a reliable foundation for subsequent feature extraction and fusion.

[0045] Further, image segmentation is performed. A sequence of T consecutive RGB image frames within a time-sliding window is acquired. ,in And normalize each frame.

[0046] At the same time, it integrates commands with the ontology state, and incorporates natural language commands. Perform text embedding encoding to convert the robot's body state data Standardization and linear mapping are performed, and then the two are concatenated and fused using a multilayer perceptron to generate action query features that are highly relevant to the current task objective. Action query features It integrates task instructions and robot status, forming a dynamic, task-related guidance signal; action query features. The guiding actions can make subsequent spatial reasoning no longer a blind general calculation, but a goal-oriented process that can focus on the visual areas and geometric properties most relevant to the task.

[0047] Then, a pre-trained vision-language model is used to analyze the RGB image sequence. Extracting multi-layered original visual word sequences The multi-layered original visual lexical sequence contains n visual lexical units; these visual lexical units are abstract feature representations of the image in the middle layer of the model, preserving rich spatial and semantic information.

[0048] Next, using action query features To guide this process, visual lexical units in the native 2D feature space are defined as topological nodes, which are the foundational material for subsequent graph construction.

[0049] In the same time frame Within, calculate any two nodes and The feature distance and cosine similarity between them are considered. If the cosine similarity is greater than a set threshold, the result is considered. Then construct spatial edges :

[0050]

[0051] in, This is the node feature similarity function. Spatial edges are used to capture the static geometry of the target (such as the outline of an object, the connection relationships between parts).

[0052] Adjacent time frames within a time sliding window (e.g.) and Between those with the same spatial coordinates or extremely high semantic cosine similarity (greater than a set threshold), the similarity is determined based on whether they have the same spatial coordinates or extremely high semantic cosine similarity (greater than a set threshold). ) nodes and Constructing time edges :

[0053]

[0054] The time edge is used to associate the state of a target at different times, thereby capturing its motion.

[0055] Then, language intent weights are introduced to dynamically recalibrate the graph connections, and the action query features of each node are calculated. The degree of impact; specifically, defining the nodes. Subject to action query features Influence on attention weights and nodes Subject to action query features Influence on attention weights .

[0056] Subsequently, by combining spatial and temporal edges, a dynamic adjacency matrix containing language intent weights is calculated. Its matrix elements Defined as:

[0057]

[0058] in, For connectivity masking functions, This is the node feature similarity function. The element values ​​of this matrix comprehensively reflect the strength of the geometric association between nodes and their relevance to the current task.

[0059] Then, implicit geometric deduction is performed, using the aforementioned dynamic adjacency matrix. As a graph structure, the original visual lexical units are used as initial node features, and feature interaction propagation and iterative updates are performed through multi-layer graph convolution:

[0060]

[0061] in, , It is an identity matrix used to add self-connections to graph nodes; For degree matrix, Indicates the first The hidden layer node feature matrix of the layer; Indicates the first Layer-learnable network weight matrix.

[0062] Let the total number of layers in the graph convolutional network be... The original visual lexical units are Nodes exchange information through edges, passing through After the first interactive propagation, the output of the final layer feature matrix is... This refers to the final visual enhancement features acquired that possess depth perception capabilities. .

[0063] As demonstrated by the steps above, by explicitly establishing spatiotemporal topological relationships and performing graph reasoning at the 2D feature layer, this method can implicitly deduce the target's 3D geometric attributes such as depth and motion vectors without requiring explicit and time-consuming 3D voxel reconstruction. Compared to methods that rely on large models to output text coordinates, this approach fundamentally eliminates coordinate deviations caused by model illusions, and the entire process is completed in 2D space, resulting in extremely low computational latency (down to the millisecond level).

[0064] S2, query features based on actions To guide the use of bridging attention mechanisms for visual enhancement features Perform nonlinear mapping and feature fusion; input the fused feature stream into a predefined policy network, and decode to output the original discrete action blocks containing the future temporal field of view length. .

[0065] This step is responsible for transforming the enhanced perceptual features into preliminary action planning. Its goal is to generate a raw action sequence that is semantically consistent with the task instructions but has not yet taken into account physical smoothness.

[0066] Specifically, a bridging attention mechanism is used to query action features. To guide, visual enhancement features Nonlinear mapping and feature fusion are performed; the bridging attention mechanism enables the model to dynamically enhance visual features according to task intent. Select the visual context information that is most relevant to the current action decision.

[0067] Next, the fused feature stream is input into a policy network built on a Transformer architecture; the network decodes and outputs the original discrete action blocks containing the future temporal view length. Original discrete action block This refers to the sequence of actions within a future time domain.

[0068] This step completes the transition from perception to preliminary planning. The generated raw discrete action blocks... In terms of high-level semantics, it is similar to natural language instructions. Maintaining consistency encompasses the basic intentions and direction of actions required to complete the task.

[0069] However, since policy networks are typically optimized for short horizons and long tasks are decomposed, these original discrete actions are divided into blocks. At the time splicing points, there are often abrupt changes in speed and acceleration, i.e., physical discontinuities, which are the direct cause of mechanical jitter and execution failure.

[0070] S3. Define a continuous physical manifold space that includes the physical limits of the robot. Construct a total energy loss function to balance semantic preservation and physical smoothing. Divide the original discrete actions into blocks Projected onto continuous physical manifold space Optimize to generate smooth motion signals And it was issued and implemented.

[0071] This step divides the original discrete actions into blocks as described above. The purpose of this input is to eliminate the physical discontinuity of the original actions described above.

[0072] Specifically, the underlying physical limit parameters of the physical robot are obtained in advance. In this embodiment, the underlying physical limit parameters include the maximum rotational speed of each joint. Maximum joint acceleration And the range of configuration singularities; in another possible embodiment, other underlying physical limit parameters may also be included.

[0073] The set of all continuous robot motion commands that satisfy the following constraints is defined as a low-dimensional continuous physical manifold space. At any given moment, the corresponding joint rotational speed is less than or equal to the maximum joint rotational speed. The corresponding joint acceleration values ​​are all less than or equal to the maximum joint acceleration. Furthermore, its configuration is not located within the range of configuration singularities.

[0074] This step explicitly defines the set of safe and feasible actions. Continuous physical manifold space. This ensures that any action command falling within it will not exceed the speed limit, overload, or enter a singular configuration, thus providing a clear physical constraint boundary for subsequent trajectory optimization.

[0075] Next, a total energy loss function is constructed. :

[0076]

[0077] The first term is the original discrete action block used to ensure that the optimized action 'a' does not deviate from the output of the large model. The first term is the reference direction; the second term is the introduced regularization penalty term, which is used to limit abrupt acceleration changes in the trajectory by integrating over time t to obtain the jerkiness.

[0078] Total energy loss function This approach aims to balance semantic fidelity and physical smoothing. Compared to existing techniques' post-processing blind smoothing, this scheme uses a different total energy loss function. The optimization process can be guided to actively eliminate the jerks that cause jitter (second item) while remaining faithful to the original task intent (first item).

[0079] Then, the original discrete actions are divided into blocks. As initial values, in the total energy loss function Under the constraints, forward projection calculations are performed to generate smooth motion signals. :

[0080]

[0081] This optimization step is used to forcibly pull discrete transition points that have exceeded physical limits back into the continuous physical manifold space while preserving the semantic direction of the action. This eliminates the physical discontinuities at the junctions of discrete blocks, ensuring that the trajectory never crosses the boundary.

[0082] Finally, the optimized smooth motion signal The pulse control commands are converted into pulses for the robot's servo motors and distributed to the actuator interface in real time to complete the final operation.

[0083] In another possible embodiment, the present invention also provides an electronic device including a processor and a memory;

[0084] The processor is connected to the memory;

[0085] The memory is used to store executable program code;

[0086] The processor reads the executable program code stored in the memory to run the program corresponding to the executable program code, in order to execute the steps of the aforementioned robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing.

[0087] In another possible embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing as described above.

[0088] Several embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing, characterized in that the method... include: S1. Receive and preprocess RGB image data, natural language commands, and robot body state data; Action query features are generated based on preprocessed natural language instructions and robot body state data. The specific method for generating these action query features includes: text embedding encoding of the natural language instructions and standardization and linear mapping of the robot body state data; merging the two data using a multilayer perceptron to generate the action query features; constructing a dynamic spatiotemporal topology map within the native 2D feature space, and performing feature interaction and implicit geometric inference through a graph neural network to output visually enhanced features with depth perception capabilities; wherein, S1 includes the following sub-steps: S11. The RGB image data is processed into a continuous T-frame RGB image sequence within a target time sliding window, and each frame image is normalized; based on a visual language model, a multi-layer original visual word sequence is extracted from the RGB image sequence; the multi-layer original visual word sequence contains multiple visual words. S12. Guided by the action query features, the visual lexical units are defined as topological nodes, and based on the feature similarity between each topological node, spatial edges within the same time frame and temporal edges across adjacent time frames are constructed respectively. S13. Calculate the dynamic adjacency matrix based on the spatial edge, the temporal edge, and the preset language intent weight; S14. Based on the dynamic adjacency matrix, feature propagation and iterative updates are performed through multi-layer graph convolution to output the visual enhancement features; S2. Guided by the action query features, the visual enhancement features are nonlinearly mapped and fused using a bridging attention mechanism; the fused feature stream is input into a preset policy network, and the decoded output contains the original discrete action blocks containing the future temporal field of view length. S3. Define a continuous physical manifold space that includes the robot's physical limits; construct a total energy loss function to balance semantic preservation and physical smoothing; project the original discrete actions into the continuous physical manifold space for optimization; generate smooth action signals and send them out for execution.

2. The robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing according to claim 1, characterized in that, In S2, the policy network is built based on the Transformer architecture.

3. The robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing according to claim 1, characterized in that, In S3, the total energy loss function includes a first term to ensure that the action direction does not deviate from the original discrete action block, and a second regularization penalty term to limit abrupt changes in trajectory acceleration.

4. The robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing according to claim 3, characterized in that, The second regularization penalty term is obtained by integrating the urgency of the action command over time.

5. The robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing according to claim 1, characterized in that, In S3, the specific method for defining the continuous physical manifold space is as follows: Obtain the underlying physical limit parameters of the physical robot, which include the maximum joint rotation speed, the maximum joint acceleration, and the range of configuration singularities; The set of all continuous robot motion commands that satisfy the following constraints is defined as a low-dimensional continuous physical manifold space: at any time, the corresponding joint rotation speed is less than or equal to the maximum joint rotation speed, the corresponding joint acceleration is less than or equal to the maximum joint acceleration, and the configuration is not located within the range of configuration singularities.

6. The robot VLA cooperative control method based on spatiotemporal enhancement and trajectory smoothing according to claim 5, characterized in that, In S3, the specific steps for optimizing the original discrete action block by projecting it onto the continuous physical manifold space are as follows: using the original discrete action block as the initial value, and under the constraint of the total energy loss function, perform forward projection calculation to generate a smooth action signal.

7. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to execute the steps of the robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot VLA cooperative control method based on spatiotemporal augmentation and trajectory smoothing as described in any one of claims 1-6.

Images