A robot operation control method and system

Abstract

This invention discloses a robot operation control method and system, belonging to the field of robot operation control technology, including the following steps: encoding the image observation data and language commands at the current moment, and fusing the encoding results to obtain the three-dimensional representation of the current moment; based on the three-dimensional representation of the current moment, acquiring scene information for multiple future trajectory queries with temporal relationships to obtain the hidden states of multiple future trajectories; concatenating the hidden states of multiple future trajectories with the robot's current end-effector state to obtain the context hidden state; based on the context hidden state, obtaining action information from multiple action queries to generate the robot's action at the next moment. This invention solves the problems of depth ambiguity and scale inconsistency between the two-dimensional image plane and the three-dimensional operation space, improving the action accuracy, task success rate, and generalization robustness of robot operation in complex multi-task situations.

Description

Technical Field

[0001] This invention relates to the field of robot operation and control technology, and in particular to a robot operation and control method and system. Background Technology

[0002] Robot visual motion policy determines the next action in real time based on camera images, natural language commands, and the robot's state. It is a core technology for multi-tasking, long-term operations, and natural human-robot interaction, and is widely used in scenarios such as home services, industrial assembly, and logistics sorting. Currently, visual motion policy uses neural networks for end-to-end learning and inference.

[0003] However, camera images are essentially projections of a 3D scene onto a 2D plane, losing true-scale depth, relative distances between objects, and pose information; robot movements are parameterized by rigid body transformations on the SE(3) group. If the network learns directly from the image plane... The mapping of actions must simultaneously complete spatial recovery, pose estimation, coordinate transformation and action parameterization. There are ambiguities in the depth and scale of multiple three-dimensional states projected onto the same two-dimensional coordinates, which are geometrically misaligned with the six-degree-of-freedom action space. This results in low network training efficiency and limited generalization ability, ultimately reducing the accuracy, stability and generalization ability of robot operations in real-world scenarios. Summary of the Invention

[0004] In view of the deficiencies of the existing technology, the present invention provides a robot operation control method and system, which solves the existing problems.

[0005] The present invention adopts the following technical solution: In a first aspect, the present invention provides a robot operation control method, comprising the following steps: Acquire the current image observation data, voice commands, and the current state of the robot's end effector; The image observation data and language commands at the current moment are encoded separately, and the encoding results are fused to obtain the three-dimensional representation of the current moment; Based on the current 3D representation, scene information is acquired for multiple future trajectory queries with temporal relationships to obtain the hidden states of multiple future trajectories; the hidden states of multiple future trajectories are concatenated with the robot's current end state to obtain the context hidden state; based on the context hidden state, action information is obtained from multiple action queries to generate the robot's action at the next moment.

[0006] Preferably, the step of encoding the image observation data and language commands at the current moment, and fusing the encoding results to obtain the three-dimensional representation at the current moment, specifically includes the following steps: Image observation data and language instructions are input into a pre-trained visual language model, which outputs visual hidden states and language hidden states. Set up multiple semantic queries, perform cross-attention calculation on multiple semantic queries and language hidden states, and obtain a task semantic representation of a set length; Image observation data is input into a pre-trained metric depth estimation model to extract the deep hidden states; The task semantic representation, visual hidden state, and deep hidden state are fused to obtain the three-dimensional representation.

[0007] Preferably, the step of acquiring scene information for multiple future trajectories with temporal relationships based on the current three-dimensional representation to obtain the hidden states of multiple future trajectories specifically includes the following steps: By performing rotational position encoding and self-attention mechanism calculation on multiple future trajectory queries, multiple future trajectory queries with temporal relationships are obtained. Cross-attention calculation is performed on multiple future trajectory queries and 3D representations with temporal relationships to obtain corresponding scene information and obtain the hidden states of multiple future trajectories.

[0008] Preferably, the step of concatenating multiple future trajectory hidden states with the robot's current end-effector state to obtain the context hidden state specifically includes the following steps: The current end-effector state of the robot is linearly projected to obtain the state embedding. The context hidden state is obtained by concatenating multiple future trajectory hidden states and state embeddings.

[0009] Preferably, the step of obtaining action information from multiple action queries based on the context-hidden state and generating the robot's next action specifically includes the following steps: Cross-attention calculation is performed on the contextual hidden state and multiple action queries to obtain the hidden state containing action information; By performing a linear mapping on the hidden state, we can obtain the action at the next moment.

[0010] Preferably, the visual language model adopts Qwen2.5-0.5B.

[0011] Preferably, the image observation data includes camera images and wrist camera images.

[0012] Preferred options also include: Map multiple hidden states of future trajectories to obtain multiple terminal states of the future; Trajectory loss is constructed by comparing multiple future terminal states with the actual terminal states; Construct motion loss by comparing the action in the next moment with the actual action; Joint training is performed using trajectory loss and motion loss.

[0013] Preferably, the current end effector state of the robot includes its three-dimensional position, rotation parameters, and gripper opening / closing amount.

[0014] In a second aspect, the present invention provides a robot operation control system, comprising: The acquisition module is used to acquire the current image observation data, language commands, and the current end-effector state of the robot. The encoding module is used to encode the image observation data and language commands at the current moment, and then fuse the encoding results to obtain the three-dimensional representation at the current moment. The generation module is used to acquire scene information for multiple future trajectory queries with temporal relationships based on the current 3D representation, and obtain the hidden states of multiple future trajectories; it then concatenates the hidden states of multiple future trajectories with the robot's current end state to obtain the context hidden state; based on the context hidden state, it retrieves action information from multiple action queries and generates the robot's action for the next moment.

[0015] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention constructs a three-dimensional representation containing spatial and semantic information by encoding and fusing the current image observation data, language commands, and robot end-effector state. Furthermore, it utilizes learnable trajectory queries to predict the future multi-step end-effector SE(3) state trajectory, and finally decodes the current state to generate precise robot actions. This method uses the SE(3) trajectory as a geometrically aligned intermediate representation between the observation space and the action space, solving the problems of depth ambiguity and scale inconsistency between the two-dimensional image plane and the three-dimensional operation space. This effectively improves the robot's action accuracy, task success rate, and generalization robustness in complex multi-task, long-term tasks, and real-world scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the overall structure of a robot operation control method according to the present invention; Figure 2 This is a visualization of the SE(3) trajectory prediction and robot execution process of the present invention; Figure 3Comparison chart of the average success rates of different methods in three out-of-distribution scenarios of real robot tasks; Figure 4 Visualization example of a representative task in four task suites of LIBERO in Embodiment 1 of the present invention; Among them, Figure 4 (a): LIBERO SPATIAL initial scene, Figure 4 (b): LIBERO SPATIAL operation example, Figure 4 (c): LIBERO GOAL initial scene, Figure 4 (d): LIBERO GOAL operation example, Figure 4 (e): LIBERO OBJECT initial scene, Figure 4 (f): LIBERO OBJECT operation example, Figure 4 (g): LIBERO LONG initial scene, Figure 4 (h): LIBERO LONG operation example; Figure 5 Visualization example of a representative task of continuously executing five instructions in the CALVIN ABC→D benchmark in Embodiment 1 of the present invention; Among them, Figure 5 (a): Environment A, Figure 5 (b): Environment B, Figure 5 (c): Environment C, Figure 5 (d): Environment D; Figure 6 Schematic diagram of the real robot platform and the external and wrist camera arrangements in Embodiment 2 of the present invention; Among them, Figure 6 (a): Robot platform setting, Figure 6 (b): Scene camera arrangement; Figure 7 Example of the execution process in the real robot multi-task and long-time sequence task in Embodiment 2 of the present invention; Among them, Figure 7 (a): Initial, Figure 7 (b): Move, Figure 7 (c) Grab, Figure 7 (d): Move, Figure 7 (e): Approach the target, Figure 7 (f): Place, Figure 7 (g): Initial, Figure 7 (h): Move, Figure 7 (i): Grab, Figure 7 (j): Move, Figure 7 (k): Approach the target, Figure 7 (l): Placement, Figure 7 (m): Initial, Figure 7 (n): Movement, Figure 7 (o): Grab the handle, Figure 7 (p): Open the drawer, Figure 7 (q): Grasp, Figure 7 (r): Placement; Figure 8 This is an execution example of the real robot in Embodiment 3 of the present invention in a scenario without a background, with a changed camera perspective, and dynamic human interference; Among them, Figure 8 (a): Initial, Figure 8 (b): Movement, Figure 8 (c): Grasp, Figure 8 (d): Movement, Figure 8 (e): Approach the target, Figure 8 (f): Placement, Figure 8 (g): Initial, Figure 8 (h): Movement, Figure 8 (i): Grasp, Figure 8 (j): Movement, Figure 8 (k): Approach the target, Figure 8 (l): Placement, Figure 8 (m): Initial, Figure 8 (n): Movement, Figure 8 (o): Grasp, Figure 8 (p): Movement, Figure 8 (q): Approach the target, Figure 8 (r): Placement. Specific implementation manner

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] In response to the above geometric inconsistency problem, the present invention proposes to first elevate the two-dimensional observation to a three-dimensional representation containing metric depth and task semantics, then predict the SE(3) state trajectory of the end effector in multiple future steps, and finally decode the action based on the SE(3) trajectory and the current state of the robot. Since the SE(3) trajectory is completely consistent with the geometric structure where the robot action is located, it can serve as a natural geometric bridge between the observation space and the action space.

[0020] This invention provides a robot operation control method and system based on semantically refined 3D representation and SE(3) trajectory prediction. The learning process is explicitly divided into a three-level structure of "2D observation → 3D representation → SE(3) trajectory → six-degree-of-freedom action". The SE(3) trajectory is used as the geometrically aligned intermediate representation between the observation space and the action space. Without relying on large-scale public robot data pre-training, the policy learning burden is significantly reduced, and the multi-task accuracy, long-term stability and generalization ability in real-world scenarios are improved.

[0021] This invention proposes a robot operation control method based on semantically refined 3D representation and SE(3) trajectory prediction. At each control moment... Acquire image observation data Language instructions and the current end state of the robot A three-dimensional representation is formed through visual language encoding, semantic refinement, and metric depth estimation; the future is output based on the three-dimensional representation. The final state trajectory of step SE(3); combined with the predicted trajectory and the current state to generate Step six degrees of freedom motion and gripper commands; after the robot executes the commands, it enters the next closed-loop moment.

[0022] The process can be summarized as follows: planar observation is lifted to obtain three-dimensional position information; three-dimensional representation is used to predict the future SE(3) state trajectory; the SE(3) trajectory is then decoded to obtain six-degree-of-freedom actions. Using the SE(3) trajectory, which is consistent with the geometry of the action space, as an intermediate representation, the originally difficult end-to-end mapping is decomposed into multiple clearly structured technical steps.

[0023] ; like Figure 1 As shown, the system of the present invention includes: a data acquisition module, a visual language encoding module, a semantic refinement module, a depth measurement module, a 3D representation fusion module, an SE(3) trajectory prediction module, an action decoding module, a training optimization module, and a robot execution module. The functions of each module and the corresponding method steps are as follows:

[0024] Step S1: Data Acquisition.

[0025] Acquire image observation data Language instructions and the current end state of the robot Image observation data can be third-person camera images, wrist camera images, or a combination of both; end state includes three-dimensional position, rotation vector (or quaternion, six-dimensional rotation representation, Lie algebra, etc., equivalent attitude parameters) and gripper opening / closing amount.

[0026] Step S2: Visual language encoding.

[0027] Will and The input is fed into a visual language model (a multimodal pre-trained model based on Qwen2.5-0.5B), which outputs the visual hidden state. ∈ Language Hidden State ∈ , Let be the length of the vector representing the visual hidden state. Let be the length of the vector representing the language hidden state. Let be the dimension of the vector. The visual hidden state is a high-dimensional vector representation containing visual information, and the language hidden state is a high-dimensional vector representation containing language information.

[0028] Step S3: Semantic refinement.

[0029] set up A learnable query serves as a task semantic anchor for a variable-length language hidden state. Perform cross attention to obtain a fixed-length task semantic representation. ∈ This step significantly reduces target confusion caused by redundant terms and irrelevant semantics in multi-task scenarios. A learnable query is a learnable high-dimensional vector representation with a length of [length missing]. N , dimension D It can learn semantic information related to the task.

[0030] Step S4: Measure the depth.

[0031] Employing a pre-trained metric depth estimation model (Depth Anything 3), by Generate real-scale multi-scale depth features The depth hidden state is obtained by average pooling and linear projection. ∈ Depth measurement, compared to depth which only provides relative proximity, directly provides physical distance information. Depth hidden features are high-dimensional vector representations that contain depth information.

[0032] Step S5: Fusion of three-dimensional representations.

[0033] Will , and By concatenating the parts, a semantically refined three-dimensional representation is obtained. It also includes task semantics, visual appearance, and real-scale spatial information.

[0034] Step S6: SE(3) trajectory prediction.

[0035] initialization Learnable trajectory query ∈ Temporal consistency is established through a self-attention layer containing Rotation Position Encoding (RoPE), and through a cross-attention layer... Read scene information and output Trajectory Hidden State ; obtain the future by performing linear mapping through a fully connected neural network. End states =[ , ]∈ , This represents the end state in the camera coordinate system. This represents the end-position state in the camera coordinate system. This represents the end-effector rotation state in the camera coordinate system. It is then further transformed into the SE(3) transformation matrix in the camera coordinate system. As shown in the following formula:

[0036] ; Learnable trajectory query is a learnable high-dimensional vector representation with a length of H , dimension D It can learn the SE(3) trajectory information, and after a series of calculations, finally let the model output the SE(3) trajectory.

[0037] Specifically, the following steps are included: S61: First, initialize H learnable trajectory queries. .

[0038] S62: Then, in the self-attention layer, query these H learnable trajectories. Rotational Position Encoding (RoPE) and self-attention mechanism computation are performed to ensure the temporal consistency of learnable trajectory queries.

[0039] S63: Next, in the cross-attention layer, the learnable trajectory query that has passed through the self-attention layer is combined with the semantically refined 3D representation. Cross-attention mechanism calculations are performed to enable the reading and understanding of scene information.

[0040] S64: Finally, output the results calculated through the above steps, which yields H trajectory hidden states. .

[0041] Step S7: Action decoding.

[0042] initialization Search for learnable actions ∈ Hide the trajectory Compared with the current state State embedding obtained by linear projection ∈ Concatenate into context-hidden state Cross-attention is performed between the contextual hidden state and the learnable action query to obtain the hidden state containing action information. h act Then, it is further linearly mapped to the action output. The action decoding module essentially learns a two-step mapping, as shown in the formula: ; The first step calculates the relative transformation matrix by predicting the SE(3) trajectory and the current end state, and the next step parameterizes the transformation matrix into a six-degree-of-freedom action. For the approximate calculation of the relative state transition of SE(3), The parameterization transformation of the relative state of SE(3) to six degrees of freedom motion.

[0043] Learnable action queries are represented as learnable high-dimensional vectors of length H and dimension D. Action information can be learned, and after a series of calculations, the model outputs the action.

[0044] Specifically, the motion decoding module implicitly learns the extrinsic parameter transformation from the camera to the world coordinate system. and the relative transformation between adjacent end states As shown in the formula: ; The module's final output action = [Δ , Δ ]∈ and gripper commands Δ , Δ This represents the relative change in position and rotation.

[0045] S5 constructs the input features, and S6 uses these features as input to predict the output as the SE(3) trajectory through a series of calculations. S7 further uses the hidden state corresponding to the SE(3) trajectory as input, combines it with the hidden state of the current terminal state, and after a series of calculations, finally outputs the executable action.

[0046] Step S8: Training optimization.

[0047] Using trajectory loss With motion loss The weighted sum is the total loss. Joint training; weight coefficients The example value is 0.1. The trajectory loss is supervised by projecting the expert's end state in the world coordinate system onto the camera coordinate system through the camera's extrinsic parameters, and the action loss is supervised by the expert's action sequence. Both are measured using ℓ1.

[0048] ; ; ; Step S9: Closed-loop execution: During the deployment phase, S1 to S7 are repeated in each control cycle. The robot performs one or more actions in the predicted sequence. In the next cycle, the prediction is re-made based on the new image and state, realizing online correction of target offset, viewpoint change and dynamic human interference.

[0049] Improved geometric consistency of robot operation tasks. Using the SE(3) trajectory as an intermediate representation for geometric alignment significantly alleviates the geometric inconsistency between the observation and action space. The SE(3) trajectory explicitly constrains the future end-effector pose, making it suitable for pose-sensitive tasks such as object grasping, placement, and alignment of tools or containers; Figure 2 The visualization shows that the predicted SE(3) trajectory is highly consistent with the actual execution trajectory.

[0050] The success rate of robot operation tasks has been improved. On the LIBERO simulation platform, the success rates reached 99.0% / 98.8% / 97.4% / 95.2% respectively (average 97.6%, exceeding UniVLA by 2.4%), as detailed in Table 1; the average completion length of CALVIN ABC→D was 4.57, and the success rate of executing five consecutive instructions was 83.3% (exceeding VPP by 8.3%), as detailed in Table 2; the average success rate of multi-task operation of the real robot was 90.4% (Goal 100%, Spatial 86.3%, Long 85%), which is better than π0.5, RDT, and ACT, as detailed in Table 3.

[0051] Table 1. Comparison of LIBERO benchmark four-task suite success rates (%) Table 2 Comparison of CALVIN ABC→D Benchmark Continuous Execution Success Rate and Average Completion Length Table 3 Comparison of success rates for multi-tasking and long-term operations of real robots (%) Reference Figure 3The out-of-distribution generalization capability is improved. In three types of OOD scenarios—no background, changed camera view, and dynamic human interference—the average success rate is 95%, 90%, and 85%, respectively, which is significantly better than strong baselines.

[0052] Deployment efficiency is good. A single NVIDIA RTX 4090 GPU uses approximately 4.5GB of video memory during inference, predicts the next 8 actions per inference, and has a single response time of approximately 0.05 seconds, which meets the requirements of real-time closed-loop control.

[0053] Example 1 Multi-task operation control of simulated robots.

[0054] Composition: This invention is deployed on the LIBERO (Spatial / Object / Goal / Long) and CALVIN ABC→D benchmarks. The policy input at each time step includes language commands, third-person and wrist RGB images, and 7-dimensional robot state. The output is six-DOF motion and gripper opening / closing amount. The visual language model uses Qwen2.5-0.5B; the depth metric model uses DA3METRIC-LARGE; the trajectory predictor has a 4-layer Transformer layer, and the motion decoder has a 2-layer layer. =896, attention count 8 =8, =16.

[0055] Training: The VLM was fine-tuned using LoRA, the deep model was frozen, and all other parameters were trained with full parameters, resulting in approximately 0.18B of trainable parameters. The AdamW optimizer was used with a learning rate of 2×10⁻⁻. 4 Total batch size 64 (4 NVIDIA A800 GPUs, 16 per card). =0.1. The training data uses only the benchmark's built-in dataset and does not use public robot data for pre-training.

[0056] Results: The success rates of the LIBERO four-task suite were 99.0%, 98.8%, 97.4%, and 95.2% (average 97.6%); the average completion length for CALVIN ABC→D was 4.57, with a final success rate of 83.3%. Compared to OpenVLA, this invention achieves a 21.1% higher average success rate with approximately 1 / 40th the number of trainable parameters compared to LIBERO. Figure 4 , Figure 5 The representative task execution processes in the two benchmarks are shown respectively.

[0057] Example 2 Real-world robots perform multitasking and long-term operations.

[0058] Composition: such as Figure 6As shown, the robotic platform is the Franka Research 3 robotic arm, with image input from an external third-person camera and a wrist camera. Tasks are divided into three categories: target-oriented (6 tasks including placing objects into containers), spatial (4 tasks including stacking blocks, building a block tower, placing a pot on a wooden stand, and placing an orange jar on a shelf), and long-term (a combination of long-term subtasks such as opening a drawer and then placing bananas from the drawer into a red bowl). Fifty teleoperation expert demonstrations were collected for each category, and multi-task training was conducted.

[0059] Effects: The present invention achieves 100%, 86.3%, and 85% (average 90.4%) in target-type, spatial-type, and long-term robot operation tasks, respectively, which is significantly better than π0.5 (82.8%), RDT (68.9%), and ACT (41.7%). Figure 7 The actual execution process of a typical task is shown.

[0060] Example 3 Out-of-Depth (OOD) generalization.

[0061] Composition: Based on the target task strategy trained in Example 2, three types of distributed off-site scenarios were constructed: changing the tablecloth and interfering objects (background not shown), changing the position and angle of the third-person camera, and the experimenter dynamically moving the target container during robot handling. No additional fine-tuning of the strategy was performed, and each scenario and task was repeated 20 times.

[0062] Results: The average success rates for the three types of OOD scenarios were 95%, 90%, and 85%, respectively, all of which were significantly better than π0.5 and RDT. Figure 8 This illustrates the execution process in a typical OOD scenario.

[0063] Based on the same concept, the present invention also provides a robot operation control system, including an acquisition module, an encoding module and a generation module.

[0064] The acquisition module is used to acquire the current image observation data, language commands, and the current end-effector state of the robot.

[0065] The encoding module is used to encode the image observation data and language commands at the current moment, and then fuse the encoding results to obtain the three-dimensional representation of the current moment.

[0066] The generation module is used to acquire scene information for multiple future trajectory queries with temporal relationships based on the current 3D representation, and obtain the hidden states of multiple future trajectories; the hidden states of multiple future trajectories are concatenated with the current end state of the robot to obtain the context hidden state; based on the context hidden state, action information is obtained from multiple action queries to generate the robot's action at the next moment.

[0067] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0068] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A robot operation control method, characterized in that, Includes the following steps: Acquire the current image observation data, voice commands, and the current state of the robot's end effector; The image observation data and language commands at the current moment are encoded separately, and the encoding results are fused to obtain the three-dimensional representation of the current moment; Based on the current three-dimensional representation, scene information is obtained for multiple future trajectories with temporal relationships, and the hidden states of multiple future trajectories are obtained. The contextual hidden state is obtained by concatenating multiple future trajectory hidden states with the robot's current end state. Based on the context-hidden state, action information is obtained from multiple action queries to generate the robot's action for the next moment.

2. The robot operation control method as described in claim 1, characterized in that, The process of encoding the current image observation data and language commands separately, and then fusing the encoding results to obtain the current three-dimensional representation, specifically includes the following steps: Image observation data and language instructions are input into a pre-trained visual language model, which outputs visual hidden states and language hidden states. Set up multiple semantic queries, perform cross-attention calculation on multiple semantic queries and language hidden states, and obtain a task semantic representation of a set length; Image observation data is input into a pre-trained metric depth estimation model to extract the deep hidden states; The task semantic representation, visual hidden state, and deep hidden state are fused to obtain the three-dimensional representation.

3. The robot operation control method as described in claim 1, characterized in that, The method of obtaining scene information for multiple future trajectories with temporal relationships based on the current three-dimensional representation to obtain the hidden states of multiple future trajectories specifically includes the following steps: By performing rotational position encoding and self-attention mechanism calculation on multiple future trajectory queries, multiple future trajectory queries with temporal relationships are obtained. Cross-attention calculation is performed on multiple future trajectory queries and 3D representations with temporal relationships to obtain corresponding scene information and obtain the hidden states of multiple future trajectories.

4. The robot operation control method as described in claim 1, characterized in that, The process of concatenating multiple future trajectory hidden states with the robot's current end-effector state to obtain the context hidden state specifically includes the following steps: The current end-effector state of the robot is linearly projected to obtain the state embedding. The context hidden state is obtained by concatenating multiple future trajectory hidden states and state embeddings.

5. The robot operation control method as described in claim 1, characterized in that, The process of obtaining action information from multiple action queries based on context-hidden states and generating the robot's next action specifically includes the following steps: Cross-attention calculation is performed on the contextual hidden state and multiple action queries to obtain the hidden state containing action information; By performing a linear mapping on the hidden state, we can obtain the action at the next moment.

6. The robot operation control method as described in claim 2, characterized in that, The visual language model used is Qwen2.5-0.5B.

7. The robot operation control method as described in claim 1, characterized in that, The image observation data includes camera images and wrist camera images.

8. The robot operation control method as described in claim 1, characterized in that, Also includes: Map multiple hidden states of future trajectories to obtain multiple terminal states of the future; Trajectory loss is constructed by comparing multiple future terminal states with the actual terminal states; Construct motion loss by comparing the action in the next moment with the actual action; Joint training is performed using trajectory loss and motion loss.

9. A robot operation control method as described in claim 1, characterized in that, The current end effector state of the robot includes its three-dimensional position, rotation parameters, and gripper opening / closing amount.

10. A robot operation control system, characterized in that, include: The acquisition module is used to acquire the current image observation data, language commands, and the current end-effector state of the robot. The encoding module is used to encode the image observation data and language commands at the current moment, and then fuse the encoding results to obtain the three-dimensional representation at the current moment. The generation module is used to obtain scene information for multiple future trajectories with temporal relationships based on the current 3D representation, and to obtain the hidden states of multiple future trajectories. The contextual hidden state is obtained by concatenating multiple future trajectory hidden states with the robot's current end state. Based on the context-hidden state, action information is obtained from multiple action queries to generate the robot's action for the next moment.

Images