Robot time sequence imitation learning method and system based on Mama coding complete history

Abstract

The invention belongs to the related technical field of artificial intelligence, and discloses a robot time sequence imitation learning method and system based on a Mama coding complete history, and the method comprises the steps: receiving a multi-modal observation sequence in a task execution process of a robot, the multi-modal observation sequence comprising observation data of at least one sensor; processing the multi-modal observation sequence by using a sequence processing module based on a state space model, and updating a time sequence output of complete historical information of one code up to the current time step at each time step; and predicting the next step or a series of future actions of the robot based on the time sequence output of the current time step so as to control the robot to simulate. The time sequence processing module based on the state space model is utilized to process and encode the complete observation history in the task execution process of the robot, so that a non-Markov decision-making imitation learning method is realized, and the learning efficiency and the execution success rate of the robot in a complex and state-dependent long time sequence operation task are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field related to artificial intelligence, and more particularly, to a robot time sequence imitation learning method and system based on Mamba coding complete history. BACKGROUND

[0002] With the rapid development of information age technology and the increasingly important role of various robot devices in today's social production and life, research on robots has become increasingly important. Imitation learning (IL) has become a powerful paradigm for teaching complex skills to robots, which learns directly from expert demonstrations, avoiding the need for complex reward engineering in reinforcement learning. Behavioral cloning (BC) is the simplest form of imitation learning, which directly learns the mapping from observations to actions through supervised learning. In recent years, methods based on powerful sequence models and generative models, such as Action Chunk Transformer (ACT) and Diffusion Policy, have made significant progress in learning visual motion control strategies for complex operations.

[0003] However, a fundamental limitation of many existing imitation learning methods lies in their dependence on the Markov assumption. These methods usually only predict actions a t or a short, fixed-length history window o t-k:t based on the current observation o t When the historical information of the task is crucial to eliminate the ambiguity of the current observation, this assumption will fail. For example, in a sequential task, the robot needs to first place object A to position 1 and then move it to position 2. At some intermediate state, the robot's visual observation and proprioceptive state can be exactly the same, regardless of whether it has successfully completed the subtask at position 1. The Markov policy that lacks memory of visiting position 1 cannot distinguish between these states with fundamentally different historical backgrounds, and may mistakenly go directly to position 2, resulting in task failure. This temporal ambiguity indicates that the task is essentially a partially observable Markov decision process (POMDP), posing a serious challenge to standard imitation learning methods.

[0004] Although there are works that attempt to use recurrent neural networks (RNNs) such as LSTM to introduce temporal context, they face problems such as gradient vanishing when dealing with long-term dependencies, and in practice often adopt fixed history windows and periodic state resets, which cannot truly capture the complete trajectory history. Transformer-based models, while leveraging self-attention mechanisms to model sequence dependencies, suffer from quadratic complexity O(L 2The calculation complexity of the above method limits the actual size of the context window, and it is difficult to efficiently process and encode the entire long trajectory.

[0005] In summary, the existing imitation learning method has the problems of relying on the Markov assumption, having a time window limit, and being prone to task failure due to time ambiguity. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a robot time sequence imitation learning method and system based on Mamba encoding complete history, which is used to solve the problems of the existing imitation learning method relying on the Markov assumption, having a time window limit, and being prone to task failure due to time ambiguity, and aims to provide a new robot imitation learning method that can effectively encode complete observation history, overcome the limitations of the Markov assumption, and efficiently process long sequences, and overcome the limitations of the imitation learning method in processing long time sequence dependence and partially observable tasks.

[0007] To achieve the above-mentioned purpose, according to one aspect of the present application, a robot time sequence imitation learning method based on Mamba encoding complete history is provided, comprising:

[0008] S1, receiving a multi-modal observation sequence in the process of a robot performing a task, wherein the multi-modal observation sequence includes observation data from at least one sensor of the robot;

[0009] S2, processing the multi-modal observation sequence by using a sequence processing module based on a state space model, and updating a time sequence output capable of encoding complete history information up to the current time step at each time step;

[0010] S3, predicting the next step or a series of future actions of the robot based on the time sequence output at the current time step which encodes complete history information, and controlling the robot to perform imitation according to the predicted actions.

[0011] According to the robot time sequence imitation learning method based on Mamba encoding complete history provided by the present application, the sequence processing module based on the state space model adopts a Mamba or Mamba-2 architecture; the Mamba or Mamba-2 architecture is used to generate a hidden state at the first time step according to the multi-modal observation sequence, and then generate a hidden state at each time step according to the multi-modal observation sequence at the current time step and the hidden state at the previous time step, and generate a time sequence output according to the hidden state at each time step.

[0012] According to the robot time sequence imitation learning method based on Mamba encoding complete history provided by the present application, S2 further comprises:

[0013] The observation data encoder converts and fuses the original data of the multi-modal observation sequence at each time step into a feature embedding vector of the multi-modal observation sequence through observation data;

[0014] Correspondingly, the sequence processing module in S2 updates a time sequence output capable of encoding complete historical information up to the current time step at each time step according to the feature embedding vector of the multi-modal observation sequence.

[0015] According to the robot time sequence imitation learning method based on Mamba encoding complete history provided by the application, the sensors include a vision sensor and a body perception sensor, which are respectively used for acquiring a vision image and a body perception state;

[0016] Correspondingly, the observation data encoder includes a vision encoder and a feature fusion module, the vision encoder is used for extracting vision features in the vision image, and the feature fusion module is used for fusing the vision features and the body perception state to obtain the feature embedding vector.

[0017] According to the robot time sequence imitation learning method based on Mamba encoding complete history provided by the application, the sequence processing module in S2 updates a time sequence output capable of encoding complete historical information up to the current time step at each time step, and the specific formula is as follows:

[0018]

[0019] Wherein, h t is the hidden state of the t time step; x t is the feature embedding vector of the t time step; is a state matrix; is an input matrix; C t is a conversion matrix; y t is the time sequence output of the t time step.

[0020] According to the robot time sequence imitation learning method based on Mamba encoding complete history provided by the application, the method further includes an offline training phase, and the specific process is as follows:

[0021] A multi-modal observation sequence and a corresponding expert action sequence are obtained from expert demonstration data, and a data set is established;

[0022] At each time step, a time sequence output capable of encoding complete historical information up to the current time step is updated by using a sequence processing module based on a state space model according to the multi-modal observation sequence in the data set;

[0023] A predicted action is obtained by using a prediction module according to the time sequence output of the current time step;

[0024] training and optimizing parameters of the sequence processing module and the prediction module by minimizing a loss function between the predicted action and the corresponding expert action in the dataset;

[0025] Correspondingly, the sequence processing module trained in S2 is used to update the timing output at each time step; the prediction module trained in S3 is used to predict the next action or a series of future actions of the robot.

[0026] According to the robot timing imitation learning method based on Mamba encoding complete history provided by the application, when the predicted action at each time step in S3 includes a series of future actions, the robot is controlled according to the predicted action to perform imitation, which specifically includes:

[0027] In the first time step, the robot is controlled to perform the first action in the corresponding predicted action;

[0028] Then, at each time step, a time aggregation strategy is used to determine the final execution action of the robot.

[0029] According to the robot timing imitation learning method based on Mamba encoding complete history provided by the application, then, at each time step, a time aggregation strategy is used to determine the final execution action of the robot, which specifically includes:

[0030] Then, at each time step t, the final execution action a t-final determined by weighted average of the actions predicted in the overlapping prediction blocks about time step t;

[0031] wherein, the predicted action at each time step forms a prediction block; the overlapping prediction blocks about time step t are the prediction blocks including the action predicted at time step t.

[0032] According to the robot timing imitation learning method based on Mamba encoding complete history provided by the application, the weight of the action predicted at time step t in the overlapping prediction blocks about time step t adopts an exponentially decaying weighted weight, which is specifically as follows:

[0033] w i = exp(-m*i);

[0034] wherein, w i is the weight of the action predicted at time step t in the i-th overlapping prediction block according to the order; m is a fixed coefficient.

[0035] According to another aspect of the present application, there is provided a robot temporal imitation learning system based on Mamba encoding complete history, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the robot temporal imitation learning method based on Mamba encoding complete history according to any one of the above aspects.

[0036] Overall, compared with the prior art, the robot temporal imitation learning method and system based on Mamba encoding complete history provided by the present application have the following advantages:

[0037] 1. A method for processing and encoding complete observation history in robot task execution process by using a temporal processing module based on state space model to realize imitation learning of non-Markov decision, which solves the limitations of traditional imitation learning methods in processing partial observable Markov decision process (POMDP) tasks and tasks requiring long temporal memory to eliminate observation ambiguity, significantly improves the learning efficiency and execution success rate of robots in complex, state-dependent long temporal operation tasks;

[0038] 2. The Mamba architecture has linear time complexity O(L) (L is the length of the sequence) and constant space complexity (for autoregressive generation) when processing sequences, which makes it very suitable for processing long sequence data that may occur in robot tasks, without facing the calculation bottleneck of square complexity like traditional Transformer, which can improve the efficiency of long sequence encoding and is suitable for long sequence encoding;

[0039] 3. When the predicted action at each time step includes a series of future actions, the final execution action at the time step is determined by adopting a time aggregation strategy to obtain a weighted average of the actions predicted in multiple prediction blocks for the same time step, which is beneficial to improve the smoothness and robustness of the robot execution action;

[0040] 4. Through the cooperative work of the above modules, the MTIL method and system proposed by the present application can effectively integrate complete task history information into each decision of the robot, thereby significantly improving its imitation learning performance and autonomous operation ability in complex, long temporal, partially observable environments. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of the robot temporal imitation learning method based on Mamba encoding complete history provided by the present application;

[0042] Figure 2 is a logic diagram of the robot temporal imitation learning method based on Mamba encoding complete history provided by the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0044] See also Figure 1 This embodiment provides a robot time sequence imitation learning method based on the complete history of Mamba coding, and the robot time sequence imitation learning method includes:

[0045] S1, receiving a multimodal observation sequence during a robot's task execution, wherein the multimodal observation sequence includes observation data from at least one sensor of the robot;

[0046] S2, processing the multimodal observation sequence using a sequence processing module based on a state-space model, and updating a time series output that can encode the complete historical information up to the current time step at each time step;

[0047] S3, based on the time series output encoding the complete historical information at the current time step, predict the robot's next step or a series of future actions, and control the robot to imitate according to the predicted actions.

[0048] This embodiment takes into account that existing tasks with temporal ambiguity are expressed as POMDPs, but human demonstrations are essentially non-Markov. Therefore, effective imitation requires a strategy with complete historical perception capabilities. Based on this, it is proposed to use a sequence processing module based on a state space model (SSM) to process the multimodal observation sequence time step by time step. At each time step t, a time series output is generated based on the multimodal observation sequence corresponding to all time steps up to time step t. The time series output encodes the complete historical information up to time step t, and then action prediction is performed based on the time series output that encodes the complete historical information. The complete historical information can be taken into account, thereby overcoming temporal ambiguity and improving the learning and execution success rate of the robot in complex, state-dependent long-time sequence operation tasks.

[0049] Further, the sequence processing module based on the state space model (SSM) adopts Mamba or its improved version Mamba-2 architecture; the embodiment provides a robot temporal imitation learning method (Mamba Temporal Imitation Learning, MTIL) based on the Mamba architecture, which is used to realize efficient coding and utilization of complete observation history in the robot task execution process; wherein for the requirement of making decisions based on complete historical information, a sequence processing module based on the Mamba architecture is designed, which can effectively compress and integrate the historical observation information into the current hidden state at each time step through its unique selective state update mechanism, thereby providing rich context information for subsequent action decision-making and improving the prediction accuracy.

[0050] Specifically, the Mamba or Mamba-2 architecture is used to generate a hidden state according to the multi-modal observation sequence at the first time step, and then generate a hidden state at the current time step according to the multi-modal observation sequence at the current time step and the hidden state at the previous time step at each time step, and generate a temporal output according to the hidden state at each time step.

[0051] That is, according to the observation data o t and the hidden state h t-1 at the previous time step, a hidden state h t at the current time step is updated to obtain a hidden state h t at the current time step, which encodes the complete observation history from the initial time step to the current time step t; and then a quantity related to the hidden state h t at the current time step is generated, that is, a temporal output.

[0052] Further, S2 further comprises:

[0053] The original data of the multi-modal observation sequence at each time step is converted and fused into a feature embedding vector of the multi-modal observation sequence by an observation data encoder (such as a visual encoder DINOv2, ResNet, etc.);

[0054] Correspondingly, the sequence processing module in S2 updates a temporal output capable of encoding complete historical information up to the current time step at each time step according to the feature embedding vector of the multi-modal observation sequence.

[0055] The sensors include visual sensors and proprioceptive sensors for acquiring visual images and proprioceptive states, respectively; correspondingly, the observation data encoder includes a visual encoder and a feature fusion module, the visual encoder is used to extract visual features in the visual images; the feature fusion module is used to fuse the visual features and the proprioceptive states to obtain the feature embedding vectors.

[0056] In some embodiments, the robot includes at least one sensor for acquiring a multi-modal observation sequence of the robot during the execution of a task. These sensor devices include but are not limited to: visual sensors such as top and side RGB cameras, palm-eye RGB cameras, depth cameras for capturing visual information of the environment; multiple cameras can be used to observe from different perspectives to obtain high-dimensional observation data, achieving more comprehensive scene understanding. Proprioceptive sensors such as robot joint encoders (providing joint angles, velocities, etc.), inertial measurement units (IMUs, providing attitude information), force / torque sensors for acquiring low-dimensional state information of the robot itself.

[0057] The raw observation data of the multi-modal observation sequence, especially high-dimensional data such as images, will first pass through a feature extraction module, namely a visual encoder module. The visual encoder module (for example, for visual data, a pre-trained convolutional neural network such as ResNet, or a visual Transformer such as DINOv2 can be used) is responsible for converting the original high-dimensional observation data into a low-dimensional feature embedding vector suitable for input into the subsequent sequence processing module. For other types of data, corresponding feature extraction methods can be used. This step helps to extract key information and reduce data dimensionality. Feature embeddings from different modalities can be fused through a feature fusion module using concatenation or more complex fusion mechanisms (cross-attention) to form a unified current time step input feature, i.e., a feature embedding vector x t The feature extraction module and the feature fusion module together form an observation data encoder.

[0058] Mamba or its improved version Mamba-2 architecture, this module is responsible for processing the feature embedding vectors x1, x2,..., x t Its working principle is as follows:

[0059] Hidden state maintenance, the module maintains a hidden state h t at each time step t. This hidden state is designed to be an effective compressed representation of the feature embedding vectors (x1,..., x t ) corresponding to all historical observation information O t = (o1,..., o t ) up to the current time step t.

[0060] The Mamba architecture realizes the update transmission of hidden states and obtains the current output through its core selective scan mechanism (SSM). Specifically, h t t-1 t , where the state transition function f is determined by the parameters of Mamba (such as state matrix, input matrix), and then the time sequence output y t is obtained through the conversion matrix C. This input-dependent feature enables the model to dynamically adjust the retention and forgetting of historical information according to the importance of current information, thereby more effectively capturing long-range dependencies.

[0061] Specifically, in S2, one time sequence output that can encode complete historical information up to the current time step is updated at each time step, and the specific formula is as follows:

[0062]

[0063] , where h t is the hidden state at time step t; x t is the feature embedding vector at time step t; is the state matrix; is the input matrix; C t is the conversion matrix; y t is the time sequence output at time step t. The output of the complete history encoding module, i.e., the sequence processing module based on the state space model, is the hidden state h t updated at each time step t and the time sequence output y t .

[0064] Further, it also includes an offline training phase, specifically as follows:

[0065] Obtain the multi-modal observation sequence and the corresponding expert action sequence from the expert demonstration data, and establish a data set;

[0066] At each time step t, use the sequence processing module based on the state space model to update a time sequence output that can encode complete historical information up to the current time step according to the multi-modal observation sequence in the data set;

[0067] Use the prediction module to obtain the predicted action according to the time sequence output at the current time step;

[0068] Train and optimize the parameters of the sequence processing module and the prediction module by minimizing the loss function between the predicted action and the corresponding expert action in the data set;

[0069] ​​Correspondingly, the sequence processing module trained in S2 is used to update the timing output at each time step; the prediction module trained in S3 is used to predict the next step or a series of future actions of the robot.

[0070] Further, the offline training stage further comprises training and optimizing the parameters of the feature fusion module in the observation data encoder by minimizing a loss function between the predicted action and the corresponding expert action in the data set.

[0071] In the training stage, the output (predicted action) of the prediction module is compared with the true action in the expert demonstration data, and the parameters of the entire MTIL model (including the observation data encoder, the Mamba module and the action prediction layer) are optimized by minimizing the loss function (such as mean square error MSE, cross entropy, etc.) between them, wherein the parameters of the visual encoder in the observation data encoder can be frozen. The training process follows a sequential manner, and the updated historical state is used to predict and learn at each time step.

[0072] The prediction module and the execution module receive the timing output y t from the complete history encoding module, predict a future action sequence (action block) containing the current and future K-1 steps, and finally execute the weighted action through a time aggregation strategy.

[0073] Action prediction module: one or more (usually fully connected) network layers take y t as input to predict an action sequence (action block) a^ t:t+K-1 containing the current and future K-1 steps that the robot should perform. The predicted action can be the joint angle of the robot, the target pose of the end effector, the velocity command, etc., depending on the task requirements and the robot control interface.

[0074] That is, at each time step t, the robot obtains the current observation o t ; then uses the trained policy network to calculate the current hidden state h t and the timing output y t by combining the current observation o t and the hidden state h t-1 maintained at the previous time step, and then obtains the predicted action a^ t:t+K-1 .

[0075] In the inference stage, the robot calculates the hidden state and the predicted action according to the current sensor observation. In order to improve the smoothness and robustness of the action, a time aggregation strategy can be used, that is, the predicted action at each time step in S3 includes a series of future actions, and the robot is controlled according to the predicted action to simulate, which specifically includes:

[0076] Controlling the robot to perform a first action in the corresponding predicted action at a first time step;

[0077] Then, a time aggregation strategy is used at each time step to determine the final execution action of the robot.

[0078] Furthermore, a time aggregation strategy is then used at each time step to determine the final execution action of the robot, specifically including:

[0079] Then at each time step t the final execution action a t-final Determined by weighted averaging of the actions predicted at time step t in the prediction blocks that overlap with respect to time step t;

[0080] The predicted action at each time step forms a prediction block; the prediction blocks that overlap with respect to time step t are the prediction blocks that include the action predicted for time step t. The finalized action is sent to the robot's underlying controller via the output interface for execution.

[0081] Specifically, the weight of the action predicted at time step t in the prediction block overlapping with time step t adopts an exponentially decaying weighted weight, as shown in the following formula:

[0082] w i =exp(-m*i);

[0083] Among them, w i is the weight of the action predicted for time step t in the i-th overlapping prediction block; m is a fixed coefficient.

[0084] In another embodiment, a robot time sequence imitation learning system based on complete history encoding is provided, comprising: at least one sensor device for acquiring multimodal observation data during the execution of a robot task; a data processing unit, wherein the data processing unit is configured with: an input interface for receiving a multimodal observation sequence from the sensor device; a sequence processing module based on a state space model (SSM) for processing the multimodal observation sequence time step by time step, and at each time step t, according to the observation data o of the current time step, t and the hidden state h at the previous time step t-1 , update to get a current hidden state h t , the current hidden state h t It encodes the complete observation history information from the initial time step to the current time step t, and then according to the current hidden state h t Get the current timing output y t ; An action prediction module that encodes the complete historical information based on the time series output y at the current time step t t , predict one or more actions a that the robot should performt:t+K-1 ; an output interface connected with the robot effector, for sending the predicted action to the robot effector.

[0085] The system further comprises an observation data encoder between the input interface and the sequence processing module, for converting and fusing high-dimensional observation data (such as images) into a low-dimensional feature embedding vector.

[0086] Further, the embodiment also provides a robot time sequence imitation learning system based on Mamba encoding complete history, characterized in that the system comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the robot time sequence imitation learning method based on Mamba encoding complete history according to any one of the above.

[0087] Specific application example: robot dual-arm cooperative task learning and execution based on MTIL.

[0088] This example describes the application of the MTIL method and system on two specially designed real-world dual-arm robot tasks, including a "sequential insertion task" and a "coordinated pouring task", which are designed to evaluate the performance of the model in complex physical environments that require long-term memory and precise coordination.

[0089] Hardware setup: The robot platform uses a platform containing two Universal Robots UR3 collaborative robot arms. Each robot arm is equipped with a custom 3D printed two-finger parallel gripper driven by a Dynamixel servo motor. The sensor devices include: visual sensors, a total of four cameras provide multi-view observation. An Intel RealSense D435i provides a side view, a Microsoft AzureKinect provides a top view, and two Intel RealSense D405 cameras are installed on the wrists of the two robot arms, providing close-up views, and these cameras capture RGB images. Proprioceptive sensors: the joint encoders built into the UR3 robot arm provide joint angle and speed information, and the state of the gripper (opening degree) is also used as input.

[0090] The system components are as follows:

[0091] The observation data encoder includes visual feature extraction, and the RGB images of all cameras are input into a pre-trained DINOv2 (ViT-L / 14 architecture) model to extract powerful visual features. The weights of DINOv2 are kept frozen during the MTIL training process. Multimodal fusion module: the extracted visual features are concatenated with low-dimensional robot state information (joint positions, gripper states) and fused through a cross-attention mechanism, and then projected to the desired input dimension of the Mamba module.

[0092] Full history encoding module, i.e., Mamba module: a sequence processor with a 4-layer Mamba-2 module is adopted. The model dimension dmodel is set to 2048, and the state expansion factor and state dimension dstate are set to 512. Through its recurrent update mechanism, the module calculates and outputs the current hidden state h t that encodes the full trajectory history at each time step t according to the fused current observation feature embedding vector x t-1 and the hidden state h t of the previous time step. t .

[0093] Action decision and output module:

[0094] Action prediction layer: a linear layer receives the time series output yt of the current Mamba module, and predicts an action block a^ t:t+K-1 containing the current and future K-1 (e.g., K = 50) steps. The predicted action is a 7-DOF control instruction (6D pose + 1D gripper) for the end effector of the dual-arm.

[0095] Output module with time aggregation: during inference execution, a weighted average method with exponential decay weights is used to aggregate multiple overlapping predicted action blocks to improve the smoothness and stability of the action.

[0096] Data processing unit: a computer equipped with a high-performance 4090 GPU is used to run the training and inference of the MTIL strategy model. Data collection: for each real-world task, 100 expert demonstrations are collected using a custom teleoperation system based on a spatial mouse.

[0097] Real-world task description and stages

[0098] Task 1: Sequential insertion task

[0099] This task is specifically designed to challenge Markov policy-based methods, as it requires long-term memory to distinguish between states that may appear similar under the current observation but differ historically and thus require different subsequent actions. The task involves sequentially inserting one tube (tube 1) into two other fixed tubes (tube 2 and tube 3) through the cooperation of the dual-arm. The four main stages of the task are as follows:

[0100] Stage 1 (left arm grasps tube 1): the left arm moves to a preset position and grasps the first tube (tube 1) on the workbench.

[0101] Stage 2 (left arm passes tube 1 to right arm): the left arm moves tube 1 to a handover position, and the right arm moves to the same position at the same time, taking tube 1 from the left arm.

[0102] Phase 3 (Right arm inserts test tube 1 into test tube 2): The right arm holds test tube 1 and precisely inserts it into the second tube, test tube 2, which is fixed on the workbench. Correct execution of this phase relies on the memory of the completion of the previous two phases, as a traditional approach cannot distinguish between inserting test tube 2 or test tube 3 based solely on the visual state of the right arm holding test tube 1 at the current moment.

[0103] Phase 4 (Right arm inserts test tube 1 into test tube 3): After successfully completing phase 3 (i.e., test tube 1 has been pulled out of test tube 2 or proceeding directly to the next step, depending on the specific demonstration), the right arm inserts test tube 1 into the third tube, test tube 3, which is fixed on the workbench.

[0104] Task Two: Coordinated Pouring Task

[0105] This task aims to evaluate the model's performance on tasks requiring precise dual-arm coordination and longer sequences of operations. The task involves one arm pouring a liquid (water) from a container into another container held by the other arm. The four main phases of the task are as follows:

[0106] Phase 1 (Left arm grasps): The left arm moves and grasps the first container, test tube 1, which is filled with water.

[0107] Phase 2 (Left arm passes test tube 1 to right arm): The left arm moves test tube 1 to a handoff position, and the right arm moves to that position and takes test tube 1 from the left arm. At this point, test tube 1 is held by the right arm.

[0108] Phase 3 (Left arm grasps test tube 2): The left arm moves and grasps the second empty container, test tube 2, which serves as the receiving container, and moves it to a suitable pouring-receiving position and pose.

[0109] Phase 4 (Right arm pours water from test tube 1 into test tube 2): The right arm precisely controls the pose of test tube 1 to pour the water inside into test tube 2 held by the left arm, while the left arm keeps test tube 2 stable.

[0110] MTIL Model Training and Inference

[0111] Reference Figure 2 , the training algorithm flow of the MTIL model:

[0112] The MTIL model of this embodiment learns through the following sequential training algorithm flow to imitate expert demonstrations:

[0113] Initialization: Prepare the expert demonstration trajectory dataset D, where each trajectory τ i contains a series of observation-action pairs (o1, a1,..., o Ti , a Ti ). Initialize the MTIL policy network π θThe parameters θ define the loss function L (e.g. mean squared error MSE), the action block size K, and the optimizer Opt (e.g. AdamW).

[0114] Iterative training: For each training cycle (epoch): For each trajectory τ in the expert demonstration dataset D i :Initialize the hidden state h0 of the Mamba module (for example, to the zero vector or a learnable initial state). Initialize the total loss L for the current trajectory traj ←0. For each time step t in the trajectory (from 0 to T i-1 ):Observation code: Get the observation o of the current time step t t , and convert it into a feature embedding x through an observation data encoder (such as DINOv2) t =Encoder(o t ).

[0115] Forward propagation and prediction: using MTIL policy network π θ The step function embeds x according to the current feature t and the hidden state h at the previous time step t-1 (For t=0, use h0), calculate the predicted action block a^ t:t+K-1 and the updated hidden state h t That is: (a^ t:t+K-1 ,h t )=π θ .step(x t ,h t-1 ).

[0116] Obtaining True Actions: From Expert Trajectory τ i Extract the real action block a of length K starting from time step t t:t+K-1 If t+K-1 exceeds the actual length of the trajectory Ti, the last real action a is used. Ti Fill in the action block.

[0117] Calculate loss: Calculate the predicted action block a^ for the current time step t:t+K-1 With real action block a t:t+K-1 The loss between t =L(a^ t:t+K-1 ,a t:t+K-1 ).

[0118] Cumulative loss: Add the loss of the current time step to the total loss of the trajectory: L traj =L traj +L t .

[0119] Parameter update: After processing a complete trajectory (or a batch of trajectory segments), the parameters θ of the MTIL policy network π are updated using the optimizer Opt based on the accumulated trajectory loss L traj (i.e. Opt.step() is executed).

[0120] Return result: After sufficient periods of training, the trained MTIL policy network π is returned. θ .

[0121] Inference algorithm flow of the MTIL model:

[0122] After training, the MTIL model performs autonomous inference and action execution on the robot through the following algorithm flow, which can optionally combine action segmentation and time aggregation strategies to improve performance:

[0123] Initialization: Load the trained MTIL policy network π θ . Get the initial observation o0 of the robot. Set the action block size K and the maximum task execution step T max . Initialize the hidden state h0 of the Mamba module (consistent with training). If time aggregation is used, initialize a prediction buffer B to store recent action block predictions, and prepare the time aggregation weights W, for example, exponential decay weights w i = exp(-m*i).

[0124] Iterative execution: For each time step t of the task (from 0 to T max-1 ):

[0125] Observation encoding: The robot obtains the observation o t at the current time step t and converts it to a feature embedding x t = Encoder(o t ) through the observation data encoder.

[0126] Forward propagation and prediction: Use the step function of the MTIL policy network π θ to calculate the predicted action block a^ t:t+K-1 and the updated hidden state h t based on the current feature embedding x t and the hidden state h t-1 of the previous time step (for t = 0, use h0). That is: (a^ t:t+K-1 , h t ) = π θ .step(x t , h t-1 ).

[0127] (Optional) Store prediction: Store the current predicted action block a^t:t+K-1 Store in prediction buffer B.

[0128] (Optional) Action aggregation: Get all actions predicted for the current time step t from the prediction buffer B. Specifically, collect all actions predicted at the previous time step j (where j ≤ t <j+K)生成的动作块a^ t:t+K-1 The action component corresponding to the current time step t (i.e., a^ t:t+K-1 [tj]). Let these actions be set P t .

[0129] Use the preset weight W to set P t The weighted average of the actions in the calculation is used to obtain the final action a to be performed at the current time step t t-final =WeightedAverage(P t ,W).

[0130] Action execution: If temporal aggregation is not used, the predicted action block a^ is directly taken t:t+K-1 The first action in a t-final The robot performs action a t-final .

[0131] Status update: h t-1 ←h t (Prepare hidden state for next time step).

[0132] (Optional) Check whether the task is complete or whether a termination condition has been reached.

[0133] Through the above training and inference process, the MTIL model is able to effectively learn from expert demonstrations and autonomously perform complex robotic tasks that require long-term memory and temporal reasoning.

[0134] Experimental results and analysis

[0135] In the sequential insertion task, MTIL (leveraging the complete history) was able to successfully complete all four stages sequentially, achieving a high overall success rate of 54% overall, 62% for stage 3, and 54% for stage 4. In contrast, state-of-the-art methods relying on limited context, such as ACT, are unable to resolve the temporal ambiguity of whether to insert test tube 2 or test tube 3 after stage 2. They often mistakenly attempt to directly perform actions similar to stage 4 (or target the wrong target) after completing stage 2, resulting in 0% success rates in stages 3 and 4, and 0% overall task success. This fully demonstrates MTIL's ability to resolve temporal ambiguity and execute long sequential dependency tasks by encoding complete history information.

[0136] In the coordinated pouring task, although the temporal ambiguity of the task is not as extreme as the sequential insertion task, precise and temporally coordinated actions are still required. MTIL also showed higher success rates than ACT in this task, with an overall success rate of 62% compared to 32% for ACT, and success rates of 94%, 80%, 74%, and 62% for each stage of MTIL, respectively. Furthermore, the trajectories executed by MTIL were generally smoother and more stable.

[0137] These real-world experimental results strongly demonstrate the MTIL method and system proposed by the present application, which can significantly improve the learning efficiency and execution robustness of robots in physical operation tasks that require long-term memory, temporal reasoning, and complex coordination by utilizing the Mamba architecture to encode complete trajectory history, surpassing existing imitation learning methods that rely on Markov assumptions or limited history.

[0138] Those skilled in the art will readily understand that the above description is only preferred embodiments of the present application and is not intended to limit the present application, and any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A robot temporal imitation learning method based on the complete history of Mamba coding, characterized in that: include: S1, receiving a multimodal observation sequence during a robot's task execution, wherein the multimodal observation sequence includes observation data from at least one sensor of the robot; S2, processing the multimodal observation sequence using a sequence processing module based on a state-space model, and updating a time series output that can encode the complete historical information up to the current time step at each time step; S3, based on the time series output encoding the complete historical information at the current time step, predict the robot's next step or a series of future actions, and control the robot to imitate according to the predicted actions.

2. The robot temporal imitation learning method based on the complete history of Mamba coding as claimed in claim 1, characterized in that: The state-space model-based sequence processing module adopts a Mamba or Mamba-2 architecture; the Mamba or Mamba-2 architecture is used to generate a hidden state according to the multimodal observation sequence at the first time step, and then generate a hidden state of the current time step according to the multimodal observation sequence of the current time step and the hidden state of the previous time step at each time step, and generate a time series output according to the hidden state of each time step.

3. The robot temporal imitation learning method based on the complete history of Mamba coding as claimed in claim 1, characterized in that: S2 and earlier also included: Converting and fusing the original data of the multimodal observation sequence at each time step into a feature embedding vector of the multimodal observation sequence through an observation data encoder; Correspondingly, the sequence processing module in S2 updates a time series output that can encode the complete historical information up to the current time step at each time step based on the feature embedding vector of the multimodal observation sequence.

4. The robot temporal imitation learning method based on the complete history of Mamba coding as claimed in claim 3 is characterized in that: The sensors include visual sensors and proprioception sensors, which are used to obtain visual images and proprioception states respectively; Correspondingly, the observation data encoder includes a visual encoder and a feature fusion module, wherein the visual encoder is used to extract visual features in the visual image; the feature fusion module is used to fuse the visual features and the proprioceptive perception state to obtain the feature embedding vector.

5. The robot temporal imitation learning method based on the complete history of Mamba coding as claimed in claim 3 is characterized in that: In S2, a time series output that can encode the complete historical information up to the current time step is updated at each time step. The specific formula is as follows: Among them, h t is the hidden state at time step t; x t is the feature embedding vector of time step t; is the state matrix; is the input matrix; C t is the transformation matrix; y t is the time series output of time step t.

6. The robot sequential imitation learning method based on the complete history of Mamba coding according to any one of claims 1 to 5, characterized in that: It also includes an offline training phase, as follows: Obtain multimodal observation sequences and corresponding expert action sequences from expert demonstration data to build a dataset; At each time step, using a sequence processing module based on a state-space model to update a time series output that can encode complete historical information up to the current time step according to the multimodal observation sequence in the data set; Use the prediction module to obtain the predicted action based on the time series output of the current time step; Training and optimizing the parameters of the sequence processing module and the prediction module by minimizing the loss function between the predicted action and the corresponding expert action in the data set; Correspondingly, in S2, the trained sequence processing module is used to update the timing output at each time step; in S3, the trained prediction module is used to predict the robot's next step or a series of future actions.

7. The robot sequential imitation learning method based on the complete history of Mamba coding according to any one of claims 1 to 5, characterized in that: When the predicted action at each time step in S3 includes a series of future actions, controlling the robot to imitate according to the predicted actions specifically includes: Controlling the robot to perform a first action in the corresponding predicted action at a first time step; Then, a time aggregation strategy is used at each time step to determine the final execution action of the robot.

8. The robot sequential imitation learning method based on the complete history of Mamba coding as claimed in claim 7, characterized in that: Then, a time aggregation strategy is used at each time step to determine the final execution action of the robot, including: Then at each time step t the final execution action a t-final Determined by weighted averaging of the actions predicted at time step t in the prediction blocks that overlap with respect to time step t; The predicted actions for each time step form a prediction block; the prediction blocks that overlap with respect to time step t are prediction blocks that include the actions predicted for time step t.

9. The robot temporal imitation learning method based on the complete history of Mamba coding as claimed in claim 8, characterized in that: The weight of the action predicted at time step t in the prediction block overlapping with time step t adopts an exponentially decaying weight, as shown in the following formula: w i =exp(-m*i); Among them, w i is the weight of the action predicted for time step t in the i-th overlapping prediction block; m is a fixed coefficient.

10. A robot temporal imitation learning system based on the complete history of Mamba coding, characterized by: The system includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the robot temporal imitation learning method based on the complete history of Mamba coding according to any one of claims 1 to 9 is executed.

Images