Robot Motion Prediction Method, Apparatus, Device, and Storage Medium

Abstract

The present application provides a robot motion prediction method, apparatus, device, and storage medium. Among them, the method includes: generating initial motion noise, generating conditional features based on the historical motion sequence and point cloud data set of the robot, encoding the initial motion noise and conditional features based on the encoder in the motion prediction model, and performing decoding processing based on the decoder in the motion prediction model to obtain the predicted motion sequence of the target robot. The present application combines point cloud data and the historical motion sequence of the robot through the Mamba module in the decoder and encoder for prediction, and can ensure the accuracy and long-term prediction ability of the model prediction results under the premise of lower computing requirements.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a robot motion prediction method, device, equipment and storage medium. Background Art

[0002] Imitation learning is an effective learning approach in the robot vision-motion framework. By observing and imitating human demonstrations, robots can learn and achieve a variety of motion control. Diffusion strategy is a method in imitation learning that can effectively manage multimodal action distribution and has been widely used in robotic applications.

[0003] However, the diffusion strategy relies on a large backbone architecture, which has huge computational overhead and is difficult to apply in resource-constrained environments or devices. Therefore, how to reduce the computational overhead while maintaining the high performance of the diffusion strategy and improve the long-term prediction ability of the model has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of this application is to provide a robot motion prediction method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems in the prior art that the diffusion strategy has high computational overhead and the model performance needs to be improved.

[0005] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0006] In a first aspect, the present application provides a robot motion prediction method, which is applied to a target robot deployed with a motion prediction model, wherein the motion prediction model includes: an encoder and a decoder, wherein both the encoder and the decoder are U-net structures, and both the encoder and the decoder include at least one Mamba module; the method includes:

[0007] During the movement of the target robot, the iterative loop performs the following steps:

[0008] A. generating initial motion noise, obtaining at least one historical motion sequence in the historical motion library of the target robot, and generating conditional features according to the historical motion sequence and the point cloud data set, wherein the conditional features include: point cloud features, state features, and time features;

[0009] B. Input the initial action noise and the conditional feature into the action prediction model, and perform encoding processing layer by layer by the Mamba module of each layer in the encoder to obtain a first encoding feature, a second encoding feature and an encoding action feature, and input the first encoding feature, the second encoding feature and the encoding action feature into the decoder, and perform decoding processing by the Mamba module in the decoder to obtain a decoded action feature;

[0010] C. Input the decoded action features into the output layer of the action prediction model to obtain a predicted action sequence. Control the movement of the target robot according to the predicted action sequence, determine the state and reward based on the actual action sequence when the target robot is moving, and add the actual action sequence, state, and reward as a piece of historical action information to the historical action library.

[0011] Optionally, the generating conditional features based on the historical action sequence and the point cloud data set includes:

[0012] Perform encoding processing on the point cloud data set based on a point cloud encoder to obtain the point cloud features;

[0013] Extract features from the state data in the historical action sequence based on a state encoder to obtain the state features;

[0014] Extract features from the time data in the historical action sequence based on a time encoder to obtain the time features.

[0015] Optionally, the encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module, and the third encoding module includes: a Mamba module;

[0016] The process of obtaining the first encoded feature, the second encoded feature, and the encoded action feature by encoding processing with the Mamba module in the encoder includes:

[0017] The Mamba module in the first encoding module extracts features from the initial action noise and the conditional features to obtain a first extracted feature, and the downsampling module performs downsampling processing on the first extracted feature to obtain the first encoded feature;

[0018] The Mamba module in the second encoding module extracts features from the first encoded feature to obtain a second extracted feature, and the downsampling module performs downsampling processing on the second extracted feature to obtain the second encoded feature;

[0019] The Mamba module in the third encoding module extracts features from the second encoded feature to obtain the encoded action feature.

[0020] Optionally, the decoder includes: a first decoding module, a second decoding module, and a third decoding module. The first decoding module includes: a Mamba module, the second decoding module includes: a Mamba module and an upsampling module, and the third decoding module includes: a Mamba module and an upsampling module;

[0021] The process of decoding by the Mamba module in the decoder to obtain the decoded action features includes:

[0022] The Mamba module in the first decoding module decodes the encoded action features to obtain the first decoded feature, and inputs the first decoded feature and the second encoded feature into the second decoding module;

[0023] The Mamba module in the second decoding module decodes the first decoded feature and the second encoded feature to obtain the initial decoded feature, and inputs the initial decoded feature into the upsampling module. The upsampling module decodes the initial decoded feature to obtain the second decoded feature;

[0024] The Mamba module in the third decoding module decodes the second decoded feature and the first encoded feature to obtain the intermediate decoded feature, and inputs the intermediate decoded feature into the upsampling module. The upsampling module decodes the intermediate decoded feature to obtain the decoded action features.

[0025] Optionally, the Mamba module includes: a linear fusion sub-module and a hybrid sub-module;

[0026] The process of the Mamba module in the first decoding module decoding the encoded action features to obtain the first decoded feature includes:

[0027] The linear fusion sub-module performs linear fusion processing on the encoded action features to obtain the processed feature;

[0028] The hybrid sub-module extracts features from the processed feature to obtain the first decoded feature.

[0029] Optionally, the encoded action features include: action noise and conditional features. The process of the linear fusion sub-module performing linear fusion processing on the encoded action features to obtain the processed feature includes:

[0030] Extract information from the action noise to obtain the preprocessed feature;

[0031] Generate modulation parameters according to the conditional features;

[0032] Perform a linear transformation on the preprocessed feature according to the modulation parameters to obtain the processed feature.

[0033] Optionally, the hybrid sub-module includes: a Mamba block, a first multi-layer perceptron, an attention module, and a second multi-layer perceptron;

[0034] The process of the hybrid sub-module extracting features from the processed feature to obtain the first decoded feature includes:

[0035] Feature extraction is performed on the processed feature by the Mamba block to obtain a first processed feature;

[0036] Mapping processing is performed on the first processed feature by the first multi-layer perceptron to obtain a second processed feature;

[0037] Feature fusion is performed on the second processed feature by the attention module to obtain a third processed feature;

[0038] Mapping processing is performed on the third processed feature by the second multi-layer perceptron to obtain the first decoded feature.

[0039] In a second aspect, the present application provides a robot motion prediction device, and the device includes:

[0040] A generation module, configured to generate an initial action noise, obtain at least one historical action sequence in the historical action library of the target robot, and generate a conditional feature according to the historical action sequence and the point cloud data set, where the conditional feature includes: a point cloud feature, a state feature, and a time feature;

[0041] An encoding module, configured to input the initial action noise and the conditional feature into the action prediction model, perform encoding processing layer by layer by the Mamba modules in each layer of the encoder to obtain a first encoded feature, a second encoded feature, and an encoded action feature, and input the first encoded feature, the second encoded feature, and the encoded action feature into the decoder, and perform decoding processing by the Mamba module in the decoder to obtain a decoded action feature;

[0042] A prediction module, configured to input the decoded action feature into the output layer of the action prediction model to obtain a predicted action sequence, control the motion of the target robot according to the predicted action sequence, determine a state and a reward according to the actual action sequence when the target robot moves, and add the actual action sequence, the state, and the reward as a historical action information to the historical action library.

[0043] Optionally, the generation module is further configured to:

[0044] Perform encoding processing on the point cloud data set based on a point cloud encoder to obtain the point cloud feature;

[0045] Perform feature extraction on the state data in the historical action sequence based on a state encoder to obtain the state feature;

[0046] Perform feature extraction on the time data in the historical action sequence based on a time encoder to obtain the time feature.

[0047] Optionally, the encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module. The third encoding module includes: a Mamba module;

[0048] The encoding module is further configured to:

[0049] The Mamba module in the first encoding module extracts features from the initial action noise and the conditional features to obtain first extracted features, and the downsampling module performs downsampling processing on the first extracted features to obtain first encoded features;

[0050] The Mamba module in the second encoding module extracts features from the first encoded features to obtain second extracted features, and the downsampling module performs downsampling processing on the second extracted features to obtain second encoded features;

[0051] The Mamba module in the third encoding module extracts features from the second encoded features to obtain the encoded action features.

[0052] Optionally, the decoder includes a first decoding module, a second decoding module, and a third decoding module. The first decoding module includes: a Mamba module. The second decoding module includes: a Mamba module and an upsampling module. The third decoding module includes: a Mamba module and an upsampling module;

[0053] The prediction module is further configured to:

[0054] The Mamba module in the first decoding module decodes the encoded action features to obtain first decoded features, and inputs the first decoded features and the second encoded features into the second decoding module;

[0055] The Mamba module in the second decoding module decodes the first decoded features and the second encoded features to obtain initial decoded features, and inputs the initial decoded features into the upsampling module. The upsampling module decodes the initial decoded features to obtain second decoded features;

[0056] The Mamba module in the third decoding module decodes the second decoded features and the first encoded features to obtain intermediate decoded features, and inputs the intermediate decoded features into the upsampling module. The upsampling module decodes the intermediate decoded features to obtain the decoded action features.

[0057] Optionally, the Mamba module includes: a linear fusion sub-module and a mixing sub-module;

[0058] The prediction module is further configured to:

[0059] The linear fusion sub-module performs linear fusion processing on the encoded action features to obtain processed features;

[0060] The hybrid sub-module extracts features from the processed features to obtain the first decoded feature.

[0061] Optionally, the encoded action features include: action noise and conditional features, and the prediction module is further configured to:

[0062] Extract information from the action noise to obtain preprocessed features;

[0063] Generate modulation parameters according to the conditional features;

[0064] Perform a linear transformation on the preprocessed features according to the modulation parameters to obtain the processed features.

[0065] Optionally, the hybrid sub-module includes: a Mamba block, a first multi-layer perceptron, an attention module, and a second multi-layer perceptron;

[0066] The prediction module is further configured to:

[0067] The Mamba block extracts features from the processed features to obtain first processed features;

[0068] The first multi-layer perceptron performs a mapping process on the first processed features to obtain second processed features;

[0069] The attention module performs feature fusion on the second processed features to obtain third processed features;

[0070] The second multi-layer perceptron performs a mapping process on the third processed features to obtain the first decoded feature.

[0071] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of a robot action prediction method according to any one of the first aspects.

[0072] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it performs the steps of a robot action prediction method according to any one of the first aspects.

[0073] The beneficial effects of this application are as follows: By denoising the initial action noise through the action prediction model of this application, a predicted action sequence can be obtained, which can generate complex and realistic trajectories, with good flexibility and effectiveness. By implementing the encoder and decoder based on the Mamba module in the action prediction model, the number of model parameters can be effectively reduced, thereby greatly reducing the computational complexity, and the Mamba module can efficiently process long sequence data, ensuring the long-term prediction ability of the model. At the same time, this application combines point cloud data and the historical action sequence of the robot for prediction. On the premise of lower computational requirements, the action prediction model of this application can also ensure the accuracy of the model prediction results.

[0074] To make the above objects, features, and advantages of this application more obvious and understandable, the following specifically presents preferred embodiments and, in conjunction with the accompanying drawings, provides detailed descriptions as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] To more clearly illustrate the technical solutions of the embodiments of this application, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0076] Figure 1 Shows a schematic diagram of the architecture of an action prediction model provided by an embodiment of this application;

[0077] Figure 2 Shows a flowchart of a robot action prediction method provided by an embodiment of this application;

[0078] Figure 3 Shows a flowchart of extracting conditional features provided by an embodiment of this application;

[0079] Figure 4 Shows a schematic diagram of the overall architecture of an action prediction model provided by an embodiment of this application;

[0080] Figure 5 Shows a flowchart of the encoder performing encoding processing provided by an embodiment of this application;

[0081] Figure 6 Shows a flowchart of the decoder performing decoding processing provided by an embodiment of this application;

[0082] Figure 7 Shows a schematic diagram of the structure of the Mamba block in an action prediction model provided by an embodiment of this application;

[0083] Figure 8Shows a flowchart for obtaining a first decoding feature provided by an embodiment of the present application;

[0084] Figure 9 Shows a flowchart for performing linear fusion provided by an embodiment of the present application;

[0085] Figure 10 Shows a flowchart for performing linear refinement provided by an embodiment of the present application;

[0086] Figure 11 Shows a schematic structural diagram of a robot motion prediction device provided by an embodiment of the present application;

[0087] Figure 12 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0089] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the subsequently stated features, but does not exclude adding other features.

[0090] Currently, diffusion strategies are widely used in imitation learning to effectively manage multimodal action distributions.

[0091] However, diffusion strategies usually rely on large backbone architectures. Although these architectures are excellent in capturing complex details and providing high performance, the models have huge computational requirements, especially when deployed in resource-constrained environments or on edge devices where the computational resources are very limited. In addition, achieving long-term prediction ability is also very important in the scenario of robot motion prediction. Therefore, how to reduce the computational overhead while maintaining the high performance of the diffusion strategy and improving the long-term prediction ability has become an urgent problem to be solved.

[0092] Based on this, the present application proposes a robot motion prediction method. By introducing the Mamba strategy into the motion prediction model, the number of parameters of the model can be significantly reduced, and better performance can be achieved. On the premise of ensuring the prediction ability of the model, the computational overhead is effectively reduced, ensuring the high performance of the model.

[0093] The method of the present application can be applied to Figure 1 the motion prediction model shown in Figure 1 . The motion prediction model includes an encoder and a decoder. Among them, the encoder and the decoder are of U-net structure, and at least one Mamba module is included in both the encoder and the decoder. The motion sequence of the past T0 time steps is input into the motion prediction model, and the motion prediction model can predict and output the motion sequence of the future T time steps.

[0094] Next, in combination with Figure 2 , the robot motion prediction method of the present application will be described. The execution subject of this method can be the target robot deployed with Figure 1 the motion prediction model shown in Figure 2 . As shown in

[0095] , during the movement of the target robot, the following steps are iteratively executed:

[0096] S201. Generate initial action noise, obtain at least one historical action sequence in the historical action library of the target robot, and generate conditional features according to the historical action sequence and the point cloud data set. The conditional features include: point cloud features, state features, and time features.

[0097] The robot can collect point cloud data through a lidar. The point cloud data can be the perception data of the robot for the surrounding environment, including the point cloud data of surrounding obstacles and surrounding paths. The point cloud features can be extracted by encoding the point cloud data.

[0098] When the robot executes an action, the state of the robot after executing the action can be recorded, and the action and the corresponding state are added to the historical action library as a piece of historical action information. When predicting future actions, the continuous historical action information within the past T0 time steps can be obtained from the historical action library in chronological order as the historical action sequence. The historical action information can include the head action and state of the robot, the upper limb action and state, and the lower limb action and state.

[0099] As a possible implementation, the actions in the historical action information can also indicate the actions of the head, upper limbs, and lower limbs of the target robot. In this case, the state includes the state of the target robot and the environmental state. Among them, the state of the target robot can be the pose of the head, the pose of the upper limbs, and the pose of the lower limbs of the target robot, and the environmental state is used to indicate the position of the target robot in the working environment, the terrain of the working environment, the position information of surrounding obstacles, the workable area, etc.

[0100] Among them, after the action prediction model denoises the initial action noise, a predicted action sequence can be obtained. The initial action noise can be directly sampled from a standard normal distribution or randomly generated through a Gaussian distribution.

[0101] S202: Input the initial action noise and conditional features into the action prediction model, and perform encoding processing layer by layer by the Mamba modules in each layer of the encoder to obtain the first encoded feature, the second encoded feature, and the encoded action feature, and input the first encoded feature, the second encoded feature, and the encoded action feature into the decoder, and perform decoding processing by the Mamba module in the decoder to obtain the decoded action feature.

[0102] Optionally, both the encoder and the decoder include Mamba modules. The Mamba module in the encoder can perform encoding processing on the initial action noise and conditional features to extract the first encoded feature, the second encoded feature, and the encoded action feature. Among them, the encoder can include multiple encoding modules, and each encoding module includes a Mamba module. Each encoding module extracts features layer by layer. For example, the encoder includes three encoding modules connected layer by layer. The first-layer encoding module performs encoding processing on the initial action noise and conditional features to obtain the first encoded feature, and inputs the first encoded feature into the second-layer encoding module for encoding processing to obtain the second encoded feature. The last-layer encoding module performs encoding processing on the second encoded feature to obtain the encoded action feature.

[0103] It should be noted that during the downsampling process, there may be a problem of spatial information loss. Based on this, when the encoding module in the encoder performs feature encoding, it can skip-connect the first encoded feature and the second encoded feature output by each layer of the encoding module to the decoder, so as to integrate features at different levels, improve the segmentation accuracy, and take into account the local details and global semantics of the features.

[0104] The decoder can include multiple decoding modules corresponding to the encoding modules in the encoder. Each decoding module includes a Mamba module. The decoding module can perform decoding processing on the first encoded feature, the second encoded feature, and the encoded action feature output by the encoder to obtain the decoded action feature.

[0105] S203. Input the decoded action features into the output layer of the action prediction model to obtain a predicted action sequence. Control the movement of the target robot according to the predicted action sequence, determine the state and reward based on the actual action sequence during the movement of the target robot, and add the actual action sequence and state as a piece of historical action information to the historical action library.

[0106] The output of the last layer decoding module of the decoder is connected to the output layer. The output layer can perform convolutional processing on the decoding result of the decoding module, thereby mapping the number of channels to the number of segmentation categories to obtain a predicted action sequence. The predicted action sequence can be multiple actions that the robot needs to execute within the next T time steps, including head actions, upper limb actions, lower limb actions, etc. that the robot needs to execute.

[0107] After obtaining the predicted action sequence, the controller of the robot can control the target robot to move according to the predicted action sequence, record the state of the target robot after each action is executed, and add the action and state as a piece of historical action information to the historical action library, so that the target robot can continue to perform the next round of action prediction based on the historical action information in the historical action library.

[0108] In the embodiments of the present application, the initial action noise is denoised through the action prediction model of the present application to obtain a predicted action sequence, which can generate complex and realistic trajectories, and has good flexibility and effectiveness. By implementing the encoder and decoder based on the Mamba module in the action prediction model, the number of parameters of the model can be effectively reduced, thereby greatly reducing the computational complexity, and the Mamba module can efficiently process long sequence data, ensuring the long-term prediction ability of the model. At the same time, the present application combines point cloud data and the historical action sequence of the robot for prediction. On the premise of lower computational requirements, the action prediction model of the present application can also ensure the accuracy of the model prediction results.

[0109] The following is a further description of generating conditional features based on the above historical action sequence and point cloud data set, as Figure 3 shown. The above S201 step includes:

[0110] S301. Perform encoding processing on the point cloud data set based on the point cloud encoder to obtain point cloud features.

[0111] Among them, the point cloud data set includes the point cloud data of obstacles in the working environment. By performing encoding processing on the point cloud data set based on the point cloud encoder, the target robot can perceive the working environment based on the obtained point cloud features.

[0112] S302. Extract features from the state data in the historical action sequence based on the state encoder to obtain state features.

[0113] S303. Extract features from the time data in the historical action sequence based on a time encoder to obtain time features.

[0114] In a possible implementation, the point cloud encoder, the state encoder, and the time encoder can be DP3 (Diffusion Policy 3, 3D diffusion policy) encoders. The DP3 encoder includes a three-layer MLP (Multi-Layer Perceptron) network, a max pooling function, and a projection head for projecting features into a compact vector.

[0115] By encoding the point cloud dataset through the point cloud encoder, the original loose point cloud can be converted into a compact point cloud, retaining key spatial information while enhancing the model's spatial generalization ability for the working area.

[0116] Figure 4 It is a schematic diagram of the architecture of an action prediction model given in this application. The encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module. The third encoding module includes: a Mamba module.

[0117] As Figure 5 shown, the process of encoding the initial action noise and conditional features by the Mamba module in the encoder to obtain the first encoded feature, the second encoded feature, and the encoded action feature includes:

[0118] S501. The Mamba module in the first encoding module extracts features from the initial action noise and conditional features to obtain the first extracted feature, and the downsampling module downsamples the first extracted feature to obtain the first encoded feature.

[0119] Referring to Figure 4 , the first encoding module includes two Mamba modules and a downsampling module. The Mamba module can extract features from the initial action noise and conditional features to obtain the first extracted feature. The first extracted feature includes: the action noise and conditional features processed by the Mamba module.

[0120] After the Mamba module extracts features, the first extracted feature can be input into the downsampling module for downsampling to obtain the first encoded feature. The first encoded feature includes: the action noise after downsampling and the conditional features after downsampling.

[0121] S502. The Mamba module in the second encoding module extracts features from the first encoded feature to obtain the second extracted feature, and the downsampling module downsamples the second extracted feature to obtain the second encoded feature.

[0122] The second encoding module includes two Mamba modules and one downsampling module. The Mamba module can perform feature extraction on the initial action noise and conditional features to obtain the second extracted features. The second extracted features include: the action noise and conditional features processed by the Mamba module.

[0123] After the Mamba module performs feature extraction, the second extracted features can be input into the downsampling module for downsampling processing to obtain the second encoded features. The second encoded features include: the action noise after downsampling processing and the conditional features after downsampling processing.

[0124] S503. The Mamba module in the third encoding module performs feature extraction on the second encoded features to obtain the encoded action features.

[0125] Continue to refer to Figure 4 , the third encoding module includes a Mamba module. The action noise and conditional features in the second encoded features are input into the Mamba module in the third encoding module. The Mamba module can perform feature extraction to obtain the encoded action features. The encoded action features include: the action noise and conditional features processed by the Mamba module in the third encoding module.

[0126] Refer to Figure 4 , the decoder includes: a first decoding module, a second decoding module, and a third decoding module. The first decoding module includes: a Mamba module. The second decoding module includes: a Mamba module and an upsampling module. The third decoding module includes: a Mamba module and an upsampling module.

[0127] As Figure 6 shown, the process of performing decoding processing by the Mamba module in the decoder to obtain the decoded action features includes:

[0128] S601. The Mamba module in the first decoding module performs decoding processing on the encoded action features to obtain the first decoded features, and inputs the first decoded features and the second encoded features into the second decoding module.

[0129] Optionally, the first decoding module includes a Mamba module. After inputting the action noise and conditional features in the encoded action features into the Mamba module in the first decoding module, the Mamba module can combine the action noise and conditional features to obtain the first decoded features.

[0130] S602. The Mamba module in the second decoding module performs decoding processing on the first decoded features and the second encoded features to obtain the initial decoded features, and inputs the initial decoded features into the upsampling module. The upsampling module performs decoding processing on the initial decoded features to obtain the second decoded features.

[0131] Optionally, the second decoding module includes a Mamba module and an upsampling module. The Mamba module can take the first decoded feature and the second encoded feature as inputs and decode to obtain an initial decoded feature.

[0132] S603. The Mamba module in the third decoding module decodes the second decoded feature and the first encoded feature to obtain an intermediate decoded feature, and inputs the intermediate decoded feature into the upsampling module. The upsampling module decodes the intermediate decoded feature to obtain a decoded action feature.

[0133] The third decoding module includes a Mamba module and an upsampling module. The Mamba module can take the second decoded feature and the first encoded feature as inputs and decode to obtain an intermediate decoded feature.

[0134] It should be noted that during the downsampling process of the encoder, there may be a problem of loss of spatial information. Based on this, the second encoded feature output by the above second encoding module can be used as the input information of the second decoding module, and the first encoded feature output by the above first encoding module can be used as the input information of the third decoding module, so as to integrate features at different levels, improve the segmentation accuracy, and take into account the local details and global semantics of the features.

[0135] It is worth noting that the number of Mamba modules in the encoder and decoder, as well as the number of levels of the encoder and decoder, can be determined based on actual needs. Figure 4 Only a possible implementation is given, and it should not be limited to this.

[0136] Figure 7 It is a schematic diagram of the internal structure of a Mamba module. Refer to Figure 7 , the Mamba module includes: a linear fusion sub-module and a hybrid sub-module.

[0137] As Figure 8 shown, the process of the Mamba module in the first encoding module extracting features from the encoded action feature to obtain the first decoded feature includes:

[0138] S801. The linear fusion sub-module performs linear fusion processing on the encoded action feature to obtain a processed feature.

[0139] Among them, the encoded action feature includes action noise and conditional features. The linear fusion sub-module can perform linear fusion processing on the action noise and conditional features based on the feature-level linear modulation method to obtain a processed feature.

[0140] S802. The hybrid sub-module extracts features from the processed feature to obtain the first decoded feature.

[0141] Optionally, the mixing sub-module can further refine and fuse the processed features to obtain the first decoded feature. Among them, the mixing sub-module can be implemented by combining a Mamba module with an attention module, which can not only effectively reduce the computational cost through the Mamba module, but also supplement the global information by combining the attention module to achieve a balance between computational efficiency and model performance.

[0142] As Figure 9 shown, the process of linearly fusing the initial action noise and the conditional feature by the linear fusion sub-module to obtain the processed feature includes:

[0143] S901. Extract information from the action noise to obtain the preprocessed feature.

[0144] Among them, extracting information from the action noise includes: performing one-dimensional convolution processing on the action noise, normalizing the convolved action noise, and finally performing activation processing on the normalized action noise to obtain the preprocessed feature.

[0145] S902. Generate modulation parameters according to the conditional feature.

[0146] Optionally, the conditional feature can be non-linearly transformed through an activation function, then the transformed conditional feature can be normalized, and finally the normalized conditional feature can be split to obtain two modulation parameters. The splitting process can be implemented through a fully connected layer or a feature segmentation function.

[0147] S903. Perform a linear transformation on the preprocessed feature according to the modulation parameter to obtain the processed feature.

[0148] In a possible implementation, the modulation parameter can be used as the parameter of the linear transformation, the preprocessed feature can be used as the independent variable, and the operation result of the modulation parameter and the independent variable can be used as the processed feature.

[0149] In this application, dynamically generating modulation parameters based on conditional features can achieve adaptive adjustment of the feature space. Incorporating conditional features into action noise through linear transformation can enhance the model's adaptability to multi-modal or dynamic scenarios.

[0150] Continuing to refer to Figure 7 , the mixing sub-module includes: a Mamba block, a first multi-layer perceptron, an attention module, and a second multi-layer perceptron.

[0151] As Figure 10 shown, the process of extracting features from the processed feature by the mixing sub-module to obtain the first decoded feature includes:

[0152] S1001. Extract features from the processed feature by the Mamba block to obtain the first processed feature.

[0153] S1002. The first multi-layer perceptron performs a mapping process on the first processed feature to obtain a second processed feature.

[0154] S1003. The attention module performs feature fusion on the second processed feature to obtain a third processed feature.

[0155] S1004. The second multi-layer perceptron performs a mapping process on the third processed feature to obtain a first decoded feature.

[0156] Optionally, the Mamba block can be a standard Mamba block. After obtaining the second processed feature, the second processed feature and the processed feature can be concatenated, and the concatenated feature can be used as the new second processed feature. The attention module performs feature fusion on the new second processed feature to obtain a third processed feature.

[0157] Optionally, after obtaining the third processed feature, the third processed feature and the new second processed feature can be concatenated, and the concatenated feature can be used as the first decoded feature.

[0158] In the embodiments of the present application, by performing skip connections on the processed feature and the second processed feature, and performing skip connections on the obtained new second processed feature and the third processed feature, multi-level feature information can be retained, enhancing the robustness and adaptability of the model.

[0159] Based on the same inventive concept, in the embodiments of the present application, there is also provided a robot motion prediction device corresponding to the robot motion prediction method. Since the principle of solving problems by the device in the embodiments of the present application is similar to that of the above-mentioned robot motion prediction method in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0160] Figure 11 Fig. shows a schematic structural diagram of a robot motion prediction device provided by an embodiment of the present application.

[0161] A generation module 1101, configured to generate initial action noise, obtain at least one historical action sequence from the historical action library of the target robot, and generate conditional features according to the historical action sequence and the point cloud data set. The conditional features include: point cloud features, state features, and time features.

[0162] An encoding module 1102, configured to input the initial action noise and the conditional features into an action prediction model, and perform encoding processing layer by layer by the Mamba blocks in each layer of the encoder to obtain a first encoded feature, a second encoded feature, and an encoded action feature, and input the first encoded feature, the second encoded feature, and the encoded action feature into the decoder, and perform decoding processing by the Mamba block in the decoder to obtain a decoded action feature.

[0163] The prediction module 1103 is configured to input the decoded action features into the output layer of the action prediction model to obtain a predicted action sequence, control the movement of the target robot according to the predicted action sequence, determine the state and reward based on the actual action sequence during the movement of the target robot, and add the actual action sequence, state, and reward as a historical action information to the historical action library.

[0164] Optionally, the generation module 1101 is further configured to:

[0165] Encode the point cloud dataset based on the point cloud encoder to obtain point cloud features;

[0166] Extract features from the state data in the historical action sequence based on the state encoder to obtain state features;

[0167] Extract features from the time data in the historical action sequence based on the time encoder to obtain time features.

[0168] Optionally, the encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module. The third encoding module includes: a Mamba module;

[0169] The encoding module 1102 is further configured to:

[0170] Extract features from the initial action noise and conditional features by the Mamba module in the first encoding module to obtain first extracted features, and perform downsampling processing on the first extracted features by the downsampling module to obtain first encoded features;

[0171] Extract features from the first encoded features by the Mamba module in the second encoding module to obtain second extracted features, and perform downsampling processing on the second extracted features by the downsampling module to obtain second encoded features;

[0172] Extract features from the second encoded features by the Mamba module in the third encoding module to obtain encoded action features.

[0173] Optionally, the decoder includes: a first decoding module, a second decoding module, and a third decoding module. The first decoding module includes: a Mamba module. The second decoding module includes: a Mamba module and an upsampling module. The third decoding module includes: a Mamba module and an upsampling module;

[0174] The prediction module 1103 is further configured to:

[0175] Decode the encoded action features by the Mamba module in the first decoding module to obtain first decoded features, and input the first decoded features and the second encoded features into the second decoding module;

[0176] The Mamba module in the second decoding module decodes the first decoding feature and the second encoding feature to obtain an initial decoding feature, and inputs the initial decoding feature into the upsampling module, and the upsampling decodes the initial decoding feature to obtain a second decoding feature;

[0177] The Mamba module in the third decoding module decodes the second decoding feature and the first encoding feature to obtain an intermediate decoding feature, and inputs the intermediate decoding feature into the upsampling module, and the upsampling module decodes the intermediate decoding feature to obtain a decoded action feature.

[0178] Optionally, the Mamba module includes: a linear fusion sub-module and a mixing sub-module;

[0179] The prediction module 1103 is further configured to:

[0180] The linear fusion sub-module linearly fuses the encoded action feature to obtain a processed feature;

[0181] The mixing sub-module extracts features from the processed feature to obtain a first decoding feature.

[0182] Optionally, the encoded action feature includes: action noise and conditional features, and the prediction module is further configured to:

[0183] Extract information from the action noise to obtain a preprocessed feature;

[0184] Generate modulation parameters according to the conditional features;

[0185] Perform a linear transformation on the preprocessed feature according to the modulation parameters to obtain a processed feature.

[0186] Optionally, the mixing sub-module includes: a Mamba block, a first multi-layer perceptron, an attention module, and a second multi-layer perceptron;

[0187] The prediction module 1103 is further configured to:

[0188] The Mamba block extracts features from the processed feature to obtain a first processed feature;

[0189] The first multi-layer perceptron maps the first processed feature to obtain a second processed feature;

[0190] The attention module fuses features of the second processed feature to obtain a third processed feature;

[0191] The second multi-layer perceptron maps the third processed feature to obtain a first decoding feature.

[0192] Figure 12The schematic structural diagram of an electronic device provided by an embodiment of the present application is shown, including: a processor 1201, a storage medium 1202, and a bus 1203. The storage medium 1202 stores machine-readable instructions executable by the processor 1201. When the electronic device runs a robot action prediction method as in the embodiment, the processor 1201 communicates with the storage medium 1202 through the bus 1203. The processor 1201 executes the machine-readable instructions, which are the preamble part of the method item of the processor 1201, to execute the steps of the above-mentioned robot action prediction method.

[0193] An embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on this computer-readable storage medium and is executed when run by a processor. The processor executes the steps of the above-mentioned robot action prediction method.

[0194] In an embodiment of the present application, when the computer program is run by a processor, it can also execute other machine-readable instructions to execute other methods described in the embodiment. For the specific method steps and principles of execution, refer to the description of the embodiment, and details will not be elaborated here.

[0195] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0196] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in the embodiments provided by the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0198] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0199] It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0200] Finally, it should be noted that: the above-mentioned embodiments are only specific implementation manners of this application, used to illustrate the technical solution of this application, rather than limiting it. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed in this application can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A robot motion prediction method, characterized in that, Applied to a target robot deployed with an action prediction model, the action prediction model includes: an encoder, a decoder, and an output layer. Both the encoder and the decoder are of U-net structure. The encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module. The third encoding module includes: a Mamba module. The decoder includes a first decoding module, a second decoding module, and a third decoding module connected in sequence. The first decoding module includes: a Mamba module. Both the second decoding module and the third decoding module include: a Mamba module and an upsampling module; The method includes: During the movement of the target robot, the following steps are iteratively executed: A. Generate initial action noise, obtain at least one historical action sequence in the historical action library of the target robot, and generate conditional features according to the historical action sequence and the point cloud data set. The conditional features include: point cloud features, state features, and time features; B. Input the initial action noise and the conditional features into the action prediction model. The Mamba module in the first encoding module extracts features from the initial action noise and the conditional features to obtain first extracted features, and the downsampling module downsamples the first extracted features to obtain first encoded features; The Mamba module in the second encoding module extracts features from the first encoded features to obtain second extracted features, and the downsampling module downsamples the second extracted features to obtain second encoded features; The Mamba module in the third encoding module extracts features from the second encoded features to obtain encoded action features, and inputs the first encoded features, second encoded features, and encoded action features into the decoder. The Mamba module in the decoder performs decoding processing to obtain decoded action features; C. Input the decoded action features into the output layer of the action prediction model. The output layer performs convolution processing on the decoded action features to map the number of channels to the number of segmentation categories, obtaining a predicted action sequence. Control the movement of the target robot according to the predicted action sequence, determine the state and reward according to the actual action sequence when the target robot moves, and add the actual action sequence, state, and reward as a historical action information to the historical action library.

2. The method according to claim 1, characterized in that, The generating conditional features according to the historical action sequence and the point cloud data set includes: Encoding the point cloud data set based on a point cloud encoder to obtain the point cloud features; Extracting features from the state data in the historical action sequence based on a state encoder to obtain the state features; Extracting features from the time data in the historical action sequence based on a time encoder to obtain the time features.

3. The method according to claim 1, characterized in that The process of obtaining decoded action features by the Mamba module in the decoder includes: The Mamba module in the first decoding module decodes the encoded action feature to obtain a first decoded feature, and inputs the first decoded feature and a second encoded feature into the second decoding module; The Mamba module in the second decoding module decodes the first decoded feature and the second encoded feature to obtain an initial decoded feature, and inputs the initial decoded feature into the upsampling module. The upsampling module decodes the initial decoded feature to obtain a second decoded feature; The Mamba module in the third decoding module decodes the second decoded feature and a first encoded feature to obtain an intermediate decoded feature, and inputs the intermediate decoded feature into the upsampling module. The upsampling module decodes the intermediate decoded feature to obtain the decoded action feature.

4. The method according to claim 3, characterized in that The Mamba module includes: a linear fusion sub-module and a mixing sub-module; The process of the Mamba module in the first decoding module decoding the encoded action feature to obtain a first decoded feature includes: The linear fusion sub-module performs linear fusion processing on the encoded action feature to obtain a processed feature; The mixing sub-module extracts features from the processed feature to obtain the first decoded feature.

5. The method according to claim 4, characterized in that, The encoded action feature includes: action noise and conditional features. The process of the linear fusion sub-module performing linear fusion processing on the encoded action feature to obtain a processed feature includes: Extracting information from the action noise to obtain a preprocessed feature; Generating a modulation parameter according to the conditional features; Performing a linear transformation on the preprocessed feature according to the modulation parameter to obtain the processed feature.

6. The method according to claim 4, wherein The mixing sub-module includes: a Mamba block, a first multi-layer perceptron, an attention module, and a second multi-layer perceptron; The process of the mixing sub-module extracting features from the processed feature to obtain the first decoded feature includes: The Mamba block extracts features from the processed feature to obtain a first processed feature; The first multi-layer perceptron performs a mapping process on the first processed feature to obtain a second processed feature; The attention module performs feature fusion on the second processed feature to obtain a third processed feature; The second multi-layer perceptron performs a mapping process on the third processed feature to obtain the first decoded feature.

7. A robot motion prediction device, characterized in that, Applied to a target robot deployed with an action prediction model, the action prediction model includes: an encoder, a decoder, and an output layer. Both the encoder and the decoder are of U-net structure, and the encoder includes a first encoding module, a second encoding module, and a third encoding module connected in sequence. Both the first encoding module and the second encoding module include: a Mamba module and a downsampling module. The third encoding module includes: a Mamba module. The decoder includes a first decoding module, a second decoding module, and a third decoding module connected in sequence. The first decoding module includes: a Mamba module. Both the second decoding module and the third decoding module include: a Mamba module and an upsampling module. The device includes: A generation module, configured to generate initial action noise, obtain at least one historical action sequence from the historical action library of the target robot, and generate conditional features based on the historical action sequence and the point cloud data set, where the conditional features include: point cloud features, state features, and time features; An encoding module, configured to input the initial action noise and the conditional features into an action prediction model. The Mamba module in the first encoding module extracts features from the initial action noise and the conditional features to obtain first extracted features, and the downsampling module performs downsampling processing on the first extracted features to obtain first encoded features; the Mamba module in the second encoding module extracts features from the first encoded features to obtain second extracted features, and the downsampling module performs downsampling processing on the second extracted features to obtain second encoded features; the Mamba module in the third encoding module extracts features from the second encoded features to obtain encoded action features, and inputs the first encoded features, second encoded features, and encoded action features into a decoder. The Mamba module in the decoder performs decoding processing to obtain decoded action features; A prediction module, configured to input the decoded action features into the output layer of the action prediction model. The output layer performs convolutional processing on the decoded action features to map the number of channels to the number of segmentation categories, obtain a predicted action sequence, control the movement of the target robot according to the predicted action sequence, determine the state and reward based on the actual action sequence when the target robot moves, and add the actual action sequence, state, and reward as a piece of historical action information to the historical action library.

8. An electronic device, characterized in that, Comprising: A processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus. The processor executes the machine-readable instructions to perform the steps of a robot action prediction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by the processor, it performs the steps of a robot action prediction method according to any one of claims 1 to 6.

Images