Robot action prediction method, device, equipment and storage medium

Abstract

The application provides a robot action prediction method and device, equipment and a storage medium, wherein the method comprises: extracting global features and local features of an action sequence based on a reinforcement learning action prediction model, and predicting a predicted action sequence based on the extracted global features and local features. By extracting the global features and local features of the action sequence respectively, the continuity of the historical action sequence in time can be concerned while focusing on the local correlation of the action sequence. By extracting the features through the Mamba module, the selectivity and context awareness of the model can be further enhanced, thereby improving the performance of the model. In addition, since the Mamba module has a significant lightweight advantage, the action prediction model improved based on the Mamba module also has the advantages of being lightweight and easy to deploy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and storage medium for predicting robot motions. Background Technology

[0002] In robot reinforcement learning, the decision-making process generally involves learning the mapping from state observation to action, thereby maximizing the cumulative discount reward. In offline reinforcement learning, a sequence modeling paradigm can be established through a decision transformer, thereby predicting the robot's future action sequence under the premise of achieving the expected reward, based on the sequence of past rewards, states, and actions.

[0003] However, sequence modeling of robot reinforcement learning trajectories differs from traditional sequence modeling. Specifically, in reinforcement learning problems, the next state depends only on the current state and action, and is independent of past states. Furthermore, due to the continuity of time steps, the features of each step are related to long-term historical information. When performing sequence modeling of robot reinforcement learning trajectories based on existing technologies, the information in historical action sequences is not fully utilized, resulting in poor reliability and robustness in action prediction. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a robot motion prediction method, apparatus, device, and storage medium, so as to solve the problem that the prior art does not make full use of information in historical motion sequences, resulting in poor reliability and robustness of motion prediction.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] In a first aspect, this application provides a robot action prediction method applied to a target robot equipped with a reinforcement learning action prediction model. The reinforcement learning action prediction model includes an encoder, a feature extractor, and a decoder. The feature extractor includes at least a global feature extraction module, a local feature extraction module, and a feature fusion module. The global feature extraction module and the local feature extraction module each include at least one Mamba module. The method includes:

[0007] Detect whether the target robot has started. If so, iteratively execute the following steps:

[0008] A. Obtain at least one historical action information from the target robot's historical action library, input the historical action information into the encoder to obtain an action vector sequence, the action vector sequence including at least one action vector, the action vector including: state element, action element and reward element;

[0009] B. Input the action vector sequence into the feature extractor, where the Mamba module in the global feature extraction module extracts global features based on the action vector sequence, and the local feature extraction module extracts local features from the action vector sequence based on the Mamba module in the local feature extraction module to obtain at least one local feature.

[0010] C. Input the global features and each of the local features into the feature fusion module to obtain the target features;

[0011] D. Input the target features into the decoder, and the decoder predicts the robot's predicted action sequence. Control the target robot's movement according to the predicted action sequence, and determine the state and reward according to the actual action sequence of the target robot's movement. Add the actual action sequence, state, and reward as historical action information to the historical action library.

[0012] Optionally, inputting the historical motion information of the target robot into the encoder to obtain a motion vector sequence includes:

[0013] The historical action information is sampled and processed to obtain multiple consecutive triplet information of the target robot. The triplet information includes: state information, action information and reward information.

[0014] The encoder inputs the multiple consecutive triplet information, and the encoder performs vectorization processing on each triplet information to obtain the action vector sequence.

[0015] Optionally, the local feature extraction module further includes a segmentation module and a transformation module;

[0016] The process by which the local feature extraction module extracts at least one local feature from the action vector sequence includes:

[0017] The segmentation module segments the action vector sequence to obtain at least one action vector sub-sequence corresponding to the action vector sequence.

[0018] The Mamba module extracts features from each of the action vector sub-sequences to obtain at least one initial feature;

[0019] The conversion module performs conversion processing on each of the initial features to obtain at least one local feature.

[0020] Optionally, the process of segmenting the action vector sequence by the segmentation module to obtain at least one action vector sub-sequence corresponding to the action vector sequence includes:

[0021] The segmentation module fills the end of the action vector sequence with a preset value until the length of the action vector sequence reaches N, thus obtaining the filled action vector sequence.

[0022] The segmentation module performs segmentation processing on the padded action vector sequence based on a preset segmentation length to obtain at least one action vector subsequence of the action vector sequence.

[0023] Optionally, the local feature extraction module further includes: a regularization module;

[0024] The step of inputting the global features and each of the local features into the feature fusion module includes:

[0025] The at least one local feature is input into the regularization module to shut down the neuron and obtain the processed local feature;

[0026] The global features and each processed local feature are input into the feature fusion module.

[0027] Optionally, the feature fusion module includes: a first connection layer and a feedforward network layer;

[0028] The step of inputting the global features and each of the local features into the feature fusion module to obtain the target features includes:

[0029] The global features and the local features are input into the first connection layer, which performs connection processing to obtain the connected features.

[0030] The connected features are input into the feedforward network layer, which performs transformation processing to obtain the transformed features.

[0031] The transformed features and the concatenated features are concatenated to obtain the target features.

[0032] Optionally, the action prediction model further includes: a linear module, which includes: a normalization layer, a first linear layer, an activation layer, a second linear layer, and a second connection layer;

[0033] Inputting the target features into the decoder includes:

[0034] The target features are input into the linear module, which performs normalization, linearization, activation, and connection processing sequentially to obtain the processed target features, which are then input into the decoder.

[0035] Secondly, this application provides a robot motion prediction device, the device comprising:

[0036] The encoding module is used to obtain at least one historical action information from the historical action library of the target robot, input the historical action information into the encoder to obtain an action vector sequence, the action vector sequence including at least one action vector, the action vector including: state element, action element and reward element;

[0037] An extraction module is used to input the action vector sequence into a feature extractor, where the Mamba module in the global feature extraction module extracts global features based on the action vector sequence, and the local feature extraction module performs local feature extraction on the action vector sequence based on the Mamba module in the local feature extraction module to obtain at least one local feature.

[0038] The fusion module is used to input the global features and each of the local features into the feature fusion module to obtain the target features;

[0039] The prediction module is used to input the target features into the decoder, which predicts the robot's predicted action sequence, controls the target robot's movement according to the predicted action sequence, determines the state and reward according to the actual action sequence of the target robot during movement, and adds the actual action sequence, state, and reward as historical action information to the historical action library.

[0040] Optionally, the encoding module is used for:

[0041] The historical action information is sampled and processed to obtain multiple consecutive triplet information of the target robot. The triplet information includes: state information, action information and reward information.

[0042] The encoder inputs the multiple consecutive triplet information, and the encoder performs vectorization processing on each triplet information to obtain the action vector sequence.

[0043] Optionally, the local feature extraction module further includes a segmentation module and a transformation module;

[0044] The extraction module is also used for:

[0045] The segmentation module segments the action vector sequence to obtain at least one action vector sub-sequence corresponding to the action vector sequence.

[0046] The Mamba module extracts features from each of the action vector sub-sequences to obtain at least one initial feature;

[0047] The conversion module performs conversion processing on each of the initial features to obtain at least one local feature.

[0048] Optionally, the extraction module is further configured to:

[0049] The segmentation module fills the end of the action vector sequence with a preset value until the length of the action vector sequence reaches N, thus obtaining the filled action vector sequence.

[0050] The segmentation module performs segmentation processing on the padded action vector sequence based on a preset segmentation length to obtain at least one action vector subsequence of the action vector sequence.

[0051] Optionally, the local feature extraction module further includes: a regularization module;

[0052] The fusion module is also used for:

[0053] The at least one local feature is input into the regularization module to shut down the neuron and obtain the processed local feature;

[0054] The global features and each processed local feature are input into the feature fusion module.

[0055] Optionally, the feature fusion module includes: a first connection layer and a feedforward network layer;

[0056] The fusion module is also used for:

[0057] The global features and the local features are input into the first connection layer, which performs connection processing to obtain the connected features.

[0058] The connected features are input into the feedforward network layer, which performs transformation processing to obtain the transformed features.

[0059] The transformed features and the concatenated features are concatenated to obtain the target features.

[0060] Optionally, the action prediction model further includes: a linear module, which includes: a normalization layer, a first linear layer, an activation layer, a second linear layer, and a second connection layer;

[0061] The prediction module is also used for:

[0062] The target features are input into the linear module, which performs normalization, linearization, activation, and connection processing sequentially to obtain the processed target features, which are then input into the decoder.

[0063] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of a robot motion prediction method as described in any one of the first aspects.

[0064] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of a robot motion prediction method as described in any one of the first aspects.

[0065] The beneficial effects of this application are: by extracting features from both global and local features of the action sequence, it is possible to consider both the local correlations of the action sequence and the temporal continuity of historical action sequences. Feature extraction using the Mamba module further enhances the model's selectivity and context awareness, thereby improving model performance. Furthermore, due to the significant lightweight advantage of the Mamba module, the action prediction model improved based on the Mamba module in this application also boasts the advantages of being lightweight and easy to deploy.

[0066] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A schematic diagram of the architecture of a reinforcement learning action prediction model provided in an embodiment of this application is shown;

[0069] Figure 2 A flowchart of a robot motion prediction method provided in an embodiment of this application is shown;

[0070] Figure 3 This application provides a flowchart of an embodiment for obtaining an action vector sequence.

[0071] Figure 4 This document illustrates a flowchart of a method for extracting local features according to an embodiment of this application.

[0072] Figure 5 A flowchart illustrating a segmentation of an action vector sequence provided in an embodiment of this application is shown;

[0073] Figure 6 This illustration shows a schematic diagram of the architecture of another reinforcement learning action prediction model provided in an embodiment of this application;

[0074] Figure 7 This document illustrates a flowchart of a method for regularizing local features according to an embodiment of this application.

[0075] Figure 8 This document illustrates a flowchart of a feature fusion method provided in an embodiment of this application.

[0076] Figure 9 This paper shows a schematic diagram of the structure of a robot motion prediction device provided in an embodiment of this application;

[0077] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0079] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0080] After being introduced into reinforcement learning, the decision transformer enables the prediction of future action sequences based on the robot's historical action sequences, states, and past rewards. The decision transformer can reinterpret the decision-making process as a sequence modeling problem, treating the robot's reinforcement learning process as a sequence modeling task, learning a state-action mapping based on reward conditions, and constructing a prediction model of the robot's actions.

[0081] However, due to the unique characteristics of reinforcement learning in robots, directly treating reinforcement learning trajectories as a sequence and modeling them using traditional methods presents many problems. Specifically, there are significant local correlations between steps in the trajectory sequence; the probability of transitioning to the next state depends only on the current state and action, thus requiring attention to these local correlations. Furthermore, since the time steps are continuous, the features of each step are related to long-term historical information, indicating that reinforcement learning trajectories also possess internal global correlations.

[0082] In summary, effectively capturing both global and local features in enhanced trajectory sequences becomes the key to improving the performance of action prediction models.

[0083] Based on this, this application proposes a robot action prediction method. By extracting global and local features from the trajectory sequence using the Mamba module, the correlation within the reinforcement learning trajectory can be better understood, thereby effectively improving the model performance of the action prediction model.

[0084] The method described in this application can be applied to target robots equipped with reinforcement learning action prediction models, with the specific execution entity being the control device within the target robot. The reinforcement learning action prediction model is used to extract features and predict actions based on the historical action sequences of various parts of the robot within time T0, thereby generating the robot's action sequences within the future time T.

[0085] Figure 1 This is a schematic diagram of the model architecture of a reinforcement learning action prediction model provided in this application, with reference to... Figure 1 The reinforcement learning action prediction model includes an encoder, a feature extractor, and a decoder. The feature extractor includes at least a global feature extraction module, a local feature extraction module, and a feature fusion module. The global and local feature extraction modules each include at least one Mamba module. The Mamba module possesses efficient sequence modeling capabilities and strong multi-scale dependency capture capabilities, thus ensuring that both global and local features in the input action trajectory are fully utilized, thereby improving model performance.

[0086] Next, combine Figure 2 The robot motion prediction method of this application is described below, with reference to... Figure 2 The method includes:

[0087] Detect whether the target robot has started. If so, iteratively execute the following steps:

[0088] S201. Obtain at least one historical action information from the target robot's historical action database, input the historical action information into the encoder to obtain an action vector sequence. The action vector sequence includes at least one action vector, and the action vector includes: a state element, an action element, and a reward element.

[0089] After the robot is started, its actions, status, and responses can be collected in real time, and the actions, status, and responses at each moment can be stored in the historical action library as historical action information for that moment.

[0090] The robot's actions include the output signals of the robot's actuators, and the robot's state can be the environmental state and its own state under the current action. The environmental state includes information about surrounding obstacles, map information, etc., and the own state includes position information and posture information, etc.

[0091] In this application, k consecutive historical action information from the past can be obtained, and the historical action information is input into an encoder for encoding processing to obtain an action vector sequence. For example, the action vector sequence can be represented as equation (1), which includes multiple action vectors, such as < >, This represents the reward element in historical action information. Represents the state element in historical action information. This represents the action elements in historical action information. This represents the original sequence of action vectors. Indicates the function after embedding The processed sequence of action vectors.

[0092] (1)

[0093] It is worth noting that the robot's actions at each moment have corresponding historical action information. In one possible implementation, the actions, states, and rewards at each moment can be used as a set of historical action information to obtain historical action information for multiple consecutive moments.

[0094] S202. Input the action vector sequence into the feature extractor. The Mamba module in the global feature extraction module extracts global features based on the action vector sequence. The local feature extraction module extracts local features based on the Mamba module in the local feature extraction module to obtain at least one local feature.

[0095] Reference Figure 1The feature extractor includes a global feature extraction module and a local feature extraction module. The action vector sequence can be processed through two branches: the global feature extraction module and the local feature extraction module, which respectively extract the global features and at least one local feature of the action vector sequence.

[0096] Both the global feature extraction module and the local feature extraction module are composed of Mamba modules. The Mamba module in the global feature extraction module can model the action vector sequence as a whole to obtain the global features of the action vector sequence. The local feature extraction module can first segment the action vector sequence to obtain multiple action vector sub-sequences, and then the Mamba module can model each action vector sub-sequence separately to obtain the local features corresponding to each action vector sub-sequence.

[0097] The Mamba module's feature extraction is primarily based on a structured state-space model and a dynamic adjustment mechanism, combining linear computational complexity with global feature modeling capabilities. Specifically, when processing action vector sequences, the Mamba module models the sequence data using state-space equations, thus efficiently modeling long-term dependencies between states. The dynamic convolution method in the Mamba module can dynamically generate convolution kernel weights based on input features, enhancing the perception of local texture and spatial differences. The channel attention method can adaptively adjust channel importance through the Squeeze-and-Excitation (SE) module to suppress redundant features. The multi-scale feature fusion method can combine a cross-scale self-attention module to capture global contextual information at different resolutions. Furthermore, for multimodal inputs, the Mamba module can also use a gating mechanism to filter cross-modal correlated features, suppress noise information, and bidirectionally propagate hidden states in the temporal dimension, enhancing the integration of complementary information between modalities.

[0098] S203. Input the global features and local features into the feature fusion module to obtain the target features.

[0099] Optionally, the feature fusion module can perform feature fusion on global features and local features. The fusion method can be, for example, to concatenate global features and local features and then standardize the concatenated features to obtain the target features.

[0100] As a possible implementation, the feature fusion module can also incorporate an attention mechanism, using self-attention to adjust feature weights and cross-attention to fuse multimodal information, thereby obtaining the fused target features.

[0101] S204. Input the target features into the decoder, and the decoder predicts the robot's predicted action sequence. Control the target robot's movement according to the predicted action sequence, and determine the state and reward according to the actual action sequence of the target robot's movement. Add the actual action sequence, state, and reward as historical action information to the historical action library.

[0102] Optionally, the decoder can decode the target features to predict the robot's predicted action sequence. This predicted action sequence can be a sequence of executable actions for the robot within a future time step T.

[0103] After obtaining the predicted action sequence, the controller in the robot can control the corresponding parts of the robot to execute the predicted action sequence. After each action is executed, the current state and reward of the robot are recorded, and the current action, current state and reward of the robot are added to the historical action library as historical action information.

[0104] In this embodiment, by extracting both global and local features from the action sequence, the system can consider both the local correlations of the action sequence and the temporal continuity of historical action sequences. Using the Mamba module for feature extraction further enhances the model's selectivity and context awareness, thereby improving its performance. Furthermore, due to the significant lightweight advantage of the Mamba module, the action prediction model improved based on the Mamba module in this application also boasts the advantages of being lightweight and easy to deploy.

[0105] The following is a further explanation of how the historical motion information of the target robot is input into the encoder to obtain a sequence of motion vectors, as described above. Figure 3 As shown, the above step S201 includes:

[0106] S301. Sample and process the historical action information to obtain multiple consecutive triplet information of the target robot. The triplet information includes: state information, action information and reward information.

[0107] Optionally, the historical action information can be sampled, which may involve obtaining the state and reward corresponding to each action in the historical action information, and treating each action, the state corresponding to the action, and the reward corresponding to the action as a triplet of information.

[0108] As an alternative implementation, the historical motion information of the same parts of the robot can be sampled sequentially according to time order to obtain multiple consecutive triplet information of each part.

[0109] S302. Input multiple consecutive triplet information into the encoder, and the encoder performs vectorization processing on each triplet information to obtain an action vector sequence.

[0110] The consecutive triples are input into the encoder in chronological order. The encoder can encode each triple and encode multiple consecutive triple information into a sequence of action vectors.

[0111] As one possible implementation, the encoder includes an autoencoder and an encoding layer. A fully connected layer or a convolutional layer can be used as the encoder to extract features from the information of consecutive triples, generate a vector representation of a preset dimension, and obtain an action vector sequence.

[0112] Optionally, the local feature extraction module may also include a segmentation module and a transformation module.

[0113] like Figure 4 As shown, the process by which the local feature extraction module extracts at least one local feature based on the action vector sequence includes:

[0114] S401. The segmentation module performs segmentation processing on the action vector sequence to obtain at least one action vector subsequence corresponding to the action vector sequence.

[0115] It should be noted that the number of actions at the past T0 time step may be different each time action prediction is performed based on historical action information. Therefore, the amount of historical action information may be different, which in turn may lead to different lengths of action vector sequences. By segmenting the action vector sequence based on the segmentation module, at least one action vector subsequence of the same length can be obtained.

[0116] S402. The Mamba module extracts features from each action vector subsequence to obtain at least one initial feature.

[0117] The Mamba module can be one or more. When there is only one Mamba module, each action vector sub-sequence can be input into the Mamba module sequentially, and the Mamba module will extract features from the action vector sub-sequence to obtain the initial features corresponding to the action vector sub-sequence. When there are multiple Mamba modules, the action vector sub-sequences can be divided into multiple groups, and the action vector sub-sequences of each group can be input into the corresponding Mamba module sequentially, and each Mamba module will extract features from the action vector sub-sequences of each group to obtain the initial features corresponding to each action vector sub-sequence.

[0118] In one possible implementation, the action vector subsequence can be first subjected to layer normalization, and then modeled based on the Mamba module to obtain the initial features of the action vector subsequence.

[0119] S403. The conversion module performs conversion processing on each initial feature to obtain at least one local feature.

[0120] The transformation module can perform dimensional transformation on the initial features while ensuring that the number of elements in the initial features remains unchanged. This includes changing the row and column arrangement of the initial features so that the dimensions of the initial features meet the processing requirements of the next step module.

[0121] The following is a further explanation of step S401 above, such as... Figure 5 As shown, the process described above, in which the segmentation module segments the action vector sequence to obtain at least one action vector subsequence corresponding to the action vector sequence, includes:

[0122] S501. The segmentation module fills the end of the action vector sequence with a preset value until the length of the action vector sequence reaches N, thus obtaining the filled action vector sequence.

[0123] Optionally, the segmentation module can pad the end of the action vector sequence with 0s until the length of the action vector sequence reaches N, resulting in a padded action vector sequence. Here, N is a positive integer greater than 0, and its value can be determined based on the maximum number of actions in the past time step T0.

[0124] S502. The segmentation module performs segmentation processing on the filled action vector sequence based on a preset segmentation length to obtain at least one action vector subsequence of the action vector sequence.

[0125] The preset segmentation length can be represented by L, where L can be any integer greater than 0. In one possible implementation, L can be a multiple of 3 to ensure that each action vector subsequence includes multiple complete historical action information.

[0126] Optional, such as Figure 6 The diagram shown is a schematic representation of the overall structure of a reinforcement learning action prediction model. (Refer to...) Figure 6 The local feature extraction module also includes a regularization module. This regularization module can be a Dropout module.

[0127] like Figure 7 As shown, the process of inputting global features and local features into the feature fusion module includes:

[0128] S701. Input at least one local feature into the regularization module for regularization processing to obtain the processed local feature.

[0129] The regularization module is used to randomly drop neurons and perform output scaling compensation during training, thereby preventing overfitting.

[0130] S702. Input the global features and each processed local feature into the feature fusion module.

[0131] In this embodiment of the application, the local features are regularized by the regularization module, which can improve the robustness of the model while maintaining the high computational efficiency of the model, thereby effectively improving the performance of the model.

[0132] Reference Figure 6 The feature fusion module includes: a first connection layer and a feedforward network layer. For example... Figure 8 As shown, the process of inputting global features and local features into the feature fusion module to obtain target features includes:

[0133] S801. Input the global features and local features into the first connection layer, and the first connection layer performs connection processing to obtain the connected features.

[0134] S802. Input the concatenated features into the feedforward network layer, which then performs transformation processing to obtain the transformed features.

[0135] Optionally, the first connection layer can concatenate global and local features into a concatenated feature, and then input the concatenated feature into the feedforward network layer. The feedforward network layer can perform feature enhancement and linear transformation on the concatenated feature. Specifically, it can combine linear transformation with activation functions to map the input data to a more complex feature space, thereby achieving efficient feature extraction and transformation of the data.

[0136] As one possible approach, the connected features can be first normalized to obtain processed connected features, and then the processed connected features can be input into the feedforward network layer for transformation to obtain transformed features.

[0137] S803. The transformed features and the concatenated features are spliced ​​together to obtain the target features.

[0138] After obtaining the transformed features, the transformed features and the concatenated features can be concatenated using the add function to obtain the target features.

[0139] Optional, continue to refer to Figure 6 The reinforcement learning action prediction model also includes a linear module, which includes a normalized layer, a first linear layer, an activation layer, a second linear layer, and a second connection layer.

[0140] The process of inputting target features into the decoder as described above includes:

[0141] The target features are input into the linear module, which performs normalization, linearization, activation and connection processing in sequence to obtain the processed target features, which are then input into the decoder.

[0142] Optionally, after the target features are input into the linear module, a normalization layer can be used to normalize the target features. The first linear layer can linearize the normalized target features. The activation layer can activate the linearized target features. The second linear layer can linearize the activated target features. The second connection layer can connect the target features processed by the second linear layer to obtain the processed target features.

[0143] As one possible implementation, the second connection layer can perform connection processing on the processed target features and the target features to obtain the connected features.

[0144] It is worth noting that the model can include multiple linear modules. After being processed by each linear module in turn, the final processed target features can be input into the encoder, which then predicts the action sequence for the next T time steps.

[0145] Based on the same inventive concept, this application also provides a robot motion prediction device corresponding to the robot motion prediction method. Since the principle of the device in this application is similar to that of the robot motion prediction method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0146] Figure 9 A schematic diagram of the structure of a robot motion prediction device provided in an embodiment of this application is shown.

[0147] The encoding module 901 is used to obtain at least one historical action information from the historical action library of the target robot, input the historical action information into the encoder, and obtain an action vector sequence. The action vector sequence includes at least one action vector, and the action vector includes: a state element, an action element, and a reward element.

[0148] The extraction module 902 is used to input the action vector sequence into the feature extractor, where the Mamba module in the global feature extraction module extracts global features based on the action vector sequence, and the local feature extraction module extracts local features from the action vector sequence based on the Mamba module in the local feature extraction module to obtain at least one local feature.

[0149] The fusion module 903 is used to input global features and various local features into the feature fusion module to obtain target features;

[0150] The prediction module 904 is used to input the target features into the decoder, which predicts the robot's predicted action sequence. The robot's movement is controlled according to the predicted action sequence. The robot's state and reward are determined according to the actual action sequence during the robot's movement. The actual action sequence, state, and reward are added to the historical action library as historical action information.

[0151] Optionally, the encoding module 901 is used for:

[0152] Historical action information is sampled and processed to obtain multiple consecutive triples of information for the target robot. The triples include: state information, action information, and reward information.

[0153] Multiple consecutive triplet information is input into the encoder, which then vectorizes each triplet information to obtain an action vector sequence.

[0154] Optionally, the local feature extraction module may also include a segmentation module and a transformation module;

[0155] Extraction module 902 is also used for:

[0156] The segmentation module segments the action vector sequence to obtain at least one action vector subsequence corresponding to the action vector sequence;

[0157] The Mamba module extracts features from each action vector subsequence to obtain at least one initial feature;

[0158] The conversion module performs conversion processing on each initial feature to obtain at least one local feature.

[0159] Optionally, the extraction module 902 is also used for:

[0160] The segmentation module fills the end of the action vector sequence with a preset value until the length of the action vector sequence reaches N, thus obtaining the filled action vector sequence.

[0161] The segmentation module segments the padded action vector sequence based on a preset segmentation length to obtain at least one action vector subsequence of the action vector sequence.

[0162] Optionally, the local feature extraction module may further include: a regularization module;

[0163] The fusion module 903 is also used for:

[0164] At least one local feature is input into the regularization module to shut down the neuron and obtain the processed local feature;

[0165] The global features and the processed local features are input into the feature fusion module.

[0166] Optionally, the feature fusion module includes: a first connection layer and a feedforward network layer;

[0167] The fusion module 903 is also used for:

[0168] Global and local features are input into the first connection layer, which performs connection processing to obtain the connected features.

[0169] The concatenated features are input into the feedforward network layer, which then performs transformation processing to obtain the transformed features.

[0170] The transformed features and the concatenated features are concatenated to obtain the target features.

[0171] Optionally, the action prediction model further includes: a linear module, which includes: a normalization layer, a first linear layer, an activation layer, a second linear layer, and a second connection layer;

[0172] Prediction module 904 is also used for:

[0173] The target features are input into the linear module, which performs normalization, linearization, activation and connection processing in sequence to obtain the processed target features, which are then input into the decoder.

[0174] Figure 10 This illustration shows a schematic diagram of an electronic device provided in an embodiment of this application, including: a processor 1001, a storage medium 1002, and a bus 1003. The storage medium 1002 stores machine-readable instructions executable by the processor 1001. When the electronic device runs a robot motion prediction method as described in the embodiment, the processor 1001 communicates with the storage medium 1002 via the bus 1003. The processor 1001 executes the machine-readable instructions, and the preamble of the method item of the processor 1001 performs the steps of the above-described robot motion prediction method.

[0175] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor. The processor performs the steps of the robot motion prediction method described above. In this application, the computer program, when executed by the processor, can also execute other machine-readable instructions to perform other methods described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0176] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0181] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A robot action prediction method characterized by, The application is applied to a target robot deployed with a reinforcement learning action prediction model for generating an action sequence of the target robot in a future T time step based on an action sequence of the target robot in a past T0 time step, the reinforcement learning action prediction model comprising: an encoder, a feature extractor, and a decoder, the feature extractor comprising at least: a global feature extraction module, a local feature extraction module, and a feature fusion module, the global feature extraction module and the local feature extraction module being two branches in the feature extractor, the global feature extraction module comprising at least one Mamba module, the local feature extraction module comprising a plurality of Mamba modules, the Mamba module comprising: a dynamic convolution method, a channel attention method, a multi-scale feature fusion method, and a gating mechanism, the dynamic convolution method being used to dynamically generate a convolution kernel weight according to an input feature, the channel attention method being used to adaptively adjust channel importance, the multi-scale feature fusion method being used to combine a cross-scale self-attention module to capture global context information at different resolutions, and the gating mechanism being used to filter cross-modal correlation features, suppress noise information, and bidirectionally pass hidden states in a time dimension, the local feature extraction module further comprising: a segmentation module and a conversion module; and the method comprising: detecting whether the target robot is started, and if so, iteratively performing the following steps: A. obtaining at least one historical action information from a historical action library of the target robot, inputting the historical action information into the encoder to obtain an action vector sequence, the action vector sequence comprising at least one action vector, the action vector comprising: a state element, an action element, and a reward element; B. inputting the action vector sequence into the feature extractor, modeling the action vector sequence as a whole by the Mamba module in the global feature extraction module to extract global features of the action vector sequence, filling a preset value at the end of the action vector sequence by the segmentation module until the length of the action vector sequence reaches N to obtain a filled action vector sequence, wherein N is a positive integer, and the value of N is determined based on the maximum value of the number of actions in the past T0 time steps; segmenting the filled action vector sequence based on a preset segmentation length by the segmentation module to obtain at least one action vector sub-sequence of the action vector sequence; dividing the action vector sub-sequence into multiple groups and inputting the action vector sub-sequences of each group into the corresponding Mamba module in turn, extracting features of the action vector sub-sequences of each group by each Mamba module to obtain initial features corresponding to each action vector sub-sequence; and converting each initial feature by the conversion module to obtain at least one local feature; C. inputting the global feature and each local feature into the feature fusion module to obtain a target feature; D., inputting the target feature into the decoder, obtaining a predicted action sequence of the robot by the decoder, controlling movement of the target robot according to the predicted action sequence, determining state and reward according to an actual action sequence when the target robot moves, and adding the actual action sequence, state and reward as a historical action information into the historical action library.

2. The method of claim 1, wherein, The inputting the historical action information into the encoder to obtain an action vector sequence comprises: sampling the historical action information to obtain a plurality of continuous triple information of the target robot, the triple information comprising state information, action information and reward information; inputting the plurality of continuous triple information into the encoder to obtain the action vector sequence by vectorizing each triple information by the encoder.

3. The method of claim 1, wherein, The local feature extraction module further comprises a regularization module. The inputting the global feature and each local feature into the feature fusion module comprises: inputting the at least one local feature into the regularization module to close neurons to obtain a processed local feature; inputting the global feature and each processed local feature into the feature fusion module.

4. The method of claim 1, wherein, The feature fusion module comprises a first connection layer and a feedforward network layer. The inputting the global feature and each local feature into the feature fusion module to obtain a target feature comprises: inputting the global feature and the local feature into the first connection layer to obtain a connected feature by connection processing of the first connection layer; inputting the connected feature into the feedforward network layer to obtain a transformed feature by transformation processing of the feedforward network layer; splicing the transformed feature and the connected feature to obtain the target feature.

5. The method of claim 4, wherein, The action prediction model further comprises a linear module, the linear module comprising a normalization layer, a first linear layer, an activation layer, a second linear layer and a second connection layer. The inputting the target feature into the decoder comprises: inputting the target feature into the linear module to obtain a processed target feature by normalization processing, linearization processing, activation processing and connection processing of the linear module in sequence, and inputting the processed target feature into the decoder.

6. A robot action prediction apparatus characterized by comprising: The action prediction model comprises: an encoding module configured to acquire at least one historical action information from a historical action library of a target robot, and input the historical action information into an encoder to obtain an action vector sequence, the action vector sequence comprising at least one action vector, the action vector comprising a state element, an action element and a reward element; The extraction module is configured to input the action vector sequence into a feature extractor, model the action vector sequence as a whole by a mamba module in a global feature extraction module, and extract global features of the action vector sequence. The segmentation module is configured to fill a preset value at the end of the action vector sequence until the length of the action vector sequence reaches N, to obtain a filled action vector sequence, where N is a positive integer, and the value of N is determined based on a maximum value of the number of actions in the past T0 time steps. The segmentation module is configured to perform segmentation processing on the filled action vector sequence based on a preset segmentation length, to obtain at least one action vector sub-sequence of the action vector sequence. The action vector sub-sequences are divided into multiple groups, and the action vector sub-sequences of each group are sequentially input into corresponding mamba modules. Each mamba module is configured to extract features of the action vector sub-sequences of each group, to obtain initial features corresponding to each action vector sub-sequence. The conversion module is configured to perform conversion processing on each initial feature, to obtain at least one local feature. The fusion module is configured to input the global features and the local features into a feature fusion module, to obtain target features. The prediction module is configured to input the target features into a decoder, predict a predicted action sequence of the robot by the decoder, control the target robot to move according to the predicted action sequence, determine a state and a reward according to an actual action sequence when the target robot moves, and add the actual action sequence, the state and the reward as a historical action information into the historical action library.

7. An electronic device, comprising: The processor, the storage medium and the bus, the storage medium stores the machine readable instructions executable by the processor, when the electronic device runs, the processor and the storage medium communicate through the bus, the processor executes the machine readable instructions, to execute the steps of the robot action prediction method in any one of claims 1 to 5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the robot action prediction method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, ​

Images