Robot action prediction method and system based on cross-modal feature enhancement

Abstract

The invention provides a robot action prediction method and system based on cross-modal feature enhancement, and relates to the technical field of universal robots. According to the method and the system, aiming at the defect that dynamic and fine-grained correlation between visual information such as RGB / depth images and a robot joint motion sequence is difficult to establish due to dependence on static feature fusion in the existing method, a cross-modal fusion feature enhancement module is designed, and an attention mechanism is introduced to drive bidirectional interaction and adaptive weighting of multi-modal features; the guidance value of the spatial geometric information of the depth image and the texture details of the RGB image on the action decision is fully mined, so that the accuracy and adaptability of action prediction in a complex scene are improved. The cross-modal motion enhancement features output by the cross-modal fusion feature enhancement module and the multi-modal visual features are spliced and input into a self-attention decoder together, the future joint motion sequence of the robot is predicted, and motion prediction is achieved.

Description

Technical Field

[0001] This invention relates to the field of general robotics technology, and in particular to a method and system for predicting robot motions based on cross-modal feature enhancement. Background Technology

[0002] Imitation learning and other general embodied intelligence methods for robots are crucial in practical applications. Their core lies in learning and reproducing complex operational skills by observing expert demonstrations (such as human operation or ideal trajectories). Current mainstream solutions typically employ behavioral cloning or inverse reinforcement learning frameworks. In behavioral cloning, models (such as deep neural networks) are directly trained to map perceived states (such as camera images or joint encoder readings) to corresponding expert actions; this is essentially supervised learning. To handle temporal dependencies, recurrent neural networks, long short-term memory networks, or Transformer encoders are often introduced to model state-action sequences, learning the temporal relationships between actions. For multimodal inputs (commonly visual RGB images and robot body states), existing technologies generally employ independent branch networks (such as convolutional neural networks for image feature extraction) for feature extraction, followed by relatively simple operations (such as feature concatenation, element-wise addition, or averaging) to achieve modality fusion at specific levels (early input layer, intermediate feature layer, or late decision layer). The learned policy ultimately outputs predicted robot actions (such as joint displacement or end effector velocity), aiming to minimize the difference from the demonstrated actions. These methods significantly reduce the complexity of robot skill programming, improve adaptability and task generalization in dynamic environments (such as changes in object position and lighting interference), and have been successfully applied in scenarios such as industrial sorting, warehousing and logistics, and surgical assistance. They are an important technical path to achieve flexible and efficient robot operation.

[0003] Existing imitation learning schemes (especially behavior cloning and end-to-end policy learning) effectively utilize robot joint motion sequences and visual perception information when handling robot manipulation tasks. However, they suffer from a significant core drawback: visual modal information is not fully mined and utilized, and there is a lack of effective, deep, dynamic interaction mechanisms between visual and action modal information. This makes it difficult for the system to accurately perceive environmental details and optimize action decisions accordingly. Specifically, existing schemes typically use robot joint motion sequences as the primary input modality, extracting motion features through temporal encoders (such as Long Short-Term Memory networks or Transformer encoders) as the main basis for prediction. Visual modal information (RGB images and depth images) is relegated to a secondary or auxiliary position, and its processing flow is relatively independent—usually, pre-trained convolutional neural networks or independent branches are used to extract image features separately. Subsequently, at a certain node of the model architecture (such as the input layer, intermediate layer, or decision layer), static, unidirectional fusion operations (such as simple feature concatenation, element-wise addition, or weighted averaging) are performed with the action features. This fusion approach is essentially a shallow, non-adaptive information merging. It implicitly assumes that visual and motion features have fixed contribution weights and association patterns across all scenarios, failing to establish fine-grained, context-dependent dynamic connections between the two modalities. Secondly, the rich spatial details contained in depth visual information (such as precise object geometry and spatial pose) and RGB information (such as texture and color cues) cannot be effectively transferred to the action generation module through simple fusion. When tasks require highly dependent visual feedback for fine-grained operations (e.g., grasping deformable objects, obstacle avoidance in cluttered scenes, or adjusting grasping strategies based on subtle changes in object posture), existing methods struggle to get the action prediction model to actively "focus" on the key regions in the visual feature map most relevant to the current action intent or the precise spatial relationships in the depth information, thus weakening the crucial guiding role of visual information. This deficiency in cross-modal interaction limits the system's sensitivity and adaptability to changes in visual cues in dynamic, unstructured environments (e.g., random object positions, changing lighting, occlusion), potentially leading to decreased action prediction accuracy and insufficient robustness, making it difficult to meet the demands of high-precision, dexterous operation tasks. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a robot motion prediction method and system based on cross-modal feature enhancement. This invention addresses the deficiency of existing methods, which rely on static feature fusion (such as stitching and weighted averaging), making it difficult to establish a dynamic and fine-grained correlation between visual information (RGB / depth images) and robot joint motion sequences. By designing a cross-modal fusion feature enhancement module, an attention mechanism is introduced to drive bidirectional interaction and adaptive weighting of multimodal features. This fully leverages the spatial geometric information of depth images and the texture details of RGB images to guide motion decisions, thereby improving the accuracy and adaptability of motion prediction in complex scenes (such as object occlusion and pose changes).

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] On one hand, the present invention provides a robot motion prediction method based on cross-modal feature enhancement, comprising:

[0007] RGB and depth images are acquired using RGB and depth cameras fixed to the robot body, and the joint motion information of the robot body is acquired simultaneously to form a continuous motion sequence of the robot body.

[0008] A series of joint motion sequences are obtained by sampling the continuous motion sequence of the robot body at a fixed length.

[0009] Embedding representations are performed on joint motion sequences and robot joint motion information to obtain joint motion sequence embedding features and robot joint motion information embedding features;

[0010] The joint motion sequence embedding features and robot joint motion information embedding features are input into a self-attention encoder, and the output features are randomly sampled according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence.

[0011] By concatenating the normally distributed random sampling features with the robot joint motion information embedding features, the fused motion features are obtained.

[0012] Two independent feature extraction networks are used to extract visual features from RGB and depth images respectively, and then the features are stitched together to obtain multimodal visual features;

[0013] Multimodal visual features and fused action features are input together into the cross-modal fusion feature enhancement module, and cross-modal action enhancement features are obtained based on the attention mechanism;

[0014] By concatenating cross-modal motion enhancement features with multimodal visual features and inputting them into an attention encoder-decoder, the robot's future joint motion sequence is predicted, thus achieving motion prediction.

[0015] Furthermore, the embedding representation of the joint motion sequence and the robot joint motion information to obtain the joint motion sequence embedding features and the robot joint motion information embedding features is specifically as follows:

[0016] set up Let D represent the robot joint motion state at time t, where D is the number of joint degrees of freedom; then a joint motion sequence of length L is represented as: ;

[0017] Combine the joint motion sequence Q and the robot joint motion state Embedding representation is performed through an embedding layer to obtain the joint motion sequence embedding features. Embedded features of robot joint motion information The embedding layer maps high-dimensional sparse inputs to a low-dimensional dense vector space.

[0018] Furthermore, the specific method for inputting the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly sampling the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence is as follows:

[0019] Embedded features of the embedded joint motion sequence Embedded features with embedded robot joint motion information splicing results in splicing features This data is then fed into a self-attention encoder, which uses a self-attention mechanism to capture temporal dependencies in joint motion sequences and outputs features. ;

[0020] Output features of a self-attention encoder Normally distributed random sampling is performed to obtain the latent spatial features z of the joint motion sequence;

[0021] Embedding the latent spatial features z of normally distributed random sampling with robot joint motion information into features By splicing the data, we can obtain the fused motion features. .

[0022] Furthermore, the specific method for extracting visual features from the RGB image and depth image using two independent feature extraction networks and then concatenating them to obtain multimodal visual features is as follows:

[0023] Visual features of RGB images are extracted using a convolutional neural network, and RGB feature maps are output. ;

[0024] The depth image features are extracted using a CNN network, and the output depth feature map is... ;

[0025] The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are concatenated along the channel dimension to obtain multimodal visual features. .

[0026] Furthermore, the cross-modal fusion feature enhancement module employs an attention mechanism to interactively fuse multimodal visual features and fused action features to obtain cross-modal action enhancement features, specifically:

[0027] By fusing action features and multimodal visual features Generate the query matrix by generating three linear transformations. Key matrix Sum matrix :

[0028] Query Matrix Based on the characteristics of fusion action generate:

[0029] ;

[0030] in, , and Both are weight matrices;

[0031] Key matrix Sum matrix Based on the characteristics of fusion action and multimodal visual features The fusion characteristics of splicing generate:

[0032] ;

[0033] ;

[0034] Calculate fusion action features Fusion features Attention weight matrix :

[0035] ;

[0036] in, For normalization function, The dimensions are the query and the key, and T is the matrix transpose.

[0037] Based on the attention weight log matrix We perform weighted summation to obtain cross-modal action enhancement features. :

[0038] .

[0039] Furthermore, the specific method for concatenating cross-modal motion enhancement features and multimodal visual features and inputting them into the attention encoder-decoder to predict the robot's future joint motion sequence is as follows:

[0040] Integrating multimodal visual features with crossmodal motion enhancement features By splicing, multimodal fusion features are obtained. ;

[0041] Multimodal fusion features The input is fed into the encoder for encoding;

[0042] The output features of the encoder are input into the decoder for decoding;

[0043] The final output of the decoder is mapped to the joint motion space through a fully connected layer to predict a future joint motion sequence of length L.

[0044] Furthermore, multimodal fusion features The specific method for inputting the data into the encoder for encoding is as follows:

[0045] The encoder uses a sine function to generate positional codes from multimodal fusion features, forming the encoder input sequence features. ;

[0046] The encoder's self-attention module captures the global dependencies of the input sequence features through a multi-head mechanism, thus obtaining self-attention features;

[0047] The self-attention features are input into the feedforward neural network, and residual connections and layer normalization are performed to obtain the encoder's output features. .

[0048] Furthermore, the specific method for inputting the encoder's output features into the decoder for decoding is as follows:

[0049] The output features of the encoder As the input sequence feature D0 of the decoder, sinusoidal position encoding is performed on the input sequence feature D0 to add position encoding to the input sequence feature D0;

[0050] Masked multi-head attention is used to handle the dependencies within the features after the position of feature D0 in the input sequence changes;

[0051] Next, cross-attention is used to process the mask multi-head attention output features and the encoder output features. Dependencies between them;

[0052] The encoder output features are further extracted using a feedforward neural network based on the cross-attention output features. .

[0053] On the other hand, the present invention also provides a robot motion prediction system based on cross-modal feature enhancement, including: a robot motion sequence acquisition module, a joint motion sequence acquisition module, an embedded feature acquisition module, a motion feature fusion module, a visual feature extraction and fusion module, a cross-modal fusion feature enhancement module, and a motion prediction module;

[0054] The robot motion sequence acquisition module uses an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquires the joint motion information of the robot body to form a continuous motion sequence of the robot body.

[0055] The joint motion sequence acquisition module performs fixed-length sampling on the continuous motion sequence of the robot body to obtain a series of joint motion sequences;

[0056] The embedding feature acquisition module embeds the joint motion sequence and robot joint motion information into a representation, and obtains the joint motion sequence embedding feature and the robot joint motion information embedding feature.

[0057] The action feature fusion module inputs the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly samples the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence; the latent spatial features are concatenated with the robot joint motion information embedding features to obtain the fused action features;

[0058] The visual feature extraction and fusion module uses two independent feature extraction networks to extract visual features from RGB images and depth images respectively and then stitches them together to obtain multimodal visual features;

[0059] The cross-modal fusion feature enhancement module uses an attention mechanism to perform cross-modal fusion enhancement of multimodal visual features and fused action features to obtain cross-modal action enhancement features;

[0060] The action prediction module concatenates cross-modal action enhancement features and multimodal visual features and inputs them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving action prediction.

[0061] Thirdly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the robot motion prediction method based on cross-modal feature enhancement.

[0062] The beneficial effects of adopting the above technical solution are as follows: Compared with the imitation learning schemes in the prior art that rely on static feature fusion (such as splicing and weighted averaging), the robot action prediction method and system based on cross-modal feature enhancement provided by this invention achieve significant performance improvement by introducing a cross-modal fusion feature enhancement module based on an attention mechanism. Its core advantage lies in establishing a dynamic and fine-grained bidirectional interaction mechanism between the visual modality (RGB image + depth image) and the action modality (joint motion sequence + latent spatial features). Specifically, this module allows action features to be used as query features, actively retrieving spatial regions and depth geometric information (such as the precise contour of the target object, texture details near the grasping point, or distance information required for obstacle avoidance) that are highly relevant to the current action intention from the visual features, and dynamically assigning weights based on relevance for feature fusion and enhancement. This mechanism overcomes the defect of static fusion assuming that modal contributions are fixed, enabling the model to adaptively focus on the most critical parts of the visual information according to the specific task context (such as object pose and environmental occlusion state). The precise three-dimensional spatial relationships contained in the depth map and the rich texture and color cues provided by the RGB image can more effectively guide action generation. Its direct effect is a significant improvement in the system's adaptability and robustness in complex, unstructured dynamic environments (such as random occlusion, lighting changes, and fine-tuning of object posture). On the one hand, motion prediction is more sensitive to changes in visual cues and can adjust the predicted trajectory in real time based on subtle environmental differences (such as the small displacement after an object slides). On the other hand, the fused and enhanced features significantly improve the accuracy and spatiotemporal continuity of joint motion sequences (especially future frames), thus demonstrating superior performance in dexterous manipulation tasks that require high-precision spatiotemporal coordination (such as fine assembly and grasping deformable objects). Attached Figure Description

[0063] Figure 1 A flowchart illustrating the robot motion prediction method based on cross-modal feature enhancement provided in an embodiment of the present invention;

[0064] Figure 2 This is a structural block diagram of an attention-based cross-modal fusion feature enhancement module provided in an embodiment of the present invention.

[0065] Figure 3 This is a structural block diagram of a robot motion prediction system based on cross-modal feature enhancement provided in an embodiment of the present invention. Detailed Implementation

[0066] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0067] Example 1:

[0068] In this embodiment, a robot motion prediction method based on cross-modal feature enhancement is described, such as... Figure 1 As shown, it includes the following steps:

[0069] Step 1: Use an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquire joint motion information of the robot body to form a continuous action sequence of the robot body.

[0070] In this embodiment, RGB and depth images are simultaneously acquired using RGB and depth cameras fixed to the robot body. The motion information of each joint is synchronously acquired in real time via the robot body's joint encoders, forming a continuous sequence of robot movements.

[0071] Step 2: Obtain the joint motion sequence of the robot body, and embed the joint motion sequence with the robot joint motion information to obtain the joint motion sequence embedding feature and the robot joint motion information embedding feature.

[0072] The joint motion sequence embedding features and robot joint motion information embedding features are input into a self-attention encoder, and the output features are randomly sampled according to a normal distribution to obtain the latent spatial features of the joint motion sequence, so as to capture the latent spatial distribution of the joint motion sequence. Then, the normally distributed random sampled features are concatenated with the robot joint motion information embedding features to obtain the fused motion features.

[0073] Step 2-1: Obtain the joint motion sequence of the robot body, and embed the joint motion sequence with the robot joint motion information to obtain the joint motion sequence embedding feature and the robot joint motion information embedding feature.

[0074] First, the continuous motion sequence of the robot body is sampled at a fixed length to obtain a series of joint motion sequences of length L; let... Let D represent the joint motion state at time t, where D is the number of joint degrees of freedom; then a joint motion sequence of length L can be represented as: ;

[0075] To facilitate subsequent processing, the joint motion sequence Q and the robot joint motion state need to be... Embedding representation is performed through an embedding layer to obtain the joint motion sequence embedding features. Embedded features of robot joint motion information Embedding layers can map high-dimensional sparse inputs to low-dimensional dense vector spaces; let the embedding function be... Then the joint motion sequence embedding features for:

[0076] ;

[0077] in, For the embedded dimension;

[0078] Robot joint motion information embedding features for:

[0079] ;

[0080] Step 2-2: Embed the joint motion sequence into features Embedded features with robot joint motion information splicing results in splicing features It is then input into a self-attention encoder, which uses a self-attention mechanism to capture the temporal dependencies in the joint motion sequence;

[0081] The output features of a self-attention encoder can be represented as follows: ,in Let be the dimension of the hidden layer of the self-attention encoder; the specific calculation process of the self-attention encoder is as follows:

[0082] Input features of the self-attention encoder The query matrix Q, the key matrix K, and the value matrix V are obtained through three linear transformations:

[0083] ;

[0084] ;

[0085] ;

[0086] in, , and All are learnable weight matrices;

[0087] Calculate the attention weight matrix A:

[0088] ;

[0089] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.

[0090] The output features of the self-attention encoder are obtained by weighted summation of the value matrix V based on the attention weights. :

[0091] ;

[0092] Steps 2-3: Analyze the output features of the self-attention encoder. Normally distributed random sampling is performed to obtain the latent spatial features of joint motion sequences in order to capture the latent spatial distribution of joint motion sequences;

[0093] First, the output features H of the self-attention encoder are mapped to the mean μ and variance through a linear transformation. :

[0094] ;

[0095] ;

[0096] in, and This is a learnable weight matrix.

[0097] Then, from the normal distribution Random sampling is performed to obtain the latent spatial features z;

[0098] Steps 2-4: Embed the latent spatial features z obtained by random sampling from a normal distribution with the robot joint motion information. By splicing the data, we can obtain the fused motion features. ;

[0099] Before entering the cross-modal fusion feature enhancement module, this invention employs independent feature extraction networks to process RGB and depth images separately, extracting their respective high-level semantic and spatial geometric features. These two heterogeneous but complementary visual features are then concatenated to form a unified, more information-rich fused visual feature representation. This step is crucial, providing a high-quality input foundation containing color texture (RGB) and accurate 3D spatial information (depth) for subsequent cross-modal attention interactions, ensuring the enhancement module can effectively utilize the advantages of both visual modalities.

[0100] Step 3: Use two independent feature extraction networks to extract visual features from the RGB image and depth image respectively and then stitch them together to obtain multimodal visual features; then input the multimodal visual features and the fused action features obtained in Step 2 into the cross-modal fusion feature enhancement module, and achieve effective fusion and feature enhancement between modalities through the attention mechanism to obtain cross-modal action enhancement features;

[0101] Step 3-1: Use two independent feature extraction networks to extract visual features from the RGB image and depth image respectively, and then stitch them together to obtain multimodal visual features, which are then input into the cross-modal fusion feature enhancement module.

[0102] Let the RGB image input to the feature extraction network be... ,in The height of the RGB image. Given the width of an RGB image, a convolutional neural network is used to extract its visual features, outputting an RGB feature map. ,in, and Here, C represents the height and width of the RGB feature map, and C represents the number of channels in the RGB feature map.

[0103] Let the depth image input to the feature extraction network be... ,in The depth image height, The depth image width is used to extract its features through a separate CNN network, outputting a depth feature map. ,in This represents the number of channels in the depth feature map.

[0104] The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are concatenated along the channel dimension to obtain multimodal visual features. ;

[0105] Step 3-2: The cross-modal fusion feature enhancement module uses an attention mechanism to interactively fuse multimodal visual features and fused action features to obtain cross-modal action enhancement features; for example... Figure 2 As shown, the specific steps are as follows:

[0106] By fusing action features and multimodal visual features Generate the query matrix by generating three linear transformations. Key matrix Sum matrix :

[0107] Query Matrix Based on the characteristics of fusion action Generate (i.e., output features from step 2):

[0108] ;

[0109] Key matrix Sum matrix Based on the characteristics of fusion action and multimodal visual features The fusion characteristics of splicing generate:

[0110] ;

[0111] ;

[0112] in, , and All are learnable weight matrices;

[0113] Calculate fusion action features Fusion features Attention weight matrix :

[0114] ;

[0115] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.

[0116] Based on the attention weight log matrix We perform weighted summation to obtain cross-modal action enhancement features. :

[0117] .

[0118] The cross-modal fusion feature enhancement module is specifically designed to achieve dynamic, bidirectional, and fine-grained interaction and feature enhancement between the visual modality (RGB image features + depth image features) and the action modality (stitched historical joint motion features + latent spatial features). It utilizes attention mechanisms (such as cross-attention) to allow action features to act as query features, actively retrieving and weighting the most relevant visual information from the fused visual features (which serve as key and value features), and aggregating them accordingly. This mechanism overcomes the limitations of static stitching or weighted averaging in existing technologies, achieving adaptive selection and deep fusion of information between modalities, significantly improving the accuracy of visual information in guiding action decisions.

[0119] Step 4: Concatenate the cross-modal motion enhancement features and multimodal visual features and input them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving accurate motion prediction;

[0120] Step 4-1: Combine multimodal visual features With cross-modal action enhancement features By splicing, multimodal fusion features are obtained. .

[0121] Step 4-2: Fuse multimodal features The input is then encoded by the encoder. The specific steps are as follows:

[0122] The encoder uses a sine function to generate positional codes from multimodal fusion features, forming the encoder input sequence features. ;

[0123] The encoder's self-attention module captures the global dependencies of the input sequence features through a multi-head mechanism, thus obtaining self-attention features;

[0124] Let the number of attention heads be h, and the query matrix be h. Key matrix Value matrix By respectively analyzing the features of the encoder input sequence Generate by performing three linear transformations:

[0125] ;

[0126] in, , All are learnable weight matrices;

[0127] Based on the query matrix Key matrix Value matrix Calculate self-attention features As shown in the formula below:

[0128] ;

[0129] in, For normalization function, The dimensions are the query and the key, and T is the matrix transpose.

[0130] Self-attention characteristics The input is fed into a feedforward neural network, and residual connections and layer normalization are performed to obtain the encoder's output features. ;

[0131] Step 4-3: Input the encoder's output features into the decoder for decoding. The specific steps are as follows:

[0132] The output features of the encoder As the input sequence feature D0 of the decoder, sinusoidal position encoding is performed on the input sequence feature D0 to add position encoding to the input sequence feature D0;

[0133] Masked multi-head attention is used to handle the dependencies within features after changes in the position D0 of the input sequence, and the query matrix is ​​used. Key matrix Value matrix The features D0 of the input sequence are generated by performing three linear transformations respectively:

[0134] ;

[0135] in, , and All are learnable weight matrices;

[0136] Attention weight Through the mask matrix Suppress information about future locations:

[0137] ;

[0138] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose. It is a lower triangular mask matrix;

[0139] The masked multi-head attention output feature D′ is the attention weight. value matrix Perform a weighted sum:

[0140] ;

[0141] Next, cross-attention is used to process the masked multi-head attention output feature D′ and the encoder output feature. Dependencies between them:

[0142] Query Matrix The key matrix is ​​generated from the decoder feature D′. Sum matrix Output from encoder generate:

[0143] ;

[0144] in, , , All are learnable weight matrices;

[0145] Cross-attention output features are :

[0146] ;

[0147] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.

[0148] The encoder output features are further extracted using a feedforward neural network based on the cross-attention output features. .

[0149] Step 4-4: The final output of the decoder... By mapping to the joint motion space through a fully connected layer, a future joint motion sequence of length L is predicted.

[0150] The cross-modal motion enhancement features (containing visual information extracted after deep interaction and enhanced motion context) output by the cross-modal fusion feature enhancement module are concatenated with multimodal visual features and input into a self-attention decoder. This decoder uses the enhanced cross-modal information to focus on predicting the future motion sequence of the robot's joints. This step directly utilizes the multimodal interaction enhancement features generated by the cross-modal fusion feature enhancement module, ensuring the accuracy and spatiotemporal continuity of the final motion prediction, especially its adaptability to subtle visual changes in complex and dynamic scenes.

[0151] Example 2:

[0152] This embodiment provides a robot motion prediction system based on cross-modal feature enhancement, such as... Figure 3 As shown, it includes: robot motion sequence acquisition module, joint motion sequence acquisition module, embedded feature acquisition module, motion feature fusion module, visual feature extraction and fusion module, cross-modal fusion feature enhancement module, and motion prediction module;

[0153] The robot motion sequence acquisition module uses an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquires the joint motion information of the robot body to form a continuous motion sequence of the robot body.

[0154] The joint motion sequence acquisition module performs fixed-length sampling on the continuous motion sequence of the robot body to obtain a series of joint motion sequences;

[0155] The embedding feature acquisition module embeds the joint motion sequence and robot joint motion information into a representation, and obtains the joint motion sequence embedding feature and the robot joint motion information embedding feature.

[0156] The action feature fusion module inputs the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly samples the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence; the latent spatial features are concatenated with the robot joint motion information embedding features to obtain the fused action features;

[0157] The visual feature extraction and fusion module uses two independent feature extraction networks to extract visual features from RGB images and depth images respectively and then stitches them together to obtain multimodal visual features;

[0158] The cross-modal fusion feature enhancement module uses an attention mechanism to perform cross-modal fusion enhancement of multimodal visual features and fused action features to obtain cross-modal action enhancement features;

[0159] The action prediction module concatenates cross-modal action enhancement features and multimodal visual features and inputs them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving action prediction.

[0160] Example 3:

[0161] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the robot motion prediction method based on cross-modal feature enhancement.

[0162] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0163] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A robot motion prediction method based on cross-modal feature enhancement, characterized in that, include: RGB and depth images are acquired using RGB and depth cameras fixed to the robot body, and the joint motion information of the robot body is acquired simultaneously to form a continuous motion sequence of the robot body. A series of joint motion sequences are obtained by sampling the continuous motion sequence of the robot body at a fixed length. Embedding representations are performed on joint motion sequences and robot joint motion information to obtain joint motion sequence embedding features and robot joint motion information embedding features; The joint motion sequence embedding features and robot joint motion information embedding features are input into a self-attention encoder, and the output features are randomly sampled according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence. By concatenating the normally distributed random sampling features with the robot joint motion information embedding features, the fused motion features are obtained. Two independent feature extraction networks are used to extract visual features from RGB and depth images respectively, and then the features are stitched together to obtain multimodal visual features; Multimodal visual features and fused action features are input together into the cross-modal fusion feature enhancement module, and cross-modal action enhancement features are obtained based on the attention mechanism; By concatenating cross-modal motion enhancement features with multimodal visual features and inputting them into an attention encoder-decoder, the robot's future joint motion sequence is predicted, thus achieving motion prediction.

2. The robot motion prediction method based on cross-modal feature enhancement according to claim 1, characterized in that, The embedding representation of the joint motion sequence and the robot joint motion information yields the joint motion sequence embedding features and the robot joint motion information embedding features, specifically as follows: set up Let D represent the robot joint motion state at time t, where D is the number of joint degrees of freedom; then a joint motion sequence of length L is represented as: ; Combine the joint motion sequence Q and the robot joint motion state Embedding representation is performed through an embedding layer to obtain the joint motion sequence embedding features. Embedded features of robot joint motion information The embedding layer maps high-dimensional sparse inputs to a low-dimensional dense vector space.

3. The robot motion prediction method based on cross-modal feature enhancement according to claim 2, characterized in that, The specific method for obtaining the latent spatial features of the joint motion sequence by inputting the joint motion sequence embedding features and the robot joint motion information embedding features into a self-attention encoder and randomly sampling the output features according to a normal distribution is as follows: Embedded features of the embedded joint motion sequence Embedded features with embedded robot joint motion information splicing results in splicing features This data is then fed into a self-attention encoder, which uses a self-attention mechanism to capture temporal dependencies in joint motion sequences and outputs features. ; Output features of a self-attention encoder Normally distributed random sampling is performed to obtain the latent spatial features z of the joint motion sequence; Embedding the latent spatial features z of normally distributed random sampling with robot joint motion information into features By splicing the data, we can obtain the fused motion features. .

4. The robot motion prediction method based on cross-modal feature enhancement according to claim 3, characterized in that, The specific method for extracting visual features from RGB and depth images using two independent feature extraction networks and then concatenating them to obtain multimodal visual features is as follows: Visual features of RGB images are extracted using a convolutional neural network, and RGB feature maps are output. ; The depth image features are extracted using a CNN network, and the output depth feature map is... ; The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are concatenated along the channel dimension to obtain multimodal visual features. .

5. The robot motion prediction method based on cross-modal feature enhancement according to claim 4, characterized in that, The cross-modal fusion feature enhancement module employs an attention mechanism to interactively fuse multimodal visual features and fused action features to obtain cross-modal action enhancement features, specifically: By fusing action features and multimodal visual features Generate the query matrix by generating three linear transformations. Key matrix Sum matrix : Query Matrix Based on the characteristics of fusion action generate: ； in, , and Both are weight matrices; Key matrix Sum matrix Based on the characteristics of fusion action and multimodal visual features The fusion characteristics of splicing generate: ； ； Calculate fusion action features Fusion features Attention weight matrix : ； in, For normalization function, The dimensions are the query and the key, and T is the matrix transpose. Based on the attention weight log matrix We perform weighted summation to obtain cross-modal action enhancement features. : 。 6. The robot motion prediction method based on cross-modal feature enhancement according to claim 5, characterized in that, The specific method for concatenating cross-modal motion enhancement features and multimodal visual features and inputting them into an attention encoder-decoder to predict the robot's future joint motion sequence is as follows: Integrating multimodal visual features with crossmodal motion enhancement features By splicing, multimodal fusion features are obtained. ; Multimodal fusion features The input is fed into the encoder for encoding; The output features of the encoder are input into the decoder for decoding; The final output of the decoder is mapped to the joint motion space through a fully connected layer to predict a future joint motion sequence of length L.

7. The robot motion prediction method based on cross-modal feature enhancement according to claim 6, characterized in that, Multimodal fusion features The specific method for inputting the data into the encoder for encoding is as follows: The encoder uses a sine function to generate positional codes from multimodal fusion features, forming the encoder input sequence features. ; The encoder's self-attention module captures the global dependencies of the input sequence features through a multi-head mechanism, thus obtaining self-attention features; The self-attention features are input into the feedforward neural network, and residual connections and layer normalization are performed to obtain the encoder's output features. .

8. The robot motion prediction method based on cross-modal feature enhancement according to claim 7, characterized in that, The specific method for inputting the encoder's output features into the decoder for decoding is as follows: The output features of the encoder As the input sequence feature D0 of the decoder, sinusoidal position encoding is performed on the input sequence feature D0 to add position encoding to the input sequence feature D0; Masked multi-head attention is used to handle the dependencies within the features after the position of feature D0 in the input sequence changes; Next, cross-attention is used to process the mask multi-head attention output features and the encoder output features. Dependencies between them; The encoder output features are further extracted using a feedforward neural network based on the cross-attention output features. .

9. A robot motion prediction system based on cross-modal feature enhancement, implemented based on the robot motion prediction method based on cross-modal feature enhancement as described in claim 1, characterized in that, include: The module includes a robot motion sequence acquisition module, a joint motion sequence acquisition module, an embedded feature acquisition module, a motion feature fusion module, a visual feature extraction and fusion module, a cross-modal fusion feature enhancement module, and a motion prediction module. The robot motion sequence acquisition module uses an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquires the joint motion information of the robot body to form a continuous motion sequence of the robot body. The joint motion sequence acquisition module performs fixed-length sampling on the continuous motion sequence of the robot body to obtain a series of joint motion sequences; The embedding feature acquisition module embeds the joint motion sequence and robot joint motion information into a representation, and obtains the joint motion sequence embedding feature and the robot joint motion information embedding feature. The action feature fusion module inputs the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly samples the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence; the latent spatial features are concatenated with the robot joint motion information embedding features to obtain the fused action features; The visual feature extraction and fusion module uses two independent feature extraction networks to extract visual features from RGB images and depth images respectively and then stitches them together to obtain multimodal visual features; The cross-modal fusion feature enhancement module uses an attention mechanism to perform cross-modal fusion enhancement of multimodal visual features and fused action features to obtain cross-modal action enhancement features; The action prediction module concatenates cross-modal action enhancement features and multimodal visual features and inputs them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving action prediction.

10. A computer program product for executing the robot motion prediction method based on cross-modal feature enhancement as described in any one of claims 1-8, characterized in that, This includes a computer program or instructions that, when executed by a processor, implement the robot motion prediction method based on cross-modal feature enhancement.

Images