A Robot Motion Prediction Method and System Based on Cross-Modal Feature Enhancement
By introducing a cross-modal feature enhancement module and an attention mechanism, the problem of underutilization of visual modal information in existing technologies is solved, thereby improving the accuracy and adaptability of robot motion prediction, especially for high-precision operation in dynamic and unstructured environments.
Patent Information
- Application Number
- CN202511659717.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-03-06
- Estimated Expiration
- 2045-11-13
AI Technical Summary
In existing imitation learning methods, visual modal information is not fully explored and utilized, and there is a lack of effective and deep dynamic interaction mechanisms between visual features and action features, resulting in insufficient accuracy and adaptability of robot action prediction in dynamic and unstructured environments.
A cross-modal feature enhancement module is introduced, which uses an attention mechanism to drive bidirectional interaction and adaptive weighting of multimodal features. Through a self-attention encoder and a cross-modal fusion feature enhancement module, dynamic and fine-grained correlation between visual information and robot joint motion sequences is achieved, thereby improving motion prediction accuracy and adaptability.
It significantly improves the accuracy and adaptability of robot motion prediction in complex scenarios, especially in situations such as object occlusion and pose changes, enabling more sensitive adjustment of motion prediction and improving adaptability and robustness to changes in visual cues.
Smart Images
Figure CN121105044B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of general robotics technology, and in particular to a method and system for predicting robot motions based on cross-modal feature enhancement. Background Technology
[0002] Imitation learning and other general embodied intelligence methods for robots are crucial in practical applications. Their core lies in learning and reproducing complex operational skills by observing expert demonstrations (such as human operation or ideal trajectories). Current mainstream solutions typically employ behavioral cloning or inverse reinforcement learning frameworks. In behavioral cloning, models (such as deep neural networks) are directly trained to map perceived states (such as camera images or joint encoder readings) to corresponding expert actions; this is essentially supervised learning. To handle temporal dependencies, recurrent neural networks, long short-term memory networks, or Transformer encoders are often introduced to model state-action sequences, learning the temporal relationships between actions. For multimodal inputs (commonly visual RGB images and robot body states), existing technologies generally employ independent branch networks (such as convolutional neural networks for image feature extraction) for feature extraction, followed by relatively simple operations (such as feature concatenation, element-wise addition, or averaging) to achieve modality fusion at specific levels (early input layer, intermediate feature layer, or late decision layer). The learned policy ultimately outputs predicted robot actions (such as joint displacement or end effector velocity), aiming to minimize the difference from the demonstrated actions. These methods significantly reduce the complexity of robot skill programming, improve adaptability and task generalization in dynamic environments (such as changes in object position and lighting interference), and have been successfully applied in scenarios such as industrial sorting, warehousing and logistics, and surgical assistance. They are an important technical path to achieve flexible and efficient robot operation.
[0003] Existing imitation learning schemes (especially behavior cloning and end-to-end policy learning) effectively utilize robot joint motion sequences and visual perception information when handling robot manipulation tasks. However, they suffer from a significant core drawback: visual modal information is not fully mined and utilized, and there is a lack of effective, deep, dynamic interaction mechanisms between visual and action modal information. This makes it difficult for the system to accurately perceive environmental details and optimize action decisions accordingly. Specifically, existing schemes typically use robot joint motion sequences as the primary input modality, extracting motion features through temporal encoders (such as Long Short-Term Memory networks or Transformer encoders) as the main basis for prediction. Visual modal information (RGB images and depth images) is relegated to a secondary or auxiliary position, and its processing flow is relatively independent—usually, pre-trained convolutional neural networks or independent branches are used to extract image features separately. Subsequently, at a certain node of the model architecture (such as the input layer, intermediate layer, or decision layer), static, unidirectional fusion operations (such as simple feature concatenation, element-wise addition, or weighted averaging) are performed with the action features. This fusion approach is essentially a shallow, non-adaptive information merging. It implicitly assumes that visual and motion features have fixed contribution weights and association patterns across all scenarios, failing to establish fine-grained, context-dependent dynamic connections between the two modalities. Secondly, the rich spatial details contained in depth visual information (such as precise object geometry and spatial pose) and RGB information (such as texture and color cues) cannot be effectively transferred to the action generation module through simple fusion. When tasks require highly dependent visual feedback for fine-grained operations (e.g., grasping deformable objects, obstacle avoidance in cluttered scenes, or adjusting grasping strategies based on subtle changes in object posture), existing methods struggle to get the action prediction model to actively "focus" on the key regions in the visual feature map most relevant to the current action intent or the precise spatial relationships in the depth information, thus weakening the crucial guiding role of visual information. This deficiency in cross-modal interaction limits the system's sensitivity and adaptability to changes in visual cues in dynamic, unstructured environments (e.g., random object positions, changing lighting, occlusion), potentially leading to decreased action prediction accuracy and insufficient robustness, making it difficult to meet the demands of high-precision, dexterous operation tasks. Summary of the Invention
[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a robot motion prediction method and system based on cross-modal feature enhancement. This invention addresses the deficiency of existing methods, which rely on static feature fusion (such as stitching and weighted averaging), making it difficult to establish a dynamic and fine-grained correlation between visual information (RGB / depth images) and robot joint motion sequences. By designing a cross-modal fusion feature enhancement module, an attention mechanism is introduced to drive bidirectional interaction and adaptive weighting of multimodal features. This fully leverages the spatial geometric information of depth images and the texture details of RGB images to guide motion decisions, thereby improving the accuracy and adaptability of motion prediction in complex scenes (such as object occlusion and pose changes).
[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0006] On one hand, the present invention provides a robot motion prediction method based on cross-modal feature enhancement, comprising:
[0007] RGB and depth images are acquired using RGB and depth cameras fixed to the robot body, and the joint motion information of the robot body is acquired simultaneously to form a continuous motion sequence of the robot body.
[0008] A series of joint motion sequences are obtained by sampling the continuous motion sequence of the robot body at a fixed length.
[0009] Embedding representations are performed on joint motion sequences and robot joint motion information to obtain joint motion sequence embedding features and robot joint motion information embedding features;
[0010] The joint motion sequence embedding features and robot joint motion information embedding features are input into a self-attention encoder, and the output features are randomly sampled according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence.
[0011] By concatenating the normally distributed random sampling features with the robot joint motion information embedding features, the fused motion features are obtained.
[0012] Two independent feature extraction networks are used to extract visual features from RGB and depth images respectively, and then the features are stitched together to obtain multimodal visual features;
[0013] Multimodal visual features and fused action features are input together into the cross-modal fusion feature enhancement module, and cross-modal action enhancement features are obtained based on the attention mechanism;
[0014] By concatenating cross-modal motion enhancement features with multimodal visual features and inputting them into an attention encoder-decoder, the robot's future joint motion sequence is predicted, thus achieving motion prediction.
[0015] Furthermore, the embedding representation of the joint motion sequence and the robot joint motion information to obtain the joint motion sequence embedding features and the robot joint motion information embedding features is specifically as follows:
[0016] set up Let D represent the robot joint motion state at time t, where D is the number of joint degrees of freedom; then a joint motion sequence of length L is represented as: ;
[0017] Combine the joint motion sequence Q and the robot joint motion state Embedding representation is performed through an embedding layer to obtain the joint motion sequence embedding features. Embedded features of robot joint motion information The embedding layer maps high-dimensional sparse inputs to a low-dimensional dense vector space.
[0018] Furthermore, the specific method for inputting the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly sampling the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence is as follows:
[0019] Embedded features of the embedded joint motion sequence Embedded features with embedded robot joint motion information splicing results in splicing features This data is then fed into a self-attention encoder, which uses a self-attention mechanism to capture temporal dependencies in joint motion sequences and outputs features. ;
[0020] Output features of a self-attention encoder Normally distributed random sampling is performed to obtain the latent spatial features z of the joint motion sequence;
[0021] Embedding the latent spatial features z of normally distributed random sampling with robot joint motion information into features By splicing the data, we can obtain the fused motion features. .
[0022] Furthermore, the specific method for extracting visual features from the RGB image and depth image using two independent feature extraction networks and then concatenating them to obtain multimodal visual features is as follows:
[0023] Visual features of RGB images are extracted using a convolutional neural network, and RGB feature maps are output. ;
[0024] The depth image features are extracted using a CNN network, and the output depth feature map is... ;
[0025] The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are concatenated along the channel dimension to obtain multimodal visual features. .
[0026] Furthermore, the cross-modal fusion feature enhancement module employs an attention mechanism to interactively fuse multimodal visual features and fused action features to obtain cross-modal action enhancement features, specifically:
[0027] By fusing action features and multimodal visual features Generate the query matrix by generating three linear transformations. Key matrix Sum matrix :
[0028] Query Matrix Based on the characteristics of fusion action generate:
[0029] ;
[0030] in, , and Both are weight matrices;
[0031] Key matrix Sum matrix Based on the characteristics of fusion action and multimodal visual features The fusion characteristics of splicing generate:
[0032] ;
[0033] ;
[0034] Calculate fusion action features Fusion features Attention weight matrix :
[0035] ;
[0036] in, For normalization function, The dimensions are the query and the key, and T is the matrix transpose.
[0037] Based on the attention weight log matrix We perform weighted summation to obtain cross-modal action enhancement features. :
[0038] .
[0039] Furthermore, the specific method for concatenating cross-modal motion enhancement features and multimodal visual features and inputting them into the attention encoder-decoder to predict the robot's future joint motion sequence is as follows:
[0040] Integrating multimodal visual features with crossmodal motion enhancement features By splicing, multimodal fusion features are obtained. ;
[0041] Multimodal fusion features The input is fed into the encoder for encoding;
[0042] The output features of the encoder are input into the decoder for decoding;
[0043] The final output of the decoder is mapped to the joint motion space through a fully connected layer to predict a future joint motion sequence of length L.
[0044] Furthermore, multimodal fusion features The specific method for inputting the data into the encoder for encoding is as follows:
[0045] The encoder uses a sine function to generate positional codes from multimodal fusion features, forming the encoder input sequence features. ;
[0046] The encoder's self-attention module captures the global dependencies of the input sequence features through a multi-head mechanism, thus obtaining self-attention features;
[0047] The self-attention features are input into the feedforward neural network, and residual connections and layer normalization are performed to obtain the encoder's output features. .
[0048] Furthermore, the specific method for inputting the encoder's output features into the decoder for decoding is as follows:
[0049] The output features of the encoder As the input sequence feature D0 of the decoder, sinusoidal position encoding is performed on the input sequence feature D0 to add position encoding to the input sequence feature D0;
[0050] Masked multi-head attention is used to handle the dependencies within the features after the position of feature D0 in the input sequence changes;
[0051] Next, cross-attention is used to process the mask multi-head attention output features and the encoder output features. Dependencies between them;
[0052] The encoder output features are further extracted using a feedforward neural network based on the cross-attention output features. .
[0053] On the other hand, the present invention also provides a robot motion prediction system based on cross-modal feature enhancement, including: a robot motion sequence acquisition module, a joint motion sequence acquisition module, an embedded feature acquisition module, a motion feature fusion module, a visual feature extraction and fusion module, a cross-modal fusion feature enhancement module, and a motion prediction module;
[0054] The robot motion sequence acquisition module uses an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquires the joint motion information of the robot body to form a continuous motion sequence of the robot body.
[0055] The joint motion sequence acquisition module performs fixed-length sampling on the continuous motion sequence of the robot body to obtain a series of joint motion sequences;
[0056] The embedding feature acquisition module embeds the joint motion sequence and robot joint motion information into a representation, and obtains the joint motion sequence embedding feature and the robot joint motion information embedding feature.
[0057] The action feature fusion module inputs the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly samples the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence; the latent spatial features are concatenated with the robot joint motion information embedding features to obtain the fused action features;
[0058] The visual feature extraction and fusion module uses two independent feature extraction networks to extract visual features from RGB images and depth images respectively and then stitches them together to obtain multimodal visual features;
[0059] The cross-modal fusion feature enhancement module uses an attention mechanism to perform cross-modal fusion enhancement of multimodal visual features and fused action features to obtain cross-modal action enhancement features;
[0060] The action prediction module concatenates cross-modal action enhancement features and multimodal visual features and inputs them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving action prediction.
[0061] Thirdly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the robot motion prediction method based on cross-modal feature enhancement.
[0062] The beneficial effects of adopting the above technical solution are as follows: Compared with the imitation learning schemes in the prior art that rely on static feature fusion (such as splicing and weighted averaging), the robot action prediction method and system based on cross-modal feature enhancement provided by this invention achieve significant performance improvement by introducing a cross-modal fusion feature enhancement module based on an attention mechanism. Its core advantage lies in establishing a dynamic and fine-grained bidirectional interaction mechanism between the visual modality (RGB image + depth image) and the action modality (joint motion sequence + latent spatial features). Specifically, this module allows action features to be used as query features, actively retrieving spatial regions and depth geometric information (such as the precise contour of the target object, texture details near the grasping point, or distance information required for obstacle avoidance) that are highly relevant to the current action intention from the visual features, and dynamically assigning weights based on relevance for feature fusion and enhancement. This mechanism overcomes the defect of static fusion assuming that modal contributions are fixed, enabling the model to adaptively focus on the most critical parts of the visual information according to the specific task context (such as object pose and environmental occlusion state). The precise three-dimensional spatial relationships contained in the depth map and the rich texture and color cues provided by the RGB image can more effectively guide action generation. Its direct effect is a significant improvement in the system's adaptability and robustness in complex, unstructured dynamic environments (such as random occlusion, lighting changes, and fine-tuning of object posture). On the one hand, motion prediction is more sensitive to changes in visual cues and can adjust the predicted trajectory in real time based on subtle environmental differences (such as the small displacement after an object slides). On the other hand, the fused and enhanced features significantly improve the accuracy and spatiotemporal continuity of joint motion sequences (especially future frames), thus demonstrating superior performance in dexterous manipulation tasks that require high-precision spatiotemporal coordination (such as fine assembly and grasping deformable objects). Attached Figure Description
[0063] Figure 1 A flowchart illustrating the robot motion prediction method based on cross-modal feature enhancement provided in an embodiment of the present invention;
[0064] Figure 2 This is a structural block diagram of an attention-based cross-modal fusion feature enhancement module provided in an embodiment of the present invention.
[0065] Figure 3 This is a structural block diagram of a robot motion prediction system based on cross-modal feature enhancement provided in an embodiment of the present invention. Detailed Implementation
[0066] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0067] Example 1:
[0068] In this embodiment, a robot motion prediction method based on cross-modal feature enhancement is described, such as... Figure 1 As shown, it includes the following steps:
[0069] Step 1: Use an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquire joint motion information of the robot body to form a continuous action sequence of the robot body.
[0070] In this embodiment, RGB and depth images are simultaneously acquired using RGB and depth cameras fixed to the robot body. The motion information of each joint is synchronously acquired in real time via the robot body's joint encoders, forming a continuous sequence of robot movements.
[0071] Step 2: Obtain the joint motion sequence of the robot body, and embed the joint motion sequence with the robot joint motion information to obtain the joint motion sequence embedding feature and the robot joint motion information embedding feature.
[0072] The joint motion sequence embedding features and robot joint motion information embedding features are input into a self-attention encoder, and the output features are randomly sampled according to a normal distribution to obtain the latent spatial features of the joint motion sequence, so as to capture the latent spatial distribution of the joint motion sequence. Then, the normally distributed random sampled features are concatenated with the robot joint motion information embedding features to obtain the fused motion features.
[0073] Step 2-1: Obtain the joint motion sequence of the robot body, and embed the joint motion sequence with the robot joint motion information to obtain the joint motion sequence embedding feature and the robot joint motion information embedding feature.
[0074] First, the continuous motion sequence of the robot body is sampled at a fixed length to obtain a series of joint motion sequences of length L; let... Let D represent the joint motion state at time t, where D is the number of joint degrees of freedom; then a joint motion sequence of length L can be represented as: ;
[0075] To facilitate subsequent processing, the joint motion sequence Q and the robot joint motion state need to be... Embedding representation is performed through an embedding layer to obtain the joint motion sequence embedding features. Embedded features of robot joint motion information Embedding layers can map high-dimensional sparse inputs to low-dimensional dense vector spaces; let the embedding function be... Then the joint motion sequence embedding features for:
[0076] ;
[0077] in, For the embedded dimension;
[0078] Robot joint motion information embedding features for:
[0079] ;
[0080] Step 2-2: Embed the joint motion sequence into features Embedded features with robot joint motion information splicing results in splicing features It is then input into a self-attention encoder, which uses a self-attention mechanism to capture the temporal dependencies in the joint motion sequence;
[0081] The output features of a self-attention encoder can be represented as follows: ,in Let be the dimension of the hidden layer of the self-attention encoder; the specific calculation process of the self-attention encoder is as follows:
[0082] Input features of the self-attention encoder The query matrix Q, the key matrix K, and the value matrix V are obtained through three linear transformations:
[0083] ;
[0084] ;
[0085] ;
[0086] in, , and All are learnable weight matrices;
[0087] Calculate the attention weight matrix A:
[0088] ;
[0089] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.
[0090] The output features of the self-attention encoder are obtained by weighted summation of the value matrix V based on the attention weights. :
[0091] ;
[0092] Steps 2-3: Analyze the output features of the self-attention encoder. Normally distributed random sampling is performed to obtain the latent spatial features of joint motion sequences in order to capture the latent spatial distribution of joint motion sequences;
[0093] First, the output features H of the self-attention encoder are mapped to the mean μ and variance through a linear transformation. :
[0094] ;
[0095] ;
[0096] in, and This is a learnable weight matrix.
[0097] Then, from the normal distribution Random sampling is performed to obtain the latent spatial features z;
[0098] Steps 2-4: Embed the latent spatial features z obtained by random sampling from a normal distribution with the robot joint motion information. By splicing the data, we can obtain the fused motion features. ;
[0099] Before entering the cross-modal fusion feature enhancement module, this invention employs independent feature extraction networks to process RGB and depth images separately, extracting their respective high-level semantic and spatial geometric features. These two heterogeneous but complementary visual features are then concatenated to form a unified, more information-rich fused visual feature representation. This step is crucial, providing a high-quality input foundation containing color texture (RGB) and accurate 3D spatial information (depth) for subsequent cross-modal attention interactions, ensuring the enhancement module can effectively utilize the advantages of both visual modalities.
[0100] Step 3: Use two independent feature extraction networks to extract visual features from the RGB image and depth image respectively and then stitch them together to obtain multimodal visual features; then input the multimodal visual features and the fused action features obtained in Step 2 into the cross-modal fusion feature enhancement module, and achieve effective fusion and feature enhancement between modalities through the attention mechanism to obtain cross-modal action enhancement features;
[0101] Step 3-1: Use two independent feature extraction networks to extract visual features from the RGB image and depth image respectively, and then stitch them together to obtain multimodal visual features, which are then input into the cross-modal fusion feature enhancement module.
[0102] Let the RGB image input to the feature extraction network be... ,in The height of the RGB image. Given the width of an RGB image, a convolutional neural network is used to extract its visual features, outputting an RGB feature map. ,in, and Here, C represents the height and width of the RGB feature map, and C represents the number of channels in the RGB feature map.
[0103] Let the depth image input to the feature extraction network be... ,in The depth image height, The depth image width is used to extract its features through a separate CNN network, outputting a depth feature map. ,in This represents the number of channels in the depth feature map.
[0104] The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are concatenated along the channel dimension to obtain multimodal visual features. ;
[0105] Step 3-2: The cross-modal fusion feature enhancement module uses an attention mechanism to interactively fuse multimodal visual features and fused action features to obtain cross-modal action enhancement features; for example... Figure 2 As shown, the specific steps are as follows:
[0106] By fusing action features and multimodal visual features Generate the query matrix by generating three linear transformations. Key matrix Sum matrix :
[0107] Query Matrix Based on the characteristics of fusion action Generate (i.e., output features from step 2):
[0108] ;
[0109] Key matrix Sum matrix Based on the characteristics of fusion action and multimodal visual features The fusion characteristics of splicing generate:
[0110] ;
[0111] ;
[0112] in, , and All are learnable weight matrices;
[0113] Calculate fusion action features Fusion features Attention weight matrix :
[0114] ;
[0115] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.
[0116] Based on the attention weight log matrix We perform weighted summation to obtain cross-modal action enhancement features. :
[0117] .
[0118] The cross-modal fusion feature enhancement module is specifically designed to achieve dynamic, bidirectional, and fine-grained interaction and feature enhancement between the visual modality (RGB image features + depth image features) and the action modality (stitched historical joint motion features + latent spatial features). It utilizes attention mechanisms (such as cross-attention) to allow action features to act as query features, actively retrieving and weighting the most relevant visual information from the fused visual features (which serve as key and value features), and aggregating them accordingly. This mechanism overcomes the limitations of static stitching or weighted averaging in existing technologies, achieving adaptive selection and deep fusion of information between modalities, significantly improving the accuracy of visual information in guiding action decisions.
[0119] Step 4: Concatenate the cross-modal motion enhancement features and multimodal visual features and input them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving accurate motion prediction;
[0120] Step 4-1: Combine multimodal visual features With cross-modal action enhancement features By splicing, multimodal fusion features are obtained. .
[0121] Step 4-2: Fuse multimodal features The input is then encoded by the encoder. The specific steps are as follows:
[0122] The encoder uses a sine function to generate positional codes from multimodal fusion features, forming the encoder input sequence features. ;
[0123] The encoder's self-attention module captures the global dependencies of the input sequence features through a multi-head mechanism, thus obtaining self-attention features;
[0124] Let the number of attention heads be h, and the query matrix be h. Key matrix Value matrix By respectively analyzing the features of the encoder input sequence Generate by performing three linear transformations:
[0125] ;
[0126] in, , All are learnable weight matrices;
[0127] Based on the query matrix Key matrix Value matrix Calculate self-attention features As shown in the formula below:
[0128] ;
[0129] in, For normalization function, The dimensions are the query and the key, and T is the matrix transpose.
[0130] Self-attention characteristics The input is fed into a feedforward neural network, and residual connections and layer normalization are performed to obtain the encoder's output features. ;
[0131] Step 4-3: Input the encoder's output features into the decoder for decoding. The specific steps are as follows:
[0132] The output features of the encoder As the input sequence feature D0 of the decoder, sinusoidal position encoding is performed on the input sequence feature D0 to add position encoding to the input sequence feature D0;
[0133] Masked multi-head attention is used to handle the dependencies within features after changes in the position D0 of the input sequence, and the query matrix is used. Key matrix Value matrix The features D0 of the input sequence are generated by performing three linear transformations respectively:
[0134] ;
[0135] in, , and All are learnable weight matrices;
[0136] Attention weight Through the mask matrix Suppress information about future locations:
[0137] ;
[0138] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose. It is a lower triangular mask matrix;
[0139] The masked multi-head attention output feature D′ is the attention weight. value matrix Perform a weighted sum:
[0140] ;
[0141] Next, cross-attention is used to process the masked multi-head attention output feature D′ and the encoder output feature. Dependencies between them:
[0142] Query Matrix The key matrix is generated from the decoder feature D′. Sum matrix Output from encoder generate:
[0143] ;
[0144] in, , , All are learnable weight matrices;
[0145] Cross-attention output features are :
[0146] ;
[0147] in, For normalization function, Let T be the dimension of the query and the key, and T be the matrix transpose.
[0148] The encoder output features are further extracted using a feedforward neural network based on the cross-attention output features. .
[0149] Step 4-4: The final output of the decoder... By mapping to the joint motion space through a fully connected layer, a future joint motion sequence of length L is predicted.
[0150] The cross-modal motion enhancement features (containing visual information extracted after deep interaction and enhanced motion context) output by the cross-modal fusion feature enhancement module are concatenated with multimodal visual features and input into a self-attention decoder. This decoder uses the enhanced cross-modal information to focus on predicting the future motion sequence of the robot's joints. This step directly utilizes the multimodal interaction enhancement features generated by the cross-modal fusion feature enhancement module, ensuring the accuracy and spatiotemporal continuity of the final motion prediction, especially its adaptability to subtle visual changes in complex and dynamic scenes.
[0151] Example 2:
[0152] This embodiment provides a robot motion prediction system based on cross-modal feature enhancement, such as... Figure 3 As shown, it includes: robot motion sequence acquisition module, joint motion sequence acquisition module, embedded feature acquisition module, motion feature fusion module, visual feature extraction and fusion module, cross-modal fusion feature enhancement module, and motion prediction module;
[0153] The robot motion sequence acquisition module uses an RGB camera and a depth camera fixed to the robot body to acquire RGB images and depth images, and simultaneously acquires the joint motion information of the robot body to form a continuous motion sequence of the robot body.
[0154] The joint motion sequence acquisition module performs fixed-length sampling on the continuous motion sequence of the robot body to obtain a series of joint motion sequences;
[0155] The embedding feature acquisition module embeds the joint motion sequence and robot joint motion information into a representation, and obtains the joint motion sequence embedding feature and the robot joint motion information embedding feature.
[0156] The action feature fusion module inputs the joint motion sequence embedding features and the robot joint motion information embedding features into the self-attention encoder and randomly samples the output features according to a normal distribution to obtain the latent spatial features of the joint motion sequence in order to capture the latent spatial distribution of the joint motion sequence; the latent spatial features are concatenated with the robot joint motion information embedding features to obtain the fused action features;
[0157] The visual feature extraction and fusion module uses two independent feature extraction networks to extract visual features from RGB images and depth images respectively and then stitches them together to obtain multimodal visual features;
[0158] The cross-modal fusion feature enhancement module uses an attention mechanism to perform cross-modal fusion enhancement of multimodal visual features and fused action features to obtain cross-modal action enhancement features;
[0159] The action prediction module concatenates cross-modal action enhancement features and multimodal visual features and inputs them into the attention encoder-decoder to predict the robot's future joint motion sequence, thereby achieving action prediction.
[0160] Example 3:
[0161] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the robot motion prediction method based on cross-modal feature enhancement.
[0162] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.
[0163] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.
Claims
1. A robot action prediction method based on cross-modal feature enhancement, characterized in that, The application relates to a robot action prediction method and device. RGB images and depth images are collected by using an RGB camera and a depth camera fixed on a robot body, and joint motion information of the robot body is synchronously acquired to form a continuous action sequence of the robot body; A continuous action sequence of the robot body is sampled to obtain a series of joint motion sequences; The joint motion sequence and the robot joint motion information are embedded to obtain joint motion sequence embedding features and robot joint motion information embedding features; The joint motion sequence embedding features and the robot joint motion information embedding features are input into a self-attention encoder, and output features are randomly sampled in a normal distribution to obtain latent space features of the joint motion sequence, so that the latent space distribution of the joint motion sequence is captured; The normal distribution random sampling features and the robot joint motion information embedding features are spliced to obtain fused action features; Two independent feature extraction networks are used to extract visual features of the RGB images and the depth images respectively, and the visual features are spliced to obtain multi-modal visual features; The multi-modal visual features and the fused action features are input into a cross-modal fusion feature enhancement module to obtain cross-modal action enhancement features based on an attention mechanism; The cross-modal action enhancement features and the multi-modal visual features are spliced and input into an attention encoding-decoding encoder to predict future joint motion sequences of the robot, so that action prediction is realized. 2.The robot action prediction method based on cross-modal feature enhancement according to claim 1, wherein, The joint motion sequence embedding features and the robot joint motion information embedding features are obtained by embedding the joint motion sequence and the robot joint motion information, and the embedding method is as follows: Setting where D is the number of joint degrees of freedom; a joint motion sequence of length L is denoted as ; joint movement sequence Q and robot joint movement state obtaining joint movement sequence embedding features by embedding representation through an embedding layer and robot joint movement information embedding features An embedding layer maps high-dimensional sparse input to a low-dimensional dense vector space. 3.The robot action prediction method based on cross-modal feature enhancement according to claim 2, characterized in that, The joint motion sequence embedding features and the robot joint motion information embedding features are input into a self-attention encoder, and output features are randomly sampled in a normal distribution to obtain latent space features of the joint motion sequence, so that the latent space distribution of the joint motion sequence is captured; The embedded joint motion sequence is embedded into a feature The embedded robot joint motion information is embedded into a feature The spliced features are obtained by splicing and input into a self-attention encoder, the self-attention encoder adopts a self-attention mechanism to capture time sequence dependency in the joint motion sequence, and outputs features ; Output features of a self-attention encoder Normal distribution random sampling is performed to obtain latent space features z of the joint motion sequence; Embedding the latent space feature z sampled by normal distribution random sampling and the robot joint motion information into the feature Perform splicing to obtain fused action features .
4. The robot action prediction method based on cross-modal feature enhancement according to claim 3, characterized in that, The two independent feature extraction networks are used to extract visual features of the RGB images and the depth images respectively, and the visual features are spliced to obtain multi-modal visual features, and the specific method is as follows: extract visual features of the RGB image through a convolutional neural network, output an RGB feature map ; The features of the depth image are extracted through a CNN network, and a depth feature map is output as ; The RGB feature map corresponding to the RGB image and the depth feature map corresponding to the depth image are spliced along the channel dimension to obtain a multi-modal visual feature .
5. The robot action prediction method based on cross-modal feature enhancement according to claim 4, characterized in that, The cross-modal fusion feature enhancement module adopts an attention mechanism to interactively fuse the multi-modal visual features and the fused action features to obtain cross-modal action enhancement features, and the specific method is as follows: By fusing action features and multi-modal visual features to generate three linear transformations to obtain a query matrix , a key matrix and a value matrix : Query matrix From fused action features Generated: ； wherein, , and are weight matrices; key matrix and value matrix fusion features and multimodal visual features concatenated fusion features generation: ； ； Computing fused action features Attention weight matrix for fused features : ； wherein, is a normalization function, is the dimension of the query and key, T is the matrix transpose; performing weighted sum on the value matrix according to the attention weight to obtain a cross-modal action enhancement feature performing weighted sum on the value matrix according to the attention weight to obtain a cross-modal action enhancement feature : 。 6. The robot action prediction method based on cross-modal feature enhancement according to claim 5, characterized in that, The cross-modal action enhancement features and the multi-modal visual features are spliced and input into an attention encoding-decoding encoder to predict future joint motion sequences of the robot, and the specific method is as follows: Multimodal visual features with cross-modal action augmented features concatenation, to obtain multimodal fusion features ; Fusing multi-modal features input to an encoder for encoding; The output features of the encoder are input into the decoder for decoding; The final output of the decoder is mapped to a joint motion space through a full connection layer to predict a future joint motion sequence with a length of L.
7. The robot action prediction method based on cross-modal feature enhancement according to claim 6, characterized in that, Fusing multi-modal features The specific method of inputting to the encoder for encoding is: The encoder generates position encoding on the multimodal fusion feature by using a sine function to form the encoder input sequence feature ; The self-attention module of the encoder captures global dependency relationships of input sequence features through a multi-head mechanism to obtain self-attention features; The self-attention features are input into a feedforward neural network, and a residual connection and layer normalization are performed to obtain output features of the encoder .
8. The robot action prediction method based on cross-modal feature enhancement according to claim 7, characterized in that, The output features of the encoder are input into the decoder for decoding, and the specific method is as follows: The output features of the encoder are The input sequence features D0 of the decoder are sinusoidal position encoded to add position encoding in the input sequence features D0 of the decoder; The input sequence features D0 are processed by using a mask multi-head attention to capture the internal dependency relationships of the features after position changes; Next, cross attention is used to process the dependency between the masked multi-head attention output features and the encoder output features between the masked multi-head attention output features and the encoder output features The cross-attention output feature continues to extract features through a feedforward neural network to obtain an encoder output feature .
9. A robot action prediction system based on cross-modal feature enhancement, implemented based on the robot action prediction method based on cross-modal feature enhancement of claim 1, characterized in that, The application relates to a robot action prediction method and device. The robot action prediction method comprises a robot action sequence acquisition module, a joint motion sequence acquisition module, an embedding feature acquisition module, an action feature fusion module, a visual feature extraction and fusion module, a cross-modal fusion feature enhancement module and an action prediction module. The robot action sequence acquisition module acquires RGB images and depth images by using an RGB camera and a depth camera fixed on the robot body, and synchronously acquires joint motion information of the robot body, to form a continuous action sequence of the robot body; The joint motion sequence acquisition module performs fixed-length sampling on the continuous action sequence of the robot body, to obtain a series of joint motion sequences; The embedded feature acquisition module performs embedded representation on the joint motion sequences and the robot joint motion information, to obtain joint motion sequence embedded features and robot joint motion information embedded features; The action feature fusion module inputs the joint motion sequence embedded features and the robot joint motion information embedded features into a self-attention encoder, and performs normal distribution random sampling on output features, to obtain latent space features of the joint motion sequences, so as to capture latent space distribution of the joint motion sequences; the latent space features are spliced with the robot joint motion information embedded features, to obtain fused action features; The visual feature extraction and fusion module extracts visual features of the RGB images and the depth images by using two independent feature extraction networks, and splices the visual features, to obtain multi-modal visual features; The cross-modal fusion feature enhancement module performs cross-modal fusion and enhancement on the multi-modal visual features and the fused action features based on an attention mechanism, to obtain cross-modal action enhanced features; The action prediction module splices the cross-modal action enhanced features and the multi-modal visual features, and inputs the spliced features into an attention encoding-decoding predictor, to predict future joint motion sequences of the robot, so as to realize action prediction.
10. A computer program product for performing the method of robot action prediction based on cross-modal feature augmentation according to any one of claims 1-8, characterized in that, The computer program or instructions are executed by a processor to realize the robot action prediction method based on cross-modal feature enhancement. The computer program or instructions are executed by a processor to realize the robot action prediction method based on cross-modal feature enhancement.