Gesture understanding and robot action generation method and system based on conditional diffusion

Abstract

This invention discloses a method and system for gesture understanding and robot motion generation based on conditional diffusion. The method includes: segmenting a panoramic image into gesture sub-images and preserving the panoramic image as a panoramic scene image; obtaining gesture visual features based on the gesture sub-images; obtaining gesture embedding features based on the gesture sub-images; obtaining environmental context features based on the panoramic scene image; fusing the gesture visual features and environmental context features to obtain a fused feature vector; performing multimodal stitching to obtain conditional features; and generating motion vectors for the robot's robotic arm to execute using a conditional diffusion model with random noise as the initial source and through multi-step iterative denoising. This invention, through multimodal feature extraction, bi-branch attention fusion, and a conditional diffusion generation network, deeply integrates gesture vision, 3D keypoints, and scene semantics to generate robot motions, enabling efficient and robust execution of gesture input into the robot's closed-loop control.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and intelligent control technology, specifically relating to a method and system for gesture understanding and robot motion generation based on conditional diffusion. Background Technology

[0002] Currently, most gesture-based robot interaction systems rely on mapping gesture images or key points to predefined action instruction tables, or using a single classification model to identify gesture categories before triggering corresponding actions. These methods only focus on the shape or trajectory of the gesture itself, ignoring environmental information such as the distribution and positional relationships of objects in the operating scene. This results in the inability to accurately determine the user's true intention in different scenarios, easily leading to execution errors or mistakes.

[0003] To address these issues, some studies have begun to concatenate or simply fuse gesture features with environmental features, hoping to improve the accuracy of intent recognition by leveraging contextual information. However, due to a lack of in-depth exploration of the deep correlation between the two, this coarse feature fusion approach still struggles to distinguish the specific target pointed to by the gesture and cannot respond to user needs in real-time and stably in complex dynamic scenarios. Furthermore, existing methods often rely on large amounts of labeled data or complex rule designs, resulting in high training and deployment costs and difficulty adapting to scene changes or new action requirements. Meanwhile, diffusion models demonstrate excellent progressive denoising capabilities in image generation and action prediction, gradually generating high-quality samples conforming to conditional distributions from random noise; however, no research has yet applied them to online gesture intent understanding and robot action generation. How to combine the denoising advantages of diffusion models with gesture / scene fusion strategies to achieve accurate and adaptive action generation remains a key area for technological breakthroughs. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for gesture understanding and robot motion generation based on conditional diffusion, to solve at least one or more problems such as ambiguity arising from relying solely on the gesture itself, insufficient utilization of environmental context, and inadequate accuracy and stability of motion generation. This invention deeply integrates gesture vision, 3D keypoints, and scene semantics through multimodal feature extraction, bi-branch attention fusion, and conditional diffusion generation networks to generate robot motions, achieving efficient and robust execution from gesture input to closed-loop robot control.

[0005] To achieve the objectives of this invention, a method for gesture understanding and robot motion generation based on a conditional diffusion strategy is proposed, comprising the following steps:

[0006] Gesture segmentation is performed on the panoramic image to obtain gesture sub-images, and the panoramic image is then preserved as a panoramic scene image;

[0007] Based on the gesture sub-image, visual features of gestures are extracted using a deep convolutional neural network.

[0008] Based on the gesture subgraph, key points are extracted and mapped to gesture embedding features;

[0009] Based on the panoramic scene image of the same frame, environmental context features are extracted through a scene feature extraction network;

[0010] A dual-branch attention module is used to fuse gesture visual features and environmental context features to highlight hand orientation and posture information, explore the deep interaction between gesture visual features and environmental context features, and obtain a fused feature vector.

[0011] The robot's state features, fused feature vectors, and gesture embedding features are multimodally concatenated to obtain conditional features;

[0012] Using conditional features as input, a conditional diffusion model is used to generate motion vectors for the robot's robotic arm to execute through multi-step iterative denoising, starting with random noise. During the operation phase, the generated robot motion vectors are sent to the robotic arm for execution.

[0013] Furthermore, a hand segmentation network is used to segment gestures in panoramic images, and pixel-level training is performed using the cross-entropy loss function.

[0014] Furthermore, the hand segmentation network employs a deep learning-based semantic segmentation structure, U-Net or DeepLabv3+.

[0015] Furthermore, during keypoint extraction, 21 three-dimensional keypoints were extracted from the gesture area.

[0016] Furthermore, based on the keypoint matrix, keypoint embedding mapping is performed using a multilayer perceptron to obtain the gesture embedding features.

[0017] Furthermore, the scene feature extraction network includes a feature extraction network, a convolutional attention module, and a global average pooling layer. The feature extraction network is used to extract deep convolutional features from the panoramic scene image. The convolutional attention module is used to enhance attention to important regions of the input and adjust the response to different features through channel attention and spatial attention. The global average pooling layer is used to output environmental context features.

[0018] Furthermore, the dual-branch attention module includes a self-attention branch and a cross-attention branch. The self-attention branch is used to perform attention weighting on the visual features of the gesture through a multi-head self-attention mechanism, capturing the global dependency between pixels at different positions within the gesture sub-image, and highlighting the hand direction and posture information. The cross-attention branch uses the output of the self-attention branch as a query and performs cross-attention alignment with the key-value pairs of the environmental context features to automatically focus on the environmental target pointed to by the gesture and extract relevant contextual information. By cascading the outputs of the self-attention branch and the cross-attention branch, a fused feature vector that includes both gesture dynamics and reflects environmental semantics is obtained.

[0019] Furthermore, when training and optimizing the conditional diffusion model, the model parameters are jointly optimized by minimizing the noise prediction error and the teaching action reconstruction error.

[0020] Furthermore, during the training phase, the real action vectors obtained from teaching are used as supervision signals to optimize the noise prediction network.

[0021] The present invention also provides a gesture understanding and robot motion generation system based on conditional diffusion.

[0022] The present invention also provides a computer device.

[0023] The present invention also provides a computer-readable storage medium.

[0024] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0025] (1) This invention deeply integrates user gestures, environmental scenes and robot state information, and combines the generation advantages of conditional diffusion stepwise denoising to achieve accurate understanding of gesture intentions in dynamic scenes and high-quality action generation, which significantly improves the naturalness, accuracy and robustness of human-machine collaboration.

[0026] (2) The present invention can output high-quality, smooth and continuous robot motion by using an iterative noise reduction method, so as to realize efficient and robust execution of gesture input to robot closed-loop control. Attached Figure Description

[0027] Figure 1 This is a flowchart of a gesture understanding and robot motion generation method based on conditional diffusion provided in an embodiment of the present invention.

[0028] Figure 2 A detailed process diagram illustrating the method of the embodiments of the present invention is provided.

[0029] Figure 3 A visualization diagram of the model input data in an embodiment of the present invention.

[0030] Figure 4A schematic diagram showing the experimental results of gesture interaction on a real robot according to an embodiment of the present invention. Detailed Implementation

[0031] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Furthermore, it should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0032] This embodiment of the gesture understanding and robot motion generation method based on the conditional diffusion strategy includes the following steps:

[0033] Gesture segmentation is performed on the panoramic image to obtain gesture sub-images, and the panoramic image is then preserved as a panoramic scene image;

[0034] Based on the gesture sub-image, visual features of gestures are extracted using a deep convolutional neural network.

[0035] Based on the gesture subgraph, key points are extracted and mapped to gesture embedding features;

[0036] Based on the panoramic scene image of the same frame, environmental context features are extracted through a scene feature extraction network;

[0037] A dual-branch attention module is used to fuse gesture visual features and environmental context features to highlight hand orientation and posture information, explore the deep interaction between gesture visual features and environmental context features, and obtain a fused feature vector.

[0038] The robot's state features, fused feature vectors, and gesture embedding features are multimodally concatenated to obtain conditional features;

[0039] Using conditional features as input, a conditional diffusion model is used to generate motion vectors for the robot's robotic arm to execute through multi-step iterative denoising, starting with random noise. During the operation phase, the generated robot motion vectors are sent to the robotic arm for execution.

[0040] In a further optimized implementation, the conditional diffusion model starts from random noise with a standard normal distribution and generates action vectors through multiple iterations according to the following formula:

[0041]

[0042] In the formula, It is the robot motion vector from the previous iteration step. It is a noise term sampled from a normal distribution. The learnable diffusion coefficient, This is for noise estimation.

[0043] In the multimodal feature extraction stage, this invention independently encodes the gesture sub-image and the panoramic scene image. Gesture sub-image For the hand region accurately cropped by a semantic segmentation network (hand segmentation network), high-dimensional visual features are extracted using a deep convolutional neural network (e.g., ResNet-18):

[0044]

[0045] It is a visual feature of gestures The dimensions represent the height, width, and number of channels, respectively.

[0046] Meanwhile, based on 21 3D key points extracted by Mediapipe Gesture embedding features are generated using multilayer perceptron (MLP) mapping. :

[0047]

[0048] Represents gesture keypoint embedding features Dimensions This indicates that the keypoint matrix is ​​flattened. For activation function, , , , These are learnable parameters. For panoramic scene image I C Another lightweight CNN is used to extract environmental context features:

[0049]

[0050] It is an environmental context feature The dimensions represent the height, width, and number of channels, respectively.

[0051] This stage simultaneously captures visual and geometric information of gestures, as well as environmental semantics and spatial layout, providing a comprehensive feature foundation for subsequent accurate fusion.

[0052] Secondly, in the dual-branch attention fusion stage, the dual-branch attention module designs two branches—self-attention and cross-attention—for gestures and scenes to uncover deep interactions between multimodal features. The self-attention branch first... Applying multi-head self-attention:

[0053]

[0054] This operation captures the global dependencies between pixels in the hand, allowing the model to focus on the most recognizable areas of the gesture; the cross-attention branch then... As a query, As a key-value alignment:

[0055]

[0056] This cross-attention mechanism automatically focuses on the environmental target being pointed at by the gesture and extracts relevant contextual information. Finally, the two outputs are cascaded.

[0057]

[0058] Combined with key features They will be moved to the next stage together. Represents the concatenated feature vector Dimensions.

[0059] Subsequently, in the condition feature construction stage, this invention further incorporates the robot's own state information to ensure that motion generation takes into account the real-time constraints of the robotic arm. The robot state features include the robot joint angle vector. and the spatial pose vector of the end effector The final conditional features obtained Represented as:

[0060]

[0061] This vector comprehensively reflects the user's intent, environmental context, and robot state, providing rigorous conditions for the next step of fine generation.

[0062] Next, in the conditional diffusion action generation stage, high-quality action commands are generated by progressively denoising based on the conditional diffusion model. The conditional diffusion model denoises random noise from a standard normal distribution. Starting from this point, the action vector is generated iteratively in multiple steps according to the following formula:

[0063]

[0064] in, For noise prediction networks, This is a learnable diffusion coefficient. This process smoothly removes noise and gradually recovers action vectors that conform to the conditional distribution. .

[0065] During the training and optimization phases, the parameters of the model are jointly optimized by minimizing the noise prediction error and the teaching action reconstruction error. The loss function is expressed as:

[0066]

[0067] in It is real Gaussian noise. For demonstrating the action, where is the weighting coefficient. This joint loss enables the model to accurately fit the noise in the diffusion process while ensuring that the final generated action is highly consistent with the human teaching intention.

[0068] Finally, in the post-processing stage of the action, to eliminate residual jitter during iteration, this invention adds exponentially weighted smoothing to the action output:

[0069]

[0070] This ensures that the robotic arm has both high responsiveness and smooth, continuous movements.

[0071] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution of this invention is provided through a specific embodiment based on a real human-computer interaction example process.

[0072] A gesture understanding and robot motion generation system based on a conditional diffusion strategy is presented. The system integrates a UR3 robotic arm and a Realsense D435i camera. Users in a desktop environment teach the robot to perform grasping and resetting operations using "pointing at target" and "OK" gestures, respectively. The specific implementation steps are as follows:

[0073] Step 1: Data Acquisition and Teaching

[0074] Step 1-1: Scene and Equipment Preparation

[0075] Two typical interactive scenarios were set up in the laboratory: Scenario A was a kitchen countertop, with graspable objects such as oranges placed on it; Scenario B was a living room tabletop, with target items such as remote controls placed on it. A RealSense D435iRGB-D camera was installed directly in front of the robotic arm's field of view, and the intrinsic and extrinsic parameters were adjusted to ensure full coverage of the target area.

[0076] Steps 1-2: Synchronize image acquisition and construct a teaching dataset.

[0077] Activate the camera and simultaneously capture RGB image I and depth map D at 30 FPS. The user sequentially demonstrates various static gestures in two scenarios, each gesture lasting 5–10 seconds, accumulating approximately 200 consecutive frames. These static gestures include, but are not limited to:

[0078] ① Pointing gesture: Extend your index finger and keep it still while pointing at the target;

[0079] ② "OK" gesture: Thumb and index finger touch to form a ring and remain still;

[0080] ③ "Clenched fist" gesture: Keep your hand clenched and still to indicate stop or confirmation;

[0081] ④ "Thumbs up" gesture: Keep your thumb raised and still to indicate agreement or start;

[0082] ⑤ "Flat palm" gesture: The palm is parallel to the ground, indicating a pause or waiting.

[0083] The system organizes each frame of image into the dataset by timestamp to ensure that there are sufficient samples of each static gesture under different background and lighting conditions, so that the subsequent model can accurately learn the gesture features and intent mapping.

[0084] The teaching dataset consists of gesture demonstrations by multiple participants under various scenario conditions to collect diverse action examples and enhance the model's generalization ability.

[0085] Steps 1-3: Recording the demonstrated actions

[0086] In parallel with image acquisition, the operator precisely drives the UR3 robotic arm to perform grasping or resetting actions via a teach pendant or ROS control interface. The system reads and saves the joint angle vectors at corresponding moments in real time. and end effector pose quaternion Together they constitute the teaching label action Through a strict synchronization mechanism (i.e., the teaching action recording and image acquisition processes are carried out in parallel), it is ensured that each frame of gesture image corresponds one-to-one with the corresponding action (the image acquired at each moment and the corresponding robot state (joint angle vector)). and end effector pose quaternion (Fully synchronized) to provide a high-quality data foundation for subsequent model training.

[0087] Step 2: Perform gesture segmentation and scene preprocessing on the panoramic image using a hand segmentation network.

[0088] Step 2-1: Perform gesture segmentation on the panoramic image using a hand segmentation network to generate an initial hand mask.

[0089] The hand segmentation network employs a deep learning-based semantic segmentation structure, U-Net or DeepLabv3+, and is trained pixel-level using a cross-entropy loss function. In one embodiment, each frame of the acquired RGB image (panoramic image) I is input into the pre-trained U-Net network to obtain a hand probability map, which is then thresholded. Binarization forms the initial mask. .

[0090] Step 2-2: Apply morphological denoising.

[0091] For the initial mask Morphological closing operations (dilation followed by erosion) are applied to remove small-area noise while preserving the largest connected region, generating the final hand mask. .

[0092] Steps 2-3: Crop the gesture sub-image.

[0093] Using hand mask Press on RGB image I Cut out a hand gesture sub-image containing only the hand. Meanwhile, the original RGB image I is preserved as a panoramic scene image I. C I C = .

[0094] Steps 2-4: Preprocessing: Hand gesture sub-image With panoramic scene map I C Size standardization and normalization are implemented.

[0095] Gesture subimage With panoramic scene map I C Scale proportionally to 224×224 pixels and average the pixel values ​​by channel. and standard deviation standardization:

[0096]

[0097] These are image features after normalization.

[0098] Through the above sub-steps, gestures and scenes can be clearly separated, and inputs with a consistent format can be generated, providing high-quality data for subsequent feature extraction.

[0099] Step 3: Extraction of visual features of gestures and key points.

[0100] Step 3-1: Based on the preprocessed gesture subgraph Visual features of gestures are extracted using a deep convolutional neural network.

[0101] The convolutional neural network uses a ResNet-18 backbone network to process the preprocessed gesture sub-images. Inputting the ResNet-18 backbone network, the feature map of the last residual block is read, and then compressed into a 512-dimensional vector using global average pooling to obtain the gesture visual features. :

[0102]

[0103] Step 3-2: Normalize the visual features of the gesture to obtain the processed gesture feature vector.

[0104] visual features of gestures Batch normalization and dropout (with a dropout rate of 0.3 in one embodiment) are performed to suppress overfitting and stabilize subsequent attention computations.

[0105]

[0106] It is the processed gesture feature vector, which has undergone batch normalization and discarding operations to improve the model's performance in gesture recognition tasks.

[0107] Step 3-3: Based on the preprocessed gesture subgraph We perform keypoint detection to obtain a keypoint matrix.

[0108] Calling the Mediapipe SDK in gesture subgraphs The coordinates of 21 3D keypoints on the hand are extracted in real time to form a keypoint matrix. Each row (corresponding to a key point) is normalized to [0,1] according to the image width, height and depth range.

[0109] Steps 3-4: Based on the key point matrix Gesture embedding features are obtained by embedding key points through a multilayer perceptron.

[0110] In one embodiment, the normalized keypoint matrix Flattened into a vector The input is fed into a two-layer multilayer perceptron (MLP) for feature mapping to obtain gesture embedding features:

[0111]

[0112] The gesture embedding features are represented by a 128-dimensional vector generated by a multilayer perceptron (MLP), which represents the normalized keypoint matrix. The processing results; This indicates the key point matrix. Flattened vector; It is a matrix containing 21 three-dimensional keypoints (21 three-dimensional coordinates). The three-dimensional coordinates of each keypoint will be flattened into one dimension, resulting in a total of 63-dimensional features. , , , All are learnable parameters. For activation function, It is a 128-dimensional real space, representing the dimension of the output feature vector; It is a 63-dimensional real space representing the dimension of the input features.

[0113] In the two-layer multilayer perceptron (MLP), the first layer of the MLP has 256 hidden units, and the second layer of the MLP outputs 128 dimensions. The activation functions used are LeakyReLU and Tanh, respectively, to fully capture the geometric information of the gesture.

[0114] Through the above sub-steps, the visual and geometric information of the gesture is fused, providing multimodal conditional input for the subsequent attention module.

[0115] Step 4: Extract panoramic scene images of the same frame using a scene feature extraction network. Environmental context feature extraction is performed.

[0116] The scene feature extraction network includes a feature extraction network, a convolutional attention module, and a global average pooling layer.

[0117] Step 4-1: Use a feature extraction network to process the panoramic scene image. Extract deep convolutional features.

[0118] Preprocessed panoramic scene image The input is a pre-trained feature extraction network to extract deep convolutional features. In one embodiment, the feature extraction network uses a ResNet-18 backbone network.

[0119] Step 4-2: Enhance the focus on important regions in the input feature map through a convolutional attention module. These important regions include those located in different channels and spatial positions. By introducing channel attention and spatial attention mechanisms, the convolutional attention module enables the network to dynamically adjust its response to different features, thereby improving its ability to extract key information.

[0120] A lightweight convolutional attention module is connected after the last residual block of the ResNet-18 backbone network in step 4-1. This module comprises two parts: channel attention (SE Block) and spatial attention, used to enhance the response to the boundaries and positions of objects in the target scene. Specifically, the channel attention is used to calculate the global average features of each channel and generate channel weights, while the spatial attention is used to perform two convolutional fusions on the feature map to generate a spatial attention map.

[0121] Step 4-3: Output the environmental context features through the global average pooling layer.

[0122] The attention-weighted feature map is then subjected to global average pooling to generate a 512-dimensional environmental context feature vector, i.e., environmental context features. :

[0123]

[0124] Step 4-4: Pre-train and fine-tune the scene feature extraction network.

[0125] In one embodiment, the scene feature extraction network is pre-trained on the COCO or ADE20K dataset to identify common objects on the desktop (such as bowls, cups, remote controls, etc.), and fine-tuned for 10–20 epochs on the teaching dataset formed in steps 1-2 to improve its adaptability to the target scene and extraction accuracy.

[0126] Step 5: Use a dual-branch attention module to fuse gesture visual features and environmental context features to obtain a gesture-scene joint representation (i.e., a fused feature vector).

[0127] The bi-branch attention module includes self-attention branch and cross-attention branch.

[0128] Step 5-1: Input the visual features of the gesture into the attention branch to highlight the hand direction and posture information.

[0129] In this step, the self-attention branch focuses on the visual features of the gesture. A multi-head self-attention mechanism is applied for attention weighting to capture the global dependencies between pixels at different locations within the gesture subgraph.

[0130] In the self-attention branch, visual features of gestures Simultaneously, multi-head attention computation is performed using the query (Q), key (K), and value (V) inputs to obtain the output of the self-attention branch. :

[0131]

[0132]

[0133]

[0134]

[0135] in The dimension of the key vector is represented by , softmax represents the normalization of the dot product result, transforming it into a probability distribution, and represents the weights at each position. It is a linear transformation matrix associated with each attention head, used to map the query vector (Q), key vector (K), and value vector (V) to different subspaces respectively; This means concatenating the outputs of all attention heads into a single large vector; It is the output linear transformation matrix, responsible for mapping the spliced ​​multi-head outputs to the final output space; This represents the output feature vector obtained after applying a multi-head self-attention mechanism. In one embodiment, the number of attention heads h=8, and the key vector dimension of each attention head is... Every attention point .

[0136] This operation enables the model to adaptively identify key regions of the gesture (such as fingertips, the center of the palm, etc.), enhancing the weight of these regions in the subsequent fusion process.

[0137] After attention weighting, the original visual features of the gesture are directly processed through residual connections. The data is then added back and layer normalization is performed to stabilize the training and preserve the original feature information.

[0138] Step 5-2: Cross-attention branch converts the output of the self-attention branch into a cross-attention branch. As a query, with environmental context features Cross-attention alignment is performed on the key-value pairs to obtain the output of the cross-attention branch. :

[0139]

[0140] This step enables the model to automatically focus on the environmental region pointed to by the gesture (such as an object in the direction indicated by the finger) and extract contextual information related to that environmental region, ensuring that the fused features contain both gesture dynamics and reflect environmental semantics.

[0141] Step 5-3: Fusion Representation Concatenation: Concatenate the outputs of the self-attention branch and the cross-attention branch along the channel dimension to obtain the preliminary fusion feature. :

[0142]

[0143] This stitching operation preserves both gesture features and scene features in parallel, ensuring that no important information from either side is lost.

[0144] Step 5-4: Preliminary fusion characteristics Linear mapping and dimensionality reduction are performed to obtain the fused feature vector.

[0145] To map the concatenated high-dimensional features back to the dimension that matches the downstream modules, and to fuse and reorganize the information between channels, a linear transformation is used for dimensionality reduction:

[0146]

[0147] For fusing feature vectors; weight matrix and bias vector All parameters are learnable. This dimensionality reduction operation not only reduces the feature dimension and computational cost, but also further integrates and strengthens the most recognizable gesture-scene interaction information through weight learning, forming a unified representation (i.e., a fused feature vector) for subsequent conditional diffusion generation. ).

[0148] Step 6: Obtain the robot's state features (including the current joint angle vector and the spatial pose vector of the end effector), and perform multimodal concatenation with the fused feature vector and gesture embedding features to form the final conditional features.

[0149] This step includes the following sub-steps:

[0150] Step 6-1: Obtain robot state features.

[0151] By calling the UR3 robotic arm's status topic through the ROS interface, the current angle vectors of each joint of the robotic arm can be obtained in real time. and the spatial pose vector of the end effector (Position + Orientation Quaternion) The number of joints in a robotic arm, in one embodiment, Joint angle vector and spatial pose vector Represented as:

[0152]

[0153]

[0154] These represent the rotation angle of each joint, usually the rotation value of each joint of the robotic arm, in degrees or radians. Represents the dimension of the joint angle vector; It is the position vector of the end effector, representing the position coordinates of the robot's end effector in three-dimensional space; It is the orientation quaternion of the end effector, representing the robot's orientation or rotation state; This represents the dimension of the spatial pose vector of the end effector.

[0155] The joint angle vector and the spatial pose vector of the end effector reflect the real-time kinematic state of the robotic arm and serve as physical constraints for the generation of subsequent actions.

[0156] Step 6-2: Perform multimodal feature splicing.

[0157] fuse feature vectors With gesture embedding features The robot state features are sequentially concatenated at the vector level to obtain the final conditional features, which include the angle vectors of each joint of the robotic arm. Spatial pose vector of the end effector Then the conditional characteristics are:

[0158]

[0159] The conditional features, which are the final feature vectors, are obtained by concatenating multiple features, including gesture features, environmental context features, and robot state features. After concatenation, these features form a comprehensive representation vector.

[0160] This splicing operation reliably merges the user's intent, environmental context, and the robot's own state to form a complete conditional input.

[0161] Step 6-3: Perform layer normalization.

[0162] To avoid instability in downstream network training caused by differences in the distribution of the concatenated vectors across different dimensions, conditional features are used. Application layer normalization:

[0163]

[0164] This represents the conditional features after normalization.

[0165] This operation calculates the mean and variance along the dimension and performs a linear transformation on each element to ensure that the input distribution of conditional features in the diffusion generative network is stable and converges faster.

[0166] Step 7: Generate conditional diffusion actions through the conditional diffusion model: Using conditional features as input, the robot generates action vectors for its robotic arm to execute by using the conditional diffusion model with random noise as initial and through multi-step iterative denoising.

[0167] Step 7-1: Model Structure Initialization

[0168] In one embodiment, a conditional diffusion model is constructed using a noise prediction network. Among them, the noise prediction network A four-layer residual fully connected architecture is adopted, with each layer accepting 653-dimensional concatenated input and mapping it to 512-dimensional hidden units for residual fully connected computation. The specific structure is as follows:

[0169]

[0170] At the network input end, the normalized conditional features are... Current noisy action (Diffusion process in iteration step) (noisy action vectors) and time step encoding The data are then spliced ​​together to form the input to the noise prediction network.

[0171] Step 7-2, Noise Initialization and Scheduling

[0172] In one embodiment, the diffusion iteration steps are set to... =50 indicates that the diffusion process will take 50 steps; a diffusion coefficient sequence is generated using linear annealing. ,satisfy The diffusion process begins by initializing the noisy action as a standard normal random vector:

[0173]

[0174] This represents the number of iterations in the diffusion process. The iterative steps representing the diffusion coefficient sequence The value of represents the value in the diffusion process. During the step, the intensity of the noise. It is generated based on a linear schedule, meaning that as the iterations proceed, It will gradually decrease; This represents the value of the last step in the diffusion coefficient sequence; This represents the final noisy action vector of the diffusion process, at the number of iterations. At this time, it represents the starting point of the diffusion process. It is sampled from the standard normal distribution and is a Gaussian noise vector, representing the beginning of the diffusion process. This represents a standard normal distribution, i.e., a multidimensional Gaussian distribution with a mean of 0 and a covariance matrix of identity matrix I. Here... This indicates that the random variables in each dimension are independent and follow a normal distribution with a mean of 0 and a variance of 1; The covariance matrix representing the standard normal distribution is a matrix where the diagonal elements are 1 and the other elements are 0.

[0175] Step 7-3: Perform iterative noise reduction.

[0176] For each iteration step First, the current noisy action Conditional features after hierarchical normalization and time step coding Input noise prediction network Noise estimation is obtained. Then update the action vector according to the reverse denoising update formula:

[0177]

[0178]

[0179] in, The learnable diffusion coefficient, and Together they determine the noise reduction intensity. It is the robot motion vector from the previous time step. It is a noise term sampled from the normal distribution, representing the random perturbation introduced during the update process.

[0180] During the training and optimization phases, the parameters of the noise prediction network are jointly optimized by minimizing the noise prediction error and the teaching action reconstruction error. Specifically, optimization is achieved by minimizing the mean squared error (MSE) loss. The specific formula for calculating the loss function is as follows:

[0181]

[0182] in Indicates the first The step is real Gaussian noise, which is noise sampled from a standard normal distribution and used as the difference between the target and the predicted value; For demonstrating the action, These are the weighting coefficients; The noise is predicted by a noise prediction network based on the conditional features of the input. Current noisy action and iteration steps Calculated. Loss function. Optimization is achieved by calculating the squared Euclidean distance (i.e., mean squared error) between the predicted noise and the actual noise. This loss function is used during the training phase to minimize the difference between the actual and predicted noise, thereby enabling the noise prediction network to learn to accurately generate outputs that closely approximate the actual noise, thus improving the network's performance.

[0183] Step 7-4: Output robot motion vectors.

[0184] After the iteration is complete, retrieve the original action vector. This serves as the target incremental command for the robot's end effector. The original motion vector includes position and orientation increments, enabling the generation of a smooth and precise motion trajectory while meeting user intent and environmental constraints. The update is performed using a reverse denoising method. Iterating step by step to The final noise-free action vector is obtained by setting =0. Depart, execute repeatedly The formula until =1, the output obtained is the original action vector. .

[0185] Step 8: Perform temporal smoothing on the generated robot motion vectors and execute the smoothing process.

[0186] Step 8-1: Perform exponential weighted smoothing on the original action vector.

[0187] The original action vector generated by the conditional diffusion model Exponentially weighted smoothing is performed to eliminate any jitter that may remain in the iterative denoising process.

[0188]

[0189] in Indicates the current control moment. When the initial action vector is obtained ; Indicates the current time after smoothing. The action vector, this variable is obtained by... and The weighted average is used to eliminate possible jitter during the generation process and ensure smooth operation. This represents the smoothing factor for exponentially weighted smoothing, used to control the current time step. The original action vector and the action vector of the previous control moment The balance, the larger (Close to 1) will make the action vector at the current control moment more dependent on the action at the previous moment, thus maintaining the smoothness of the action. Smaller This would make the current action vector have a greater impact and a faster response, but it might not be smooth enough. In one embodiment, ; Indicates the current time The original action vector is generated by the conditional diffusion model. This operation ensures that the action output is both timely and continuously smooth.

[0190] Step 8-2: Perform instruction format conversion.

[0191] Smoothed motion vectors Decomposed into end position increments and attitude increment quaternion Based on the UR3 kinematic model (an existing model, which will not be elaborated here), the increment is converted into target joint angles or Cartesian space target pose, generating a MotionPlanRequest message conforming to the MoveIt! interface specification.

[0192] Step 8-3: Perform security checks and collision avoidance.

[0193] Before execution, the planned end trajectory is simulated and verified using the robot's built-in collision detection module, including:

[0194] ① Perform collision detection with the environment model (the loaded scene model);

[0195] ② Perform reachability verification with the current robotic arm joint state and kinematic constraints (joint limits, velocity and acceleration limits). If the test passes, proceed to the next level; otherwise, adjust the smoothing coefficient or regenerate the motion vector.

[0196] Step 8-4: Command issuance and execution monitoring.

[0197] The MotionPlanRequest message is sent to the UR3 control node via the ROS MoveIt! interface to initiate planning and execution. During execution, the robot arm status and force sensor feedback are monitored in real time to ensure safe movement along the expected path. In one embodiment, the average latency from camera acquisition to robot arm motion initiation is less than 100ms, and the measured task success rate exceeds 90%, effectively verifying the superiority of the method of this invention in terms of real-time performance, smoothness, and robustness.

[0198] Figure 3 This demonstrates how a robot performs tasks by perceiving user gestures, environmental context, and its own state. (a) The gesture image in Figure 1 shows the user's gestures, providing the robot with intention input; (b) The scene image in Figure 2 shows the robot's task environment, which helps with task planning; (c) The figure shows the extracted key points of the hand, which help the robot recognize gesture details; (d) The robot's human state in Figure 3 shows the robot's position and posture, ensuring that it performs the task correctly.

[0199] Figure 4This image demonstrates the complete process of a robot placing fruit into a box based on user gesture commands. (a) shows the robot grasping the fruit, and (b) shows it placing the fruit in the designated location. In frame 1, the robot responds to the user's gesture command and prepares to move the grasped fruit towards the target box. In frame 2, the robot's robotic arm begins moving the fruit towards the box and approaches the target position. In frame 3, the robot accurately places the fruit into the box. In frame 4, the robotic arm returns to its starting point, ready to perform the next task. The entire process demonstrates how the robot performs precise grasping, placing, and repositioning actions based on gesture commands.

[0200] In one embodiment, a gesture intent understanding and robot motion generation system based on conditional diffusion is provided to implement the method provided in the foregoing embodiments. The system includes the following modules:

[0201] The gesture segmentation module is used to segment the panoramic image into gesture sub-images and retain the panoramic image as a panoramic scene image.

[0202] The gesture feature extraction module is used to extract visual features of gestures based on gesture subgraphs using a deep convolutional neural network.

[0203] The key point extraction module is used to extract key points based on the gesture subgraph and map them into gesture embedding features;

[0204] The environmental feature extraction module is used to extract environmental context features based on the panoramic scene map through a scene feature extraction network.

[0205] The fusion module is used to fuse gesture visual features and environmental context features through a dual-branch attention module, to explore the deep interaction between gesture visual features and environmental context features, and to obtain a fused feature vector.

[0206] The multimodal feature stitching module is used to stitch together robot state features, fused feature vectors, and gesture embedding features in a multimodal manner to obtain conditional features;

[0207] The motion generation module is used to generate motion vectors for the robot's robotic arm to perform, by taking conditional features as input, using a conditional diffusion model with random noise as initial, and through multi-step iterative denoising.

[0208] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the foregoing embodiments.

[0209] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0210] The embodiments of the present invention can adaptively solve the semantic ambiguity problem of the same gesture in different scenarios, significantly improve the accuracy of gesture intent recognition and the robustness and adaptability of action generation, and are suitable for dynamic human-computer collaboration scenarios.

[0211] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for gesture understanding and robot motion generation based on conditional diffusion, characterized in that, Includes the following steps: Gesture segmentation is performed on the panoramic image to obtain gesture sub-images, and the panoramic image is then preserved as a panoramic scene image; Based on the gesture sub-image, visual features of gestures are extracted using a deep convolutional neural network. Based on the gesture subgraph, key points are extracted and mapped to gesture embedding features; Based on the panoramic scene map, environmental context features are extracted through a scene feature extraction network. A dual-branch attention module is used to fuse gesture visual features and environmental context features, uncovering the deep interaction between them to obtain a fused feature vector. The dual-branch attention module includes a self-attention branch and a cross-attention branch. The self-attention branch uses a multi-head self-attention mechanism to weight the gesture visual features, capturing the global dependencies between pixels at different positions within the gesture subimage and highlighting hand orientation and posture information. The cross-attention branch uses the output of the self-attention branch as a query and performs cross-attention alignment with the key-value pairs of environmental context features to automatically focus on the environmental target pointed to by the gesture and extract relevant contextual information. By concatenating the outputs of the self-attention branch and the cross-attention branch, a fused feature vector that includes both gesture dynamics and reflects environmental semantics is obtained. The robot's state features, fused feature vectors, and gesture embedding features are multimodally concatenated to obtain conditional features; Using conditional features as input, a conditional diffusion model is used to generate motion vectors for the robot's robotic arm to execute, starting with random noise and then denoising through multi-step iterations.

2. The gesture understanding and robot motion generation method based on conditional diffusion according to claim 1, characterized in that, Gesture segmentation is performed on panoramic images using a hand segmentation network, and pixel-level training is conducted using the cross-entropy loss function.

3. The gesture understanding and robot motion generation method based on conditional diffusion according to claim 1, characterized in that, Based on the keypoint matrix, the gesture embedding features are obtained by keypoint embedding mapping through a multilayer perceptron.

4. The gesture understanding and robot motion generation method based on conditional diffusion according to claim 1, characterized in that, The scene feature extraction network includes a feature extraction network, a convolutional attention module, and a global average pooling layer. The feature extraction network is used to extract deep convolutional features from the panoramic scene image, and the convolutional attention module is used to enhance attention to important regions of the input and adjust the response to different features through channel attention and spatial attention. The global average pooling layer is used to output environmental context features.

5. The gesture understanding and robot motion generation method based on conditional diffusion according to claim 1, characterized in that, When training and optimizing the conditional diffusion model, the model parameters are jointly optimized by minimizing the noise prediction error and the teaching action reconstruction error.

6. The gesture understanding and robot motion generation method based on conditional diffusion according to any one of claims 1-5, characterized in that, The conditional diffusion model starts from random noise distributed in a standard normal distribution and generates action vectors through multiple iterations according to the following formula: In the formula, It is the robot motion vector from the previous iteration step. It is a noise term sampled from a normal distribution. The learnable diffusion coefficient, For noise estimation, This is the current noisy action.

7. A gesture understanding and robot motion generation system based on conditional diffusion, characterized in that, For implementing the method according to any one of claims 1-6, the system comprises the following modules: The gesture segmentation module is used to segment the panoramic image into gesture sub-images and retain the panoramic image as a panoramic scene image. The gesture feature extraction module is used to extract visual features of gestures based on gesture subgraphs using a deep convolutional neural network. The key point extraction module is used to extract key points based on the gesture subgraph and map them into gesture embedding features; The environmental feature extraction module is used to extract environmental context features based on the panoramic scene map through a scene feature extraction network. The fusion module is used to fuse gesture visual features and environmental context features through a dual-branch attention module, to explore the deep interaction between gesture visual features and environmental context features, and to obtain a fused feature vector. The multimodal feature stitching module is used to stitch together robot state features, fused feature vectors, and gesture embedding features in a multimodal manner to obtain conditional features; The motion generation module is used to generate motion vectors for the robot's robotic arm to perform, by taking conditional features as input, using a conditional diffusion model with random noise as initial, and through multi-step iterative denoising.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images