Method and system for cross-domain human-robot motion retargeting based on skeleton graph convolutional network
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU UNIV
- Filing Date
- 2026-03-22
- Publication Date
- 2026-06-19
AI Technical Summary
Existing technologies for human-computer motion retargeting based on deep learning suffer from low motion generation efficiency, making it difficult to meet real-time requirements. Furthermore, traditional models cannot effectively extract complex human motion information, affecting prediction accuracy and stability.
A cross-domain human-robot motion retargeting method based on skeleton graph convolutional networks is adopted. By constructing graph encoding and graph decoding modules to extract human motion features, and combining diffusion models and gradient guidance mechanisms, robot joint angle representations are generated to achieve high-precision motion retargeting.
It improves the efficiency and prediction accuracy of motion retargeting, ensures the consistency between generated robot motions and human motions, and enhances overall motion stability and applicability.
Smart Images
Figure CN121904322B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-machine motion redirection technology, and in particular to a cross-domain human-machine motion redirection method and system based on skeleton graph convolutional networks. Background Technology
[0002] Human-robot motion retargeting technology refers to generating a sequence of corresponding actions that a robot can perform, based on preset transformation mapping rules, given a human demonstration action. With the continuous development of robotics technology, the demand for motion generation in various application scenarios such as service, manufacturing, and healthcare is increasing. How to efficiently and accurately generate robot actions that conform to human movement characteristics has become an important issue in current research and application.
[0003] To improve the convenience and naturalness of robot motion generation, existing technologies involve directly mapping large amounts of collected human motion data to robot platforms with different structures and degrees of freedom, thereby achieving human-robot motion retargeting. Consequently, human-robot motion retargeting methods can be divided into three categories: the first category is retargeting methods based on direct mapping rules, which typically rely on manually designed joint or pose mapping relationships; the second category is retargeting methods based on objective function optimization, which solve for actions that conform to robot motion constraints by constructing constraints and optimization objectives; and the third category is retargeting methods based on deep learning models, which use a data-driven approach to learn the mapping relationship between human and robot actions. Among these, deep learning-based human-robot motion retargeting methods have shown good results in applications such as animation production and robot control, and are gradually becoming an important research direction in this field.
[0004] In deep learning-based human-robot motion redirection methods, to achieve motion redirection between heterogeneous robots of different sizes and kinematic configurations, there are methods that learn the latent spatial representation of actions through deep reinforcement learning. This method can, to some extent, avoid dependence on solving the complete inverse kinematics and enhance the ability to extract human motion features. However, it usually relies on a complex training process, resulting in low motion generation efficiency in practical applications and making it difficult to meet the needs of applications with high real-time requirements.
[0005] To mitigate the latency introduced by perception and computation during real-time motion redirection, prediction-based motion redirection methods have emerged. These methods construct a redirection pipeline with a spatiotemporal graph structure to predict future movements in the latent space and achieve trajectory tracking based on the predicted joint movements. While this approach has achieved some improvement in dynamic tracking performance, its prediction accuracy and stability still require further enhancement. Meanwhile, in skeletal motion feature extraction, existing technologies often employ traditional variational autoencoder (VAE) models or common graph neural network (GNN) models. VAE-based methods typically only vectorize joint sequences without explicitly mapping the topological structure of the human skeleton, resulting in incomplete preservation of joint connections and topological constraints. Graph neural network-based methods can model the skeletal structure, but the model structures used are relatively basic and cannot effectively extract complex human motion information, further impacting prediction accuracy and stability. Summary of the Invention
[0006] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a cross-domain human-computer motion redirection method and system based on skeleton graph convolutional networks, which can improve the motion generation efficiency, prediction accuracy and stability of motion redirection.
[0007] To address the aforementioned technical problems, this invention provides a cross-domain human-machine motion redirection method based on skeleton graph convolutional networks, comprising:
[0008] The human motion dataset is acquired and preprocessed. Based on the human body structure and the target robot structure, human skeleton diagrams and robot skeleton diagrams are generated respectively from the human motion dataset.
[0009] A graph encoding module and a graph decoding module are constructed. Based on the human skeleton map and the robot skeleton map, the graph encoding module extracts the latent feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation.
[0010] A loss function is set according to the constraints of the motion retargeting task, wherein the constraints are used to measure the difference between the generated robot motion and the human motion.
[0011] The graph encoding module and graph decoding module are trained based on the loss function, and the trained graph encoding module and graph decoding module are used to obtain robot joint angle representation data;
[0012] A diffusion model is constructed based on a convolutional neural network, and the diffusion model is trained using the robot joint angle representation data;
[0013] During the diffusion model inference stage, a gradient guidance mechanism is introduced to generate and optimize the robot's joint angles, and output the final executable actions of the robot.
[0014] Furthermore, based on the human body structure and the target robot structure, human skeleton maps and robot skeleton maps are generated respectively based on the human motion dataset, including:
[0015] Based on the human body structure, the nodes of each joint are numbered to obtain the node set of the human skeleton diagram. Based on the connection relationship between the joints, the edge set of the human skeleton diagram is obtained. Thus, the human skeleton diagram is constructed. Based on the target robot structure, the nodes of each joint are numbered to obtain the node set of the robot skeleton diagram. Based on the joint structure and kinematic connection relationship of the target robot, the edge set of the robot skeleton diagram is obtained. Thus, the robot skeleton diagram is constructed.
[0016] Each node in the human skeleton diagram and robot skeleton diagram is assigned node feature information, which includes the position and orientation information of the joints; each edge in the human skeleton diagram and robot skeleton diagram is assigned edge feature information, which is obtained based on the feature difference between adjacent nodes.
[0017] Furthermore, the graph encoding module includes multiple channel-based topology-refining graph convolution modules, and the graph decoding module includes multiple graph convolution modules.
[0018] Furthermore, the loss function includes end effector position loss, end effector pose direction loss, elbow structure loss, and regularization loss. The end effector position loss is used to constrain the difference between the generated robot end effector position and the human end effector position. The end effector pose direction loss is used to constrain the consistency between the generated robot end effector pose and the human end effector pose. The elbow structure loss is used to constrain the relative direction between the robot elbow and wrist to be consistent with the corresponding structure of the human body. The regularization loss is used to limit the complexity of the model parameters. The weight coefficients of the end effector position loss, the end effector pose direction loss, and the elbow structure loss are adaptively adjusted based on the training state.
[0019] Furthermore, the loss function is specifically as follows:
[0020] ,
[0021] In the formula, Represents the loss function. This indicates the position loss of the end effector. The weighting coefficients represent the position loss of the end effector. This indicates the end-effector attitude direction loss. The weighting coefficients represent the end-effector orientation loss. This indicates elbow structural damage. The weighting coefficients representing the elbow structure loss are... This represents the regularization loss.
[0022] Furthermore, the method for calculating the weighting coefficients of the end-effector attitude direction loss is as follows:
[0023] ,
[0024] In the formula, The weighting coefficients represent the end-effector orientation loss. This represents the maximum weight value of the end-effector attitude direction loss. This indicates the relative decrease in the position loss of the end effector. The exponential adjustment parameter represents the end-effector orientation loss, and exp represents the natural exponential function.
[0025] Furthermore, the method for calculating the weighting coefficient of the elbow structure loss is as follows:
[0026] ,
[0027] In the formula, The weighting coefficients representing the elbow structure loss are... This represents the maximum weight value for elbow structure loss. This indicates the relative decrease in the position loss of the end effector. The exponential adjustment parameter represents the loss of elbow structure, and exp represents the natural exponential function.
[0028] Further, training the diffusion model using the robot joint angle representation data includes:
[0029] The trained graph coding module is used to extract latent feature representations of human motion, which are then used as external observation condition inputs for the diffusion model.
[0030] The robot joint angle representation data is normalized.
[0031] Random noise is introduced into the normalized robot joint angle representation data to construct noisy joint angle data and realize the forward diffusion process;
[0032] The noisy joint angle data is input into the diffusion model, and the noise information at the corresponding time step is predicted through a conditionally modulated convolutional sampling process.
[0033] The time step and latent layer are used as conditional inputs, which are then passed through a linear layer and used as embeddings to generate scale and bias and predict noise.
[0034] A training loss function is constructed based on the error between predicted noise and actual noise, and the parameters of the diffusion model are updated through backpropagation;
[0035] Repeat the above training process until the diffusion model meets the preset training convergence condition.
[0036] Furthermore, the diffusion model inference stage includes:
[0037] Input the target human motion skeleton map into the trained graph encoding module to obtain the latent feature representation of human motion;
[0038] Generate initial random noise that matches the input dimension of the diffusion model, and use it as the initial input to the diffusion model;
[0039] Construct the time step sequence for diffusion inference and execute the denoising inference process according to the preset time step order;
[0040] At each time step, the current time step information and the potential characteristics of the human motion are used as conditional inputs, and random noise or intermediate joint angle results are input into the diffusion model.
[0041] The noise information corresponding to the current time step is predicted by the diffusion model, and the robot joint angle parameters are gradually recovered based on the predicted noise. The corresponding end effector position and elbow position are calculated by the forward kinematics model.
[0042] Based on the end effector position and elbow position, calculate the pose error corresponding to the target human body movement, and construct a guidance loss based on the pose error;
[0043] The current joint angle parameters are updated using gradient guidance based on the aforementioned guidance loss, so as to constrain the generated robot motion to move closer to the target human motion.
[0044] During the diffusion inference process, the generated joint angle results are evaluated based on the guiding loss, and the robot joint angle parameters that meet the preset optimization criteria are output as the final robot executable actions.
[0045] This invention also provides a cross-domain human-machine motion redirection system based on a skeleton graph convolutional network, comprising:
[0046] The data acquisition module is used to acquire human motion datasets and perform preprocessing.
[0047] The skeleton map construction module is used to generate human skeleton maps and robot skeleton maps based on the human motion dataset, according to the human body structure and the target robot structure.
[0048] The graph encoding module and graph decoding module construction module are used to construct the graph encoding module and graph decoding module. Based on the human skeleton map and robot skeleton map, the graph encoding module extracts the latent feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation.
[0049] The graph encoding module and graph decoding module training module are used to set a loss function according to the constraints of the motion redirection task. The constraints are used to measure the difference between the generated robot motion and the human motion. The graph encoding module and graph decoding module are trained based on the loss function, and the trained graph encoding module and graph decoding module are used to obtain robot joint angle representation data.
[0050] A diffusion model construction module is used to construct a diffusion model based on a convolutional neural network and to train the diffusion model using the robot joint angle representation data;
[0051] The diffusion model inference module is used to introduce a gradient guidance mechanism during the diffusion model inference stage to generate and optimize the robot's joint angles and output the final executable actions of the robot.
[0052] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:
[0053] This invention accurately extracts human motion information through graph encoding and decoding modules, and achieves high-precision motion redirection between human bodies and humanoid robots of different configurations using a diffusion model. A gradient guidance mechanism is introduced during the inference phase of the diffusion model, ensuring that the generated robot motion maintains a high degree of consistency with human motion in overall movement style and key joint positions, thereby improving the stability and applicability of motion redirection. Simultaneously, the motion generation process reduces reliance on complex iterative optimization, improving motion generation efficiency. Attached Figure Description
[0054] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:
[0055] Figure 1 This is a flowchart of a method in a preferred embodiment of the present invention.
[0056] Figure 2 This is an architecture diagram of the graph encoding module and the graph decoding module in a preferred embodiment of the present invention.
[0057] Figure 3 This is a diagram illustrating the training process of the graph encoding module and the graph decoding module in a preferred embodiment of the present invention.
[0058] Figure 4 This is a diagram of the diffusion model in a preferred embodiment of the present invention.
[0059] Figure 5 This is a diagram illustrating the training process of the diffusion model in a preferred embodiment of the present invention.
[0060] Figure 6 This is a reasoning guidance diagram for the gradient guidance mechanism in the diffusion model in a preferred embodiment of the present invention.
[0061] Figure 7 This is a diagram illustrating the reasoning process of the diffusion model in a preferred embodiment of the present invention.
[0062] Figure 8 An example diagram showing the results of motion redirection using this invention. Detailed Implementation
[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.
[0064] Reference Figure 1 As shown, this invention discloses a cross-domain human-computer motion redirection method based on a skeleton graph convolutional network, comprising:
[0065] S1: Acquire and preprocess the human motion dataset. The human motion dataset can be a publicly available human motion dataset or obtained through motion capture equipment.
[0066] In this embodiment, a publicly available human sign language dataset is used as the source of human motion data; the original data is normalized and scaled to unify the numerical range of joint coordinates and ensure that the input data is adapted to the training requirements of the subsequent graph encoding module.
[0067] S2: Based on the human body structure and the target robot structure, generate a human skeleton map and a robot skeleton map respectively based on the human motion dataset. The human skeleton map and the robot skeleton map are used to describe the topological relationship between each joint.
[0068] S2-1: Based on the human body structure, number each joint point to obtain the node set of the human skeleton diagram. Based on the connection relationship of the human skeleton, establish the connection structure between the joint points to obtain the edge set of the human skeleton diagram, thus constructing the human skeleton diagram. Based on the target robot structure, number each joint point of the target robot to obtain the node set of the robot skeleton diagram. Based on the joint structure and kinematic connection relationship of the target robot, establish the connection structure between the joint points to obtain the edge set of the robot skeleton diagram, thus constructing the robot skeleton diagram.
[0069] S2-2: Assign node feature information to each node in the human skeleton diagram and robot skeleton diagram. The node feature information includes the position information and posture information of the joints. At the same time, assign edge feature information to each edge in the human skeleton diagram and robot skeleton diagram. The edge feature information is obtained based on the feature difference between adjacent nodes.
[0070] In this embodiment, the human sign language motion dataset mainly includes six joints of both arms: the shoulder joint, the elbow joint, and the distal interphalangeal joint. Each joint is numbered, and each joint is used as a node in the graph, while the physical connections between joints are used as edges, thus constructing the topological structure of the human skeletal graph. In the human skeletal graph, the position and pose information of each joint is used as node features, where pose information is represented by a three-dimensional rotation angle. The features of the edges in the graph are obtained by subtracting the position vector of the parent node from the position vector of the corresponding child node, used to characterize the relative motion relationships between adjacent joints to meet the computational requirements of the subsequent graph convolutional network.
[0071] For the target robot, the YuMi dual-arm robot was selected as the motion retargeting object. By parsing the robot's corresponding URDF file, the structural information of each joint was obtained, and a topological structure of the robot's skeleton graph was constructed using robot joints as nodes and joint connections as edges. In the robot skeleton graph, the features of the edges are represented by the position and pose information of the corresponding sub-joints. Simultaneously, the original length information between the end joint and the shoulder joint, and between the elbow joint and the shoulder joint, was further calculated and saved for subsequent normalization and scale alignment of the human and robot skeleton structures.
[0072] S3: Construct a graph encoding module and a graph decoding module. Based on the human skeleton map and the robot skeleton map, the graph encoding module extracts the potential feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation, thereby realizing the mapping from human motion to robot motion and ensuring the accuracy of human motion feature extraction by the graph encoding module.
[0073] like Figure 2 As shown, the graph encoding module includes multiple channel-wise topology refinement graph convolution (CTR-GC) modules, which are sequentially connected to form a feature extraction network. The graph decoding module includes multiple graph convolutions.
[0074] The graph encoding module performs channel-based topological refinement graph convolution on the input skeletal graph features, fuses the outputs of each CTR-GC, and enhances the features using a non-linear activation function. Subsequently, the fused features are residually connected to the original input features to improve the stability and continuity of the feature representation. This feature extraction process is repeated multiple times to gradually improve the high-level semantic representation of human motion features. After feature extraction, the graph encoding module compresses the feature dimension through a linear mapping layer to obtain a latent feature representation for motion redirection. This latent feature representation is then used as the input to the external observation conditions in the subsequent diffusion model.
[0075] S4: Set a loss function based on the constraints of the motion redirection task, wherein the constraints are used to measure the difference between the generated robot motion and the human motion.
[0076] The loss function includes end effector position loss, end effector pose direction loss, elbow structure loss, and regularization loss. The end effector position loss is used to constrain the difference between the generated robot end effector position and the human end effector position. The end effector pose direction loss is used to constrain the consistency between the generated robot end effector pose and the human end effector pose. The elbow structure loss is used to constrain the relative direction between the robot elbow and wrist to be consistent with the corresponding structure of the human body. The regularization loss is used to limit the complexity of model parameters and improve the stability of the training process. The weight coefficients of the end effector position loss, the end effector pose direction loss, and the elbow structure loss are adaptively adjusted based on the training state.
[0077] In this embodiment, the loss function is specifically:
[0078] ,
[0079] In the formula, Represents the loss function. This indicates the position loss of the end effector. The weighting coefficient representing the position loss of the end effector is adjusted according to the actual situation. This indicates the end-effector attitude direction loss. The weighting coefficients represent the end-effector orientation loss. This indicates elbow structural damage. The weighting coefficients representing the elbow structure loss are... This represents the regularization loss.
[0080] The calculation methods for end effector position loss, end effector attitude direction loss, and elbow structure loss are as follows:
[0081] ,
[0082] ,
[0083] ;
[0084] In the formula, This indicates the normalized position of the human extremities. This represents the normalized position of the robot's end effector. Indicates the robot's end-effector posture. Indicates the end-body posture. This represents the normalized vector from the robot's elbow to its wrist. This represents the normalized vector from the elbow to the wrist of the human body. This represents the mean square error.
[0085] The method for calculating the weighting coefficients of the end-effector attitude direction loss is as follows:
[0086] ,
[0087] In the formula, The weighting coefficients represent the end-effector orientation loss. This represents the maximum weight value of the end-effector orientation loss. The value is adjusted based on actual conditions and is used to limit the maximum contribution of the end-effector orientation loss in the later stages of training. This represents the relative decrease in the end-effector position loss and is used to characterize the convergence of the model on the end-effector position alignment task. The exponential adjustment parameter representing the end-effector attitude direction loss is used for control. The rate at which position loss decreases, and its value, should be adjusted according to actual conditions; a larger value is preferable. The value corresponds to a smoother weight growth process, and exp represents the natural exponential function.
[0088] In this embodiment, the method for calculating the relative decrease ratio of the end effector position loss is as follows:
[0089] ,
[0090] In the formula, This indicates the relative decrease in the current end effector position loss. This indicates the current position loss of the end effector. This represents the end-position loss value in the initial stage of training (i.e., before the first iteration of training). This is used to normalize the relative decrease in loss at the current location, thereby eliminating the impact of differences in loss values under different data scales and task scenarios.
[0091] The method for calculating the weighting coefficient of the elbow structure loss is as follows:
[0092]
[0093] In the formula, The weighting coefficients representing the elbow structure loss are... This represents the maximum weight value for elbow structure loss, used to limit the maximum contribution of elbow structure loss in the later stages of training. This indicates the relative decrease in the position loss of the end effector. An exponentially adjusted parameter representing elbow structural loss, used for control. The rate at which position loss decreases, and its value, should be adjusted according to actual conditions; a larger value is preferable. The value corresponds to a smoother weighting process.
[0094] In the initial training phase, the focus is on optimizing the end-effector position loss to ensure that the robot's end-effector trajectory is basically aligned with the human end-effector motion in space. As the end-effector position loss gradually decreases and reaches a preset convergence level, the weights of the end-effector pose direction loss and elbow structure loss are gradually increased through an exponential adjustment function. This introduces higher-order pose and structural constraints without disrupting the learned position mapping, achieving a phased optimization from coarse to fine, thereby improving the stability of the training process and the accuracy of motion retargeting results.
[0095] The training process of the graph encoding module and the graph decoding module is as follows: Figure 3 As shown in the figure. In this embodiment, the training process consists of 50 iterations, with the initial learning rate set to... And gradually decrease during training. The model output consists of joint angle parameters for each joint of the robot. These joint angle parameters are calculated using a forward kinematics model to obtain the pose information of the robot's end effector and elbow joints. Based on this pose information, a loss function is constructed. Regularization loss is introduced to constrain the range of potential feature values, thereby improving the stability of training.
[0096] S5: Train the graph encoding module and graph decoding module based on the loss function until the loss function converges, and use the trained graph encoding module and graph decoding module to obtain robot joint angle representation data; the trained graph encoding module and graph decoding module will be used as trained modules in subsequent diffusion models.
[0097] S6: Construct a diffusion model based on a convolutional neural network, and train the diffusion model using the robot joint angle representation data.
[0098] In this embodiment, as Figure 4 As shown, the diffusion model includes a downsampling module, an intermediate module, and an upsampling module. The diffusion model is trained using the robot joint angle representation data, and the training process is as follows: Figure 5 As shown, it includes:
[0099] S6-1: Use the trained graph coding module to extract latent feature representations of human motion as input to the external observation conditions of the diffusion model.
[0100] S6-2: Normalize the robot joint angle representation data to meet the input requirements of the diffusion model.
[0101] S6-3: Introduce random noise into the normalized robot joint angle representation data to construct noisy joint angle data and realize the forward diffusion process.
[0102] The noisy joint angle data are as follows:
[0103] ,
[0104] In the formula, Indicates the first Joint angle data after adding noise at each time step Indicates the relationship with the first The cumulative noise attenuation coefficient corresponding to each time step This represents the raw robot joint angle representation data without added noise. This represents random noise that follows a standard normal distribution.
[0105] S6-4: Input the noisy joint angle data into the diffusion model, and predict the noise information at the corresponding time step through the conditionally modulated convolutional sampling process.
[0106] S6-5: The time step and the latent layer are used as conditional inputs, which are then passed through a linear layer and used as embeddings to generate the scale and bias and predict the noise.
[0107] S6-6: Construct a training loss function based on the error between predicted noise and real noise, and update the diffusion model parameters through backpropagation.
[0108] S6-7: Repeat the above training process until the diffusion model meets the preset training convergence condition.
[0109] In S6, the parameters of the pre-trained graph encoding and decoding modules are fixed. Each set of human motion features corresponds to a set of robot joint angle movements. The noise addition time step in the diffusion process is set to 100. Random noise is gradually added to the joint angle data. The diffusion model takes the noise-added joint angles as input and uses the time step and latent human motion features as conditions to predict the added noise value.
[0110] S7: In the diffusion model inference stage, introduce, for example... Figure 6 The gradient guidance mechanism shown generates and optimizes the robot's joint angles, outputting the final executable actions of the robot.
[0111] During the inference phase, the graph encoding module is first used to extract features from the target human motion skeleton map to obtain the corresponding latent feature representation, and pure noise of the same dimension as that generated in the training phase is used as the initial input. Setting the denoising time step to 100, the diffusion model progressively denoises the current joint angle data under the constraints of the time step conditions and latent feature conditions. Each iteration removes a portion of the noise and generates the joint angle estimate for the next time step. After each denoising iteration, based on the end-effector position and elbow position calculated by the forward kinematics model for the current joint angle, the end-effector position loss and elbow position loss are calculated respectively, and the gradient of the loss function is obtained. This gradient is applied to the current joint angle to guide its optimization towards the target direction. Throughout the entire denoising loop, the joint angle result with the minimum end-effector loss and elbow loss is retained as the final robot joint angle output. Figure 7 As shown, the reasoning process of the diffusion model is as follows:
[0112] S7-1: Input the target human motion skeleton map into the graph encoding module trained in S5 to obtain the latent feature representation of human motion.
[0113] S7-2: Generate initial random noise that matches the input dimension of the diffusion model, and use it as the initial input to the diffusion model.
[0114] S7-3: Construct the time step sequence of diffusion inference and execute the denoising inference process according to the preset time step order.
[0115] S7-4: At each time step, the current time step information and the potential characteristics of the human motion are used as conditional inputs, and random noise or intermediate joint angle results are input into the diffusion model.
[0116] S7-5: The noise information corresponding to the current time step is predicted by the diffusion model, and the robot joint angle parameters are gradually recovered based on the predicted noise. The corresponding end effector position and elbow position are calculated by the forward kinematics model.
[0117] S7-6: Calculate the pose error corresponding to the target human body movement based on the position of the end effector and the elbow position, and construct the guidance loss based on the pose error.
[0118] S7-7: Based on the guidance loss, perform gradient guidance updates on the current joint angle parameters to constrain the generated robot motion to move closer to the target human motion.
[0119] S7-8: During the diffusion inference process, the generated joint angle results are evaluated based on the guiding loss, and the robot joint angle parameters that meet the preset optimization criteria are output as the final robot executable actions.
[0120] This invention also discloses a cross-domain human-machine motion redirection system based on a skeleton graph convolutional network, comprising:
[0121] The data acquisition module is used to acquire human motion datasets and perform preprocessing.
[0122] The skeleton map construction module is used to generate human skeleton maps and robot skeleton maps based on the human motion dataset, according to the human body structure and the target robot structure.
[0123] The graph encoding module and graph decoding module construction module are used to construct the graph encoding module and graph decoding module. Based on the human skeleton map and robot skeleton map, the graph encoding module extracts the latent feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation.
[0124] The graph encoding module and graph decoding module training module are used to set a loss function according to the constraints of the motion redirection task. The constraints are used to measure the difference between the generated robot motion and the human motion. The graph encoding module and graph decoding module are trained based on the loss function, and the trained graph encoding module and graph decoding module are used to obtain robot joint angle representation data.
[0125] A diffusion model construction module is used to construct a diffusion model based on a convolutional neural network and to train the diffusion model using the robot joint angle representation data;
[0126] The diffusion model inference module is used to introduce a gradient guidance mechanism during the diffusion model inference stage to generate and optimize the robot's joint angles and output the final executable actions of the robot.
[0127] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a cross-domain human-machine motion redirection method based on a skeleton graph convolutional network.
[0128] The present invention also discloses an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a cross-domain human-machine motion redirection method based on a skeleton graph convolutional network.
[0129] Compared with the prior art, the advantages of the present invention are:
[0130] 1. Accurately extract human motion information through graph encoding and graph decoding modules. A channel-based topology refinement graph convolution module and graph convolution are introduced. By adapting the channel-based topology refinement graph convolution module, it can simultaneously express joint topology and cross-configuration motion features during skeleton graph encoding, thereby improving the completeness and accuracy of human motion information modeling. Graph convolution can perform in-depth representation of human motion features, improving the ability to capture human motion information.
[0131] 2. The diffusion model generates robot joint movements based on the extracted features, making the robot's movements more natural and highly consistent with human movement styles, so as to achieve high-precision motion redirection between human bodies and humanoid robots with different configurations.
[0132] 3. Introduce a gradient guidance mechanism in the inference stage of the diffusion model to ensure that the generated robot motion is consistent with human motion in terms of overall motion style and key joint positions, thereby improving the accuracy and stability of motion retargeting.
[0133] 4. The motion generation process reduces the reliance on complex iterative optimization and improves the efficiency of motion generation.
[0134] To further demonstrate the beneficial effects of the present invention, this embodiment utilizes the method of the present invention and DMPMR (see “Y. Liang, W. Li, Y. Wang, and R. Xiong, Y. Mao, and J. Zhang, “Dynamicmovement primitive based motion retargeting for dual-arm sign languagemotions,” in Proc. IEEE Int. Conf. Robot. Automat., 2021, pp. 8195–8201”), NMG (see “S. Choi and J. Kim, “Towards a natural motion generator: A pipeline to control a humanoid based on motion data,” in Proc. IEEE / RSJ Int. Conf. Intell. Robots Syst., 2019, pp. 4373–4380”), and C-3PO (see “T. Kim and J.-H. Lee, “C-3PO: Cyclic-three-phase optimization for human-robot motion retargeting based on reinforcement learning,” in Proc. IEEE Int. Conf. Robot. Automat., 2020, pp. 8425–8432"), NN (for details, see "S. Choi, M. Pan, and J. Kim, "Nonparametricmotion retargeting for humanoid robots on shared latent space," in Proc. Robot.: Sci. Syst., Jul. 2020"), NLO (for details, see "H. Zhang, W. Li, J. Liu, Z. Chen, Y. Cui, Y. Wang, and R. Xiong, “Kinematic motion retargeting via neural latent optimization for learning sign language,” IEEE Robotics and Automation Letters, vol.The motion redirection experiment was conducted using the method described in "7, no. 2, pp. 4582–4589, 2022". The data used in the experiment came from a publicly available sign language dataset, and 25 movement sequences were tested. The average elbow and distal end positions were calculated based on the experimental results, and the similarity of the generated trajectories was measured using the average Fréchet distance to compare the motion redirection effects. A smaller average Fréchet distance indicates a more similar trajectory. The motion redirection results of the method of this invention are shown below. Figure 8 As shown in Table 1, the average Fréchet distance results for different methods are presented.
[0135] Table 1. Comparison of average Fréchet distances using different methods
[0136]
[0137] As can be seen from Table 1, the present invention can improve the accuracy of motion redirection.
[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0142] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A cross-domain human-computer motion redirection method based on skeleton graph convolutional networks, characterized in that, include: The human motion dataset is acquired and preprocessed. Based on the human body structure and the target robot structure, human skeleton diagrams and robot skeleton diagrams are generated respectively from the human motion dataset. A graph encoding module and a graph decoding module are constructed. Based on the human skeleton map and the robot skeleton map, the graph encoding module extracts the latent feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation. A loss function is set according to the constraints of the motion retargeting task, wherein the constraints are used to measure the difference between the generated robot motion and the human motion. The graph encoding module and graph decoding module are trained based on the loss function, and the trained graph encoding module and graph decoding module are used to obtain robot joint angle representation data; A diffusion model is constructed based on a convolutional neural network, and the diffusion model is trained using the robot joint angle representation data; In the diffusion model inference stage, a gradient guidance mechanism is introduced to generate and optimize the robot's joint angles, and output the final robot-executable actions. The diffusion model inference phase includes: Input the target human motion skeleton map into the trained graph encoding module to obtain the latent feature representation of human motion; Generate initial random noise that matches the input dimension of the diffusion model, and use it as the initial input to the diffusion model; Construct the time step sequence for diffusion inference and execute the denoising inference process according to the preset time step order; At each time step, the current time step information and the potential characteristics of the human motion are used as conditional inputs, and random noise or intermediate joint angle results are input into the diffusion model. The noise information corresponding to the current time step is predicted by the diffusion model, and the robot joint angle parameters are gradually recovered based on the predicted noise. The corresponding end effector position and elbow position are calculated by the forward kinematics model. Based on the end effector position and elbow position, calculate the pose error corresponding to the target human body movement, and construct a guidance loss based on the pose error; The current joint angle parameters are updated using gradient guidance based on the aforementioned guidance loss, so as to constrain the generated robot motion to move closer to the target human motion. During the diffusion inference process, the generated joint angle results are evaluated based on the guiding loss, and the robot joint angle parameters that meet the preset optimization criteria are output as the final robot executable actions.
2. The skeleton graph convolution network based cross-domain human motion retargeting method according to claim 1, characterized in that, The step of generating human skeleton maps and robot skeleton maps based on the human motion dataset, according to the human body structure and the target robot structure, includes: Based on the human body structure, the nodes of each joint are numbered to obtain the node set of the human skeleton diagram. Based on the connection relationship between the joints, the edge set of the human skeleton diagram is obtained. Thus, the human skeleton diagram is constructed. Based on the target robot structure, the nodes of each joint are numbered to obtain the node set of the robot skeleton diagram. Based on the joint structure and kinematic connection relationship of the target robot, the edge set of the robot skeleton diagram is obtained. Thus, the robot skeleton diagram is constructed. Each node in the human skeleton diagram and robot skeleton diagram is assigned node feature information, which includes the position and orientation information of the joints; each edge in the human skeleton diagram and robot skeleton diagram is assigned edge feature information, which is obtained based on the feature difference between adjacent nodes.
3. The skeleton graph convolutional network based cross-domain human motion retargeting method of claim 1, wherein, The graph encoding module includes multiple channel-based topology-refining graph convolution modules, and the graph decoding module includes multiple graph convolution modules.
4. The cross-domain human-computer motion redirection method based on skeleton graph convolutional networks according to claim 1, characterized in that, The loss function includes end effector position loss, end effector pose direction loss, elbow structure loss, and regularization loss. The end effector position loss is used to constrain the difference between the generated robot end effector position and the human end effector position. The end effector pose direction loss is used to constrain the consistency between the generated robot end effector pose and the human end effector pose. The elbow structure loss is used to constrain the relative direction between the robot elbow and wrist to be consistent with the corresponding structure of the human body. The regularization loss is used to limit the complexity of the model parameters. The weight coefficients of the end effector position loss, the end effector pose direction loss, and the elbow structure loss are adaptively adjusted based on the training state.
5. The skeleton graph convolution network based cross-domain human motion retargeting method according to claim 4, characterized in that, The loss function is specifically as follows: ， In the formula, Represents the loss function. This indicates the position loss of the end effector. The weighting coefficients represent the position loss of the end effector. This indicates the end-effector attitude direction loss. The weighting coefficients represent the end-effector orientation loss. This indicates elbow structural damage. The weighting coefficients representing elbow structure loss This represents the regularization loss.
6. The cross-domain human-computer motion redirection method based on skeleton graph convolutional networks according to claim 5, characterized in that, The method for calculating the weighting coefficients of the end-effector attitude direction loss is as follows: ， In the formula, The weighting coefficients represent the end-effector orientation loss. This represents the maximum weight value of the end-effector attitude direction loss. This indicates the relative decrease in the position loss of the end effector. The exponential adjustment parameter represents the end-effector orientation loss, and exp represents the natural exponential function.
7. The skeleton graph convolution network based cross-domain human motion retargeting method according to claim 5, characterized in that, The method for calculating the weighting coefficient of the elbow structure loss is as follows: ， In the formula, The weighting coefficients representing the elbow structure loss are... This represents the maximum weight value for elbow structure loss. This indicates the relative decrease in the position loss of the end effector. The exponential adjustment parameter represents the loss of elbow structure, and exp represents the natural exponential function.
8. The skeleton graph convolution network based cross-domain human motion retargeting method according to any one of claims 1-7, characterized in that, Training the diffusion model using the robot joint angle representation data includes: The trained graph coding module is used to extract latent feature representations of human motion, which are then used as external observation condition inputs for the diffusion model. The robot joint angle representation data is normalized, and random noise is introduced into the normalized robot joint angle representation data to construct noisy joint angle data, thereby realizing the forward diffusion process. The noisy joint angle data is input into the diffusion model, and the noise information at the corresponding time step is predicted through a conditionally modulated convolutional sampling process. The time step and latent layer are used as conditional inputs, which are then passed through a linear layer and used as embeddings to generate scale and bias and predict noise. A training loss function is constructed based on the error between predicted noise and actual noise, and the parameters of the diffusion model are updated through backpropagation; Repeat the above training process until the diffusion model meets the preset training convergence condition. 9.A skeleton graph convolution network based cross-domain human motion re-targeting system, characterized in that, include: The data acquisition module is used to acquire human motion datasets and perform preprocessing. The skeleton map construction module is used to generate human skeleton maps and robot skeleton maps based on the human motion dataset, according to the human body structure and the target robot structure. The graph encoding module and graph decoding module construction module are used to construct the graph encoding module and graph decoding module. Based on the human skeleton map and robot skeleton map, the graph encoding module extracts the latent feature representation of human motion, and the graph decoding module generates the corresponding robot joint angle representation. The graph encoding module and graph decoding module training module are used to set a loss function according to the constraints of the motion redirection task. The constraints are used to measure the difference between the generated robot motion and the human motion. The graph encoding module and graph decoding module are trained based on the loss function, and the trained graph encoding module and graph decoding module are used to obtain robot joint angle representation data. A diffusion model construction module is used to construct a diffusion model based on a convolutional neural network and to train the diffusion model using the robot joint angle representation data; The diffusion model inference module is used to introduce a gradient guidance mechanism during the diffusion model inference stage to generate and optimize the robot's joint angles and output the final robot-executable actions. The diffusion model inference phase includes: Input the target human motion skeleton map into the trained graph encoding module to obtain the latent feature representation of human motion; Generate initial random noise that matches the input dimension of the diffusion model, and use it as the initial input to the diffusion model; Construct the time step sequence for diffusion inference and execute the denoising inference process according to the preset time step order; At each time step, the current time step information and the potential characteristics of the human motion are used as conditional inputs, and random noise or intermediate joint angle results are input into the diffusion model. The noise information corresponding to the current time step is predicted by the diffusion model, and the robot joint angle parameters are gradually recovered based on the predicted noise. The corresponding end effector position and elbow position are calculated by the forward kinematics model. Based on the end effector position and elbow position, calculate the pose error corresponding to the target human body movement, and construct a guidance loss based on the pose error; The current joint angle parameters are updated using gradient guidance based on the aforementioned guidance loss, so as to constrain the generated robot motion to move closer to the target human motion. During the diffusion inference process, the generated joint angle results are evaluated based on the guiding loss, and the robot joint angle parameters that meet the preset optimization criteria are output as the final robot executable actions.