Robot redirection method based on graph neural network
Through the graph neural network combined with the latent neural network, the structural relationship between the human body and the robot is automatically learned, solving the problems of complex computing and poor interpretation in the existing technology, and achieving efficient and accurate robot motion redirection mapping.
Patent Information
- Application Number
- CN202411084199.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-08
AI Technical Summary
In the prior art, in visual remote operation, traditional optimization algorithms have complex calculations and poor interpretability, deep learning methods have difficulty in debugging black box operations, reinforcement learning methods have high calculation costs and lack interpretability, making it difficult to achieve high-precision robot motion redirection mapping.
Using a graph neural network-based method, the graph neural network encoder and potential neural network are combined to automatically learn the structure mapping relationship between human body and robot, capture the different characteristics of motion, and use the latent space to optimize joint angle and link position information to achieve end-to-end visual-motion mapping at the end.
The mapping process is simplified, the computational complexity is reduced, the mapping accuracy and generalization ability is improved, manual design and parameter debugging are avoided, and efficient robot motion redirection is achieved.
Smart Images

Figure CN119068148A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot computing, and in particular to a robot redirection method based on graph neural network. Background Art
[0002] With the rapid development of robotics, vision-guided robot teleoperation technology has received widespread attention. This technology captures the operator's movements through various equipment, and then maps them to the robot platform in real time to achieve remote control. It has important applications in special environment operations, surgical assistance and other fields. The key to visual teleoperation is to achieve high-precision motion redirection mapping.
[0003] Traditional methods are mainly based on optimization algorithms, such as inverse kinematics and motion planning, which require manual design of objective functions and constraints, are computationally complex and easily fall into local optimality. In recent years, data-driven methods based on deep learning have emerged, attempting to directly learn mapping models end-to-end from visual data, overcoming the shortcomings of traditional methods. However, due to black-box operations, they have poor interpretability and controllability, are difficult to debug and optimize, and have poor adaptability to new postures that are not covered.
[0004] Secondly, existing technologies, such as reinforcement learning-based methods, model motion redirection as a Markov decision process and solve the state-to-action mapping strategy through deep reinforcement algorithms. The advantage of such methods is strong generalization, but the disadvantage is that they need to build an accurate environment model, have high computational cost and lack interpretability, which are subject to certain limitations in practical applications. At the same time, they also have the problem of insufficient interpretability. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a robot redirection method based on graph neural network.
[0006] The technical solution of the present invention is: a robot redirection method based on graph neural network, comprising the following steps:
[0007] S1), collecting the operator's action video; and extracting the human skeleton structure from the action video;
[0008] S2), modeling the human skeleton structure and the robot structure in the form of a graph into a tree topology graph;
[0009] S3), using the corresponding graph neural network encoder to respectively optimize the encoding of the tree topology graphs of the human body and the robot, and obtain the latent space representation of the tree topology graphs of the human body and the robot;
[0010] S4), inputting the optimized latent space representation of the human-machine skeleton graph obtained in step S3) into the latent neural network, capturing the latent features of the human-machine motion difference through the latent neural network, thereby obtaining the joint angle and connecting rod position information of the robot;
[0011] S5) Through the differentiable forward kinematics layer, the joint angle and link pose information output by the graph decoder are converted into the desired position of the robot end effector; that is, the final motion redirection mapping is completed.
[0012] Preferably, in step S1), after acquiring the action video, denoising, color correction and frame rate adjustment are performed on it.
[0013] Preferably, in step S1), the visual detection algorithm OpenPose and the estimation algorithm Frankmocap extract the three-dimensional joint point coordinate data and posture quaternion information of the human body from the action video as the human skeletal structure.
[0014] Preferably, in step S2), the joints of the robot are taken as nodes, and the connecting rods connecting two joints are taken as undirected edges; therefore, the structure of the robot is modeled as an acyclic tree topology graph Gr=(Vr, Er); wherein Vr is the set of robot joint points; and Er is the set of robot connecting rod edges.
[0015] Preferably, in step S2), the human skeleton structure is modeled as an acyclic tree topology graph Gh=(Vh, Eh); wherein Vh is a set of human joint points; and Er is a set of human skeleton connection edges.
[0016] Preferably, in step S3), the graph neural network encoder adopts a layered convolution structure, and the graph neural network encoder includes multiple stages, each stage includes multiple convolutional layers of the same type, and each stage processes features of different scales.
[0017] Preferably, in step S3), the graph neural network encoders are respectively a hand encoder and a body encoder; the hand encoder and the body encoder are respectively used to optimize the encoding of the tree topology graphs of the human body and the robot, and the convolution operation of the hand encoder is used to capture the subtle changes in the hand movements of the human body and the robot; the body encoder uses the backbone path encoding operation to learn the macroscopic features of the overall body movement from the human body and robot full-body skeleton graphs.
[0018] Preferably, in step S3), the tree topology graphs of the human body and the robot are encoded and optimized using the graph neural network encoder, which specifically includes the following steps:
[0019] S31), for the human body tree topology, at each node, the input feature is the rotation angle of the corresponding joint, and at each edge, the input feature is the initial offset and rotation between adjacent edges;
[0020] For the tree topology of the robot, at each node, the output feature will represent the rotation angle of the corresponding joint, and at each edge, the output feature will represent the initial offset and rotation between the links;
[0021] S32), using a graph neural network encoder to encode the tree topology graphs of the human body and the robot respectively to generate a latent space representation;
[0022] S33), transmitting and updating node features and edge features in the graph neural network encoder, and repeatedly performing multi-level graph convolution operations;
[0023] S34) Optimize the initial features of the input through multi-level feature extraction and updating of the graph neural network encoder; and output a concise and efficient latent space representation.
[0024] Preferably, in step S4), the latent neural network includes a graph encoder and a graph decoder; wherein the graph encoder performs node and edge update operations on the optimized encoding of the input human-machine skeleton graph through a graph convolutional network GCN, wherein the graph convolutional network GCN is stacked by multiple layers of graph convolutional layers, each layer of graph convolutional layer includes two operations of node update and edge update, the node update fuses the current node feature and the aggregated feature from the adjacent nodes, and the edge update fuses the current edge feature and the aggregated feature from the adjacent edges; the graph convolutional network GCN is used to extract high-level node features and edge features, and capture higher-level motion patterns and differences layer by layer;
[0025] The graph decoder decodes the potential features through a fully connected layer or a transposed convolutional layer, and outputs the robot joint angle and connecting rod posture information.
[0026] Preferably, in step S4), the joint angle and connecting rod posture information of the robot are extracted using a potential neural network, which specifically includes the following steps:
[0027] S41), mapping the optimized encodings of the skeleton graphs of the human body and the robot to the latent neural network respectively, and then performing continuous constrained optimization in the latent space through the graph encoder to obtain the latent features that can minimize the representation of the difference between human and machine;
[0028] S42), input the optimized aligned latent code into the graph decoder to decode the corresponding joint angle and connecting rod posture information, and in the decoding process, the robot's prior structure information is used to ensure that the decoded output satisfies the robot's kinematic constraints.
[0029] Preferably, in step S4), the graph encoder updates the expression for fusing node features and edge features as follows:
[0030] x_v t+1 =Φ(x_v t ,∑_{u∈N(v)}ψ(x_v t ,x_u t ,e_attr(v,u)));
[0031] In the formula, x_v t+1 represents the updated feature vector of node v at time t+1; x_v t represents the feature vector of node v at time t; N(v) is the set of neighbor nodes of node v, u represents a neighbor node of v, x_u t represents the feature vector of node u at time t; e_attr(v,u) is the feature vector of the edge connecting v and u, ψ is the message function that calculates the information vector based on the node and edge information; ∑_{u∈N(v)} is used to summarize the influence of all neighboring nodes; Φ is the update function that integrates its own information and the influence of neighboring nodes to update node features.
[0032] Preferably, in step S4), the objective function of the graph decoder training is:
[0033]
[0034] Where z is the potential code; is the graph decoder parameter; Represents the reconstruction loss function, which is used to measure the output of the graph decoder The difference between Ω(z) and the input X; Ω(z) is the regularization term applied to the latent code z; λ is the weight coefficient of the regularization term.
[0035] Preferably, in step S5), the joint angle and connecting rod posture information output by the graph decoder are converted into the desired position of the robot end effector using a differentiable positive kinematics layer, which specifically includes the following steps:
[0036] S51), passing the robot joint angle and connecting rod posture information output by the graph encoder as input to the differentiable positive kinematics layer;
[0037] S52), the forward kinematics layer calculates the transformation matrix of each joint according to the robot's kinematic model and DOF settings;
[0038] S53), calculating the overall motion transformation from the base coordinate system to the end effector through the transformation matrix of each joint, and obtaining the desired position of the end effector;
[0039] S54), comparing the expected position of the end effector with the actual target position to obtain a position error;
[0040] S55), based on the position error, back-propagating to the graph decoder and the graph encoder to optimize the potential representation of the graph encoder and the output of the graph decoder;
[0041] S56) Through multiple iterations of optimization, the position error of the end effector is minimized to obtain accurate motion redirection mapping; thereby obtaining a robot joint angle sequence that can accurately reach the desired position, completing the motion redirection mapping task.
[0042] Preferably, in step S5), the goal of the motion redirection is to determine a set of robot joint angles r* so that the robot's motion posture R (r*) can imitate and fit the given human posture H to the greatest extent under given constraints; at the same time, it cannot exceed the robot's own joint angle limit range, and by optimizing the objective function and satisfying the constraints, obtain the motion trajectory that allows the robot to best imitate human movements, that is:
[0043] r * =argminL(H,R(r));
[0044] str lower ≤r≤r upper ;
[0045] Where r* is the robot joint angle; L represents the target mapping function from the joint angle to the robot's final posture; H is the human body posture information; R(r) represents the robot's movement, which is determined by the joint angle r; r lower 、r upper They are the lower and upper limits of the robot's motion, respectively, reflecting the constraints of the robot's motion range.
[0046] Preferably, in step S5), the robot's joint angles and connecting rod posture information output by the graph decoder are mapped to a robot end effector motion trajectory that can best imitate human body movements by optimizing the target mapping function L; the target mapping function L of the motion redirection is expressed as:
[0047] L=λ ees L ees +λ ee L ee +λ ori L ori +λ fin L fin ;
[0048] Where, L ees is the speed loss function; L ee is the end effector loss function; Lori is the direction loss function; L fin is the finger loss function; ees , ee , ori , fin The weights of the corresponding loss functions respectively.
[0049] Preferably, in step S5), the speed loss function L ees The expression is:
[0050]
[0051] In the formula, s i and r i is the velocity of the robot in the i-th frame and is the normalization coefficient; and is the velocity and normalization coefficient of the human body in the i-th frame.
[0052] Preferably, in step S5), the direction loss function L ori The expression is:
[0053]
[0054] Where: D i is the rotation matrix of the robot’s end effector in the i-th frame; is the rotation matrix of the end effector of the human body in the i-th frame.
[0055] Preferably, in step S5), the finger loss function L fin The expression is:
[0056]
[0057] Where: f i tip is the fingertip position of the robot in the i-th frame; f i meta is the fingertip position of the robot in the i-th frame; r tm is the normalized coefficient of robot arm i; is the fingertip position of the human body in the i-th frame; is the fingertip position of the human body in the i-th frame; is the normalized coefficient of human arm i.
[0058] Preferably, in step S5), the end effector loss function L ee The expression is:
[0059]
[0060] Where: ei and is r i The position and normalization coefficient of the robot's end effector at the i-th frame, and are the end effectors and normalized coefficients of the person in the i-th frame.
[0061] The beneficial effects of the present invention are:
[0062] 1. The present invention combines the graph neural network encoder with the latent neural network, thereby fully utilizing the powerful representation capability of the latent space while retaining the robot structure prior; automatically learning the structural mapping relationship between the human body and the robot by using the graph neural network encoder; the graph neural network encoder can efficiently capture the topological features in the graph structure data; adaptively extract the association pattern between humans and machines; greatly simplify the mapping process, reduce labor costs, and improve mapping accuracy and generalization ability;
[0063] 2. The present invention uses a latent coding optimization strategy to map the original joint space into a compact latent space, transforming the complex motion mapping problem into an optimization problem in the latent space; the gradient descent algorithm is used to search for the optimal human-machine motion mapping in the latent space, which greatly reduces the computational complexity of the mapping process and improves the real-time performance;
[0064] 3. The present invention uses an RGB camera to capture human joint information, and performs graph modeling on the human skeleton and robot structure; uses a graph encoder to learn the structural mapping relationship between humans and machines; and avoids complex manual design and parameter debugging;
[0065] 4. The present invention comprehensively considers the spatiotemporal characteristics of human-machine motion through a composite optimization objective function including speed loss, acceleration loss, end effector loss, direction loss, and finger loss, making the mapping result more accurate and natural;
[0066] 5. The present invention uses a graph decoder to recover the robot joint angle information from the optimized latent code, and converts it into the robot end position through a differentiable positive kinematics layer to achieve end-to-end visual-motion mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is a schematic diagram of the process of the present invention;
[0068] Figure 2 It is a framework diagram of the method of the present invention;
[0069] Figure 3 It is a schematic diagram of the framework of the image encoder of the present invention;
[0070] Figure 4 Schematic diagram of the framework of the decoder of the present invention. DETAILED DESCRIPTION
[0071] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:
[0072] like Figure 1 and 2 As shown, this embodiment provides a robot redirection method based on a graph neural network, comprising the following steps:
[0073] S1), using an RGB camera to collect the operator's action video; and using a visual detection algorithm and an estimation algorithm to extract the human body's three-dimensional joint point coordinate data and posture quaternion information from the collected operator image or video as the human body skeleton structure.
[0074] In this embodiment, the visual detection algorithm and the estimation algorithm respectively adopt OpenPose and Frankmocap algorithms.
[0075] Furthermore, after acquiring the action video of the operator, this embodiment needs to perform denoising, color correction and frame rate adjustment processing on it to improve the accuracy of subsequent posture extraction.
[0076] S2) Modeling is performed in the form of a graph according to the human skeleton structure and the robot structure into a tree topology graph.
[0077] In this embodiment, the robot structure is constructed as a tree topological graph, specifically: the joints of the robot are taken as nodes, and the connecting rods connecting two joints are taken as undirected edges; therefore, the structure of the robot is modeled as an acyclic tree topological graph Gr = (Vr, Er); wherein Vr is the set of robot joint points; and Er is the set of robot connecting rod edges.
[0078] Correspondingly, the human skeleton structure is modeled as an acyclic tree topology graph Gh=(Vh, Eh); wherein Vh is a set of human joint points; and Er is a set of human skeleton connection edges.
[0079] S3), using the corresponding graph neural network encoder to respectively optimize the encoding of the tree topology graphs of the human body and the robot, and obtain the latent space representation of the tree topology graphs of the human body and the robot.
[0080] In this embodiment, the graph neural network encoder adopts a layered convolution structure. The graph neural network encoder includes multiple stages, each stage includes multiple convolutional layers of the same type, and each stage processes features of different scales.
[0081] In addition, the graph neural network encoders are respectively a hand encoder and a body encoder; the hand encoder and the body encoder are respectively used to optimize the encoding of the tree topology graphs of the human body and the robot, and the convolution operation of the hand encoder is used to capture the subtle changes in the hand movements of the human body and the robot; the body encoder uses the backbone path encoding operation to learn the macroscopic features of the overall body movements from the full-body skeleton graphs of the human body and the robot.
[0082] The tree topology graphs of the human body and the robot are encoded and optimized using the graph neural network encoder, which specifically includes the following steps:
[0083] S31), for the tree topology of the human body, at each node, the input feature is the rotation angle of the corresponding joint, and at each edge, the input feature is the initial offset and rotation between adjacent edges;
[0084] For the tree topology of the robot, at each node, the output feature will represent the rotation angle of the corresponding joint, and at each edge, the output feature will represent the initial offset and rotation between the links;
[0085] S32), using a graph neural network encoder to encode the tree topology graphs of the human body and the robot respectively to generate a latent space representation;
[0086] S33), transmitting and updating node features and edge features in the graph neural network encoder, and repeatedly performing multi-level graph convolution operations;
[0087] S34) Optimize the initial features of the input through multi-level feature extraction and updating of the graph neural network encoder; and output a concise and efficient latent space representation.
[0088] S4), inputting the optimized latent space representation of the human-machine skeleton graph obtained in step S3) into the latent neural network, capturing the latent features of the human-machine motion difference through the latent neural network, thereby obtaining the joint angle and connecting rod position information of the robot.
[0089] In this implementation, as a preferred embodiment, in step S4), the latent neural network includes a graph encoder and a graph decoder; wherein the graph encoder performs node and edge update operations on the optimized encoding of the input human-machine skeleton graph through a graph convolutional network GCN, wherein the graph convolutional network GCN is stacked by multiple layers of graph convolutional layers, each layer of graph convolutional layer includes two operations of node update and edge update, the node update fuses the current node features and the aggregated features from the adjacent nodes, and the edge update fuses the current edge features and the aggregated features from the adjacent edges; the graph convolutional network GCN is used to extract high-level node features and edge features, and capture higher-level motion patterns and differences layer by layer;
[0090] The graph decoder decodes the latent features through a fully connected layer or a transposed convolutional layer, and outputs the robot joint angle and link pose information. Figure 3 and 4 shown.
[0091] S5) Through the differentiable forward kinematics layer, the joint angle and link pose information output by the graph decoder are converted into the desired position of the robot end effector; that is, the final motion redirection mapping is completed.
[0092] As a preferred embodiment of this embodiment, in step S4), the potential neural network is used to extract the joint angle and connecting rod posture information of the robot, which specifically includes the following steps:
[0093] S41), mapping the optimized codes of the skeleton graphs of the human body and the robot to the latent neural network respectively, and the graph encoder of the latent neural network uses the graph convolutional network GCN to constrain the optimized code of the input human-machine skeleton graph; by alternately updating the node / edge features of the human body and the robot through the graph convolutional network GCN, higher-level motion patterns and differences are captured layer by layer; thereby obtaining latent features that can minimize the representation of human-machine differences;
[0094] S42), input the optimized aligned latent code into the graph decoder to decode the corresponding joint angle and connecting rod posture information, and in the decoding process, the robot's prior structure information is used to ensure that the decoded output satisfies the robot's kinematic constraints.
[0095] As a preferred embodiment of this invention, in step S41), the graph encoder updates the expression of fused node features and edge features as follows:
[0096] x_v t+1 =Φ(x_v t ,∑_{u∈N(v)}ψ(x_v t ,x_u t ,e_attr(v,u)));
[0097] In the formula, x_v t+1 represents the updated feature vector of node v at time t+1; x_v t represents the feature vector of node v at time t; N(v) is the set of neighbor nodes of node v, u represents a neighbor node of v, x_u t represents the feature vector of node u at time t; e_attr(v,u) is the feature vector of the edge connecting v and u, ψ is the message function that calculates the information vector based on the node and edge information; ∑_{u∈N(v)} is used to summarize the influence of all neighboring nodes; Φ is the update function that integrates its own information and the influence of neighboring nodes to update node features.
[0098] As a preferred embodiment of this invention, in step S42), the objective function of the graph decoder training is:
[0099]
[0100] Where z is the potential code; is the graph decoder parameter; Represents the reconstruction loss function, which is used to measure the output of the graph decoder The difference between Ω(z) and the input X; Ω(z) is the regularization term applied to the latent code z; λ is the weight coefficient of the regularization term.
[0101] As a preferred embodiment of this embodiment, in step S5), the joint angle and connecting rod posture information output by the graph decoder are converted into the desired position of the robot end effector using a differentiable positive kinematics layer, which specifically includes the following steps:
[0102] S51), passing the robot joint angle and connecting rod posture information output by the graph encoder as input to the differentiable positive kinematics layer;
[0103] S52), the forward kinematics layer calculates the transformation matrix of each joint according to the robot's kinematic model and DOF settings;
[0104] S53), calculating the overall motion transformation from the base coordinate system to the end effector through the transformation matrix of each joint, and obtaining the desired position of the end effector;
[0105] S54), comparing the expected position of the end effector with the actual target position to obtain a position error;
[0106] S55), based on the position error, back-propagating to the graph decoder and the graph encoder to optimize the potential representation of the graph encoder and the output of the graph decoder;
[0107] S56) Through multiple iterations of optimization, the position error of the end effector is minimized to obtain accurate motion redirection mapping; thereby obtaining a robot joint angle sequence that can accurately reach the desired position, completing the motion redirection mapping task.
[0108] Preferably, in step S5), the goal of the motion redirection is to determine a set of robot joint angles r* so that the robot's motion posture R (r*) can imitate and fit the given human posture H to the greatest extent under given constraints; at the same time, it cannot exceed the robot's own joint angle limit range, and by optimizing the objective function and satisfying the constraints, obtain the motion trajectory that allows the robot to best imitate human movements, that is:
[0109] r * =argminL(H,R(r));
[0110] str lower ≤r≤r upper ;
[0111] Where r* is the robot joint angle; L represents the target mapping function from the joint angle to the robot's final posture; H is the human body posture information; R(r) represents the robot's movement, which is determined by the joint angle r; r lower 、r upper They are the lower and upper limits of the robot's motion, respectively, reflecting the constraints of the robot's motion range.
[0112] Preferably, in step S5), the robot's joint angles and connecting rod posture information output by the graph decoder are mapped to a robot end effector motion trajectory that can best imitate human body movements by optimizing the target mapping function L; the target mapping function L of the motion redirection is expressed as:
[0113] L=λ ees L ees +λ ee L ee +λ ori L ori +λ fin L fin ;
[0114] Where, L ees is the speed loss function; L ee is the end effector loss function; L ori is the direction loss function; L fin is the finger loss function; ees , ee , ori , fin The weights of the corresponding loss functions respectively.
[0115] Preferably, in step S5), the speed loss function L ees The expression is:
[0116]
[0117] In the formula, s i and r i is the velocity of the robot in the i-th frame and is the normalization coefficient; and is the velocity and normalization coefficient of the human body in the i-th frame.
[0118] Preferably, in step S5), the direction loss function L ori The expression is:
[0119]
[0120] Where: D i is the rotation matrix of the robot’s end effector in the i-th frame; is the rotation matrix of the end effector of the human body in the i-th frame.
[0121] Preferably, in step S5), the finger loss function L fin The expression is:
[0122]
[0123] Where: f i tip is the fingertip position of the robot in the i-th frame; f i meta is the fingertip position of the robot in the i-th frame; r tm is the normalized coefficient of robot arm i; is the fingertip position of the human body in the i-th frame; is the fingertip position of the human body in the i-th frame; is the normalized coefficient of human arm i.
[0124] Preferably, in step S5), the end effector loss function L ee The expression is:
[0125]
[0126] Where: e i and is r i The position and normalization coefficient of the robot's end effector at the i-th frame, and are the end effectors and normalized coefficients of the person in the i-th frame.
[0127] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.
Claims
1. A robot redirection method based on graph neural network, It is characterized in that The steps include: S1), collecting the operator's action video; and extracting the human skeleton structure from the action video; S2), constructing a corresponding tree topology diagram according to the human skeleton structure and the robot structure; S3), using a graph neural network encoder to respectively optimize the encoding of the tree topology graphs of the human body and the robot, and obtain the latent space representation of the tree topology graphs of the human body and the robot; S4), inputting the optimized latent space representation of the human-machine skeleton graph obtained in step S3) into the latent neural network, capturing the latent features of the human-machine motion difference through the latent neural network, and obtaining the joint angle and connecting rod position information of the robot; S5) Through the differentiable forward kinematics layer, the joint angle and link pose information output by the graph decoder are converted into the desired position of the robot end effector; that is, the final motion redirection mapping is completed.
2. A robot redirection method based on graph neural network according to claim 1, Features: In step S1), after obtaining the action video, it is subjected to denoising, color correction and frame rate adjustment processing; and the visual detection algorithm OpenPose and the estimation algorithm Frankmocap algorithm are used to extract the three-dimensional joint point coordinate data and posture quaternion information of the human body.
3. A robot redirection method based on graph neural network according to claim 2, Features: In step S2), the joints of the robot are taken as nodes, and the connecting rods connecting the two joints are taken as undirected edges; the structure of the robot is modeled as an acyclic tree topology graph Gr=(Vr, Er); wherein Vr is the point set of the robot joints; Er is the edge set of the robot connecting rods; the acyclic tree topology graph of the human skeleton structure is Gh=(Vh, Eh); wherein Vh is the point set of the human joints; and Er is the edge set of the human skeleton connection.
4. The robot redirection method based on graph neural network according to claim 1, Features: In step S3), the graph neural network encoder adopts a layered convolution structure, and the graph neural network encoder includes multiple stages, each stage includes multiple convolutional layers of the same type, and each stage processes features of different scales.
5. A robot redirection method based on graph neural network according to claim 4, Features: In step S3), the tree topology graphs of the human body and the robot are encoded and optimized using the graph neural network encoder, which specifically includes the following steps: S31), for the human body tree topology, at each node, the input feature is the rotation angle of the corresponding joint, and at each edge, the input feature is the initial offset and rotation between adjacent edges; For the tree topology of the robot, at each node, the output feature will represent the rotation angle of the corresponding joint, and at each edge, the output feature will represent the initial offset and rotation between the links; S32), using a graph neural network encoder to encode the tree topology graphs of the human body and the robot respectively to generate a latent space representation; S33), transmitting and updating node features and edge features in the graph neural network encoder, and repeatedly performing multi-level graph convolution operations; S34), optimizing the initial features of the input through multi-level feature extraction and updating of the graph neural network encoder; And output a concise and efficient latent space representation.
6. A robot redirection method based on graph neural network according to claim 1, Features: In step S4), the latent neural network includes a graph encoder and a graph decoder; wherein the graph encoder performs node and edge update operations on the optimized encoding of the input human-machine skeleton graph through a graph convolutional network GCN, wherein the graph convolutional network GCN is stacked by multiple layers of graph convolutional layers, each layer of graph convolutional layer includes two operations of node update and edge update, the node update fuses the current node feature and the aggregated feature from the adjacent nodes, and the edge update fuses the current edge feature and the aggregated feature from the adjacent edges; the graph convolutional network GCN is used to extract high-level node features and edge features, and capture higher-level motion patterns and differences layer by layer; The graph decoder decodes the potential features through a fully connected layer or a transposed convolutional layer, and outputs the robot joint angle and connecting rod posture information.
7. A robot redirection method based on graph neural network according to claim 6, Features: In step S4), the potential neural network is used to extract the joint angle and connecting rod posture information of the robot, which specifically includes the following steps: S41), mapping the optimized codes of the skeleton graphs of the human body and the robot to the latent neural network respectively, and then performing continuous constraint optimization in the latent space through the graph encoder to obtain the latent features that can minimize the representation of the human-machine difference; the expression of the graph encoder updating the fusion node features and edge features is: x_v t+1 =Φ(x_v t ,∑_{u∈N(v)}ψ(x_v t ,x_u t ,e_attr(v,u))); In the formula, x_v t+1 represents the updated feature vector of node v at time t+1; x_v t represents the feature vector of node v at time t; N(v) is the set of neighbor nodes of node v, u represents a neighbor node of v, x_u t represents the feature vector of node u at time t; e_attr(v,u) is the feature vector of the edge connecting v and u, ψ is the message function that calculates the information vector based on the node and edge information; ∑ - {u∈N(v)} is used to summarize the influence of all neighbor nodes; Φ is an update function that integrates its own information and the influence of neighboring nodes to update node features; S42), inputting the optimized aligned latent code into the graph decoder to decode the corresponding joint angle and connecting rod posture information, and using the robot's prior structure information to ensure that the decoded output meets the robot's kinematic constraints during the decoding process; the objective function of the graph decoder training is: Where z is the potential code; is the graph decoder parameter; Represents the reconstruction loss function, which is used to measure the output of the graph decoder The difference between Ω(z) and the input X; Ω(z) is the regularization term applied to the latent code z; λ is the weight coefficient of the regularization term.
8. A robot redirection method based on graph neural network according to claim 7, Features: In step S5), the joint angle and link pose information output by the graph decoder are converted into the desired position of the robot end effector using a differentiable forward kinematics layer, which specifically includes the following steps: S51), passing the robot joint angle and connecting rod posture information output by the graph encoder as input to the differentiable positive kinematics layer; S52), the forward kinematics layer calculates the transformation matrix of each joint according to the robot's kinematic model and DOF settings; S53), calculating the overall motion transformation from the base coordinate system to the end effector through the transformation matrix of each joint, and obtaining the desired position of the end effector; S54), comparing the expected position of the end effector with the actual target position to obtain a position error; S55), based on the position error, back-propagating to the graph decoder and the graph encoder to optimize the potential representation of the graph encoder and the output of the graph decoder; S56) Through multiple iterations of optimization, the position error of the end effector is minimized to obtain accurate motion redirection mapping; thereby obtaining a robot joint angle sequence that can accurately reach the desired position, completing the motion redirection mapping task.
9. A robot redirection method based on graph neural network according to claim 8, Features: In step S5), the goal of the motion redirection is to determine a set of robot joint angles r* so that the robot's motion posture R(r*) can imitate and fit the given human posture H to the greatest extent under given constraints; at the same time, it cannot exceed the robot's own joint angle limit range. By optimizing the objective function and satisfying the constraints, the motion trajectory that allows the robot to best imitate human movements is obtained, that is: r * =argminL(H,R(r)); s.t.r lower ≤r≤r upper ; Where r* is the robot joint angle; L represents the target mapping function from the joint angle to the robot's final posture; H is the human body posture information; R(r) represents the robot's movement, which is determined by the joint angle r; r lower 、r upper They are the lower and upper limits of the robot's motion, respectively, reflecting the constraints of the robot's motion range.
10. A robot redirection method based on graph neural network according to claim 9, Features: In step S5), the robot's joint angles and connecting rod posture information output by the graph decoder are mapped to the robot end effector motion trajectory that can best imitate human body movements by optimizing the target mapping function L; the expression of the target mapping function L of the motion redirection is: L=λ ees L ees +λ ee L ee +λ ori L ori +λ fin L fin ; Where, L ees is the speed loss function; L ee is the end effector loss function; L ori is the direction loss function; L fin is the finger loss function; ees , ee , ori , fin The weights of the corresponding loss functions respectively.
Citation Information
Patent Citations
Motion capture missing data recovery method combining graph neural network and bone length constraint
CN116563952A
Sign language recognition method and device, electronic equipment and storage medium
CN117437694A
Multi-view human action recognition method based on hypergraph learning
US20240177525A1
Cited By
Double-arm smart teleoperation method and system based on motion redirection and adaptive force
CN121670679A
Cross-domain man-machine motion redirection method and system based on skeleton graph convolutional network
CN121904322A
