Humanoid robot action simulation method and system based on human body linear model

Abstract

The invention provides a humanoid robot action simulation method and system based on a human body linear model, and the method comprises the steps: S1, collecting the video input of human body actions, and extracting the action information frame by frame; s2, carrying out modeling based on the action information, and obtaining SMPL representation and three-dimensional representation of a human body posture; s3, based on the SMPL representation and the three-dimensional representation of the human body posture, generating SMPL sequence data in the time dimension and the three-dimensional representation of the human body action, and constructing a dynamics tree model of the robot; and S4, planning a motion track of the robot through the dynamic tree model, and generating a corresponding control instruction. According to the method, mesh modeling is carried out on human body actions and the SMPL model is extracted, so that human body dynamics are accurately captured. Based on the SMPL sequence of the motion, the power tree model adapted to the motion of the humanoid robot is constructed, real-time simulation of the robot to human actions is realized, and the accuracy and real-time performance of the actions are improved.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction technology, and in particular, relates to a humanoid robot motion imitation method and system based on a human linear model, and more particularly, to a humanoid robot motion imitation method based on a skinned human linear model. Background Art

[0002] At present, most of the methods for humanoid robots to imitate human movements are limited to local movements such as gait walking and two-handed operation, and usually only focus on partial movements of the human body, such as the simulation of arms or a single limb. These methods often ignore the dynamic modeling of the entire human body and fail to achieve comprehensive imitation of complex and dynamic human movements. Therefore, such methods lack accurate capture and real-time simulation of the entire human body movement process, and cannot achieve more natural and smooth humanoid robot movement imitation.

[0003] Patent document CN116079727A discloses a method and device for humanoid robot motion imitation based on 3D human body posture estimation. The method collects human body posture point cloud data and estimates the position of human body skeletal points to obtain the coordinates of key points of the human body, and then maps them to the robot's arm joint angles and performs filtering and smoothing, thereby realizing the humanoid robot's imitation of the instructor's movements. However, this type of method is mainly aimed at the static movements of the human body and can only achieve the movement imitation of the arms. Its disadvantage is that it cannot capture the dynamic changes in the human body during movement, and lacks the motion dynamics modeling of the humanoid robot, resulting in the inability to achieve real-time and continuous imitation of human movements.

[0004] Patent document CN106078752A discloses a human behavior imitation method for a humanoid robot based on Kinect, which includes: 1) extracting human skeleton information from RGB 3D images collected by Kinect to construct a human motion model; 2) constructing a humanoid robot motion model based on the limb structure characteristics of the humanoid robot; 3) converting the collected human actions into actions in the robot motion model through a mapping model; 4) adjusting the robot joint angles using balance control technology so that the robot remains stable during the imitation process; 5) after performing self-collision avoidance processing on the robot's state, sending the final joint configuration data to the robot to complete the imitation action. This solution cannot enable the robot to more accurately imitate the complex actions of the human body and improve the naturalness and fluency of the actions. This problem needs to be solved urgently. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a method and system for imitating the motion of a humanoid robot based on a human linear model.

[0006] A humanoid robot motion imitation method based on a human linear model provided by the present invention comprises:

[0007] Step S1: Collecting video input of human body movements and extracting movement information frame by frame;

[0008] Step S2: Modeling is performed based on the action information, and a SMPL representation and a three-dimensional representation of the human body posture are obtained;

[0009] Step S3: Based on the SMPL representation and the three-dimensional representation of the human body posture, SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion are generated to construct a dynamic tree model of the robot;

[0010] Step S4: planning the robot motion trajectory through the dynamic tree model and generating corresponding control instructions.

[0011] Preferably, in step S1, the action information is the visual features of the human body shape and the motion features of the changes between frames; the visual features include the human body contour and joint morphology; the motion features include the movement and posture of the human body;

[0012] The action information is extracted based on a deep network.

[0013] Preferably, in the step S2, the modeling is to construct a Mesh model;

[0014] In the step S2, based on the motion information, a multi-layer perceptron is used to estimate the human body posture and obtain a Mesh model of the human body; the Mesh model of the human body is mapped to a three-dimensional space through an SMPL model to generate a three-dimensional representation of the human body posture;

[0015] In step S3, the SMPL sequence data is a sequence consisting of three-dimensional posture parameters output by the SMPL model in each frame and the motion trajectory of each joint; the three-dimensional posture parameters include joint positions, rotation matrices, and shape parameters of the human body;

[0016] The process of establishing and optimizing the kinetic tree model includes:

[0017] Step A1: Construct the dynamic tree model of the robot according to the SMPL sequence data output by the human motion modeling module; use the Newton-Lagrange method to obtain the relationship between joint torque and joint motion;

[0018] Step A2: constructing a complete robot dynamics model based on the relationship between the joint torque and the joint motion and the dynamics tree model of the robot;

[0019] The complete robot dynamics model includes the physical connection relationship between robot joints, inertia parameters and mass distribution;

[0020] Step A3: based on the complete robot dynamics model, extract the target joint position from each frame of the SMPL sequence data, and calculate the angle of each joint according to the target joint position, so as to generate a motion trajectory as an excitation trajectory; the target joint position includes the position of the hand and the position of the foot;

[0021] Step A4: Based on the motion trajectory, collecting the joint position, speed, and torque of the robot as training data;

[0022] Step A5: low-pass filtering the training data to obtain filtered training data;

[0023] Step A6: training a deep learning network through filtered training data; the deep learning network outputs a kinetic tree model; optimizing and updating the control strategy based on the parameters of the kinetic tree model, and judging whether the optimization error is less than a preset threshold value, if the result is yes, then the process ends, if the result is no, then re-execute step A3;

[0024] The control strategy includes: adjusting the torque control parameters of the joint, optimizing the motion trajectory and redefining the rules of feedback control;

[0025] In the step A1, the dynamic tree model provides the kinetic energy and gravitational potential energy of the robot joints;

[0026] The mathematical expression of the kinetic energy is:

[0027]

[0028] Where T represents the kinetic energy of the robot joints, n represents the total number of robot joints; m i represents the mass of the i-th joint; v i represents the velocity of the i-th joint;

[0029] The mathematical expression of the gravitational potential energy is:

[0030]

[0031] Where V represents the gravitational potential energy of the robot joint; g represents the gravitational acceleration; h i represents the height of the center of gravity of the i-th joint;

[0032] The relationship between the joint torque and the joint motion is expressed mathematically as follows:

[0033]

[0034] Among them, qj represents the generalized coordinates of the jth joint, represents the generalized velocity of the jth joint, τ j is the joint torque of the jth joint;

[0035] In step A2, the complete robot dynamics model is expressed as follows:

[0036]

[0037] in, are the Coriolis and centrifugal force matrices; M(q) represents the mass matrix; G(q) represents the gravity term; q represents the generalized coordinates of the joint, represents the generalized velocity of the joint; represents the generalized acceleration of the joint.

[0038] Preferably, the step A1 includes:

[0039] Step A1.1: Take each frame of SMPL sequence data as input data and input it into a deep network composed of multi-layer perceptrons, i.e., a deep neural network;

[0040] Step A1.2: extracting the features of the input data based on the deep network and reducing the dimension to obtain the first layer of features; mapping the first layer of features into dynamic tree model parameters; the dynamic tree model parameters include: joint mass, inertia matrix, damping coefficient and stiffness coefficient;

[0041] Step A1.3: outputting a dynamic tree model of the robot based on the dynamic tree model parameters;

[0042] Step A1.4: Based on the dynamic tree model parameters, the kinetic energy and potential energy of each joint are calculated and obtained; the kinetic energy and potential energy are solved by Lagrange equations to obtain the explicit relationship between joint torque and joint position and the explicit relationship between velocity and acceleration, and then the relationship between joint torque and joint motion is obtained;

[0043] In the step A3, it includes:

[0044] Step A3.1: extracting target joint positions from the SMPL posture data of each frame; the target joint positions include the positions of the hands and the feet;

[0045] Step A3.2: Based on the target joint position, the angles of each joint are obtained, and based on inverse kinematics, a set of joint angle values ​​that satisfy the target joint position are obtained;

[0046] Step A3.3: Based on the dynamic constraints and kinematic restrictions of each joint, the joint angle values ​​are optimized; the dynamic constraints include: speed restrictions, acceleration restrictions and torque restrictions; the kinematic restrictions include: joint angle range and degrees of freedom; the optimization is to set numerical boundaries for the joint angle values;

[0047] Step A3.4: Connect the optimized joint angle values ​​into a smooth and continuous trajectory, i.e., the target motion trajectory, by using the cubic spline interpolation method;

[0048] The smooth and continuous trajectory refers to a trajectory with a smoothness index less than or equal to 0.1 and continuous;

[0049] The mathematical expression of the smoothness index is:

[0050]

[0051] Among them, S represents the smoothness index, t0 represents the starting time of the trajectory; t f represents the end time of the trajectory; θ represents the joint angle; t represents time;

[0052] In step A6, the process of optimizing the parameters of the kinetic tree model includes:

[0053] Step B1: deriving predicted values ​​through the dynamics tree model; the predicted values ​​include: joint position, velocity and torque;

[0054] Step B2: Calculating the loss function value of the predicted value and the actual data through the loss function;

[0055] Step B3: Calculate the gradient of the loss function to the network parameters by back propagation algorithm, and update the network parameters by gradient descent method, so as to obtain the optimized dynamic tree model;

[0056] In step B2, the mathematical expression of the loss function is:

[0057]

[0058] Where N represents the number of motion trajectory frames, represents the predicted joint position of the i-th frame, q i Indicates the actual joint position; represents the prediction speed, v i Indicates actual speed; represents the prediction moment, τ i represents the actual torque. α, β and γ are a weight coefficient, another weight coefficient and another weight coefficient respectively; || represents the L2 norm of the vector, that is, the Euclidean distance, Represents the final loss function value;

[0059] In step B3, the mathematical expression of the gradient of the loss function to the network parameters is:

[0060]

[0061] in, Represents the gradient of the loss function with respect to the network parameters, and the superscript T represents the transpose of the matrix or vector; represents the predicted joint position of the i-th frame; q i represents the actual joint position of the i-th frame; represents the predicted speed of the i-th frame; v i Indicates the actual speed of the i-th frame.

[0062] The gradient of the joint position is expressed mathematically as:

[0063]

[0064] in, The gradient representing the joint position;

[0065] The gradient of velocity is expressed mathematically as:

[0066]

[0067] in, represents the gradient of velocity;

[0068] The gradient of the torque is expressed mathematically as:

[0069]

[0070] in, represents the gradient of the moment, W represents the network parameter;

[0071] The mathematical expression of the network parameter, i.e. the optimization error, is:

[0072]

[0073] Among them, W represents the network parameters, that is, the optimization error, ← represents the assignment operation, represents the partial derivative, Represents the final loss function value, and η represents the learning rate.

[0074] A humanoid robot motion imitation system based on a human linear model provided by the present invention comprises:

[0075] Module M1: collects video input of human movements and extracts movement information frame by frame;

[0076] Module M2: Modeling based on the action information, and obtaining a SMPL representation and a three-dimensional representation of the human body posture;

[0077] Module M3: Based on the SMPL representation and the three-dimensional representation of the human body posture, generate SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion, and construct a dynamic tree model of the robot;

[0078] Module M4: plans the robot motion trajectory through the dynamic tree model and generates corresponding control instructions.

[0079] Preferably, in the module M1, the action information is the visual features of the human body shape and the motion features of the changes between frames; the visual features include the human body contour and joint morphology; the motion features include the movement and posture of the human body;

[0080] The action information is extracted based on a deep network.

[0081] Preferably, in the module M2, the modeling is to construct a Mesh model;

[0082] In the module M2, based on the motion information, a multi-layer perceptron is used to estimate the human body posture and obtain a Mesh model of the human body; the Mesh model of the human body is mapped to a three-dimensional space through an SMPL model to generate a three-dimensional representation of the human body posture;

[0083] In the module M3, the SMPL sequence data is a sequence consisting of three-dimensional posture parameters output by the SMPL model in each frame and the motion trajectory of each joint; the three-dimensional posture parameters include joint positions, rotation matrices, and shape parameters of the human body;

[0084] The establishment and optimization of the kinetic tree model includes:

[0085] Module A1: Construct the dynamic tree model of the robot based on the SMPL sequence data output by the human motion modeling module; use the Newton-Lagrange method to obtain the relationship between joint torque and joint motion;

[0086] Module A2: Based on the relationship between the joint torque and the joint motion and the dynamics tree model of the robot, a complete robot dynamics model is constructed;

[0087] The complete robot dynamics model includes the physical connection relationship between robot joints, inertia parameters and mass distribution;

[0088] Module A3: Based on the complete robot dynamics model, extract the target joint position from the SMPL sequence data of each frame, and calculate the angle of each joint according to the target joint position, so as to generate a motion trajectory as an excitation trajectory; the target joint position includes the position of the hand and the position of the foot;

[0089] Module A4: Based on the motion trajectory, collecting the joint position, speed and torque of the robot as training data;

[0090] Module A5: performing low-pass filtering on the training data to obtain filtered training data;

[0091] Module A6: train the deep learning network through the filtered training data; the deep learning network outputs a kinetic tree model; optimize and update the control strategy based on the parameters of the kinetic tree model, and judge whether the optimization error is less than a preset threshold. If the result is yes, end; if the result is no, re-trigger the module A3 to work;

[0092] The control strategy includes: adjusting the torque control parameters of the joint, optimizing the motion trajectory and redefining the rules of feedback control;

[0093] In the module A1, the dynamic tree model provides the kinetic energy and gravitational potential energy of the robot joints;

[0094] The mathematical expression of the kinetic energy is:

[0095]

[0096] Where T represents the kinetic energy of the robot joints, n represents the total number of robot joints; m i represents the mass of the i-th joint; v i represents the velocity of the i-th joint;

[0097] The mathematical expression of the gravitational potential energy is:

[0098]

[0099] Where V represents the gravitational potential energy of the robot joint; g represents the gravitational acceleration; h i represents the height of the center of gravity of the i-th joint;

[0100] The relationship between the joint torque and the joint motion is expressed mathematically as follows:

[0101]

[0102] Among them, q j represents the generalized coordinates of the jth joint, represents the generalized velocity of the jth joint, τ jis the joint torque of the jth joint;

[0103] In the module A2, the complete robot dynamics model is expressed as follows:

[0104]

[0105] in, are the Coriolis and centrifugal force matrices; M(q) represents the mass matrix; G(q) represents the gravity term; q represents the generalized coordinates of the joint, represents the generalized velocity of the joint; represents the generalized acceleration of the joint.

[0106] Preferably, the module A1 includes:

[0107] Module A1.1: Take each frame of SMPL sequence data as input data and input it into a deep network composed of multi-layer perceptrons, i.e., a deep neural network;

[0108] Module A1.2: extracting the features of the input data based on the deep network and reducing the dimension to obtain the first layer of features; mapping the first layer of features into dynamic tree model parameters; the dynamic tree model parameters include: joint mass, inertia matrix, damping coefficient and stiffness coefficient;

[0109] Module A1.3: outputting a dynamic tree model of the robot based on the dynamic tree model parameters;

[0110] Module A1.4: Based on the dynamic tree model parameters, the kinetic energy and potential energy of each joint are calculated and obtained; the kinetic energy and potential energy are solved by Lagrange equations to obtain the explicit relationship between joint torque and joint position and the explicit relationship between velocity and acceleration, and then the relationship between joint torque and joint motion is obtained;

[0111] The module A3 includes:

[0112] Module A3.1: Extracting target joint positions from the SMPL posture data of each frame; the target joint positions include the positions of the hands and the feet;

[0113] Module A3.2: Based on the target joint position, obtain the angle of each joint, and obtain a set of joint angle values ​​that meet the target joint position based on inverse kinematics;

[0114] Module A3.3: Optimizing the joint angle value based on the dynamic constraints and kinematic restrictions of each joint; the dynamic constraints include: speed restriction, acceleration restriction and torque restriction; the kinematic restrictions include: joint angle range and degree of freedom; the optimization is to set numerical boundaries for the joint angle value;

[0115] Module A3.4: Connect the optimized joint angle values ​​into a smooth and continuous trajectory, i.e., the target motion trajectory, through cubic spline interpolation;

[0116] The smooth and continuous trajectory refers to a trajectory with a smoothness index less than or equal to 0.1 and continuous;

[0117] The mathematical expression of the smoothness index is:

[0118]

[0119] Among them, S represents the smoothness index, t0 represents the starting time of the trajectory; t f represents the end time of the trajectory; θ represents the joint angle; t represents time;

[0120] In the module A6, the process of optimizing the parameters of the kinetic tree model includes:

[0121] Module B1: deriving predicted values ​​through the dynamic tree model; the predicted values ​​include: joint position, velocity and torque;

[0122] Module B2: Calculate the loss function value of the predicted value and the actual data through the loss function;

[0123] Module B3: Calculate the gradient of the loss function to the network parameters through the back propagation algorithm, and update the network parameters through the gradient descent method to obtain the optimized dynamic tree model;

[0124] In the module B2, the mathematical expression of the loss function is:

[0125]

[0126] Where N represents the number of motion trajectory frames, represents the predicted joint position of the i-th frame, q i Indicates the actual joint position; represents the prediction speed, v i Indicates actual speed; represents the prediction moment, τ i represents the actual torque. α, β and γ are a weight coefficient, another weight coefficient and another weight coefficient respectively; || represents the L2 norm of the vector, that is, the Euclidean distance, Represents the final loss function value;

[0127] In the module B3, the mathematical expression of the gradient of the loss function to the network parameters is:

[0128]

[0129] in, Represents the gradient of the loss function with respect to the network parameters, and the superscript T represents the transpose of the matrix or vector; represents the predicted joint position of the i-th frame; q i represents the actual joint position of the i-th frame; represents the predicted speed of the i-th frame; v i Indicates the actual speed of the i-th frame.

[0130] The gradient of the joint position is expressed mathematically as:

[0131]

[0132] in, The gradient representing the joint position;

[0133] The gradient of velocity is expressed mathematically as:

[0134]

[0135] in, represents the gradient of velocity;

[0136] The gradient of the torque is expressed mathematically as:

[0137]

[0138] in, represents the gradient of the moment, and W represents the network parameter;

[0139] The mathematical expression of the network parameter, i.e. the optimization error, is:

[0140]

[0141] Among them, W represents the network parameters, that is, the optimization error, ← represents the assignment operation, represents the partial derivative, Represents the final loss function value, and η represents the learning rate.

[0142] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the humanoid robot motion imitation method based on a human linear model are implemented.

[0143] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the humanoid robot motion imitation method based on the human linear model are implemented.

[0144] Compared with the prior art, the present invention has the following beneficial effects:

[0145] 1. The present invention accurately captures human body dynamics by mesh modeling and extracting SMPL models of human body movements. Based on the SMPL sequence of the movement, a dynamic tree model adapted to the movement of humanoid robots is constructed to achieve real-time imitation of human movements by robots, thereby improving the accuracy and real-time performance of the movements.

[0146] 2. The present invention can accurately capture human motion data by combining RGB camera with computer vision algorithm.

[0147] 3. The present invention uses the SMPL model to achieve high-quality three-dimensional human posture representation, ensuring the accuracy and efficiency of input data.

[0148] 4. The key innovation of the present invention lies in the integration of the motion optimization unit and the deep learning method, which effectively solves the accuracy problem in robot motion control. Especially in the case of robot structural complexity and inconsistent dynamic parameters, the optimized robot dynamics model significantly improves the accuracy of motion control.

[0149] 5. The present invention optimizes the robot's dynamic parameters by introducing deep learning network training, which can automatically adapt and accurately adjust according to the characteristics of different humanoid robots, further improving the accuracy and stability of motion control. During the optimization process, the natural fluency of the robot's motion imitation is significantly improved.

[0150] 6. In terms of the control module, the present invention successfully overcomes external interference and internal structural errors through real-time motion planning and error compensation mechanism, and significantly improves the robot's execution ability in complex environments. Experiments show that the robot's accuracy and execution efficiency in performing imitation actions have been significantly improved.

[0151] 7. The method proposed in the present invention is highly scalable and universal, and is applicable to various types of humanoid robots. It not only reduces the difficulty of robot programming, but also improves the flexibility and adaptability of robot motion imitation. It can achieve efficient and accurate motion imitation in a variety of scenarios, and has broad application prospects and market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0152] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0153] Figure 1 A schematic diagram of the workflow provided by the present invention;

[0154] Figure 2This is a schematic diagram of the humanoid robot motion imitation effect of the human linear model provided by the present invention. DETAILED DESCRIPTION

[0155] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0156] The present invention proposes an imitation learning scheme for a humanoid robot. Specifically, the present invention combines the multi-module collaboration of human motion acquisition, motion modeling, dynamics optimization and control strategy to propose an efficient and accurate robot motion imitation method.

[0157] The present invention solves the problem in the prior art that humanoid robots cannot capture dynamic changes when imitating human movements and lack dynamic modeling of the entire humanoid body by adopting a comprehensive solution of a human motion acquisition module, a human motion modeling module, a robot power tree modeling module and a robot motion control module.

[0158] The present invention collects human motion data through an RGB camera and extracts and optimizes the SMPL model in combination with a deep learning algorithm, so that the robot can more accurately imitate the complex movements of the human body. Furthermore, by constructing a dynamic tree model adapted to the humanoid robot and optimizing the robot motion control accuracy in combination with a deep learning network, it is ensured that the robot can imitate the movements of the human body in real time and continuously. Ultimately, the present invention achieves the effect of enabling the robot to accurately imitate the dynamic movements of the human body and effectively improves the fluency and naturalness of the imitated movements.

[0159] According to a humanoid robot motion imitation method based on a human linear model provided by the present invention, that is, a method for directly imitating human motion by a humanoid robot, the flow of its software system includes:

[0160] Step S1: collecting human motion data, and obtaining motion information based on the human motion data;

[0161] Step S2: constructing a mesh model based on the motion information, extracting and constructing a three-dimensional representation of human motion through an SMPL model;

[0162] Step S3: outputting SMPL sequence data based on the three-dimensional representation of the human body motion;

[0163] Step S4: constructing a dynamic tree model of the robot based on the SMPL sequence data;

[0164] Step S5: Plan the motion trajectory of the robot and generate control instructions through the dynamic tree model.

[0165] In other words, a humanoid robot motion imitation method based on a human linear model provided by the present invention comprises:

[0166] Step S1: Collect video input of human movements and extract appearance and motion features frame by frame through a deep network;

[0167] Step S2: Based on the appearance and motion features extracted frame by frame, human body Mesh modeling is performed, and the SMPL representation and the three-dimensional representation of human body posture are obtained using the SMPL model;

[0168] Step S3: Based on the SMPL representation constructed frame by frame and the three-dimensional representation of the human body posture, generate SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion;

[0169] Step S4: constructing a dynamic tree model of the robot based on the SMPL sequence and the three-dimensional representation of human body movements;

[0170] Step S5: Plan the robot motion trajectory through the dynamic tree model and generate corresponding control instructions.

[0171] In step S1, the motion information refers to the visual features of the human body shape and the motion features of the changes between frames;

[0172] The visual features include: human appearance, contour and joint morphology;

[0173] The motion characteristics include: movement and posture of the human body;

[0174] Specifically, the action information is extracted based on a deep network.

[0175] Specifically, the motion information is extracted based on a deep network, which means that the motion features are analyzed and extracted by using a deep learning algorithm of computer vision through an RGB video input of human body movements. This feature is represented in the form of an intermediate vector and is ultimately mapped to the parameters of the SMPL model.

[0176] Motion optimization refers to the optimization of the robot's dynamics, with SMPL model parameters as input and the robot's dynamics tree parameters as output. A mapping relationship between the two is established through a deep neural network, and with the help of deep learning optimization algorithms, this mapping is continuously optimized to generate dynamics tree parameters that are more in line with human movements, and ultimately build a more accurate robot dynamics model that is more in line with human motion characteristics.

[0177] In the step S2, human body Mesh modeling is performed, and the SMPL representation and the three-dimensional representation of the human body posture are obtained using the SMPL model, which is specifically achieved as follows;

[0178] In step S2, the posture of the human body Mesh is estimated by combining visual features and motion features, which is specifically achieved as follows:

[0179] First, based on the visual features extracted in step S1, i.e., the motion information, a multi-layer perceptron, i.e., MLP, is used to estimate the Mesh appearance of the human body;

[0180] Then, combined with the motion features, the posture of the human body Mesh is further estimated;

[0181] Finally, the estimated human body mesh is mapped to three-dimensional space through the SMPL model to generate a three-dimensional representation of the human body posture;

[0182] In step S3, the SMPL sequence data is the three-dimensional posture parameters output by the SMPL model of each frame in the time dimension, including the sequence composed of joint positions, rotation matrices, shape parameters of the human body, and the motion trajectory of each joint;

[0183] Specifically, it is obtained as follows: by processing the SMPL three-dimensional posture parameters of each frame obtained in step S2, extracting the joint position and rotation matrix of each frame, and combining with the shape parameters of the human body, a three-dimensional representation of the human body posture of each frame is constructed; then, based on the joint position information, the motion trajectory of each joint is calculated and time-serialized, and finally the three-dimensional postures and motion trajectories of all frames are connected in time sequence to form a complete SMPL sequence data representing the evolution of human body movements.

[0184] In step S4, the kinetic tree model is specifically established as follows:

[0185] Based on SMPL sequence data, the Newton-Lagrangian method and deep network are used to establish the relationship between joint torque and motion, and a preliminary dynamic tree model is obtained.

[0186] Then, the motion optimization unit is used to perform iterative optimization on the dynamics tree. In the motion optimization unit, the excitation trajectory is used to generate data using signals such as joint position and velocity to train the deep network. At the same time, a low-pass filter is used to filter out high-frequency noise and burr signals to reduce interference.

[0187] The signal data after filter processing will be passed to the optimization unit for further optimization. The iterative process will end when the optimization error of the kinetic tree model is lower than the preset threshold, indicating that the accuracy of the model has reached the required level.

[0188] In step S5, the motion trajectory is specifically planned through the dynamic tree model as follows: based on the joint torque and motion relationship in the dynamic tree model, combined with the dynamic constraints and kinematic restrictions of each joint, the trajectory planning algorithm is used to calculate the optimal motion trajectory of the robot from the initial position to the target position.

[0189] Specifically, this trajectory planning algorithm is the motion trajectory generation method mentioned in step A4. The specific steps of the trajectory algorithm are as follows:

[0190] First, based on the target position of the given end effector, such as the hands and feet, the angles of each joint are calculated using the inverse kinematics algorithm in combination with the kinematic model and joint degrees of freedom of the humanoid robot. The inverse kinematics equations are solved by the Newton iteration method or analytical method to generate a set of joint angle values ​​that meet the target position of the end effector.

[0191] Then, based on the relationship between joint torque and motion in the dynamics tree model, combined with the dynamic constraints of each joint, such as speed, acceleration and torque limits, and kinematic constraints, such as joint angle range and degrees of freedom, the joint angles are optimized and adjusted to ensure that the generated motion trajectory conforms to the dynamic characteristics of human motion;

[0192] Next, the cubic spline interpolation method is used to connect the optimized joint angle values ​​into a smooth and continuous trajectory;

[0193] Finally, feedback control combined with PID controllers is used to adjust joint motion in real time to ensure that the robot can accurately track the generated trajectory.

[0194] Specifically, the control instructions are generated through the motion trajectory: the motion trajectory generated by the trajectory planning module will be discretized and transformed by the control module to form a series of control instructions, which define the motion trajectory and speed of each joint or actuator of the robot. The control instructions are then transmitted to the robot through the drive system, and the execution unit monitors the posture through real-time sensor feedback, and corrects the deviation in the execution process in combination with the error compensation unit.

[0195] The optimization steps of the kinetic tree model and its software flow include:

[0196] Step A1: Based on the SMPL sequence data output by the human motion modeling module, a preliminary dynamic tree model suitable for humanoid robot motion is constructed in combination with a deep network. The Newton-Lagrange method is used to establish the relationship between joint torque and joint motion to ensure that the robot motion is highly consistent with the human motion.

[0197] In the step A1, it includes:

[0198] Step A1.1: Take each frame of SMPL sequence data as input data and input it into a deep network composed of multi-layer perceptrons, i.e., a deep neural network;

[0199] Step A1.2: extracting the features of the input data based on the deep network and reducing the dimension to obtain the first layer of features; mapping the first layer of features into dynamic tree model parameters; the dynamic tree model parameters include: joint mass, inertia matrix, damping coefficient and stiffness coefficient;

[0200] Step A1.3: outputting a dynamic tree model of the robot based on the dynamic tree model parameters;

[0201] Step A1.4: Based on the dynamic tree model parameters, the kinetic energy and potential energy of each joint are calculated and obtained; the kinetic energy and potential energy are solved by Lagrange equations to obtain the explicit relationship between joint torque and joint position and the explicit relationship between velocity and acceleration, and then the relationship between joint torque and joint motion is obtained;

[0202] Specifically, in the step A1, in the process of constructing the humanoid robot motion dynamics tree model according to the SMPL sequence data output by the human motion modeling module;

[0203] First, the SMPL sequence data of each frame is input into a deep network composed of two layers of multi-layer perceptrons. The first layer of the network extracts the features of the input data and reduces the dimension through a nonlinear activation function. The second layer of the network maps the extracted features into the parameters required by the dynamic tree model. These parameters include joint mass, inertia matrix, damping coefficient and stiffness coefficient, etc., which fully describe the dynamic characteristics of each joint of the humanoid robot. Through this process, the deep network directly outputs the initial dynamic tree model of the humanoid robot's motion.

[0204] After obtaining the initial dynamics tree model, the Newton-Lagrange method is used to establish the relationship between the joint torque and joint motion of the humanoid robot. Specifically, based on the parameters of the dynamics tree model, the kinetic energy and potential energy of each joint are first calculated. The kinetic energy is determined by the joint mass and velocity, and the potential energy is determined by the joint position and external force fields such as gravity. Then the kinetic energy and potential energy are substituted into the Lagrange equation. By solving the Lagrange equation, the explicit relationship between the joint torque and the joint position, velocity and acceleration is obtained. This relationship can be described as the dynamics equation of the humanoid robot, so that the required joint torque can be directly calculated according to the motion state of the robot.

[0205] The entire method establishes an accurate mapping between human motion and humanoid robot motion. The human motion features extracted from SMPL sequence data are mapped to the parameters of the robot dynamics tree model through a deep network, and the dynamic equations based on the Newton-Lagrangian method further map the human joint motion to the robot joint torque, thereby ensuring that the motion of the humanoid robot can accurately imitate human motion and meet the physical constraints and control requirements of the robot, ultimately achieving efficient conversion and precise control from human motion to humanoid robot motion.

[0206] Specifically, the Lagrangian function is first defined: L=TV, where the kinetic energy T and potential energy V accumulate the kinetic energy and gravitational potential energy of all joints.

[0207] T and V are expressed as:

[0208]

[0209]

[0210] Among them, m i is the mass of the ith joint, v i is its linear velocity, g is the acceleration due to gravity, h i is the center of gravity height.

[0211] Then, through the Lagrange equation Derive the dynamic equations that describe the relationship between the joints.

[0212] In other words, in step A1, the dynamic tree model provides the kinetic energy and gravitational potential energy of the robot joints;

[0213] The mathematical expression of the kinetic energy is:

[0214]

[0215] Where T represents the kinetic energy of the robot joints, n represents the total number of robot joints; m i represents the mass of the i-th joint; v i represents the velocity of the i-th joint;

[0216] The mathematical expression of the gravitational potential energy is:

[0217]

[0218] Where V represents the gravitational potential energy of the robot joint; g represents the gravitational acceleration; h i represents the height of the center of gravity of the i-th joint;

[0219] The relationship between the joint torque and the joint motion is expressed mathematically as follows:

[0220]

[0221] Among them, q j represents the generalized coordinates of the jth joint, represents the generalized velocity of the jth joint, τ j is the joint torque of the jth joint;

[0222] In step A2, the complete robot dynamics model is expressed as follows:

[0223]

[0224] in, are the Coriolis and centrifugal force matrices; M(q) represents the mass matrix; G(q) represents the gravity term; q represents the generalized coordinates of the joint, represents the generalized velocity of the joint; represents the generalized acceleration of the joint.

[0225] Step A2: Based on the SMPL sequence data, a complete robot dynamics model is constructed, including the physical connection relationship between joints, inertia parameters, mass distribution, etc.

[0226] Inertial parameters such as mass, center of mass position and moment of inertia are combined in the dynamic equations through the mass matrix M(q) and the gravity term G(q), which ultimately forms

[0227] in, are the Coriolis and centrifugal force matrices.

[0228] This model comprehensively reflects the motion and torque relationship of each joint of the robot, and combines the physical connection and mass distribution of each joint.

[0229] Step A3: Generate the excitation trajectory according to the SMPL sequence data. Through the inverse kinematics algorithm, i.e., inverse kinematics, the target joint position is extracted from the SMPL posture data of each frame, and then the motion trajectory of each joint is solved by inverse kinematics according to the robot's degrees of freedom and physical constraints. Inverse kinematics learns to calculate the angles of each joint according to the given end effector, such as hands and feet, and thus generates a motion trajectory that conforms to human movements, ensuring that the robot can accurately imitate human movements in the SMPL model;

[0230] In the step A3, it includes:

[0231] Step A3.1: extracting the target joint position from the SMPL posture data of each frame; the target joint position includes the position of the hand and the position of the foot;

[0232] Step A3.2: Based on the target joint position, the angles of each joint are obtained, and based on inverse kinematics, a set of joint angle values ​​that satisfy the target joint position are obtained;

[0233] Step A3.3: Optimizing the joint angle value based on the dynamic constraints and kinematic restrictions of each joint; the dynamic constraints include: speed restriction, acceleration restriction and torque restriction; the kinematic restrictions include: joint angle range and degree of freedom; the optimization sets a numerical boundary for the joint angle value;

[0234] Step A3.4: Connect the optimized joint angle values ​​into a smooth and continuous trajectory, i.e., the target motion trajectory, by using the cubic spline interpolation method; in other words, connect the optimized joint angle values ​​into a smoothness index satisfying And the continuous trajectory, i.e., the target motion trajectory;

[0235] Step A4: Collect training data such as joint position, velocity, torque, etc.;

[0236] Step A5: low-pass filter the collected data to filter out high-frequency noise and burr signals to reduce interference;

[0237] Step A6: Use the processed data to train the deep learning network, optimize the parameters of the kinetic tree model, and update the control strategy.

[0238] In step A6, in this step, the joint position, velocity, and torque data preprocessed in step A5 are used as true labels to optimize the deep network. The optimization process first calculates the output of the deep network, i.e., the dynamic tree model, through forward propagation, and derives the corresponding joint position, velocity, and torque prediction values ​​based on this model. The prediction values ​​are then compared with the true label data, and the loss function is calculated to measure the difference between the predicted value and the true value. The gradient of the loss function to the network parameters is then calculated using the back propagation algorithm, and the network parameters are updated by the gradient descent method to reduce the loss function value and improve the model accuracy. In addition, according to the optimized dynamic tree model, the joint control strategy of the robot is updated, including adjusting the torque control parameters of the joints, optimizing the motion trajectory, and redefining the rules of feedback control.

[0239] In step A6, the process of optimizing the parameters of the kinetic tree model includes:

[0240] Step B1: deriving predicted values ​​through the dynamics tree model; the predicted values ​​include: joint position, velocity and torque;

[0241] Step B2: Calculating the loss function value of the predicted value and the actual data through the loss function;

[0242] Step B3: Calculate the gradient of the loss function to the network parameters by back propagation algorithm, and update the network parameters by gradient descent method, so as to obtain the optimized dynamic tree model;

[0243] In step B2, the mathematical expression of the loss function is:

[0244]

[0245] Where N represents the number of motion trajectory frames, represents the predicted joint position of the i-th frame, q i Indicates the actual joint position; represents the prediction speed, v i Indicates actual speed; represents the prediction moment, τ i represents the actual torque. α, β and γ are a weight coefficient, another weight coefficient and another weight coefficient, respectively, which are used to balance the proportion of joint position, velocity and torque in the loss function; |·| represents the L2 norm of the vector, that is, the Euclidean distance, and L is the final loss function value. α, β and γ can be adjusted according to the needs of specific tasks.

[0246] In step B3, the mathematical expression of the gradient of the loss function to the network parameters is:

[0247]

[0248] in, Represents the gradient of the loss function with respect to the network parameters, and the superscript T represents the transpose of the matrix or vector; represents the predicted joint position of the i-th frame; q i represents the actual joint position of the i-th frame; represents the predicted speed of the i-th frame; v i Indicates the actual speed of the i-th frame.

[0249] The gradient of the joint position is expressed mathematically as:

[0250]

[0251] in, The gradient representing the joint position;

[0252] The gradient of velocity is expressed mathematically as:

[0253]

[0254] in, represents the gradient of velocity;

[0255] The gradient of the torque is expressed mathematically as:

[0256]

[0257] in, represents the gradient of the moment, and W represents the network parameters.

[0258] Specifically, the main purpose of solving the position gradient, velocity gradient, and torque gradient is to calculate the gradient of the loss function with respect to the network parameters through the chain rule, thereby updating the network parameters and optimizing the model performance.

[0259] In the optimization process of deep learning, the chain rule is the core method for calculating gradients, which requires passing the error of the loss function back to the network parameters step by step. Position, velocity, and torque are the key outputs of model prediction. They are directly related to the loss function, so their gradients need to be calculated first. The gradient acts as a "bridge" to pass the error of the loss function to the network parameters, helping the operator understand the contribution of each parameter to the error.

[0260] Finally, by calculating these gradients, we can use the gradient descent method to update the network parameters and optimize the performance of the model. Therefore, the fundamental purpose of solving these gradients is to find the network parameters and optimize the model.

[0261] The mathematical expression of the network parameter, i.e. the optimization error, is:

[0262]

[0263] Among them, W represents the network parameters, that is, the optimization error; ← represents the assignment operation, that is, updating the variables on the left with the values ​​on the right; Represents partial derivatives, used to calculate gradients; represents the final loss function value; η is the learning rate, which is used to control the step size of parameter update.

[0264] Step A7: Iterate until the optimization error of the kinetic tree model is lower than the preset threshold, indicating that the accuracy of the model has reached the requirement and the optimization process ends.

[0265] Human motion acquisition module, which collects human motion data through RGB cameras and uses computer vision algorithms, such as human posture estimation, to extract the two-dimensional or three-dimensional joint data of the human body from the video frames. The video data captured by the RGB camera will be transmitted to the computer vision algorithm, and the posture and motion information of the human body will be obtained through algorithm processing.

[0266] To ensure data quality, the collected data is further processed by the preprocessing unit, including noise removal, smoothing and standardization, to ensure that the data is suitable for subsequent modeling processing. The preprocessed data will be passed to the human motion modeling module for subsequent operations.

[0267] The human motion modeling module meshes the collected human motion data and extracts SMPL, the Skinned Multi-Person Linear model, to construct a three-dimensional representation of human motion. This module uses a deep learning algorithm to extract the SMPL model based on the preprocessed data through the data modeling unit to accurately represent the various postures and movements of the human body. The extracted SMPL model data will be passed to the motion optimization unit for further processing. The motion optimization unit will optimize the SMPL model, adjust the joint angles and smooth the transition movements, so that the robot can imitate the movements more naturally and smoothly. The optimized data will be passed to the robot power tree modeling module to ensure the accuracy and smoothness of motion control.

[0268] The robot dynamic tree modeling module builds a dynamic tree model suitable for humanoid robot motion based on the SMPL sequence data output by the human motion modeling module, thereby optimizing the robot's motion control algorithm. The dynamic modeling unit of this module builds the dynamic model of the humanoid robot based on the SMPL model sequence, and uses the Newton-Lagrange method to establish the relationship between joint torque and joint motion, ensuring that the robot's motion is highly consistent with the human body's motion.

[0269] However, due to the complexity of humanoid robot design, there are deviations in elastic components, mass and inertial parameters in the mechanical structure, and the inevitable vibration, friction, coupling effects and noise may cause the dynamic calculation of the theoretical model to be inaccurate.

[0270] To solve this problem, the system uses deep learning methods to optimize the dynamic parameters of humanoid robots. The optimization unit uses deep learning training networks to adapt to different humanoid robots and accurately obtain their dynamic parameters, thereby improving motion control accuracy.

[0271] The training data includes information such as joint position, speed and current. The deep learning network is trained based on these data, and the optimized dynamic parameters are finally output. These optimized parameters will be passed to the robot motion control module to update the control strategy. During the training process, in order to generate enough data for deep learning network training, the system uses an excitation trajectory design unit. This unit designs appropriate excitation trajectories according to system requirements to ensure that recognizable dynamic parameters can be generated. With the data generated by these excitation trajectories, the deep learning network can be better trained and optimized. The collected robot joint torque data will be transmitted to the low-pass filter for signal processing to filter out high-frequency noise and burr signals to reduce interference. The signal data processed by the filter will be passed to the optimization unit for further model optimization and training.

[0272] The control module is responsible for generating the robot's control instructions based on the robot's dynamic model and hardware configuration, and executing these instructions through the execution unit, thereby completing the action imitation and ensuring the synchronization of the robot's actions with the target human body's actions. The control module includes multiple functional units.

[0273] The first is the real-time motion planning unit, which plans a reasonable motion trajectory based on the dynamic model, real-time sensor data and task requirements. Motion planning takes into account the accessibility, stability and dynamic constraints of the robot's joints to ensure the accuracy and efficiency of the robot's movements.

[0274] Next is the execution unit, which generates control instructions based on the planned trajectory and executes these instructions through the robot drive system to ensure that the robot accurately performs the action imitation. In order to further improve the robot's motion accuracy, the control module also includes an error compensation unit. This unit monitors and adjusts the deviations that occur during the execution process in real time to ensure that the robot can still perform actions with high precision in complex environments. The error compensation unit adjusts the control strategy through sensor feedback to effectively overcome the impact of external interference or internal structural errors.

[0275] In addition, the control module also integrates an adaptive learning mechanism, which enables the robot to continuously optimize the control strategy during long-term training and adapt to different motion tasks.

[0276] The key innovation of this method lies in the integration of motion optimization unit and deep learning method, which effectively solves the accuracy problem in robot motion control. Especially in the face of robot structural complexity and inconsistent dynamic parameters, the optimized robot dynamics model significantly improves the accuracy of motion control, and the error is reduced to less than 5%.

[0277] The present invention also provides a humanoid robot motion imitation system based on a human body linear model. The humanoid robot motion imitation system based on a human body linear model can be realized by executing the process steps of the humanoid robot motion imitation method based on a human body linear model, that is, those skilled in the art can understand the humanoid robot motion imitation method based on a human body linear model as a preferred implementation of the humanoid robot motion imitation system based on a human body linear model.

[0278] A humanoid robot motion imitation system based on a human linear model provided by the present invention comprises:

[0279] Module M1: collects video input of human movements and extracts movement information frame by frame;

[0280] Module M2: Modeling based on the action information, and obtaining a SMPL representation and a three-dimensional representation of the human body posture;

[0281] Module M3: Based on the SMPL representation and the three-dimensional representation of the human body posture, generate SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion, and construct a dynamic tree model of the robot;

[0282] Module M4: plans the robot motion trajectory through the dynamic tree model and generates corresponding control instructions.

[0283] The modules M1, M2, M3 and M4 correspond to the human motion acquisition module, human motion modeling module, robot power tree modeling module and robot motion control module in the embodiment respectively;

[0284] The robot is a humanoid robot.

[0285] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0286] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0287] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A humanoid robot motion imitation method based on a human linear model, characterized in that: include: Step S1: Collecting video input of human body movements and extracting movement information frame by frame; Step S2: Modeling is performed based on the action information, and a SMPL representation and a three-dimensional representation of the human body posture are obtained; Step S3: Based on the SMPL representation and the three-dimensional representation of the human body posture, SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion are generated to construct a dynamic tree model of the robot; Step S4: planning the robot motion trajectory through the dynamic tree model and generating corresponding control instructions.

2. The humanoid robot motion imitation method based on human linear model according to claim 1, characterized in that: In step S1, the action information is the visual features of the human body shape and the motion features of the changes between frames; the visual features include the human body contour and joint morphology; the motion features include the movement and posture of the human body; The action information is extracted based on a deep neural network.

3. The humanoid robot motion imitation method based on human linear model according to claim 2, characterized in that: In the step S2, the modeling is to construct a Mesh model; In the step S2, based on the motion information, a multi-layer perceptron is used to estimate the human body posture and obtain a Mesh model of the human body; the Mesh model of the human body is mapped to a three-dimensional space through an SMPL model to generate a three-dimensional representation of the human body posture; In step S3, the SMPL sequence data is a sequence consisting of three-dimensional posture parameters output by the SMPL model in each frame and the motion trajectory of each joint; The three-dimensional posture parameters include joint positions, rotation matrices, and shape parameters of the human body; The process of establishing and optimizing the kinetic tree model includes: Step A1: Construct the dynamic tree model of the robot according to the SMPL sequence data output by the human motion modeling module; use the Newton-Lagrange method to obtain the relationship between joint torque and joint motion; Step A2: constructing a complete robot dynamics model based on the relationship between the joint torque and the joint motion and the dynamics tree model of the robot; The complete robot dynamics model includes the physical connection relationship between robot joints, inertia parameters and mass distribution; Step A3: based on the complete robot dynamics model, extract the target joint position from each frame of the SMPL sequence data, and calculate the angle of each joint according to the target joint position, so as to generate a motion trajectory as an excitation trajectory; the target joint position includes the position of the hand and the position of the foot; Step A4: Based on the motion trajectory, collecting the joint position, speed, and torque of the robot as training data; Step A5: low-pass filtering the training data to obtain filtered training data; Step A6: training a deep learning network through filtered training data; the deep learning network outputs a kinetic tree model; optimizing and updating the control strategy based on the parameters of the kinetic tree model, and judging whether the optimization error is less than a preset threshold value, if the result is yes, then the process ends, if the result is no, then re-execute step A3; The control strategy includes: adjusting the torque control parameters of the joint, optimizing the motion trajectory and redefining the rules of feedback control; In the step A1, the dynamic tree model provides the kinetic energy and gravitational potential energy of the robot joints; The mathematical expression of the kinetic energy is: Where T represents the kinetic energy of the robot joints, n represents the total number of robot joints; m i represents the mass of the i-th joint; v i represents the velocity of the i-th joint; The mathematical expression of the gravitational potential energy is: Where V represents the gravitational potential energy of the robot joint; g represents the gravitational acceleration; h i represents the height of the center of gravity of the i-th joint; The relationship between the joint torque and the joint motion is expressed mathematically as follows: Among them, q j represents the generalized coordinates of the jth joint, represents the generalized velocity of the jth joint, τ j is the joint torque of the jth joint; In step A2, the complete robot dynamics model is expressed as follows: in, are the Coriolis and centrifugal force matrices; M(q) represents the mass matrix; G(q) represents the gravity term; q represents the generalized coordinates of the joint, represents the generalized velocity of the joint; represents the generalized acceleration of the joint.

4. The humanoid robot motion imitation method based on human linear model according to claim 3 is characterized in that: In the step A1, it includes: Step A1.1: Take each frame of SMPL sequence data as input data and input it into a deep network composed of multi-layer perceptrons, i.e., a deep neural network; Step A1.2: extracting the features of the input data based on the deep network and reducing the dimension to obtain the first layer of features; mapping the first layer of features into dynamic tree model parameters; the dynamic tree model parameters include: joint mass, inertia matrix, damping coefficient and stiffness coefficient; Step A1.3: outputting a dynamic tree model of the robot based on the dynamic tree model parameters; Step A1.4: Based on the dynamic tree model parameters, the kinetic energy and potential energy of each joint are calculated and obtained; the kinetic energy and potential energy are solved by Lagrange equations to obtain the explicit relationship between joint torque and joint position and the explicit relationship between velocity and acceleration, and then the relationship between joint torque and joint motion is obtained; In the step A3, it includes: Step A3.1: extracting target joint positions from the SMPL posture data of each frame; the target joint positions include the positions of the hands and the feet; Step A3.2: Based on the target joint position, the angles of each joint are obtained, and based on inverse kinematics, a set of joint angle values ​​that satisfy the target joint position are obtained; Step A3.3: Based on the dynamic constraints and kinematic restrictions of each joint, the joint angle values ​​are optimized; the dynamic constraints include: speed restrictions, acceleration restrictions and torque restrictions; the kinematic restrictions include: joint angle range and degrees of freedom; the optimization is to set numerical boundaries for the joint angle values; Step A3.4: Connect the optimized joint angle values ​​into a smooth and continuous trajectory, i.e., the target motion trajectory, by using the cubic spline interpolation method; The smooth and continuous trajectory refers to a trajectory with a smoothness index less than or equal to 0.1 and continuous; The mathematical expression of the smoothness index is: Among them, S represents the smoothness index, t0 represents the starting time of the trajectory; t f represents the end time of the trajectory; θ represents the joint angle; t represents time; In step A6, the process of optimizing the parameters of the kinetic tree model includes: Step B1: deriving predicted values ​​through the dynamics tree model; the predicted values ​​include: joint position, velocity and torque; Step B2: Calculating the loss function value of the predicted value and the actual data through the loss function; Step B3: Calculate the gradient of the loss function to the network parameters by back propagation algorithm, and update the network parameters by gradient descent method, so as to obtain the optimized dynamic tree model; In step B2, the mathematical expression of the loss function is: Where N represents the number of motion trajectory frames, represents the predicted joint position of the i-th frame, q i Indicates the actual joint position; represents the prediction speed, v i Indicates actual speed; represents the prediction moment, τ i represents the actual torque. α, β and γ are a weight coefficient, another weight coefficient and another weight coefficient respectively; || represents the L2 norm of the vector, that is, the Euclidean distance, Represents the final loss function value; In step B3, the mathematical expression of the gradient of the loss function to the network parameters is: in, Represents the gradient of the loss function with respect to the network parameters, and the superscript T represents the transpose of the matrix or vector; represents the predicted joint position of the i-th frame; q i represents the actual joint position of the i-th frame; represents the predicted speed of the i-th frame; v i Indicates the actual speed of the i-th frame. The gradient of the joint position is expressed mathematically as: in, The gradient representing the joint position; The gradient of velocity is expressed mathematically as: in, represents the gradient of velocity; The gradient of the torque is expressed mathematically as: in, represents the gradient of the moment, W represents the network parameter; The mathematical expression of the network parameter, i.e. the optimization error, is: Among them, W represents the network parameters, that is, the optimization error, ← represents the assignment operation, represents the partial derivative, Represents the final loss function value, and η represents the learning rate.

5. A humanoid robot motion imitation system based on a human linear model, characterized in that: include: Module M1: collects video input of human movements and extracts movement information frame by frame; Module M2: Modeling based on the action information, and obtaining a SMPL representation and a three-dimensional representation of the human body posture; Module M3: Based on the SMPL representation and the three-dimensional representation of the human body posture, generate SMPL sequence data in the time dimension and the three-dimensional representation of the human body motion, and construct a dynamic tree model of the robot; Module M4: plans the robot motion trajectory through the dynamic tree model and generates corresponding control instructions.

6. The humanoid robot motion imitation system based on human linear model according to claim 5, characterized in that: In the module M1, the action information is the visual features of the human body shape and the motion features of the changes between frames; the visual features include the human body contour and joint morphology; the motion features include the movement and posture of the human body; The action information is extracted based on a deep neural network.

7. The humanoid robot motion imitation system based on human linear model according to claim 6, characterized in that: In the module M2, the modeling is to construct a Mesh model; In the module M2, based on the motion information, a multi-layer perceptron is used to estimate the human body posture and obtain a Mesh model of the human body; the Mesh model of the human body is mapped to a three-dimensional space through an SMPL model to generate a three-dimensional representation of the human body posture; In the module M3, the SMPL sequence data is a sequence consisting of three-dimensional posture parameters output by the SMPL model in each frame and the motion trajectory of each joint; The three-dimensional posture parameters include joint positions, rotation matrices, and shape parameters of the human body; The establishment and optimization of the kinetic tree model includes: Module A1: Construct the dynamic tree model of the robot based on the SMPL sequence data output by the human motion modeling module; use the Newton-Lagrange method to obtain the relationship between joint torque and joint motion; Module A2: Based on the relationship between the joint torque and joint motion and the dynamics tree model of the robot, a complete robot dynamics model is constructed; The complete robot dynamics model includes the physical connection relationship between robot joints, inertia parameters and mass distribution; Module A3: Based on the complete robot dynamics model, extract the target joint position from the SMPL sequence data of each frame, and calculate the angle of each joint according to the target joint position, so as to generate a motion trajectory as an excitation trajectory; the target joint position includes the position of the hand and the position of the foot; Module A4: Based on the motion trajectory, collecting the joint position, speed and torque of the robot as training data; Module A5: performing low-pass filtering on the training data to obtain filtered training data; Module A6: train the deep learning network through the filtered training data; the deep learning network outputs a kinetic tree model; optimize and update the control strategy based on the parameters of the kinetic tree model, and judge whether the optimization error is less than a preset threshold. If the result is yes, end; if the result is no, re-trigger the module A3 to work; The control strategy includes: adjusting the torque control parameters of the joint, optimizing the motion trajectory and redefining the rules of feedback control; In the module A1, the dynamic tree model provides the kinetic energy and gravitational potential energy of the robot joints; The mathematical expression of the kinetic energy is: Where T represents the kinetic energy of the robot joints, n represents the total number of robot joints; m i represents the mass of the i-th joint; v i represents the velocity of the i-th joint; The mathematical expression of the gravitational potential energy is: Where V represents the gravitational potential energy of the robot joint; g represents the gravitational acceleration; h i represents the height of the center of gravity of the i-th joint; The relationship between the joint torque and the joint motion is expressed mathematically as follows: Among them, q j represents the generalized coordinates of the jth joint, represents the generalized velocity of the jth joint, τ j is the joint torque of the jth joint; In the module A2, the complete robot dynamics model is expressed as follows: in, are the Coriolis and centrifugal force matrices; M(q) represents the mass matrix; G(q) represents the gravity term; q represents the generalized coordinates of the joint, represents the generalized velocity of the joint; represents the generalized acceleration of the joint.

8. The humanoid robot motion imitation system based on human linear model according to claim 7, characterized in that: The module A1 includes: Module A1.1: Take each frame of SMPL sequence data as input data and input it into a deep network composed of multi-layer perceptrons, i.e., a deep neural network; Module A1.2: extracting the features of the input data based on the deep network and reducing the dimension to obtain the first layer of features; mapping the first layer of features into dynamic tree model parameters; the dynamic tree model parameters include: joint mass, inertia matrix, damping coefficient and stiffness coefficient; Module A1.3: outputting a dynamic tree model of the robot based on the dynamic tree model parameters; Module A1.4: Based on the dynamic tree model parameters, the kinetic energy and potential energy of each joint are calculated and obtained; the kinetic energy and potential energy are solved by Lagrange equations to obtain the explicit relationship between joint torque and joint position and the explicit relationship between velocity and acceleration, and then the relationship between joint torque and joint motion is obtained; The module A3 includes: Module A3.1: Extracting target joint positions from the SMPL posture data of each frame; the target joint positions include the positions of the hands and the feet; Module A3.2: Based on the target joint position, obtain the angle of each joint, and obtain a set of joint angle values ​​that meet the target joint position based on inverse kinematics; Module A3.3: Optimizing the joint angle value based on the dynamic constraints and kinematic restrictions of each joint; the dynamic constraints include: speed restriction, acceleration restriction and torque restriction; the kinematic restrictions include: joint angle range and degree of freedom; the optimization is to set numerical boundaries for the joint angle value; Module A3.4: Connect the optimized joint angle values ​​into a smooth and continuous trajectory, i.e., the target motion trajectory, through cubic spline interpolation; The smooth and continuous trajectory refers to a trajectory with a smoothness index less than or equal to 0.1 and continuous; The mathematical expression of the smoothness index is: Among them, S represents the smoothness index, t0 represents the starting time of the trajectory; t f represents the end time of the trajectory; θ represents the joint angle; t represents time; In the module A6, optimizing the parameters of the kinetic tree model includes: Module B1: deriving predicted values ​​through the dynamic tree model; the predicted values ​​include: joint position, velocity and torque; Module B2: Calculate the loss function value of the predicted value and the actual data through the loss function; Module B3: Calculate the gradient of the loss function to the network parameters through the back propagation algorithm, and update the network parameters through the gradient descent method to obtain the optimized dynamic tree model; In the module B2, the mathematical expression of the loss function is: Where N represents the number of motion trajectory frames, represents the predicted joint position of the i-th frame, q i Indicates the actual joint position; represents the prediction speed, v i Indicates actual speed; represents the prediction moment, τ i represents the actual torque. α, β and γ are a weight coefficient, another weight coefficient and another weight coefficient respectively; || represents the L2 norm of the vector, that is, the Euclidean distance, Represents the final loss function value; In the module B3, the mathematical expression of the gradient of the loss function to the network parameters is: in, Represents the gradient of the loss function with respect to the network parameters, and the superscript T represents the transpose of the matrix or vector; represents the predicted joint position of the i-th frame; q i represents the actual joint position of the i-th frame; represents the predicted speed of the i-th frame; v i Indicates the actual speed of the i-th frame. The gradient of the joint position is expressed mathematically as: in, The gradient representing the joint position; The gradient of velocity is expressed mathematically as: in, represents the gradient of velocity; The gradient of the torque is expressed mathematically as: in, represents the gradient of the moment, W represents the network parameter; The mathematical expression of the network parameter, i.e. the optimization error, is: Among them, W represents the network parameters, that is, the optimization error, ← represents the assignment operation, represents the partial derivative, Represents the final loss function value, and η represents the learning rate.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for imitating humanoid robot motion based on a human linear model according to any one of claims 1 to 4 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method for imitating humanoid robot motion based on a human linear model according to any one of claims 1 to 4 are implemented.

Images