Humanoid robot motion imitation method and device based on 3D human posture estimation

Abstract

The present invention discloses a method and device for imitating humanoid robot motion based on 3D human posture estimation. The method comprises: data acquisition and preprocessing: using a color-depth camera to acquire color images and depth images, and obtaining human posture point cloud data of a demonstrator in a world coordinate system through data preprocessing; human limb motion capture: according to the human posture point cloud data, the three-dimensional coordinates of human joints are captured based on a 3D human posture estimation network; motion behavior mapping: based on coordinate transformation and space vector calculation, the obtained three-dimensional coordinates of human joints are converted into joint angles of both arms of a humanoid robot; mapping joint angle filtering: smoothing the joint angles through Kalman filtering, and limiting the output of the joint angles according to the range of motion of each joint of the robot, so as to realize the humanoid robot's imitation of the demonstrator's arm motion.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular to a method and device for imitating humanoid robot movements based on 3D human body posture estimation. Background Art

[0002] The promotion of robot applications has liberated people from monotonous and repetitive work, greatly improved productivity and production quality, and therefore has been increasingly widely used in various fields such as industrial production and social life. However, most robots are deployed in structured working environments to complete repetitive routine work according to pre-set programs. With the diversification of application scenarios, the working environment of robots is becoming more and more complex. Therefore, the manually designed motion strategies for a single task cannot adapt to a large number of flexible and complex working scenarios, which will limit the further development of robot technology. Robot motion imitation gives robots the ability to operate flexibly in complex scenarios and tasks, reduces the requirements for robot operators and the difficulty of robot deployment, and is of great significance to the realization of robot intelligence and autonomy.

[0003] Robot motion imitation includes two parts: human limb motion capture and motion behavior mapping, among which motion capture is the basis of motion imitation. Common human limb motion capture relies on optical or inertial motion capture devices. The data captured by this wearable sensor is very accurate, but it is expensive and has very high requirements for the experimental environment, resulting in very limited application scenarios. The Microsoft Kinect sensor uses computer vision technology to obtain human skeleton data with depth information. However, Kinect skeleton tracking technology is integrated in the Kinect camera and does not support the use of other visual sensors. 3D human pose estimation technology can capture the spatial position information of key points of the human body, but most 3D human pose estimation methods rely on the results of 2D pose estimation. Since 2D to 3D mapping is an ill-posed problem, the prediction process is usually complex and the accuracy is not high. Many methods use multi-view image information fusion to predict the 3D pose of the human body. However, fusing image information from multiple perspectives is also a challenge.

[0004] For the problem of action behavior mapping, most methods use inverse kinematics to solve the problem based on the position information of the robot's end effector to obtain the angle information of each degree of freedom of the robot. However, this method has a large amount of calculation and a complex calculation process, which may affect the similarity of action imitation. The method of solving using space vectors can directly calculate the mapping joint angle. This method is simple to calculate and can achieve imitation actions with high similarity. However, at present, this type of method lacks accurate analysis of the relationship between the human skeletal structure and the corresponding degrees of freedom of the robot, so there is still a certain gap in achieving the similarity and accuracy of robot action imitation. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for humanoid robot motion imitation based on 3D human posture estimation in order to address the deficiencies of the prior art.

[0006] According to a first aspect of an embodiment of the present application, a method for imitating humanoid robot motion based on 3D human posture estimation is provided, comprising:

[0007] (1) Data acquisition and preprocessing: Use a color-depth camera to acquire color images and depth images, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing;

[0008] (2) Human body motion capture: Based on the human body posture point cloud data obtained in step (1), the three-dimensional coordinates of human body joints are captured based on the 3D human body posture estimation network;

[0009] (3) Action behavior mapping: Based on coordinate transformation and space vector calculation, the three-dimensional coordinates of the human joints predicted in step (2) are converted into the joint angles of the humanoid robot’s arms;

[0010] (4) Mapping joint angle filtering: The robot joint angles obtained in step (3) are smoothed by Kalman filtering, and the joint angles are restricted and output according to the range of motion of each joint of the robot, so that the humanoid robot can imitate the arm movements of the instructor.

[0011] Furthermore, step (1) includes the following sub-steps:

[0012] (1.1) using a color-depth camera to collect a color image and a depth image, and matching the color image with the depth image to generate color point cloud data;

[0013] (1.2) performing point cloud filtering and clustering on the color point cloud data to generate human body posture point cloud data;

[0014] (1.3) Using data enhancement methods to process the human body point cloud data.

[0015] Furthermore, step (2) includes the following sub-steps:

[0016] (2.1) quantizing the human body posture point cloud data;

[0017] (2.2) building a 3D human pose estimation network, wherein the 3D human pose estimation network includes a mask network and a pose estimation network;

[0018] (2.3) After training the mask network until convergence, loading the trained mask network and setting it to inference state, and training the posture estimation network according to the mask features generated by the trained mask network;

[0019] (2.4) Based on the human body posture point cloud data quantized in step (2.1), the trained 3D human body posture estimation network is used to capture the three-dimensional coordinates of human body joints.

[0020] Furthermore, the 3D human posture estimation network includes:

[0021] The mask network is used to roughly search the point cloud around the human joints. It uses the point cloud data within a certain range around the true value of the joint as the supervision signal to generate mask features indicating the importance of the point cloud.

[0022] A posture estimation network is used to prune the point cloud according to the mask features transmitted by the mask network, and generate a three-dimensional heat map representing the probability distribution of joints, and predict the precise position of each joint by integrating and regressing the three dimensions separately;

[0023] The mask network and the pose estimation network are both symmetric encoder-decoder structures composed of convolution and transposed convolution, and share the same pruning layer.

[0024] Furthermore, step (3) includes the following sub-steps:

[0025] (3.1) By establishing the robot's left and right arm spatial coordinate systems and constructing human joint vectors, according to the corresponding relationship between the human joint vectors and the humanoid robot's double-arm structure, the three-dimensional coordinates of the human joints are converted into the robot's left and right arm spatial coordinate systems;

[0026] (3.2) Using the three-dimensional coordinates of the human joints in the spatial coordinate system of the robot's left and right arms, the upper arm vector and the lower arm vector are solved respectively, and then the three-dimensional coordinates of the human joints in the spatial coordinate system of the robot's left and right arms are mapped to the joint angles of the robot's left and right arms, including the shoulder joint pitch angle, shoulder joint roll angle, elbow joint yaw angle and elbow joint roll angle of both arms.

[0027] Furthermore, in step (3.2), taking the left arm as an example, the process of obtaining the shoulder joint pitch angle, shoulder joint roll angle, elbow joint yaw angle and elbow joint roll angle of the left arm includes:

[0028] (3.2.1) Using the three-dimensional coordinates of the left shoulder, left elbow, and left wrist in the robot's left arm spatial coordinate system, obtain the spatial vectors representing the left upper arm and left lower arm respectively;

[0029] (3.2.2) According to the projection relationship of the left upper arm vector on the three coordinate axes, the pitch angle and roll angle of the left shoulder joint are calculated;

[0030] (3.2.3) By rotating the left arm spatial coordinate system, the vector representation of the left lower arm vector in the new coordinate system after the shoulder rotation is obtained;

[0031] (3.2.4) Based on the projection relationship of the left lower arm vector on the three coordinate axes after rotation, the yaw angle and roll angle of the left elbow joint are calculated.

[0032] Furthermore, step (4) includes the following sub-steps:

[0033] (4.1) performing Kalman filtering on the joint angles of the humanoid robot arms calculated in step (3) to avoid violent shaking of the joints during movement and achieve smooth joint movement;

[0034] (4.2) The joint angles are limited in range according to the range of motion of each joint of the robot, and the joint angles that meet the range limits are output to the humanoid robot so that the humanoid robot can imitate the arm movements of the instructor.

[0035] According to a second aspect of an embodiment of the present application, there is provided a humanoid robot motion imitation device based on 3D human posture estimation, comprising:

[0036] The data acquisition and preprocessing module is used to acquire color images and depth images using a color-depth camera, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing;

[0037] The human body motion capture module is used to capture the three-dimensional coordinates of human joints based on the 3D human posture estimation network according to the human posture point cloud data obtained by the data acquisition and preprocessing module;

[0038] The action behavior mapping module is used to convert the three-dimensional coordinates of the human joints predicted by the human limb motion capture module into the joint angles of the humanoid robot's arms based on coordinate conversion and space vector calculation;

[0039] The mapping joint angle filtering module is used to smooth the robot joint angles obtained in the action behavior mapping module through Kalman filtering, and to limit the output of the joint angles according to the range of motion of each joint of the robot, so that the humanoid robot can imitate the arm movements of the instructor.

[0040] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0041] one or more processors;

[0042] A memory for storing one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0044] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0045] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0046] It can be seen from the above embodiments that the present application solves the calculation and storage problems of color-depth (RGB-D) data as input by building a 3D human posture estimation network, improves the accuracy of 3D human posture estimation, and realizes a universal human limb motion capture technology. This method supports the use of any RGB-D sensor and is both economical and scalable. The motion mapping method proposed in the present invention eliminates the process of presetting motion instructions for the robot, greatly reduces the requirements for programming expertise of the robot operator and the difficulty of programming and deploying the robot, thereby improving the flexibility and diversity of the robot's movement. The humanoid robot motion imitation method proposed in the present invention has the advantages of high imitation similarity and high calculation accuracy.

[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] Figure 1 The present invention is a flowchart of a method for imitating humanoid robot motion based on 3D human posture estimation according to an exemplary embodiment.

[0050] Figure 2 is a schematic diagram of a 3D human posture estimation network according to an exemplary embodiment.

[0051] Figure 3 is a schematic diagram of the coordinate system of the left and right arms of a humanoid robot according to an exemplary embodiment.

[0052] Figure 4 The figure is a schematic diagram showing human body joints according to an exemplary embodiment.

[0053] Figure 5The diagram is a schematic diagram of the joint degrees of freedom of the left and right arms of a humanoid robot according to an exemplary embodiment, wherein (a) is the joint degrees of freedom of the left arm of the humanoid robot, and (b) is the joint degrees of freedom of the right arm of the humanoid robot.

[0054] Figure 6 The present invention is a schematic diagram of calculating the left shoulder joint angle of a humanoid robot in a robot coordinate system according to an exemplary embodiment, wherein (a) is a schematic diagram of calculating the pitch angle of the left shoulder joint of the humanoid robot in the robot coordinate system, and (b) is a schematic diagram of calculating the roll angle of the left shoulder joint of the humanoid robot in the robot coordinate system.

[0055] Figure 7 The present invention is a schematic diagram of calculating the left elbow joint angle of a humanoid robot in a robot coordinate system according to an exemplary embodiment, wherein (a) is a schematic diagram of calculating the yaw angle of the left elbow joint of the humanoid robot in the robot coordinate system, and (b) is a schematic diagram of calculating the roll angle of the left elbow joint of the humanoid robot in the robot coordinate system.

[0056] Figure 8 The figure is a schematic diagram showing the interactive relationship among a color-depth camera, a PC and a humanoid robot according to an exemplary embodiment.

[0057] Fig. 9 The present invention is a block diagram of a humanoid robot motion imitation device based on 3D human posture estimation according to an exemplary embodiment.

[0058] Fig.10 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0060] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0061] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] The core technology of the present invention is to realize human limb motion capture based on a 3D human posture estimation method and thereby realize motion imitation on a humanoid robot arm, so as to realize a human joint capture technology applicable to all RGB-D sensors, while reducing the difficulty of robot deployment and programming, enabling the robot to have the ability to imitate human motion, and realizing the similarity, fluency and stability of humanoid robot motion imitation.

[0063] Based on this, the present application provides a humanoid robot motion imitation method based on 3D human posture estimation, such as Figure 1 As shown, after the method is deployed on a PC and the operation is completed, the application display is realized on a humanoid robot, which may include the following steps:

[0064] (1) Data acquisition and preprocessing: Use a color-depth camera to acquire color images and depth images, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing;

[0065] (2) Human body motion capture: Based on the human body posture point cloud data obtained in step (1), the three-dimensional coordinates of human body joints are captured based on the 3D human body posture estimation network;

[0066] (3) Action behavior mapping: Based on coordinate transformation and space vector calculation, the three-dimensional coordinates of the human joints predicted in step (2) are converted into the joint angles of the humanoid robot’s arms;

[0067] (4) Mapping joint angle filtering: The robot joint angles obtained in step (3) are smoothed by Kalman filtering, and the joint angles are restricted and output according to the range of motion of each joint of the robot, so that the humanoid robot can imitate the arm movements of the instructor.

[0068] It can be seen from the above embodiments that the present application solves the calculation and storage problems of RGB-D data as input by building a 3D human posture estimation network, improves the accuracy of 3D human posture estimation, and realizes a universal human limb motion capture technology. This method supports the use of any RGB-D sensor and is both economical and scalable. The motion mapping method proposed in the present invention eliminates the process of presetting motion instructions for the robot, greatly reduces the requirements for the programming expertise of the robot operator and the difficulty of programming and deploying the robot, thereby improving the flexibility and diversity of the robot's movement. The humanoid robot motion imitation method proposed in the present invention has the advantages of high imitation similarity and high calculation accuracy.

[0069] In the specific implementation of step (1), data collection and preprocessing: using a color-depth camera to collect color images and depth images, and obtaining the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing;

[0070] It includes the following sub-steps:

[0071] (1.1) using a color-depth camera to collect a color image and a depth image, and matching the color image with the depth image to generate color point cloud data;

[0072] According to the camera's intrinsic parameter matrix K, the change from the camera coordinate system to the pixel coordinate system is as follows:

[0073]

[0074] Where (u, v) and (x, y, z) represent the coordinates in the pixel coordinate system and the camera coordinate system respectively, and f x ,f y ,u0,v0 represent the horizontal and vertical focal lengths and the horizontal and vertical optical centers respectively.

[0075] Therefore, according to the depth value at the uth row and vth column in the depth image, the depth pixel can be converted from the depth image 2D coordinate system to the depth camera 3D coordinate system, which can be expressed as:

[0076]

[0077] From the above, we can calculate the 3D coordinates of the point cloud in the camera coordinate system. Then, we can obtain the color point cloud information by matching it with the color image. Its definition is as follows:

[0078] PCD={Point i (x i ,y i ,z i ,r i ,g i ,b i):i=1,…,n}

[0079] Where (x i ,y i ,z i ) is the 3D coordinate, (r i ,g i ,b i ) is the color value in the RGB system, n is the total number of point clouds, Point i Represents the i-th point cloud.

[0080] In order to align the color image with the depth image, we first need to convert the point cloud coordinates (x, y, z) into the color camera coordinate system, which can be expressed as:

[0081]

[0082] Where (x RGB ,y RGB ,z RGB ) represents the three-dimensional coordinate representation of the point cloud coordinates under the color camera, and R and T represent the rotation matrix and translation matrix between the depth camera and the color camera respectively. Next, using the intrinsic parameters of the color camera, it is mapped to the color image coordinate system to obtain its corresponding pixel index, as shown below:

[0083]

[0084] Among them, (u RGB ,v RGB ) represents the coordinates in the color camera pixel coordinate system, are the internal parameters of the color camera, representing the focal length and optical center in the horizontal and vertical directions respectively.

[0085] Therefore, according to the color image pixel (u RGB ,v RGB ) is used to read the RGB value of the corresponding point cloud, and combined with the 3D coordinates (x, y, z) of the point cloud in the camera coordinate system, the color point cloud information Point in the camera coordinate system can be restored. i (x i ,y i ,z i ,r i ,g i ,b i ).

[0086] The above steps realize the conversion of point cloud information from pixel coordinates to camera coordinates, and restore the color point cloud information Point in the camera coordinate system. iThen, the 3D coordinates (x, y, z) of the point cloud in the camera coordinate system are transformed from the camera coordinate system to the world coordinate system, and the point cloud coordinates in the world coordinate system are obtained as follows:

[0087]

[0088] Where R′ and T′ are the rotation matrix and translation matrix from the camera coordinate system to the world coordinate system, respectively.

[0089] Finally, we get the color point cloud information Point′ in the world coordinate system. i (x′ i ,y′ i ,z′ i ,r i ,g i ,b i ).

[0090] (1.2) performing point cloud filtering and clustering on the color point cloud data to generate human body posture point cloud data;

[0091] Since the original image contains a lot of information that is not related to the human body, this will increase the amount of calculation and make it difficult for the model to converge. Based on this, it is necessary to crop the original image by restricting coordinates for some areas with obvious coordinate distinction, such as the ground and wall areas. For point clouds distributed around the human body that are difficult to filter out by restricting coordinates, point clouds are clustered according to point density and the distance between neighboring points, that is, points within a certain distance range are considered for density calculation and the minimum number of points required to form a category is determined. Each point in the original point cloud will get a category label, and the category representing the person should have the largest number of point clouds. Finally, you only need to extract all point cloud sets representing the human body to obtain point cloud data about human posture information.

[0092] (1.3) Processing the human body posture point cloud data using a data enhancement method;

[0093] In order to enhance the robustness and generalization ability of the model, the data enhancement processing of the human body posture point cloud data obtained by (1.2) is completed through several geometric transformations, including rotation, folding, scaling and random occlusion. The rotation is based on the human pelvis as the rotation center, and the Euler angles α, β, γ) are randomly generated to complete the rotation transformation based on the rotation vector (axis angle). Folding is to perform a 180-degree folding transformation on the three coordinates with the pelvis as the center. Scaling is to achieve the scale scaling of the human body point cloud with the pelvis as the center to obtain the scaled human body representation. By randomly occluding part of the data, the diversity of the data and the robustness of the model can be enhanced, so that the model can successfully predict the position of the joint points even when the human body is partially occluded.

[0094] According to the above data preprocessing method, the data form suitable for model input is obtained.

[0095] In the specific implementation of step (2), human body motion capture: according to the human body posture point cloud data obtained in step (1), the three-dimensional coordinates of human body joints are captured based on the 3D human body posture estimation network;

[0096] It includes the following sub-steps:

[0097] (2.1) Quantify the human body posture point cloud data.

[0098] For 3D data, its dense representation is inefficient because the effective information only occupies a small part of the space. Therefore, the present invention hopes to save the information in the non-empty area of ​​the space and discard those areas that do not contain information. This representation of sparse data is called a sparse tensor. Minkowski Engine (Choy C, Gwak JY, Savarese S. 4Dspatio-temporal convnets: Minkowski convolutional neural networks [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 3075-3084.) is a self-differentiation library based on sparse tensors. By using MinkowskiEngine, the input three-dimensional point cloud data is quantized to obtain a sparse tensor to solve the problem of time-consuming and memory-occupied 3D data calculation. The introduction of sparse tensors and sparse tensor networks helps to expand the 3D human posture estimation network to a greater depth and can also better handle scenes with huge amounts of data.

[0099] After quantization, the three-dimensional point cloud data can be represented as a sparse tensor It consists of two parts: coordinate matrix and the feature matrix Its representation is as follows:

[0100]

[0101] Where C records the coordinates of non-empty elements, (x i ,y i ,z i ) represents the coordinates of the point cloud, b i Indicates which batch the point cloud coordinates belong to. The feature matrix F records the eigenvalues ​​of the elements in the corresponding non-empty coordinate matrix C. i Trepresents the RGB feature value of the point cloud. N represents the number of non-empty elements, D represents the spatial dimension, and N F The number of channels representing the features.

[0102] (2.2) Build a 3D human pose estimation network, which includes a mask network and a pose estimation network. The network structure diagram is as follows: Figure 2 shown.

[0103] Specifically, the mask network is used to roughly search the point cloud around the human joints, and generate mask features representing the importance of the point cloud by taking the point cloud data within a certain range around the true value of the joint as the supervision signal; the posture estimation network is used to prune the point cloud according to the mask features transmitted by the mask network, and generate a three-dimensional heat map representing the probability distribution of the joint, and predict the exact position of each joint by integrating and regressing the three dimensions respectively; the mask network and the posture estimation network are both symmetric encoder-decoder structures composed of convolution and transposed convolution. Among them, the encoder first downsamples through convolution to achieve a reduction in resolution, and the decoder then upsamples through transposed convolution to achieve an increase in resolution. The encoder part consists of 6 convolution layers, each of which consists of a convolution with a convolution kernel size of 3*3*3 and a channel number of 32, 64, 128, 256, 512, 1024, followed by batch normalization and RELU activation function. When the lowest resolution is reached, the decoder restores the resolution through 6 transposed convolution layers. Each layer of transposed convolution consists of a generative transposed convolution with a kernel size of 2*2*2 and output channels of 512, 256, 128, 64, 32, and 1, batch normalization, and RELU activation function. In the process of upsampling and downsampling, the sparse tensors with the same number of channels in the encoder and decoder are added with element-wise addition through the skip-layer connection structure, which is conducive to achieving a more refined prediction effect. It can be expressed as follows:

[0104]

[0105] in, represents the sparse tensor output by the encoder, represents the sparse tensor output by the decoder, Represents the final output of layer l.

[0106] As the number of transposed convolution layers in the decoder increases, the dimension in the sparse tensor will grow exponentially, which will lead to overflow of video memory and increase of computational cost. However, in this process, not all voxels are meaningful for the final joint prediction result. It is generally believed that the probability of voxels far away from the true value of the joint should be very low in the heat map, and the contribution to regression is very small. Therefore, the present invention removes these points far away from the true value of the joint by adding a pruning layer after each transposed convolution layer, thereby deleting unnecessary sparse voxels while increasing the spatial resolution, thereby achieving compression of memory and computation.

[0107] The pruning layer defines a function that represents the probability of point cloud features. s (·, then all point clouds with feature probabilities less than a certain value are cropped, and the calculation method can be expressed as follows:

[0108]

[0109]

[0110] Among them, p represents the probability value of the point cloud, is the input sparse tensor, W p represents the spatial weight matrix, Represents the output sparse tensor after pruning the point cloud whose feature probability is less than a certain threshold τ.

[0111] The mask network is used to predict the probability of point cloud features at different resolutions, providing an effective basis for the pruning layer. Its mask loss function represents the target mask of each layer * The binary cross entropy loss between the predicted mask M can be expressed as:

[0112] L MASK =BCE(Sigmoid(M), * )

[0113] The prediction mask M is obtained by adding a convolution kernel of size 1*1*1 after each transposed convolution layer, and the number of output channels is the convolution of the output dimension of the corresponding layer. * Covering those point clouds whose distance to the true value of the joint is less than a certain range, it can be expressed as:

[0114]

[0115]

[0116] in represents the coordinates of the kth joint truth point, v is the center coordinate of the point cloud, l v represents the minimum distance from v to all joint true values, τ MRepresents the mask length, which is a hyperparameter where the distance to the joint truth value is less than τ M The point cloud of , whose target mask feature value is 1.

[0117] The pose estimation network prunes the sparse tensor according to the eigenvalues ​​of the predicted mask to solve the problem of exponential growth of data caused by upsampling, which can be expressed as:

[0118]

[0119] in, represents the output of the lth layer after upsampling, v represents the predicted mask feature probability value predicted by the mask network, τ P Represents the pruning threshold.

[0120] Finally, a 3D heat map representing the probability distribution of k joints is generated by a convolution kernel with a size of 1*1*1 and a number of channels equal to the number of joints. k (p), H k (p) represents the probability that the point is the kth joint. The predicted joint coordinates are obtained by normalizing and integrating the generated 3D heat map, which can be expressed as:

[0121]

[0122]

[0123] in, represents the normalized three-dimensional heat map, Ω is its domain, and p represents all positions in the domain. k Represents the predicted joint coordinates, which are obtained by integrating the three dimensions separately. Its discrete form is expressed as follows:

[0124]

[0125] Where D,,represent the resolution of the heat map in three dimensions.

[0126] The loss function of the pose estimation network consists of two parts: joint loss and heat map loss. The joint loss is defined as the predicted joint coordinates k The joint coordinates with the true value The L1 distance between . The heatmap loss is defined as the predicted heatmap and true value heat map The mean square error loss between . It can be expressed as:

[0127]

[0128]

[0129] L comb =αL joint +(1)L heatmap

[0130] Where α is a hyperparameter. The true value heat map is composed of the true value joint For expectations, 2 is the variance (σ 2 The three-dimensional Gaussian distribution function of (where is the hyperparameter) can be expressed as:

[0131]

[0132] (2.3) After training the mask network until convergence, loading the trained mask network and setting it to inference state, and training the posture estimation network according to the mask features generated by the trained mask network;

[0133] Specifically, the prediction results of the mask network provide a basis for the point cloud pruning of the pose estimation network. The pose estimation network prunes the point cloud according to the predicted mask of the mask network and generates a 3D heat map representing the joint position. During the training process, the model is trained in two steps, that is, the mask network is trained first, and after it converges, its model parameters are saved and then the pose estimation network is trained. When the pose estimation network is trained, the trained mask network model is loaded and set to the inference state. The training datasets are public datasets used for pose estimation evaluation, including the CMU Panoptic dataset and the Berkeley MHAD dataset.

[0134] (2.4) Based on the human body point cloud posture data quantized in step (2.1), the trained 3D human body posture estimation network is used to capture the three-dimensional coordinates of human body joints;

[0135] Specifically, this step is a conventional technical means of performing reasoning using a trained network, which will not be described in detail here.

[0136] In the specific implementation of step (3), the action behavior mapping: based on coordinate transformation and space vector calculation, the three-dimensional coordinates of the human joints predicted in step (2) are converted into the joint angles of the humanoid robot's arms;

[0137] It includes the following sub-steps:

[0138] (3.1) By establishing the robot's left and right arm spatial coordinate systems and constructing human joint vectors, according to the corresponding relationship between the human joint vectors and the humanoid robot's double-arm structure, the three-dimensional coordinates of the human joints are converted into the robot's left and right arm spatial coordinate systems;

[0139] Specifically, based on the seven joint coordinates of the left shoulder, left elbow, left wrist, right hand, right shoulder, right wrist, and pelvis extracted in step (2), a human joint vector is constructed, and according to the correspondence between the human joint vector and the humanoid robot's double-arm structure, the three-dimensional coordinates of the human joint are mapped to the robot's left and right arm coordinate system. Since human arms and humanoid robot arms differ in connection length, joint degrees of freedom, and spatial coordinates, the three-dimensional coordinates of the joints obtained by 3D posture estimation cannot be directly transferred to the robot. Figure 3 The space coordinate system of the left and right arms of the humanoid robot (including the space coordinate system X L -Y L -Z L and the robot right arm space coordinate system X R -Y R -Z R ), whose origins are located at the left and right shoulders of the robot, the x-axis points to the front of the robot, the y-axis is parallel to the vector from the right shoulder to the left shoulder, and the z-axis is orthogonal to the plane formed by the x-axis and y-axis. Figure 4 It is a vector representation of the human limb joints. In the figure, 1 to 7 represent the coordinates of the left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, and hip respectively. The spatial transformation relationship from the world coordinate system to the left and right arm coordinate systems of the humanoid robot can be expressed as:

[0140] P i ′ =T·(P i -P1),i=1,2,3

[0141] P′ i =T·(P i -P4),i=4,5,6

[0142] Where P i (i∈1~7) represents the coordinates of each joint in the world coordinate system, P′ i (i∈1~7) represents the coordinates of each joint in the left and right arm coordinate system of the humanoid robot. The left and right arm coordinates of the human body are converted to the left and right arm coordinate systems of the robot. T represents the transformation matrix from the world coordinate system to the left and right arm coordinate systems of the humanoid robot, which can be expressed by unit vectors in three directions:

[0143]

[0144] The present invention uses the vector triangle composed of the left shoulder P1, the right shoulder P4 and the hip 7 to solve the space mapping coordinate transformation matrix T, where Perpendicular to the plane enclosed by the triangle, parallel to the vector from the right shoulder to the left shoulder, and and The plane orthogonality of the formed plane can be calculated as follows:

[0145]

[0146]

[0147]

[0148] (3.2) Using the three-dimensional coordinates of the human joints in the space coordinate system of the robot's left and right arms, the upper arm vector and the lower arm vector are solved respectively. Then, according to the three-dimensional coordinates of the human joints in the space coordinate system of the robot's left and right arms, the joint angles of the robot's left and right arms are calculated, including the shoulder joint pitch angle, shoulder joint roll angle, elbow joint yaw angle, and elbow joint roll angle of both arms, as shown in Figure 5 shown.

[0149] Since the two arms of the humanoid robot are bilaterally symmetrical in structure, the present invention takes the left arm as an example to solve the mapping angle. The vectors representing the left upper arm and left lower arm respectively can be expressed as:

[0150]

[0151]

[0152] like Figure 6 As shown in (a), the pitch angle of the left shoulder joint It can be expressed as:

[0153]

[0154] in, represents the component of the left upper arm vector on the z-axis, Represents the component of the left upper arm vector on the coordinate axis.

[0155] like Figure 6 As shown in (b), the left shoulder joint roll angle It can be expressed as:

[0156]

[0157] in, represents the component of the left upper arm vector on the y-axis, Represents the two-norm of the projection of the left upper arm vector on the z plane.

[0158] After calculating the pitch and roll angles of the left shoulder joint, rotate around the y-axis The rotation matrix is ​​defined as:

[0159]

[0160] Rotate around the z axis The rotation matrix is ​​defined as:

[0161]

[0162] Then the vector of the left lower arm in the new coordinate system after the shoulder rotation is expressed as

[0163]

[0164] like Figure 7 As shown in (a), the yaw angle of the left elbow joint It can be expressed as:

[0165]

[0166] in, represents the component of the left lower arm vector on the y-axis, Represents the component of the left lower arm vector on the z-axis.

[0167] like Figure 7 As shown in (b), the left elbow joint roll angle It can be expressed as:

[0168]

[0169] in, represents the component of the left lower arm vector on the coordinate axis, Represents the two-norm of the projection of the left lower arm vector on the yz plane.

[0170] In the specific implementation of step (4), mapping joint angle filtering: smoothing the robot joint angles obtained in step (3) by Kalman filtering, and limiting the output of the joint angles according to the range of motion of each joint of the robot;

[0171] It includes the following sub-steps:

[0172] (4.1) performing Kalman filtering on the joint angles of the humanoid robot arms calculated in step (3) to avoid violent shaking of the joints during movement and achieve smooth joint movement;

[0173] Specifically, a motion model is constructed based on the Kalman filter algorithm, and the mapped joint angles are filtered to avoid violent shaking of the joints during movement and achieve smoothness of joint movement. The angles of the joints of the human body change randomly during actual movement. In order to achieve the filtering effect, this patent can approximate that the changes in the angles of the human joints are uniform, thereby updating the motion state based on the uniform motion model and the actual measured values. In the actual filtering process, a better filtering effect is achieved by adjusting the error covariance matrix, thereby eliminating points with large noise during movement and avoiding violent shaking of the joints. In this embodiment, a uniform motion model is established for the pitch angles of the shoulder joints, the roll angles of the shoulder joints, the yaw angles of the elbow joints, and the roll angles of the elbow joints of the left and right arms, and angle filtering is performed according to the time update equation and the state update equation of the Kalman filter. The time update equation of the Kalman filter is as follows:

[0174]

[0175]

[0176] The state update equation of the Kalman filter is as follows:

[0177]

[0178]

[0179]

[0180] in, represents the prior state estimate (prediction value) at the kth moment; x k represents the posterior state estimate (estimated value); A represents the state transfer matrix; represents the prior error covariance matrix; P k represents the posterior error covariance matrix; K k represents the Kalman gain; H represents the measurement matrix; Q represents the model error covariance matrix; R represents the measurement error covariance matrix. In the process, the ideal filtering effect is obtained by adjusting the Q and R matrices.

[0181] In this embodiment, the state variable is Where x represents the position variable, Represents the speed variable.

[0182] In this embodiment, the state transfer matrix A of the Kalman filter is expressed as:

[0183]

[0184] Wherein, Δt represents the time difference between the kth moment and the k-1th moment.

[0185] The measurement matrix H of Kalman filter is expressed as:

[0186] H

[10]

[0187] In the initial state, the error covariance matrix P k-1 Can be simply set as:

[0188]

[0189] Subsequently, the error covariance matrix will be continuously updated.

[0190] The model error covariance matrix can be set as a 2×2 diagonal matrix, and the measurement error covariance matrix can be set as a 1×1 matrix. By adjusting the sizes of the elements in these two matrices, a balance between measurement and model errors can be achieved, thereby obtaining a more ideal filtering effect.

[0191] (4.2) limiting the range of the joint angles according to the range of motion of each joint of the humanoid robot, and outputting the joint angles that meet the range limit to the humanoid robot so as to enable the humanoid robot to imitate the arm movements of the human body;

[0192] The range of motion of each joint of a humanoid robot is limited to a certain extent. Before the final joint angle output is performed, the mapping angle needs to be limited to a certain range to meet the mechanical structure requirements of the robot itself.

[0193] In one embodiment, for a general humanoid robot Nao, the range of motion of the pitch angle of the left shoulder joint is -119.5 ° ~119.5 ° ; The range of motion of the left shoulder joint roll angle is -18 ° ~76 ° ; The range of motion of the left elbow joint yaw angle is -119.5 ° ~119.5 ° ; The range of motion of the left elbow joint roll angle is -88.5 ° ~-2 ° The right shoulder joint pitch angle range is -119.5 ° ~119.5 ° ; The range of motion of the right shoulder joint roll angle is -76 ° ~18 ° ; The range of motion of the right elbow joint yaw angle is -119.5 ° ~119.5 ° ; The range of motion of the right elbow joint roll angle is 2 ° ~88.5 ° .

[0194] Ultimately, the joint angle data that meets the robot's joint motion range limits will be calculated and converted into command information to drive the robot's motion through the robot's own motion and control system, thereby completing the robot's arm motion imitation task for the input data.

[0195] like Figure 8 As shown, the color-depth camera is used for data collection and is connected to a PC via USB. The PC is used for data processing, model reasoning, and mapping angle calculation. The PC establishes a connection with the humanoid robot via a wired network and transmits the corresponding angle data to the robot. The robot finally drives the corresponding joint motor to complete the movement through its own motion control system.

[0196] Corresponding to the aforementioned embodiment of the humanoid robot motion imitation method based on 3D human posture estimation, the present application also provides an embodiment of a humanoid robot motion imitation device based on 3D human posture estimation.

[0197] Fig. 9 is a block diagram of a humanoid robot motion imitation device based on 3D human posture estimation according to an exemplary embodiment. Fig. 9 , the device may include:

[0198] The data acquisition and preprocessing module 21 is used to acquire color images and depth images using a color-depth camera, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing;

[0199] A human body motion capture module 22 is used to capture the three-dimensional coordinates of human joints based on a 3D human body posture estimation network according to the human body posture point cloud data obtained by the data acquisition and preprocessing module;

[0200] The action behavior mapping module 23 is used to convert the three-dimensional coordinates of the human joints predicted by the human limb motion capture module into the joint angles of the humanoid robot's arms based on coordinate conversion and space vector calculation;

[0201] The mapping joint angle filtering module 24 is used to smooth the robot joint angles obtained in the action behavior mapping module through Kalman filtering, and limit the output of the joint angles according to the range of motion of each joint of the robot, so as to realize the humanoid robot's imitation of the instructor's arm movements.

[0202] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0203] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0204] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned humanoid robot motion imitation method based on 3D human posture estimation. Fig.10 As shown, a hardware structure diagram of a humanoid robot motion imitation method based on 3D human posture estimation provided by an embodiment of the present invention is provided for any device with data processing capability, except Fig.10 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0205] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the humanoid robot motion imitation method based on 3D human posture estimation as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0206] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0207] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A humanoid robot motion imitation method based on 3D human posture estimation, characterized in that: include: (1) Data acquisition and preprocessing: Use a color-depth camera to collect color images and depth images, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing; (2) Human limb motion capture: Based on the human body posture point cloud data obtained in step (1), the three-dimensional coordinates of human joints are captured based on the 3D human body posture estimation network; (3) Action behavior mapping: Based on coordinate transformation and space vector calculation, the three-dimensional coordinates of the human joints predicted in step (2) are converted into the joint angles of the humanoid robot’s arms; (4) Mapping joint angle filtering: The robot joint angles obtained in step (3) are smoothed by Kalman filtering, and the joint angles are restricted and output according to the range of motion of each joint of the robot, so that the humanoid robot can imitate the arm movements of the instructor; Wherein, step (2) includes the following sub-steps: (2.1) quantifying the human body posture point cloud data; (2.2) Building a 3D human pose estimation network, wherein the 3D human pose estimation network includes a mask network and a pose estimation network; (2.3) After training the mask network until convergence, loading the trained mask network and setting it to an inference state, and training the posture estimation network according to the mask features generated by the trained mask network; (2.4) Based on the human body posture point cloud data quantized in step (2.1), the trained 3D human body posture estimation network is used to capture the three-dimensional coordinates of human body joints; Wherein, the 3D human posture estimation network includes: The mask network is used to roughly search the point cloud around the human joints. It uses the point cloud data within a certain range around the true value of the joint as the supervision signal to generate mask features indicating the importance of the point cloud. A posture estimation network is used to prune the point cloud according to the mask features transmitted by the mask network, and generate a three-dimensional heat map representing the probability distribution of joints, and predict the precise position of each joint by integrating and regressing the three dimensions separately; The mask network and the pose estimation network are both symmetric encoder-decoder structures composed of convolution and transposed convolution, and share the same pruning layer.

2. The method according to claim 1, characterized in that Step (1) includes the following sub-steps: (1.1) using a color-depth camera to collect a color image and a depth image, and matching the color image with the depth image to generate color point cloud data; (1.2) performing point cloud filtering and clustering on the color point cloud data to generate human body posture point cloud data; (1.3) Using data enhancement methods to process the human body posture point cloud data.

3. The method according to claim 1, characterized in that: Step (3) includes the following sub-steps: (3.1) By establishing the robot's left and right arm spatial coordinate systems and constructing human joint vectors, according to the corresponding relationship between the human joint vectors and the humanoid robot's double-arm structure, the three-dimensional coordinates of the human joints are converted into the robot's left and right arm spatial coordinate systems; (3.2) Using the three-dimensional coordinates of the human joints in the spatial coordinate system of the robot’s left and right arms, the upper arm vector and the lower arm vector are solved respectively, and then the three-dimensional coordinates of the human joints in the spatial coordinate system of the robot’s left and right arms are mapped to the joint angles of the robot’s left and right arms, including the shoulder joint pitch angle, shoulder joint roll angle, elbow joint yaw angle, and elbow joint roll angle of both arms.

4. The method according to claim 3, characterized in that: In step (3.2), taking the left arm as an example, the process of obtaining the shoulder joint pitch angle, shoulder joint roll angle, elbow joint yaw angle and elbow joint roll angle of the left arm includes: (3.2.1) Using the three-dimensional coordinates of the left shoulder, left elbow, and left wrist in the robot’s left arm spatial coordinate system, obtain the spatial vectors representing the left upper arm and left lower arm, respectively; (3.2.2) According to the projection relationship of the left upper arm vector on the three coordinate axes, the pitch angle and roll angle of the left shoulder joint are calculated; (3.2.3) By rotating the left arm spatial coordinate system, the vector representation of the left lower arm vector in the new coordinate system after the shoulder rotation is obtained; (3.2.4) Based on the projection relationship of the left lower arm vector on the three coordinate axes after rotation, the yaw angle and roll angle of the left elbow joint are calculated.

5. The method according to claim 1, characterized in that Step (4) includes the following sub-steps: (4.1) Performing Kalman filtering on the joint angles of the humanoid robot's arms calculated in step (3) to avoid violent shaking of the joints during movement and to achieve smooth joint movement; (4.2) Limiting the range of the joint angles according to the range of motion of each joint of the robot, and outputting the joint angles that meet the range limits to the humanoid robot so that the humanoid robot can imitate the arm movements of the instructor.

6. A device for a humanoid robot motion imitation method based on 3D human posture estimation as claimed in claim 1, characterized in that: include: The data acquisition and preprocessing module is used to acquire color images and depth images using a color-depth camera, and obtain the human body posture point cloud data of the instructor in the world coordinate system through data preprocessing; The human body motion capture module is used to capture the three-dimensional coordinates of human joints based on the 3D human posture estimation network according to the human posture point cloud data obtained by the data acquisition and preprocessing module; The action behavior mapping module is used to convert the three-dimensional coordinates of the human joints predicted by the human limb motion capture module into the joint angles of the humanoid robot's arms based on coordinate conversion and space vector calculation; The mapping joint angle filtering module is used to smooth the robot joint angles obtained in the action behavior mapping module through Kalman filtering, and to limit the output of the joint angles according to the range of motion of each joint of the robot, so that the humanoid robot can imitate the arm movements of the instructor.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images