Robot attitude estimation method based on deep learning

Abstract

The invention discloses a robot attitude estimation method based on deep learning. The method solves the problems of labor intensity of marking attitude annotations in real world images and difference contradiction between a virtual training domain and a real domain. The invention provides an unsupervised learning normal form which comprises four parts. The method comprises the following steps: firstly, on the basis of rendered robot pose estimation, predicting pose updating of a rendered image of current state estimation and a real image; detecting key points of the robot, inputting a real image into a network structure of an encoder-decoder, and predicting a confidence map; the projection key points of pose estimation and the detected key points are converted into a distribution situation, and a divergence-based objective function is constructed; and finally, a divergence objective function is approximated by sampling-based integration. This integrated design enables unsupervised learning to stably enhance the network.

Description

Technical Field

[0001] The present invention belongs to the field of robot hand calibration, and in particular to a robot posture estimation method based on deep learning. Background Art

[0002] Restoring the 6-DOF pose of a robot from a monocular image is a key task in robotics and computer vision, and has been widely used in various fields such as robot grasping, manipulation, and rearrangement. Although traditional offline methods such as hand-eye calibration have been the mainstream for many years, with the development of deep learning, online pose estimation techniques have emerged, which are both practical and economical, and offer great prospects for dynamic and unconstrained environments.

[0003] Mainstream deep learning-based methods can be divided into keypoint-based and rendering-based methods. However, these deep learning-based methods are mainly trained on synthetic data because annotating diverse real-world images is very labor-intensive. Therefore, these methods have difficulties in estimating the pose of real-world data due to the difference between training and testing, which is often referred to as the "simulation-to-real domain gap".

[0004] Similar challenges are evident in the field of object pose estimation, which has attracted significant research attention. Notably, unsupervised learning techniques that leverage real-world data have shown superior performance due to its targeted and explicit adaptation to the real-world domain. Typically, this learning paradigm associates pose estimation with another attribute extracted from the image, such as mask segmentation. Specifically, the pose is rendered as a robot (object) mask image and compared to the foreground mask obtained through segmentation via a pre-trained network as an unsupervised loss for pose estimation.

[0005] While this paradigm is effective in addressing the simulation-to-reality gap for generic objects, it is elusive to simply adapt it for robotics because: robots are often in heavily self-occluded configurations, which leads to ambiguity in masks and affects segmentation reliability for unsupervised learning; rendering is time-consuming as it needs to be differentiable, hindering online pose optimization at test time. Summary of the invention

[0006] The purpose of the present invention is to provide a robot posture estimation method based on deep learning, which uses unsupervised learning of unlabeled real-world images to solve the problem of the gap between simulation and reality in existing methods.

[0007] The technical solution to achieve the purpose of the present invention is: a robot posture estimation method based on deep learning, comprising the following steps:

[0008] Step 1: Rendering-based robot pose estimation: First, select a robotic arm part, render the known robot model into a 2D image based on the current pose estimate, concatenate the rendered image with the observed RGB image of the robot, and give it as input to a deep convolutional neural network for end-to-end learning of pose updates. The pose update is predicted so that the rendered image is aligned with the observed RGB image. Then, the pose estimate is projected onto the pixel plane based on the camera extrinsic parameters to obtain the projection key points corresponding to the pose estimate.

[0009] Step 2: Robot key point detection: input the observed RGB image into the encoder-decoder deep convolutional neural network structure to predict the confidence map of each robot key point;

[0010] Step 3: Convert the projected key points of pose estimation and the detected key points into the distribution in 2D pixel space. The projected key points must first be processed to obtain a confidence map consistent with the detected key points, and then normalized to obtain the distribution. The detected key points can be directly normalized in the pixel space to obtain the distribution, and then the objective function based on the distribution divergence is constructed.

[0011] In step 4, the divergence objective function is approximated by sampling-based integration, and the Adam learning rate adaptive update algorithm is used to control the optimization of the pose estimation, so as to finally align the projected key points with the detected key points in the image.

[0012] Furthermore, in step 1, based on the rendered robot pose estimation,

[0013] Step 1.1: Estimation of the current robot pose Render the image corresponding to the current pose estimate using a non-learning, robot-based 3DCAD model renderer Its height is 240 and width is 320;

[0014] Step 1.2, cut the RGB image of the observed robot from 480 in height and 640 in width to 240 in height and 320 in width, which is consistent with the rendered image;

[0015] Step 1.3, concatenate the cut image and the rendered image in the RGB color channel to obtain a feature image with 6 channels, a height of 240, and a width of 320;

[0016] Step 1.4: Input the feature map into the deep convolutional neural network f θ , the update of the predicted pose estimate:

[0017]

[0018] The above neural network f θResidual block connection is adopted; specifically, for the convolution module including convolution layer, normalization layer and activation layer, two convolution modules are set on the network trunk and one on the branch, and the residual block is obtained by skip connection. Finally, 5 residual blocks are stacked to obtain the feature extraction part of the neural network; f θ It also includes a multi-layer perceptron MLP for pose estimation, which accepts the mean of the features on the pixels after feature extraction to predict the pose update; in addition, the pose update ΔT follows the decoupled parameterization design method, representing the pose as a rotation in the object coordinate system and a translation in the camera coordinate system.

[0019] Furthermore, in step 2, the robot key point detection in the RGB image is

[0020] 2.1 Define the key points of the robot arm. Define each joint axis connection of the robot arm as the 3D key point of the robot arm. Its projection to the 2D pixel space is the key point of the RGB image of the robot arm.

[0021] 2.2 Use the encoder-decoder structure to train the neural network to predict key points, with the RGB image of the robot arm as input; the encoder passes through 4 downsampling convolutional layers, which include convolution, normalization, and activation operations, and the decoder is responsible for upsampling the image features obtained by the encoder into heat maps corresponding to key points, one heat map for each point; the decoder consists of 4 transposed convolutional layers; represented as, in the input scene graph The encoder-decoder processes the image to generate a series of d keypoint confidence maps, where d corresponds to the number of pre-set keypoints.

[0022]

[0023] Each confidence map corresponds to a key point, as a 2D array with height w and width h respectively, where the pixel value represents the possibility of the key point being projected onto this pixel; f φ A keypoint detection network representing an encoder-decoder structure.

[0024] Furthermore, in step 3: the divergence between the two distributions of posture projection and key point detection is used as the objective function, and the posture estimation obtained in step 1 is Project it to the pixel space and get its corresponding 2D key point; take this point as the center of the 2D Gaussian kernel function and 5 as the variance to build the confidence heat map corresponding to the projected key point; compare it with the detection key point heat map obtained in step 2 Figure 1 Starting from, by normalizing all pixel values, the key point probability density distribution function on the overall confidence map is obtained as follows:

[0025]

[0026] After obtaining these two distributions, the global difference between the two key point distributions is quantified by the KL divergence formula, which is used to fine-tune the pose estimation and rendering-based pose estimation networks, and the distribution divergence is used as the objective function loss.

[0027]

[0028] Only 6D pose projection key points It is optimized by the above distribution divergence objective function and then further back-propagated to the parameters of the pose estimation network; at the same time, the distribution of the detection key points is fixed as the alignment target for pose estimation.

[0029] Furthermore, in step 4, the divergence objective function is approximated based on the sampling integral,

[0030] 4.1 First, we make an assumption that applies to most hand-eye calibration scenarios, that is, the robotic arm is within the image;

[0031] 4.2 In the confidence map, the probability values ​​of the key point projection distribution and the detection distribution are sampled at all pixel coordinates, and the divergence between the two releases of approximate pose estimation and key point detection is as follows

[0032]

[0033] In the calculation process, the value of each pixel u = (x, y) in the confidence map is directly used instead of the probability density. The predicted value of the network is directly used as follows

[0034]

[0035] In the above formula, const1 and const2 are constant values ​​resulting from the normalization process, which have nothing to do with pose estimation and are directly set to 1 and 0;

[0036] 4.3 Take the above formula as the objective function, use the Adam learning rate adaptive update algorithm to control the optimization of posture estimation, and update the posture estimation so that the projected key points are aligned with the detected key points in the image.

[0037] Compared with the prior art, the present invention has the following significant advantages:

[0038] (1) This paper proposes a novel unsupervised learning paradigm that effectively bridges the gap between simulation and real scenes by combining rendering-based and keypoint-based pose estimation techniques, thereby achieving fine-tuning of network parameters and optimizing pose estimation when testing with unlabeled real-world images. Compared with previous methods, this paper makes full use of image information, which is beneficial to improving the stability of fine-tuning and the final performance.

[0039] (2) This paper introduces a novel objective function based on distribution divergence, which promotes robustness in unsupervised learning and approximates the divergence by comprehensively considering the key point distribution and sampling integration. Compared with previous methods, this paper fine-tunes the posture according to different detections from a more fine-grained perspective, and uses a consistent solution without manual debugging. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall framework diagram of the method of the present invention.

[0041] Figure 2 It is a rendering-based pose estimation framework diagram in the method of the present invention.

[0042] Figure 3 This is a qualitative comparison of the proposed method. The first, second and third rows are the original image, the baseline and our estimated results, respectively. The important contrasts are magnified in the black box. The red dots represent the estimated joint points, and the green circles represent the true values. Note that we show the result of the pose estimation projected onto the 2D image. DETAILED DESCRIPTION

[0043] The present invention provides a robot pose estimation method based on deep learning, aiming to solve the gap problem between the simulated source domain for training and the real target domain for testing, by combining the advantages of rendering-based pose estimation and key point detection networks, including an unsupervised learning framework covering the two networks and associating 2D key points with pose estimation, and an objective function quantifying the distribution difference between the distribution derived from the pose projection and the key points, and finally approximating the divergence of the target distribution based on sampling integrals. The overview of the present invention is shown in FIG. Figure 1 shown.

[0044] A robot posture estimation method based on deep learning of the present invention uses a rendering network to estimate posture, and uses a key point network to provide pseudo-labeled 2D key points for real-world images. The present invention pre-trains the rendering network and the key point-based network in a synthetic dataset. Then, in the real-world image, the rendering-based posture estimation is projected into the pixel space and aligned with the image key point detection to train the rendering network. Specifically, the following steps are included:

[0045] Step 1, rendering-based robot pose estimation, is derived from the rendering comparison paradigm in general object pose estimation, and is modified according to the multi-joint characteristics of the robotic arm. First, a robotic arm part is selected, and the known robot model is rendered into a 2D image based on the current pose estimation, which is then concatenated with the input image and given as input to a deep neural network for end-to-end learning of pose updates, with the goal of making the rendered image of the robotic arm more and more consistent with the real image.

[0046] Step 2, image key point detection, is similar to human posture detection and is based on key points. The robot image is used as input to predict the coordinates of the predefined robot key points in the image, aiming to extract the key point features from the image through the network. Based on a large number of key point detection schemes, an encoder-decoder structure is used to predict the key point confidence map corresponding to the image, with each key point corresponding to a confidence map.

[0047] Step 3: Based on the divergence objective function, we construct the key point distribution on the 2D image corresponding to pose estimation and key point detection, and use an objective function based on distribution divergence to quantify the gap between pose estimation and key point detection, which is used to fine-tune the rendering-based pose estimation network. The confidence map is converted into key point distribution, making full use of the fine-grained information of key point detection and avoiding the deviation caused by direct reasoning of coordinates.

[0048] Step 4, four processes of sampling-based integration, treat the distribution objective function solution as an integration process based on sampling, avoiding the need for difficult-to-obtain analytical expressions for direct analytical solutions. The present invention adopts a general method to approximate the objective function value based on sampling. Based on reasonable assumptions, such as the robotic arm is in the image, it is applicable to most hand-eye calibration scenarios.

[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] In the image-based pose estimation scenario, a known robot is usually considered, which includes an available 3D robot model and a kinematic model. Different from the pose estimation of general objects, the robot contains multiple parts connected by joints. Due to this unique multi-joint feature, it is necessary to define the joint angles. d is the number of joints. Based on the joint angles and robot model, the robot forward kinematics can be used to obtain predefined 3D key points:

[0051]

[0052] In most robotic applications, joint angles are measured by sensors, and the robot model is known as prior information. Therefore, the 6D pose T∈SE(3) from the camera to the robot arm is the only variable that determines the robot state. For known camera intrinsic parameters, projecting the above key points into the image space yields

[0053]

[0054] The above is an efficient process of associating 3D and 2D space, which supports the application of online optimization process in the present invention compared to the previous differentiable rendering. The unsupervised learning proposed in the present invention includes the following stages: pre-training the rendering-based pose estimation network and the key point detection network in the synthetic data domain; comparing the projection of the pose and the key point detection to fine-tune the two networks, and using the distribution-based objective function to quantify the difference between the two in the unlabeled real data domain; and finally, the sampling-based integral is used to approximate the theoretically unsolvable target distribution function.

[0055] Step 1: Pose estimation based on rendering and comparison

[0056] An initial pose estimate is iteratively fine-tuned by a deep neural network according to a rendering and contrast paradigm. Due to the general difficulty of regressing 3D information directly from 2D images, this paradigm first estimates the pose of a randomly selected robot part based on the current Render together, then render the image With the input image concatenated and input together into the deep convolutional neural network f θ The changes to the network output pose and the update of the current pose estimate are as follows:

[0057]

[0058] The above ΔT follows a decoupled parameterization design approach, derived from general object pose estimation, in order to be suitable for image extraction of more predictable decoupled rotation and translation features. At this point, the parameters θ of the deep convolutional neural network can be fine-tuned according to a simple gap loss function to align the keypoints projected to the pose with the high-confidence keypoint detections:

[0059]

[0060] The above objective function is effective in using the entire useful keypoint detection information to fine-tune the parameters θ of the deep convolutional neural network to make it applicable to the real-world domain. However, direct explicit utilization has several disadvantages:

[0061] (1) The selection of key point detection requires a user-specified confidence threshold;

[0062] (2) The reasoning of key point coordinates introduces a non-learnable algorithm, which leads to deviations

[0063] (3) The objective function is important for fine-tuning the convolutional neural network f θ It is relatively aggressive and can easily lead to a decrease in the pose estimation performance when the distance between the pose projection and the key point detection is far.

[0064] Therefore, the global objective function based on distribution divergence proposed in the present invention is needed to quantify the differences between poses and key points with high quality, ensure the stability of the entire unsupervised learning framework, and do not require additional processes and processing.

[0065] Step 2: Key point detection

[0066] The present invention adopts a general key point detection paradigm to predict the key points corresponding to the parts of the robot. The specific implementation process is as follows: an encoder-decoder structure processes the image to generate a series of d key point confidence maps, where d corresponds to the number of preset key points.

[0067]

[0068] Each confidence map corresponds to a key point, as a 2D array with height w and width h respectively, where the pixel value represents the probability of the key point being projected onto this pixel. φ Key point detection network representing an encoder-decoder structure. In most key point detection applications, a series of post-processing is applied to the confidence map to infer the exact coordinates of the key points. The simple gap loss function mentioned above uses this. Although effective, the original confidence map has more information, potentially for fine-tuning the convolution-based pose estimation neural network. These reasoning processes may also introduce biases when extracting discrete key point coordinates, especially for scenarios where the confidence map is relatively flat or in multimodal conditions. To address this issue, the present invention explores the global information of the confidence map to discover and represent the possible locations and uncertainties of the key points. In addition, the present invention can also fine-tune the encoder-decoder network parameters φ by using pose projections as pseudo-labels for key points. Because the pose estimation contains 3D information related to the entire robot, it is very beneficial to assist in the detection of some uncertain key points. The present invention sets the pseudo-label of the confidence map through a Gaussian kernel function to project each key point Take it as the peak value and use a larger variance σ, for each pixel with coordinates u = (x, y)

[0069]

[0070] The keypoint detector is fine-tuned by using the uncertainty confidence map B i With pseudo labels The L2 loss function is used to drive the

[0071] So far the unsupervised learning framework of the present invention has been introduced, which is used to integrate the advantages of pose estimation and key point detection networks to train each other, and can significantly bridge the gap from virtual to reality only by training on unlabeled real data.

[0072] Step 3: Objective function based on distribution divergence

[0073] The core of the present invention is the pose estimation of the robot, so fine-tuning the rendering-based pose estimation network is the key point. However, the simple difference objective function cannot fully utilize the information of key point detection. Generally, the detected confidence map can be regarded as the distribution of key points, which contains fine-grained coordinate and confidence information, which is very helpful for promoting the performance of deep unsupervised learning. In fact, the pixel value in each confidence map represents the probability that this pixel corresponds to a key point. Therefore, the key point probability density distribution function on the overall confidence map can be obtained by normalizing all pixel values

[0074]

[0075] The above key point probability density distribution function makes the projected key points of 6D pose It can be aligned with key point detections in a globally consistent way, rather than just with inferred coordinates. In order to quantify the similarity between pose projections and key point detections, it is necessary to represent the pose projection as another key point probability distribution, using the same normalized confidence map. The corresponding confidence map of the pose projection is as follows

[0076]

[0077] After obtaining these two distributions, the global difference between the two key point distributions can be quantified by the KL divergence formula (Kullback-Leibler Divergence), which can be used to fine-tune the pose estimation and rendering-based pose estimation networks. As long as the distribution divergence is used as the objective function loss

[0078]

[0079] Note that there are only 6D pose projection key points It is optimized by the distribution divergence objective function mentioned above and then further back-propagated to the parameters of the pose estimation network. At the same time, the distribution of the key points detected is fixed as the alignment target for pose estimation. This objective function greatly improves the stability of unsupervised learning because it is applicable to key point detection in various situations.

[0080] Step 4: Sampling-based integration

[0081] The above objective function based on distribution divergence can be calculated by its definition

[0082]

[0083] This integral is difficult to calculate analytically because the distribution pi (x, y) is essentially an unknown distribution expression. Empirically, this unknown distribution can be approximated by a parameter model, but this will result in information loss in key point detection. Instead, the present invention uses a common measure in the case of unknown analytical expressions, which directly approximates the distribution divergence value based on sampling.

[0084] The first step is to determine the upper and lower limits of the integral, or the sampling area, which the present invention limits to the confidence map. This assumption is based on the fact that the robot recorded in the image contains all joints and parts in the field of view, which is also the most common requirement for robot hand-eye calibration. Therefore, the probability values ​​of the two distributions are randomly and uniformly sampled in the confidence map to obtain the divergence between the two distributions of pose estimation and key point detection as follows:

[0085]

[0086] In the calculation process, the value of each pixel u = (x, y) in the confidence map can be used directly instead of the probability density to directly use the predicted value of the network as follows

[0087]

[0088] In the above formula, const1 and const2 are constant values ​​resulting from the normalization process, which have nothing to do with pose estimation and can be ignored in unsupervised training. Note that

[0089]

[0090] The relationship between the distribution divergence objective and the pose estimation projected keypoints is implied to drive the fine-tuning of pose estimation and rendering-based networks.

[0091] The embodiments described below are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] The overall framework of the method of the present invention is as follows Figure 1 As shown in Figure 1, a pose estimation method based on deep learning is designed to address the difference between the virtual domain and the real-world target domain. The specific steps are:

[0093] Step 1: Pose estimation based on rendering and comparison

[0094] An initial pose estimate is iteratively fine-tuned by a deep neural network according to a rendering and contrast paradigm. Due to the general difficulty of regressing 3D information directly from 2D images, this paradigm first estimates the pose of a randomly selected robot part based on the current Render together, then render the image With the input image concatenated and input together into the deep convolutional neural network f θ The changes to the network output pose and the update of the current pose estimate are as follows:

[0095]

[0096] The above ΔT follows a decoupled parameterization design approach, derived from general object pose estimation, in order to be suitable for image extraction of more predictable decoupled rotation and translation features. At this point, the parameters θ of the deep convolutional neural network can be fine-tuned according to a simple gap loss function to align the keypoints projected to the pose with the high-confidence keypoint detections:

[0097]

[0098] The above objective function is effective in using the entire useful keypoint detection information to fine-tune the parameters θ of the deep convolutional neural network to make it applicable to the real-world domain. However, direct explicit utilization has several disadvantages:

[0099] (1) The selection of key point detection requires a user-specified confidence threshold;

[0100] (2) The reasoning of key point coordinates introduces a non-learnable algorithm, which leads to deviations

[0101] (3) The objective function is important for fine-tuning the convolutional neural network f θ It is relatively aggressive and can easily lead to a decrease in the pose estimation performance when the distance between the pose projection and the key point detection is far.

[0102] Therefore, the global objective function based on distribution divergence proposed in the present invention is needed to quantify the differences between poses and key points with high quality, ensure the stability of the entire unsupervised learning framework, and do not require additional processes and processing.

[0103] Step 2: Key point detection

[0104] The present invention adopts a general key point detection paradigm to predict the key points corresponding to the parts of the robot. The specific implementation process is as follows: an encoder-decoder structure processes the image to generate a series of d key point confidence maps, where d corresponds to the number of preset key points.

[0105]

[0106] Each confidence map corresponds to a key point, as a 2D array with height w and width h respectively, where the pixel value represents the probability of the key point being projected onto this pixel. φ Key point detection network representing an encoder-decoder structure. In most key point detection applications, a series of post-processing is applied to the confidence map to infer the exact coordinates of the key points. The simple gap loss function mentioned above uses this. Although effective, the original confidence map has more information, potentially for fine-tuning the convolution-based pose estimation neural network. These reasoning processes may also introduce biases when extracting discrete key point coordinates, especially for scenarios where the confidence map is relatively flat or in multimodal conditions. To address this issue, the present invention explores the global information of the confidence map to discover and represent the possible locations and uncertainties of the key points. In addition, the present invention can also fine-tune the encoder-decoder network parameters φ by using pose projections as pseudo-labels for key points. Because the pose estimation contains 3D information related to the entire robot, it is very beneficial to assist in the detection of some uncertain key points. The present invention sets the pseudo-label of the confidence map through a Gaussian kernel function to project each key point Take it as the peak value and use a larger variance σ, for each pixel with coordinates

[0107] u=(x,y)

[0108]

[0109] The keypoint detector is fine-tuned by using the uncertainty confidence map B i With pseudo labels The L2 loss function is used to drive the

[0110] So far the unsupervised learning framework of the present invention has been introduced, which is used to integrate the advantages of pose estimation and key point detection networks to train each other, and can significantly bridge the gap from virtual to reality only by training on unlabeled real data.

[0111] Step 3: Objective function based on distribution divergence

[0112] The core of the present invention is the pose estimation of the robot, so fine-tuning the rendering-based pose estimation network is the key point. However, the simple difference objective function cannot fully utilize the information of key point detection. Generally, the detected confidence map can be regarded as the distribution of key points, which contains fine-grained coordinate and confidence information, which is very helpful for promoting the performance of deep unsupervised learning. In fact, the pixel value in each confidence map represents the probability that this pixel corresponds to a key point. Therefore, the key point probability density distribution function on the overall confidence map can be obtained by normalizing all pixel values

[0113]

[0114] The above key point probability density distribution function makes the projected key points of 6D pose It can be aligned with key point detections in a globally consistent way, rather than just with inferred coordinates. In order to quantify the similarity between pose projections and key point detections, it is necessary to represent the pose projection as another key point probability distribution, using the same normalized confidence map. The corresponding confidence map of the pose projection is as follows

[0115]

[0116] After obtaining these two distributions, the global difference between the two key point distributions can be quantified by the KL divergence formula (Kullback-Leibler Divergence), which can be used to fine-tune the pose estimation and rendering-based pose estimation networks. As long as the distribution divergence is used as the objective function loss

[0117]

[0118] Note that there are only 6D pose projection key points It is optimized by the distribution divergence objective function mentioned above and then further back-propagated to the parameters of the pose estimation network. At the same time, the distribution of the key points detected is fixed as the alignment target for pose estimation. This objective function greatly improves the stability of unsupervised learning because it is applicable to key point detection in various situations.

[0119] Step 4: Sampling-based integration

[0120] The above objective function based on distribution divergence can be calculated by its definition

[0121]

[0122] This integral is difficult to calculate analytically because the distribution p i (x, y) is essentially an unknown distribution expression. Empirically, this unknown distribution can be approximated by a parameter model, but this will result in information loss in key point detection. Instead, the present invention uses a common measure in the case of unknown analytical expressions, which directly approximates the distribution divergence value based on sampling.

[0123] The first step is to determine the upper and lower limits of the integral, or the sampling area, which the present invention limits to the confidence map. This assumption is based on the fact that the robot recorded in the image contains all joints and parts in the field of view, which is also the most common requirement for robot hand-eye calibration. Therefore, the probability values ​​of the two distributions are randomly and uniformly sampled in the confidence map to obtain the divergence between the two distributions of pose estimation and key point detection as follows:

[0124]

[0125] In the calculation process, the value of each pixel u = (x, y) in the confidence map can be used directly instead of the probability density to directly use the predicted value of the network as follows

[0126]

[0127] In the above formula, const1 and const2 are constant values ​​resulting from the normalization process, which have nothing to do with pose estimation and can be ignored in unsupervised training. Note that

[0128]

[0129] The relationship between the distribution divergence objective and the pose estimation projected keypoints is implied to drive the fine-tuning of pose estimation and rendering-based networks.

[0130] In order to further verify the feasibility and effectiveness of the method of the present invention, the technical effect of the present invention is further described in combination with experiments. The hardware platform used in the experiment of the present invention is: Intel (R) Xeon (R) Gold 6230 CPU @ 2.10GHz × 16, 64G memory, two GPUs with 24G video memory, and the model is NVIDIA TITAN RTX. The software platform used in the experiment of the present invention is: Ubuntu 18.04.6LTS operating system and Python3.8.17, Pytorch1.8.0 deep learning framework. Four real-world data sets are used to analyze the robot pose estimation effect obtained by this method. These data sets are for the 7-degree-of-freedom robot Franka Emika Panda, including data sets Azure Kinect (AK), XBOX 360 Kinect (XK) and Realsense (RS) and Panda-ORB. Here, the mean and integral value (Area Under Curve, AUC) of the average distance (Average Distance, ADD) are used as evaluation indicators to evaluate the performance of pose estimation.

[0131] The method of the present invention is used to estimate the pose of the robot images of the above four data sets, and the pose estimation results corresponding to each scene are obtained. In order to test the performance of the method of the present invention, the proposed pose estimation method based on deep learning is compared with several existing methods, including: DREAM, a pose estimation method based on key point detection proposed by Timothy E. Lee, et al. in their published paper "Camera-to-robot pose estimation from a single image, ICRA, 2020.", and RoboPose, a pose estimation method based on rendering proposed by Yann Labbé et al. in their published paper "Single-view robot pose andjoint angle estimation via render compare, CVPR, 2021." The quantitative results are listed in Table 1, using the above evaluation index AUC as the evaluation standard, the value is a percentage, and the larger the value, the better.

[0132] Table 1 Quantitative test results of the method of the present invention and the prior art on the validation set

[0133] method AK XK RS ORB DREAM 60.51 64.01 78.83 69.05 RoboPose 76.49 85.97 76.90 80.29 Method of the present invention 86.95 87.94 87.45 86.21

[0134] From the results in Table 1, it can be seen that the test results obtained by the method of the present invention on the four data sets are higher than those of the existing methods, indicating that the present invention can be more applicable to the real world domain. Figure 3 As shown, the pose results of the method of the present invention are more accurate than those of the existing methods.

[0135] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A robot posture estimation method based on deep learning, characterized in that: The following steps are involved: Step 1: Rendering-based robot pose estimation: First, select a robotic arm part, render the known robot model into a 2D image based on the current pose estimate, concatenate the rendered image with the observed RGB image of the robot, and give it as input to a deep convolutional neural network for end-to-end learning of pose updates. The pose update is predicted so that the rendered image is aligned with the observed RGB image. Then, the pose estimate is projected onto the pixel plane based on the camera extrinsic parameters to obtain the projection key points corresponding to the pose estimate. Step 2: Robot key point detection: input the observed RGB image into the encoder-decoder deep convolutional neural network structure to predict the confidence map of each robot key point; Step 3: Convert the projected key points of pose estimation and the detected key points into a distribution in 2D pixel space. The projected key points must first be processed to obtain a confidence map consistent with the detected key points, and then normalized to obtain a distribution. The detected key points can be directly normalized in the pixel space to obtain a distribution, and then a target function based on the distribution divergence is constructed. In step 4, the divergence objective function is approximated by sampling-based integration, and the Adam learning rate adaptive update algorithm is used to control the optimization of the pose estimation, so as to finally align the projected key points with the detected key points in the image.

2. The robot posture estimation method based on deep learning according to claim 1, characterized in that: In step 1, the robot pose is estimated based on rendering. Step 1.1: Estimation of the current robot pose Render the image corresponding to the current pose estimate using a non-learning, robot-based 3DCAD model renderer Its height is 240 and width is 320; Step 1.2, cut the RGB image of the observed robot from 480 in height and 640 in width to 240 in height and 320 in width, which is consistent with the rendered image; Step 1.3, concatenate the cut image and the rendered image in the RGB color channel to obtain a feature image with 6 channels, a height of 240, and a width of 320; Step 1.4: Input the feature map into the deep convolutional neural network f θ , the update of the predicted pose estimate: The above neural network f θ Use residual block connection; specifically, for the convolution module containing convolution layer, normalization layer and activation layer, set two of this convolution module on the network trunk and one on the branch, jump connection to obtain residual block, and finally stack 5 residual blocks to obtain the feature extraction part of the neural network; f θ It also includes a multi-layer perceptron MLP for pose estimation, which accepts the mean of the features on the pixels after feature extraction to predict the pose update; in addition, the pose update ΔT follows the decoupled parameterization design method, representing the pose as a rotation in the object coordinate system and a translation in the camera coordinate system.

3. The robot posture estimation method based on deep learning according to claim 1, characterized in that: In step 2, the robot key point detection in the RGB image, 2.1 Define the key points of the robot arm. Define each joint axis connection of the robot arm as the 3D key point of the robot arm. Its projection to the 2D pixel space is the key point of the RGB image of the robot arm. 2.2 Use the encoder-decoder structure to train the neural network to predict key points, with the RGB image of the robotic arm as input; the encoder passes through 4 downsampling convolutional layers, which include convolution, normalization, and activation operations, and the decoder is responsible for upsampling the image features obtained by the encoder into heat maps corresponding to key points, one heat map for each point; The decoder consists of 4 transposed convolutional layers; represented as, in the input scene graph The encoder-decoder processes the image to generate a series of d keypoint confidence maps, where d corresponds to the number of pre-set keypoints. Each confidence map corresponds to a key point, as a 2D array with height w and width h respectively, where the pixel value represents the possibility of the key point being projected onto this pixel; f φ A keypoint detection network representing an encoder-decoder structure.

4. The robot posture estimation method based on deep learning according to claim 1, characterized in that: In step 3, the divergence between the two distributions of posture projection and key point detection is used as the objective function, and the posture estimation obtained in step 1 is Project it to the pixel space to get its corresponding 2D key point; take this point as the center of the 2D Gaussian kernel function and 5 as the variance to construct the confidence heat map corresponding to the projected key point; together with the detection key point heat map obtained in step 2, normalize all pixel values ​​to get the key point probability density distribution function on the overall confidence map as follows: After obtaining these two distributions, the global difference between the two key point distributions is quantified by the KL divergence formula, which is used to fine-tune the pose estimation and rendering-based pose estimation networks, and the distribution divergence is used as the objective function loss. Only 6D pose projection key points It is optimized by the above distribution divergence objective function and then further back-propagated to the parameters of the pose estimation network; at the same time, the distribution of the detection key points is fixed as the alignment target for pose estimation.

5. The method for posture estimation based on deep learning according to claim 1, characterized in that: In step 4, the divergence objective function is approximated based on the sampling integral, 4.1 First, we make an assumption that applies to most hand-eye calibration scenarios, that is, the robotic arm is within the image; 4.2 In the confidence map, the probability values ​​of the key point projection distribution and the detection distribution are sampled at all pixel coordinates, and the divergence between the two releases of approximate pose estimation and key point detection is as follows In the calculation process, the value of each pixel u = (x, y) in the confidence map is directly used instead of the probability density. The predicted value of the network is directly used as follows In the above formula, const1 and const2 are constant values ​​resulting from the normalization process, which have nothing to do with pose estimation and are directly set to 1 and 0; 4.3 Take the above formula as the objective function, use the Adam learning rate adaptive update algorithm to control the optimization of posture estimation, and update the posture estimation so that the projected key points are aligned with the detected key points in the image.

Images