A robot pose estimation method based on deep learning

Abstract

The application discloses a robot pose estimation method based on deep learning, which solves the contradiction between labor-intensive pose annotation in real-world images and the difference between virtual training domain and real domain. The application proposes an unsupervised learning paradigm, which includes four parts. First, based on rendered robot pose estimation, the rendered image estimated in the current state is updated together with the real image to predict the pose. Then, the robot key point detection is input into the encoder-decoder network structure to predict the confidence map. Then, the projected key points of the pose estimation and the detected key points are converted into a distributed situation to construct a divergence-based objective function. Finally, the divergence objective function is approximated by using the integral based on sampling. The comprehensive design enables the unsupervised learning to stably enhance the network.

Description

Technical Field

[0001] This invention belongs to the field of robot hand calibration, and in particular, it is a robot pose estimation method based on deep learning. Background Technology

[0002] From monocular image recovery to robot 6DOF pose estimation, this is a critical task in robotics and computer vision, and has been widely applied in various fields such as robot grasping, manipulation, and rearrangement. While traditional offline methods such as hand-eye calibration have been mainstream for many years, the development of deep learning has led to the emergence of online pose estimation techniques that are both practical and cost-effective, offering great promise for dynamic and unrestricted environments.

[0003] Mainstream deep learning-based methods can be categorized into keypoint-based and rendering-based methods. However, these methods are primarily trained on synthetic data because annotating diverse real-world images is extremely labor-intensive. Consequently, due to the discrepancy between training and testing, these methods struggle to estimate poses in real-world data, a phenomenon often referred to as the "simulation-to-reality gap."

[0004] Similar challenges are evident in the field of object pose estimation, which has attracted significant research attention. Notably, unsupervised learning techniques utilizing real-world data have demonstrated superior performance because they are specifically and explicitly adapted to the real-world domain. Typically, this learning paradigm associates pose estimation with another attribute extracted from the image, such as mask segmentation. Specifically, the pose is rendered as a robot (object) mask image and compared with a foreground mask segmented by a pre-trained network as the unsupervised loss for pose estimation.

[0005] While this paradigm is effective in bridging the gap between simulation and reality for general objects, it is difficult to apply it simply to robots because: robots are often in configurations with severe self-occlusion, which leads to mask ambiguity and affects the segmentation reliability of unsupervised learning; rendering is time-consuming because it needs to be differentiable, thus hindering online pose optimization at test time. Summary of the Invention

[0006] The purpose of this invention is to provide a robot pose estimation method based on deep learning, which uses unsupervised learning from unlabeled real-world images to solve the problem of the gap between simulation and reality in existing methods.

[0007] The technical solution to achieve the objective of this invention is: a robot pose estimation method based on deep learning, comprising the following steps:

[0008] Step 1, Render-based robot pose estimation: First, select a robotic arm part, render the known robot model into a 2D image based on the current pose estimation, stitch the rendered image together with the observed RGB image of the robot, and feed it as input to a deep convolutional neural network to learn the pose update end-to-end. The predicted pose update makes the rendered image and the observed RGB image consistent and aligned. Then, the pose estimation is projected onto the pixel plane according to the camera extrinsic parameters to obtain the projection key points corresponding to the pose estimation.

[0009] Step 2, Robot key point detection: The observed RGB image is input into the deep convolutional neural network structure of encoder-decoder to predict the confidence map of each robot key point;

[0010] Step 3: Transform the projected keypoints of pose estimation and the detected keypoints into a distribution in 2D pixel space. The projected keypoints are first processed to obtain a confidence map consistent with the detected keypoints, and then normalized to obtain a distribution. The detected keypoints can be directly normalized in pixel space to obtain a distribution. Then, an objective function based on the distribution divergence is constructed.

[0011] Step 4: The divergence objective function is approximated by sampling-based integral, and the Adam learning rate adaptive update algorithm is used to control the optimization of pose estimation, so that the projected key points are aligned with the detected key points in the image.

[0012] Furthermore, in step 1, the robot pose estimation is based on rendering.

[0013] Step 1.1, Estimating the current robot pose Render the image corresponding to the current pose estimation using a renderer of a non-learning, robot-based 3D CAD model. Its height is 240 and its width is 320;

[0014] Step 1.2: Cut the RGB image of the observation robot from 480 in height and 640 in width to 240 in height and 320 in width, consistent with the rendered image;

[0015] Step 1.3: Concatenate the cut image and the rendered image in the RGB color channels to obtain a feature map with 6 channels, a height of 240, and a width of 320.

[0016] Step 1.4: Input the feature map into the deep convolutional neural network f θ In the middle, the update of the predicted pose estimation:

[0017]

[0018] The aforementioned neural network f θResidual block connections are used; specifically, for a convolutional module containing convolutional layers, normalization layers, and activation layers, two of this convolutional module are placed on the network backbone and one on a branch, with skip connections to obtain residual blocks. Finally, the five residual blocks are stacked to obtain the feature extraction part of the neural network; f θ It also includes a multilayer perceptron (MLP) for pose estimation, which accepts the mean of features on pixels after feature extraction to predict pose updates; in addition, the pose update ΔT follows a decoupled parametric design method, which represents the pose as rotation in the object coordinate system and translation in the camera coordinate system.

[0019] Furthermore, in step 2, robot keypoint detection is performed in the RGB image.

[0020] 2.1 Define the key points of the robotic arm. Define each joint axis connection of the robotic arm as the 3D key point of the robotic arm. Its projection onto the 2D pixel space is the key point of the RGB image of the robotic arm.

[0021] 2.2 A neural network is trained using an encoder-decoder architecture to predict keypoints, with the RGB image of the robotic arm as input. The encoder uses four downsampling convolutional layers, including convolution, normalization, and activation operations. The decoder upsamples the image features obtained from the encoder to create heatmaps corresponding to each keypoint, one heatmap per keypoint. The decoder consists of four transposed convolutional layers. This is represented as follows: In the input scene image... Above, the encoder-decoder processes the image to generate a series of d keypoint confidence maps, where d corresponds to a pre-set number of keypoints.

[0022]

[0023] Each confidence map corresponds to a keypoint, represented as a 2D array with height w and width h, where the pixel value represents the probability that the keypoint will be projected onto that pixel; f φ Key point detection network representing the encoder-decoder structure.

[0024] Furthermore, in step 3: the method of using the divergence between the two distributions of pose projection and keypoint detection as the objective function, the pose estimate obtained in step 1 is... Projecting the data into pixel space yields the corresponding 2D keypoints. Using these keypoints as the center of a 2D Gaussian kernel function and a variance of 5, a confidence heatmap is constructed for each projected keypoint. This heatmap is then compared with the heatmap of the detected keypoints obtained in step two. Figure 1 Starting from this point, the probability density distribution function of key points on the overall confidence map is obtained by normalizing all pixel values ​​as follows:

[0025]

[0026] These two distributions were obtained, and the global difference between the two keypoint distributions was quantified using the KL divergence formula. This difference was then used to fine-tune the pose estimation and rendering-based pose estimation networks, with the distribution divergence used as the objective function loss.

[0027]

[0028] Only 6D pose projection key points The parameters are optimized by the above distribution divergence objective function and then backpropagated to the pose estimation network; at the same time, the distribution of the key points is fixed and used as the alignment target for pose estimation.

[0029] Furthermore, in step 4, the divergence objective function is approximated based on the integral of the samples.

[0030] 4.1 First, we make an assumption applicable to most hand-eye calibration scenarios, namely, that the robotic arm is within the image;

[0031] 4.2 Within the confidence map, the probability values ​​in the keypoint projection distribution and detection distribution are sampled at all pixel coordinates. The divergence between the pose estimation and keypoint detection publications is approximated as follows:

[0032]

[0033] During the calculation, the value of each pixel u = (x, y) in the confidence map is directly used to replace the probability density, and the network's predicted value is used directly as follows:

[0034]

[0035] In the above formula, const1 and const2 are constant values ​​resulting from the normalization process, which are irrelevant to pose estimation, and are directly set to 1 and 0;

[0036] 4.3 The above equation is the objective function. The Adam learning rate adaptive update algorithm is used to control the optimization of pose estimation. The updated pose estimation aligns the projected keypoints with the detected keypoints in the image.

[0037] Compared with the prior art, the present invention has the following significant advantages:

[0038] (1) This invention proposes a novel unsupervised learning paradigm that effectively bridges the gap between simulation and real-world scenarios by combining rendering-based and keypoint-based pose estimation techniques. This enables fine-tuning of network parameters and optimization of pose estimation when testing with unlabeled real-world images. Compared to previous methods, this invention makes full use of image information, which helps improve the stability of fine-tuning and the final performance.

[0039] (2) This invention introduces a novel objective function based on distribution divergence, which promotes robustness in unsupervised learning. It approximates the divergence by comprehensively considering the distribution of key points and sampling ensemble. Compared with previous methods, this invention provides a more granular approach to fine-tune the pose according to different detections, achieving a consistent solution without the need for manual adjustments. Attached Figure Description

[0040] Figure 1 This is an overall framework diagram of the method of the present invention.

[0041] Figure 2 This is a rendering-based pose estimation framework diagram in the method of this invention.

[0042] Figure 3 This section compares the qualitative results of the method presented in this invention. The first, second, and third rows show the original image, the baseline, and our estimation results, respectively. Important comparisons are enlarged within the black boxes. Red dots represent estimated joints, and green circles represent ground truth values. Note that we are showing the results of the pose estimation projected onto the 2D image. Detailed Implementation

[0043] This invention provides a deep learning-based robot pose estimation method aimed at addressing the gap between the simulated source domain during training and the real target domain during testing. It combines the advantages of rendering-based pose estimation and keypoint detection networks, including an unsupervised learning framework encompassing both networks and associated 2D keypoints with pose estimation. It also employs an objective function to quantify the difference between the distribution derived from pose projection and the distribution of keypoints, and finally approximates the divergence of the target distribution based on sampled integrals. An overview of the invention is shown in the figure below. Figure 1 As shown.

[0044] This invention discloses a deep learning-based robot pose estimation method. It uses a rendering network to estimate pose and a keypoint network to provide pseudo-labeled 2D keypoints for real-world images. The invention pre-trains both the rendering network and the keypoint-based network on a synthetic dataset. Then, in real-world images, the rendering-based pose estimation is projected into pixel space and aligned with image keypoint detection to train the rendering network. Specifically, it includes the following steps:

[0045] Step one, rendering-based robot pose estimation, originates from the rendering contrast paradigm in general object pose estimation and is modified according to the multi-joint characteristics of the robotic arm. First, a robotic arm section is selected. The known robot model is rendered into a 2D image based on the current pose estimation, and this image is concatenated with the input image. This concatenation is then fed into a deep neural network for end-to-end learning of pose updates, aiming to make the rendered image of the robotic arm increasingly consistent with the real image.

[0046] Step two, image keypoint detection, similar to human pose detection, is based on keypoints. Using the robotic arm image as input, the coordinates of predefined robotic arm keypoints in the image are predicted, aiming to extract keypoint features from the image through a network. Based on numerous keypoint detection schemes, an encoder-decoder structure is used to predict the corresponding keypoint confidence map for the image, with one confidence map for each keypoint.

[0047] Step 3: Based on the divergence objective function, construct the keypoint distributions on the 2D image corresponding to pose estimation and keypoint detection, respectively. Use a divergence-based objective function to quantify the gap between pose estimation and keypoint detection, which is then used to fine-tune the rendering-based pose estimation network. Specifically, the confidence map is transformed into a keypoint distribution to fully utilize the fine-grained information from keypoint detection and avoid biases caused by direct inference from coordinates.

[0048] Step four involves four processes based on sampling integration. Solving the objective function of the distribution is viewed as a sampling integration process, avoiding the need for difficult-to-obtain analytical expressions for direct analytical solutions. This invention employs a universal method to approximate the objective function value based on sampling. Based on reasonable assumptions, such as the robotic arm being within the image, it is applicable to most hand-eye calibration scenarios.

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] In image-based pose estimation scenarios, a known robot is typically considered, which includes available 3D robot models and kinematic models. Unlike pose estimation for general objects, robots consist of multiple parts connected by joints. Due to this unique multi-joint characteristic, it is necessary to define joint angles. d represents the number of joints. Based on the joint angles and the robot model, predefined 3D keypoints can be obtained using robot forward kinematics:

[0051]

[0052] In most robotic applications, joint angles are measured using sensors, and the robot model serves as known prior information. Therefore, the 6D pose T∈SE(3) from the camera to the robotic arm is the only variable determining the robot's state. For known camera intrinsics, projecting the aforementioned keypoints into the image space yields...

[0053]

[0054] The above describes an efficient process for associating 3D and 2D spaces. Compared to previous differentiable rendering, this invention supports online optimization. The unsupervised learning proposed in this invention includes the following stages: pre-training a rendering-based pose estimation network and a keypoint detection network in the synthetic data domain; comparing the projection of the pose with the keypoint detection to fine-tune the two networks in the unlabeled real data domain, using a distribution-based objective function to quantify the difference between them; and finally, approximating a theoretically unsolvable objective distribution function based on sampling integrals.

[0055] Step 1: Pose Estimation Based on Rendering and Contrast

[0056] An initial pose estimation Iteratively fine-tuned by a deep neural network, based on rendering and contrast paradigms. Since a common difficulty is regressing 3D information directly from 2D images, this paradigm first estimates the randomly selected robot portion based on its current pose. Render together, then render the image. With input image The data is concatenated and fed together into a deep convolutional neural network f. θ The changes to the network output pose and the updates to the current pose estimate are as follows:

[0057]

[0058] The aforementioned ΔT follows a decoupled parameterized design method derived from general object pose estimation, specifically designed to be applicable to more easily predictable decoupled rotational and translational features extracted from images. At this point, the parameters θ of the deep convolutional neural network can be fine-tuned using a simple gap loss function to align the pose-projected keypoints with the high-confidence keypoint detections.

[0059]

[0060] The above objective function is effective in using all the useful keypoint detection information to fine-tune the parameters θ of the deep convolutional neural network to make it suitable for the real world domain. However, directly and explicitly utilizing this information has several drawbacks:

[0061] (1) The selection of key point detection requires a user-specified confidence threshold;

[0062] (2) The inference of key point coordinates introduces an unlearnable algorithm, which leads to bias.

[0063] (3) The form of this objective function for fine-tuning convolutional neural networks f θ Overly aggressive approaches can easily lead to a decrease in pose estimation performance, especially when the distance between the pose projection and the key point detection is far.

[0064] Therefore, the global, distribution-divergence-based objective function proposed below is needed to quantify the differences between poses and keypoints with high quality, ensuring the stability of the entire unsupervised learning framework, and without requiring additional processes or processing.

[0065] Step 2: Key Point Detection

[0066] This invention employs a general keypoint detection paradigm to predict keypoints corresponding to parts of the robot, based on the input scene graph. The specific implementation process is as follows: an encoder-decoder structure processes the image to generate a series of d keypoint confidence maps, where d corresponds to the pre-set number of keypoints.

[0067]

[0068] Each confidence map corresponds to a keypoint, represented as a 2D array with height w and width h, where the pixel value represents the probability that the keypoint will be projected onto that pixel. φ This invention represents a keypoint detection network with an encoder-decoder architecture. In most keypoint detection applications, a series of post-processing steps are applied to the confidence map to infer the accurate coordinates of keypoints. The simple gap loss function mentioned above utilizes these steps. While effective, the original confidence map contains more information, potentially useful for fine-tuning convolution-based pose estimation neural networks. These inference processes can also introduce bias when extracting discrete keypoint coordinates, especially in scenarios where the confidence map is relatively flat or in a multimodal context. To address this issue, this invention explores the global information of the confidence map, discovering and representing the possible locations and uncertainties of keypoints. Furthermore, this invention can fine-tune the encoder-decoder network parameters φ by using pose projection as pseudo-labels for keypoints. Because pose estimation contains 3D information relevant to the entire robot, it is highly advantageous for assisting in the detection of uncertain keypoints. This invention sets the pseudo-labels of the confidence map by projecting a Gaussian kernel function onto each keypoint. This is taken as the peak value, and a large variance σ is used for each pixel with coordinates u = (x, y).

[0069]

[0070] Fine-tuning of the keypoint detector is achieved through uncertain confidence plot B. i With pseudo-tags It is driven by the L2 loss function.

[0071] This concludes the introduction of the unsupervised learning framework of the present invention. The network used to integrate pose estimation and key point detection trains itself to significantly bridge the gap between virtual and reality by training on unlabeled real data alone.

[0072] Step 3: Objective function based on distribution divergence

[0073] The core of this invention is robot pose estimation, so fine-tuning the rendering-based pose estimation network is key. However, a simple difference objective function cannot fully utilize the information from keypoint detection. Typically, the detected confidence map can be viewed as a distribution of keypoints, containing fine-grained coordinates and confidence information, which is very helpful for improving the performance of unsupervised learning in depth. In fact, each pixel value in the confidence map represents the probability that the pixel corresponds to a keypoint. Therefore, the keypoint probability density distribution function on the overall confidence map can be obtained by normalizing all pixel values.

[0074]

[0075] The aforementioned keypoint probability density distribution function makes the projected keypoints of the 6D pose... Alignment with keypoint detection can be achieved in a globally consistent manner, rather than merely with inferred coordinates. To quantify the similarity between pose projection and keypoint detection, it is necessary to represent the pose projection as another keypoint probability distribution. The confidence map corresponding to the pose projection, using the same normalized confidence map, is as follows:

[0076]

[0077] Having obtained these two distributions, the global difference between the two keypoint distributions can be quantified using the Kullback-Leibler Divergence formula. This can be used to fine-tune pose estimation and rendering-based pose estimation networks, as long as the distribution divergence is used as the objective function loss.

[0078]

[0079] Note that only 6D pose projection keypoints The parameters are optimized by the aforementioned distribution divergence objective function and then backpropagated to the pose estimation network. Simultaneously, the distribution of detected keypoints is fixed, serving as the alignment target for pose estimation. This objective function significantly improves the stability of unsupervised learning because it is applicable to keypoint detection in various situations.

[0080] Step 4: Integration based on sampling

[0081] The objective function based on the distribution divergence mentioned above can be calculated through its definition.

[0082]

[0083] This integral is difficult to calculate using mathematical analysis because of the distribution pi (x, y) is essentially an unknown distribution expression. Empirically, this unknown distribution can be approximated using a parametric model, but this leads to the loss of information in keypoint detection. Instead, this invention uses a common approach for dealing with analytical unknowns: directly approximating the distribution divergence value based on sampling.

[0084] First, the upper and lower limits of integration, or the sampling region, are determined. This invention restricts it to the confidence map. This assumption is based on the fact that the robot recorded in the image includes all joints and parts within the viewpoint, which is also the most common requirement for robot hand-eye calibration. Therefore, probability values ​​from two distributions are randomly and uniformly sampled within the confidence map to determine the divergence between the pose estimation and keypoint detection distributions as follows:

[0085]

[0086] During the calculation, the value of each pixel u = (x, y) in the confidence map can be directly used to replace the probability density and the network's predicted value, as follows:

[0087]

[0088] In the above equation, const1 and const2 are constant values ​​resulting from the normalization process, which are irrelevant to pose estimation and can therefore be ignored in unsupervised training. Note that...

[0089]

[0090] This implies the relationship between the distribution divergence target and the projected keypoints for pose estimation, which can be used to drive the fine-tuning of pose estimation and rendering-based networks.

[0091] The embodiments described below are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0092] The overall framework of the method of this invention is as follows: Figure 1 As shown, a pose estimation method based on deep learning aims to address the difference between the virtual domain and the real-world target domain. The specific steps are as follows:

[0093] Step 1: Pose Estimation Based on Rendering and Contrast

[0094] An initial pose estimation Iteratively fine-tuned by a deep neural network, based on rendering and contrast paradigms. Since a common difficulty is regressing 3D information directly from 2D images, this paradigm first estimates the randomly selected robot portion based on its current pose. Render together, then render the image. With input image The data is concatenated and fed together into a deep convolutional neural network f. θ The changes to the network output pose and the updates to the current pose estimate are as follows:

[0095]

[0096] The aforementioned ΔT follows a decoupled parameterized design method derived from general object pose estimation, specifically designed to be applicable to more easily predictable decoupled rotational and translational features extracted from images. At this point, the parameters θ of the deep convolutional neural network can be fine-tuned using a simple gap loss function to align the pose-projected keypoints with the high-confidence keypoint detections.

[0097]

[0098] The above objective function is effective in using all the useful keypoint detection information to fine-tune the parameters θ of the deep convolutional neural network to make it suitable for the real world domain. However, directly and explicitly utilizing this information has several drawbacks:

[0099] (1) The selection of key point detection requires a user-specified confidence threshold;

[0100] (2) The inference of key point coordinates introduces an unlearnable algorithm, which leads to bias.

[0101] (3) The form of this objective function for fine-tuning convolutional neural networks f θ Overly aggressive approaches can easily lead to a decrease in pose estimation performance, especially when the distance between the pose projection and the key point detection is far.

[0102] Therefore, the global, distribution-divergence-based objective function proposed below is needed to quantify the differences between poses and keypoints with high quality, ensuring the stability of the entire unsupervised learning framework, and without requiring additional processes or processing.

[0103] Step 2: Key Point Detection

[0104] This invention employs a general keypoint detection paradigm to predict keypoints corresponding to parts of the robot, based on the input scene graph. The specific implementation process is as follows: an encoder-decoder structure processes the image to generate a series of d keypoint confidence maps, where d corresponds to the pre-set number of keypoints.

[0105]

[0106] Each confidence map corresponds to a keypoint, represented as a 2D array with height w and width h, where the pixel value represents the probability that the keypoint will be projected onto that pixel. φ This invention represents a keypoint detection network with an encoder-decoder architecture. In most keypoint detection applications, a series of post-processing steps are applied to the confidence map to infer the accurate coordinates of keypoints. The simple gap loss function mentioned above utilizes these steps. While effective, the original confidence map contains more information, potentially useful for fine-tuning convolution-based pose estimation neural networks. These inference processes can also introduce bias when extracting discrete keypoint coordinates, especially in scenarios where the confidence map is relatively flat or in a multimodal context. To address this issue, this invention explores the global information of the confidence map, discovering and representing the possible locations and uncertainties of keypoints. Furthermore, this invention can fine-tune the encoder-decoder network parameters φ by using pose projection as pseudo-labels for keypoints. Because pose estimation contains 3D information relevant to the entire robot, it is highly advantageous for assisting in the detection of uncertain keypoints. This invention sets the pseudo-labels of the confidence map by projecting a Gaussian kernel function onto each keypoint. This is taken as the peak value, and a large variance σ is used for each pixel with coordinates.

[0107] u = (x, y)

[0108]

[0109] Fine-tuning of the keypoint detector is achieved through uncertain confidence plot B. i With pseudo-tags It is driven by the L2 loss function.

[0110] This concludes the introduction of the unsupervised learning framework of the present invention. The network used to integrate pose estimation and key point detection trains itself to significantly bridge the gap between virtual and reality by training on unlabeled real data alone.

[0111] Step 3: Objective function based on distribution divergence

[0112] The core of this invention is robot pose estimation, so fine-tuning the rendering-based pose estimation network is key. However, a simple difference objective function cannot fully utilize the information from keypoint detection. Typically, the detected confidence map can be viewed as a distribution of keypoints, containing fine-grained coordinates and confidence information, which is very helpful for improving the performance of unsupervised learning in depth. In fact, each pixel value in the confidence map represents the probability that the pixel corresponds to a keypoint. Therefore, the keypoint probability density distribution function on the overall confidence map can be obtained by normalizing all pixel values.

[0113]

[0114] The aforementioned keypoint probability density distribution function makes the projected keypoints of the 6D pose... Alignment with keypoint detection can be achieved in a globally consistent manner, rather than merely with inferred coordinates. To quantify the similarity between pose projection and keypoint detection, it is necessary to represent the pose projection as another keypoint probability distribution. The confidence map corresponding to the pose projection, using the same normalized confidence map, is as follows:

[0115]

[0116] Having obtained these two distributions, the global difference between the two keypoint distributions can be quantified using the Kullback-Leibler Divergence formula. This can be used to fine-tune pose estimation and rendering-based pose estimation networks, as long as the distribution divergence is used as the objective function loss.

[0117]

[0118] Note that only 6D pose projection keypoints The parameters are optimized by the aforementioned distribution divergence objective function and then backpropagated to the pose estimation network. Simultaneously, the distribution of detected keypoints is fixed, serving as the alignment target for pose estimation. This objective function significantly improves the stability of unsupervised learning because it is applicable to keypoint detection in various situations.

[0119] Step 4: Integration based on sampling

[0120] The objective function based on the distribution divergence mentioned above can be calculated through its definition.

[0121]

[0122] This integral is difficult to calculate using mathematical analysis because of the distribution p i (x, y) is essentially an unknown distribution expression. Empirically, this unknown distribution can be approximated using a parametric model, but this leads to the loss of information in keypoint detection. Instead, this invention uses a common approach for dealing with analytical unknowns: directly approximating the distribution divergence value based on sampling.

[0123] First, the upper and lower limits of integration, or the sampling region, are determined. This invention restricts it to the confidence map. This assumption is based on the fact that the robot recorded in the image includes all joints and parts within the viewpoint, which is also the most common requirement for robot hand-eye calibration. Therefore, probability values ​​from two distributions are randomly and uniformly sampled within the confidence map to determine the divergence between the pose estimation and keypoint detection distributions as follows:

[0124]

[0125] During the calculation, the value of each pixel u = (x, y) in the confidence map can be directly used to replace the probability density and the network's predicted value, as follows:

[0126]

[0127] In the above equation, const1 and const2 are constant values ​​resulting from the normalization process, which are irrelevant to pose estimation and can therefore be ignored in unsupervised training. Note that...

[0128]

[0129] This implies the relationship between the distribution divergence target and the projected keypoints for pose estimation, which can be used to drive the fine-tuning of pose estimation and rendering-based networks.

[0130] To further verify the feasibility and effectiveness of the method of this invention, the technical effects of this invention are further described in conjunction with experiments. The hardware platform used in the experiments of this invention is: Intel(R) Xeon(R) Gold 6230 CPU@2.10GHz×16, 64G of memory, and two GPUs with 24G of video memory each, model NVIDIA TITAN RTX. The software platform used in the experiments of this invention is: Ubuntu 18.04.6LTS operating system and Python 3.8.17, PyTorch 1.8.0 deep learning framework. The robot pose estimation effect obtained by this method is analyzed using four real-world datasets. These datasets are for the 7-DOF robot Franka Emika Panda, including the datasets Azure Kinect (AK), XBOX 360 Kinect (XK), Realsense (RS), and Panda-ORB. Here, the mean and integral value of the average distance (ADD) (Area Under Curve, AUC) are used as evaluation metrics to evaluate the performance of pose estimation.

[0131] The method of this invention is used to estimate the pose of robot images from the four datasets mentioned above, obtaining the pose estimation results for each scene. To test the performance of the method of this invention, the proposed deep learning-based pose estimation method is compared with several existing methods, including: DREAM, a pose estimation method based on keypoint detection proposed by Timothy E. Lee et al. in their paper "Camera-to-robot pose estimation from a single image, ICRA, 2020," and RoboPose, a pose estimation method based on rendering proposed by Yann Labbé et al. in their paper "Single-view robot pose and joint angle estimation via render compare, CVPR, 2021." Quantitative results are listed in Table 1. The AUC metric is used as the evaluation standard, with values ​​representing percentages; a higher value is better.

[0132] Table 1. Quantitative test results on the validation set using the method of the present invention and existing technologies.

[0133] method AK XK RS ORB DREAM 60.51 64.01 78.83 69.05 RoboPose 76.49 85.97 76.90 80.29 Method of the present invention 86.95 87.94 87.45 86.21

[0134] As can be seen from the results in Table 1, the test results obtained by the method of the present invention on all four datasets are higher than those of existing methods, indicating that the present invention is more applicable to real-world domains. Qualitative test results are as follows: Figure 3 As shown, the pose results of the method of the present invention are more accurate than those of existing methods.

[0135] The above embodiments should be considered as exemplary and non-limiting, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A robot pose estimation method based on deep learning, characterized in that, Includes the following steps: Step 1, Render-based robot pose estimation: First, select a robotic arm part, render the known robot model into a 2D image based on the current pose estimation, stitch the rendered image together with the observed RGB image of the robot, and feed it as input to a deep convolutional neural network to learn the pose update end-to-end. The predicted pose update makes the rendered image and the observed RGB image consistent and aligned. Then, the pose estimation is projected onto the pixel plane according to the camera extrinsic parameters to obtain the projection key points corresponding to the pose estimation. Step 2, Robot key point detection: The observed RGB image is input into the deep convolutional neural network structure of encoder-decoder to predict the confidence map of each robot key point; Step 3: Transform the projected keypoints of pose estimation and the detected keypoints into a distribution in 2D pixel space. The projected keypoints are first processed to obtain a confidence map consistent with the detected keypoints, and then normalized to obtain the distribution. The detected keypoints can be directly normalized in pixel space to obtain the distribution. Then, an objective function based on the distribution divergence is constructed. Step 4: The divergence objective function is approximated by sampling-based integral, and the Adam learning rate adaptive update algorithm is used to control the optimization of pose estimation, so that the projected key points are aligned with the detected key points in the image. In step 1, the robot pose estimation is based on rendering. Step 1.1, Estimating the current robot pose The renderer uses a non-learning, robot-based 3D CAD model to render the image corresponding to the current pose estimation. Its height is 240 and its width is 320; Step 1.2: Cut the RGB image of the observation robot from 480 in height and 640 in width to 240 in height and 320 in width, consistent with the rendered image; Step 1.3: Concatenate the cut image and the rendered image in the RGB color channels to obtain a feature map with 6 channels, a height of 240, and a width of 320. Step 1.4: Input the feature map into the deep convolutional neural network. In the process of updating the predicted pose estimation: ， The above neural network Residual block connections are used; specifically, for a convolutional module containing convolutional layers, normalization layers, and activation layers, two of this convolutional module are set on the main trunk of the network and one on a branch, and a jump connection is used to obtain residual blocks. Finally, the five residual blocks are stacked to obtain the feature extraction part of the neural network. It also includes a multilayer perceptron (MLP) for pose estimation, which takes the mean of features across pixels after feature extraction and uses it to predict pose updates; additionally, pose updates... Following a decoupled parametric design approach, pose is represented as rotation in the object coordinate system and translation in the camera coordinate system; In step 2, robot key point detection is performed in the RGB image. 2.1 Define the key points of the robotic arm. Define each joint axis connection of the robotic arm as a 3D key point of the robotic arm. Its projection onto the 2D pixel space is the key point of the RGB image of the robotic arm. 2.2 A neural network is trained using an encoder-decoder structure to predict key points, with the RGB image of the robotic arm as input. The encoder uses four downsampling convolutional layers, which include convolution, normalization, and activation operations. The decoder is responsible for upsampling the image features obtained by the encoder to create heatmaps corresponding to the key points, with one heatmap for each key point. The decoder consists of four transposed convolutional layers; this is represented as follows: In the input scene graph... Above, the encoder-decoder processes the image to generate a series of d keypoint confidence maps, where d corresponds to a pre-set number of keypoints. ， Each confidence map corresponds to a key point, which is a 2D array with height w and width h, where the pixel value represents the probability that the key point will be projected onto that pixel. Key point detection network representing the encoder-decoder structure.

2. The robot pose estimation method based on deep learning according to claim 1, characterized in that, In step 3: the method of using the divergence between the two distributions of pose projection and keypoint detection as the objective function, the pose estimate obtained in step 1 is transformed into... Projecting onto pixel space yields the corresponding 2D key points; Using this point as the center of the 2D Gaussian kernel function and a variance of 5, a confidence heatmap corresponding to this projected keypoint is constructed. Together with the detected keypoint heatmap obtained in step two, the probability density distribution function of the keypoints on the overall confidence map is obtained by normalizing all pixel values ​​as follows: ， These two distributions were obtained, and the global difference between the two keypoint distributions was quantified using the KL divergence formula. This difference was then used to fine-tune the pose estimation and rendering-based pose estimation networks, with the distribution divergence used as the objective function loss. ， Only 6D pose projection key points The parameters are optimized by the above-mentioned distribution divergence objective function and then backpropagated to the pose estimation network; at the same time, the distribution of the key points is fixed and used as the alignment target for pose estimation.

3. The robot pose estimation method based on deep learning according to claim 1, characterized in that: In step 4, the divergence objective function is approximated based on the integral of the samples. 4.1 First, we make an assumption applicable to most hand-eye calibration scenarios, namely, that the robotic arm is within the image; 4.2 Within the confidence map, the probability values ​​in the keypoint projection distribution and detection distribution are sampled at all pixel coordinates. The divergence between the pose estimation and keypoint detection publications is approximated as follows: ， During the calculation process, each pixel is used directly. The values ​​in the confidence plot are used to replace the probability density and directly represent the network's predicted values, as follows: ， In the above formula and These are constant values ​​resulting from the normalization process, unrelated to pose estimation, and can be directly set to 1 and 0; 4.3 The above equation is the objective function. The Adam learning rate adaptive update algorithm is used to control the optimization of pose estimation. The updated pose estimation aligns the projected keypoints with the detected keypoints in the image.

Images