A six-dimensional pose estimation method and system for small objects in images

Abstract

The present invention belongs to the technical field of computer vision, and provides a six-dimensional pose estimation method and system for small objects in an image, comprising obtaining an original object image and performing preprocessing to obtain a first object image; using a trained scale-adaptive jumping hole space pyramid pixel point voting network to perform six-dimensional pose estimation on the object in the first object image; wherein the six-dimensional pose estimation process is specifically as follows: based on the first object image, predicting a vector field feature map of an object mask and two-dimensional key points; locating the two-dimensional key points by voting on the object mask and the vector field feature map of the two-dimensional key points; scaling the located two-dimensional key points back to the original object image to solve the six-dimensional pose of the object; adaptively scaling the small objects in the image to a suitable size, and scaling the output 2D key points back to the original image during multi-point perspective imaging, thereby solving the problem that the pose of small objects in the image cannot be accurately predicted in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a six-dimensional pose estimation method and system for small objects in an image. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The 6D pose estimation of an object is to predict the 3D rotation and 3D translation of the object coordinate system relative to the camera coordinate system. In recent years, pose estimation has been required in many fields, such as augmented reality, robot grasping, and autonomous driving. Many traditional methods rely on manually extracted features to establish the correspondence between the image plane and the 3D object model, and then solve the 6D pose of the object through some variant of the PnP (Perspective-N-point) / RANSAC algorithm. However, these methods are sensitive to lighting environment, background clutter, weak texture, target occlusion, etc. In recent years, with the development of deep learning, some work has been devoted to training end-to-end neural networks to directly regress the 6D pose of the object with a single RGB image as input. These direct methods take advantage of the powerful image feature processing capabilities of deep neural networks and achieve impressive results. However, this method makes it difficult to explain how the neural network regresses the 6D pose of the object, so it is not known whether the neural network has learned enough features to achieve generalization applications.

[0004] In recent years, in order to combine the advantages of traditional methods and direct methods, some studies have adopted a keypoint-based method, which first trains a neural network to detect the 2D keypoints of objects in the image, and then solves the 6D pose of the object through a variant of the PnP / RANSAC algorithm. As a further improvement, PVNet proposed a pixel voting strategy that uses a vector field representation for 2D keypoints and uses this voting strategy to locate the final 2D keypoints. These keypoint-based methods have good interpretability and versatility, thanks to the robust detection of 2D keypoints by convolutional neural networks. However, these keypoint-based methods cannot accurately predict the pose of small objects in images. Summary of the invention

[0005] Although keypoint-based methods have recently shown good results, their performance is still poor when dealing with small objects in images. To address this problem, the present invention proposes a six-dimensional pose estimation method and system for small objects in images. The present invention conducts an in-depth pose error analysis on keypoint-based methods and proposes a scale-adaptive skip atrous spatial pyramid pixel voting network (SASA-PVNet), which can adaptively scale small objects in the input image to a suitable size and scale the output 2D keypoints back to the original input image during multi-point perspective imaging (PnP). A skip atrous spatial pyramid (Skip-ASP) module is also designed in the middle of the network, which can reduce image distortion and blur caused by scaling objects in the input image through multi-scale information fusion.

[0006] According to some embodiments, a first solution of the present invention provides a six-dimensional pose estimation method for small objects in an image, using the following technical solution:

[0007] A six-dimensional pose estimation method for small objects in an image, comprising:

[0008] Acquire an original object image and perform preprocessing to obtain a first object image;

[0009] Using the trained scale-adaptive jumping hole spatial pyramid pixel voting network to perform six-dimensional pose estimation on the object in the first object image;

[0010] The process of six-dimensional pose estimation is specifically as follows:

[0011] Based on the first object image, predict an object mask and a vector field of two-dimensional key points;

[0012] The 2D key points are located by voting on the object mask and the vector field of the 2D key points;

[0013] Scale the located two-dimensional key points back to the original object image and solve the six-dimensional pose of the object.

[0014] Furthermore, the obtaining of the original object image and preprocessing to obtain the first object image is specifically as follows:

[0015] Get the original object image;

[0016] Detect objects in the original object image using an object detection algorithm;

[0017] adding 10% padding around the object in the original object image;

[0018] The object-filled image is scaled to obtain a first object image.

[0019] Furthermore, the process of training the scale-adaptive jumping hole spatial pyramid pixel voting network is specifically as follows:

[0020] Get the original image of the object and build the training set and test set;

[0021] Preprocess the training set;

[0022] The preprocessed training set is used to train the scale-adaptive jumping hole spatial pyramid pixel voting network and correct its various parameters.

[0023] Furthermore, the preprocessing of the training set includes:

[0024] Dynamically crop and fill objects in the original images of objects in the training set;

[0025] It is then scaled to the required size to obtain the preprocessed training set.

[0026] Furthermore, the scale-adaptive jumping hole spatial pyramid pixel voting network includes an encoder network, a jumping hole spatial pyramid module and a decoder network;

[0027] The encoder network is used to extract high-level features of the first object image;

[0028] The jumping hole spatial pyramid module is used to perform multi-scale feature fusion on the high-level features of the first object image;

[0029] The decoder network is used to perform convolution and bilinear upsampling on the fused multi-scale features to obtain a first object image feature map;

[0030] A convolution operation is performed on the first object image feature map to output an object mask and a vector field of two-dimensional key points.

[0031] Furthermore, the two-dimensional key points are located by voting on the object mask and the vector field of the two-dimensional key points, specifically:

[0032] Use the object mask to segment the vector field feature map of the 2D key points, ensuring that the vector pointing to each 2D key point comes from the pixel points on the visible surface of the object in the image;

[0033] Randomly select two pixel points and take the intersection of their vectors as a hypothetical point of a two-dimensional key point; repeat the above process to obtain N hypothetical points;

[0034] From the N hypothetical points of each two-dimensional key point, select the hypothetical point with the highest confidence as the predicted two-dimensional key point.

[0035] Furthermore, the located two-dimensional key points are scaled back to the original object image to solve the six-dimensional pose of the object, specifically:

[0036] Scaling the coordinates of the two-dimensional key points based on the inverse of the scaling ratio when the original image is scaled to the first object image;

[0037] According to the position of the upper left corner of the image after object filling in the original image, the located two-dimensional key points are scaled back to the original image;

[0038] The six-dimensional pose of the object is solved based on the camera intrinsic parameters of the original image, the predicted two-dimensional key points, and the key points on the preset object 3D model.

[0039] According to some embodiments, a second solution of the present invention provides a six-dimensional pose estimation system for small objects in an image, which adopts the following technical solution:

[0040] A six-dimensional pose estimation system for small objects in an image, comprising:

[0041] An image acquisition module is configured to acquire an original object image and perform preprocessing to obtain a first object image;

[0042] An image object pose estimation module is configured to perform six-dimensional pose estimation on the object in the first object image using a trained scale-adaptive jump hole spatial pyramid pixel voting network;

[0043] The process of six-dimensional pose estimation is specifically as follows:

[0044] Based on the first object image, predict an object mask and a vector field of two-dimensional key points;

[0045] The 2D key points are located by voting on the object mask and the vector field of the 2D key points;

[0046] Scale the located two-dimensional key points back to the original object image and solve the six-dimensional pose of the object.

[0047] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a six-dimensional pose estimation method for small objects in an image as described in the first aspect above.

[0049] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0050] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a six-dimensional pose estimation method for small objects in an image as described in the first aspect above are implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. Estimating the 6-dimensional pose of an object from a single RGB image is a challenging task in the field of computer vision. Although keypoint-based methods have recently shown good results, their performance is still poor when dealing with small objects in images. To address this problem, the present invention conducts an in-depth pose error analysis of keypoint-based methods and proposes a scale-adaptive skip atrous spatial pyramid pixel voting network (SASA-PVNet), which can adaptively scale small objects in the input image to a suitable size and scale the output 2D keypoints back to the original input image during multi-point perspective imaging (PnP). A skip atrous spatial pyramid (Skip-ASP) module is also designed in the middle of the network, which can reduce image distortion and blurring caused by scaling objects in the input image through multi-scale information fusion.

[0053] 2. The present invention conducts an in-depth analysis of the pose error of the key point-based method and finds that the accuracy of object pose estimation can be improved not only by reducing the prediction error of the 2D key points of the object in the image, but also by scaling the size of the object in the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0055] Figure 1 is a flow chart of a six-dimensional pose estimation method for small objects in an image according to Embodiment 1 of the present invention;

[0056] FIG. 2( a ) is a schematic diagram of the original image preprocessing process according to the first embodiment of the present invention;

[0057] FIG2( b ) is a scale-adaptive jumping hole spatial pyramid pixel voting network diagram according to the first embodiment of the present invention;

[0058] FIG2( c ) is a schematic diagram of a jumping hole space pyramid module according to Embodiment 1 of the present invention;

[0059] Figure 3 is a diagram of the size change of the experimental object for the posture error analysis described in the first embodiment of the present invention;

[0060] Figure 4 1 is a graph showing the experimental results of the posture error analysis described in Example 1 of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0063] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0064] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0065] Embodiment 1

[0066] This embodiment conducts an in-depth analysis of the pose error of the key point-based method and finds that improving the pose estimation accuracy can be achieved by reducing the prediction error of the 2D key points or by scaling the size of the object in the input image. However, most of the key point-based methods currently focus only on reducing the prediction error of the 2D key points. Based on the analysis, this embodiment proposes a new network framework, the Scale Adaptive Skip-ASP Pixel Voting Network (SASA-PVNet), which can adaptively scale the objects in the input image to a suitable size and then scale the output 2D key points back to the original input image during PnP. The framework of this embodiment significantly improves the accuracy of estimating the pose of small objects in the image. However, scaling the image will cause severe image distortion and blurring, especially at large scaling rates, which limits the performance of the network.

[0067] In order to reduce the impact of image distortion and blur, this embodiment designs a Skip Atrous Spatial Pyramid (Skip-ASP) module between the encoder and decoder of the network. DeepLabv3+ proves that the Atrous Spatial Pyramid Pooling (ASPP) module can capture multi-scale feature information. This embodiment believes that image distortion and blur can be repaired by multi-scale information fusion. However, this embodiment finds that the pooling branch and large atrous rate in the ASPP module interfere with the neural network's prediction of the vector field of 2D key points. Therefore, this embodiment removes the Pooling branch in the ASPP module and adds a skip connection (SkipConnection) to the remaining Atrous Spatial Pyramid (ASP) module.

[0068] The method of this embodiment is tested on two widely used benchmark datasets: LINEMOD and Occlusion LINEMOD. Experiments show that the method of this embodiment performs far better than the most advanced methods on the LINEMOD dataset and achieves competitive results on the Occlusion LINEMOD dataset.

[0069] like Figure 1 As shown, this embodiment provides a six-dimensional pose estimation method for small objects in an image, the method comprising the following steps:

[0070] Step S1: Acquire an original object image and perform preprocessing to obtain a first object image;

[0071] Step S2: using the trained scale-adaptive jumping hole spatial pyramid pixel voting network to perform six-dimensional pose estimation on the object in the first object image;

[0072] Among them, in step S2, the process of six-dimensional pose estimation is specifically as follows:

[0073] Step S2.1: predicting an object mask and a vector field of two-dimensional key points based on the first object image;

[0074] Step S2.2: Locate the two-dimensional key points by voting on the object mask and the vector field of the two-dimensional key points;

[0075] Step S2.3: Scale the located two-dimensional key points back to the original object image and solve the six-dimensional pose of the object.

[0076] Specifically, in step S1, the original object image is acquired and preprocessed to obtain a first object image, specifically:

[0077] Get the original object image;

[0078] Detect objects in the original object image using an object detection algorithm;

[0079] adding 10% padding around the object in the original object image;

[0080] The object-filled image is scaled to obtain a first object image.

[0081] Specifically, in step S2, the process of training the scale-adaptive jumping hole spatial pyramid pixel voting network is as follows:

[0082] Get the original image of the object and build the training set and test set;

[0083] Preprocess the training set;

[0084] The preprocessed training set is used to train the scale-adaptive jumping hole spatial pyramid pixel voting network and correct its various parameters.

[0085] Wherein, the preprocessing of the training set includes:

[0086] Dynamically crop and fill objects in the original images of objects in the training set;

[0087] It is then scaled to the required size to obtain the preprocessed training set.

[0088] In step S2, the scale-adaptive jumping hole spatial pyramid pixel voting network includes an encoder network, a jumping hole spatial pyramid module and a decoder network;

[0089] The encoder network is used to extract high-level features of the first object image;

[0090] The jumping hole spatial pyramid module is used to perform multi-scale feature fusion on the high-level features of the first object image;

[0091] The decoder network is used to perform convolution and bilinear upsampling on the fused multi-scale features to obtain a first object image feature map;

[0092] A convolution operation is performed on the first object image feature map to output an object mask and a vector field of two-dimensional key points.

[0093] As shown in Figure 2(b), the framework of the scale-adaptive skip hole spatial pyramid pixel voting network (SASA-PVNet) proposed in this embodiment;

[0094] The input image of this embodiment is a preprocessed first object image. Specifically, during the training process, the object in the original object image is dynamically cropped and filled, and then scaled to a desired size to obtain the first object image; during the testing process, this embodiment uses an off-the-shelf target detector to crop and fill the object in the original object image.

[0095] The network of this embodiment is an encoder-jump hole spatial pyramid module-decoder structure. Similar to PVNet, the encoder is a pre-trained ResNet-18, and the decoder is several convolutional layers and upsampling layers. Given an RGB image as input, the encoder module of this embodiment is used to extract high-level features. Specifically, in order to obtain an accurate vector field of 2D key points, this embodiment reduces the output step size of ResNet-18 from 32 to 8, and replaces the last two residual blocks with hole convolutions with atrous rate (Atrous Rate) = 2 and 4, respectively. After that, this embodiment sends the high-level features to the jump hole spatial pyramid module for multi-scale feature fusion, and then performs three consecutive convolutions and bilinear upsampling. Finally, this embodiment performs 1×1 convolution on the feature map and outputs the object mask and the vector field of 2D key points.

[0096] In step S2, the jumping hole spatial pyramid module in the scale-adaptive jumping hole spatial pyramid pixel voting network is specifically:

[0097] The input of the scale-adaptive skip atrous spatial pyramid pixel voting network described in this embodiment is a scaled image, which may suffer from severe distortion and blur. To solve this problem, this embodiment designs a skip atrous spatial pyramid (Skip-ASP) module between the encoder and decoder of the network. We believe that the image distortion and blur caused by the scaled image can be repaired by multi-scale information fusion, however, we find that the pooling branch and large atrous rate of the ASPP module interfere with the neural network's prediction of the vector field of the 2D key points.

[0098] Therefore, we abandon the Pooling branch of the ASPP module and add a skip connection to the ASP (Atrous Spatial Pyramid) module.

[0099] As shown in Figure 2(c), the Skip-ASP module includes:

[0100] (a) A 1×1Conv-BN-ReLU;

[0101] (b) Three 3×3 AtrousConv-BN-ReLUs, with dilation rate = (6, 12, 18);

[0102] (c) Skip connection.

[0103] Then, the (a, b) branches are concatenated and fed into another 1×1 Conv-BN-ReLU to achieve the required spatial dimension. In this network, in order to ensure the dominance of the main channel, the output spatial size of the Skip-ASP module described in this embodiment is one-fourth of the input. Finally, the fused features are concatenated with the main channel.

[0104] Scale-adaptive input and return

[0105] We found that the pose estimation accuracy of small objects can be improved by enlarging small objects in the input image. In this section, we present a scale-adaptive input and return strategy based on the keypoint approach.

[0106] As shown in Figure 2(a), for objects of different sizes in the input image, we first crop and pad them. To make any object detector robust, we add 10% padding to the detected objects before cropping. Afterwards, to balance the running time of the entire network, we choose to scale the objects to 240×240 pixels. Note that to reduce training time, we do not use an object detector during training. Instead, we use dynamic cropping and padding during training to accommodate detectors with different performance. Finally, we record the cropping location (x, y) and scale S.

[0107] As shown in Figure 2(b), the scaled image is fed into our network, which outputs the vector field of the 2D key points and the object mask (Mask). After voting, the 2D key points in the scaled image are obtained. After that, in the scale adaptive return module, we first scale the 2D key points by 1 / S, and then add the cropping position (x, y), so that the 2D key points can be adaptively scaled to the original input image.

[0108] In step S2.2, the two-dimensional key points are located by voting on the object mask and the vector field of the two-dimensional key points, specifically:

[0109] Step S2.2.1: Use the object mask to segment the vector field feature map of the two-dimensional key points, ensuring that the vector pointing to each two-dimensional key point comes from the pixel points on the visible surface of the object in the image;

[0110] Step S2.2.2: Randomly select two pixel points and take the intersection of their vectors as a hypothetical point of a two-dimensional key point; repeat the above process to obtain N hypothetical points;

[0111] From the N hypothetical points of each two-dimensional key point, select the hypothetical point with the highest confidence as the predicted two-dimensional key point.

[0112] Specifically, the process of obtaining the hypothesis points was repeated 128 times, that is, for each 2D key point, we obtained 128 hypothesis points;

[0113] From the 128 hypothetical points of each 2D key point, select the point with the highest confidence as the predicted 2D key point. The number of repetitions is determined based on experience. The more repetitions, the better and more robust the effect, but the greater the computation time and cost. So in order to balance the effect and cost, 128 is selected based on experience.

[0114] In step S2.2.2, the confidence calculation method is:

[0115] For a pixel point p on the surface of an object, its pointing vector to the two-dimensional key point is X1 (obtained by the output of the neural network), and its pointing vector X2 to the hypothetical point is calculated by the coordinates of the pixel point and the coordinates of the hypothetical point;

[0116] Calculate the angle between X1 and X2. If the angle is less than 8.1°, the hypothesis point is considered credible and the confidence level is +1. Otherwise, the confidence level is +0.

[0117] Repeat the above process so that all pixels on the surface of the object participate in the above process. Finally, the hypothesis point with the highest confidence is selected as the predicted two-dimensional key point.

[0118] Step S2.3: scaling the located two-dimensional key points back to the original object image to solve the six-dimensional pose of the object, specifically:

[0119] Step S2.3.1: scaling the coordinates of the two-dimensional key points based on the inverse of the scaling ratio when the original image is scaled to the first object image;

[0120] When the original object image is preprocessed to obtain the first object image, we first add a certain amount of filler to the object and then crop it from the original object image. The cropped data is (x, y, h, w), where (x, y) is the position of the upper left corner of the cropped image in the original object image, and (h, w) is the length and width of the cropped image; then, we scale the cropped image by S times to obtain the first object image. From the above, we get the data (x, y) and S during preprocessing.

[0121] We need to scale the 2D key points back to the original object image to perform the PnP algorithm, because we can only get the camera intrinsics of the original object image. The scaling process is: first, we scale the coordinates of the 2D key points by 1 / S times, and then we add (x, y) to scale the located 2D key points from the first object image back to the original object image.

[0122] Step S2.3.2: scaling the located two-dimensional key points back to the original image according to the position of the upper left corner of the image after object filling in the original image;

[0123] Step S2.3.3: Based on the camera intrinsic parameters of the original image, the predicted two-dimensional key points, and the key points on the preset object 3D model, the six-dimensional pose of the object is solved through the PnP algorithm in the CV2 library in Python.

[0124] Because the PnP algorithm is used to solve the object pose, a necessary condition of the algorithm is to know the camera intrinsic parameters. The camera intrinsic parameters correspond one-to-one to the original image taken by the camera. If the original image is cropped, scaled, etc., the camera intrinsic parameters cannot be used. The camera intrinsic parameters corresponding to the new image cannot be obtained, so the two-dimensional key points need to be scaled back to the original image. Based on the predicted coordinate positions of the two-dimensional key points in the original image, the known camera intrinsic parameters, and the preset 3D model key points, the pose can be solved by the PnP algorithm.

[0125] Among them, in step S3.3, the pose error analysis based on the key point method is as follows:

[0126] In this section, through mathematical analysis of the PnP algorithm process, we reveal the reason why keypoint-based methods are difficult to accurately predict the pose of small objects in images.

[0127] Given a monocular camera, assuming that the center point of the object is on the optical axis of the camera, only the translational motion of the object on the optical axis of the camera is considered. According to the principle of camera projection, we can derive the following formula:

[0128]

[0129] Among them, S pix Represents the size of the object in the pixel coordinate system, S c Represents the size of the object in the camera coordinate system, d c Represents the distance from the center point of the object to the optical center of the camera, and K1 is a constant coefficient.

[0130] Assume that the 2D keypoint error predicted by the neural network is δ pix , according to the camera projection transformation principle, the error in the object coordinate system can be deduced as:

[0131] δ o =K2·δ pix ·d c (2)

[0132] Among them, δ o is the error in the object coordinate system, K2 is the constant coefficient. According to equations (1) and (2), we can obtain:

[0133]

[0134] Among them, K=K1·K2.

[0135] Through the above derivation, we get the formula (3) for calculating the error of the key points of the object. According to the principle of the PnP (Perspective-n-Point) algorithm, we can infer that the pose error of the object roughly conforms to formula (3). In addition, when we judge whether the pose is accurate, we usually use relative error based on the size of the object itself. Relative error δ re Defined as:

[0136]

[0137] Note that for simplicity, the above derivation does not consider the rotation of the object and the translation of its center of mass not on the optical axis of the camera. At the same time, different PnP algorithms have different optimization effects on pose estimation. Therefore, considering all influencing factors, the relative error of the object's pose can be roughly expressed as:

[0138]

[0139] According to formula (5), we can conclude that keypoint-based methods can reduce the predicted pose error by improving the accuracy of predicted 2D keypoints and enlarging the size of objects in the image. However, most keypoint-based methods focus on reducing the estimation error of 2D keypoints.

[0140] Experimental verification

[0141] 1. Dataset

[0142] This embodiment conducts experiments on two datasets: LINEMOD and Occlusion LINEMOD.

[0143] LINEMOD is a benchmark dataset for 6D pose estimation of objects, consisting of 13 objects and a total of 15,783 RGB images. The main challenges of this dataset are: cluttered background, weak texture, changing lighting conditions and small objects.

[0144] Occlusion LINEMOD is also a benchmark dataset for 6D pose estimation of objects. It consists of 8 objects and a total of 1214 RGB images, each of which contains annotations for multiple objects. Compared with LINEMOD, the objects in this dataset have severe occlusion.

[0145] 2. Evaluation Metrics

[0146] This embodiment uses two standard evaluation indicators: ADD(-S), n°, n cm;

[0147] ADD(-S) indicator. This indicator consists of ADD and ADD-S. For asymmetric objects, the ADD indicator is used to measure whether the average distance between the 3D model points converted from the estimated pose and the 3D model points of the true pose is less than 10% (0.1d) of the object diameter. For symmetric objects, the ADD-S indicator uses the nearest neighbor distance to measure the average distance. Note that, consistent with GDR-Net, the experiments in this embodiment also use stricter 0.05d and 0.02d indicators.

[0148] n°, n cm indicator. This indicator measures whether the rotation error is less than n° and whether the translation error is less than n cm. In the experiment of this embodiment, this embodiment uses the widely used 5°, 5 cm indicator and the more stringent 2°, 2 cm indicator.

[0149] The ADD(-S) indicator can evaluate the relative error of the pose estimation, while the n°,n cm indicator can evaluate the absolute error of the pose estimation. Therefore, this embodiment can evaluate whether the pose estimator can be applied to various situations through these two indicators. In addition, this embodiment also uses two very strict metrics, ADD(-S)0.02d and 2°,2cm, which can be used to evaluate whether the pose estimator can be used in situations requiring high precision, such as robot grasping or other high-level tasks.

[0150] 3. Experimental Details

[0151] Training set and test set. For fair comparison, the training set and test set of all experiments in this embodiment are consistent with the baseline method PVNet of this embodiment. Specifically, this embodiment uses 15% of the LINEMOD dataset for training and 85% for testing, while the Occlusion LINEMOD dataset is only used for testing. In addition, this embodiment uses 10,000 rendered images for each object and 10,000 "cut and paste" synthetic images for all objects.

[0152] Training details. All experiments in this example are implemented in Pytorch. This example uses Adamoptimizer to train the network of this example, with a batch size of 24, a base learning rate of 1e-3, and a learning rate divided by 0.94 at each epoch until it reaches 1e-5. Please note that this example only trains the network for 80 epochs, while PVNet is trained for 200 epochs.

[0153] 4. Comparison with State-of-the-Art Methods

[0154] On these two datasets, this embodiment compares the method of this embodiment with the most advanced RGB image based methods.

[0155] Results on the LINEMOD dataset. In Table 1, the method of this embodiment achieves the best performance on the 5°5cm and 2°2cm metrics without pose optimization. Note that compared with GDR-Net using the stricter 2°2cm metric, this embodiment improves from 62.1% to 89.2%.

[0156] In Table 2, the method of this embodiment outperforms other competitors and even outperforms the method using pose optimization. Since the ADD(-S) indicator is widely used, this embodiment lists detailed experimental results for comparison. Specifically, for each type of object in the dataset, the method of this embodiment reaches the state-of-the-art level in 10 out of 13 objects. In particular, compared with the baseline method PVNet of this embodiment, the accuracy of the method of this embodiment for small objects "ape" and "duck" is improved by 38% and 33.6%, respectively, which shows that the method of this embodiment can indeed accurately estimate the pose of small objects.

[0157] Table 3 shows the comparison of the method of this embodiment with the most advanced method on the LINEMOD dataset, using more stringent ADD(-S) indicators, including ADD(-S)0.05d and ADD(-S)0.02d. As far as the present embodiment knows, GDR-Net is the first to use these two more stringent indicators to evaluate whether the pose estimator can be used for high-precision situations. Compared with the baseline method GDR-Net of this embodiment, the method of this embodiment improves by 8.6% and 14.8% for these two indicators. This shows that the method of this embodiment is very suitable for high-precision applications.

[0158] Table 1: Comparison results with the state-of-the-art methods on the LINEMOD dataset using the n°n cm metric

[0159]

[0160] Table 2: Comparison results with state-of-the-art methods on the LINEMOD dataset using the ADD(-S) metric. Baseline methods: DPOD, HybridPose, PVNet, GDR-Net.

[0161]

[0162]

[0163] Table 3: Comparison results with state-of-the-art methods on the LINEMOD dataset using the more stringent ADD(-S) metric

[0164]

[0165] Table 4: Comparison with state-of-the-art RGB-based methods on the Occlusion LINEMOD dataset using the ADD(-S) metric. Baseline methods: DPOD, HybridPose, PVNet, GDR-Net.

[0166]

[0167] Results on the Occlusion LINEMOD dataset. As shown in Table 4, compared with keypoint-based methods such as PVNet, HybridPose, and DPOD, the method of this embodiment is also the most advanced. However, the method of this embodiment is lower than the direct regression method GDR-Net. This embodiment believes that methods like GDR-Net, which first predict intermediate features and then directly regress, perform better for occluded objects. This embodiment infers that this may be because the direct regression method can remember the appearance of objects that are severely occluded.

[0168] 5. Ablation Experiment

[0169] This embodiment conducts ablation experiments to explore the effects of different components in the method proposed in this embodiment. As shown in Table 5, PVNet and DPVL are the benchmark methods of this embodiment. DPVL uses L pv_dp The loss function is used to replace the loss function of PVNet and achieves better performance. Therefore, in this work, this embodiment also uses L pv_dp Loss function.

[0170] This embodiment first studies the effectiveness of the scale adaptation strategy proposed in this embodiment. Compared with the baseline method of this embodiment, Ours (SA_PVNet) has been greatly improved. Then this embodiment studies the role of the Skip-ASP module proposed in this embodiment. Compared with Ours (SA_PVNet), Ours (SASA_PVNet) has improved in various indicators, especially in the two more stringent indicators of ADD (-S) 0.02d and 2 ° 2cm. This shows that the Skip-ASP module of this embodiment can reduce the effects of distortion and blur caused by image scaling, thereby improving the network performance of this embodiment.

[0171] Table 5: Ablation experiments on the LINEMOD dataset using different metrics. Ours (SA_PVNet) represents the scale-adaptive pixel voting network of this embodiment, and Ours (SASA_PVNet) represents the scale-adaptive jump hole spatial pyramid pixel voting network of this embodiment.

[0172]

[0173] 6. Posture error analysis experiment based on key point method

[0174] like Figure 3 As shown, this embodiment uses "cat" in the LINEMOD data set to verify the conclusion formula (5). Figure 4 As shown, in this embodiment, the error of the 2D key point is manually set to 1-3 pixels to observe the ADD (average distance deviation of the object's 3D model points) error corresponding to objects of different sizes.

[0175] like Figure 4 As shown, as the object becomes larger in the image, the same pixel error will become smaller when transformed into the camera coordinate system. The experimental results of this embodiment confirm the correctness of the mathematical derivation of this embodiment.

[0176] Note that this embodiment selects the maximum side length of an object in the image to represent the size of the object.

[0177] This embodiment conducts an in-depth pose error analysis on the key point based method and proposes a novel scale adaptive skip atrous spatial pyramid pixel voting network (SASA-PVNet) to improve the pose estimation accuracy of small objects in images. In particular, a scale adaptive input and return strategy is proposed on the key point based method, and a skip atrous spatial pyramid (Skip-ASP) module is designed. Experimental results show that the proposed SASA-PVNet can significantly improve the pose estimation accuracy of small objects.

[0178] Embodiment 2

[0179] This embodiment provides a six-dimensional pose estimation system for small objects in an image, including:

[0180] An image acquisition module is configured to acquire an original object image and perform preprocessing to obtain a first object image;

[0181] An image object pose estimation module is configured to perform six-dimensional pose estimation on the object in the first object image using a trained scale-adaptive jump hole spatial pyramid pixel voting network;

[0182] The process of six-dimensional pose estimation is specifically as follows:

[0183] Based on the first object image, predict an object mask and a vector field of two-dimensional key points;

[0184] The 2D key points are located by voting on the object mask and the vector field of the 2D key points;

[0185] Scale the located two-dimensional key points back to the original object image and solve the six-dimensional pose of the object.

[0186] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0187] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0188] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0190] Embodiment 3

[0191] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of a six-dimensional pose estimation method for small objects in an image as described in the first embodiment above are implemented.

[0192] Embodiment 4

[0193] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in a six-dimensional pose estimation method for small objects in an image as described in the first embodiment above are implemented.

[0194] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0195] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0199] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A six-dimensional pose estimation method for small objects in an image, characterized in that: include: The original object image is obtained and preprocessed to obtain the first object image, specifically: Get the original object image; Detect objects in the original object image using an object detection algorithm; Adding 10% padding around the object in the original object image; Scaling the object-filled image to obtain a first object image; Using the trained scale-adaptive jumping hole spatial pyramid pixel voting network to perform six-dimensional pose estimation on the object in the first object image; The process of six-dimensional pose estimation is specifically as follows: Based on the first object image, predict an object mask and a vector field feature map of two-dimensional key points; The 2D key points are located by voting on the object mask and the vector field feature map of the 2D key points; Scale the located two-dimensional key points back to the original object image to solve the six-dimensional pose of the object; The scale-adaptive jumping hole spatial pyramid pixel voting network includes an encoder network, a jumping hole spatial pyramid module and a decoder network; The encoder network is used to extract high-level features of the first object image; The jumping hole spatial pyramid module is used to perform multi-scale feature fusion on the high-level features of the first object image; The decoder network is used to perform convolution and bilinear upsampling on the fused multi-scale features to obtain a first object image feature map; A convolution operation is performed on the first object image feature map to output an object mask and a vector field feature map of two-dimensional key points.

2. A six-dimensional pose estimation method for small objects in an image as claimed in claim 1, characterized in that: The process of training the scale-adaptive jumping hole spatial pyramid pixel voting network is specifically as follows: Get the original image of the object and build the training set and test set; Preprocess the training set; The preprocessed training set is used to train the scale-adaptive jumping hole spatial pyramid pixel voting network and correct its various parameters.

3. A six-dimensional pose estimation method for small objects in an image as claimed in claim 2, characterized in that: The preprocessing of the training set includes: Dynamically crop and fill objects in the original images of objects in the training set; It is then scaled to the required size to obtain the preprocessed training set.

4. A six-dimensional pose estimation method for small objects in an image as claimed in claim 1, characterized in that: The two-dimensional key points are located by voting on the object mask and the vector field feature map of the two-dimensional key points, specifically: Use the object mask to segment the vector field feature map of the 2D key points, ensuring that the vector pointing to each 2D key point comes from the pixel points on the visible surface of the object in the image; Randomly select two pixel points and take the intersection of their vectors as a hypothetical point of a two-dimensional key point; repeat the above process to obtain N hypothetical points; From the N hypothetical points of each two-dimensional key point, select the hypothetical point with the highest confidence as the predicted two-dimensional key point.

5. A six-dimensional pose estimation method for small objects in an image as claimed in claim 1, characterized in that: The located two-dimensional key points are scaled back to the original object image to solve the six-dimensional pose of the object, specifically: Scaling the coordinates of the two-dimensional key points based on the inverse of the scaling ratio when the original image is scaled to the first object image; According to the position of the upper left corner of the image after object filling in the original image, the located two-dimensional key points are scaled back to the original image; The six-dimensional pose of the object is solved based on the camera intrinsic parameters of the original image, the predicted two-dimensional key points, and the key points on the preset object 3D model.

6. A six-dimensional pose estimation system for small objects in an image, characterized in that: include: The image acquisition module is configured to acquire the original object image and perform preprocessing to obtain the first object image, specifically: Get the original object image; Detect objects in the original object image using an object detection algorithm; Adding 10% padding around the object in the original object image; Scaling the object-filled image to obtain a first object image; An image object pose estimation module is configured to perform six-dimensional pose estimation on the object in the first object image using a trained scale-adaptive jump hole spatial pyramid pixel voting network; The process of six-dimensional pose estimation is specifically as follows: Based on the first object image, predict an object mask and a vector field of two-dimensional key points; The 2D key points are located by voting on the object mask and the vector field of the 2D key points; Scale the located two-dimensional key points back to the original object image to solve the six-dimensional pose of the object; The scale-adaptive jumping hole spatial pyramid pixel voting network includes an encoder network, a jumping hole spatial pyramid module and a decoder network; The encoder network is used to extract high-level features of the first object image; The jumping hole spatial pyramid module is used to perform multi-scale feature fusion on the high-level features of the first object image; The decoder network is used to perform convolution and bilinear upsampling on the fused multi-scale features to obtain a first object image feature map; A convolution operation is performed on the first object image feature map to output an object mask and a vector field feature map of two-dimensional key points.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a six-dimensional pose estimation method for small objects in an image as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a six-dimensional pose estimation method for small objects in an image as described in any one of claims 1-5 are implemented.

Images