Two-stage object six-degree-of-freedom pose estimation method based on monocular camera

Abstract

The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera solves the problem of how to achieve high-precision and high-efficiency pose estimation of weak-texture industrial parts, and belongs to the technical field of monocular camera pose estimation. The present invention is a two-stage pose estimation method from coarse to fine. The coarse pose estimation network roughly estimates the six-degree-of-freedom pose value based on the fusion of mask features and 2D position features. The coarse pose estimation stage provides the initial value for precise pose estimation, avoiding the optimization process of the precise pose estimation algorithm from falling into the local optimal value. The precise pose estimation network introduces a dynamic snake-shaped convolution layer to construct an object contour feature extraction network, improves the quality of contour feature generation, and filters out background contours. Through the differentiable Gauss-Newton optimization method, iterative optimization is performed at multiple scales to minimize the reprojection error between the projection contour and the object contour, thereby achieving accurate pose estimation.

Description

Technical Field

[0001] The present invention relates to a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, belonging to the technical field of monocular camera pose estimation. Background Art

[0002] Pose estimation of weakly textured parts is a challenging task, mainly because the surface features of weakly textured parts are not obvious, and changes in lighting conditions as well as occlusion and interference in actual application scenarios will affect the accuracy of pose estimation.

[0003] There are currently two main approaches for pose estimation of weakly textured parts: deep learning-based pose estimation and feature matching-based pose estimation. Deep learning-based approaches use deep neural networks to learn from images to obtain object pose information, but this method requires a large amount of labeled data for training and has high algorithmic overhead. Deep learning-based pose estimation algorithms have significantly improved real-time performance, but due to the weak texture characteristics of industrial parts, estimation accuracy is low. Summary of the Invention

[0004] To address the problem of how to achieve high-precision and high-efficiency pose estimation of weak-texture industrial parts, the present invention provides a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera.

[0005] The present invention provides a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, comprising:

[0006] The monocular camera RGB image of the object to be measured is input into the coarse pose estimation network and the fine pose estimation network at the same time;

[0007] The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image and inputs the coarse pose estimation value as the initial pose to the precise pose estimation network; the coarse pose estimation value includes the translation vector and the rotation quaternion q;

[0008] The coarse pose estimation network includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network;

[0009] The RGB image from the monocular camera is input into the multi-scale feature encoder to extract features of different scales and input into the mask prediction branch network;

[0010] The mask prediction branch network recovers the object binary mask from features of different scales through upsampling;

[0011] The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the feature of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) Using the pinhole camera model to recover the translation vector;

[0012] The rotation prediction branch network uses the object binary mask as the region of interest, performs regional ROI pooling on the features of the multi-scale feature encoder output at a scale of 512, and positionally encodes the object binary mask to obtain a 2D position feature encoding at a scale of 256. Feature embedding and fusion are performed based on the ROI pooled features and the 2D position feature encoding at a scale of 256. After fully connecting the fused features, the rotation quaternion q is obtained. The true pose value of the coarse pose estimation network and the fine pose estimation network during training is obtained through pose estimation based on template matching.

[0013] The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose.

[0014] As a preference, feature embedding fusion is performed using a position-aware state space network based on the features after ROI pooling and the 2D position feature encoding of scale 256 to obtain fused features;

[0015] The position-aware state-space network includes a first normalization layer LN, a first linear layer Linear, a first activation function Silu, a depth-separable convolutional layer DWConv, a global receptive field module SS2D, a second normalization layer LN, a second linear layer Linear, a second activation function Silu, a third normalization layer LN, a third linear layer Linear and a fusion layer;

[0016] The features after ROI pooling pass through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depth-wise separable convolution layer DWConv, the global receptive field module SS2D, and the second normalization layer LN.

[0017] The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN;

[0018] The output of the No. 2 normalization layer LN and the output of the No. 3 normalization layer LN are multiplied and then input into the No. 3 linear layer Linear;

[0019] The output of the Linear layer 3 and the feature after ROI pooling are fused into the fusion layer to obtain the fused features.

[0020] Preferably, the accurate pose estimation network includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network and a differentiable Newton optimization module;

[0021] Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map;

[0022] Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales;

[0023] The sampled contour feature map and contour feature maps of different scales are simultaneously input into the edge probability map prediction module, which performs statistically-based contour extraction on the input contour feature map to generate an edge probability map.

[0024] The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate.

[0025] Preferably, the contour feature extraction network is a contour feature extraction network based on dynamic snake convolution, including 5 encoders, 5 decoders, 10 bottleneck layers, 1 standard two-dimensional convolution layer, 5 two-dimensional transposed convolution layers and 4 snake convolution layers;

[0026] The monocular camera RGB image is input to the first encoder. The outputs of the five encoders are input to the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolutional layer.

[0027] The outputs of the first four encoders each enter a snake convolution layer;

[0028] The output of the standard 2D convolution layer enters the first 2D transposed convolution layer. The output of the first 2D transposed convolution layer and the output of the fourth snake convolution layer are concatenated and enter the first decoder. The output of the first decoder enters the second 2D transposed convolution layer after a bottleneck layer. The output of the second 2D transposed convolution layer and the output of the third snake convolution layer are concatenated and enter the second decoder. The output of the second decoder enters the third 2D transposed convolution layer after a bottleneck layer. The output of the convolution layer is concatenated with the output of the second snake convolution layer and then enters the third decoder. The output of the third decoder enters the fourth two-dimensional transposed convolution layer after passing through a bottleneck layer. The output of the fourth two-dimensional transposed convolution layer is concatenated with the output of the first snake convolution layer and then enters the fourth decoder. The output of the fourth decoder enters the fifth two-dimensional transposed convolution layer after passing through a bottleneck layer. The output of the fifth two-dimensional transposed convolution layer enters the fifth decoder. The outputs of the third, fourth, and fifth decoders are 320 2 , 160 2 , 80 2 High-resolution feature maps.

[0029] Preferably, the encoder is a pre-trained MobileNetV2.

[0030] As a preferred method, the line corresponding model template library is obtained by:

[0031] Projection rendering is performed based on the object CAD model to generate the object contours and normals corresponding to the contour points under different viewing angles offline to form a contour template library.

[0032] Preferably, the pose true values ​​of the coarse pose estimation network and the fine pose estimation network during training are obtained by pose estimation based on template matching, and the method includes:

[0033] Based on the object CAD model, a pose hierarchy template library of object edge information is constructed. The edge templates of the objects in the pose hierarchy template library are used to perform similarity matching operations with the edges of the objects in the image to be annotated taken by the monocular camera to find the best template as the pose value.

[0034] The similarity matching operation method is: by moving the virtual camera position to render the projection outline of the CAD model in the pose hierarchy template library, until an edge template that coincides with the outline of the object in the image to be annotated is found.

[0035] The beneficial effects of the present invention are as follows: in the first stage, the present invention achieves a rough estimation of the pose based on the object mask and 2D position features; in the second stage, based on the object contour information, the deviation of the rough estimation result is optimized and corrected by a differentiable Gauss-Newton optimization algorithm. Compared with the existing pose estimation method based on template matching, the real-time performance of the two-stage pose estimation method proposed in the present invention is significantly improved. Compared with the existing six-degree-of-freedom pose estimation method of objects based on deep learning methods, the accuracy of pose estimation is improved. The proposed two-stage pose estimation method based on a monocular camera achieves high-precision and high-efficiency pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the two-stage pose estimation method;

[0037] Figure 2 Schematic diagram of the pose estimation dataset for industrial metal parts;

[0038] Figure 3 The Apriltag code pose annotation results of the metal calibration plate, where (a) is the Apriltag code calibration result and (b) is the template matching annotation result;

[0039] Figure 4 Schematic diagram of the coarse pose estimation method based on mask and 2D position features;

[0040] Figure 5 Schematic diagram of the location-aware state-space network;

[0041] Figure 6 Schematic diagram of the contour feature extraction network based on dynamic snake convolution;

[0042] Figure 7 Schematic diagram of the accurate pose estimation method;

[0043] Figure 8 The estimation results of the two-stage pose estimation method, where (a) is the annotation result, (b) is the coarse pose estimation, and (c) is the precise pose estimation. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0047] The two-stage monocular camera-based six-degree-of-freedom pose estimation method of an object in this embodiment includes:

[0048] The monocular camera RGB image of the object to be measured is input into the coarse pose estimation network and the fine pose estimation network at the same time;

[0049] The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image and inputs the coarse pose estimation value as the initial pose into the precise pose estimation network;

[0050] The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose;

[0051] This implementation is a two-stage pose estimation method from coarse to fine, such as Figure 1 As shown in the figure, this algorithm uses a two-stage pose estimation strategy, from coarse to fine, to ensure accuracy while achieving faster speed than traditional template matching methods. The coarse pose estimation network roughly estimates the six-degree-of-freedom pose value, while the fine pose estimation network is used to correct the coarse pose estimation error. The coarse pose estimation method provides the initial value for the fine pose estimation, preventing the fine pose estimation algorithm from falling into a local optimum during the optimization process.

[0052] The true pose values ​​during the training of the coarse pose estimation network and the fine pose estimation network are obtained through pose estimation based on template matching.

[0053] To train the proposed two-stage pose estimation algorithm, a comprehensive pose estimation dataset for industrial metal parts was constructed. This dataset contains object RGB images, CAD models, six-degree-of-freedom (6DOF) pose ground truth, instance segmentation images (mask images), and 2D bounding boxes of the objects. The RGB images were acquired using a 640×480 resolution camera calibrated using a checkerboard grid. The six-degree-of-freedom pose ground truth was annotated using HALCON, which estimates the pose based on a template matching algorithm. The instance segmentation images serve as object masks, obtained by projecting the annotated pose ground truth onto the image plane. 2D bounding boxes were computed based on the instance segmentation masks and annotated using the YOLO format.

[0054] A pose estimation algorithm based on template matching is used to annotate an object's six degrees of freedom (6DOF) pose. Implemented via a HALCON script, this algorithm is widely used for pose estimation in industrial scenarios due to its excellent accuracy. A pose hierarchy template library containing the object's edge information is constructed based on the object's CAD model. A similarity matching operation is performed between the edge templates in the pose hierarchy template library and the edges of the object in the image to be annotated, captured by a monocular camera, to find the optimal template as the pose value. This similarity matching operation is performed by moving the virtual camera position to render the projected outline of the CAD model in the pose hierarchy template library until an edge template that coincides with the outline of the object in the image to be annotated is found.

[0055] In order to verify the accuracy of the six-degree-of-freedom annotation, the Apriltag benchmark system is used for pose calibration, such as Figure 3 As shown in the figure, this calibration plate was annotated using a pose annotation method based on template matching, with the Apriltag code pose calibration value used as the ground truth to verify the accuracy of the pose annotation method based on template matching. A total of 500 images were tested. When the template matching method was used for pose annotation, the mean rotation estimation error was 0.77°. When the translation distance was between [0.2m and 0.3m], the estimation error accuracy could reach 0.22% of the Apriltag code calibration value.

[0056] The coarse pose estimation network of this embodiment addresses the problem that the template matching algorithm has high accuracy but slow speed for weak texture objects. A coarse pose estimation method based on the fusion of mask features and 2D position features is proposed, which directly regresses the pose through deep learning to improve the real-time performance of pose estimation. The coarse pose estimation algorithm uses PoseCNN as the baseline model, calculates the 2D position through the predicted object mask, uses sine and cosine coding to implement position feature encoding, and injects the encoded position features into the pooled multi-scale features through a position-aware state space network (PSM) to improve the robustness of the prediction. Figure 4 As shown, the coarse pose estimation network of this embodiment includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network;

[0057] The RGB image from the monocular camera is input into the multi-scale feature encoder, which extracts features of different scales and inputs them into the mask prediction branch network. The pre-trained VGG16 model is used as the multi-scale feature encoder.

[0058] The mask prediction branch network recovers the object binary mask from features of different scales through upsampling;

[0059] The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the feature of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T Z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) The pinhole camera model is used to restore the translation vector; the center map performs Hough voting to select the center value in the segmented area. This mechanism avoids the interference of background pixels in the voting process, and also reduces the number of pixels that need to be calculated in the voting stage, reducing the number of operations.

[0060] The rotation prediction branch network uses the object binary mask as the region of interest (ROI) and performs regional ROI pooling on the 512-scale features output by the multi-scale feature encoder. This retains only the multi-scale feature values ​​of the object region, removing background features and improving the robustness of the algorithm. The object binary mask is positionally encoded using sine-cosine coding to obtain a 2D position feature encoding at a scale of 256. Feature embedding is performed based on the ROI pooled features and the 2D position feature encoding at a scale of 256. The embedded features are then used to implement rotation quaternion q regression through a fully connected network with two hidden layers.

[0061] This embodiment provides a position-aware state-space network (PSM) to compensate for the loss of position information in the pooling process, realize 2D position feature encoding and feature embedding, and improve the accuracy of coarse pose estimation of weakly textured industrial parts.

[0062] This embodiment uses the proposed Position-Aware State Space Network (PSM) to embed the position encoding features into the ROI pooling features to achieve rotation estimation robustness. The two data stream features of the PSM come from the pooled features and the position encoding features, respectively, so that the pooled features do not lose position information. The specific network structure is as follows: Figure 5 As shown. After passing through the linear layer, activation layer and normalization layer, the position code is multiplied with the pooled feature to achieve position feature fusion. The position-aware state space network of this embodiment includes the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depth-wise separable convolution layer DWConv, the global receptive field module SS2D, the second normalization layer LN, the second linear layer Linear, the second activation function Silu, the third normalization layer LN, the third linear layer Linear and the fusion layer;

[0063] The features after ROI pooling pass through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depth-wise separable convolution layer DWConv, the global receptive field module SS2D, and the second normalization layer LN.

[0064] The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN;

[0065] The output of the No. 2 normalization layer LN and the output of the No. 3 normalization layer LN are multiplied and then input into the No. 3 linear layer Linear;

[0066] The output of the Linear layer 3 and the feature after ROI pooling are fused into the fusion layer to obtain the fused features.

[0067] The second phase of this implementation uses the initial position values ​​provided by the first phase to refine the pose, preventing it from falling into a local optimum during the pose update phase. This pose refinement method uses the DeepAC algorithm as a baseline model and introduces a snake-like convolutional layer (DSConv) to finely extract object contours.

[0068] Based on an accurate pose estimation algorithm based on contour information, a contour template library is generated offline from the object's CAD model. The template library contains the object's contours and the normals corresponding to the contour points at different viewpoints. This offline template library avoids the time-consuming rendering process during the inference phase.

[0069] The precise pose estimation network of this embodiment includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network and a differentiable Newton optimization module;

[0070] Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map;

[0071] Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales;

[0072] In order to solve the problem that traditional convolutional neural networks have weak ability to extract local contours of images and are interfered by background contours, a dynamic snake convolution layer is introduced to construct an object contour feature extraction network to improve the quality of contour feature generation and filter out background contours. The contour feature extraction network of this embodiment is a contour feature extraction network based on dynamic snake convolution. The contours of industrial metal parts and tubular structure features both have slender shape features, so dynamic snake convolution is introduced to improve the ability to extract local contour features. The improved contour feature extraction network, such as Figure 6 As shown;

[0073] The contour feature extraction network of this embodiment includes a downsampling path and an upsampling path. The pre-trained MobileNetV2 is used as the encoder to achieve downsampling of the original image. The same Bottleneck structure as MobileNetV2 is used to refine the features. The upsampling branch uses a conventional 2D convolution structure, and integrates the structural features of different levels of the encoder through jump connections to output three feature maps with different resolutions. After the jump connection, a convolution layer is used to refine the contour features, and the contour features outside the object are filtered through a standard 2D convolution layer to avoid background contour interference.

[0074] The encoder’s features are added to the decoder’s generated features through the dynamic snake convolution layer DSConv represented by the green module to generate 320 2 , 160 2 , 80 2 The three scale feature maps of , the red module indicates the output feature position.

[0075] The blue squares represent the features of different scales generated by the encoder and decoder, and the yellow squares represent the features of different scales generated by the encoder and decoder. 2 The yellow squares of the resolution represent the features of the bottleneck part, 20 2 with 40 2 The feature of the resolution represents the feature map after the decoder and encoder features are fused. The red square represents the output of 320 2 , 160 2 , 80 2 Three scale feature maps.

[0076] The contour feature extraction network of this embodiment is a contour feature extraction network based on dynamic snake convolution, including 5 encoders, 5 decoders, 10 bottleneck layers Bottleneck, 1 standard two-dimensional convolution layer Conv2d, 5 Convtranspose2d, 4 snake convolution layers DSConv and 4 splicing layers Concat;

[0077] The RGB image from the monocular camera is input into the first encoder. The outputs of the five encoders are input into the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolutional layer Conv2d.

[0078] The outputs of the first four encoders each enter a snake convolution layer DSConv;

[0079] The output of the standard two-dimensional convolution layer Conv2d enters the first Conv transpose2d, the output of the first Convtranspose2d and the output of the fourth snake convolution layer DSConv are spliced ​​and enter the first decoder, the output of the first decoder enters the second Conv transpose2d after a bottleneck layer Bottleneck, the output of the second Convtranspose2d and the output of the third snake convolution layer DSConv are spliced ​​and enter the second decoder, the output of the second decoder enters the third Conv transpose2d after a bottleneck layer Bottleneck, the output of the third Convtranspose2d and the output of the second snake convolution layer DSConv are spliced ​​and enter the third decoder, the output of the third decoder enters the fourth Conv after a bottleneck layer Bottleneck. transpose2d, the output of the 4th Convtranspose2d and the output of the 1st snake convolution layer DSConv are concatenated and enter the 4th decoder. The output of the 4th decoder enters the 5th Conv transpose2d after a bottleneck layer Bottleneck. The output of the 5th Convtranspose2d enters the 5th decoder. The 3rd, 4th, and 5th decoders output 320 2 , 160 2 , 80 2 High-resolution feature maps.

[0080] The sampled contour feature map and contour feature maps of different scales are simultaneously input into a lightweight edge probability map prediction module. The edge probability map prediction module performs statistically-based contour extraction on the input contour feature map to generate an edge probability map; the edge probability map represents the projected distance between the observed image and the template edge, providing an optimized gradient.

[0081] The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate. This process does not rely on the texture information of the object, and uses the second-order differentiable Gauss-Newton optimization method to improve the real-time performance of the algorithm. The specific algorithm process is as follows: Figure 7 The final pose estimation result is shown as Figure 8 shown.

[0082] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, characterized by: The method comprises: The monocular camera RGB image of the object to be measured is input into the coarse pose estimation network and the fine pose estimation network at the same time; The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image and inputs the coarse pose estimation value as the initial pose to the precise pose estimation network; the coarse pose estimation value includes the translation vector and the rotation quaternion q; The coarse pose estimation network includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network; The RGB image from the monocular camera is input into the multi-scale feature encoder to extract features of different scales and input into the mask prediction branch network; The mask prediction branch network recovers the object binary mask from features of different scales through upsampling; The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the feature of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T Z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) Using the pinhole camera model to recover the translation vector; The rotation prediction branch network uses the object binary mask as the region of interest, performs regional ROI pooling on the features of the multi-scale feature encoder output at a scale of 512, and positionally encodes the object binary mask to obtain a 2D position feature encoding at a scale of 256. Feature embedding and fusion are performed based on the ROI pooled features and the 2D position feature encoding at a scale of 256. After fully connecting the fused features, the rotation quaternion q is obtained. The true pose value of the coarse pose estimation network and the fine pose estimation network during training is obtained through pose estimation based on template matching. The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose.

2. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1, characterized in that: Based on the features after ROI pooling and the 2D position feature encoding of scale 256, the position-aware state space network is used to perform feature embedding fusion to obtain the fused features; The position-aware state-space network includes a first normalization layer LN, a first linear layer Linear, a first activation function Silu, a depth-separable convolutional layer DWConv, a global receptive field module SS2D, a second normalization layer LN, a second linear layer Linear, a second activation function Silu, a third normalization layer LN, a third linear layer Linear and a fusion layer; The features after ROI pooling pass through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depth-wise separable convolution layer DWConv, the global receptive field module SS2D, and the second normalization layer LN. The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN; The output of the No. 2 normalization layer LN and the output of the No. 3 normalization layer LN are multiplied and input into the No. 3 linear layer Linear; The output of the Linear layer 3 and the feature after ROI pooling are fused into the fusion layer to obtain the fused features.

3. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1, characterized in that: The precise pose estimation network includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network, and a differentiable Newton optimization module; Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map; Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales; The sampled contour feature map and contour feature maps of different scales are simultaneously input into the edge probability map prediction module, which performs statistically-based contour extraction on the input contour feature map to generate an edge probability map. The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate.

4. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3, characterized in that: The contour feature extraction network is based on dynamic snake convolution, which includes 5 encoders, 5 decoders, 10 bottleneck layers, 1 standard 2D convolution layer, 5 2D transposed convolution layers and 4 snake convolution layers. The monocular camera RGB image is input to the first encoder. The outputs of the five encoders are input to the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolutional layer. The outputs of the first four encoders each enter a snake convolution layer; The output of the standard 2D convolution layer enters the first 2D transposed convolution layer. The output of the first 2D transposed convolution layer and the output of the fourth snake convolution layer are concatenated and enter the first decoder. The output of the first decoder enters the second 2D transposed convolution layer after a bottleneck layer. The output of the second 2D transposed convolution layer and the output of the third snake convolution layer are concatenated and enter the second decoder. The output of the second decoder enters the third 2D transposed convolution layer after a bottleneck layer. The output of the convolution layer is concatenated with the output of the second snake convolution layer and then enters the third decoder. The output of the third decoder enters the fourth two-dimensional transposed convolution layer after passing through a bottleneck layer. The output of the fourth two-dimensional transposed convolution layer is concatenated with the output of the first snake convolution layer and then enters the fourth decoder. The output of the fourth decoder enters the fifth two-dimensional transposed convolution layer after passing through a bottleneck layer. The output of the fifth two-dimensional transposed convolution layer enters the fifth decoder. The outputs of the third, fourth, and fifth decoders are 320 2 , 160 2 , 80 2 High-resolution feature maps.

5. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3, characterized in that: The encoder is a pre-trained MobileNetV2.

6. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3, characterized in that: The method for obtaining the line corresponding model template library is: Projection rendering is performed based on the object CAD model to generate the object contours and normals corresponding to the contour points under different viewing angles offline to form a contour template library.

7. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1, characterized in that: The pose truth values ​​during the training of the coarse pose estimation network and the fine pose estimation network are obtained through pose estimation based on template matching. The method includes: Based on the object CAD model, a pose hierarchy template library of object edge information is constructed. The edge templates of the objects in the pose hierarchy template library are used to perform similarity matching operations with the edges of the objects in the image to be annotated taken by the monocular camera to find the best template as the pose value. The similarity matching operation method is: by moving the virtual camera position to render the projection outline of the CAD model in the pose hierarchy template library, until an edge template that coincides with the outline of the object in the image to be annotated is found.

8. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the two-stage object six-degree-of-freedom pose estimation method based on a monocular camera are implemented as described in any one of claims 1 to 7.

9. A two-stage object six-degree-of-freedom pose estimation device based on a monocular camera, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of the two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the two-stage object six-degree-of-freedom pose estimation method based on a monocular camera are implemented as described in any one of claims 1 to 7.

Images