Two-stage object six-degree-of-freedom pose estimation method based on monocular camera

Abstract

The invention discloses a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, solves the problem of how to carry out high-precision and high-efficiency pose estimation on weak-texture industrial parts, and belongs to the technical field of monocular camera pose estimation. The invention relates to a coarse-to-fine two-stage pose estimation method. The coarse pose estimation network roughly estimates the pose value of six degrees of freedom based on mask feature and 2D position feature fusion, and the coarse pose estimation stage provides an initial value for precise pose estimation, so that the optimization process of the precise pose estimation algorithm is prevented from falling into a local optimal value. A dynamic snakelike convolutional layer is introduced into an accurate pose estimation network to construct an object contour feature extraction network, the contour feature generation quality is improved, and a background contour is filtered out. Through a differentiable Gaussian-Newton optimization method, iterative optimization is carried out on multiple scales, the re-projection error of the projection contour and the object contour is minimized, and accurate pose estimation is realized.

Description

Technical Field

[0001] The invention relates to a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, and belongs to the technical field of monocular camera pose estimation. Background Art

[0002] Pose estimation of weakly textured parts is a challenging task, mainly because the surface features of weakly textured parts are not obvious, and changes in lighting conditions as well as occlusion and interference in actual application scenarios will affect the accuracy of pose estimation.

[0003] There are currently two main methods for pose estimation of weakly textured parts: pose estimation based on deep learning and pose estimation based on feature matching. The deep learning-based method uses deep neural networks to learn images to obtain the pose information of objects, but this method requires a large amount of labeled data for training and has a large algorithm overhead. The pose estimation algorithm based on deep learning has significantly improved in real time, but due to the weak texture characteristics of industrial parts, the estimation accuracy is low. Summary of the invention

[0004] In order to solve the problem of how to achieve high-precision and high-efficiency pose estimation of weak-textured industrial parts, the present invention provides a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera.

[0005] A two-stage object six-degree-of-freedom pose estimation method based on a monocular camera of the present invention comprises:

[0006] The monocular camera RGB image of the object to be measured is simultaneously input into the coarse pose estimation network and the precise pose estimation network;

[0007] The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image, and inputs the coarse pose estimation value as the initial pose into the precise pose estimation network; the coarse pose estimation value includes the translation vector and the rotation quaternion q;

[0008] The coarse pose estimation network includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network;

[0009] The RGB image of the monocular camera is input into the multi-scale feature encoder to extract features of different scales and input into the mask prediction branch network;

[0010] The mask prediction branch network recovers the object binary mask from features of different scales through upsampling;

[0011] The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the features of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) Using the pinhole camera model to recover the translation vector;

[0012] The rotation prediction branch network uses the object binary mask as the region of interest, performs regional ROI pooling on the features of the multi-scale feature encoder output scale 512, performs position encoding on the object binary mask, and obtains a 2D position feature encoding of scale 256. Feature embedding fusion is performed based on the features after ROI pooling and the 2D position feature encoding of scale 256. After the fused features are fully connected, the rotation quaternion q is obtained. The true pose value during the training of the coarse pose estimation network and the precise pose estimation network is obtained through pose estimation based on template matching.

[0013] The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose.

[0014] Preferably, the position-aware state space network is used to perform feature embedding fusion based on the ROI pooled features and the 2D position feature encoding of scale 256 to obtain the fused features;

[0015] The position-aware state space network includes a first normalization layer LN, a first linear layer Linear, a first activation function Silu, a depth-separable convolution layer DWConv, a global receptive field module SS2D, a second normalization layer LN, a second linear layer Linear, a second activation function Silu, a third normalization layer LN, a third linear layer Linear and a fusion layer;

[0016] The features after ROI pooling are sequentially passed through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depthwise separable convolutional layer DWConv, the global receptive field module SS2D, and the second normalization layer LN;

[0017] The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN in sequence;

[0018] The output of the No. 2 normalization layer LN is multiplied by the output of the No. 3 normalization layer LN and then input to the No. 3 linear layer Linear;

[0019] The output of the linear layer 3 and the features after ROI pooling are fused into the fusion layer to obtain the fused features.

[0020] Preferably, the accurate pose estimation network includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network and a differentiable Newton optimization module;

[0021] Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map;

[0022] Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales;

[0023] The sampled contour feature map and contour feature maps of different scales are simultaneously input into the edge probability map prediction module, and the edge probability map prediction module performs a statistically-based contour extraction on the input contour feature map to generate an edge probability map;

[0024] The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate.

[0025] Preferably, the contour feature extraction network is a contour feature extraction network based on dynamic snake convolution, including 5 encoders, 5 decoders, 10 bottleneck layers, 1 standard two-dimensional convolution layer, 5 two-dimensional transposed convolution layers and 4 snake convolution layers;

[0026] The monocular camera RGB image is input into the first encoder. The outputs of the five encoders are input into the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolutional layer.

[0027] The outputs of the first four encoders each enter a snake convolution layer;

[0028] The output of the standard 2D convolution layer enters the first 2D transposed convolution layer. The output of the first 2D transposed convolution layer and the output of the fourth snake convolution layer are concatenated and enter the first decoder. The output of the first decoder enters the second 2D transposed convolution layer after a bottleneck layer. The output of the second 2D transposed convolution layer and the output of the third snake convolution layer are concatenated and enter the second decoder. The output of the second decoder enters the third 2D transposed convolution layer after a bottleneck layer. The output of the convolution layer is concatenated with the output of the second snake convolution layer and then enters the third decoder. The output of the third decoder enters the fourth two-dimensional transposed convolution layer after a bottleneck layer. The output of the fourth two-dimensional transposed convolution layer is concatenated with the output of the first snake convolution layer and then enters the fourth decoder. The output of the fourth decoder enters the fifth two-dimensional transposed convolution layer after a bottleneck layer. The output of the fifth two-dimensional transposed convolution layer enters the fifth decoder. The outputs of the third, fourth, and fifth decoders are 320, respectively. 2 , 160 2 , 80 2 Resolution feature maps.

[0029] Preferably, the encoder is a pre-trained MobileNetV2.

[0030] Preferably, the method for obtaining the line corresponding model template library is:

[0031] Projection rendering is performed offline based on the object CAD model to generate the object contours and normals corresponding to the contour points at different viewing angles to form a contour template library.

[0032] Preferably, the pose true values ​​of the coarse pose estimation network and the precise pose estimation network during training are obtained by pose estimation based on template matching, and the method includes:

[0033] Based on the object CAD model, a pose hierarchical template library of the object edge information is constructed. The edge template of the object in the pose hierarchical template library is used to perform similarity matching operation with the edge of the object in the image to be annotated taken by the monocular camera to find the best template as the pose value.

[0034] The method of similarity matching operation is: rendering the projection outline of the CAD model in the pose hierarchy template library by moving the virtual camera position until an edge template that coincides with the outline of the object in the image to be annotated is found.

[0035] The beneficial effects of the present invention are as follows: in the first stage, the present invention realizes a rough estimation of the pose based on the object mask and 2D position features; in the second stage, based on the object contour information, the deviation of the rough estimation result is optimized and corrected by a differentiable Gauss-Newton optimization algorithm. Compared with the existing pose estimation method based on template matching, the real-time performance of the two-stage pose estimation method proposed in the present invention is significantly improved, and compared with the existing six-degree-of-freedom pose estimation method of objects based on deep learning methods, the accuracy of pose estimation is improved. The proposed two-stage pose estimation method based on a monocular camera realizes high-precision and high-efficiency pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the two-stage pose estimation method;

[0037] Figure 2 This is a schematic diagram of the pose estimation dataset for industrial metal parts;

[0038] Figure 3 The Apriltag code pose annotation results of the metal calibration plate, where (a) is the Apriltag code calibration result, and (b) is the template matching annotation result;

[0039] Figure 4 The schematic diagram of the rough pose estimation method based on mask and 2D position features;

[0040] Figure 5 Schematic diagram of a location-aware state-space network;

[0041] Figure 6 Schematic diagram of contour feature extraction network based on dynamic snake convolution;

[0042] Figure 7 Schematic diagram of the accurate pose estimation method;

[0043] Figure 8 It is the estimation result of the two-stage pose estimation method, where (a) is the annotation result, (b) is the coarse pose estimation, and (c) is the precise pose estimation. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0046] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0047] The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera of this embodiment includes:

[0048] The monocular camera RGB image of the object to be measured is simultaneously input into the coarse pose estimation network and the precise pose estimation network;

[0049] The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image, and inputs the coarse pose estimation value as the initial pose into the precise pose estimation network;

[0050] The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose;

[0051] This implementation is a two-stage pose estimation method from coarse to fine, such as Figure 1 As shown in the figure. Through the two-stage pose estimation strategy from coarse to fine, the algorithm ensures accuracy while achieving faster speed than the traditional template matching method. The coarse pose estimation network roughly estimates the pose value of six degrees of freedom, and the precise pose estimation network is used to correct the coarse pose estimation error. The coarse pose estimation method stage provides the initial value of the precise pose estimation, avoiding the optimization process of the precise pose estimation algorithm from falling into the local optimal value.

[0052] The true pose value during the training of the coarse pose estimation network and the precise pose estimation network is obtained through pose estimation based on template matching.

[0053] In order to realize the proposed two-stage pose estimation algorithm model training, a complete pose estimation dataset for industrial metal parts is constructed. The dataset contains object RGB images, CAD models, six-degree-of-freedom pose truth values, instance segmentation images (mask images), and object 2D bounding boxes. The RGB images are collected by a camera with a resolution of 640×480, and the camera is calibrated by a checkerboard. The six-degree-of-freedom pose truth values ​​are annotated using HALCON, which estimates the pose based on the template matching algorithm. The instance segmentation image is the object mask, and the CAD model is projected onto the image plane through the annotated pose truth values. The 2D bounding box is calculated based on the instance segmentation mask and annotated using the YOLO format.

[0054] The pose estimation algorithm based on template matching is used to annotate the six-degree-of-freedom pose of the object. This is achieved through the HALCON script. Due to its excellent accuracy, it is widely used for pose estimation in industrial scenarios. A pose hierarchy template library of the object edge information is constructed based on the object CAD model. The edge template of the object in the pose hierarchy template library and the edge of the object in the image to be annotated taken by the monocular camera are used for similarity matching operations to find the best template as the pose value. The similarity matching operation method is: by moving the virtual camera position to render the projection contour of the CAD model in the pose hierarchy template library, until an edge template that coincides with the contour of the object in the image to be annotated is found.

[0055] In order to verify the accuracy of the six-degree-of-freedom annotation, the Apriltag benchmark system is used for pose calibration. Figure 3 As shown. The calibration plate is annotated using the pose annotation method based on template matching, and the pose calibration value of the Apriltag code is used as the true value to verify the accuracy of the pose annotation method based on template matching. A total of 500 images were used for testing. When the pose annotation was performed using the template matching method, the mean rotation estimation error was 0.77°, and when the translation distance was between [0.2m, 0.3m], the estimation error accuracy could reach 0.22% of the Apriltag code calibration value.

[0056] The coarse pose estimation network of this embodiment addresses the problem that the template matching algorithm based on weak texture objects has high accuracy but slow speed. A coarse pose estimation method based on the fusion of mask features and 2D position features is proposed, and the pose is directly regressed through deep learning to improve the real-time performance of pose estimation. The coarse pose estimation algorithm uses PoseCNN as the baseline model, calculates the 2D position through the predicted object mask, uses sine and cosine coding to implement position feature encoding, and injects the encoded position features into the pooled multi-scale features through a position-aware state space network (PSM) to improve the robustness of the prediction. Figure 4 As shown, the coarse pose estimation network of this embodiment includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network;

[0057] The RGB image of the monocular camera is input into the multi-scale feature encoder to extract features of different scales and input into the mask prediction branch network; the pre-trained VGG16 model is used as the multi-scale feature encoder;

[0058] The mask prediction branch network recovers the object binary mask from features of different scales through upsampling;

[0059] The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the features of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T Z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) The pinhole camera model is used to restore the translation vector; the center map performs Hough voting in the segmented area to select the center value. This mechanism avoids the interference of background pixels in the voting process, and also reduces the number of pixels that need to be calculated in the voting stage, thereby reducing the number of operations.

[0060] The rotation prediction branch network uses the object binary mask as the region of interest, and performs regional ROI pooling on the features of the multi-scale feature encoder output scale 512. Only the multi-scale feature values ​​of the object area are retained, which removes the background feature values ​​and improves the robustness of the algorithm. The object binary mask is position-encoded using sine-cosine coding to obtain a 2D position feature encoding of scale 256. Feature embedding is performed based on the features after ROI pooling and the 2D position feature encoding of scale 256. The embedded features are used to implement rotation quaternion q regression through a fully connected layer with two hidden layers.

[0061] This embodiment provides a position-aware state space network (PSM) to supplement the loss of position information in the pooling process, realize 2D position feature encoding and feature embedding, and improve the accuracy of coarse pose estimation of weakly textured industrial parts.

[0062] This embodiment uses the proposed position-aware state space network (PSM) to embed the position encoding feature into the ROI pooling feature to achieve rotation estimation robustness. The two data stream features of the PSM come from the pooled features and the position encoding features, respectively, so that the pooled features do not lose position information. The specific network structure is as follows: Figure 5 As shown. After passing through the linear layer, activation layer and normalization layer, the position code is multiplied with the pooled feature to achieve position feature fusion. The position-aware state space network of this embodiment includes a first layer of normalization layer LN, a first linear layer Linear, a first activation function Silu, a depth-separable convolution layer DWConv, a global receptive field module SS2D, a second layer of normalization layer LN, a second linear layer Linear, a second activation function Silu, a third layer of normalization layer LN, a third linear layer Linear and a fusion layer;

[0063] The features after ROI pooling are sequentially passed through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depthwise separable convolutional layer DWConv, the global receptive field module SS2D, and the second normalization layer LN;

[0064] The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN in sequence;

[0065] The output of the No. 2 normalization layer LN is multiplied by the output of the No. 3 normalization layer LN and then input to the No. 3 linear layer Linear;

[0066] The output of the linear layer 3 and the features after ROI pooling are fused into the fusion layer to obtain the fused features.

[0067] The second stage of this implementation uses the initial position value provided in the first stage to refine the pose, avoiding falling into the local optimal value in the pose update stage. The pose refinement method uses the DeepAC algorithm as the baseline model and introduces the snake convolution layer DSConv to achieve fine extraction of object contours.

[0068] Based on the accurate pose estimation algorithm of contour information, the contour template library is generated offline according to the object CAD model. The template library contains the object contours and normals corresponding to the contour points under different viewing angles. The offline generation of the template library avoids the time-consuming rendering process in the inference stage.

[0069] The accurate pose estimation network of this embodiment includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network and a differentiable Newton optimization module;

[0070] Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map;

[0071] Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales;

[0072] In order to solve the problem that the traditional convolutional neural network has weak ability to extract local contours of images and is disturbed by background contours, a dynamic snake convolution layer is introduced to construct an object contour feature extraction network to improve the quality of contour feature generation and filter out background contours. The contour feature extraction network of this embodiment is a contour feature extraction network based on dynamic snake convolution. The contours of industrial metal parts and tubular structure features both have slender shape features, so dynamic snake convolution is introduced to improve the ability to extract local contour features. The improved contour feature extraction network, such as Figure 6 As shown;

[0073] The contour feature extraction network of this embodiment includes a downsampling path and an upsampling path. The pre-trained MobileNetV2 is used as the encoder to achieve downsampling of the original image. The same Bottleneck structure as MobileNetV2 is used to refine the features. The upsampling branch uses a conventional 2D convolution structure, and integrates the structural features of different levels of the encoder through jump connections to output three feature maps with different resolutions. After the jump connection, a convolution layer is used to refine the contour features, and the contour features outside the object are filtered through a standard 2D convolution layer to avoid background contour interference.

[0074] The encoder features are added to the decoder's generated features through the dynamic snake convolution layer DSConv represented by the green module to generate 320 2 , 160 2 , 80 2 Three scale feature maps, the red module indicates the output feature position.

[0075] The blue squares represent the features of different scales generated by the encoder and decoder, and the yellow squares represent the features of different scales generated by the encoder and decoder. 2 The yellow squares at the resolution 20 represent the features of the bottleneck part. 2 With 40 2 The feature of the resolution represents the feature map after the decoder and encoder features are fused. The red square represents the output of 320 2 , 160 2 , 80 2 Three scale feature maps.

[0076] The contour feature extraction network of this embodiment is a contour feature extraction network based on dynamic snake convolution, including 5 encoders, 5 decoders, 10 bottleneck layers Bottleneck, 1 standard two-dimensional convolution layer Conv2d, 5 Convtranspose2d, 4 snake convolution layers DSConv and 4 concatenation layers Concat;

[0077] The RGB image of the monocular camera is input into the first encoder. The outputs of the five encoders are input into the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolution layer Conv2d.

[0078] The outputs of the first four encoders each enter a snake convolution layer DSConv;

[0079] The output of the standard two-dimensional convolution layer Conv2d enters the first Conv transpose2d. The output of the first Convtranspose2d is concatenated with the output of the fourth snake convolution layer DSConv and enters the first decoder. The output of the first decoder enters the second Conv transpose2d after a bottleneck layer Bottleneck. The output of the second Convtranspose2d is concatenated with the output of the third snake convolution layer DSConv and enters the second decoder. The output of the second decoder enters the third Conv transpose2d after a bottleneck layer Bottleneck. The output of the third Convtranspose2d is concatenated with the output of the second snake convolution layer DSConv and enters the third decoder. The output of the third decoder enters the fourth Conv after a bottleneck layer Bottleneck. transpose2d, the output of the 4th Convtranspose2d and the output of the 1st snake convolution layer DSConv are concatenated and enter the 4th decoder. The output of the 4th decoder enters the 5th Conv transpose2d after a bottleneck layer Bottleneck, and the output of the 5th Convtranspose2d enters the 5th decoder; the 3rd, 4th, and 5th decoders output 320 2 , 160 2 , 80 2 Resolution feature maps.

[0080] The sampled contour feature map and contour feature maps of different scales are simultaneously input into a lightweight edge probability map prediction module. The edge probability map prediction module performs statistical-based contour extraction on the input contour feature map to generate an edge probability map; the edge probability map represents the projection distance between the observed image and the template edge, and provides an optimized gradient.

[0081] The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate. This process does not rely on the texture information of the object, and uses the second-order differentiable Gauss-Newton optimization method to improve the real-time performance of the algorithm. The specific algorithm flow is as follows: Figure 7 The final pose estimation result is shown in Figure 8 shown.

[0082] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A two-stage object six-degree-of-freedom pose estimation method based on a monocular camera, characterized in that: The method comprises: The monocular camera RGB image of the object to be measured is simultaneously input into the coarse pose estimation network and the precise pose estimation network; The coarse pose estimation network obtains the coarse pose estimation value of the input monocular camera RGB image, and inputs the coarse pose estimation value as the initial pose into the precise pose estimation network; the coarse pose estimation value includes the translation vector and the rotation quaternion q; The coarse pose estimation network includes a multi-scale feature encoder, a mask prediction branch network, a translation prediction branch network and a rotation prediction branch network; The RGB image of the monocular camera is input into the multi-scale feature encoder to extract features of different scales and input into the mask prediction branch network; The mask prediction branch network recovers the object binary mask from features of different scales through upsampling; The object binary mask is input to the translation prediction branch network and the rotation prediction branch network at the same time. The translation prediction branch network predicts the vector graph of the features of the multi-scale feature encoder output scale 128 according to the object binary mask, and obtains the x and y direction vector graphs and the depth T Z , and then select the center point of the object (c x ,c y ), according to the center point of the object (c x ,c y ) Using the pinhole camera model to recover the translation vector; The rotation prediction branch network uses the object binary mask as the region of interest, performs regional ROI pooling on the features of the multi-scale feature encoder output scale 512, performs position encoding on the object binary mask, and obtains a 2D position feature encoding of scale 256. Feature embedding fusion is performed based on the features after ROI pooling and the 2D position feature encoding of scale 256. After the fused features are fully connected, the rotation quaternion q is obtained. The true pose value during the training of the coarse pose estimation network and the precise pose estimation network is obtained through pose estimation based on template matching. The precise pose estimation network estimates the precise pose in the input monocular camera RGB image based on the initial pose.

2. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1 is characterized in that: Based on the features after ROI pooling and the 2D position feature encoding of scale 256, the position-aware state space network is used to perform feature embedding fusion to obtain the fused features; The position-aware state space network includes a first normalization layer LN, a first linear layer Linear, a first activation function Silu, a depth-separable convolution layer DWConv, a global receptive field module SS2D, a second normalization layer LN, a second linear layer Linear, a second activation function Silu, a third normalization layer LN, a third linear layer Linear and a fusion layer; The features after ROI pooling are sequentially passed through the first normalization layer LN, the first linear layer Linear, the first activation function Silu, the depthwise separable convolutional layer DWConv, the global receptive field module SS2D, and the second normalization layer LN; The 256-scale 2D position feature encoding passes through the second linear layer Linear, the second activation function Silu, and the third normalization layer LN in sequence; The output of the No. 2 normalization layer LN is multiplied by the output of the No. 3 normalization layer LN and then input to the No. 3 linear layer Linear; The output of the linear layer 3 and the features after ROI pooling are fused into the fusion layer to obtain the fused features.

3. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1, characterized in that: The precise pose estimation network includes a line correspondence model template library, a KNN network, an edge probability map prediction module, a contour feature extraction network, and a differentiable Newton optimization module; Use the KNN algorithm to find the template contour closest to the initial pose in the online corresponding model template library; sample the found template contour to obtain the sampled contour feature map; Input the monocular camera RGB image into the contour feature extraction network to obtain contour feature maps of different scales; The sampled contour feature map and contour feature maps of different scales are simultaneously input into the edge probability map prediction module, and the edge probability map prediction module performs a statistically-based contour extraction on the input contour feature map to generate an edge probability map; The generated edge probability map is input into the differentiable Newton optimization module to optimize the gradient and pose update, minimize the reprojection error of the contour points, and obtain an accurate pose estimate.

4. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3 is characterized in that: The contour feature extraction network is a contour feature extraction network based on dynamic snake convolution, including 5 encoders, 5 decoders, 10 bottleneck layers, 1 standard 2D convolution layer, 5 2D transposed convolution layers and 4 snake convolution layers; The monocular camera RGB image is input into the first encoder. The outputs of the five encoders are input into the next encoder in sequence after passing through a bottleneck layer. The output of the fifth encoder enters the standard two-dimensional convolutional layer. The outputs of the first four encoders each enter a snake convolution layer; The output of the standard 2D convolution layer enters the first 2D transposed convolution layer. The output of the first 2D transposed convolution layer and the output of the fourth snake convolution layer are concatenated and enter the first decoder. The output of the first decoder enters the second 2D transposed convolution layer after a bottleneck layer. The output of the second 2D transposed convolution layer and the output of the third snake convolution layer are concatenated and enter the second decoder. The output of the second decoder enters the third 2D transposed convolution layer after a bottleneck layer. The output of the convolution layer is concatenated with the output of the second snake convolution layer and then enters the third decoder. The output of the third decoder enters the fourth two-dimensional transposed convolution layer after a bottleneck layer. The output of the fourth two-dimensional transposed convolution layer is concatenated with the output of the first snake convolution layer and then enters the fourth decoder. The output of the fourth decoder enters the fifth two-dimensional transposed convolution layer after a bottleneck layer. The output of the fifth two-dimensional transposed convolution layer enters the fifth decoder. The outputs of the third, fourth, and fifth decoders are 320, respectively. 2 , 160 2 , 80 2 Resolution feature maps.

5. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3 is characterized in that: The encoder is a pre-trained MobileNetV2.

6. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 3 is characterized in that: The method for obtaining the line corresponding model template library is: Projection rendering is performed offline based on the object CAD model to generate the object contours and normals corresponding to the contour points at different viewing angles to form a contour template library.

7. The two-stage object six-degree-of-freedom pose estimation method based on a monocular camera according to claim 1, characterized in that: The pose truth values ​​of the coarse pose estimation network and the precise pose estimation network during training are obtained through pose estimation based on template matching, and the method includes: Based on the object CAD model, a pose hierarchical template library of the object edge information is constructed. The edge template of the object in the pose hierarchical template library is used to perform similarity matching operation with the edge of the object in the image to be annotated taken by the monocular camera to find the best template as the pose value. The method of similarity matching operation is: rendering the projection outline of the CAD model in the pose hierarchy template library by moving the virtual camera position until an edge template that coincides with the outline of the object in the image to be annotated is found.

8. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the two-stage object six-degree-of-freedom pose estimation method based on a monocular camera are implemented as described in any one of claims 1 to 7.

9. A two-stage object six-degree-of-freedom pose estimation device based on a monocular camera, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of a two-stage object six-degree-of-freedom pose estimation method based on a monocular camera as claimed in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the two-stage object six-degree-of-freedom pose estimation method based on a monocular camera as described in any one of claims 1 to 7 are implemented.

Images