Object six-degree-of-freedom pose estimation method based on deep learning

Abstract

The invention belongs to the technical field of attitude estimation, and particularly discloses an object six-degree-of-freedom pose estimation method based on deep learning. Comprising the following steps: S1, obtaining an original RGBD image, and processing the original RGBD image through a mask technology; s2, constructing a pyramid network, inputting the processed RGBD image into the pyramid network, endowing different features with different weights, and generating rotation and translation prediction of an object; and S3, constructing a loss function, carrying out training and verification through the 6D attitude estimation data set model, and evaluating the precision and robustness of the model when the model processes a 6D attitude estimation task. The invention provides an object six-degree-of-freedom pose estimation method based on deep learning. The object six-degree-of-freedom pose estimation method comprises a pyramid network. The network structure extracts rich color and geometric information in a complex scene, so that accurate attitude estimation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of posture estimation, and in particular relates to a method for estimating the six-degree-of-freedom posture of an object based on deep learning. Background Art

[0002] The six degrees of freedom (6DoF) pose estimation of objects plays a vital role in the field of modern technology, especially in the fields of augmented reality, robotic manipulation and autonomous driving. The ability to accurately estimate the position and orientation of an object in three-dimensional space is extremely important for achieving complex interactions and tasks. For example, in robotics, accurately acquiring the 6D pose of the target is the key to achieving efficient grasping and manipulation. At the same time, in virtual reality (VR) and augmented reality (AR) environments, accurate 6D pose estimation can ensure seamless integration and interaction between virtual objects and the real world. Therefore, whether in industrial automation, interactive entertainment, or future transportation systems, accurate estimation of the 6D pose of the target object is the cornerstone for realizing advanced functions and improving user experience.

[0003] Target pose estimation is a classic problem in the field of computer vision. Traditional methods usually match local features extracted from images with features in the 3D model of the object, and achieve pose estimation by establishing a correspondence between the target image and the model. However, this method usually relies on an accurate model to identify specific objects and has difficulty in dealing with textureless objects and occlusion problems in scenes. This is because only limited local features can be extracted for these objects. At the same time, the extraction of some local features depends on 2D bounding boxes, which is difficult to achieve in complex occluded scenes. With the rapid development of deep learning technology, trainable convolutional neural networks (CNNs) have shown great potential in dealing with symmetrical or textureless objects, solving the problems that traditional methods based on image gradient features are difficult to overcome. Some studies regress 2D key points through CNNs and then use the PnP algorithm to calculate 6DoF parameters. However, these methods still face great challenges in dealing with occluded and truncated objects because some key points may not be detected. Another method is to train an end-to-end neural network through deep learning, take images as input, and output the corresponding pose. These methods can handle textureless objects, but since they cannot achieve high-precision pose estimation, small errors in the regression stage directly lead to pose mismatches. At the same time, generalization remains a challenge as it is unclear whether end-to-end methods learn sufficient and robust feature representations for pose estimation.

[0004] In summary, the existing technology usually relies on accurate models to identify specific objects, and it is difficult to deal with textureless objects and occlusion problems in scenes; the extraction of some local features relies on 2D bounding boxes, which are difficult to implement in complex occluded scenes; when using CNNs to regress 2D key points and then using the PnP algorithm to calculate 6DoF parameters, it is difficult to deal with the problem that some key points may not be detected when objects are occluded and truncated; the end-to-end neural network method can handle textureless objects, but cannot achieve high-precision pose estimation; at the same time, it is not certain whether the end-to-end method has learned sufficient and robust feature representations for pose estimation. Summary of the invention

[0005] The present invention provides a method for estimating the six-degree-of-freedom posture of an object based on deep learning, so as to solve the problem that high-precision posture estimation cannot be achieved at present.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: the method for estimating the six-degree-of-freedom pose of an object based on deep learning, wherein the method comprises the following steps:

[0007] S1: Obtain the original RGBD image and process the original RGBD image through mask technology;

[0008] S2: construct a pyramid network, input the processed RGBD image into the pyramid network, assign different weights to different features, and generate rotation and translation predictions of the object;

[0009] S3: Construct a loss function, train and validate the model using the 6D pose estimation dataset, and evaluate the accuracy and robustness of the model in handling the 6D pose estimation task.

[0010] In a preferred embodiment of the present invention, step S1 specifically includes:

[0011] S11: Crop the original RGBD image using the mask;

[0012] S12: Enlarge the cropped image based on its aspect ratio, centering on the centroid of the cropped image.

[0013] In a preferred embodiment of the present invention, in step S2, the pyramid network includes an attention pyramid network and a feature pyramid network, and the attention pyramid network includes a convolutional layer, a SE module and a spatial attention module.

[0014] In a preferred embodiment of the present invention, step S2 specifically includes:

[0015] S21: Input the processed RGBD image into a convolution layer with a convolution kernel of 3×3 to extract the basic features of the image;

[0016] S22: inputting the basic features into the SE module, assigning different weights to different features and performing global learning on channel importance;

[0017] S23: The spatial attention module generates a spatial attention map by aggregating channel responses and applying convolution to assign different weights to different features;

[0018] S24: Input the spatial attention map into the feature pyramid network and generate rotation and translation predictions of the object.

[0019] In a preferred embodiment of the present invention, the step S22 of "assigning different weights to different features" specifically includes the following steps:

[0020] First, the convolutional layer inputs a H×W×C size network to the SE module. 1 Features F 1 , where H, W, C 1 are the length, width, and number of channels of the feature respectively; 1 Perform a Squeeze operation and obtain the global information of the feature map through global average pooling to obtain a 1×1×C 1 Features F 2 ;

[0021] Secondly, perform excititon operation to 1 It is sent to a multi-layer perceptron (MLP) consisting of a fully connected layer, a relu layer, and a logistic function activation layer to obtain the importance weight coefficient M 1 ;

[0022] Finally, the scale operation is performed and the input features are combined with M 1 Multiply to get the channel attention feature F 3 .

[0023] In a preferred embodiment of the present invention, the step S23 of "assigning different weights to different features" specifically includes the following steps:

[0024] First, the SE module takes the channel attention feature F 3 Output to the SAM module, the SAM block generates a feature map F by combining maximum pooling and average pooling 4 and F 5 , generate a spatial attention map;

[0025] Secondly, concatenate them and input them into the perceptron composed of the fully connected layer and sigmoid to obtain the spatial attention weight coefficient M 2 ;

[0026] Finally, the input features are combined with M2 Multiply to get the enhanced feature F 6 .

[0027] In a preferred embodiment of the present invention, the loss function L in step S3 is obtained by formula 1:

[0028] L=λ 1 l MSE +λ 2 l dist (1)

[0029] where λ 1 , 2 is the equilibrium constant coefficient; l MSE , l dist There are two parts designed for the loss function.

[0030] In a preferred embodiment of the present invention, MSE , l dist Obtained by formula 2:

[0031]

[0032] Among them, r i and t j are the elements of the actual rotation matrix and translation vector respectively; and are the elements of the predicted rotation matrix and translation vector respectively; M represents the set of randomly selected 3D points of the target object; k represents the number of 3D points; R and T represent the actual rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

[0033] In a preferred embodiment of the present invention, in step S3, the 6D pose estimation dataset model is the INEMOD dataset and the Occlusion LINEMOD dataset, and the evaluation indicators of the accuracy and robustness of the model when processing the 6D pose estimation task are the 2D projection index, the average distance and the closest point distance.

[0034] In a preferred embodiment of the present invention, the 2D projection index Pro-2d is obtained by formula 3:

[0035]

[0036] Where: M is the object's 3D point set; m is the number of 3D points; K is the camera's intrinsic parameter matrix; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively;

[0037] The average distance ADD is obtained by formula 4:

[0038]

[0039] Where m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively;

[0040] The closest point distance ADD-S is obtained by formula 5:

[0041]

[0042] Where, m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

[0043] The technical solution provided by the present invention has the following advantages compared with the prior art:

[0044] This paper proposes a deep learning-based six-degree-of-freedom pose estimation method for objects, which includes an innovative attention pyramid network (PPN). The network structure cleverly combines the feature pyramid network (FPN) and the channel-space attention mechanism, which can effectively learn from a single RGBD image and extract rich color and geometric information in complex scenes, thereby achieving accurate pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of an attention pyramid network in a six-degree-of-freedom pose estimation method of an object based on deep learning described in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] For ease of understanding, a method for estimating the six-degree-of-freedom pose of an object based on deep learning is described below in conjunction with embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0048] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation and positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0051] The present invention discloses a method for estimating the six-degree-of-freedom pose of an object based on deep learning, which comprises the following steps:

[0052] S1: Obtain the original RGBD image and process the original RGBD image through mask technology.

[0053] In image processing, extracting effective features for pose estimation is particularly difficult if the image background is complex and cluttered. Noise and irrelevant information in the background may interfere with the extraction of key features, thereby reducing the effectiveness of the features. To overcome this challenge, an effective strategy is to use masking technology to accurately crop the original RGBD (a combination of color and depth images) images.

[0054] Masking technology plays a key role in image processing, which can focus on a specific area in the image, namely the Region of Interest (ROI), while ignoring other irrelevant or interfering background information. By applying a mask, the target object can be effectively isolated, making it easier to extract features that are crucial for subsequent analysis such as pose estimation.

[0055] In the present invention, step S1 mainly includes the following two steps:

[0056] S11: Use the mask to crop the original RGBD image to highlight the target object in the image.

[0057] S12: Centered on the centroid of the cropped image, the image is enlarged according to the aspect ratio to ensure the consistency of the neural network input dimension.

[0058] This approach not only ensures the preservation of important features, but also reduces the interference of background noise on target recognition and feature extraction to a certain extent, which in turn affects the accuracy of feature extraction and subsequent analysis. Maintaining a consistent aspect ratio ensures that the image retains its original geometric and visual properties during the cropping process, which helps improve the reliability of the analysis.

[0059] S2: Build a pyramid network, input the processed RGBD image into the pyramid network, assign different weights to different features, and generate rotation and translation predictions of the object;

[0060] The pyramid network includes the Attention Pyramid Network (APN) and the Feature Pyramid Network (FPN). The Attention Pyramid Network includes a convolutional layer, a SE module (Squeeze-and-Excitation, SEBlock) and a Spatial Attention Module (SAM).

[0061] APN combines FPN and hybrid attention mechanism. The overall architecture is as follows Figure 1 As shown in Figure 2, the input of APN is an RGBD image. Under ideal conditions, the proposed APN can perform pose estimation well in complex backgrounds, different lighting conditions, and occlusion conditions.

[0062] In the present invention, step S2 mainly includes the following steps:

[0063] S21: Input the processed RGBD image into a convolution layer with a convolution kernel of 3×3 to extract the basic features of the image.

[0064] S22: Input the basic features into the SE module, assign different weights to different features and perform global learning on the importance of channels, so that the network can dynamically adjust the response of each channel and highlight the key information.

[0065] The attention mechanism distinguishes the importance of different features by giving them different weights. This approach allows the model to focus more on features that are closely related to the target task, rather than distracting attention from background information that is irrelevant to the task or redundant. Therefore, the present invention uses a hybrid attention mechanism that combines the SE module and the SAM module.

[0066] Here, "giving different weights to different features" specifically includes the following steps:

[0067] First, the convolutional layer inputs a H×W×C size network to the SE module. 1 Features F 1 , where H, W, C 1 are the length, width, and number of channels of the feature respectively; 1 Perform a Squeeze operation and obtain the global information of the feature map through global average pooling to obtain a 1×1×C 1 Features F 2 ;

[0068] Secondly, perform excititon operation to 1 It is sent to a multi-layer perceptron (MLP) consisting of a fully connected layer, a relu layer, and a logistic function activation layer to obtain the importance weight coefficient M 1 ;

[0069] Finally, a scale (weighting) operation is performed, and the input features are combined with M 1 Multiply to get the channel attention feature F 3 .

[0070] The SE module can dynamically adjust the feature map of each channel based on global information, effectively enhancing key features and suppressing unimportant features. This dynamic calibration mechanism significantly improves the model's sensitivity to key features, especially when performing pose estimation in complex backgrounds, and can more accurately highlight key parts.

[0071] S23: The spatial attention module aggregates channel responses and applies convolution to assign different weights to different features, generating a spatial attention map to emphasize key areas in the image.

[0072] Here, "giving different weights to different features" specifically includes the following steps:

[0073] First, the SE module takes the channel attention feature F 3 Output to the SAM module, the SAM block generates a feature map F by combining maximum pooling and average pooling 4 and F 5 , generate a spatial attention map;

[0074] Secondly, they are concatenated and input into the perceptron consisting of a fully connected layer and a logistic function activation layer to obtain the spatial attention weight coefficient M 2 ;

[0075] Finally, the input features are combined with M 2 Multiply to get the enhanced feature F 6 .

[0076] The SAM module indicates which spatial regions are more critical, helping the model focus on these regions. For pose estimation applications, this means that the model can focus more on key areas related to shape and pose, thereby improving the overall estimation accuracy.

[0077] S24: The spatial attention map is fed into the feature pyramid network and generates rotation and translation predictions of the object.

[0078] This structure fuses features from different scales, ensuring that details and contextual information are maintained even at deep levels. Finally, high-level features are converted into rotation and translation predictions of the object through a network block consisting of a linear layer and an activation function (LRL for short). These predictions are usually generated by fully connected layers, which convert complex features into specific parameters for pose estimation.

[0079] The FPN (Feature Pyramid Network) is designed to enhance the model's ability to handle multi-scale features. It consists of multiple convolutional layers and lateral connections. The core of this structure is that the top convolution first processes the deepest features, which are then upsampled and combined with shallow features through lateral connections. This feature combination is repeated at different levels to form a comprehensive, multi-level feature map. This structure is particularly suitable for handling small-scale object detection because it provides rich contextual information, thereby enhancing the model's feature recognition capabilities at different scales.

[0080] S3: Construct a loss function, train and validate the model using the 6D pose estimation dataset, and evaluate the accuracy and robustness of the model in handling the 6D pose estimation task.

[0081] APN is a neural network that directly regresses the pose of an object using RGBD images. In order to effectively train the network, a loss function consisting of two parts is designed. In the present invention, the loss function L is obtained by formula 1:

[0082] L=λ 1 l MSE +λ 2 l dist (1)

[0083] where λ 1 ,2 is the equilibrium constant coefficient; l MSE , l dist There are two parts designed for the loss function.

[0084] l MSE , l dist Obtained by formula 2:

[0085]

[0086] Among them, r i and t j are the elements of the actual rotation matrix and translation vector respectively; and are the elements of the predicted rotation matrix and translation vector respectively; M represents the set of randomly selected 3D points of the target object; k represents the number of 3D points; R and T represent the actual rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

[0087] The loss function combines the average squared error between the predicted value and the true value and the average distance of the model points. In order to suppress the large gradient changes that may be caused by outliers, the balance coefficient is fine-tuned to ensure the stability of the loss during training. This design is intended to help the network learn more accurate object postures and improve generalization capabilities in various scenarios.

[0088] The 6D pose estimation dataset models are the INEMOD dataset and the Occlusion LINEMOD dataset.

[0089] The INEMOD dataset is a standard dataset in the field of 6D pose estimation of objects, containing more than 15,000 images, covering 13 objects with weak textures. Due to the lack of obvious texture of the objects, the complexity of the image background and the variability of lighting, these characteristics show many challenges in 6D pose estimation of the LINEMOD dataset.

[0090] Occlusion LINEMOD is created by annotating a subset of the LINEMOD dataset to represent occlusion. It contains 8 objects that are severely occluded. This occlusion greatly increases the difficulty of pose estimation. This dataset can be used to test and verify the robustness of pose estimation algorithms when dealing with occlusion.

[0091] The evaluation indicators of the model's accuracy and robustness in processing 6D pose estimation tasks are 2D projection indicators, average distance, and closest point distance.

[0092] In order to accurately evaluate the performance of the model in 6D pose estimation, two main indicators are used, including 2D projection indicator and average distance (Average Distance of Deviation, ADD(-S)) indicator. These indicators provide a comprehensive and detailed method to evaluate the accuracy and robustness of the model when handling 6D pose estimation tasks.

[0093] The 2D projection index mainly calculates the average pixel distance between the 3D point projection under the predicted pose and the 3D point projection under the true pose. When using the reprojection error evaluation, if the average distance is less than 5 pixels, the predicted pose is considered accurate. The 2D projection index Pro-2d is obtained by formula 3:

[0094]

[0095] Where: M is the object's 3D point set; m is the number of 3D points; K is the camera's intrinsic parameter matrix; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

[0096] The average distance (ADD(-S)) metric is widely used to quantify the error of 6D pose. It is evaluated by calculating the average distance of the object's 3D points between the true annotated pose and the predicted pose. For asymmetric objects, the ADD metric measures the average distance between the object's 3D points under the transformation of the predicted pose and the true pose. For symmetric objects, the closest point distance (ADD-S) is used as the evaluation criterion. When the calculated distance is less than 10% of the object diameter, the estimated pose is considered correct.

[0097] Among them, the average distance ADD is obtained by formula 4:

[0098]

[0099] Where m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively;

[0100] The closest point distance ADD-S is obtained by formula 5:

[0101]

[0102] Where, m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

[0103] Two public datasets are used: LINEMOD dataset and Occlusion LINEMOD dataset. 85% of the samples of LINEMOD dataset are used as training set, and the remaining 15% are used for testing. For Occlusion LINEMOD dataset, it is only used as a test set to evaluate the performance of the model in handling occlusion.

[0104] During the model training process, the initial learning rate was set to 0.001, the bathsize was set to 16, the epoch was set to 100, and the Adam optimizer was used. The learning rate decayed by 0.01 after each epoch.

[0105] In order to verify the performance of the APN network, the 6D pose estimated by the APN network is evaluated using the 2D projection index and the ADD(-S) index, and compared with different methods in the field of object 6D pose estimation, including BB8, PVNet, YOLO6D, DPOD, and CDPN. The experimental results are shown in Tables 1 and 2.

[0106] Table 1 Comparison of various methods on the LINEMOD dataset using 2D projection indicators Unit: %

[0107]

[0108] Table 2 Comparison of various methods on the LINEMOD dataset using the ADD(-S) indicator Unit: %

[0109]

[0110] Table 1 shows the comparison results of the method of the present invention and the prior art in terms of 2D projection accuracy. Table 2 further reveals the comparison of the method of the present invention with the prior art in terms of the ADD(-S) index. The results show that the method of the present invention exhibits superior performance compared with other methods. Compared with the PVNet method with better performance, the method of the present invention shows similar performance in 2D projection accuracy. However, in terms of the more critical ADD(-S) index, the method of the present invention has improved significantly by 3.7 percentage points. Even without the addition of a refined network, the method can achieve high-precision posture estimation. This proves that APN has obvious advantages in dealing with objects with complex backgrounds and weak textures, highlighting its superiority in accurately estimating rotation matrices and translation vectors, and can detect targets stably by identifying key features.

[0111] Estimating the 6D pose of an object under occlusion conditions is a challenge in this field. This paper uses the model trained on the LINEMOD dataset to test on the Occlusion LINEMOD dataset. The experimental results of Oberweger, SegDriven, PVNet, SSPE, and PoseCNN on the Occlusion LINEMOD dataset are compared, as shown in Tables 4 and 5.

[0112] Table 3 Comparison of various methods on the Occlusion LINEMOD dataset using 2D projection indicators Unit: %

[0113] method ape can cat driller duck eggbox glue holepuncher average value Oberweger 69.9 82.6 65.1 73.8 61.4 13.1 54.9 66.4 60.9 SegDriven 59.1 59.8 46.9 59.0 42.6 11.9 16.5 63.6 44.9 PVNet 69.1 86.1 65.1 73.1 61.4 8.4 55.4 69.8 61.1 APN 68.7 85.4 67.4 73.5 62.1 10.9 55.3 70.1 61.6

[0114] Table 4 Comparison of various methods on the Occlusion LINEMOD dataset using the ADD(-S) indicator Unit: %

[0115] method ape can cat driller duck eggbox glue holepuncher average value Oberweger 17.6 53.6 3.3 62.4 19.2 25.9 39.6 21.3 30.4 SegDriven 12.1 39.9 8.2 45.2 17.2 22.1 35.8 36.0 27.0 SSPE 19.2 65.1 18.9 69.0 25.3 52.0 51.4 45.6 43.3 PVNet 15.8 63.3 16.7 65.7 25.2 50.2 49.6 39.7 20.8 PoseCNN 9.6 45.2 0.9 41.4 19.6 22.0 38.5 22.1 24.9 APN 22.1 68.5 20.4 71.1 24.8 48.2 49.6 48.8 44.2

[0116] Table 3 shows the comparison results between the method of the present invention and the prior art in terms of 2D projection indicators, while Table 4 shows the comparison results between the method of the present invention and the ADD(-S) indicator. Through these comparisons, it can be clearly seen that the method of the present invention performs well in a variety of different technologies, and shows above-average performance in both 2D projection indicators and ADD-S indicators. In terms of 2D projection indicators, the method of the present invention is 0.5 percentage points higher than the PVNet with better performance. In terms of ADD-S indicators, it is 0.9 percentage points higher than the outstanding SSPE. Even when the object is partially occluded, the method of the present invention can still effectively maintain accurate posture estimation, demonstrating its stability and high adaptability to various situations. This not only proves the robustness of the technical solution of the present invention in dealing with occlusion situations, but also highlights its wide applicability and effectiveness in complex environments.

[0117] The present invention proposes a deep learning-based six-degree-of-freedom pose estimation method for objects, which includes an innovative attention pyramid network (PPN). The network structure cleverly integrates the feature pyramid network (FPN) and the channel-space attention mechanism, which can effectively learn from a single RGBD image and extract rich color and geometric information in complex scenes, thereby achieving accurate pose estimation. Experimental verification was carried out on two widely used datasets, and the results showed that APN exhibited excellent pose estimation performance when dealing with objects with complex backgrounds and weak textures. Especially when dealing with occlusion problems, APN not only demonstrated its strong robustness, but also confirmed its wide applicability in various scenarios.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the six-degree-of-freedom pose of an object based on deep learning, characterized in that: The following steps are involved: S1: Obtain the original RGBD image and process the original RGBD image through mask technology; S2: construct a pyramid network, input the processed RGBD image into the pyramid network, assign different weights to different features, and generate rotation and translation predictions of the object; S3: Construct a loss function, train and validate the model using the 6D pose estimation dataset, and evaluate the accuracy and robustness of the model in handling the 6D pose estimation task.

2. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 1, characterized in that: Step S1 specifically includes: S11: Crop the original RGBD image using the mask; S12: Enlarge the cropped image based on its aspect ratio, centering on the centroid of the cropped image.

3. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 1, characterized in that: In step S2, the pyramid network includes an attention pyramid network and a feature pyramid network, and the attention pyramid network includes a convolutional layer, a SE module and a spatial attention module.

4. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 3, characterized in that: The step S2 specifically includes: S21: Input the processed RGBD image into a convolution layer with a convolution kernel of 3×3 to extract the basic features of the image; S22: inputting the basic features into the SE module, assigning different weights to different features and performing global learning on channel importance; S23: The spatial attention module generates a spatial attention map by aggregating channel responses and applying convolution to assign different weights to different features; S24: Input the spatial attention map into the feature pyramid network and generate rotation and translation predictions of the object.

5. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 4, characterized in that: The step S22 of "giving different weights to different features" specifically includes the following steps: First, the convolutional layer inputs a feature F1 of size H×W×C1 to the SE module, where H, W, and C1 are the length, width, and number of channels of the feature, respectively. F1 is squeezed and the global information of the feature map is obtained through global average pooling to obtain a feature F2 of size 1×1×C1. Secondly, perform an excititon operation and send F1 to a multi-layer perceptron (MLP) consisting of a fully connected layer, a relu layer, and a logistic function activation layer to obtain the importance weight coefficient M1. Finally, a scale operation is performed, and the input feature is multiplied by M1 to obtain the channel attention feature F3.

6. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 5, characterized in that: The step S23 of "giving different weights to different features" specifically includes the following steps: First, the SE module outputs the channel attention feature F3 to the SAM module. The SAM block generates a spatial attention map by combining the feature maps F4 and F5 generated by maximum pooling and average pooling. Secondly, concatenate them and input them into the perceptron composed of the fully connected layer and sigmoid to obtain the spatial attention weight coefficient M2; Finally, the input feature is multiplied by M2 to obtain the enhanced feature F6.

7. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 1, characterized in that: The loss function L in step S3 is obtained by formula 1: L=λ1l MSE +λ2l dist (1) Among them, λ1 and λ2 are equilibrium constant coefficients; l MSE , l dist There are two parts designed for the loss function.

8. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 7, characterized in that: l MSE , l dist Obtained by formula 2: Among them, r i and t j are the elements of the actual rotation matrix and translation vector respectively; and are the elements of the predicted rotation matrix and translation vector respectively; M represents the set of randomly selected 3D points of the target object; k represents the number of 3D points; R and T represent the actual rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

9. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 1, characterized in that: In step S3, the 6D pose estimation dataset model is the INEMOD dataset and the Occlusion LINEMOD dataset, and the evaluation indicators of the accuracy and robustness of the model when processing the 6D pose estimation task are the 2D projection index, the average distance and the closest point distance.

10. The method for estimating the six-degree-of-freedom pose of an object based on deep learning according to claim 9, characterized in that: The 2D projection index Pro-2d is obtained by formula 3: Where: M is the set of 3D points of the object; m is the number of 3D points; K is the camera intrinsic parameter matrix; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively; The average distance ADD is obtained by formula 4: Where m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively; The closest point distance ADD-S is obtained by formula 5: Where, m is the number of 3D points; R and T represent the real rotation matrix and translation vector respectively; and Represent the predicted rotation matrix and translation vector respectively.

Images