Six-degree-of-freedom pose estimation method for objects based on 3D geometric information registration

Abstract

This application belongs to the field of image processing technology, and specifically relates to a method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration, including a target detection module, a three-dimensional point cloud generation model, and a geometric information registration pose estimation module. The present invention combines geometric reconstruction and high-precision geometric information registration technology with the object pose estimation task. The geometric reconstruction module introduces object geometric feature extraction and feature decoding to achieve an in-depth understanding and modeling of the algorithm's geometric features. The entire process conforms to geometric disorder, further improving the algorithm's accuracy and scene adaptability. The present invention can achieve accurate estimation of the six-degree-of-freedom pose of an object, providing strong support for fields such as industrial automation, robotic vision, AR / VR, autonomous driving, and medical robotics.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration. Background Art

[0002] To meet the growing demand for object positioning and pose accuracy in fields such as industrial automation, precision robotic grasping, intelligent manufacturing, and augmented reality, various industries are constantly seeking more efficient and accurate six-degree-of-freedom pose estimation technologies. Traditional pose estimation methods often rely on two-dimensional image features or low-precision sensors. These methods are easily affected by factors such as noise, occlusion, and lighting changes in complex scenes, resulting in insufficient positioning accuracy and poor robustness.

[0003] Traditional algorithms and hand-crafted feature matching methods often suffer from high computational complexity, slow convergence, and a tendency to fall into local optimality when faced with high-precision requirements. This makes it difficult to achieve real-time monitoring and accurate detection at all angles and all times. Furthermore, while existing point cloud processing technologies have somewhat compensated for the shortcomings of two-dimensional information, bottlenecks such as low computational efficiency and error accumulation exist during point cloud data acquisition, preprocessing, and registration, making it difficult to meet the requirements of modern high-precision industrial applications.

[0004] With the rapid development of high-precision 3D data acquisition technologies such as laser scanning and depth cameras, as well as the widespread application of new algorithms such as deep learning and global optimization in computer vision, high-precision geometric information registration has gradually become an effective approach to solving the problem of estimating the six-degree-of-freedom pose of objects. Advanced geometric information registration technology can fully extract and utilize an object's 3D structural information, achieving comprehensive, multi-level, and accurate estimation of its position and pose. This significantly improves the accuracy and stability of pose detection, providing strong technical support for the refined operation of industrial automation and intelligent control systems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration, so as to improve the accuracy and robustness of the six-degree-of-freedom pose estimation of the object, and provide strong support for industrial automation, robotic vision, AR / VR, autonomous driving, medical robots and other fields.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration, comprising a target detection module, a three-dimensional point cloud generation model, and a geometric information registration pose estimation module;

[0008] Object detection module, for object detection and bounding box prediction;

[0009] The 3D point cloud generation model includes a deep set feature extraction network and a spatial 3D point cloud decoding network;

[0010] Deep ensemble feature extraction network: It combines RGB and depth information and converts RGBD images into global feature representations through an encoder;

[0011] Spatial 3D point cloud decoding network: Generates an unordered point cloud through the decoder, and reconstructs the collective information in the object and camera coordinate systems based on depth information, that is, 3D point cloud generation, thereby achieving high-quality geometric information reconstruction in multiple coordinate systems driven by depth images;

[0012] The geometric information registration pose estimation module aligns the decoded three-dimensional point cloud in the object coordinate system and the calculated point cloud in the real camera coordinate system. The former is used as the source point cloud and the latter as the target point cloud. The rotation and translation transformation from the source point cloud to the target point cloud is calculated to realize the six-degree-of-freedom pose estimation of the corresponding object.

[0013] Preferably, the specific steps of the target detection module training phase are as follows:

[0014] (1) Based on the RGBD data in the pose estimation dataset and the bounding box information of each object in the image, the YOLO format training dataset is constructed, screened and enhanced in the pose estimation scenario;

[0015] (2) By introducing the PGI mechanism, the model is allowed to learn different types of targets in a more accurate and efficient manner through an adjustable gradient update strategy during training. The gradient weighting module dynamically adjusts the gradient weight according to the object category and detection difficulty.

[0016] (3) Optimize the feature extraction and information fusion process of the YOLOv9 network. In the multi-level feature aggregation stage, information aggregation is performed between the various convolutional layers of the YOLOv9 network, integrating low-level detail features with high-level semantic features to capture object information at different scales and complex backgrounds. At the same time, the adaptive feature fusion module dynamically adjusts the fusion strategy of different levels of features according to the complexity of the object and background interference, so that the model can more accurately identify and locate the target;

[0017] (4) The loss is calculated based on cross entropy, and the gradient is calculated by back propagation to update the parameters to complete the model training.

[0018] Preferably, the steps for training the 3D point cloud generation model are as follows:

[0019] (1) Based on the trained YOLOv9 target detection model, the pose estimation dataset is preprocessed. For the input RGB image , corresponding to the depth map , use the RGB part of the image for target detection, and get the bounding box of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence;

[0020] (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes , based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object , so far the dataset is created;

[0021] (3) For the RGBD image of the object in each batch during the training phase, calculate the point cloud in the real camera coordinate system , based on the real pose label of each object, calculate the point cloud in the real object coordinate system ;

[0022] (4) For the RGBD image of the object in each batch during the training phase, the spatial 3D point cloud decoding network predicts the point cloud in the object coordinate system ,calculate and The Chamfer Distance is used as the loss and the back-propagation training model is performed.

[0023] Preferably, the training steps of the geometric information registration pose estimation module are as follows:

[0024] (1) Based on the object RGBD dataset, for each batch of object RGBD images in the training phase, calculate the point cloud in the real camera coordinate system ;

[0025] (2) For the RGBD image of the object in each batch during the training phase, the point cloud in the object coordinate system is predicted based on the trained 3D point cloud generation model. ;

[0026] (3) Based on the DCP geometric information registration network, the point cloud under the object coordinates is predicted As the source point cloud, the point cloud in the real camera coordinate system As the target point cloud, predict the object's pose, including the rotation matrix With translation vector ;

[0027] (4) Calculate the true rotation matrix and Cross entropy loss , calculate the true translation vector and Cross entropy loss , + Backpropagation gradient calculation and model parameter update are performed as loss.

[0028] Preferably, the target detection module implements object recognition and positioning in RGB images based on YOLOv9, and segments RGBD images of different objects. The specific steps of the inference stage are as follows:

[0029] (1) For the input RGB image , the corresponding depth map is , use the RGB part of the image for target detection, and get the bounding box of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence;

[0030] (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes , based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object .

[0031] Preferably, the specific steps of the 3D point cloud generation model reasoning stage are as follows:

[0032] (1) For RGBD image input, the input channels need to be expanded from 3 to 4 to accommodate the input data. The modified convolution kernel size is , after the first layer of convolution, the first feature map is obtained The entire feature extraction part is composed of multiple residual stacks. From the input to the final feature map after passing through all the residual blocks, the pooling layer is added to compress the spatial dimension of the feature map into a feature vector. ;

[0033] (2) The spatial 3D point cloud decoding network further decodes the features extracted from the image and defines the tree structure of the decoding network , where the tree structure satisfies The number of point cloud points N, the initial input is the initial feature vector , high-dimensional features are obtained after layer 0 , the dimension changes to obtain the first layer feature set ;

[0034] (3) The next layers 1 to L-2 cannot directly use the fully connected layer to realize the spatial mapping of the two-dimensional matrix. One-dimensional convolution is used for feature mapping. In order to maintain the feature direction of the decoding process, the initial feature vector is concatenated before each calculation to obtain , according to the tree structure described above, the first layer can be expanded times the number of features, that is, at this time, the features are further mapped to At this time, the feature dimension transformation is performed to further expand the number of feature sets to obtain the first layer output feature set ;

[0035] (4) Similarly, from the 2nd layer to the L-2th layer, feature decoding and feature set accumulation are continuously performed, and the number of features in the set is continuously approached to the number of points in the point cloud to be generated, until the L-2 layer, at which time the feature set is obtained. Similarly, after concatenating the initial feature vectors, the last layer maps each feature to a 3D space to obtain the generated point cloud .

[0036] Preferably, the geometric information registration pose estimation module realizes the registration of the source point cloud and the target point cloud. The specific steps of the inference stage are as follows:

[0037] (1) Based on the RGBD image of the object, the depth information and the object segmentation information, the point cloud in the camera coordinate system is calculated at this time, and the farthest point sampling is further performed to reduce the number of point cloud points to N, and the result is , , based on the 3D point cloud generation model, the point cloud in the object coordinate system is obtained , ; Consider the point cloud in the object coordinate system as the source point cloud, and the point cloud in the camera coordinate system as the target point cloud. That is, the transformation from the source point cloud to the target point cloud is the pose of the object in the image at this time;

[0038] (2) Based on the feature mapping part of the PointNet network as the DCP feature extraction network, each point is processed independently by a multi-layer perceptron with shared parameters, and the high-dimensional features of each point are extracted to obtain the feature set corresponding to the target point cloud and the source point cloud. and , further apply maximum pooling to the two feature sets to obtain the global feature vectors and , and then concatenate it with the feature set output by the last layer to obtain and , use MLP to achieve equal-dimensional mapping for each feature in the set at this time, and obtain the final high-dimensional feature set and ;

[0039] (3) Two point cloud high-dimensional feature sets are used for self-attention calculation, and the input point cloud high-dimensional features go through , , Generate query (Query, Q), key (Key, K) and value (Value, V), , , , map the features to the feature_dim dimension, calculate the attention weight matrix A through the dot product operation, and then model the global information learned by the attention mechanism into the value vector by clicking A and the value vector, so as to realize dynamic attention to different features in the entire feature set, and obtain , the two point cloud feature sets are self-attention calculated separately, and we get and , , , Respectively represent the mapping matrices of query vector, key vector, and value vector in self-attention calculation;

[0040] (4) Perform cross-attention calculation to learn the correspondence between the two, and use the query matrix to map the feature vector of the first point cloud to obtain , a key and value mapping matrix to map the feature vectors of the second point cloud, , , after performing cross attention calculations, we can get and ;

[0041] (5) Calculate the similarity matrix of two point cloud feature sets ,in Indicates the similarity between the i-th point of the source point cloud and the j-th point of the target point cloud. At this time, the position with the highest similarity becomes the correspondence between the two point clouds, and a two-way consistency check is adopted, that is, the best match from the source point cloud point to the target point cloud point. and the best match of the target point cloud point to the source point cloud point Must be consistent, ,

[0042] (6) Extract the source point cloud that meets the corresponding relationship With the target point cloud , calculate the covariance matrix Represent the geometric relationship between point clouds and perform singular value decomposition on the covariance matrix , calculate the pose rotation matrix , further calculate the translation vector based on the center of mass position ,in and They represent the centroids of the source point cloud and target point cloud calculated according to the correspondence.

[0043] Preferably, the target detection module implements YOLOv9 target detection model training based on the bounding box information in the pose estimation dataset; the target detection module performs object target detection on the pose estimation dataset, and cuts out RGBD images of all objects to construct a subsequent training dataset; the three-dimensional point cloud generation model implements the training of a three-dimensional space geometric reconstruction model composed of an image feature extractor and a TopNet point cloud decoder based on the constructed object image RGBD dataset and the pose labels of all objects; the geometric information registration module completes the training of the geometric information registration model DCP based on the point cloud in the camera coordinate system calculated based on the object RGBD image and the point cloud in the object coordinate system generated by the three-dimensional space geometric reconstruction model, and combines the object's real pose label.

[0044] Preferably, an RGBD image containing multiple objects is input, and the target detection model detects the category information and bounding box information of each object in the image, and cuts out each object according to the bounding box information; for the RGBD image of each object, the high-dimensional feature extraction of the image and the decoding and generation of the three-dimensional point cloud are completed based on the three-dimensional point cloud generation model to obtain the point cloud in the object coordinate system, and at the same time, the real camera coordinate system geometric information registration model is calculated based on the image depth information to realize the estimation of the object pose.

[0045] Compared with the prior art, this application has the following beneficial effects:

[0046] This method combines deep learning technology with object pose estimation, fully utilizing the object category and bounding box information extracted by the target detection model, and combining it with the depth data in the RGBD image to achieve high-precision object segmentation and three-dimensional point cloud generation. In this way, the high-dimensional features of each object can be accurately extracted, and point cloud data in the object coordinate system can be generated. This method not only improves the accuracy of object pose estimation, but also avoids the over-reliance on image features in traditional methods during the processing process, ensuring that the estimation results remain highly stable in complex environments and under different viewing angles.

[0047] By leveraging the DCP geometric information registration model, the present invention enables efficient and precise alignment of source and target point clouds, thereby accurately estimating the object's pose. This technology not only addresses the noise interference and error accumulation issues inherent in traditional image-based pose estimation methods, but also significantly improves processing speed and robustness. It is particularly suitable for applications such as industrial automation and robotic grasping, where high-precision pose estimation is required.

[0048] The six-degree-of-freedom pose estimation method of the present invention eliminates the need for complex surface calibration or excessive manual design. It boasts a high degree of automation and adaptability, enabling rapid deployment and real-time operation in a variety of scenarios, significantly reducing system operation and maintenance costs. Overall, the present invention offers significant advantages in improving object recognition and positioning accuracy, shortening processing time, and reducing hardware reliance, promising broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0050] Figure 1 A schematic diagram disclosed in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the process disclosed in the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0053] A method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration, comprising a target detection module, a three-dimensional point cloud generation model, and a geometric information registration pose estimation module;

[0054] The object detection module uses the YOLOv9 network to complete object detection tasks in object pose estimation scenarios. It fully utilizes the network's programmable gradient information mechanism (PGI) and generalized efficient layer aggregation network mechanism (GELAN) to ensure that local details of small targets are preserved during feature transfer, significantly improving the robustness of recognition of partially visible targets. At the same time, GELAN supports flexible multi-scale feature fusion and reduces the potential mismatching problems that may occur when objects are densely arranged through a hierarchical interaction mechanism.

[0055] The 3D point cloud generation model consists of a deep ensemble feature extraction network and a spatial 3D point cloud decoding network. Based on image encoding and 3D geometric information decoding, it achieves the generation of 3D point clouds and reconstruction of object geometry in the object coordinate system. The deep ensemble feature extraction network integrates RGB and depth information, converting the RGBD image into a global feature representation via an encoder. The spatial 3D point cloud decoding network generates an unordered point cloud using decoders such as TopNet, significantly improving the quality of 3D spatial geometric reconstruction. Simultaneously, depth information is used to reconstruct geometric information and generate 3D point clouds in the object and camera coordinate systems, enabling high-quality geometric reconstruction in multiple coordinate systems driven by depth images.

[0056] The geometric information registration pose estimation module is responsible for registering the decoded 3D point cloud in the object coordinate system and the calculated point cloud in the real camera coordinate system. The former is used as the source point cloud and the latter as the target point cloud. The rotation and translation transformation from the source point cloud to the target point cloud is calculated to achieve the six-degree-of-freedom pose estimation of the corresponding object. At the same time, the registration process of this module conforms to the disorder of the point cloud, ensuring that the registration result is not affected by the order of the point cloud input, thereby improving the robustness and adaptability of the pose estimation method. The specific method flow is as follows: Figure 2 As shown, the following steps are included:

[0057] Training phase: Unlike traditional pose estimation methods that use end-to-end regression for object center points, the target detection module in this method implements YOLOv9 target detection model training based on the bounding box information in the pose estimation dataset; the target detection module performs object target detection on the pose estimation dataset, and cuts out RGBD images of all objects to construct subsequent training datasets; the 3D point cloud generation model implements the training of the 3D space geometric reconstruction model composed of the image feature extractor and the TopNet point cloud decoder based on the constructed object image RGBD dataset and the pose labels of all objects; the geometric information registration module completes the training of the geometric information registration model DCP (Deep Closest Point) based on the point cloud in the camera coordinate system calculated from the object RGBD image and the point cloud in the object coordinate system generated by the 3D space geometric reconstruction model, and combines the object's true pose label.

[0058] Inference stage: Input RGBD image, the image contains multiple objects, the target detection model detects the category information and bounding box information of each object in the image, and cuts out each object according to the bounding box information; for each object's RGBD image, the high-dimensional feature extraction and 3D point cloud decoding generation of the image are completed based on the 3D point cloud generation model to obtain the point cloud in the object coordinate system, and at the same time, the point cloud in the real camera coordinate system is calculated based on the image depth information; the predicted point cloud in the object coordinate system is used as the source point cloud, and the point cloud in the real camera coordinate system is used as the target point cloud, and the object pose is estimated through the DCP geometric information alignment model.

[0059] 1. Target detection model training:

[0060] (1) Based on the RGBD data in the pose estimation dataset and the bounding box information of each object in the image, the YOLO format training dataset is constructed, screened and enhanced in the pose estimation scenario;

[0061] (2) Adopting the Programmable Gradient Information (PGI) mechanism, the gradient optimization capability of the YOLOv9 network in target detection tasks is enhanced by introducing the PGI mechanism. The PGI mechanism allows the model to learn different types of targets in a more accurate and efficient manner through an adjustable gradient update strategy during training. The gradient weighting module dynamically adjusts the gradient weight according to the object category and detection difficulty. For objects that are more difficult to detect (such as small objects or occluded objects), PGI increases their gradient weight, allowing the model to pay more attention to these difficult-to-detect targets during training. On the other hand, in the gradient clipping part, gradient clipping is used to avoid the gradient explosion problem during the gradient calculation process, ensuring stable convergence of the network.

[0062] (3) The generalized efficient layer aggregation network mechanism (GELAN) is used to optimize the feature extraction and information fusion process of the YOLOv9 network. The GELAN mechanism improves the model's ability to detect multi-scale targets by effectively aggregating features at different levels in the network. In the multi-level feature aggregation stage, information is aggregated between the various convolutional layers of the YOLOv9 network, combining low-level detail features with high-level semantic features to capture object information at different scales and in complex backgrounds. At the same time, the adaptive feature fusion module dynamically adjusts the fusion strategy of different levels of features according to the complexity of the object and background interference, enabling the model to more accurately identify and locate targets.

[0063] (4) The loss is calculated based on cross entropy, and the gradient is calculated by back propagation to update the parameters to complete the model training.

[0064] The reasoning process of the target detection module is as follows:

[0065] (1) For the input RGB image , the corresponding depth map is Use the RGB part of the image for target detection to obtain the bounding box (BouningBox) of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence.

[0066] (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes Based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object. .

[0067] 2. The training steps for the 3D point cloud generation model are as follows:

[0068] (1) Based on the trained YOLOv9 target detection model, the pose estimation dataset is preprocessed. For the input RGB image , corresponding to the depth map , use the RGB part of the image for target detection, and get the bounding box of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence;

[0069] (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes , based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object , so far the dataset is created;

[0070] (3) For the RGBD image of the object in each batch during the training phase, calculate the point cloud in the real camera coordinate system , based on the real pose label of each object, calculate the point cloud in the real object coordinate system ;

[0071] (4) For the RGBD image of the object in each batch during the training phase, the spatial 3D point cloud decoding network predicts the point cloud in the object coordinate system ,calculate and The Chamfer Distance is used as the loss and the back-propagation training model is performed.

[0072] The inference steps of the 3D point cloud generation model are as follows:

[0073] (1) For RGBD image input, the input channels need to be expanded from 3 to 4 to accommodate the input data. The modified convolution kernel size is , after the first layer of convolution, the first feature map is obtained The entire feature extraction part is composed of multiple residual stacks. From the input to the final feature map after passing through all the residual blocks, the pooling layer is added to compress the spatial dimension of the feature map into a feature vector. ;

[0074] (2) The spatial 3D point cloud decoding network further decodes the features extracted from the image and defines the tree structure of the decoding network , where the tree structure satisfies The number of point cloud points N, when the number of point cloud points is set to 2048, the common decoding tree structure is {2: [32, 64], 4: [4, 8, 8, 8], 6: [2,4, 4, 4, 4, 4], 8: [2, 2, 2, 2, 2, 4, 4, 4]}. The initial input is the initial feature vector , high-dimensional features are obtained after layer 0 , the dimension changes to obtain the first layer feature set ;

[0075] (3) The next layers 1 to L-2 cannot directly use the fully connected layer to realize the spatial mapping of the two-dimensional matrix. One-dimensional convolution is used for feature mapping. In order to maintain the feature direction of the decoding process, the initial feature vector is concatenated before each calculation to obtain , according to the tree structure described above, the first layer can be expanded times the number of features, that is, at this time, the features are further mapped to At this time, the feature dimension transformation is performed to further expand the number of feature sets to obtain the first layer output feature set ;

[0076] (4) Similarly, from the 2nd layer to the L-2th layer, feature decoding and feature set accumulation are continuously performed, and the number of features in the set is continuously approached to the number of points in the point cloud to be generated, until the L-2 layer, at which time the feature set is obtained. Similarly, after concatenating the initial feature vectors, the last layer maps each feature to a 3D space to obtain the generated point cloud .

[0077] 3. The training steps of the geometric information registration pose estimation module are as follows:

[0078] (1) Based on the object RGBD dataset, for each batch of object RGBD images in the training phase, calculate the point cloud in the real camera coordinate system ;

[0079] (2) For the RGBD image of the object in each batch during the training phase, the point cloud in the object coordinate system is predicted based on the trained 3D point cloud generation model. ;

[0080] (3) Based on the DCP geometric information registration network, the point cloud under the object coordinates is predicted As the source point cloud, the point cloud in the real camera coordinate system As the target point cloud, predict the object's pose, including the rotation matrix With translation vector ;

[0081] (4) Calculate the true rotation matrix and Cross entropy loss , calculate the true translation vector and Cross entropy loss , + Backpropagation gradient calculation and model parameter update are performed as loss.

[0082] The geometric information registration pose estimation module realizes the registration of the source point cloud and the target point cloud. The specific steps of the inference stage are as follows:

[0083] (1) Based on the RGBD image of the object, the depth information and the object segmentation information, the point cloud in the camera coordinate system is calculated at this time, and the farthest point sampling is further performed to reduce the number of point cloud points to N, and the result is , , based on the 3D point cloud generation model, the point cloud in the object coordinate system is obtained , ; Consider the point cloud in the object coordinate system as the source point cloud, and the point cloud in the camera coordinate system as the target point cloud. That is, the transformation from the source point cloud to the target point cloud is the pose of the object in the image at this time;

[0084] (2) Based on the feature mapping part of the PointNet network as the DCP feature extraction network, each point is processed independently by a multi-layer perceptron with shared parameters, and the high-dimensional features of each point are extracted to obtain the feature set corresponding to the target point cloud and the source point cloud. and , further apply maximum pooling to the two feature sets to obtain the global feature vectors and , and then concatenate it with the feature set output by the last layer to obtain and , use MLP to achieve equal-dimensional mapping for each feature in the set at this time, and obtain the final high-dimensional feature set and ;

[0085] (3) Two point cloud high-dimensional feature sets are used for self-attention calculation, and the input point cloud high-dimensional features go through , , Generate query (Query, Q), key (Key, K) and value (Value, V), , , , map the features to the feature_dim dimension, calculate the attention weight matrix A through the dot product operation, and then model the global information learned by the attention mechanism into the value vector by clicking A and the value vector, so as to realize dynamic attention to different features in the entire feature set, and obtain , the two point cloud feature sets are self-attention calculated separately, and we get and , , , Respectively represent the mapping matrices of query vector, key vector, and value vector in self-attention calculation;

[0086] (4) Perform cross-attention calculation to learn the correspondence between the two, and use the query matrix to map the feature vector of the first point cloud to obtain , a key and value mapping matrix to map the feature vectors of the second point cloud, , , after performing cross attention calculations, we can get and ;

[0087] (5) Calculate the similarity matrix of two point cloud feature sets ,in Indicates the similarity between the i-th point of the source point cloud and the j-th point of the target point cloud. At this time, the position with the highest similarity becomes the correspondence between the two point clouds, and a two-way consistency check is adopted, that is, the best match from the source point cloud point to the target point cloud point. and the best match of the target point cloud point to the source point cloud point Must be consistent, ,

[0088] (6) Extract the source point cloud that meets the corresponding relationship With the target point cloud , calculate the covariance matrix Represent the geometric relationship between point clouds and perform singular value decomposition on the covariance matrix , calculate the pose rotation matrix , further calculate the translation vector based on the center of mass position ,in and Respectively represent the center of mass of the source point cloud and the target point cloud calculated according to the correspondence relationship. At this point, the pose estimation from the source point cloud to the target point cloud (including the rotation matrix R and the translation vector t) is completed.

[0089] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 2 The methods provided in the various optional methods are therefore not described in detail here.

[0090] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0091] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] The methods and related devices provided in the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device, or are transmitted through a computer-readable storage medium. Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The instruction device is implemented in the process Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

Claims

1. A method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration, characterized in that: Including target detection module, 3D point cloud generation model, geometric information registration pose estimation module; Object detection module, for object detection and bounding box prediction; The 3D point cloud generation model includes a deep set feature extraction network and a spatial 3D point cloud decoding network; Deep ensemble feature extraction network: It combines RGB and depth information and converts RGBD images into global feature representations through an encoder; Spatial 3D point cloud decoding network: The decoder generates an unordered point cloud and reconstructs the geometric information in the object and camera coordinate systems based on depth information, i.e., 3D point cloud generation, thereby achieving high-quality geometric information reconstruction in multiple coordinate systems driven by depth images. The specific steps for generating 3D point clouds are as follows: (1) For RGBD image input, the input channels need to be expanded from 3 to 4 to accommodate the input data. The modified convolution kernel size is , after the first layer of convolution, we get the first feature map The entire feature extraction part is composed of multiple residual stacks. From the input to the final feature map after passing through all the residual blocks, the pooling layer is added to compress the spatial dimension of the feature map into a feature vector. ; (2) The spatial 3D point cloud decoding network further decodes the features extracted from the image and defines the tree structure of the decoding network , where the tree structure satisfies The number of point cloud points N, the initial input is the initial feature vector , high-dimensional features are obtained after layer 0 , the dimension changes to obtain the first layer feature set ; (3) The next layers 1 to L-2 cannot directly use the fully connected layer to realize the spatial mapping of the two-dimensional matrix. One-dimensional convolution is used for feature mapping. In order to maintain the feature direction of the decoding process, the initial feature vector is concatenated before each calculation to obtain , according to the tree structure described above, the first layer can be expanded times the number of features, that is, at this time, the features are further mapped to At this time, the feature dimension transformation is performed to further expand the number of feature sets to obtain the first layer output feature set ; (4) Similarly, from the 2nd layer to the L-2th layer, feature decoding and feature set accumulation are continuously performed, and the number of features in the set is continuously approached to the number of points in the point cloud to be generated, until the L-2 layer, at which time the feature set is obtained. Similarly, after concatenating the initial feature vectors, the last layer maps each feature to a 3D space to obtain the generated point cloud ; The geometric information registration pose estimation module aligns the decoded three-dimensional point cloud in the object coordinate system and the calculated point cloud in the real camera coordinate system. The former is used as the source point cloud and the latter as the target point cloud. The rotation and translation transformation from the source point cloud to the target point cloud is calculated to realize the six-degree-of-freedom pose estimation of the corresponding object.

2. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The specific steps of the target detection module training phase are as follows: (1) Based on the RGBD data in the pose estimation dataset and the bounding box information of each object in the image, the YOLO format training dataset is constructed, screened and enhanced in the pose estimation scenario; (2) By introducing the PGI mechanism, the model is allowed to learn different types of targets in a more accurate and efficient manner through an adjustable gradient update strategy during training. The gradient weighting module dynamically adjusts the gradient weight according to the object category and detection difficulty. (3) Optimize the feature extraction and information fusion process of the YOLOv9 network. In the multi-level feature aggregation stage, information aggregation is performed between the various convolutional layers of the YOLOv9 network, integrating low-level detail features with high-level semantic features to capture object information at different scales and complex backgrounds. At the same time, the adaptive feature fusion module dynamically adjusts the fusion strategy of different levels of features according to the complexity of the object and background interference, so that the model can more accurately identify and locate the target; (4) The loss is calculated based on cross entropy, and the gradient is calculated by back propagation to update the parameters to complete the model training.

3. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The training steps for the 3D point cloud generation model are as follows: (1) Based on the trained YOLOv9 target detection model, the pose estimation dataset is preprocessed. For the input RGB image , corresponding to the depth map , use the RGB part of the image for target detection, and get the bounding box of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence; (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes , based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object , so far the dataset is created; (3) For the RGBD image of the object in each batch during the training phase, calculate the point cloud in the real camera coordinate system , based on the real pose label of each object, calculate the point cloud in the real object coordinate system ; (4) For the RGBD image of the object in each batch during the training phase, the spatial 3D point cloud decoding network predicts the point cloud in the object coordinate system ,calculate and The Chamfer Distance is used as the loss and the back-propagation training model is performed.

4. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The training steps of the geometric information registration pose estimation module are as follows: (1) Based on the object RGBD dataset, for each batch of object RGBD images in the training phase, calculate the point cloud in the real camera coordinate system ; (2) For the RGBD image of the object in each batch during the training phase, the point cloud in the object coordinate system is predicted based on the trained 3D point cloud generation model. ; (3) Based on the DCP geometric information registration network, the point cloud under the object coordinates is predicted As the source point cloud, the point cloud in the real camera coordinate system As the target point cloud, predict the object's pose, including the rotation matrix With translation vector ; (4) Calculate the true rotation matrix and Cross entropy loss , calculate the true translation vector and Cross entropy loss , + Backpropagation gradient calculation and model parameter update are performed as loss.

5. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The object detection module uses YOLOv9 to identify and locate objects in RGB images and segment RGBD images of different objects. The specific steps of the inference stage are as follows: (1) For the input RGB image , the corresponding depth map is , use the RGB part of the image for target detection, and get the bounding box of each object in the image as well as the category and confidence information ,in Indicates the object category, Indicates confidence level; (2) Setting the confidence threshold And filter out all high-confidence objects that meet the conditions to obtain a set of valid bounding boxes , based on the bounding box, the RGB image and depth map are intercepted to obtain an image set containing the RGBD information of each object .

6. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The geometric information registration pose estimation module realizes the registration of the source point cloud and the target point cloud. The specific steps of the inference stage are as follows: (1) Based on the RGBD image of the object, the depth information and the object segmentation information, the point cloud in the camera coordinate system is calculated at this time, and the farthest point sampling is further performed to reduce the number of point cloud points to N, and the result is , , based on the 3D point cloud generation model, the point cloud in the object coordinate system is obtained , ; Consider the point cloud in the object coordinate system as the source point cloud, and the point cloud in the camera coordinate system as the target point cloud. That is, the transformation from the source point cloud to the target point cloud is the pose of the object in the image at this time; (2) Based on the feature mapping part of the PointNet network as the DCP feature extraction network, each point is processed independently by a multi-layer perceptron with shared parameters, and the high-dimensional features of each point are extracted to obtain the feature set corresponding to the target point cloud and the source point cloud. and , further apply maximum pooling to the two feature sets to obtain the global feature vectors and , and then concatenate it with the feature set output by the last layer to obtain and , use MLP to achieve equal-dimensional mapping for each feature in the set at this time, and obtain the final high-dimensional feature set and ; (3) Two point cloud high-dimensional feature sets are used for self-attention calculation, and the input point cloud high-dimensional features go through , , Generate query (Query, Q), key (Key, K) and value (Value, V), , , , map the features to the feature_dim dimension, calculate the attention weight matrix A through the dot product operation, and then model the global information learned by the attention mechanism into the value vector by clicking A and the value vector, so as to realize dynamic attention to different features in the entire feature set, and obtain , the two point cloud feature sets are self-attention calculated separately, and we get and , , , Respectively represent the mapping matrices of query vector, key vector, and value vector in self-attention calculation; (4) Perform cross-attention calculation to learn the correspondence between the two, and use the query matrix to map the feature vector of the first point cloud to obtain , a key and value mapping matrix to map the feature vectors of the second point cloud, , , after performing cross attention calculations, we can get and ; (5) Calculate the similarity matrix of two point cloud feature sets ,in Indicates the similarity between the i-th point of the source point cloud and the j-th point of the target point cloud. At this time, the position with the highest similarity becomes the correspondence between the two point clouds, and a two-way consistency check is adopted, that is, the best match from the source point cloud point to the target point cloud point. and the best match of the target point cloud point to the source point cloud point Must be consistent, , (6) Extract the source point cloud that meets the corresponding relationship With the target point cloud , calculate the covariance matrix Represent the geometric relationship between point clouds and perform singular value decomposition on the covariance matrix , calculate the pose rotation matrix , further calculate the translation vector based on the center of mass position ,in and They represent the centroids of the source point cloud and target point cloud calculated according to the correspondence.

7. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: The object detection module implements YOLOv9 object detection model training based on the bounding box information in the pose estimation dataset; The target detection module performs object target detection on the pose estimation dataset and cuts out RGBD images of all objects to construct subsequent training datasets; the 3D point cloud generation model implements the training of the 3D space geometric reconstruction model composed of the image feature extractor and the TopNet point cloud decoder based on the constructed object image RGBD dataset and the pose labels of all objects; the geometric information registration module completes the training of the geometric information registration model DCP based on the point cloud in the camera coordinate system calculated based on the object RGBD image and the point cloud in the object coordinate system generated by the 3D space geometric reconstruction model, and combines the object's real pose label.

8. The method for estimating the six-degree-of-freedom pose of an object based on three-dimensional geometric information registration according to claim 1, characterized in that: Input RGBD image, the image contains multiple objects, the object detection model detects the category information and bounding box information of each object in the image, and cuts out each object based on the bounding box information; For each object's RGBD image, high-dimensional feature extraction and three-dimensional point cloud decoding are completed based on the three-dimensional point cloud generation model to obtain the point cloud in the object coordinate system. At the same time, the real camera coordinate system geometric information registration model is calculated based on the image depth information to realize the estimation of the object's pose.

Images