Object pose estimation method and system based on three-dimensional feature mapping and perspective aggregation

Abstract

The application provides an object pose estimation method and system based on three-dimensional feature mapping and perspective aggregation, and belongs to the technical field of pose estimation.The method is applied to a system comprising an image processing module, a three-dimensional feature mapping module, a perspective aggregation module and a pose estimation module, and specifically comprises the following steps: performing feature extraction on a template image and a target image, performing three-dimensional space mapping on the template image features, determining the template three-dimensional features and perspective information, determining the perspective information of each target image feature and the corresponding template three-dimensional features through feature matching, obtaining the matching relationship between the target image features and the template three-dimensional features through perspective aggregation, selecting a pose estimation algorithm to solve an alternative rough pose estimation result, and iteratively optimizing the initial value of a pose optimization network to obtain an object pose estimation result.The application can effectively reduce the error transmission risk in pose estimation, guarantee the accuracy, reduce the calculation complexity and improve the estimation efficiency.

Description

Technical Field

[0001] This invention relates to the field of attitude estimation technology, and in particular to an object attitude estimation method and system based on three-dimensional feature mapping and view aggregation. Background Technology

[0002] Existing 6D object pose estimation techniques generally suffer from insufficient accuracy, slow speed, poor robustness, and high computational cost due to the inherent complexity of recovering the 3D rotation and translation pose of objects from 2D images. When attempting to improve performance, they often fall into a dilemma of balancing accuracy and efficiency, making it difficult to meet the needs of practical scenarios such as industrial sorting and intelligent service robots. To alleviate these problems, most current approaches employ a two-stage model: first performing a coarse pose estimation, then refining and optimizing the pose. This step-by-step processing reduces the difficulty of direct estimation, thereby optimizing the estimation accuracy and efficiency of 6D object pose estimation.

[0003] However, in this two-stage object pose estimation process, the coarse pose or matching template output from the first stage is often directly used as the input to the second stage. If there is a deviation in the initial estimation of the first stage, this deviation will directly propagate and affect the convergence direction and final result of the pose optimization in the second stage, and the accuracy of the pose estimation result cannot be guaranteed. Although the accuracy of the pose estimation result can be improved by increasing the number of templates or using complex feature extraction networks, this will also increase the computational complexity and affect the efficiency of pose estimation. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies that use a two-stage processing mode of coarse attitude estimation and attitude optimization to obtain object attitude, which suffers from high error propagation and computational complexity, resulting in low accuracy and efficiency of attitude estimation. This invention provides an object attitude estimation method and system based on three-dimensional feature mapping and view aggregation. By using three-dimensional feature mapping and view aggregation, multiple coarse attitude candidates are generated simultaneously in the coarse attitude estimation stage, reducing the risk of error propagation. Furthermore, feature matching replaces global search, and network iteration is optimized based on reliable initial values, reducing computational complexity and accelerating convergence, thereby improving the accuracy and efficiency of attitude estimation.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] Object pose estimation methods based on 3D feature mapping and view aggregation include:

[0007] The template image and target image of the target to be detected are obtained, and features are extracted using a pre-trained feature extraction model to obtain the features of the template image and the target image.

[0008] The template image features are mapped in three-dimensional space to determine the template three-dimensional features and viewpoint information of each template image feature;

[0009] The template image features and target image features are matched, and the viewpoint information of each target image feature and the corresponding template 3D features are determined by combining the matching results.

[0010] Viewpoint aggregation is performed based on the viewpoint information of each target image feature, and the matching relationship between the target image features and the template 3D features is obtained based on the viewpoint aggregation result.

[0011] By selecting the corresponding pose estimation algorithm and combining the matching relationship between the target image features and the template 3D features, a candidate coarse pose estimation result containing several coarse poses is obtained.

[0012] The candidate coarse pose estimation results are used as the initial values ​​for the pose optimization network to perform iterative optimization and obtain the object pose estimation results.

[0013] The reliability of the input data in the coarse pose estimation stage is ensured by using 3D feature mapping and view aggregation. Multiple coarse pose candidates are generated during this stage to prevent the error of a single initial pose from being amplified, thus reducing the risk of error propagation. Furthermore, feature matching replaces the global search in the high-dimensional pose space, reducing the computational load in the coarse pose estimation stage. Each candidate coarse pose is used as an initial value for optimization, enabling rapid iteration and optimization of pose estimation, thereby reducing the computational complexity of the pose optimization network and improving its efficiency.

[0014] Furthermore, acquiring the template image and target image of the detection target includes:

[0015] Obtain the CAD model of the target to be detected, and use a rendering tool to obtain a template image containing the target to be detected;

[0016] The system acquires input image data containing the target to be detected, and uses a pre-trained target detection model to detect and segment the input image data to obtain the target image of the target.

[0017] Furthermore, the step of performing three-dimensional spatial mapping on the template image features to determine the template three-dimensional features and viewpoint information of each template image feature includes:

[0018] Extract the rendering parameters of the rendering tool, and perform three-dimensional spatial mapping on the template image features based on the rendering parameters;

[0019] Based on the three-dimensional spatial mapping results, obtain the spatial position of each template image feature in three-dimensional space, and obtain the template three-dimensional feature and the corresponding viewpoint information corresponding to each template image feature based on the corresponding spatial position.

[0020] Furthermore, the matching of template image features and target image features, and the determination of the viewpoint information of each target image feature and the corresponding template 3D features based on the matching results, includes:

[0021] Calculate the cosine similarity between each target image feature and each template image feature;

[0022] For each target image feature, the template image feature with the highest cosine similarity is selected to determine the corresponding template 3D feature;

[0023] Based on the perspective information of the corresponding template's 3D features, obtain the perspective information of each target image feature.

[0024] Furthermore, the step of performing viewpoint aggregation based on the viewpoint information of each target image feature, and obtaining the matching relationship between the target image features and the template 3D features based on the viewpoint aggregation result, includes:

[0025] The number of times each viewpoint appears is counted based on the viewpoint information of each target image feature, and the corresponding confidence level is calculated.

[0026] Viewpoint information is aggregated based on viewpoint confidence and spatial correlation, and several viewpoint clusters are obtained based on the aggregation results;

[0027] Treat all viewpoints within each viewpoint cluster as a super-viewpoint, and merge the target image features and template 3D features corresponding to all viewpoints within the super-viewpoint.

[0028] Within the local spatial range of each super-viewpoint, a matching relationship is established between the target image features and the template 3D features.

[0029] Furthermore, establishing the matching relationship between target image features and template 3D features within the local spatial range of each super-viewpoint includes:

[0030] Within the local spatial range of each super-viewpoint, the cosine similarity between the corresponding target image features and the template 3D features is recalculated;

[0031] For each target image feature within each super-viewpoint, the template 3D feature with the highest cosine similarity is selected, and a matching relationship is established with it.

[0032] Furthermore, the selection of the corresponding pose estimation algorithm, combined with the matching relationship between the target image features and the template 3D features, yields candidate coarse pose estimation results containing several coarse poses, including:

[0033] Obtain the input image data containing the detected target corresponding to the target image, and identify the image type of the input image data;

[0034] When the input image data is an image type containing color information, the random sampling consistency algorithm with perspective N points is selected, and the candidate coarse pose estimation results of the detected target are obtained by solving the matching relationship between the target image features and the template 3D features.

[0035] When the input image data is a multimodal image type containing color information and depth information, the depth information of the input image data is obtained, and the target image features are mapped to three-dimensional space based on the depth information to obtain the target three-dimensional features of each target image feature.

[0036] Establish the matching relationship between the target's 3D features and the template's 3D features based on the matching relationship between the target image features and the template's 3D features;

[0037] An improved random sampling consensus algorithm incorporating Kabusch is selected to obtain candidate coarse pose estimation results for the detected target based on the matching relationship between the target's 3D features and the template's 3D features.

[0038] Furthermore, the iterative optimization using the candidate coarse pose estimation results as the initial values ​​for the pose optimization network to obtain the object pose estimation results includes:

[0039] A coarse pose is randomly selected from the candidate coarse pose estimation results and input into the pose optimization network as the initial value for iterative optimization. The target image and the template image are also input into the pose optimization network.

[0040] The pose optimization network renders a corresponding virtual image based on the input coarse pose, combined with the template image and its rendering parameters.

[0041] The virtual image and the target image are compared in terms of features. Based on the comparison results, the rotation correction and translation correction of the current pose are predicted, and the coarse pose is updated based on the prediction results.

[0042] Repeat the virtual image rendering and coarse pose update until the iteration termination condition is met, and obtain the optimized pose corresponding to the current coarse pose.

[0043] From the candidate coarse pose estimation results, a new coarse pose is selected, and the corresponding optimized pose is obtained through the pose optimization network until the optimized pose corresponding to each coarse pose in the candidate coarse pose estimation results is obtained.

[0044] The object pose estimation result is obtained by taking the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result.

[0045] Furthermore, the step of obtaining the object pose estimation result based on the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result includes:

[0046] For each coarse pose, the corresponding optimized pose is rendered to produce the corresponding virtual image;

[0047] Using the feature matching degree and geometric consistency between the virtual image and the target image as evaluation metrics, the pose confidence of each optimized pose is calculated.

[0048] The optimized pose with the highest confidence level is selected as the stage pose estimation result for the detected target.

[0049] An object pose estimation system based on 3D feature mapping and view aggregation, and an object pose estimation method for any of the above, including:

[0050] The image processing module is used to acquire template images and target images of the target to be detected, and to extract features through a pre-trained feature extraction model to obtain features of the template image and the target image.

[0051] The 3D feature mapping module, connected to the image processing module, is used to perform 3D spatial mapping on the template image features to determine the template 3D features and viewpoint information of each template image feature.

[0052] The view aggregation module is connected to the image processing module and the 3D feature mapping module respectively. It is used to match the template image features and the target image features, determine the view information of each target image feature and the corresponding template 3D features based on the matching results, perform view aggregation based on the view information of each target image feature, and obtain the matching relationship between the target image features and the template 3D features based on the view aggregation results.

[0053] The pose estimation module, connected to the view aggregation module, is used to select the corresponding pose estimation algorithm. It combines the matching relationship between the target image features and the template 3D features to solve for the candidate coarse pose estimation results containing several coarse poses. The candidate coarse pose estimation results are used as the initial values ​​of the pose optimization network for iterative optimization to obtain the object pose estimation result.

[0054] The beneficial effects of this invention are:

[0055] (1) The reliability of the input data in the coarse pose estimation stage is ensured by three-dimensional feature mapping and view aggregation, and multiple coarse pose candidates are generated in the coarse pose estimation stage to avoid the error of a single initial pose being amplified and reduce the risk of error propagation. In addition, feature matching is used to replace the global search in the high-dimensional pose space to reduce the computational amount in the coarse pose estimation stage, and each candidate coarse pose is used as the initial value for optimization to quickly perform the optimization iteration of pose estimation, thereby reducing the computational complexity of the pose optimization network and optimizing the pose estimation efficiency.

[0056] (2) Initial feature matching is achieved using cosine similarity. View aggregation filters out isolated incorrect viewpoints and retains effective viewpoint clusters with high consistency. Then, the bias is corrected by rematching in the super-viewpoint local space. This reduces interference from unmatched features and ensures the reliability of the pose estimation input data.

[0057] (3) Adapt pose estimation algorithms to different image types to ensure that reliable coarse poses can be generated for various scenarios, thereby improving the generalization of coarse estimation. Furthermore, optimize all candidate poses one by one to avoid local optima, and then filter high-confidence results through feature matching degree and geometric consistency, which not only improves optimization efficiency but also ensures the accuracy and stability of the final output pose. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of a process of the present invention;

[0059] Figure 2 This is a flowchart of an RGB image object pose estimation method according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of a structure according to an embodiment of the present invention.

[0061] The module consists of: 1. Image processing module; 2. 3D feature mapping module; 3. View aggregation module; and 4. Pose estimation module. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] Example:

[0064] 6D object pose estimation is a technique that estimates the pose of a target object based on RGB (color) or RGBD (color + depth) images and known camera parameters. Its core purpose is to quickly and accurately estimate the translation and rotation transformations of the target object relative to the camera coordinate system when it is being photographed. It can be applied to fields such as augmented reality, virtual reality, autonomous driving, and robotic arm grasping.

[0065] However, current mainstream 6D object pose estimation methods generally suffer from error propagation and computational complexity. To improve the accuracy and efficiency of 6D object pose estimation, this embodiment proposes an object pose estimation method based on 3D feature mapping and view aggregation, such as... Figure 1 As shown, it includes:

[0066] The template image and target image of the target to be detected are obtained, and features are extracted using a pre-trained feature extraction model to obtain the features of the template image and the target image.

[0067] The template image features are mapped in three-dimensional space to determine the template three-dimensional features and viewpoint information of each template image feature;

[0068] The template image features and target image features are matched, and the viewpoint information of each target image feature and the corresponding template 3D features are determined by combining the matching results.

[0069] Viewpoint aggregation is performed based on the viewpoint information of each target image feature, and the matching relationship between the target image features and the template 3D features is obtained based on the viewpoint aggregation result.

[0070] By selecting the corresponding pose estimation algorithm and combining the matching relationship between the target image features and the template 3D features, a candidate coarse pose estimation result containing several coarse poses is obtained.

[0071] The candidate coarse pose estimation results are used as the initial values ​​for the pose optimization network to perform iterative optimization and obtain the object pose estimation results.

[0072] The realization of 6D object pose estimation mainly relies on the comparison between a known reference and an unknown solution. Through feature correlation and geometric matching between the two, the derivation of three-dimensional pose parameters from two-dimensional image information is realized. Therefore, before carrying out pose estimation, the template image of the target is first obtained as a known reference to provide reliable three-dimensional parameter information. At the same time, the target image is obtained as an unknown solution to provide visual and geometric information of the object in the real scene. Subsequently, through comparative analysis of the two, accurate object pose estimation can be achieved.

[0073] The acquisition of the template image and target image of the detection target includes:

[0074] Obtain the CAD model of the target to be detected, and use a rendering tool to obtain a template image containing the target to be detected;

[0075] The system acquires input image data containing the target to be detected, and uses a pre-trained target detection model to detect and segment the input image data to obtain the target image of the target.

[0076] When acquiring the template image, the CAD model of the target is first extracted. The extracted CAD model contains the complete three-dimensional geometric structure and texture information of the target, such as size, outline, surface details, color and material properties.

[0077] Based on the CAD model, the Blenderproc rendering tool is used to render the CAD model. During the rendering process, camera intrinsic parameters that match the actual scene being captured need to be preset, such as focal length, pixel size, and principal point coordinates. Then, multiple sets of virtual scenes with different perspectives are generated according to preset viewing intervals. Specifically, the perspective of the generated virtual scene can be adjusted by adjusting the position and angle of the camera relative to the CAD model.

[0078] After rendering, the generated images need further preprocessing. First, image segmentation techniques are used to remove the virtual background from the rendered virtual scene, retaining only the foreground region of the detected target to ensure that the template image contains only the visual information of the detected target itself. Furthermore, each template image is labeled with corresponding rendering viewpoint parameters, such as camera rotation angle and offset, and with corresponding 3D coordinate mapping relationships, so that the template image not only possesses visual information but also is associated with corresponding 3D geometric references.

[0079] The final output template image can be an RGB image containing the detected target or an RGBD image. The rendering parameters of the rendering tool can be adjusted according to the needs to add depth rendering functionality so that the output template image contains the corresponding depth information.

[0080] The target images are mainly sourced from input image data in real-world scenarios, which consists of video frames or single images captured in actual application scenarios.

[0081] After acquiring the corresponding input image data, a pre-trained object detection model is used to detect and segment the input image data. In this embodiment, the YOLO11 model is specifically used as the object detection model for detection and segmentation. The object detection model first locates the bounding box of the target in the input image to determine the spatial position of the target in the image. Then, through instance segmentation, it separates the pixel region of the target from the original input image and removes background pixel interference. After segmentation, the segmented target region is cropped to a fixed size to align with the size of the template image. Furthermore, if the input image is an RGBD image, the depth information of the target region and the distance between each pixel on the target surface and the camera must also be preserved simultaneously.

[0082] After obtaining the template image and target image of the target to be detected, feature extraction is performed using a pre-trained feature extraction model to provide a data foundation for subsequent pose estimation.

[0083] In this embodiment, the pre-trained feature extraction model is specifically the ViT feature extraction model. This model is based on the ViT-L / 14 (Vision Transformer Large with 14x14 patches, image encoder) network, initialized with the pre-trained weights of DINOv2 (pre-trained image encoder), takes matching points that have only undergone in-plane rotation as positive samples, and other unmatched point pairs as negative samples, and uses Info-NCE loss (information-noise contrastive estimation loss) for contrastive learning training to finally obtain the required feature extraction model.

[0084] By inputting the template image and the target image into the feature extraction model, the template image features and target image features required for object pose estimation can be obtained.

[0085] Given that template image features are extracted from 2D rendered images and contain only pixel-level visual information, they cannot be directly associated with the object's true position in the 3D coordinate system. Therefore, relying solely on visual similarity when matching with target image features can easily lead to feature confusion. Furthermore, the lack of viewpoint identification means that 2D features cannot reflect their respective rendering viewpoints, which can easily result in incorrect viewpoint associations during subsequent matching, thus reducing the reliability of subsequent feature matching.

[0086] Therefore, before feature matching, the template image features are mapped in three-dimensional space to determine the template three-dimensional features and viewpoint information of each template image feature, including:

[0087] Extract the rendering parameters of the rendering tool, and perform three-dimensional spatial mapping on the template image features based on the rendering parameters;

[0088] Based on the three-dimensional spatial mapping results, obtain the spatial position of each template image feature in three-dimensional space, and obtain the template three-dimensional feature and the corresponding viewpoint information corresponding to each template image feature based on the corresponding spatial position.

[0089] When generating a template image, the rendering tool records all parameters during the rendering process. Based on these parameters, camera intrinsic and extrinsic parameters are extracted. Intrinsic parameters include camera focal length, stationary point coordinates, and image resolution, describing the optical characteristics of the camera projecting points from 3D space onto a 2D plane. Extrinsic parameters, on the other hand, are the camera's pose parameters relative to the CAD model during rendering, including rotation matrices and translation vectors. The rotation matrix describes the rotational transformation of the camera coordinate system relative to the object coordinate system, while the translation vector describes the position of the camera coordinate system's origin in the object coordinate system.

[0090] Then, based on the position of the template image features in the image, the corresponding pixel coordinates are determined, and then converted into normalized coordinates in the camera coordinate system. Combining this with the depth information of the template image, the 3D coordinates in the camera coordinate system are obtained; this depth information is generated during the rendering process. Finally, using camera extrinsic parameters, the 3D coordinates in the camera coordinate system are converted into 3D coordinates in the object coordinate system.

[0091] After the 3D spatial mapping is completed, the template 3D features corresponding to the template image features are obtained based on the calculated 3D coordinates. At the same time, the corresponding viewpoint information is obtained based on the camera extrinsic parameters corresponding to the template image, forming a correspondence with the corresponding template image features.

[0092] 6D object pose estimation largely relies on the correspondence between 3D points in the object coordinate system and 2D points in the image coordinate system to solve for the pose. Therefore, after binding the template image features and the corresponding template 3D features, the template image features and target image features are matched, and then the viewpoint information of each target image feature and the corresponding template 3D feature are determined by combining the matching results, including:

[0093] Calculate the cosine similarity between each target image feature and each template image feature;

[0094] For each target image feature, the template image feature with the highest cosine similarity is selected to determine the corresponding template 3D feature;

[0095] Based on the perspective information of the corresponding template's 3D features, obtain the perspective information of each target image feature.

[0096] Specifically, cosine similarity is used to measure the similarity between the features of the target image and the features of the template image. The formula for calculating cosine similarity is as follows:

[0097] ;

[0098] in, For the first The target image features and the first Cosine similarity between features of template images For the first Each target image feature For the first Template image features.

[0099] By traversing all target image features and all template image features, a three-dimensional correlation matrix is ​​generated, consisting of target features, template features, and cosine similarity. The higher the cosine similarity, the closer the two features are visually and semantically.

[0100] Then, for each target image feature, the cosine similarity value between it and all template image features is extracted from the three-dimensional correlation matrix, and the template image feature corresponding to the maximum value is selected as the optimal match.

[0101] Once the optimal matching template image features are determined, the template 3D features obtained by mapping these features in 3D space can be directly associated with them and bound to the target image features. Furthermore, since each template image feature is bound with unique viewpoint information during the 3D space mapping stage, the viewpoint information of the optimally matched template image features can also be used to provide bound viewpoint information for the target image features.

[0102] Although the optimal matching template image features for each target image feature are selected through cosine similarity, this matching based on single feature similarity may result in incorrect matching due to similar textures on the object surface, or due to image noise, local occlusion, or differences in lighting. These incorrect matching will cause deviations in the viewpoint information of the target image features, affecting the accuracy of subsequent pose estimation results.

[0103] Since the same object has only one pose in a real scene, the viewpoint information of all its features should theoretically be highly consistent. Therefore, we further verify this consistency through viewpoint aggregation, thereby verifying the reliability of the obtained matching results, so as to finally obtain a reliable matching pair with consistent viewpoints.

[0104] The step of performing viewpoint aggregation based on the viewpoint information of each target image feature, and obtaining the matching relationship between the target image features and the template 3D features based on the viewpoint aggregation result, includes:

[0105] The number of times each viewpoint appears is counted based on the viewpoint information of each target image feature, and the corresponding confidence level is calculated.

[0106] Viewpoint information is aggregated based on viewpoint confidence and spatial correlation, and several viewpoint clusters are obtained based on the aggregation results;

[0107] Treat all viewpoints within each viewpoint cluster as a super-viewpoint, and merge the target image features and template 3D features corresponding to all viewpoints within the super-viewpoint.

[0108] Within the local spatial range of each super-viewpoint, a matching relationship is established between the target image features and the template 3D features.

[0109] Each target image feature, after matching, is associated with unique viewpoint information. Therefore, the viewpoint information of all target image features can be traversed, and identical or highly similar viewpoints can be categorized and counted to obtain the frequency of occurrence of each viewpoint. The higher the frequency, the greater the consensus of that viewpoint in the target image, and the stronger its potential reliability. Furthermore, viewpoint reliability is quantified using viewpoint confidence metrics. The formula for calculating the viewpoint confidence level is:

[0110] ;

[0111] in, From the perspective confidence level For the first The viewpoint corresponding to each target image feature The number of features in the target image. This is an indicator function; it is 1 when the condition is true and 0 otherwise.

[0112] Then, spatial distance between viewpoints is defined, such as the angle between rotation matrices or the Euclidean distance between translation vectors, to measure the spatial correlation between two viewpoints. The smaller the distance, the closer the camera poses of the two viewpoints are, and the higher the probability that they belong to the same true pose. Subsequently, density clustering or hierarchical clustering algorithms are used, with viewpoint confidence as the weight and spatial distance as the clustering criterion, to cluster viewpoints with spatial distance less than a preset threshold and viewpoint confidence higher than the corresponding threshold.

[0113] Furthermore, after obtaining all view clusters, it is necessary to remove clusters with excessively low total confidence or excessively large spatial dispersion to ensure the reliability of subsequent aggregation results.

[0114] After the filtering process, all the obtained view clusters, each representing a group of highly similar viewpoints, are considered as a hyperviewpoint. When determining the hyperviewpoint, the mean of the rotation matrix or translation vector of all viewpoints within the view cluster is first used as the center viewpoint, and the maximum dispersion of the viewpoints within the cluster is used as the range of the hyperviewpoint, thus representing the camera pose range covered by the hyperviewpoint. Then, all target image features within the view cluster and their corresponding template 3D features are merged.

[0115] Then, local spatial constraints are set using the center viewpoint and range of the super-viewpoint to limit the matching of target image features and template 3D features to meet the consistency of 3D spatial position. That is, the position of the template 3D features corresponding to the target image features projected onto the image plane under the camera pose of the super-viewpoint must be close to the actual pixel coordinates of the target features.

[0116] Based on this, establishing a matching relationship between target image features and template 3D features within the local spatial range of each super-viewpoint includes:

[0117] Within the local spatial range of each super-viewpoint, the cosine similarity between the corresponding target image features and the template 3D features is recalculated;

[0118] For each target image feature within each super-viewpoint, the template 3D feature with the highest cosine similarity is selected, and a matching relationship is established with it.

[0119] During global matching, target image features may produce false similarities with template image features across super-viewpoints. These template image features do not conform to geometric consistency in the local space of the current super-viewpoint. Therefore, such erroneous matching relationships can be eliminated by recalculating the cosine similarity within the local space.

[0120] Furthermore, the super-viewpoint has already aggregated and filtered out feature clusters with consistent viewpoints, and the similarity calculation is limited to their local space, which can further narrow the matching range, reduce irrelevant computation, and improve computational efficiency.

[0121] Based on the recalculated cosine similarity, the optimal matching template 3D features are reselected for each target image feature within the super-viewpoint, and the final matching relationship is established to ensure the reliability of the constructed matching pairs between target image features and template 3D features.

[0122] After processing the relevant feature matching pairs, a suitable pose estimation algorithm can be selected for preliminary coarse pose estimation. Considering that the final accuracy of subsequent pose estimation optimization highly depends on the initial values, if the initially obtained coarse pose deviates too much from the true pose, this error will be propagated into the optimization process, potentially causing the optimization process to get stuck in a local optimum. Therefore, during coarse state estimation, multiple alternative coarse poses are solved, thus providing multiple differentiated initial values ​​for the subsequent optimization network.

[0123] Specifically, the selection of the corresponding pose estimation algorithm, combined with the matching relationship between the target image features and the template 3D features, yields candidate coarse pose estimation results containing several coarse poses, including:

[0124] Obtain the input image data containing the detected target corresponding to the target image, and identify the image type of the input image data;

[0125] When the input image data is an image type containing color information, the random sampling consistency algorithm with perspective N points is selected, and the candidate coarse pose estimation results of the detected target are obtained by solving the matching relationship between the target image features and the template 3D features.

[0126] When the input image data is a multimodal image type containing color information and depth information, the depth information of the input image data is obtained, and the target image features are mapped to three-dimensional space based on the depth information to obtain the target three-dimensional features of each target image feature.

[0127] Establish the matching relationship between the target's 3D features and the template's 3D features based on the matching relationship between the target image features and the template's 3D features;

[0128] An improved random sampling consensus algorithm incorporating Kabusch is selected to obtain candidate coarse pose estimation results for the detected target based on the matching relationship between the target's 3D features and the template's 3D features.

[0129] The input image data is the raw data containing the target to be detected. The corresponding image type can be determined by the data format. Specifically, the input image data includes image types containing color information, namely RGB images, and multimodal image types containing color information and depth information, namely RGBD images.

[0130] For different image types, select appropriate pose estimation algorithms to ensure the accuracy of the obtained coarse pose.

[0131] For RGB images, the PnP-RANSAC (Perspective N-Point Random Sample Consensus) algorithm is specifically chosen. In this fusion algorithm, the perspective N-point algorithm can solve the pose parameters that satisfy the projection constraints through the least squares method, which is well-suited to the two-dimensional and three-dimensional mapping requirements of RGB images. The random sample consensus algorithm can solve the problem by randomly sampling interior points, i.e., effective matching pairs, and can effectively eliminate residual exterior points in the matching relationship, i.e., erroneous matches, thereby reducing and avoiding the interference of a few noises on the pose solution and improving the robustness of the results.

[0132] For RGBD images, their depth information allows the target image features to be mapped from two-dimensional pixels to three-dimensional space, thereby forming a corresponding matching relationship with the template's three-dimensional features.

[0133] For each target image feature, its three-dimensional coordinates in three-dimensional space can be obtained based on the corresponding depth information:

[0134] ;

[0135] in, Let be the coordinates of the pixel corresponding to one of the target image features in the depth map. For pixels depth, , , and All of these are camera internal parameters.

[0136] Based on the existing matching relationship between the target image features and the template 3D features, the target image features are replaced with their corresponding target 3D features, thus obtaining the matching relationship between the corresponding target 3D features and the template 3D features.

[0137] The solution for matching pairs of 3D features relies more heavily on 3D geometric constraints. Therefore, this embodiment specifically uses the improved Kabsch-MAGSAC++ algorithm (a fusion of Kabsch and MAGSAC++) for coarse pose determination. In this fusion algorithm, the Kabsch algorithm can solve for the optimal rotation matrix and translation vector between two sets of 3D point sets, namely the target 3D features and the template 3D features, through singular value decomposition, making it more suitable for 3D features and more accurate. The MAGSAC++ algorithm, on the other hand, can more efficiently identify interior points through a probabilistic model, reducing the number of iterations, while also having a higher tolerance for depth noise, making it more suitable for RGBD image data processing.

[0138] After obtaining several coarse poses through the adaptive pose estimation algorithm, they are combined into a candidate coarse pose estimation result, which is used as the input for pose optimization, and the second stage of pose optimization is started in parallel.

[0139] The step of iteratively optimizing the pose optimization network using the candidate coarse pose estimation results as initial values ​​to obtain the object pose estimation results includes:

[0140] A coarse pose is randomly selected from the candidate coarse pose estimation results and input into the pose optimization network as the initial value for iterative optimization. The target image and the template image are also input into the pose optimization network.

[0141] The pose optimization network renders a corresponding virtual image based on the input coarse pose, combined with the template image and its rendering parameters.

[0142] The virtual image and the target image are compared in terms of features. Based on the comparison results, the rotation correction and translation correction of the current pose are predicted, and the coarse pose is updated based on the prediction results.

[0143] Repeat the virtual image rendering and coarse pose update until the iteration termination condition is met, and obtain the optimized pose corresponding to the current coarse pose.

[0144] From the candidate coarse pose estimation results, a new coarse pose is selected, and the corresponding optimized pose is obtained through the pose optimization network until the optimized pose corresponding to each coarse pose in the candidate coarse pose estimation results is obtained.

[0145] The object pose estimation result is obtained by taking the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result.

[0146] The pose optimization network can employ existing pose optimization algorithms. In this embodiment, the FoundationPose algorithm is specifically used. The specific iteration termination condition can be reaching a certain number of iterations or the convergence of the optimization results.

[0147] During the pose optimization process, multiple differentiated starting points are provided for the pose optimization network by iterating through the alternative coarse pose estimation results to select initial values, thereby covering a wider pose space and reducing the risk of getting trapped in local optima.

[0148] After obtaining the optimization results for all coarse poses, it is considered that each optimized pose is a theoretically optimal solution obtained through iterative correction. However, there may be cases where the numerical convergence is not visually consistent. For example, the optimization process may get stuck in a local optimum, and the pose parameters may converge numerically, but there may be significant misalignment between the visual features of the virtual image and the target image, such as non-overlapping edges or texture shifts. Therefore, it is necessary to further comprehensively evaluate the visual consistency and geometric correctness of each optimization result to determine the final output that is closest to the real pose from the candidate results, and to avoid erroneous results caused by random errors in a single optimization process.

[0149] The step of obtaining the object pose estimation result based on the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result includes:

[0150] For each coarse pose, the corresponding optimized pose is rendered to produce the corresponding virtual image;

[0151] Using the feature matching degree and geometric consistency between the virtual image and the target image as evaluation metrics, the pose confidence of each optimized pose is calculated.

[0152] The optimized pose with the highest confidence level is selected as the stage pose estimation result for the detected target.

[0153] By rendering a virtual image corresponding to the optimized pose, pose parameters can be transformed into specific visual representations. The visual consistency between the virtual and target images is then reflected by calculating their feature matching degree. A higher matching degree indicates a closer match between the object's appearance under the optimized pose and the real-world scene, and a stronger visual plausibility of the pose. This feature matching degree can be evaluated using cosine similarity.

[0154] Considering that visual consistency only reflects the similarity of two-dimensional images and cannot guarantee the accuracy of pose in three-dimensional space, geometric consistency is further introduced as an evaluation index to directly quantify the correctness of the pose in three-dimensional space. The higher the geometric consistency, the better the three-dimensional position and orientation of the object under the optimized pose matches the spatial constraints of the real scene. The geometric consistency can be evaluated through parameters such as the projection coordinates of the object's key points in the virtual image and the pixel distance between the key points in the target image, or the Euclidean distance between the template's three-dimensional features after being transformed into the camera coordinate system through optimized pose and the target's three-dimensional features.

[0155] Furthermore, the feature matching degree and geometric consistency are weighted and summed to obtain the corresponding pose confidence. The optimized pose with the highest pose confidence is selected as the final result, ensuring that the final output pose estimation result is consistent with the target at the image level and meets the geometric constraints in three-dimensional space, belonging to the most reliable solution that is closest to the true value among all candidates.

[0156] Taking an RGB image as the input data as an example, the entire process of object pose estimation is as follows: Figure 2 As shown.

[0157] Another aspect of this embodiment also provides an object pose estimation system based on 3D feature mapping and view aggregation, such as... Figure 3 As shown, it includes:

[0158] Image processing module 1 is used to acquire template images and target images of the target to be detected, and to extract features through a pre-trained feature extraction model to obtain features of the template image and the target image.

[0159] The 3D feature mapping module 2 is connected to the image processing module and is used to perform 3D spatial mapping on the template image features to determine the template 3D features and viewpoint information of each template image feature.

[0160] The view aggregation module 3 is connected to the image processing module and the three-dimensional feature mapping module respectively. It is used to match the template image features and the target image features, determine the view information of each target image feature and the corresponding template three-dimensional features by combining the matching results, perform view aggregation based on the view information of each target image feature, and obtain the matching relationship between the target image features and the template three-dimensional features based on the view aggregation results.

[0161] The pose estimation module 4, connected to the view aggregation module, is used to select the corresponding pose estimation algorithm, solve the matching relationship between the target image features and the template 3D features to obtain candidate coarse pose estimation results containing several coarse poses, and use the candidate coarse pose estimation results as the initial values ​​of the pose optimization network to perform iterative optimization to obtain the object pose estimation result.

[0162] In this embodiment, the image processing module, 3D feature mapping module, view aggregation module, and pose estimation module are all data processing components with built-in corresponding processing algorithms, such as microprocessors and computers.

[0163] The image processing module is equipped with corresponding algorithms for image processing and feature extraction, the 3D feature mapping module is equipped with related algorithms for 3D spatial mapping, the view aggregation module is equipped with related algorithms for feature matching and view aggregation, and the pose estimation module is equipped with related algorithms for pose estimation and pose optimization.

[0164] Through the image processing module, 3D feature mapping module, view aggregation module, and pose estimation module, fast and accurate 6D object pose estimation can be achieved for targets detected within RGB or RGBD images.

[0165] This embodiment further verifies the advantages of the object pose estimation method proposed in this embodiment based on the Linemod-Occluded dataset.

[0166] Based on the Linemod-Occluded dataset, different coarse pose estimation algorithms were used for corresponding coarse pose estimation. These algorithms included MegaPose, GigaPose, and FoundPose, as well as PnP-RANSAC and Kabsch-MAGSAC++ used in this embodiment. The performance comparison results of different coarse pose estimation algorithms are shown in Table 1.

[0167] Table 1. Performance comparison results of different coarse attitude estimation algorithms

[0168]

[0169] As shown in Table 1, the PnP-RANSAC and Kabsch-MAGSAC++ algorithms used in this embodiment both exhibit significant performance and accuracy advantages in the coarse attitude estimation stage. The main difference between the two lies in the fact that the latter can utilize depth information during estimation, resulting in more accurate translation estimation compared to the former.

[0170] Based on the Linemod-Occluded dataset, the performance of pose estimation by combining different coarse pose estimation algorithms and pose optimization algorithms is compared. The comparison results are shown in Table 2.

[0171] Table 2 Comparison of attitude estimation performance of different combinations of coarse attitude estimation algorithms and attitude optimization algorithms

[0172]

[0173] As shown in Table 2, the PnP-RANSAC algorithm presented in this embodiment, when combined with the existing FoundationPose optimization algorithm, suffers only a small performance loss but significantly reduces the time required, effectively verifying the performance and speed advantages of the estimation method described in this embodiment.

[0174] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Other variations and modifications are possible without departing from the technical solutions described in the claims.

Claims

1. An object pose estimation method based on 3D feature mapping and view aggregation, characterized in that, include: The template image and target image of the target to be detected are obtained, and features are extracted using a pre-trained feature extraction model to obtain the features of the template image and the target image. The template image features are mapped in three-dimensional space to determine the template three-dimensional features and viewpoint information of each template image feature; The template image features and target image features are matched, and the viewpoint information of each target image feature and the corresponding template 3D features are determined by combining the matching results. Viewpoint aggregation is performed based on the viewpoint information of each target image feature, and the matching relationship between the target image features and the template 3D features is obtained based on the viewpoint aggregation result. By selecting the corresponding pose estimation algorithm and combining the matching relationship between the target image features and the template 3D features, a candidate coarse pose estimation result containing several coarse poses is obtained. The alternative coarse pose estimation results are used as the initial values ​​for the pose optimization network to perform iterative optimization and obtain the object pose estimation results. The selected pose estimation algorithm, by combining the matching relationship between target image features and template 3D features, obtains candidate coarse pose estimation results containing several coarse poses, including: Obtain the input image data containing the detected target corresponding to the target image, and identify the image type of the input image data; When the input image data is an image type containing color information, the random sampling consistency algorithm with perspective N points is selected, and the candidate coarse pose estimation results of the detected target are obtained by solving the matching relationship between the target image features and the template 3D features. When the input image data is a multimodal image type containing color information and depth information, the depth information of the input image data is obtained, and the target image features are mapped to three-dimensional space based on the depth information to obtain the target three-dimensional features of each target image feature. Establish the matching relationship between the target's 3D features and the template's 3D features based on the matching relationship between the target image features and the template's 3D features; An improved random sampling consensus algorithm incorporating Kabusch is selected to obtain candidate coarse pose estimation results for the detected target based on the matching relationship between the target's 3D features and the template's 3D features.

2. The object pose estimation method based on 3D feature mapping and view aggregation according to claim 1, characterized in that, The process of obtaining the template image and target image of the target to be detected includes: Obtain the CAD model of the target to be detected, and use a rendering tool to obtain a template image containing the target to be detected; The system acquires input image data containing the target to be detected, and uses a pre-trained target detection model to detect and segment the input image data to obtain the target image of the target.

3. The object pose estimation method based on 3D feature mapping and view aggregation according to claim 2, characterized in that, The step of performing three-dimensional spatial mapping on template image features to determine the template three-dimensional features and viewpoint information of each template image feature includes: Extract the rendering parameters of the rendering tool, and perform three-dimensional spatial mapping on the template image features based on the rendering parameters; Based on the three-dimensional spatial mapping results, obtain the spatial position of each template image feature in three-dimensional space, and obtain the template three-dimensional feature and the corresponding viewpoint information corresponding to each template image feature based on the corresponding spatial position.

4. The object pose estimation method based on 3D feature mapping and view aggregation according to claim 1, characterized in that, The process of matching template image features and target image features, and determining the viewpoint information and corresponding template 3D features for each target image feature based on the matching results, includes: Calculate the cosine similarity between each target image feature and each template image feature; For each target image feature, the template image feature with the highest cosine similarity is selected to determine the corresponding template 3D feature; Based on the perspective information of the corresponding template's 3D features, obtain the perspective information of each target image feature.

5. The object pose estimation method based on 3D feature mapping and view aggregation according to claim 1, characterized in that, The step of performing viewpoint aggregation based on the viewpoint information of each target image feature, and obtaining the matching relationship between the target image features and the template 3D features based on the viewpoint aggregation result, includes: The number of times each viewpoint appears is counted based on the viewpoint information of each target image feature, and the corresponding confidence level is calculated. Viewpoint information is aggregated based on viewpoint confidence and spatial correlation, and several viewpoint clusters are obtained based on the aggregation results; Treat all viewpoints within each viewpoint cluster as a super-viewpoint, and merge the target image features and template 3D features corresponding to all viewpoints within the super-viewpoint. Within the local spatial range of each super-viewpoint, a matching relationship is established between the target image features and the template 3D features.

6. The object pose estimation method based on three-dimensional feature mapping and view aggregation according to claim 5, characterized in that, The process of establishing a matching relationship between target image features and template 3D features within the local spatial range of each super-viewpoint includes: Within the local spatial range of each super-viewpoint, the cosine similarity between the corresponding target image features and the template 3D features is recalculated; For each target image feature within each super-viewpoint, the template 3D feature with the highest cosine similarity is selected, and a matching relationship is established with it.

7. The object pose estimation method based on 3D feature mapping and view aggregation according to claim 1, characterized in that, The step of iteratively optimizing the pose optimization network using the candidate coarse pose estimation results as initial values ​​to obtain the object pose estimation results includes: A coarse pose is randomly selected from the candidate coarse pose estimation results and input into the pose optimization network as the initial value for iterative optimization. The target image and the template image are also input into the pose optimization network. The pose optimization network renders a corresponding virtual image based on the input coarse pose, combined with the template image and its rendering parameters. The virtual image and the target image are compared in terms of features. Based on the comparison results, the rotation correction and translation correction of the current pose are predicted, and the coarse pose is updated based on the prediction results. Repeat the virtual image rendering and coarse pose update until the iteration termination condition is met, and obtain the optimized pose corresponding to the current coarse pose. From the candidate coarse pose estimation results, a new coarse pose is selected, and the corresponding optimized pose is obtained through the pose optimization network until the optimized pose corresponding to each coarse pose in the candidate coarse pose estimation results is obtained. The object pose estimation result is obtained by taking the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result.

8. The object pose estimation method based on three-dimensional feature mapping and view aggregation according to claim 7, characterized in that, The step of obtaining the object pose estimation result based on the optimized pose corresponding to each coarse pose in each candidate coarse pose estimation result includes: For each coarse pose, the corresponding optimized pose is rendered to produce the corresponding virtual image; Using the feature matching degree and geometric consistency between the virtual image and the target image as evaluation metrics, the pose confidence of each optimized pose is calculated. The optimized pose with the highest confidence level is selected as the stage pose estimation result for the detected target.

9. An object pose estimation system based on three-dimensional feature mapping and view aggregation, used to execute the object pose estimation method according to any one of claims 1 to 8, characterized in that, include: The image processing module is used to acquire template images and target images of the target to be detected, and to extract features through a pre-trained feature extraction model to obtain features of the template image and the target image. The 3D feature mapping module, connected to the image processing module, is used to perform 3D spatial mapping on the template image features to determine the template 3D features and viewpoint information of each template image feature. The view aggregation module is connected to the image processing module and the 3D feature mapping module respectively. It is used to match the template image features and the target image features, determine the view information of each target image feature and the corresponding template 3D features based on the matching results, perform view aggregation based on the view information of each target image feature, and obtain the matching relationship between the target image features and the template 3D features based on the view aggregation results. The pose estimation module, connected to the view aggregation module, is used to select the corresponding pose estimation algorithm. It combines the matching relationship between the target image features and the template 3D features to solve for the candidate coarse pose estimation results containing several coarse poses. The candidate coarse pose estimation results are used as the initial values ​​of the pose optimization network for iterative optimization to obtain the object pose estimation result.

Images