A metrology warehouse multi-form target pose recognition method and system

Abstract

The application discloses a kind of metrology warehouse multi-form target pose recognition method and system, it is related to target pose recognition technical field.The metrology warehouse multi-form target pose recognition method, by using coarse positioning, time sequence visible region segmentation, depth boundary correction, category shape priori completion and occlusion rate calculation, and according to occlusion rate triggers active supplement view angle acquisition, again combining two-dimensional and three-dimensional semantic key point's double-domain consistency matching, robust geometric estimation and local point cloud registration obtains initial pose, finally utilize multi-frame re-projection consistency constraint and local geometry joint optimization output final pose, the application can still be stable in the case where target object is seriously occluded, stack placement and local structure missing, obtain high-precision six degrees of freedom pose, improve pose recognition success rate under complex occlusion scene.

Description

Technical Field

[0001] This invention relates to the field of target pose recognition technology, specifically to a method and system for recognizing the poses of multi-form targets in a metering warehouse. Background Technology

[0002] With the deepening of intelligent transformation of power supply metering warehouses, more and more warehouses are beginning to use intelligent robots to perform tasks such as material entry and exit, inventory, and grasping. Target objects in metering warehouses are usually characterized by a wide variety of types, large differences in shape, random placement orientation, and obvious stacking and occlusion. At the same time, there are also practical problems such as strong background interference, changes in lighting, and irregular placement. For such scenarios, robots not only need to identify the type of item to which the target belongs, but also need to accurately obtain the six-degree-of-freedom pose information of the target object in the workspace to support subsequent grasping, positioning, and operation execution. At present, most of the research on target recognition in warehouses still mainly relies on traditional visual recognition or small model detection methods. These methods usually rely on the appearance, outline, or local features of the object for matching, and are prone to false detection, missed detection, and inaccurate positioning under conditions of occlusion, overlap, changes in lighting, and complex background interference.

[0003] In existing technologies, some studies have begun to attempt to combine RGB-D point clouds, video streams, multimodal representation alignment, and end-to-end visual language action frameworks to improve the adaptability to complex scenes. However, for target objects with significant differences in shape, such as transformers, meters, switches, communication modules, and meter boxes, the above methods can often only complete category recognition. They are difficult to stably output accurate pose information when the target is partially occluded or stacked or overlapped. Traditional pose estimation methods generally rely on complete observation information in a single frame image. When the target object is partially occluded or depth information is missing, the visible area is insufficient to support accurate rigid body transformation solutions, which can easily lead to pose calculation failures or excessive errors. Moreover, most existing methods adopt passive observation strategies and lack the ability to actively adjust the viewing angle according to the degree of occlusion to obtain better observation data. Therefore, it is difficult to meet the actual needs of robots to perceive position and posture with high precision at the same time in grasping operations. Therefore, how to achieve high-precision pose recognition of multi-shaped target objects in metering warehouses under complex occlusion conditions remains an urgent technical problem to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for recognizing the pose of multi-morphological targets in a metering warehouse, thereby solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides a method for recognizing the pose of multi-morphological targets in a metering warehouse, comprising the following steps: Step S1: Obtain the RGB image and depth map of the current scene, perform coarse localization of the target object, segment the visible area of ​​the target object in the time-series image sequence, correct the visible area by combining depth boundary information, and then use category shape prior to complete the complete outline of the target object, while calculating the occlusion rate of the target object. Step S2: Determine whether active supplementary view acquisition is needed based on the occlusion rate. If so, control the camera to perform local micro-scanning around the target candidate area to acquire multiple frames of RGB-D images, and perform spatial registration and quality screening on the multiple frames of RGB-D images to form a set of multiple effective observations. Step S3: Based on the complete contour and the set of multiple effective observations, extract the two-dimensional semantic key points and three-dimensional semantic key points of the target object, perform dual-domain consistency matching with the category shape prior template, and use robust geometric estimation and local point cloud registration to obtain the initial six-degree-of-freedom pose of the target object. Step S4: Using the multi-frame effective observation set, perform multi-frame reprojection consistency constraint and local geometry joint optimization on the initial six-degree-of-freedom pose, and output the final six-degree-of-freedom pose and confidence of the target object.

[0006] Preferably, step S1 further includes: The RGB image and the target category text are input into the YOLO-World model to obtain a set of candidate boxes, and the candidate region with the highest score is selected as the region of interest. Centered on the region of interest, from the current frame and its preceding Lm A short temporal ROI image sequence is cropped from a single frame image and input into the SAM2 model to obtain the initial visible mask and segmentation confidence map; The depth map is projected onto the region of interest, the depth gradient map and the surface normal abrupt change map are calculated, and the initial visible mask is corrected based on the consistency of the depth boundary to obtain the corrected visible mask. A prior library of the category and shape of the target object in the measurement warehouse is pre-built. Based on the contour features and depth structure features of the corrected visible mask, candidate prior templates are retrieved from the prior library of the corresponding category. Input the current ROI visual features, SAM2 temporal memory features, corrected visible mask and candidate prior templates into the AmodalCompletion branch to complete the target object; The occlusion rate is calculated based on the area ratio of the modified visible mask to the full mask.

[0007] Preferably, in step S2, when the occlusion rate is greater than a preset occlusion trigger threshold, the active supplementary view acquisition process is triggered. The local micro-scanning specifically involves: extracting the main occlusion boundaries and missing contours of the target based on the difference region between the complete mask and the corrected visible mask; using the outward normal direction of the missing region as the priority compensation view direction; and, with the current observation pose as the center, performing angle-range operations in the horizontal and pitch directions respectively. and Small-range oscillations, planned A supplementary viewpoint pose is obtained, and RGB images and depth maps are acquired at each pose point to form a supplementary viewpoint observation sequence.

[0008] Preferably, step S2, which involves spatial registration and quality screening of multiple RGB-D images, further includes: Using the corrected visible mask as the initial cue, the SAM2 model is called in each supplementary view frame to perform temporal mask propagation, thereby obtaining the visible mask corresponding to each supplementary view frame; ORB-SLAM3 is used to estimate the camera pose between adjacent supplementary view frames, obtain the pose transformation matrix of each frame relative to the initial reference frame, and perform fine registration using point-to-plane ICP. Based on the registered multi-frame poses and target masks of each frame, the depth point cloud belonging to the target region in each frame is back-projected onto a unified coordinate system and fused to form an enhanced target local point cloud. Calculate the visible area gain ratio of each supplementary viewpoint relative to the initial viewpoint, and calculate the information quality score by combining segmentation confidence, depth efficiency, and image sharpness. Only retain observation frames with a gain ratio greater than a preset gain threshold and an information quality score greater than a preset information threshold to form a multi-frame effective observation set.

[0009] Preferably, step S3, which involves extracting two-dimensional and three-dimensional semantic key points, specifically includes: Using the complete mask as a constraint, within the target ROI region of the effective observation image, the visual feature map of the target is extracted using DINOv2, and the contour corners, main axis endpoints, structural turning points, connection boundary points, and related edge points of the target are predicted by the key point heatmap regression head to obtain a two-dimensional semantic key point set. Using the fused target local point cloud as input, PointNet++ is used for hierarchical feature encoding, and the corresponding 3D semantic key point set is extracted through the key point regression branch.

[0010] Preferably, the bi-domain consistency matching and robust geometric estimation in step S3 specifically include: Read the predefined set of template key points from the candidate category shape prior template; The template key points are matched with the two-dimensional semantic key point set and the three-dimensional semantic key point set respectively. The two-dimensional matching is based on the relative positional relationship of the key points on the complete contour, the local edge direction and the topological relationship of the semantic components. The three-dimensional matching is based on the local curvature of the point cloud, the distribution of normal vectors and the neighborhood geometric relationship. A dual-domain consistency screening mechanism is introduced to retain only the correspondence of key points that satisfy the consistency constraints in both the two-dimensional contour domain and the three-dimensional structural domain, forming a set of effective matching point pairs. The effective set of matching points is input into TEASER++ to perform robust initial pose solution, which yields the initial rotation matrix and initial translation vector. Then, the initial rotation matrix and initial translation vector are used to perform point-to-plane ICP local fine registration to obtain the initial six-degree-of-freedom pose.

[0011] Preferably, step S3 is followed by an initial pose confidence evaluation step: The initial six-DOF pose is applied to the target category prior template, reprojected onto the reference image plane to obtain the predicted contour, and matched with the contour boundary of the complete mask to calculate the contour reprojection error. Simultaneously, the point-to-plane registration error between the template point cloud and the fused target local point cloud in the current pose is calculated; An initial pose confidence score is constructed by combining the average residual of key point matching, contour reprojection error, point cloud registration error, and multi-frame observation consistency error. When the initial pose confidence score is lower than the preset initial pose threshold, return to step S2 to re-execute the supplementary view acquisition or switch to the backup prior template to solve again.

[0012] Preferably, the multi-frame reprojection consistency constraint in step S4 specifically includes: The initial six-DOF pose is applied to the target category prior template. PyTorch3D is used to perform differentiable rendering in each frame of the effective observation image to generate the template's predicted contour, prediction mask, and set of visible boundaries. The predicted contour is matched point-by-point with the boundary of the complete contour of the frame to calculate the contour reprojection error, and the overlap between the predicted mask and the target mask of the frame is compared to calculate the mask consistency error. The reprojection error of multiple frames is weighted and fused, and frames with clearer outlines, higher segmentation confidence, and more complete visible areas are given greater weight.

[0013] Preferably, the local geometry joint optimization in step S4 specifically includes: The target category prior template is transformed into the observation coordinate system under the current pose to obtain the current template point cloud. Point-to-plane ICP registration is performed with the fused target local point cloud, and normal consistency constraints are added to minimize the angle difference between the template normal and the observation normal. Construct a joint optimization objective function: ; in This is a combined term of multi-frame contour reprojection error and mask consistency error; For point-to-plane registration error, This refers to the normal consistency error; The consistency error of pose projection results under multi-frame observation; α, β, γ, δ are weighting coefficients; Using the initial six-DOF pose as the initial value for iteration, the pose increment is iteratively optimized using CeresSolver combined with the Levenberg-Marquardt algorithm until the joint objective function converges or the maximum number of iterations is reached, and the final six-DOF pose is output. Calculate the multi-frame contour reprojection error, 3D point cloud registration error, normal consistency error, and optimized convergence residual of the final six-DOF pose. Construct the final pose confidence score. When the score is higher than the preset final confidence threshold, output the final six-DOF pose to the grasping and planning module.

[0014] This invention also discloses a multi-morphological target pose recognition system for a metering warehouse, applied to the aforementioned multi-morphological target pose recognition method for a metering warehouse, comprising: Image acquisition module: used to acquire RGB images and depth maps of the current scene, and control the camera to perform local micro-scanning in the active view completion process to acquire multiple frames of RGB-D images; Contour restoration and occlusion assessment module: used to coarsely locate the target object, segment the visible area in the time-series image sequence, combine depth boundary correction and category shape prior to complete the complete contour, and calculate the occlusion rate of the target object; Active observation enhancement module: used to plan local micro-scanning trajectories when the occlusion rate exceeds the threshold, and to perform spatial registration, quality screening and fusion of multiple RGB-D images to form a multi-frame effective observation set; The pose initial solution module is used to extract two-dimensional and three-dimensional semantic key points from the complete contour and multi-frame effective observation set, perform dual-domain consistency matching with the category shape prior template, and obtain the initial six-degree-of-freedom pose by robust geometric estimation and local point cloud registration. Pose refinement module: used to perform multi-frame reprojection consistency constraints and local geometry joint optimization on the initial six-DOF pose, and output the final six-DOF pose and confidence score; Storage and execution module: Used to store the target object's category shape prior library, pose recognition parameters and final pose result, and transmit the final six-degree-of-freedom pose to the robotic arm grasping planning unit.

[0015] The technical effects and advantages of this invention are as follows: 1. This method for recognizing the pose of multi-morphological targets in a metrology warehouse employs coarse localization, temporal visible region segmentation, depth boundary correction, category shape prior completion, and occlusion rate calculation. It triggers active viewpoint acquisition based on the occlusion rate, and then combines dual-domain consistency matching of 2D and 3D semantic key points, robust geometric estimation, and local point cloud registration to obtain the initial pose. Finally, it utilizes multi-frame reprojection consistency constraints and local geometric joint optimization to output the final pose. This invention can still stably obtain high-precision six-degree-of-freedom poses even when the target object is severely occluded, stacked, or has missing local structures, thus improving the pose recognition success rate in complex occlusion scenarios.

[0016] 2. This method for recognizing the pose of multi-morphological targets in a metrology warehouse employs an active viewpoint acquisition process. Specifically, it extracts the main occlusion boundaries and missing contours of the target based on the difference between the complete mask and the corrected visible mask. The outward normal direction of the missing region is used as the priority viewpoint direction. Multiple frames of RGB-D images are acquired through small-range swinging in the horizontal and pitch directions. Spatial registration, visible region gain rate filtering, and information quality scoring are then performed on the multi-frame observations. This invention effectively overcomes the limitation of insufficient information from a single frame observation, obtaining more comprehensive and reliable target observation data with less robotic arm movement, and enhancing the system's adaptability to heavily occluded targets.

[0017] 3. This method for recognizing the pose of multi-morphological targets in a metrology warehouse, by constructing a joint optimization objective function, unifies the multi-frame contour reprojection error, the point-to-plane registration error between the template point cloud and the observation point cloud, the normal consistency error, and the multi-frame observation consistency error into the same optimization framework. It also uses differentiable rendering and a nonlinear optimization solver for iterative refinement. This invention enables the final pose to achieve optimal performance in both two-dimensional complete contour alignment and three-dimensional local geometric fitting, thereby improving the geometric consistency of pose estimation and the positioning accuracy required for grasping operations. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 This is a logic diagram for contour restoration and occlusion evaluation in this invention; Figure 3 This is the active perspective acquisition, processing, and judgment diagram of the present invention; Figure 4 This is a diagram illustrating the dual-domain consistency matching and initial pose solution of the present invention. Figure 5 This is the multi-frame joint optimization and final pose output diagram of the present invention; Figure 6 This is a system module architecture diagram of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This embodiment discloses a method for recognizing the pose of multi-morphological targets in a metering warehouse, as shown in the attached figure. Figure 1 To be continued Figure 6 As shown, it includes: Step S1: Obtain the RGB image and depth map of the current scene, perform coarse localization of the target object, segment the visible area of ​​the target object in the temporal image sequence, correct the visible area by combining depth boundary information, and then use category shape prior to complete the complete outline of the target object, while calculating the occlusion rate of the target object. Step S2: Determine whether active supplementary view acquisition is needed based on the occlusion rate. If so, control the camera to perform local micro-scanning around the target candidate area to acquire multiple RGB-D images. Then, perform spatial registration and quality screening on the multiple RGB-D images to form a set of multiple effective observations. Step S3: Based on the complete contour and multiple frames of effective observation set, extract the two-dimensional semantic key points and three-dimensional semantic key points of the target object, perform dual-domain consistency matching with the category shape prior template, and use robust geometric estimation and local point cloud registration to obtain the initial six-degree-of-freedom pose of the target object. Step S4: Using a set of multiple effective observations, perform multi-frame reprojection consistency constraints and local geometry joint optimization on the initial six-degree-of-freedom pose, and output the final six-degree-of-freedom pose and confidence of the target object.

[0022] Furthermore, in step S1, the present invention first acquires the RGB image at the current moment. and depth map It also receives the target category text output by the upper-level task module. RGB image With target category text The YOLO-World model is input together to perform coarse localization of target objects in the current scene that are semantically relevant to the task, and outputs a set of candidate bounding boxes. ; in This represents the i-th candidate box. Indicates the confidence level of the detection. The candidate bounding box represents the target category. Then, the candidate bounding boxes are filtered based on the detection confidence and category semantic matching score. The candidate region B^ with the highest score is selected as the region of interest for the current target object. After obtaining the target candidate region B^, the region is used as the center, and the current frame and its preceding L_m are analyzed. Extracting a short temporal ROI image sequence from a single frame: ; The ROI image sequence is then input into the SAM2 model, which utilizes its temporal memory capability to segment and associate the visible parts of the target object in consecutive frames, and outputs the initial visible mask for the current frame: ; and the corresponding segmentation confidence plot To suppress boundary drift caused by RGB segmentation alone, this invention uses depth maps... Project the image onto the target ROI region and calculate the depth gradient map:

[0023] and surface normal abrupt change map ; Then, combine the depth boundary constraints with the initial visible mask Make corrections to obtain a more accurate visible mask that better matches the contours of the real object. , where R(·) represents the mask correction function based on depth boundary consistency.

[0024] Furthermore, this invention pre-constructs a priori library of the categories and shapes of target objects in the metrology warehouse. For each type of target c, its standard two-dimensional contour template, typical depth section template, and key structural edge set are stored offline, denoted as: ; During online inference, based on the modified visibility mask Based on the contour features, current depth structure features, and target category information, the system retrieves the most similar object from the shape prior library for the corresponding category. A set of prior templates is formed to create a candidate prior set: ; Then, the visual features of the current ROI are... Temporal memory characteristics of SAM2 Correcting visible masks and candidate category prior set The common input AmodalCompletion branch infers the occluded portion of the target object through a prior attention fusion mechanism, and outputs the complete mask of the target object in the unoccluded state:

[0025] And complete the confidence graph Finally, calculate the occlusion rate of the target object: ; in This represents the visible pixel area of ​​the target object from the current viewpoint. This represents the complete pixel area of ​​the target object in the unobstructed state, and ε is a stabilizing term to prevent the denominator from being zero.

[0026] Specifically disclosed, in step S2, the system first determines the occlusion rate. With preset occlusion trigger threshold Compare; when ≤ If the current observation remains unchanged, proceed directly to the next step; if > When the target is determined to be in a heavily occluded state, the active view acquisition process is triggered, based on the complete mask obtained in S1. With modified visible mask In the difference region, the main occlusion boundary and missing contour of the target are extracted, and the outward normal direction of the missing region is used as the priority compensation view direction to construct a local micro-scanning trajectory set. For the RGB-D camera installed at the end of the robotic arm, with the current observation pose as the center, the angle range in the horizontal and pitch directions is respectively [missing information]. and Small-range oscillations, planned A supplementary perspective pose: , At each pose point, the system acquires the corresponding RGB image and depth map to form a supplementary view observation sequence: ; in, , , and the interval between adjacent sampling times These are the core control parameters for this step.

[0027] Specifically disclosed, after obtaining the supplementary view observation sequence, this invention further performs target region tracking, spatial registration, and observation quality enhancement on multiple frames of RGB-D data, using the target candidate region output in S1. and visible mask As an initial prompt, the SAM2 model is invoked again in each supplementary view frame to perform temporal mask propagation and segmentation updates on the target region, obtaining the set of visible masks corresponding to each supplementary view frame: ; ORB-SLAM3 is used to estimate the camera pose between adjacent supplementary view frames, and the pose transformation matrix of each frame relative to the initial reference frame is obtained: ; Then, by combining the depth point cloud within the target area with point-to-plane ICP fine registration, the camera pose is further corrected. Based on the registered multi-frame pose and the target mask of each frame, the depth point cloud belonging to the target area in each frame is back-projected onto a unified coordinate system and fused to form an enhanced target local point cloud set. Define the visible region gain rate: ; in This indicates the corrected visible mask from the initial viewpoint. This represents the visible mask under the i-th complementary viewpoint, only Greater than the preset gain threshold Only when the time is right will the observation be retained, and at the same time, the overall information quality score of each frame of supplementary viewpoint observation will be calculated: ; in This represents the average segmentation confidence of the target mask in the i-th frame. This indicates the percentage of effective depth pixels within the target area. Indicates an image sharpness index. For weight parameters, when Greater than the preset information threshold When this frame is selected, it is considered a valid observation frame. After filtering, the final set of valid supplementary view frames constitutes a multi-frame valid observation set. ; in This indicates the number of valid observation frames after filtering. This represents the pose transformation from the corresponding frame to the reference coordinate system.

[0028] It is particularly important to emphasize that in step S3, the present invention extracts semantic key structural features from the two-dimensional complete contour and the three-dimensional fused point cloud, respectively. For the two-dimensional part, a complete mask is used. To constrain this process, within the target ROI region of each valid observation image frame, DINOv2 is used to extract the target visual feature map, and a keypoint heatmap regression head is constructed on it to predict the target's contour corners, principal axis endpoints, structural inflection points, connection boundary points, and grasping related edge points, resulting in a two-dimensional semantic keypoint set. ; in This indicates the number of key points; for the 3D portion, it is represented by the fused local point cloud of the target. As input, PointNet++ is used to perform hierarchical feature encoding on the target point cloud, and the corresponding 3D semantic keypoint set is extracted through the keypoint regression branch: ; For each template in the candidate category prior set, read its predefined template keypoint set. Then, combine the template key points with the current set of observed key points. and Similarity matching is performed separately. On the 2D side, matching is based on the relative positions of key points on the complete contour, local edge directions, and semantic component topological relationships. On the 3D side, matching is based on the local curvature of the point cloud, normal vector distribution, and neighborhood geometric relationships. A dual-domain consistency filtering mechanism is introduced, retaining only the correspondences of key points that satisfy consistency constraints in both the 2D contour domain and the 3D structural domain, forming a set of valid matching point pairs. ; in This indicates the number of valid corresponding point pairs.

[0029] It is particularly important to emphasize that after establishing the set of valid corresponding point pairs C, the method of this invention first uses TEASER++ to perform robust initial pose solution on the matching point pairs. The 3D keypoint set on the observation side and the corresponding keypoint set on the template side are input into TEASER++. Abnormal matching points are suppressed through truncated least squares and graph optimization mechanisms. The initial rigid body transformation of the target object is solved to obtain the initial rotation matrix. and initial translation vector , and then with and As initial values, the 3D structural model corresponding to the category prior template is projected onto the observation point cloud coordinate system, and the point-to-plane ICP method is used to further register and optimize the local geometry to obtain a more accurate initial pose. At the same time, the currently obtained initial pose is... The target category prior template is applied and reprojected onto the reference image plane to obtain the predicted contour of the template in the current viewpoint. Then combine it with the complete mask. contour boundary Perform matching and calculate the contour reprojection error: ; Where d(·) represents the minimum Euclidean distance from the contour point to the predicted boundary, and the point-to-plane registration error between the template point cloud and the observed point cloud in the current pose is calculated: ; in This represents the normal vector at observation point q. Represent the template point cloud at the current pose and its corresponding projection point, and construct the initial pose confidence score: ; Where e_match represents the average residual of keypoint matching. This represents the consistency error of pose projection results across multiple observation frames. For weighting coefficients, when Higher than the preset initial pose threshold At that time, This serves as the input for the subsequent pose refinement module; otherwise, return to S2 to re-execute the supplementary view acquisition or switch to a backup prior template.

[0030] It is particularly important to emphasize that in step S4, the present invention will set the initial pose. Applying to the prior template of the target category, PyTorch3D is used to perform differentiable rendering of the template mesh or template point cloud, generating the predicted contour of the template in each effective observation view frame. Predictive mask and the set of visible boundaries Then, the predicted contour will be Complete outline boundary of the real target in this frame Perform point-by-point matching and calculate the contour reprojection error; simultaneously, apply the predicted mask... The overlap with the target mask of this frame is compared to calculate the mask consistency error. A weighted fusion of reprojection errors from multiple frames is performed, with frames having clearer contours, higher segmentation confidence, and more complete visible areas receiving greater weight. In 3D space, the target category prior template is placed at the current pose. The point cloud of the current template is obtained by down-transforming it to the observation coordinate system. Then, the target local point cloud is obtained by fusing it with S2. Local geometric registration is performed using a combination of point-to-plane ICP and normal consistency constraints to locally optimize the template surface and the observed point cloud surface, thus constructing a joint objective function for pose refinement: ; in This represents a combined term of multi-frame contour reprojection error and mask consistency error. This represents the point-to-plane registration error between the template point cloud and the observed point cloud. This represents the normal consistency error between the template normal and the observed normal. The expression represents the consistency error of pose projection results under multiple frame observations, where α, β, γ, and δ are the weighting coefficients of each error term. CeresSolver is used as the nonlinear optimization solver, combined with Levenberg... Iterative optimization using the Marquardt algorithm, to As initial values ​​for iteration, the pose parameters are represented as a combination of the current pose and the incremental pose. In each iteration, the Jacobian matrix is ​​calculated based on the current error term, the pose increment Δξ is updated, and the rotation matrix and translation vector are corrected accordingly, until the objective function converges or the maximum number of iterations is reached. HuberLoss is introduced as a robust loss function in both the contour error term and the point cloud error term. The loss function is applied when the decrease in the objective function over two consecutive iterations is less than a preset convergence threshold. When the optimization has reached convergence, the current pose is output as the final refined pose. ; Finally, calculate the multi-frame contour reprojection error. 3D point cloud registration error Normal consistency error And optimize convergence residuals Construct the final pose confidence score: ; in For weight parameters, when Higher than the preset final confidence threshold At that time, The final output pose of the target object is then passed to the subsequent grasping planning module.

[0031] This invention also discloses a multi-morphological target pose recognition system for a metrology warehouse, applied to a method for multi-morphological target pose recognition in a metrology warehouse, comprising: Image acquisition module: used to acquire RGB images and depth maps of the current scene, and control the camera to perform local micro-scanning in the active view completion process to acquire multiple frames of RGB-D images; Contour restoration and occlusion assessment module: used to coarsely locate the target object, segment the visible area in the time-series image sequence, combine depth boundary correction and category shape prior to complete the complete contour, and calculate the occlusion rate of the target object; Active observation enhancement module: used to plan local micro-scanning trajectories when the occlusion rate exceeds the threshold, and to perform spatial registration, quality screening and fusion of multiple RGB-D images to form a multi-frame effective observation set; The pose initial solution module is used to extract two-dimensional and three-dimensional semantic key points from the complete contour and multi-frame effective observation set, perform dual-domain consistency matching with the category shape prior template, and obtain the initial six-degree-of-freedom pose by robust geometric estimation and local point cloud registration. Pose refinement module: used to perform multi-frame reprojection consistency constraints and local geometry joint optimization on the initial six-DOF pose, and output the final six-DOF pose and confidence score; Storage and Execution Module: Used to store the target object's category shape prior library, pose recognition parameters, and final pose results, and to transmit the final six-degree-of-freedom pose to the robotic arm's grasping planning unit.

[0032] The workflow of the present invention will be described in detail below with reference to specific embodiments: Example 1: This example uses current transformers in a metering warehouse as the target object, and describes the workflow of the present invention in detail under a scenario of partial stacking and obstruction, in conjunction with the appendix. Figure 1 To be continued Figure 6 .

[0033] As attached Figure 1 As shown, after system initialization, the RGB-D camera installed at the end of the robotic arm acquires the RGB image and depth map of the current scene. The task target issued by the host computer is "current transformer". The system inputs the image and text into the YOLO-World model and outputs candidate boxes. Since the current transformer is partially obscured by two stacked meters, the initial detection box includes the exposed terminals and part of the housing of the current transformer. From the current frame and the previous 4 frames ( =5) Input the ROI sequence into SAM2 to obtain the initial visible mask. Calculate the depth gradient map after projection of the depth map, correct the mask boundaries, and remove drift caused by the meter edges. Retrieve the standard contour template (including key structures such as terminal blocks, core windows, and mounting base plates) from the transformer category prior library. AmodalCompletion completes the branch and outputs the complete mask, setting the occlusion rate. The calculated value is 0.62, which is greater than the threshold. =0.4, triggering the active view compensation process.

[0034] As attached Figure 2 As shown, the system plans a horizontal direction of ±15° based on the outward normal direction of the missing area (the middle part of the current transformer blocked by the meter). =30°), pitch direction ±10° ( =20°) A total of 9 ( =9) Supplementing the viewpoint: The robotic arm moves the camera sequentially to each pose point, acquiring RGB-D image sequences. For each supplemented viewpoint frame, SAM2 performs mask propagation using the initial mask as a cue to obtain the visible masks for each frame. ORB-SLAM3 estimates the inter-frame pose, and after ICP fine registration, the point clouds are fused, and the gain ratio for each frame is calculated. In the third frame (tilt +10°), the gain ratio reached 0.28 and the information quality score was 0.91, both exceeding the threshold. =0.15, =0.7), retained; the 7th frame was discarded due to severe depth loss, scoring only 0.32, resulting in 7 effective observation frames N_e=7, forming an enhanced point cloud. .

[0035] As attached Figure 3 As shown, from the complete mask The DINOv2 algorithm is used to extract two-dimensional key points of the current transformer, such as terminal corner points, core window corner points, and base plate edge points. =12); from the merged point cloud PointNet++ was used to extract the corresponding 3D keypoints, and dual-domain matching was performed with the prior template of the mutual inductor. A total of 9 keypoint pairs were successfully matched in both 2D and 3D. The initial pose was obtained by inputting TEASER++. , Then, through ICP fine registration, it is obtained Calculate the contour reprojection error =0.023, point cloud registration error =0.0018, initial confidence level =0.87, higher than the threshold. =0.7, entering the fine-tuning stage.

[0036] As attached Figure 4As shown, As initial values, differentiable rendering is performed in PyTorch3D over 7 valid observations to compute the joint objective function. CeresSolver converges after 8 iterations, yielding the final pose. Its contour reprojection error was reduced to 0.009, point cloud registration error was reduced to 0.0006, and the final confidence level was [missing information]. =0.94, higher than =0.8, the system will The data was sent to the robotic arm grasping planning unit, and the robotic arm successfully grasped the current transformer, verifying the effectiveness of the invention.

[0037] Example 2: This example uses a switch in a metering warehouse as the target object. It details the workflow of the invention in a scenario where the switch is against a wall and closely obstructed by adjacent meter boxes, in conjunction with the attached... Figure 1 To be continued Figure 6 .

[0038] As attached Figure 5 As shown, the switch is placed close to the wall, with approximately 70% of its right side obscured by an adjacent meter box. The switch itself is made of black plastic with weak texture. The initial YOLO-World detection confidence score is 0.78. The visible mask segmented by SAM2 only includes the left edge of the switch and part of the button. After depth gradient map correction, Amodal Completion is performed using a shape prior library for the switch category (including standard switch outline, rocker button position, terminal block, etc.) to obtain a complete mask with a low occlusion rate. =0.83, the system triggers active view compensation. Due to the switch being against the wall, the outward normal direction is limited, and the planned horizontal direction is ±20°. =40°), pitch direction ±5° ( =10°) A total of 6 viewpoints were obtained. The right supplementary viewpoint was invalid due to wall obstruction, but the left and upper viewpoints captured the upper edge of the switch and button details. After registration and fusion, the button protrusion and terminal pit features of the switch in the point cloud were restored. DINOv2 extracted key points such as button corners and switch housing turning points within the complete mask. PointNet++ extracted the corresponding 3D points in the fused point cloud. During dual-domain matching, the confidence of 2D matching was low due to weak texture, but 3D geometric matching (curvature change) provided sufficient constraints, ultimately forming 7 effective matching point pairs. After TEASER++ solution, the initial pose error was large (rotation error of about 8°), but after ICP fine registration, the error was reduced to 1.2°. After joint optimization, the final pose error was less than 0.5°. The success rate of the grasping experiment was increased from 32% of the traditional method to 91%, which fully demonstrates the superiority of this invention in weak texture and closely occluded scenes.

[0039] Example 3: This example uses a communication module in a metering warehouse as the target object, illustrating the batch processing and real-time performance of the present invention in a multi-target random stacking scenario, in conjunction with the appendix. Figure 6 .

[0040] The communication module is small in size and roughly cuboid in shape, with gold fingers and snap-fit ​​features on its surface. In stacked scenarios, multiple modules overlap each other, and only the end faces or snap-fits of some modules can be seen in a single frame image. This invention first outputs multiple candidate boxes (corresponding to each module) through YOLO-World, and executes processes S1 to S4 in parallel for each candidate box. Since the prior library of the communication module's category shape contains parameters such as standard length, width, and gold finger position, AmodalCompletion can restore the complete outline of the module that is pressed underneath. The active view compensation strategy targets the upper module in the stacked area, and only requires a micro-scan in the horizontal direction of ±10° to expose the side edges of the lower module. Through multi-threaded processing, the average processing time for each module is about 0.8 seconds (excluding the time of the robotic arm movement), which meets the requirements of real-time warehouse operations. Finally, the pose of each module is output, and the grasping system successfully grasps all modules in the order from the top layer to the bottom layer, solving the planning problem of "identifying first and then grasping" in stacked scenarios.

[0041] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing the pose of multi-morphological targets in a metering warehouse, characterized in that, Includes the following steps: Step S1: Obtain the RGB image and depth map of the current scene, perform coarse localization of the target object, segment the visible area of ​​the target object in the time-series image sequence, correct the visible area by combining depth boundary information, and then use category shape prior to complete the complete outline of the target object, while calculating the occlusion rate of the target object. Step S2: Determine whether active supplementary view acquisition is needed based on the occlusion rate. If so, control the camera to perform local micro-scanning around the target candidate area to acquire multiple frames of RGB-D images, and perform spatial registration and quality screening on the multiple frames of RGB-D images to form a set of multiple effective observations. Step S3: Based on the complete contour and the set of multiple effective observations, extract the two-dimensional semantic key points and three-dimensional semantic key points of the target object, perform dual-domain consistency matching with the category shape prior template, and use robust geometric estimation and local point cloud registration to obtain the initial six-degree-of-freedom pose of the target object. Step S4: Using the multi-frame effective observation set, perform multi-frame reprojection consistency constraint and local geometry joint optimization on the initial six-degree-of-freedom pose, and output the final six-degree-of-freedom pose and confidence of the target object.

2. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 1, characterized in that, Step S1 further includes: The RGB image and the target category text are input into the YOLO-World model to obtain a set of candidate boxes, and the candidate region with the highest score is selected as the region of interest. Centered on the region of interest, from the current frame and its preceding Lm A short temporal ROI image sequence is cropped from a single frame image and input into the SAM2 model to obtain the initial visible mask and segmentation confidence map; The depth map is projected onto the region of interest, the depth gradient map and the surface normal abrupt change map are calculated, and the initial visible mask is corrected based on the consistency of the depth boundary to obtain the corrected visible mask. A prior library of the category and shape of the target object in the measurement warehouse is pre-built. Based on the contour features and depth structure features of the corrected visible mask, candidate prior templates are retrieved from the prior library of the corresponding category. Input the current ROI visual features, SAM2 temporal memory features, corrected visible mask and candidate prior templates into the AmodalCompletion branch to complete the target object; The occlusion rate is calculated based on the area ratio of the modified visible mask to the full mask.

3. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 2, characterized in that, In step S2, when the occlusion rate is greater than the preset occlusion trigger threshold, the active supplementary view acquisition process is triggered. The local micro-scanning specifically involves: extracting the main occlusion boundaries and missing contours of the target based on the difference region between the complete mask and the corrected visible mask; using the outward normal direction of the missing region as the priority compensation view direction; and, with the current observation pose as the center, performing angle-range operations in the horizontal and pitch directions respectively. and Small-range oscillations, planned A supplementary viewpoint pose is obtained, and RGB images and depth maps are acquired at each pose point to form a supplementary viewpoint observation sequence.

4. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 3, characterized in that, The spatial registration and quality screening of multiple RGB-D images in step S2 further includes: Using the corrected visible mask as the initial cue, the SAM2 model is called in each supplementary view frame to perform temporal mask propagation, thereby obtaining the visible mask corresponding to each supplementary view frame; ORB-SLAM3 is used to estimate the camera pose between adjacent supplementary view frames, obtain the pose transformation matrix of each frame relative to the initial reference frame, and perform fine registration using point-to-plane ICP. Based on the registered multi-frame poses and target masks of each frame, the depth point cloud belonging to the target region in each frame is back-projected onto a unified coordinate system and fused to form an enhanced target local point cloud. Calculate the visible area gain ratio of each supplementary viewpoint relative to the initial viewpoint, and calculate the information quality score by combining segmentation confidence, depth efficiency, and image sharpness. Only retain observation frames with a gain ratio greater than a preset gain threshold and an information quality score greater than a preset information threshold to form a multi-frame effective observation set.

5. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 4, characterized in that, The extraction of two-dimensional and three-dimensional semantic key points in step S3 specifically includes: Using the complete mask as a constraint, within the target ROI region of the effective observation image, the visual feature map of the target is extracted using DINOv2, and the contour corners, main axis endpoints, structural turning points, connection boundary points, and related edge points of the target are predicted by the key point heatmap regression head to obtain a two-dimensional semantic key point set. Using the fused target local point cloud as input, PointNet++ is used for hierarchical feature encoding, and the corresponding 3D semantic key point set is extracted through the key point regression branch.

6. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 5, characterized in that, The bi-domain consistency matching and robust geometric estimation in step S3 specifically include: Read the predefined set of template key points from the candidate category shape prior template; The template key points are matched with the two-dimensional semantic key point set and the three-dimensional semantic key point set respectively. The two-dimensional matching is based on the relative positional relationship of the key points on the complete contour, the local edge direction and the topological relationship of the semantic components. The three-dimensional matching is based on the local curvature of the point cloud, the distribution of normal vectors and the neighborhood geometric relationship. A dual-domain consistency screening mechanism is introduced to retain only the correspondence of key points that satisfy the consistency constraints in both the two-dimensional contour domain and the three-dimensional structural domain, forming a set of effective matching point pairs. The effective set of matching points is input into TEASER++ to perform robust initial pose solution, which yields the initial rotation matrix and initial translation vector. Then, the initial rotation matrix and initial translation vector are used to perform point-to-plane ICP local fine registration to obtain the initial six-degree-of-freedom pose.

7. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 6, characterized in that, The step S3 is followed by an initial pose confidence evaluation step: The initial six-DOF pose is applied to the target category prior template, reprojected onto the reference image plane to obtain the predicted contour, and matched with the contour boundary of the complete mask to calculate the contour reprojection error. Simultaneously, the point-to-plane registration error between the template point cloud and the fused target local point cloud in the current pose is calculated; An initial pose confidence score is constructed by combining the average residual of key point matching, contour reprojection error, point cloud registration error, and multi-frame observation consistency error. When the initial pose confidence score is lower than the preset initial pose threshold, return to step S2 to re-execute the supplementary view acquisition or switch to the backup prior template to solve again.

8. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 1, characterized in that, The multi-frame reprojection consistency constraint in step S4 specifically includes: The initial six-DOF pose is applied to the target category prior template. PyTorch3D is used to perform differentiable rendering in each frame of the effective observation image to generate the template's predicted contour, prediction mask, and set of visible boundaries. The predicted contour is matched point-by-point with the boundary of the complete contour of the frame to calculate the contour reprojection error, and the overlap between the predicted mask and the target mask of the frame is compared to calculate the mask consistency error. The reprojection error of multiple frames is weighted and fused, and frames with clearer outlines, higher segmentation confidence, and more complete visible areas are given greater weight.

9. The method for recognizing the pose of multi-morphological targets in a metering warehouse according to claim 8, characterized in that, The local geometry joint optimization in step S4 specifically includes: The target category prior template is transformed into the observation coordinate system under the current pose to obtain the current template point cloud. Point-to-plane ICP registration is performed with the fused target local point cloud, and normal consistency constraints are added to minimize the angle difference between the template normal and the observation normal. Construct a joint optimization objective function: ； in This is a combined term of multi-frame contour reprojection error and mask consistency error; For point-to-plane registration error, This refers to the normal consistency error; The consistency error of pose projection results under multi-frame observation; α, β, γ, δ are weighting coefficients; Using the initial six-DOF pose as the initial value for iteration, the pose increment is iteratively optimized using CeresSolver combined with the Levenberg-Marquardt algorithm until the joint objective function converges or the maximum number of iterations is reached, and the final six-DOF pose is output. Calculate the multi-frame contour reprojection error, 3D point cloud registration error, normal consistency error, and optimized convergence residual of the final six-DOF pose. Construct the final pose confidence score. When the score is higher than the preset final confidence threshold, output the final six-DOF pose to the grasping and planning module.

10. A multi-morphological target pose recognition system for a metering warehouse, applied to the multi-morphological target pose recognition method for a metering warehouse as described in any one of claims 1-9, characterized in that, include: Image acquisition module: used to acquire RGB images and depth maps of the current scene, and control the camera to perform local micro-scanning in the active view completion process to acquire multiple frames of RGB-D images; Contour restoration and occlusion assessment module: used to coarsely locate the target object, segment the visible area in the time-series image sequence, combine depth boundary correction and category shape prior to complete the complete contour, and calculate the occlusion rate of the target object; Active observation enhancement module: used to plan local micro-scanning trajectories when the occlusion rate exceeds the threshold, and to perform spatial registration, quality screening and fusion of multiple RGB-D images to form a multi-frame effective observation set; The pose initial solution module is used to extract two-dimensional and three-dimensional semantic key points from the complete contour and multi-frame effective observation set, perform dual-domain consistency matching with the category shape prior template, and obtain the initial six-degree-of-freedom pose by robust geometric estimation and local point cloud registration. Pose refinement module: used to perform multi-frame reprojection consistency constraints and local geometry joint optimization on the initial six-DOF pose, and output the final six-DOF pose and confidence score; Storage and execution module: Used to store the target object's category shape prior library, pose recognition parameters and final pose result, and transmit the final six-degree-of-freedom pose to the robotic arm grasping planning unit.

Images