A 6D Pose Estimation Method for Industrial Targets Based on Multimodal Information Fusion and Geometric Constraints

Abstract

This invention relates to the field of machine vision, specifically disclosing a 6D pose estimation method for industrial targets based on multimodal information fusion and geometric constraints. The method includes: acquiring binocular images and target images under two lighting conditions; performing depth estimation based on the binocular images; performing grayscale statistics and binarization fusion on the dual-lit images to obtain a fused image; performing rotational target detection on the fused image to obtain a bounding box, and performing instance segmentation under its constraints to obtain a target mask; generating a scene target point cloud based on the depth results, coordinate transformation, and the mask; performing top-plane fitting on the point cloud and extracting the four corner points of the observed top surface; establishing candidate correspondences between the observed four corner points and the CAD four corner points; constructing an orthogonal coordinate base based on the long side, short side, and plane normal of the top surface; and solving for the 6D pose. This invention solves the problem of low pose estimation accuracy caused by the inseparable gaps between weakly textured, highly reflective, and closely arranged industrial targets through multimodal information fusion and geometric constraints.

Description

Technical Field

[0001] This invention belongs to the field of machine vision, specifically a method for 6D pose estimation of industrial targets based on multimodal information fusion and geometric constraints. Background Technology

[0002] In the field of industrial automation, six-degree-of-freedom pose estimation is a core component in applications such as unordered grasping, workpiece loading, and assembly by robots. Common pose estimation methods are mainly divided into two categories: one is based on two-dimensional image features, which directly regresses the pose through template matching or deep learning networks, and has good performance in environments with rich textures and clear target boundaries; the other is based on three-dimensional point cloud registration, which uses algorithms such as iterative nearest point to align the scene point cloud with the CAD model to obtain the pose transformation. However, for industrial targets such as magnesium ingots and aluminum ingots, their surfaces usually exhibit weak or even no texture, and the metal material is prone to high reflectivity. At the same time, multiple target objects are often closely arranged along the stacking direction, with extremely narrow gaps between the targets.

[0003] In the aforementioned complex scenarios, images acquired from a single illumination direction are difficult to clearly show the boundaries between targets, leading to problems such as target adhesion, inaccurate boundary positioning, and increased depth direction errors in pose estimation methods based on two-dimensional image features. On the other hand, the method of directly using point cloud registration is prone to convergence to an incorrect pose because the target point cloud contains adjacent target points, background points, or edge missing points.

[0004] Therefore, how to overcome the key defect that the gaps between targets are inseparable under a single illumination condition in industrial scenes with weak texture, high reflectivity and close proximity of targets, and stably determine the six-degree-of-freedom pose of the target objects is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a 6D pose estimation method for industrial targets based on multimodal information fusion and geometric constraints. This method solves the problem of low pose estimation accuracy caused by the indivisible gaps between weak textured, highly reflective, and closely packed industrial targets under single illumination conditions.

[0006] This invention provides a 6D pose estimation method for industrial targets based on multimodal information fusion and geometric constraints, comprising the following steps:

[0007] Acquire binocular images of the same target scene and target images acquired under two lighting methods, wherein the target scene includes a target object whose pose is to be determined;

[0008] Depth estimation is performed based on the binocular images to obtain binocular depth results;

[0009] Based on the target images under the two lighting methods, grayscale statistics and binarization fusion are performed to obtain the fused target image;

[0010] Rotational target detection is performed on the fused target image to obtain the rotational target bounding box corresponding to the target object;

[0011] Instance segmentation is performed within the target area constrained by the rotated target bounding box to obtain the target instance mask corresponding to the target object;

[0012] Based on the binocular depth results, coordinate transformation, and target instance mask, a scene target point cloud corresponding to the target object is generated;

[0013] The target point cloud of the scene is fitted with a top plane, and the four corner points of the observed top plane are extracted based on the fitted top plane.

[0014] Based on the four corner points of the observed top surface and the four corner points of the CAD top surface in the 3D CAD model of the target object, determine at least one candidate corner point correspondence;

[0015] Based on the correspondence of the candidate four corner points, an observation-side orthogonal coordinate base and a CAD-side orthogonal coordinate base are constructed according to the constraints of the long side direction of the top surface, the short side direction of the top surface, and the plane normal. The six-degree-of-freedom pose of the target object is determined according to the observation-side orthogonal coordinate base and the CAD-side orthogonal coordinate base.

[0016] Preferably, the grayscale statistics and binarization fusion of the target image based on the two lighting methods specifically includes:

[0017] The first target image acquired under the first lighting method is converted into a first grayscale image, and the second target image acquired under the second lighting method is converted into a second grayscale image;

[0018] The low grayscale values ​​corresponding to the target gap region in the first grayscale image and the second grayscale image are counted respectively, and the median of the low grayscale values ​​is determined as the binarization threshold under the corresponding lighting mode;

[0019] The first grayscale image and the second grayscale image are binarized based on the binarization threshold, and the binarization results are fused to enhance the target gap in different directions.

[0020] The fused target image is generated based on the binarization result after fusion.

[0021] Preferably, the rotating target detection is implemented using the YOLO-OBB model, and the instance segmentation is implemented using the FastSAM model to generate a target instance mask within the target area constrained by the rotating target bounding box.

[0022] Preferably, the step of generating scene target point clouds based on binocular depth results, coordinate transformation, and the target instance mask specifically includes:

[0023] The depth map in the binocular depth results is back-projected into an initial point cloud in the left eye camera coordinate system using the intrinsic parameters of the left eye camera.

[0024] Perform invalid depth filtering and depth range filtering on the initial point cloud;

[0025] The filtered point cloud is transformed to the RGB camera coordinate system using the extrinsic parameter matrix from the left-eye camera coordinate system to the RGB camera coordinate system.

[0026] The point cloud in the RGB camera coordinate system is projected onto the RGB image plane, and the target points are filtered by combining the target instance mask to obtain the scene target point cloud;

[0027] Specifically, the selection of target points includes: for pixel positions in the RGB image plane, using a z-buffer to record the nearest depth of the points projected onto the pixel positions; and determining the points whose projection positions are located within the target instance mask and whose depth values ​​are different from the nearest depth of the corresponding pixel positions, satisfying the foreground depth threshold, as target points.

[0028] Preferably, the step of performing top plane fitting on the target point cloud of the scene specifically includes:

[0029] Perform random sampling consistent plane segmentation on the target point cloud of the scene to obtain an initial top plane and an initial set of points within the top plane;

[0030] Perform singular value decomposition plane fitting on the initial set of points inside the top surface to obtain the first plane parameters;

[0031] The target point cloud of the scene is re-screened for the inner points of the top surface according to the preset fine screening distance threshold, and the singular value decomposition plane fitting is performed again on the re-screened inner points of the top surface to obtain the target top surface plane.

[0032] Adjust the normal vector of the target top plane so that the normal vector is oriented toward the camera or satisfies a preset orientation constraint.

[0033] Preferably, the step of extracting the four corner points of the observed top surface based on the fitted top surface plane specifically includes:

[0034] Using the center of the in-plane point on the top surface of the target as the local coordinate origin, the local two-dimensional coordinate axes of the target top surface are constructed through singular value decomposition.

[0035] Project the points in the top surface onto the local two-dimensional coordinate system defined by the local two-dimensional coordinate axes;

[0036] Find the minimum bounding rectangle of the points in the top surface in the local two-dimensional coordinate system;

[0037] The four corner points of the minimum bounding rectangle are transformed from the local two-dimensional coordinate system to the camera coordinate system to obtain the four corner points of the observed top surface.

[0038] Preferably, determining the correspondence between at least one candidate four corner points specifically includes:

[0039] Obtain the four corner points of the CAD top surface defined in a preset order in the 3D CAD model;

[0040] The four corner points of the observed top surface are enumerated in clockwise, counterclockwise and cyclic starting point changes to obtain multiple candidate four corner point arrangements;

[0041] From the multiple candidate four-corner point arrangements, select the arrangement that satisfies that the first adjacent side is the long side and the second adjacent side is the short side, and establish a correspondence between the selected arrangement and the four corner points of the CAD top surface.

[0042] Preferably, determining the six-degree-of-freedom pose of the target object specifically includes:

[0043] The first axis of the CAD side is constructed according to the long side direction of the four corner points of the CAD top surface, the second axis of the CAD side is constructed according to the short side direction of the four corner points of the CAD top surface, and the third axis of the CAD side is determined by the first axis of the CAD side and the second axis of the CAD side, thus obtaining the orthogonal coordinate base of the CAD side.

[0044] The first axis of the observation side is constructed based on the direction of the long side among the four corner points of the observation top surface, the second axis of the observation side is constructed based on the direction of the short side among the four corner points of the observation top surface, and the third axis of the observation side is determined based on the normal vector of the top surface plane, thus obtaining the orthogonal coordinate base of the observation side.

[0045] The rotation matrix is ​​determined based on the product relationship between the observation-side orthogonal coordinate base and the CAD-side orthogonal coordinate base;

[0046] The translation vector is determined based on the positional relationship between the center point of the observed top surface and the center point of the CAD top surface after the transformation by the rotation matrix;

[0047] A 4×4 pose matrix is ​​generated from the CAD object coordinate system to the camera coordinate system based on the rotation matrix and the translation vector.

[0048] Preferably, the method further includes: sorting the candidate six-degree-of-freedom poses generated by the correspondence between different candidate four corner points, and determining the candidate six-degree-of-freedom pose with the best sorting as the six-degree-of-freedom pose of the target object;

[0049] The sorting criteria include: whether the z-axis of the candidate six-degree-of-freedom pose satisfies the same direction constraint as the normal vector of the top plane; the root mean square error between the four corner points of the CAD top surface after transformation by the candidate six-degree-of-freedom pose and the four corner points of the observed top surface; and the absolute value of the cosine of the angle between the z-axis of the candidate six-degree-of-freedom pose and the normal vector of the top plane.

[0050] Preferably, when the target scene includes multiple target objects, each target instance mask is treated as an independent target instance, and scene target point cloud generation, top plane fitting, extraction of four corner points of the observed top surface, determination of the correspondence of candidate four corner points, and determination of six degrees of freedom pose are performed respectively to obtain the six degrees of freedom pose corresponding to each of the multiple target objects.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] By fusing grayscale statistics and adaptive binarization of images illuminated from both directions, the separability of gaps between weakly textured and highly reflective industrial targets arranged in different directions is significantly enhanced, fundamentally solving the problems of target boundary adhesion and unclear gaps under single illumination conditions.

[0053] Based on this, a rotating target bounding box is used to accurately locate individual target regions and constrain the segmentation range of instances, effectively avoiding interference from neighboring targets and background regions on the segmentation results, and obtaining target instance masks with better boundary fit and higher quality.

[0054] Furthermore, this invention unifies the binocular depth estimation results to the RGB camera coordinate system through coordinate transformation and uses a z-buffer to filter foreground visible points, ensuring strict consistency between the generated target point cloud and the instance mask, eliminating the mixing of background points and occlusion points. By performing top-plane fitting on the target point cloud using a combination of random sampling consistency segmentation and singular value decomposition fine fitting, and extracting the four corner points of the observed top surface using the minimum bounding rectangle in the local two-dimensional coordinate system, the stability of the plane normal and the accuracy of corner point extraction are guaranteed. Based on the multi-candidate enumeration correspondence between the four corner points of the observed top surface and the four corner points of the CAD model top surface, an orthogonal coordinate basis is constructed by combining the long side, short side and plane normal of the top surface, thereby stably determining the rotation matrix and translation vector, avoiding the problems of uncertain normal direction, confusion of long and short sides and attitude flipping that are easy to occur in pure four-point fitting or point cloud registration.

[0055] Finally, by comprehensively considering the constraint that the object's z-axis is in the same direction as the plane normal, the corner reprojection error, and the consistency of the normal angle, the candidate poses are ranked, and the optimal six-DOF pose is output. This invention integrates multi-illumination images, accurate instance masks, depth point clouds, and CAD geometric priors throughout the entire process, achieving stable and accurate estimation of the six-DOF pose of target objects in industrial scenes with weak texture, high reflectivity, and closely arranged targets. Attached Figure Description

[0056] Figure 1 This is a flowchart of the 6D pose estimation method for industrial targets in Embodiment 1 of the present invention;

[0057] Figure 2 This is a schematic diagram of the image preprocessing results in Embodiment 1 of the present invention;

[0058] Figure 3 This is a schematic diagram of the target detection and segmentation results in Embodiment 1 of the present invention;

[0059] Figure 4 This is a schematic diagram of the target point cloud top surface fitting and four corner point extraction results in Embodiment 1 of the present invention;

[0060] Figure 5 This is a schematic diagram showing the correspondence between the four corners of the CAD top surface definition and the four observation points in Embodiment 1 of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1: This example provides a 6D pose estimation method for industrial targets based on multimodal information fusion and geometric constraints, specifically using magnesium ingots as the target object for illustration;

[0063] It should be understood that this method is also applicable to aluminum ingots, long strip-shaped metal workpieces, regular block-shaped industrial parts, or other industrial targets with observable principal planes.

[0064] The hardware system used in this embodiment includes:

[0065] An Intel RealSense D435i binocular depth camera integrates a left-eye infrared camera, a right-eye infrared camera, and an RGB color camera. The left and right eyes are used to acquire binocular images to generate depth maps, while the RGB camera is used to acquire target images. The left-eye camera and the RGB camera communicate via an extrinsic parameter matrix. Coordinate correlation was established, and the matrix was obtained using the checkerboard calibration method.

[0066] Two sets of LED strip light sources are arranged in the front-back and left-right directions of the target scene, respectively, to achieve lighting from two different directions;

[0067] An industrial computer used to perform image processing and pose calculation.

[0068] like Figure 1 As shown, the specific steps of this method include:

[0069] Step 101: Acquire binocular images and dual-light target images.

[0070] Control the first group of light sources (front and back direction) to work, and simultaneously capture the first target image through the RealSense D435i's RGB camera, recorded as... Control the second set of light sources (left and right directions) to acquire images of the second target, and record them as follows: Simultaneously, images of the left eye are acquired using the left and right eye infrared cameras of the RealSense D435i. And right eye image All images maintain a consistent resolution, such as 1920×1080 pixels. The above acquisition process can be completed simultaneously or within a very short time interval to ensure that the position of the target object remains essentially unchanged.

[0071] Step 102: Perform depth estimation based on the binocular images to obtain the binocular depth results.

[0072] Left eye image And right eye image Input a pre-trained binocular depth estimation model; in this embodiment, the model uses a FoundationStereo network structure; FoundationStereo outputs a depth map aligned with the left eye image. Each pixel value represents the distance (in meters) from the point in the left eye camera coordinate system to the optical center plane of the left eye camera. Invalid points (such as sky or reflective areas) may exist in the depth map, which are marked with 0 or NaN and filtered out in subsequent processing.

[0073] Step 103: Perform grayscale statistics and binarization fusion based on the dual-lighting image to generate the fused target image.

[0074] like Figure 2 As shown, firstly, the first target image... Convert to grayscale , the second target image Convert to grayscale The conversion formula uses the standard luminance formula: .

[0075] Statistical analysis of grayscale images separately and The low grayscale value corresponding to the gap region between the target; the specific method is as follows: perform pixel grayscale value statistics on the entire grayscale image. Since the gap between the magnesium ingots appears as a dark stripe in the image, its grayscale value is usually lower than that of the target and the background; select the grayscale level near the 5th quantile on the left side of the grayscale histogram, and count the specific value corresponding to this grayscale level. Take the median of multiple samples as the binarization threshold under this lighting method; for example, for the first lighting method, the threshold is obtained by statistics. For the second lighting method, the threshold was statistically determined. .

[0076] Based on threshold grayscale image Binarization is performed: pixel grayscale values ​​greater than or equal to Set to 255 (white) and the rest to 0 (black) to obtain a binary image. Also based on right Processing yields a binary image Then on and Perform a logical "OR" fusion to obtain a fused binary image. In the fused binary image, the gaps between targets under both lighting directions are preserved as black areas.

[0077] Finally, with Inverted and used as a mask, it is superimposed on the original RGB image (e.g. Generate a fused target image on the ) This image enhances the gap features, making subsequent detection more accurate.

[0078] Step 104: Perform rotation target detection on the fused target image to obtain the rotated target bounding box.

[0079] fused target images Input a YOLO-OBB model; the YOLO-OBB model is pre-trained and can output a rotated bounding box surrounding a single magnesium ingot, such as... Figure 3 As shown; each rotated target box contains five parameters: center point coordinates Length of the longer side Length of the shorter side Rotation angle (in radians); for cases with multiple magnesium ingots in the figure, the model outputs multiple rotating target boxes; in this embodiment, YOLO-OBB achieves an average accuracy mAP of 0.99 on the test set, demonstrating high detection reliability.

[0080] Step 105: Perform instance segmentation under the constraints of the rotated target bounding box to obtain the target instance mask.

[0081] For each rotated target bounding box, it is used as a box prompt for the FastSAM model; specifically, the merged target images are... The coordinate parameters of the axis-aligned bounding box (AABB) obtained by transforming the rotated target box are input into the FastSAM model. The center point coordinates, long side length, short side length, and rotation angle of the rotated target box are used to determine the range of the axis-aligned bounding box. Based on the box cues, the model generates a pixel-level mask for a single magnesium ingot within this region. Since the rotated target box has roughly defined the range of a single magnesium ingot, FastSAM only needs to perform fine segmentation within this small region, avoiding interference from adjacent magnesium ingots and the background. Figure 3 As shown; the output mask It is a binary image of the same size as the original RGB image. A pixel value of 1 in the mask indicates that it belongs to the target, and otherwise it is 0.

[0082] For multiple rotated target bounding boxes, corresponding instance masks are generated for each, and each instance mask is treated as an independent target.

[0083] Step 106: Generate scene target point cloud based on stereo depth results, coordinate transformation and instance mask.

[0084] This step details how to generate a corresponding 3D point cloud for a single instance mask.

[0085] First, using the intrinsic parameter matrix of the left eye camera Depth map The effective pixel back projection is a 3D point in the left eye camera coordinate system; for pixel coordinates and its depth value (Unit: meters), the back projection formula is:

[0086]

[0087] in , The focal length of the left-eye camera. , Principal point coordinates; for depth values ​​of 0 or exceeding a reasonable range (such as setting...). , Points that are not in the range of ) are removed.

[0088] Then, using the extrinsic parameter matrix from the left-eye camera coordinate system to the RGB camera coordinate system (4×4 rigid body transformation matrix, pre-obtained using the checkerboard calibration method), transform the left-eye point cloud to the RGB camera coordinate system:

[0089]

[0090] Next, the point cloud in the RGB camera coordinate system is projected onto the RGB image plane; for each 3D point ( , , Using RGB camera intrinsics Calculate its projected coordinates:

[0091]

[0092] For each RGB pixel position Use a z-buffer to record the depth values ​​of all points projected onto that pixel. The smallest point (i.e., the point closest to the camera) is denoted as... .

[0093] Finally, points belonging to the target instance are selected; a 3D point is retained if:

[0094] Its projected coordinates Instance mask falling on the target Inside;

[0095] Its depth value With the corresponding pixel The difference between them is less than the preset foreground depth threshold. (e.g., 0.01 meters).

[0096] Points that meet the above conditions constitute the scene target point cloud of the target object. .

[0097] Step 107: Perform top-plane fitting on the scene target point cloud and extract the four corner points of the observed top surface, such as... Figure 4 As shown.

[0098] (a) Top plane fitting

[0099] Input point cloud Perform the following operations:

[0100] ① Random Sample Consensus (RANSAC) Plane Partitioning: Setting the maximum number of iterations to 1000 and the distance threshold to 0.005 meters, the initial principal plane equations are obtained. and the initial set of interior points .

[0101] ② To Perform Singular Value Decomposition (SVD) plane fitting: Decenter the interior point coordinates, construct the covariance matrix, and perform SVD decomposition. The eigenvector corresponding to the minimum singular value is the plane normal vector. Then calculate based on the center point Thus, the first plane parameters are obtained.

[0102] ③ Fine screening: Set stricter distance thresholds Meters, traversing the original point cloud All points in the plane, if the point is on the plane The distance is less than Then add it to the set of interior points of the fine sieve. .

[0103] ④ To Perform SVD plane fitting again to obtain the target top surface plane. To ensure the normal vector points towards the camera, check the normal vector. relative to the camera's optical axis (usually If the dot product (in the RGB camera coordinate system) is negative, then the entire normal vector is inverted, and the parameters are adjusted accordingly. The final normal vector points to the camera.

[0104] (ii) Extracting the four corner points of the observation top surface

[0105] ① Calculate the set of points inside the top surface center .

[0106] ② with The decentralized coordinate matrix is ​​used for SVD decomposition, and the first two principal component directions are taken as local two-dimensional coordinate axes. and These two axes lie in a plane and are orthogonal to each other.

[0107] ③ Convert each interior point Projected onto a local two-dimensional coordinate system: .

[0108] ④ In Find the minimum bounding rectangle of a set of points in a plane; the method is: calculate the bounding rectangle of all points. , , , The local coordinates of the four corners of the rectangle are obtained as follows: .

[0109] ⑤ Transform the local coordinates back to the RGB camera coordinate system: The three-dimensional coordinates of the four corner points of the observed top surface are obtained and recorded in the current order as follows: , , , .

[0110] Step 108: Determine the candidate correspondence between the four corner points of the CAD top surface and the four corner points of the observed top surface, such as... Figure 5 As shown.

[0111] Load the 3D CAD model file of the target object (magnesium ingot) (e.g., obj_000001.PLY); in the definition of the CAD model, the four corner points of the top surface are given in a fixed order: TL (top left), TR (top right), BR (bottom right), BL (bottom left). The specific values ​​(unit: meters) are:

[0112]

[0113]

[0114]

[0115]

[0116] Note: The origin of the CAD coordinate system is located at the center of the top surface. The X-axis is along the long side (from BL to BR), the Y-axis is along the short side (from TL to BL or from TR to BR), and the Z-axis is perpendicular to the top surface and upwards (satisfying the right-hand rule). The actual dimensions of the top surface are 0.551 meters long and 0.062 meters wide.

[0117] Since the order of the observed four corner points output in step 107 is uncertain, this embodiment uses an enumeration method to establish the correspondence with the CAD four corner points; specific operation:

[0118] Observation of the four corner points , , , Perform permutation enumeration, including:

[0119] Clockwise (0,1,2,3), (1,2,3,0), (2,3,0,1), (3,0,1,2);

[0120] Counterclockwise (0,3,2,1), (3,2,1,0), (2,1,0,3), (1,0,3,2).

[0121] For each permutation, determine its geometric relationship:

[0122] The length of the first adjacent side (from point 0 to point 1) should be close to the target long side (0.551 meters), with an allowable error of ±0.02 meters;

[0123] The length of the second adjacent side (from point 1 to point 2) should be close to the target short side (0.062 meters), with an allowable error of ±0.01 meters.

[0124] At the same time, the angle between the first adjacent side and the second adjacent side should be close to 90° (±10° allowed).

[0125] Permutations that satisfy the above conditions are retained and interpreted as Each retained permutation constitutes a candidate correspondence.

[0126] Step 109: Solve the six-degree-of-freedom pose based on the local coordinate system of the top surface and the plane normal constraint.

[0127] For each candidate mapping, perform the following calculation:

[0128] (a) Constructing the orthogonal coordinate base on the CAD side

[0129] Find the four corner points in the CAD file: , , (BR can be derived from others); Calculation:

[0130]

[0131]

[0132]

[0133] Then, normalization and orthogonalization (e.g., Gram-Schmidt) are performed to obtain an orthogonal coordinate basis. Each axis is a column vector.

[0134] (ii) Constructing an orthogonal coordinate base for the observation side

[0135] Use the candidate correspondence to observe the four corner points , , :

[0136]

[0137]

[0138] Meanwhile, the target top surface plane normal vector obtained in step 107 (Already normalized) as the third axis on the observation side To ensure orthogonality, use Correction and :

[0139]

[0140] Then, by normalization, we obtain... , and guarantee (Make the coordinate system right-handed). Finally, establish an orthogonal coordinate base on the observation side. .

[0141] (iii) Calculate the rotation matrix and translation vector

[0142] Rotation matrix satisfy ,therefore:

[0143]

[0144] in yes The transpose, because It is an orthogonal matrix.

[0145] Calculate the center of the observed top surface CAD top center Translation vector:

[0146]

[0147] The final 4×4 homogeneous pose matrix is ​​obtained:

[0148]

[0149] This matrix represents the rigid body transformation from the CAD object coordinate system to the RGB camera coordinate system.

[0150] Step 110: Candidate pose sorting and output.

[0151] Because there may be multiple candidate correspondences (usually...) (Number of candidates), each calculated pose needs to be evaluated and ranked; this embodiment adopts the following priority rule:

[0152] Prioritize z-axis same-direction constraints: Check the z-axis of the object in the candidate pose (i.e., The third column) and the top plane normal vector obtained in step 107 The dot product; if the dot product is greater than 0.9 (basically in the same direction), it is marked as... Otherwise Preferred selection Candidates.

[0153] The next step is to perform corner point RMSE transformation: transform the four corner points of the CAD top surface to the camera coordinate system using candidate poses to obtain the transformed four points. Calculate the root mean square error (RMSE) between the four observed points:

[0154]

[0155] exist Among the candidates, select the one with the smallest RMSE.

[0156] Finally, for consistency of normal angles: if the RMSE of two candidates are very close (difference less than 1e-4 meters), then compare. Select the candidate with the largest absolute value (i.e., the object's z-axis is most consistent with the plane normal).

[0157] The final determined optimal candidate , , This refers to the six-degree-of-freedom pose of the target object.

[0158] Step 111: Multi-objective processing.

[0159] When there are multiple rotated target boxes and corresponding instance masks in the fused target image, steps 106 to 110 are repeated for each instance mask to output the independent pose of each magnesium ingot; all pose results can be saved in a list for subsequent robot grasping or assembly.

[0160] Step 112 (optional): Geometric consistency evaluation and visualization.

[0161] To verify the reliability of the pose results, the CAD model point cloud can be transformed to the camera coordinate system using the calculated pose, resulting in the transformed point cloud of the model. .calculate With scene target point cloud If the average nearest neighbor distance between the two is less than a preset threshold (e.g., 0.005 meters), the pose is considered valid. At the same time, the transformed CAD model outline can be projected onto an RGB image and overlaid with the instance mask for easy manual inspection.

[0162] This embodiment achieves stable 6D pose estimation of magnesium ingot targets in industrial scenarios with weak texture, high reflectivity, and close arrangement through the above steps. Experimental results show that the pose angle error of this method is less than 2° and the translation error is less than 3 mm, which meets the requirements of industrial grasping.

[0163] Example 2: This example provides a pose determination device for executing the method described in Example 1. The device includes:

[0164] Image acquisition unit: used to acquire binocular images of the same target scene and target images acquired under two lighting methods, the target scene including target objects whose poses need to be determined; also used to acquire camera parameters (left eye intrinsic parameters, RGB intrinsic parameters, checkerboard calibration extrinsic parameters, etc.).

[0165] Depth estimation unit: used to perform depth estimation based on the stereo image to obtain stereo depth results.

[0166] Image fusion unit: used to perform grayscale statistics and binarization fusion on the target images under the two lighting methods to obtain a fused target image.

[0167] Detection unit: used to perform rotation target detection on the fused target image to obtain the rotation target bounding box corresponding to the target object.

[0168] Segmentation unit: used to perform instance segmentation within the target area constrained by the rotated target box to obtain the target instance mask corresponding to the target object.

[0169] Point cloud generation unit: used to generate scene target point cloud corresponding to the target object based on the stereo depth result, coordinate transformation and the target instance mask.

[0170] Geometric extraction unit: used to fit the top plane of the target point cloud of the scene, and extract the four corner points of the observed top plane based on the fitted top plane.

[0171] Correspondence determination unit: used to determine at least one candidate corner point correspondence based on the four corner points of the observed top surface and the four corner points of the CAD top surface in the three-dimensional CAD model of the target object.

[0172] Pose determination unit: for constructing an observation-side orthogonal coordinate base and a CAD-side orthogonal coordinate base based on the long side direction of the top surface, the short side direction of the top surface, and the plane normal constraint for the correspondence of the candidate four corner points, and determining the six-degree-of-freedom pose of the target object according to the observation-side orthogonal coordinate base and the CAD-side orthogonal coordinate base.

[0173] Sorting and Output Unit: Used to sort the candidate six-DOF poses generated by different candidate four-corner point correspondences, and determine the candidate six-DOF pose with the best sorting as the final output.

[0174] The specific implementation methods of each unit are the same as the corresponding steps in Embodiment 1, and will not be repeated here; the device can be deployed in an industrial computer, an embedded vision processor or a robot controller.

[0175] Example 3: This example provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the pose estimation method described in Example 1 is implemented. The storage medium includes, but is not limited to, non-temporary storage media such as ROM, RAM, hard disk, solid-state hard disk, optical disk, and USB flash drive.

[0176] This embodiment also provides a computer program product that, when run on a computer, executes the method described in Embodiment 1; the computer program product can be provided to the user in the form of a software installation package, online update, or pre-installation.

[0177] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any changes, modifications, substitutions and variations made by those skilled in the art to the above embodiments within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for 6D pose estimation of industrial targets based on multimodal information fusion and geometric constraints, characterized in that, Includes the following steps: Acquire binocular images of the same target scene and target images acquired under two lighting methods, wherein the target scene includes a target object whose pose is to be determined; Depth estimation is performed based on the binocular images to obtain binocular depth results; Based on the target images under the two lighting methods, grayscale statistics and binarization fusion are performed to obtain the fused target image; Rotational target detection is performed on the fused target image to obtain the rotational target bounding box corresponding to the target object; Instance segmentation is performed within the target area constrained by the rotated target bounding box to obtain the target instance mask corresponding to the target object; Based on the binocular depth results, coordinate transformation, and target instance mask, a scene target point cloud corresponding to the target object is generated; The target point cloud of the scene is fitted with a top plane, and the four corner points of the observed top plane are extracted based on the fitted top plane. Based on the four corner points of the observed top surface and the four corner points of the CAD top surface in the 3D CAD model of the target object, determine at least one candidate corner point correspondence; Based on the correspondence of the candidate four corner points, an observation-side orthogonal coordinate base and a CAD-side orthogonal coordinate base are constructed according to the constraints of the long side direction of the top surface, the short side direction of the top surface, and the plane normal. The six-degree-of-freedom pose of the target object is determined according to the observation-side orthogonal coordinate base and the CAD-side orthogonal coordinate base.

2. The method according to claim 1, characterized in that, The grayscale statistics and binarization fusion of the target image based on the two lighting methods specifically includes: The first target image acquired under the first lighting method is converted into a first grayscale image, and the second target image acquired under the second lighting method is converted into a second grayscale image; The low grayscale values ​​corresponding to the target gap region in the first grayscale image and the second grayscale image are counted respectively, and the median of the low grayscale values ​​is determined as the binarization threshold under the corresponding lighting mode; The first grayscale image and the second grayscale image are binarized based on the binarization threshold, and the binarization results are fused to enhance the target gap in different directions. The fused target image is generated based on the binarization result after fusion.

3. The method according to claim 1, characterized in that, The rotating target detection is implemented using the YOLO-OBB model, and the instance segmentation is implemented using the FastSAM model to generate a target instance mask within the target area constrained by the rotating target bounding box.

4. The method according to claim 1, characterized in that, The generation of scene target point clouds based on binocular depth results, coordinate transformation, and the target instance mask specifically includes: The depth map in the binocular depth results is back-projected into an initial point cloud in the left eye camera coordinate system using the intrinsic parameters of the left eye camera. Perform invalid depth filtering and depth range filtering on the initial point cloud; The filtered point cloud is transformed to the RGB camera coordinate system using the extrinsic parameter matrix from the left-eye camera coordinate system to the RGB camera coordinate system. The point cloud in the RGB camera coordinate system is projected onto the RGB image plane, and the target points are filtered by combining the target instance mask to obtain the scene target point cloud; Specifically, the selection of target points includes: for pixel positions in the RGB image plane, using a z-buffer to record the nearest depth of the points projected onto the pixel positions; and determining the points whose projection positions are located within the target instance mask and whose depth values ​​are different from the nearest depth of the corresponding pixel positions, satisfying the foreground depth threshold, as target points.

5. The method according to claim 1, characterized in that, The step of fitting the top plane of the target point cloud in the scene specifically includes: Perform random sampling consistent plane segmentation on the target point cloud of the scene to obtain an initial top plane and an initial set of points within the top plane; Perform singular value decomposition plane fitting on the initial set of points inside the top surface to obtain the first plane parameters; The target point cloud of the scene is re-screened for the inner points of the top surface according to the preset fine screening distance threshold, and the singular value decomposition plane fitting is performed again on the re-screened inner points of the top surface to obtain the target top surface plane. Adjust the normal vector of the target top plane so that the normal vector is oriented toward the camera or satisfies a preset orientation constraint.

6. The method according to claim 1, characterized in that, The extraction of the four corner points of the observed top surface based on the fitted top surface plane specifically includes: Using the center of the in-plane point on the top surface of the target as the local coordinate origin, the local two-dimensional coordinate axes of the target top surface are constructed through singular value decomposition. Project the points in the top surface onto the local two-dimensional coordinate system defined by the local two-dimensional coordinate axes; Find the minimum bounding rectangle of the points in the top surface in the local two-dimensional coordinate system; The four corner points of the minimum bounding rectangle are transformed from the local two-dimensional coordinate system to the camera coordinate system to obtain the four corner points of the observed top surface.

7. The method according to claim 1, characterized in that, The determination of at least one candidate four-corner point correspondence specifically includes: Obtain the four corner points of the CAD top surface defined in a preset order in the 3D CAD model; The four corner points of the observed top surface are enumerated in clockwise, counterclockwise and cyclic starting point changes to obtain multiple candidate four corner point arrangements; From the multiple candidate four-corner point arrangements, select the arrangement that satisfies that the first adjacent side is the long side and the second adjacent side is the short side, and establish a correspondence between the selected arrangement and the four corner points of the CAD top surface.

8. The method according to claim 1, characterized in that, Determining the six-degree-of-freedom pose of the target object specifically includes: The first axis of the CAD side is constructed according to the long side direction of the four corner points of the CAD top surface, the second axis of the CAD side is constructed according to the short side direction of the four corner points of the CAD top surface, and the third axis of the CAD side is determined by the first axis of the CAD side and the second axis of the CAD side, thus obtaining the orthogonal coordinate base of the CAD side. The first axis of the observation side is constructed based on the direction of the long side among the four corner points of the observation top surface, the second axis of the observation side is constructed based on the direction of the short side among the four corner points of the observation top surface, and the third axis of the observation side is determined based on the normal vector of the top surface plane, thus obtaining the orthogonal coordinate base of the observation side. The rotation matrix is ​​determined based on the product relationship between the observation-side orthogonal coordinate base and the CAD-side orthogonal coordinate base; The translation vector is determined based on the positional relationship between the center point of the observed top surface and the center point of the CAD top surface after the transformation by the rotation matrix; A 4×4 pose matrix is ​​generated from the CAD object coordinate system to the camera coordinate system based on the rotation matrix and the translation vector.

9. The method according to claim 1, characterized in that, Also includes: The candidate six-degree-of-freedom poses generated by different candidate four-corner point correspondences are sorted, and the candidate six-degree-of-freedom pose with the best sort is determined as the six-degree-of-freedom pose of the target object; The sorting criteria include: whether the z-axis of the candidate six-degree-of-freedom pose satisfies the same direction constraint as the normal vector of the top plane; the root mean square error between the four corner points of the CAD top surface after transformation by the candidate six-degree-of-freedom pose and the four corner points of the observed top surface; and the absolute value of the cosine of the angle between the z-axis of the candidate six-degree-of-freedom pose and the normal vector of the top plane.

10. The method according to claim 1, characterized in that, When the target scene includes multiple target objects, each target instance mask is treated as an independent target instance, and scene target point cloud generation, top plane fitting, extraction of four corner points of the observed top surface, determination of the correspondence of candidate four corner points, and determination of six degrees of freedom pose are performed respectively to obtain the six degrees of freedom pose corresponding to each of the multiple target objects.

Images