Palletizing pose solution method and system based on fusion of visual identifier and point cloud

Abstract

This invention discloses a method and system for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds, relating to the fields of industrial visual inspection, 3D point cloud processing, and visual positioning. The invention first simultaneously acquires 2D grayscale images and 3D point cloud data of the packing box using a 3D camera; then, it utilizes AprilTag visual identifiers to complete the packing box identification and 6D coarse pose calculation; using the coarse pose as prior information, it performs ROI dynamic cropping and multi-level preprocessing on the 3D point cloud to obtain a clean point cloud; next, it obtains the 6D fine pose of the packing box through plane fitting, Euclidean clustering, and coordinate system orthogonalization; finally, it combines an embedded process database to recursively obtain the target pose. This invention integrates the semantic recognition advantages of visual identifiers with the high-precision positioning advantages of point clouds, solving problems such as random packing box poses, internal structural occlusion, poor point cloud quality due to metal reflection, and difficulty in adapting to multiple types of packing boxes. It boasts high positioning accuracy and robustness, making it suitable for industrial automated packing scenarios.

Description

Technical Field

[0001] This invention relates to the fields of industrial visual inspection, 3D point cloud processing, and visual positioning technology, and in particular to a method and system for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The pose of a workpiece frame refers to its position and orientation information in three-dimensional space, and is a crucial foundation for achieving automated robotic grasping and packing. In industrial automation production scenarios such as automotive stamping, sheet metal processing, and home appliance manufacturing, workpiece frames serve as carriers for workpiece storage and transfer, and accurate pose calculation is a prerequisite for automated packing. In actual production, workpiece frames are typically moved and placed manually by forklifts or AGVs, resulting in positional deviations of ±50mm and angular deviations of ±30°. Long-term heavy-load turnover can also cause plastic deformation. Simultaneously, the complex internal structure of the workpiece frame, with interlayer supports and slots easily causing visual obstruction, and the metal surface prone to reflection, leads to voids, flying points, and noise in the 3D point cloud, posing challenges to accurate pose calculation of the workpiece frame.

[0004] Existing methods for calculating the pose of material frames mainly fall into two categories: The first category is pose calculation methods based on point cloud registration. These methods utilize the ICP iterative nearest point algorithm and its improved versions to calculate the pose by solving the rigid transformation matrix between the on-site point cloud and the CAD standard template point cloud. However, these methods are susceptible to poor point cloud quality caused by metal reflections and structural occlusion. Registration is prone to getting stuck in local optima, and they have poor robustness to random and deformable material frame poses. Furthermore, they cannot calculate the pose of the packing placement points within the visual blind spots inside the material frame, making it difficult to meet actual production needs.

[0005] The second category is pose calculation methods based on visually encoded tags, with AprilTag as a typical example. This method identifies visual tags affixed to the surface of the packing frame and calculates the frame's pose using PnP-type algorithms. This method is easy to deploy and has strong recognition robustness, but it is affected by manual application tolerances and label warping, resulting in insufficient positioning accuracy. It can only achieve coarse positioning and cannot meet the pose requirements of high-precision packing. Furthermore, when changing production lines for different types of packing frames, parameters need to be readjusted, indicating poor adaptability.

[0006] In summary, existing technologies cannot simultaneously address the four major industry pain points: random material frame pose, visual occlusion of internal structures, failure of metal reflective point clouds, and rapid adaptation of multi-variety material frames. They suffer from defects such as insufficient positioning accuracy, poor robustness, low production changeover efficiency, and inability to adapt to pose calculation in visual blind spots, making it difficult to meet the accuracy and robustness pose calculation requirements of industrial automated packing scenarios. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds. It combines the ease of semantic recognition of AprilTag visual identifiers with the high-precision positioning advantages of point cloud geometric features. The method employs a three-step progressive approach: first, coarse semantic positioning; then, fine point cloud positioning; and finally, blind zone pose calculation based on a database. By precisely defining the pose calculation logic through a homogeneous transformation matrix, it solves the technical challenges of random packing box poses, internal occlusion, metal reflection, and difficulties in changing production across multiple product types. The method features high positioning accuracy, strong robustness, and wide adaptability, and can be directly applied to various industrial automated packing scenarios.

[0008] On the one hand, a method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds is provided, including: The 2D grayscale image and 3D point cloud data of the material frame are acquired simultaneously using a 3D camera. Based on the AprilTag visual identifier in the 2D grayscale image, combined with the intrinsic parameters of the 3D camera, the unique TagID of the material frame is decoded, and 6D coarse pose calculation is performed to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; based on the TagID, the type of the corresponding material frame and the packing process parameters are retrieved from the pre-built embedded process database. Based on the 6D coarse pose homogeneous transformation matrix, the 3D point cloud data is subjected to ROI dynamic clipping and multi-level preprocessing to obtain a clean point cloud of the top surface of the rigid column of the material frame. Based on the pure point cloud, the 6D fine pose homogeneous transformation matrix is ​​obtained by plane fitting, Euclidean clustering and coordinate system orthogonalization. Based on the material frame type determined by the TagID, combined with the 6D fine pose homogeneous transformation matrix and the packing process parameters, the target pose of the packing placement point inside the material frame is obtained recursively through homogeneous coordinate transformation.

[0009] Furthermore, the 6D coarse pose calculation specifically involves: The 2D grayscale image is subjected to adaptive binarization, quadrilateral contour extraction, and sub-pixel corner optimization to complete AprilTag decoding, thereby obtaining the unique TagID and corner coordinates of the material frame. Combined with the corner coordinates of the marker in the 3D camera, the 6D pose of the AprilTag visual marker relative to the 3D camera coordinate system is calculated using the IPPE algorithm to obtain the 6D coarse pose homogeneous transformation matrix.

[0010] Furthermore, the embedded process database uses AprilTag's unique TagID as the primary key to store at least one packing process parameter among the following: material frame type, geometric dimensions, layer spacing, slot spacing, tilt angle, and safety release distance, in order to achieve index matching and rapid adaptation of multi-variety material frames.

[0011] Furthermore, the ROI dynamic clipping specifically refers to: the Z-axis translation amount of the 6D coarse pose homogeneous transformation matrix. For depth reference, set the pass-through filter interval to Where δ is the depth expansion margin, the invalid point cloud in the background is removed by direct filtering, and only the point cloud ROI area on the top surface of the column at the top edge of the material frame is retained.

[0012] Furthermore, the multi-stage preprocessing sequentially includes voxel downsampling, statistical filtering for noise reduction, and normal consistency constraint.

[0013] Furthermore, the formula for constructing the 6D fine pose homogeneous transformation matrix is ​​as follows: ; in, These are the unit vectors of the X, Y, and Z axes of the material frame coordinate system, respectively. is the translation vector of the origin of the material frame coordinate system relative to the 3D camera coordinate system.

[0014] Furthermore, the specific steps for calculating the 6D precise pose homogeneous transformation matrix include: The reference plane at the upper edge of the material frame is fitted by the RANSAC algorithm to obtain the plane normal vector and perform L2 normalization. The dot product is then checked in conjunction with the Z-axis direction of the 6D coarse pose homogeneous transformation matrix. If the dot product is negative, the normal vector is inverted to eliminate the ambiguity of the normal vector and obtain the Z-axis unit vector of the material frame coordinate system. By separating the point cloud subsets of the top surfaces of each column of the material frame through Euclidean clustering, setting the clustering distance threshold and the minimum clustering point threshold, the effective point cloud subsets of each column top surface are obtained. The geometric center of each subset is calculated as the reference point of the column top surface. The reference point closest to the origin of the AprilTag coordinate system is locked as the origin of the material frame coordinate system, and the translation vector of the material frame coordinate system relative to the 3D camera coordinate system is obtained. Starting from the origin of the material frame coordinate system, the reference points of adjacent short side columns are connected to obtain the initial direction vector of the X-axis. The projection component of the initial direction vector on the Z-axis is removed by Schmidt orthogonalization and L2 normalization is performed to obtain the unit vector of the X-axis perpendicular to the Z-axis. The Y-axis unit vector is obtained by cross product of the Z-axis unit vector and the X-axis unit vector using the right-hand rule.

[0015] Furthermore, the recursive formula for the target pose of the box placement point inside the material frame, obtained through homogeneous coordinate transformation, is as follows: ; in, This is the 6D fine pose homogeneous transformation matrix. Let be the homogeneous transformation matrix of the packing placement point in the material frame coordinate system. The target pose of the box placement point relative to the 3D camera coordinate system.

[0016] On the other hand, a system for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds is provided, including: The data acquisition module is configured to simultaneously acquire 2D grayscale images and 3D point cloud data of the material frame using a 3D camera. The coarse pose calculation module is configured to: decode the unique TagID of the material frame based on the AprilTag visual identifier in the 2D grayscale image and the intrinsic parameters of the 3D camera, and perform 6D coarse pose calculation to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; and retrieve the type and packing process parameters of the corresponding material frame from the pre-built embedded process database based on the TagID. The point cloud preprocessing module is configured to: perform ROI dynamic cropping and multi-level preprocessing on the three-dimensional point cloud data based on the 6D coarse pose homogeneous transformation matrix to obtain a clean point cloud of the top surface of the rigid column of the material frame. The fine pose calculation module is configured to: based on the pure point cloud, calculate the 6D fine pose homogeneous transformation matrix through plane fitting, Euclidean clustering and coordinate system orthogonalization. The target pose recursion module is configured to: based on the material frame type determined by the TagID, combined with the 6D fine pose homogeneous transformation matrix and the packing process parameters, recursively obtain the target pose of the packing placement point inside the material frame through homogeneous coordinate transformation.

[0017] In another aspect, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the method described in the first aspect.

[0018] The above technical solution has the following advantages or beneficial effects: (1) In terms of positioning accuracy, it can meet the high-precision pose positioning requirements of material frames in industrial settings. This invention adopts a positioning strategy that integrates AprilTag visual identifiers with point cloud geometric features. It constructs the pose calculation relationship using a homogeneous transformation matrix, first quickly obtaining the coarse pose through visual identifiers, and then achieving precise positioning based on the geometric features of the rigid columns of the material frame, effectively compensating for the positioning deviation caused by label mounting tolerances. Experimental results show that the maximum overall spatial positioning error of this method does not exceed 1.6mm, which is significantly improved in accuracy compared to the single visual identifier positioning method, and can stably meet the high-precision pose positioning requirements of material frames in industrial settings.

[0019] (2) In terms of adaptability to complex working conditions, it can adapt to complex industrial working conditions such as metal reflection and structural occlusion. The present invention uses coarse pose as prior information to perform ROI dynamic trimming on the point cloud, retaining only the effective area of ​​the upper edge column of the material frame, which can remove more than 99% of the background redundant data. While reducing the amount of computation, it effectively suppresses point cloud noise and holes, and improves the stability and robustness of the algorithm in the field interference environment.

[0020] (3) In terms of multi-variety adaptation and blind zone pose calculation, it can meet the engineering requirements of rapid production changeover of multi-variety material frames and blind zone pose calculation for packing. This invention constructs an embedded process database with TagID as the primary key, realizing code-free rapid adaptation and rapid switching of material frames of different specifications; at the same time, relying on the built-in geometric parameters of the database, the pose of the packing placement point in the visual blind zone inside the material frame is directly deduced through homogeneous coordinate transformation, thereby solving the technical problem that the bottom layer and inner station cannot be directly located due to internal occlusion, and ensuring that the packing pose is complete and solvable.

[0021] (4) In terms of computational efficiency and engineering integration, the computational process of this invention is simple, the coordinate system is rigorously constructed, the three-axis orthogonality of the material frame coordinate system is stable and controllable, and it can be easily integrated into existing industrial vision systems. Under normal working conditions, the pose calculation time is controlled within 100ms, with good real-time performance and strong engineering adaptability, and it has good practicality and promotion application value. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is an overall flowchart of the method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds in one or more embodiments of the present invention; Figure 2 This is a flowchart illustrating the AprilTag visual identifier recognition and 6D coarse pose calculation in one or more embodiments of the present invention, wherein... Figure 2 In the image (a), the original 2D grayscale image was acquired. Figure 2 (b) in the figure shows the binarized result and the quadrilateral contour fitting. Figure 2 Image (c) shows the sub-pixel corner extraction result. Figure 2 (d) in the image represents the result of tag decoding. Figure 3 This is a comparison image of the effects of point cloud ROI clipping and preprocessing in one or more embodiments of the present invention, wherein... Figure 3 (a) in the image is the original global scene point cloud map. Figure 3 (b) in the image is the point cloud image after ROI pass-through filtering and cropping. Figure 3(c) in the figure is the point cloud image after voxel downsampling and statistical filtering for denoising. Figure 3 (d) in the figure is the point cloud diagram of the top surface of the column after the normal consistency constraint; Figure 4 This is a schematic diagram illustrating the principle of 6D precise pose calculation of the material frame coordinate system in one or more embodiments of the present invention; Figure 5 This is a classification and packing execution logic diagram in one or more embodiments of the present invention; Figure 6 This is a schematic diagram of the internal structure and packing position parameters of a flat-lying stacked material frame in one or more embodiments of the present invention, wherein, Figure 6 (a) is a top view of a flat material frame. Figure 6 (b) is a side view of a flat material frame; Figure 7 This is a schematic diagram of the internal structure and packing position parameters of the vertical slot-type material frame in one or more embodiments of the present invention, wherein, Figure 7 (a) is a top view of a vertically placed material frame. Figure 7 (b) is a side view of a vertically placed material frame; Figure 8 This is a schematic diagram of the mounting position and coordinate system reference of the AprilTag visual identifier on the material frame in one or more embodiments of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. Those skilled in the art should understand that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] Example 1 This embodiment provides a method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds. Figure 1 The overall flowchart of the method in Embodiment 1 of the present invention includes the following steps: S101: Synchronously acquire 2D grayscale images and 3D point cloud data of the material frame using a 3D camera; S102: Based on the AprilTag visual identifier in the 2D grayscale image and combined with the intrinsic parameters of the 3D camera, decode to obtain the unique TagID of the material frame, and perform 6D coarse pose calculation to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; based on the TagID, retrieve the type of the corresponding material frame and the packing process parameters from the pre-built embedded process database. S103: Based on the 6D coarse pose homogeneous transformation matrix, the ROI dynamic clipping and multi-level preprocessing of the 3D point cloud data are performed to obtain the pure point cloud of the top surface of the rigid column of the material frame. S104: Based on the pure point cloud, the 6D fine pose homogeneous transformation matrix is ​​obtained by plane fitting, Euclidean clustering and coordinate system orthogonalization. S105: Based on the material frame type determined by TagID, combined with the 6D fine pose homogeneous transformation matrix and packing process parameters, the target pose of the packing placement point inside the material frame is obtained recursively through homogeneous coordinate transformation.

[0027] Specifically, in step S101, this embodiment uses an industrial-grade surface structured light 3D camera to obtain the camera calibration intrinsic parameters and establish the transformation relationship between the pixel coordinate system and the 3D camera coordinate system. The camera intrinsic parameters are represented in matrix form K, and their expression is: ; in, , These are the focal lengths along the X and Y axes of the image, respectively. Let X be the pixel coordinates of the principal point of the image along the X-axis in the pixel coordinate system. This represents the pixel coordinates of the principal point of the image along the Y-axis in the pixel coordinate system. In this embodiment, camera intrinsic parameter calibration and accuracy verification can be completed using a standard checkerboard calibration board to ensure the accuracy of pose calculation.

[0028] This invention constructs an embedded process database based on SQLite. The database uses the unique TagID of AprilTag as the primary key to build a main table of process parameters, which stores packing process parameters such as material frame type, geometric dimensions, layer spacing, slot spacing, tilt angle, and maximum capacity, enabling index matching and code-free rapid adaptation of material frames of various types.

[0029] Understandably, the embedded process database is a pre-built database. Each record in this database uses the unique AprilTagTagID as the primary key, which is associated with and stores the complete set of process parameters for that material frame. These parameters include, but are not limited to, the material frame type identifier (e.g., flat stacking, vertical slot type), external geometric dimensions (length, width, height), interlayer distance, slot spacing, slot tilt angle, maximum allowable stacking layers, and safety release distance. Through this database, once the material frame TagID is obtained, the corresponding process parameters can be retrieved, thereby enabling code-free, rapid adaptation and production changeover for material frames of different specifications.

[0030] In step S102: For situations where the material frame is randomly placed by an AGV or forklift, has significant pose deviations, and may deform, this invention proposes using AprilTag visual identifiers for 6D coarse pose calculation to quickly obtain the global prior pose of the material frame. Specifically, the process of AprilTag visual identifier recognition and 6D coarse pose calculation is as follows: Figure 2 As shown, it specifically includes: (1) Material frame mounting specifications: The AprilTag visual identifier uses the Tag25h9 coding family, with a physical size of 31.11mm × 31.11mm. It is rigidly mounted on the horizontal top surface of the reference column at the lower left corner of the material frame. The X-axis of the label is parallel to the short side of the material frame, and the positive Y-axis extends along the long side of the material frame to ensure that the mounting reference of different material frames is completely consistent. The mounting position and coordinate system direction are defined as follows: Figure 8 As shown.

[0031] (2) Image and Point Cloud Acquisition: 2D grayscale images and 3D point cloud data of the material frame are simultaneously acquired using an industrial-grade structured light 3D camera. The camera's working distance is 0.85m, and its field of view can completely cover the material frame's positional deviation of ±50mm. The acquired original 2D grayscale images are shown below. Figure 2 As shown in (a) above. In this embodiment, the acquisition command of the 3D camera can be triggered by devices with computing and communication capabilities, such as general-purpose desktop computers, laptops, and industrial PCs.

[0032] (3) AprilTag Decoding and Corner Extraction: First, the 2D grayscale image is locally binarized using an adaptive Gaussian thresholding algorithm to separate the black outer border of the tag from the background. The result is as follows: Figure 2 As shown in (b) of the figure; secondly, the quadrilateral contour is extracted and sub-pixel corner point optimization is performed to obtain the sub-pixel coordinates of the four corner points of the label. The extraction results are shown in the figure. Figure 2 As shown in (c); finally, orthorectification and decoding are performed on the label area, and false detections are filtered out by Hamming distance verification. When the minimum Hamming distance is less than the preset fault tolerance threshold, the decoding is deemed valid, and the unique label identifier of the material frame and the sub-pixel coordinates of the four corner points are output. The result is as follows. Figure 2 As shown in (d) in the figure.

[0033] (4) 6D coarse pose calculation: Combining the intrinsic coordinates of the 3D camera calibration with the corner coordinates of the tag, the 6D pose of the AprilTag relative to the 3D camera coordinate system is calculated using the IPPE algorithm to obtain the homogeneous transformation matrix, which is used as the 6D coarse pose homogeneous transformation matrix of the material frame. Simultaneously, using TagID as the primary key, all process parameters for the current material frame are retrieved from a pre-built embedded process database. Homogeneous transformation matrix. It is a 4×4 rigid transformation matrix, specifically in the form of: ; in, It is a 3×3 unit orthogonal rotation matrix, representing the spatial pose of the AprilTag tag relative to the 3D camera coordinate system; It is a 3×1 translation vector, representing the three-dimensional spatial coordinates of the AprilTag origin in the 3D camera coordinate system. The Z-axis component of the translation vector, i.e., the AprilTag identifier, is the depth reference value relative to the 3D camera, providing a spatial prior reference for subsequent dynamic pruning of point cloud ROIs.

[0034] In step S103, addressing the issue that the metal frame, with its highly reflective surface, easily produces voids, flying points, and noise in the 3D point cloud data acquired by the 3D camera, severely interfering with the stability of the point cloud processing algorithm, this invention proposes using the 6D coarse pose obtained in step S102 as prior information to perform dynamic ROI clipping and multi-level preprocessing on the 3D point cloud, thereby obtaining a clean point cloud of the column's top surface. A comparison of the effects of point cloud ROI clipping and preprocessing is shown in the figure below. Figure 3 As shown.

[0035] Specifically, firstly, the 6D coarse pose homogeneous transformation matrix obtained in step S102... Extracting the Z-axis translation (i.e., the depth value identified by the AprilTag) is used as the depth reference, and the pass-through filter interval is set to... ,in, δ This is the depth expansion margin (20mm in this embodiment). By performing a pass-through filter on the original 3D point cloud, invalid point clouds such as those on the ground and background are removed, retaining only the ROI region of the point cloud on the top surface of the upper edge of the material frame column. Figure 3 (a) in the image is the original global scene point cloud map, which contains approximately 880,000 points. Figure 3 (b) in the figure is the point cloud map after ROI pass-through filtering and cropping. After cropping, only about 5,500 points are retained, eliminating more than 99% of invalid data, which greatly reduces the computing power consumption of subsequent point cloud processing.

[0036] The multi-stage preprocessing includes voxel downsampling, statistical filtering for noise reduction, and normal consistency constraints, specifically including: (1) Voxel downsampling: A voxel grid filter with a resolution of 1.5 mm is used to downsample the ROI point cloud. The original point cloud is replaced by calculating the voxel center, which compresses the amount of data while preserving the geometric contour features.

[0037] (2) Statistical filtering for noise reduction: Set the number of nearest neighbors k=50 and the confidence coefficient α=1.0. Calculate the average distance from each point to its 50 nearest neighbors, and remove outliers whose average distance exceeds μ+ασ, where μ is the mean of the average distances of all points and σ is the standard deviation. Figure 3 As shown in (c), Figure 3 (c) in the figure is the point cloud map after voxel downsampling and statistical filtering for denoising.

[0038] (3) Normal Consistency Constraint: Calculate the local normal vector of the point cloud, compare it with the coarse pose Z-axis direction of AprilTag, and remove residual points on the sidewall of the material frame where the angle between the normal vectors is greater than 30°, finally obtaining the pure point cloud of the top surface of the four rigid columns of the material frame. Figure 3 As shown in (d) in the figure, Figure 3 (d) in the figure is the point cloud diagram of the top surface of the column after the normal consistency constraint.

[0039] In step S104, after completing multi-level preprocessing, considering the structural characteristics of the material frame being supported by four rigid columns with flat and coplanar top surfaces, this invention proposes to utilize this natural geometric datum. Based on the pure point cloud obtained in step S103, the 6D precise pose is calculated through plane fitting, Euclidean clustering, and coordinate system orthogonalization. The principle of the 6D precise pose calculation for the material frame coordinate system is as follows: Figure 4 As shown, the specific steps include the following: (1) Z-axis unit vector calculation: The reference plane at the upper edge of the material frame is fitted based on the RANSAC random sampling consensus algorithm to obtain the plane normal vector, which is then L2 normalized; the Z-axis direction of the coarse pose in AprilTag is used for verification. If the dot product of the two is negative, the normal vector is inverted to eliminate the ambiguity of the normal vector, and finally the Z-axis unit vector of the material frame coordinate system is obtained. This vector corresponds to Figure 4 In .

[0040] (2) Coordinate system origin locking: The point cloud subsets of a single column are separated by Euclidean clustering. The clustering distance threshold is set to 60mm and the minimum number of cluster points threshold is set to 100 to obtain 4 effective point cloud subsets corresponding to the top surface of the column. The geometric center of each subset is calculated as the reference point of the top surface of the column and named as spatial reference points. , , , Lock the reference point closest to the origin of the AprilTag coordinate system. As the origin of the material frame coordinate system That is, the translation vector of the frame coordinate system relative to the 3D camera coordinate system. .

[0041] (3) X-axis unit vector calculation: with the origin of the material frame coordinate system Starting from the reference point, connect the adjacent short-side columns. The initial direction vector of the X-axis is obtained; the projection component of the initial vector on the Z-axis is removed by Schmitt orthogonalization, and L2 normalization is performed to obtain the unit vector of the X-axis that is strictly perpendicular to the Z-axis. This vector corresponds to Figure 4 In axis.

[0042] (4) Y-axis unit vector calculation: The Y-axis unit vector is obtained by cross product of the Z-axis unit vector and the X-axis unit vector using the right-hand rule. This vector corresponds to Figure 4 In The axis, the formula is: ; (5) Construction of 6D fine pose matrix: Combining the three-axis unit vector and the origin translation vector obtained above, construct the 6D fine pose homogeneous transformation matrix of the frame coordinate system relative to the 3D camera coordinate system: ; Experimental results show that the maximum positioning error of the material frame coordinate system calculated in this step is controlled within 1.6mm, which meets the positional requirements of high-precision industrial packing.

[0043] In step S105, addressing the issue that the complex internal structure of the material frame (commonly flat stacked brackets and upright slot structures) can easily be obscured by upper or outer components from the camera's perspective, creating a visual blind spot and preventing traditional visual positioning methods from directly obtaining the pose information of the bottom or inner workstations, this invention proposes using the 6D precise pose calculated in step S104 as a benchmark, combined with geometric parameters from the embedded process database, to directly deduce the target pose of the packing placement point within the blind spot through homogeneous coordinate transformation.

[0044] Furthermore, since the packing process logic, pose parameter definitions, and recursion rules differ for different material frame types (such as flat stacking and upright slotted types), it is necessary to determine the material frame type based on the TagID obtained in step S102 and perform branching recursion. Specifically, the classification packing execution logic in this embodiment is as follows: Figure 5 As shown, the core principle is to use the unique tag number of the material frame obtained in step S102 as the classification basis to achieve branched pose calculation for two types of material frames: flat and upright. The specific recursive process is as follows: (1) Using the material frame coordinate system as a reference and combining the prior parameters in the process database, the packing placement point pose is recursively derived for different types of material frames. The recursive formula is as follows: ; in, The 6D fine pose homogeneous transformation matrix of the frame coordinate system relative to the 3D camera coordinate system obtained in step S104 is a unified reference for the pose transformation of the two types of frames and is independent of the frame type. This is the homogeneous transformation matrix of the box placement point in the material frame coordinate system, which is the target placement point pose matrix calculated and constructed for subsequent flat stacked material frames and vertical slotted material frames, respectively. The target pose of the box placement point relative to the 3D camera coordinate system.

[0045] (2) For example Figure 6 As shown, Figure 6 This is a schematic diagram of a flat, stacked material frame structure and its packing position parameters. Figure 6 (a) is a top view of a flat material frame. Figure 6 (b) is a side view of a flat material frame. Specifically, the recursive formula for the translation vector of the workpiece placement center in the material frame coordinate system of the i-th layer is: ; in, , This is the center offset of the bracket. The depth of the first-floor bracket. Interlayer spacing, For workpiece thickness, Release distance for safety.

[0046] Regarding attitude constraints, the target attitude of the material frame in the local coordinate system is set as: [reflection / perspective]. The axis yaw angle is 0°, around , The pitch and roll angles of the shaft are both 0°.

[0047] By combining the calculated center position and attitude constraints, the homogeneous transformation matrix of the pose of the target placement point of the flat material frame relative to the coordinate system of the material frame can be constructed. .

[0048] (3) such as Figure 7 As shown, Figure 7 This is a schematic diagram of the upright slot-type material frame structure and its packing position parameters. Figure 7 (a) is a top view of a vertically placed material frame. Figure 7 (b) is a side view of the vertically placed material frame. Specifically, the recursive formula for the translation vector of the workpiece placement center in the i-th slot in the material frame coordinate system is: ; in, This is the X-axis offset of the slot center. The Y-axis reference coordinate of the first slot, The slot spacing is... For safe insertion depth.

[0049] Regarding attitude constraints, a material wrapping frame needs to be constructed to ensure that the workpiece conforms to the slot tilt angle. Rotation matrix of axis : ; In the formula, The nominal tilt angle of the slot in the process database, i.e., the deviation of the workpiece center axis from the specified angle. The angle of direction.

[0050] By combining the calculated center position and rotation matrix, the homogeneous pose transformation matrix of the target placement point of the vertical placement frame relative to the frame coordinate system can be constructed. .

[0051] Using the above formula, the poses of all packing points within the visual blind zone of the material frame can be recursively derived, solving the problem of pose calculation blind zone caused by internal structural occlusion, and finally outputting the target pose of the packing points inside the material frame relative to the 3D camera coordinate system.

[0052] Example 2 This embodiment provides a material frame packing pose calculation system based on visual identification and point cloud fusion, including: The data acquisition module is configured to simultaneously acquire 2D grayscale images and 3D point cloud data of the material frame using a 3D camera. The coarse pose calculation module is configured to: decode the unique TagID of the material frame based on the AprilTag visual identifier in the 2D grayscale image and the intrinsic parameters of the 3D camera, and perform 6D coarse pose calculation to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; and retrieve the type of the corresponding material frame and the packing process parameters from the pre-built embedded process database based on the TagID. The point cloud preprocessing module is configured to: perform ROI dynamic clipping and multi-level preprocessing on the 3D point cloud data based on the 6D coarse pose homogeneous transformation matrix to obtain a clean point cloud of the top surface of the rigid column of the material frame. The fine pose calculation module is configured to: based on the clean point cloud, calculate the 6D fine pose homogeneous transformation matrix through plane fitting, Euclidean clustering and coordinate system orthogonalization. The target pose recursion module is configured to: based on the material frame type determined by TagID, combined with the 6D fine pose homogeneous transformation matrix and packing process parameters, recursively obtain the target pose of the packing placement point inside the material frame through homogeneous coordinate transformation.

[0053] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0054] Example 3 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds, characterized in that, include: The 2D grayscale image and 3D point cloud data of the material frame are acquired simultaneously using a 3D camera. Based on the AprilTag visual identifier in the 2D grayscale image, combined with the intrinsic parameters of the 3D camera, the unique TagID of the material frame is decoded, and 6D coarse pose calculation is performed to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; based on the TagID, the type of the corresponding material frame and the packing process parameters are retrieved from the pre-built embedded process database. Based on the 6D coarse pose homogeneous transformation matrix, the 3D point cloud data is subjected to ROI dynamic clipping and multi-level preprocessing to obtain a clean point cloud of the top surface of the rigid column of the material frame. Based on the pure point cloud, the 6D fine pose homogeneous transformation matrix is ​​obtained by plane fitting, Euclidean clustering and coordinate system orthogonalization. Based on the material frame type determined by the TagID, combined with the 6D fine pose homogeneous transformation matrix and the packing process parameters, the target pose of the packing placement point inside the material frame is obtained recursively through homogeneous coordinate transformation.

2. The method according to claim 1, characterized in that, The 6D coarse pose calculation specifically involves: The 2D grayscale image is subjected to adaptive binarization, quadrilateral contour extraction, and sub-pixel corner optimization to complete AprilTag decoding, thereby obtaining the unique TagID and corner coordinates of the material frame. Combined with the corner coordinates of the marker in the 3D camera, the 6D pose of the AprilTag visual marker relative to the 3D camera coordinate system is calculated using the IPPE algorithm to obtain the 6D coarse pose homogeneous transformation matrix.

3. The method according to claim 1, characterized in that, The embedded process database uses AprilTag's unique TagID as the primary key and stores at least one packing process parameter among the following: material frame type, geometric dimensions, layer spacing, slot spacing, tilt angle, and safety release distance. This is used to achieve index matching and rapid adaptation of material frames of various varieties.

4. The method according to claim 1, characterized in that, The ROI dynamic clipping specifically involves: a Z-axis translation using the 6D coarse pose homogeneous transformation matrix. For depth reference, set the pass-through filter interval to Where δ is the depth expansion margin, the invalid point cloud in the background is removed by direct filtering, and only the point cloud ROI area on the top surface of the column at the top edge of the material frame is retained.

5. The method according to claim 1, characterized in that, The multi-stage preprocessing includes voxel downsampling, statistical filtering for noise reduction, and normal consistency constraint.

6. The method according to claim 1, characterized in that, The formula for constructing the 6D fine pose homogeneous transformation matrix is ​​as follows: ； in, These are the unit vectors of the X, Y, and Z axes of the material frame coordinate system, respectively. is the translation vector of the origin of the material frame coordinate system relative to the 3D camera coordinate system.

7. The method according to claim 1, characterized in that, The specific steps for calculating the 6D precise pose homogeneous transformation matrix include: The reference plane at the upper edge of the material frame is fitted by the RANSAC algorithm to obtain the plane normal vector and perform L2 normalization. The dot product is then checked in conjunction with the Z-axis direction of the 6D coarse pose homogeneous transformation matrix. If the dot product is negative, the normal vector is inverted to eliminate the ambiguity of the normal vector and obtain the Z-axis unit vector of the material frame coordinate system. By separating the point cloud subsets of the top surfaces of each column of the material frame through Euclidean clustering, setting the clustering distance threshold and the minimum clustering point threshold, the effective point cloud subsets of each column top surface are obtained. The geometric center of each subset is calculated as the reference point of the column top surface. The reference point closest to the origin of the AprilTag coordinate system is locked as the origin of the material frame coordinate system, and the translation vector of the material frame coordinate system relative to the 3D camera coordinate system is obtained. Starting from the origin of the material frame coordinate system, the reference points of adjacent short side columns are connected to obtain the initial direction vector of the X-axis. The projection component of the initial direction vector on the Z-axis is removed by Schmidt orthogonalization and L2 normalization is performed to obtain the unit vector of the X-axis perpendicular to the Z-axis. The Y-axis unit vector is obtained by cross product of the Z-axis unit vector and the X-axis unit vector using the right-hand rule.

8. The method according to claim 1, characterized in that, The recursive formula for obtaining the target pose of the packing point inside the material frame through homogeneous coordinate transformation is as follows: ； in, This is the 6D fine pose homogeneous transformation matrix. Let be the homogeneous transformation matrix of the packing placement point in the material frame coordinate system. The target pose of the box placement point relative to the 3D camera coordinate system.

9. A material frame packing pose calculation system based on visual identification and point cloud fusion, characterized in that, include: The data acquisition module is configured to simultaneously acquire 2D grayscale images and 3D point cloud data of the material frame using a 3D camera. The coarse pose calculation module is configured to: decode the unique TagID of the material frame based on the AprilTag visual identifier in the 2D grayscale image and the intrinsic parameters of the 3D camera, and perform 6D coarse pose calculation to obtain the 6D coarse pose homogeneous transformation matrix of the AprilTag visual identifier relative to the 3D camera coordinate system; and retrieve the type and packing process parameters of the corresponding material frame from the pre-built embedded process database based on the TagID. The point cloud preprocessing module is configured to: perform ROI dynamic cropping and multi-level preprocessing on the three-dimensional point cloud data based on the 6D coarse pose homogeneous transformation matrix to obtain a clean point cloud of the top surface of the rigid column of the material frame. The fine pose calculation module is configured to: based on the pure point cloud, calculate the 6D fine pose homogeneous transformation matrix through plane fitting, Euclidean clustering and coordinate system orthogonalization. The target pose recursion module is configured to: based on the material frame type determined by the TagID, combined with the 6D fine pose homogeneous transformation matrix and the packing process parameters, recursively obtain the target pose of the packing placement point inside the material frame through homogeneous coordinate transformation.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method for calculating the pose of a packing box based on the fusion of visual identifiers and point clouds as described in any one of claims 1-8.

Images