A method and system for feature extraction of multi-layer stacked cartons based on 3D vision

Abstract

The application discloses a kind of multi-layer stacked carton feature extraction method and system based on 3D vision.The present application collects the original point cloud and RGB image of stacked scene in view of the problem that prior art cannot accurately quantify the abnormality of carton surface and multi-dimensional stacking characteristics in complex stacking scene;After preprocessing, construct height distribution histogram, and get initial point cloud cluster by clustering;By curvature analysis and normal vector consistency, remove warped abnormal points to obtain accurate upper surface point cloud;Get the length, width and normal vector direction of carton by principal component analysis;Combined with RGB image, extract basic geometric features, damage features, occlusion relationship, suspended / overlap features, inclination features and multi-layer interlaced features;Finally, the feature vector of each carton is output in structured form.The present application realizes the fine processing of point cloud, effectively removes the warped points, quantifies the complex stacking characteristics, fuses multi-modal data, and significantly improves the carton recognition and feature extraction capability in complex logistics scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and more particularly to a method and system for feature extraction of multi-layer stacked cardboard boxes based on 3D vision. This invention also relates to electronic devices and storage media for implementing the above method. Background Technology

[0002] Currently, automated identification technology for stacked cardboard boxes in scenarios such as warehousing and ports mainly relies on 3D vision systems to obtain the basic geometric information of the boxes. Although existing technical solutions have made some progress, they still have significant bottlenecks in recognition capabilities when dealing with increasingly complex logistics scenarios.

[0003] The following existing patented technologies are cited:

[0004] 1. CN118123802B (Method, system, device, electronic device and storage medium for robotic arm grasping): It uses 3D vision for object localization, but its technical solution only focuses on basic size and position recognition and does not involve the quantitative extraction of features such as surface damage of carton and complex stacking relationships (such as interlayer dependence and overlap stability).

[0005] 2. CN119503442B (Material grabbing method, apparatus, equipment and storage medium): A method for extracting geometric features of objects through 3D point clouds, but it is limited to extracting basic geometric features of objects, such as size and center coordinates, and does not design a dedicated feature quantification extraction algorithm for complex stacking relationships (such as upper carton overlapping, carton tilting and tilting, and multiple layers of crisscrossing).

[0006] Summary of core defects in existing technologies:

[0007] 1. Insufficient scene recognition capability: Existing 3D vision technology can only extract the basic geometric features of cardboard boxes. It lacks dedicated algorithms for multi-layer stacking scenarios and cannot effectively identify and quantify complex stacking features such as upper layer overhang (requiring identification of the number of overhang support points and overhang length), large-angle tilt (attitude angle 15-45°), and multi-layer interlacing (requiring marking of interlayer interference relationship and load-bearing strength). This results in the system's lack of "understanding" of complex scenes.

[0008] 2. Lack of surface anomaly handling capability: Existing technologies do not consider the impact of cardboard surface warping (dents, bulges, creases) on extraction during point cloud processing, and cannot identify and remove warped abnormal points, which may lead to subsequent extraction points being selected in abnormal areas, resulting in adsorption failure or cardboard damage.

[0009] 3. Single feature extraction dimension: Existing technologies mainly rely on point cloud data for geometric feature extraction, failing to fully utilize the synchronously acquired two-dimensional RGB image information, and thus cannot effectively identify issues such as damaged cardboard boxes and interference from printed patterns.

[0010] Therefore, this invention aims to provide a feature extraction method for multi-layer stacked cardboard boxes based on 3D vision. Through multi-level point cloud separation, refined point cloud optimization processing, and multi-modal data fusion, it achieves accurate quantitative extraction of multi-dimensional features (including basic geometric features, damage features, occlusion relationships, suspended / overlapping features, tilting features, and multi-layer interlacing features) of each independent cardboard box in complex stacking scenarios, providing a complete data foundation for subsequent extraction decisions. Summary of the Invention

[0011] To address the technical problem of the inability to accurately quantify surface anomalies and multidimensional stacking features of cardboard boxes in complex stacking scenarios, this invention provides a feature extraction method for multi-layer stacked cardboard boxes based on 3D vision.

[0012] Therefore, the first objective of this invention is to provide a method for feature extraction of multi-layer stacked cardboard boxes based on 3D vision, which includes the following steps:

[0013] Acquire raw point cloud data and 2D RGB images of the stacked scene;

[0014] The original point cloud data is preprocessed to obtain a sorted point cloud. Based on the sorted point cloud, the number of points in each height interval is counted to construct a height distribution histogram.

[0015] The sorted point cloud is divided into different height layers according to the trough positions of the height distribution histogram, and the point cloud of each height layer is obtained. Then, the sorted point cloud of each layer is clustered to identify the initial point cloud clusters representing individual cardboard boxes.

[0016] For each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud for each carton.

[0017] Based on the precise point cloud of the upper surface of each carton and the synchronously acquired two-dimensional RGB image, relevant features are quantitatively extracted and captured.

[0018] Based on the grasping-related features, a feature vector is formed for each carton.

[0019] Preferably, the method further includes: obtaining the known rated specifications of the carton to be identified, wherein the known rated specifications include at least one of rated length, rated width, rated height and flatness parameters;

[0020] When constructing the height distribution histogram, the width of the height interval is adaptively determined by combining the known rated height;

[0021] When dividing the sorted point cloud into different height layers, the known rated height constraint layer height segmentation threshold is combined;

[0022] When clustering the sorted point cloud at each layer, the clustering results are constrained by the known rated length and rated width.

[0023] When removing warped abnormal points, the curvature threshold is adaptively set in conjunction with the known flatness parameters.

[0024] Preferably, for each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud of each carton. This specifically includes:

[0025] For each point in the cluster and its neighborhood, fit a quadratic surface and calculate the average curvature of the point. If the curvature is greater than a preset threshold, it is identified as a warped point and removed, resulting in a point cloud after removing warped points.

[0026] Perform a normal vector consistency analysis on the point cloud after removing warped points, calculate the angle between the normal vector of each point and the average normal vector of its neighboring points, and remove points if the angle is greater than 15°, thus obtaining the accurate upper surface point cloud of each carton.

[0027] Principal component analysis was performed on the precise upper surface point cloud to obtain the length direction, width direction, and upper surface normal vector direction of the carton.

[0028] Preferably, the extraction of relevant features includes:

[0029] Based on the precise point cloud of the upper surface, basic geometric features are extracted through principal component analysis. The basic geometric features include: precise dimensions of the carton, center coordinates, attitude angle and tilt angle, as well as length direction, width direction and upper surface normal vector direction.

[0030] Based on the RGB image region corresponding to the cardboard box, the damaged area is identified and the damage features are extracted, including the damaged area and the scratch length.

[0031] By calculating the ratio of the actual point cloud number on the upper surface of the cardboard box to its theoretical point cloud number, it is determined whether the box is unobstructed or partially obstructed, and the obstruction ratio is output.

[0032] Based on the basic geometric features and point cloud layering results, complex stacking features are extracted, including suspended / overlapping features, tilted features, and multi-layered interlaced features.

[0033] Preferably, the suspended / overlapping feature is quantified by analyzing the number of support point clouds under the carton and the projected overlap length, specifically including:

[0034] Determine the projection area of ​​the cardboard box on the horizontal plane, and query the point cloud below the current cardboard box that is located within the projection area and whose height is lower than the current cardboard box;

[0035] Connectivity analysis is performed on the lower point cloud that meets the height condition to obtain support point clusters. The number of support points is the number of clusters.

[0036] Calculate the projected length of each support area along the current length of the carton, and take the maximum projected length as the overlap length.

[0037] Preferably, the tilting feature is quantified by analyzing the normal vector of the carton and the bottom contact support area, specifically including:

[0038] Based on the principal component analysis results of the point cloud on the upper surface of the cardboard box, the normal vector of the upper surface of the cardboard box is calculated, and the tilt angle around the X-axis and the tilt angle around the Y-axis are calculated according to the normal vector to obtain the total attitude angle;

[0039] Project the point cloud of the carton vertically onto its bottom plane to obtain the lower support point cloud within the bottom projection area. Perform convex hull calculation on the support point cloud to obtain the actual contact support area at the bottom of the carton, and calculate the contact support area.

[0040] Preferably, the multi-layered interlacing feature is quantified by analyzing the layer number of the carton, the interlayer projection overlap relationship, and the load-bearing strength of the lower layer of the carton, specifically including:

[0041] The layer number of each carton is determined based on the trough locations of the height distribution histogram;

[0042] Analyze the overlap between the projection area of ​​the current cardboard box on the horizontal plane and the projection area of ​​the adjacent cardboard box directly below. If it partially exceeds the projection area of ​​all the cardboard boxes below, calculate the proportion of the excess area and record the overlapping cardboard box number.

[0043] Measure the vertical spacing between layers, that is, the vertical distance between the bottom of the upper carton and the top of the lower carton.

[0044] The second objective of this invention is to provide a feature extraction system for multi-layer stacked cardboard boxes based on 3D vision, comprising:

[0045] The data acquisition module is used to acquire raw point cloud data and two-dimensional RGB images of the stacked scene;

[0046] The point cloud preprocessing module is used to preprocess the original point cloud, remove outliers, sort them by Z-axis height, and construct a height distribution histogram.

[0047] The point cloud hierarchical clustering module is used to divide the sorted point cloud into different height layers according to the trough position of the height distribution histogram, and to cluster the point cloud in each height layer to identify the initial point cloud clusters representing individual cardboard boxes.

[0048] The point cloud refinement module is used to remove warped outliers from each initial point cloud cluster through curvature analysis and optimize the carton boundary through normal vector consistency analysis to obtain the accurate top surface point cloud of each carton.

[0049] The feature extraction module is used to quantitatively extract relevant features based on the precise upper surface point cloud of each carton and the synchronously acquired two-dimensional RGB image.

[0050] The feature output module is used to structure and organize all the feature data extracted by quantization to form the feature vector of each carton.

[0051] A third objective of this invention is to provide an electronic device comprising:

[0052] A memory for storing computer programs, the memory including non-volatile memory and random access memory, the non-volatile memory for storing system programs and the random access memory for caching runtime data;

[0053] A processor is used to execute a program stored in memory to implement the steps of any of the above-mentioned methods for feature extraction of multi-layer stacked cartons based on 3D vision.

[0054] The fourth objective of this invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for feature extraction of multi-layer stacked cartons based on 3D vision.

[0055] Beneficial effects

[0056] The beneficial effects of this invention are as follows:

[0057] 1. Point cloud refinement capability: Through the multi-layer point cloud separation technology of "layer height threshold + connected component analysis", combined with curvature analysis and normal vector consistency optimization, it has achieved for the first time the accurate segmentation of the point cloud of each independent cardboard box in the multi-layer stacking scene, and can effectively remove warped abnormal points on the surface of the cardboard box, ensuring that the extracted upper surface point cloud is flat and complete, laying a high-quality data foundation for subsequent feature extraction.

[0058] 2. Quantification Capability of Complex Stacking Features: For the first time, a systematic quantification algorithm is proposed for complex stacking features such as upper layer suspension / overlap (number of support points ≥ 2, overlap length ≥ 1 / 3 of the side length), tilting angle ≤ 45°, and multi-layer staggered stacking. Through multi-dimensional parameters such as the number of support point clouds, projected overlap length, normal vector analysis, bottom contact support area, inter-layer projection overlap rate, and lower layer load-bearing strength, a precise quantitative description of complex spatial relationships is achieved, enabling the system to "understand" the complex spatial relationships between cartons.

[0059] 3. Multimodal feature fusion capability: By combining 3D point cloud data with 2D RGB images, and using a deep learning semantic segmentation model to perform pixel-level recognition of damaged areas in RGB images, the system can accurately extract key features such as damage and occlusion of cardboard boxes, making up for the shortcomings of pure point cloud analysis in dimensions such as surface texture and color.

[0060] 4. Scene coverage capability: This method is compatible with various cardboard boxes with a length and width of 30-80cm and a thickness of 5-20cm. It can effectively handle single and mixed complex scenes such as suspended, tilted (≤45°), and multi-layered overlapping, with a scene coverage rate of ≥98%. It fundamentally solves the limitation of existing technologies that can only handle scenes with regular stacking.

[0061] 5. Optional Prior Specification Enhancement: Known box specifications can be introduced as prior constraints to further improve the accuracy of layering, clustering, and warpage removal in complex scenarios with dense stacking and similar sizes, thereby enhancing the robustness and adaptability of the system. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0063] Figure 1 This is a flowchart illustrating a method for feature extraction of multi-layer stacked cardboard boxes based on 3D vision, according to an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram illustrating the specific implementation process of extracting and capturing relevant features based on accurate upper surface point cloud and RGB image quantization in an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram showing the visualization results of point cloud optimization before and after in an embodiment of the present invention, wherein... Figure 3 Image (a) in the image is the original RGB color point cloud image collected. Figure 3 (b) in the figure is a schematic diagram of the visualization result after segmenting a single cardboard box from the total point cloud.

[0066] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0068] Terminology Explanation

[0069] Warped points: These are points in 3D point cloud data that are height anomalies caused by localized deformations (such as dents, bulges, creases, and damage) on the surface of a cardboard box. These points deviate significantly from the ideal plane of the cardboard box surface, appearing as localized bulges or depressions in the point cloud model.

[0070] Specific definitions in this invention:

[0071] Detection method: Identification is performed by calculating the local curvature of the point cloud. When the curvature k of a point is greater than 0.3, it is determined to be a warped point.

[0072] Principal Component Analysis (PCA) is a commonly used data analysis method used to extract key features from high-dimensional data and reduce data dimensionality. Essentially, it identifies the main direction of data distribution.

[0073] PCA transforms the original data into a new orthogonal coordinate system through linear transformation, such that the variances of the data in the new coordinate system are arranged in descending order. These new coordinate axes are called principal components, where:

[0074] First principal component: The direction of maximum data variance;

[0075] Second principal component: The direction that is orthogonal to the first principal component and has the second largest variance;

[0076] The third principal component is the direction that is orthogonal to the first two and has the smallest variance.

[0077] In 3D point cloud processing, PCA can analyze a set of spatial points (xi, yi, zi) and calculate the eigenvalues ​​and eigenvectors of its covariance matrix. The three eigenvectors form a new orthogonal coordinate system, corresponding to the three principal directions of the point cloud; the magnitude of the eigenvalue indicates the degree of data dispersion in that direction.

[0078] Example 1

[0079] This embodiment provides a feature extraction method for multi-layer stacked cardboard boxes based on 3D vision. The invention uses an "ultra-high dynamic range 3D structured light camera" as its hardware foundation and "multi-layer point cloud separation and complex feature quantization extraction" as its core, constructing a complete technical chain from raw data to multi-dimensional features.

[0080] This invention provides a method for feature extraction from multi-layer stacked cardboard boxes based on 3D vision, such as... Figure 1 As shown, it includes the following steps:

[0081] First, the hardware system is deployed. The system includes: a 3D vision perception system consisting of an ultra-high dynamic range (HDR) structured light 3D main camera; and an industrial-grade PLC controller. All hardware is connected to the central control system via a bus.

[0082] Before the operation begins, the size range of the carton whose features are to be extracted is imported into the control system (in this embodiment, a length and width of 30-80cm and a thickness of 5-20cm are used as an example) for system parameter adaptation.

[0083] As an optional implementation, if the scenario is complex (such as dense stacking or similar sizes), the known rated specifications of each carton (including rated length, rated width, rated height, and flatness parameters) can be further imported as prior constraints to enhance recognition accuracy.

[0084] S1 acquires raw point cloud data and 2D RGB images of the stacked scene.

[0085] A 3D structured light main camera deployed on the flange at the end of the robotic arm triggers data acquisition when the robotic arm stops at a preset photographing pose. The camera projects a structured light pattern, calculates the depth of each pixel based on the phase of the reflected grating, and generates a point cloud. Simultaneously, a built-in RGB camera is triggered to capture a two-dimensional RGB image, denoted as I_rgb. Both are time-aligned via hardware trigger signals, and pixel coordinates and point cloud coordinates are associated through a pre-calibrated extrinsic parameter matrix. In this embodiment, the robotic arm moves to a position 1.5m directly in front of the container door, at a height of 1.2m, with the camera's optical axis pointing horizontally towards the internal stacking area. One acquisition is triggered, obtaining a frame of two-dimensional RGB image and the corresponding raw point cloud data. The image depth resolution can reach up to 1280×800 pixels, corresponding to 1280×800=1024000 points / frame in raw point cloud data, covering an area of ​​2.5m×2m. The scene contains three layers of stacked cardboard boxes: the top layer has two boxes (A and B) that are tilted (at an angle of approximately 25° and 30°), the middle layer has two boxes (C and D) stacked regularly, and the bottom layer has three boxes (E, F, and G) arranged regularly. All the boxes are made of kraft paper and have no damage to their surfaces.

[0086] As an optional implementation, while acquiring the raw point cloud data and 2D RGB image of the stacked scene, the known rated specifications of the cardboard box to be identified are obtained. The known rated specifications include at least one of the following parameters: rated length, rated width, rated height, and flatness. The purpose of each parameter is as follows:

[0087] When constructing the height distribution histogram, the width of the height interval is adaptively determined by combining the known rated height;

[0088] When dividing the sorted point cloud into different height layers, the known rated height constraint layer height segmentation threshold is combined;

[0089] When clustering the sorted point cloud at each layer, the clustering results are constrained by the known rated length and rated width.

[0090] When removing warped abnormal points, the curvature threshold is adaptively set in conjunction with the known flatness parameters.

[0091] S2, preprocess the original point cloud data to obtain sorted point clouds, and count the number of points in each height interval based on the sorted point clouds to construct a height distribution histogram.

[0092] The raw point cloud data obtained from step S1 is denoted as P_raw. In this embodiment, the point cloud density generated by the 3D structured light main camera in a single acquisition is 1,024,000 points / frame, covering an area of ​​2.5m×2m. Each point has three-dimensional coordinates (x,y,z) and reflection intensity information.

[0093] For complex scenarios involving multiple stacked and crisscrossed layers within shipping containers, this method employs a dual algorithm combining layer height thresholding and connected component analysis. This is achieved by integrating point cloud curvature analysis and normal vector consistency optimization to separate multi-layered point clouds. Furthermore, relevant features for each carton are quantized and extracted using simultaneously acquired 2D RGB images. The specific implementation process is as follows:

[0094] S21, Perform statistical filtering on the original point cloud P_raw.

[0095] Outlier removal: For each point, if there are fewer than 10 points in its neighborhood of radius r, it is identified as an outlier and removed, resulting in the filtered point cloud P_filtered. The purpose of outlier removal is to eliminate isolated points caused by sensor noise or environmental interference, preventing them from interfering with subsequent clustering and feature extraction.

[0096] S22, sort P_filtered according to the Z-axis coordinate values ​​from smallest to largest to obtain the sorted point set P_sorted.

[0097] Using the Z-coordinate value of each point as the sorting key, all points in P_filtered are sorted in ascending order to obtain an ordered point set P_sorted. Sort by the Z-axis to organize the point cloud vertically, facilitating subsequent height layer segmentation and histogram statistics based on the Z-coordinate values.

[0098] S23, Calculate the height distribution histogram of the filtered point cloud P_filtered.

[0099] The number of points within each height interval is counted with a width of 5mm. The height distribution histogram can intuitively reflect the distribution of the point cloud in the vertical direction: when there are multiple stacks, a large number of point clouds will accumulate on the upper surface of each layer of the carton, forming the peak area in the histogram; while the point cloud is sparse in the gaps between layers, forming the trough area. Based on the trough position of the histogram, the set of layer height segmentation thresholds H_thresholds={h1,h2, ...,hk} can be initially determined, where k is the number of layers minus 1. In this embodiment, approximately 950,000 points remain in the filtered point cloud. The height distribution histogram shows three obvious peak areas, corresponding to: 0–15cm (lower layer), 15–30cm (middle layer), and 30–45cm (upper layer). The troughs are located at 15cm and 30cm. Based on this, the layer height threshold is set to H_thresholds={0.15m,0.30m}, and the sorted point cloud is divided into three height layers: L1 (z≤0.15m), L2 (z≤0.15m), L3 (z≤0.30m), L4 (z≤0.15m), L5 (z≤0.15m), L6 (z≤0.15m), L7 (z≤0.15m), L8 (z≤0.15m), L9 (z≤0.15m), L10 (z≤0.15m), L11 (z≤0.15m), L22 (z≤0.15m), L12 (z≤0.15m), L13 (z≤0.15m), L14 (z≤0.15m), L15 (z≤0.15m), L12 (z≤0.1<z≤0.30m)、L3(z> 0.30m).

[0100] As an optional implementation, if the known rated height has been imported, the height distribution histogram can be constructed by adaptively determining the height interval width (for example, setting the interval width to the smaller of H0 / 10 and a fixed value of 5mm) so that the trough position of the histogram more accurately reflects the real interlayer boundary, which is especially suitable for scenarios where the thickness of the carton changes little.

[0101] S3, the sorted point cloud is divided into different height layers according to the trough position of the height distribution histogram to obtain the point cloud of each height layer; and the sorted point cloud of each layer is clustered to identify the initial point cloud cluster representing an independent cardboard box.

[0102] S31, using the trough position of the height distribution histogram as the layer height segmentation threshold, the sorted point cloud is divided into height layers corresponding to different stacking layers.

[0103] Based on the height distribution histogram constructed in step S2, the valley positions are used as the layer height segmentation thresholds to initially divide the sorted point cloud into height layers corresponding to different stacking layers, thus obtaining the point cloud of each height layer.

[0104] In this embodiment, the wave valleys are located at 15 cm and 30 cm. Accordingly, the sorted point cloud is divided into a lower layer L1 (z ≤ 0.15 m), a middle layer L2 (0.15 m < z ≤ 0.30 m), and an upper layer L3 (z > 0.30 m).

[0105] As an optional implementation, if the known rated height has been imported, when determining the floor height segmentation threshold, the known rated height can be combined for threshold verification and correction (for example, the difference between adjacent floor heights should be greater than the known rated height / 2 and less than 2 times the known rated height, otherwise merge or adjust) to avoid incorrect stratification caused by noise or local point cloud loss.

[0106] S32. For the point cloud of each height layer, use a clustering algorithm based on Euclidean distance, with the neighborhood radius less than the minimum gap between adjacent cartons and the number of neighborhood points estimated based on point cloud density and not less than the preset density threshold as parameters to identify the initial point cloud clusters representing independent cartons, and calculate the minimum oriented bounding box and normal vector of each point cloud cluster.

[0107] After completing the height layer division, for the point cloud Lj (j = 1, 2, 3) of each height layer, use a clustering algorithm based on Euclidean distance (DBSCAN) to identify the initial point cloud clusters representing independent cartons. The DBSCAN algorithm is a density-based spatial clustering algorithm that can discover clusters of arbitrary shapes and effectively handle noise points. In this embodiment, the algorithm parameters are set as follows:

[0108] Neighborhood radius ε: The value should be less than the minimum gap between adjacent cartons (the minimum gap in this embodiment is 0.5 cm) to ensure that the point clouds of different cartons are not merged; at the same time, this value should be large enough so that the point clouds inside the same carton can form a connected cluster. In this embodiment, ε = 3 cm, which meets the above requirements.

[0109] Minimum number of neighborhood points MinPts: Determined based on point cloud density. In this embodiment, the point cloud density is approximately 204,800 points / m². In a circular area with a diameter of 3 cm (area approximately 7 cm 2 ), the theoretical number of points is 7×10 -4 m 2 ×204,800 points / m 2 ≈143 points. Take MinPts = 50 to ensure that only dense point cloud regions of a certain scale are identified as valid clusters and avoid misidentifying a small number of noise points as cartons.

[0110] For each identified point cloud cluster, calculate its minimum oriented bounding box (OBB) and normal vector as a preliminary carton candidate. The minimum oriented bounding box can more accurately describe the boundary and orientation of the carton and is more suitable for tilted cartons than the axis-aligned bounding box (AABB).

[0111] In this embodiment, DBSCAN clustering is performed on the already layered point cloud, including:

[0112] DBSCAN clustering was performed on the lower point cloud L1 to obtain three point cloud clusters, corresponding to the cardboard boxes E, F and G respectively. Each cluster has approximately 300,000 to 350,000 points, and the normal vectors are all vertically upward with an attitude angle of 0°.

[0113] Clustering the middle-layer point cloud L2 yields two point cloud clusters, corresponding to cardboard boxes C and D, with normal vectors pointing vertically upwards and an attitude angle of 0°.

[0114] Clustering the upper point cloud L3 yields two point cloud clusters, corresponding to boxes A and B. Box A measures approximately 50.4cm × 40.3cm × 12.2cm, with a center at (1.25m, 0.8m, 0.38m) and a normal vector at (0.15, 0, 0.99). Box B measures approximately 50.3cm × 40.1cm × 12.1cm, with a center at (1.28m, 0.83m, 0.41m) and a normal vector at (0.18, 0, 0.98). Since the upper box may be tilted, its precise spatial pose will be calculated in subsequent steps.

[0115] As an optional implementation, if the known rated length and rated width have been imported, the clustering results can be constrained by the known rated length and rated width during clustering identification (such as removing invalid clusters whose OBB size deviates from the rated specifications by more than ±20%), effectively preventing missegmentation caused by carton adhesion or point cloud noise.

[0116] S4. For each initial point cloud cluster, remove warped outliers and analyze and optimize the carton boundary to obtain the accurate upper surface point cloud of each carton.

[0117] For each initial point cloud cluster identified in step S3, further refinement processing is performed to remove surface anomalies and optimize boundaries, ultimately obtaining a precise point cloud of the cardboard box's upper surface. This step is one of the key innovations of this invention, distinguishing it from existing technologies, and effectively addresses the impact of local deformations on the cardboard box surface (such as dents, bulges, and creases) on subsequent extraction. Specifically, it includes the following sub-steps:

[0118] S41, fit a quadratic surface to each point and its neighborhood in the initial point cloud cluster, calculate the average curvature of the point, and if the curvature is greater than a preset threshold, it is determined to be a warped point and removed, thus obtaining the point cloud after removing the warped points.

[0119] Fit a quadratic surface to each point and its neighborhood (radius 1 cm) within the initial point cloud cluster:

[0120] z=ax 2 +by 2 +cxy+dx+ey+f

[0121] The surface parameters a, b, c, d, e, and f are solved using the least squares method. The average curvature k at each point is calculated based on the fitting results. A curvature threshold of 0.3 is set. If the curvature k > 0.3 at a point, it is identified as a warped point (corresponding to local deformation areas such as dents, bulges, or creases on the cardboard surface) and is removed. After removing warped points, the minimum directed bounding box (OBB) and normal vector are recalculated for the remaining point cloud, resulting in a more accurate boundary.

[0122] S42, Perform normal vector consistency analysis on the point cloud after removing warped points, calculate the angle between the normal vector of each point and the average normal vector of its neighboring points, and remove the point if the angle is greater than a preset angle threshold, to obtain the accurate upper surface point cloud of each carton.

[0123] For each cardboard box, after removing warped points, a normal vector consistency analysis is performed on the point cloud: the angle between the normal vector of each point and the average normal vector of its neighboring points is calculated. If the angle is greater than a preset angle threshold, the point is determined to belong to a different surface (such as the side of the cardboard box or other environmental interference) and is removed. In this embodiment, the preset angle threshold is set to 15°.

[0124] After the above two steps, the remaining point cloud is the precise point cloud of the upper surface of the carton.

[0125] In this embodiment, after the above treatment is applied to the clusters of cartons A and B, the proportion of warping points is extremely small (less than 0.1%), and the stability of the normal vector is significantly improved.

[0126] As an optional implementation, if known flatness parameters have been imported, the curvature threshold can be adaptively set according to the flatness parameters when removing warp points in curvature analysis (e.g., a curvature threshold of 0.3 is used for new boxes with good flatness, and the threshold is appropriately reduced for old boxes that may be deformed), making the removal of warp points more accurate and adaptable to different carton surface conditions.

[0127] S5. Based on the precise point cloud of the upper surface of each carton and the synchronously acquired two-dimensional RGB image, the relevant features are quantitatively extracted and captured.

[0128] Specifically, this step is implemented sequentially through the following sub-steps: First, the precise upper surface point cloud is registered with the RGB image to obtain the image mask corresponding to the cardboard box; then, the point cloud boundaries are corrected by backprojection using the image segmentation results, and basic geometric features are extracted based on the corrected point cloud; simultaneously, damaged areas are identified by combining the RGB image, the occlusion ratio is calculated, and complex stacking features are quantified based on the point cloud layering results. The specific implementation methods of each sub-step are as follows, such as... Figure 2 As shown:

[0129] S51, the precise upper surface point cloud of each carton is projected onto a two-dimensional image plane, and a binary mask corresponding to each carton is generated using the synchronously acquired RGB image.

[0130] The precise upper surface point cloud of each carton obtained in step S4 is projected onto a two-dimensional image plane, and the RGB image I_rgb acquired in step S1 is used for region focusing. The specific steps are as follows:

[0131] First, a mapping relationship between 3D point cloud coordinates and 2D image pixel coordinates is established using pre-calibrated camera intrinsic and extrinsic parameters. The intrinsic parameter matrix describes the camera's internal optical parameters such as focal length and principal point coordinates, while the extrinsic parameter matrix describes the camera's pose in the world coordinate system.

[0132] Then, based on the calibration parameters, the 3D coordinates (x, y, z) of the point cloud of each cardboard box are mapped to the image pixel coordinates (u, v), generating a binary mask mask_i for each cardboard box. This mask identifies the corresponding region of the point cloud on the upper surface of the cardboard box in the RGB image.

[0133] S52, GrabCut segmentation is performed on the RGB image based on the binary mask, and the boundary of the precise upper surface point cloud is corrected by back projection to obtain the precise upper surface point cloud.

[0134] For each RGB image region corresponding to mask_i, the GrabCut algorithm is used for accurate segmentation. The GrabCut algorithm is an interactive image segmentation algorithm based on graph cut. It can automatically iteratively optimize the segmentation boundary based on the statistical model of the foreground (top surface of the cardboard box) and the background (surrounding environment) to obtain a finer outline of the top surface of the cardboard box than the initial projection mask.

[0135] Based on the GrabCut segmentation results, the boundary of the precise upper surface point cloud is corrected by back projection, and edge noise that does not belong to the upper surface of the carton in the original point cloud is removed to obtain the precise upper surface point cloud.

[0136] S53, Based on the precise upper surface point cloud, basic geometric features are extracted through principal component analysis. The basic geometric features include: precise dimensions of the carton, center coordinates, attitude angle and tilt angle, as well as length direction, width direction and upper surface normal vector direction.

[0137] The basic geometric features include: precise dimensions of the carton (length L, width W, thickness H), center coordinates (x_c, y_c, z_c), attitude angles (yaw angle, pitch angle, roll angle), and tilt angle. The extraction steps are as follows:

[0138] 1. Principal Component Analysis (PCA): Calculate the centroid coordinates of the precise upper surface point cloud, i.e., use the average coordinates of all points as the center point of the precise upper surface point cloud.

[0139] Secondly, a covariance matrix is ​​constructed with the centroid coordinates as the origin. The covariance matrix describes the dispersion of the precise upper surface point cloud in various directions, and its elements are obtained by calculating the covariance of each dimension coordinate.

[0140] Then, the covariance matrix is ​​decomposed using eigenvalues ​​to solve for its eigenvalues ​​and corresponding eigenvectors. The magnitude of the eigenvalues ​​reflects the degree of dispersion of the precise upper surface point cloud along the corresponding eigenvector directions.

[0141] Finally, the feature vectors are arranged in descending order of eigenvalues, corresponding to the first, second, and third principal components of the precise upper surface point cloud, respectively. The first principal component corresponds to the direction of maximum variance in the point cloud, i.e., the length direction of the cardboard box; the second principal component corresponds to the width direction; and the third principal component corresponds to the normal vector direction of the upper surface.

[0142] In this embodiment, the principal component analysis results show that the first principal component corresponds to the length direction of the carton (along the X-axis), the second principal component corresponds to the width direction (along the Y-axis), and the third principal component corresponds to the direction of the normal vector of the upper surface. The calculated normal vector of carton A is n_A=(0.315,0,0.97), and the corresponding tilt angle about the Y-axis is Φy=arctan(0.315 / 0.97)≈18°, while the tilt angle about the X-axis is negligible. Similarly, PCA analysis of carton B yields its normal vector n_B=(0.27,0,0.96), and the corresponding tilt angle about the Y-axis is Φy≈15.7°. It is confirmed that the tilt direction of both carton A and B is in the positive direction about the Y-axis (i.e., tilted towards the camera).

[0143] The result of this step is as follows: Figure 3 As shown. Among them, Figure 3 (a) in the image is the original RGB color point cloud image collected; Figure 3 (b) in the diagram is a schematic representation of the visualization result after segmenting a single cardboard box from the total point cloud. Figure 3 In (b), the upper surface of the segmented cardboard box is precisely outlined with a blue border, the four corner points are marked with red dots, and the geometric center is marked with a green dot. The overall outline of the cardboard box is defined by a yellow minimum directed bounding box rectangle, thus fully representing the boundary, key points and spatial pose of the cardboard box.

[0144] Calculate dimensions (L, W, H): Construct minimum directed bounding boxes (OBBs) along the three principal directions. Project the point cloud onto each principal direction axis, take the minimum and maximum values ​​of the projection, and use the difference between the maximum and minimum values ​​as the side length of that direction to obtain the length L, width W, and thickness H of the carton.

[0145] Calculate the center coordinates (x_c, y_c, z_c): Take the center points of the three main directions of the span as the center coordinates of the cardboard box.

[0146] Calculate the attitude angle and tilt angle: Using the camera coordinate system as a reference, convert the three principal directions into Euler angles around the camera axis (yaw angle around the Z-axis, pitch angle around the Y-axis, and roll angle around the X-axis). The angle between the normal vector of the upper surface and the vertical direction is the tilt angle of the carton.

[0147] The final output includes the following basic geometric features: precise dimensions of the carton (L, W, H), center coordinates (x_c, y_c, z_c), attitude angles (yaw angle, pitch angle, roll angle), tilt angle, as well as the length direction, width direction, and the direction of the normal vector of the upper surface.

[0148] S54, based on the RGB image area corresponding to the cardboard box, identify the damaged area and extract the damage features, the damage features including the damaged area and the scratch length.

[0149] The two-dimensional RGB image region corresponding to the cardboard box is input into a pre-trained deep learning semantic segmentation model to identify the damaged area.

[0150] The RGB image region corresponding to the cardboard box is input into a pre-trained deep learning semantic segmentation model (such as U-Net). The deep learning semantic segmentation model is trained on a large number of cardboard box images with damage annotations and can identify damaged areas (dents, scratches, damage, etc.) at the pixel level.

[0151] The deep learning semantic segmentation model outputs a probability map of each pixel belonging to a damaged region. After thresholding (probability > 0.5), a binary mask of the damaged region is obtained. The area and length of the connected components in the mask are calculated as follows:

[0152] 1. Area: The total number of pixels in a connected region is multiplied by the actual area represented by a single pixel.

[0153] 2. Length: Calculates the length of the minimum bounding rectangle or skeleton of the connected component.

[0154] The final output includes the quantized data of the damaged area, including: the binary mask of the damaged area, its area, and its length.

[0155] If the area of ​​any connected region is ≥800mm² or the length of any linear scratch is ≥80mm, then the carton is determined to be unremovable.

[0156] In this embodiment, all RGB images of the cartons were found to have no damaged areas after model detection, and were therefore determined to be extractable.

[0157] S55 determines whether the surface of the cardboard box is unobstructed or partially obstructed by calculating the ratio of the actual point cloud quantity to the theoretical point cloud quantity, and outputs the obstruction ratio.

[0158] By calculating the ratio of the actual point cloud count to the theoretical point cloud count on the upper surface of the cardboard box, it can be determined whether the surface is unobstructed or partially obstructed.

[0159] By calculating the proportion of point cloud on the surface of each cardboard box to the theoretically complete surface number of points, the occlusion ratio is output for subsequent decision-making. If the occlusion ratio is less than 50%, it is considered partial occlusion; otherwise, it is considered unoccluded. The specific calculation method is as follows:

[0160] Based on the precise dimensions of the cardboard box (length L, width W) obtained in step S53, and the point cloud density ρ (unit: points / m) of the 3D structured light camera. 2 The theoretical number of dots on the top surface of the cardboard box under unobstructed conditions can be calculated:

[0161] Ntheory = L×W×ρ

[0162] The point cloud density ρ can be obtained through calibration. The calibration method is to collect point clouds on a standard plane and count the number of points per unit area.

[0163] In this embodiment, the total number of point clouds collected in a single frame in step S1 is 1,024,000 points, covering an area of ​​2.5m × 2m = 5m. 2 Therefore, the point cloud density ρ = 1,024,000 / 5 = 204,800 points / m 2 .

[0164] The actual number of points, Nactual, is the number of points contained in the precise point cloud on the upper surface of the carton obtained after step S4 (i.e., the number of point cloud clusters after removing warped points and optimizing for normal vector consistency). The occlusion ratio is calculated using the following formula:

[0165] Ratio = Nactual / Ntheory

[0166] It should be noted that, due to the potential tilt of the cardboard box surface, the actual surface area is slightly larger than the horizontal projected area. Therefore, the actual number of point clouds collected on the upper surface may be slightly higher than the theoretical number of horizontal projected points, but it usually will not exceed 1.1 times the theoretical value. The occlusion ratio reflects the degree of exposure of the upper surface of the cardboard box; a higher ratio indicates less occlusion.

[0167] Judgment criteria: If the proportion is less than 50%, it is judged as "partial occlusion"; otherwise, it is "unoccluded".

[0168] In this embodiment, taking carton A as an example: its dimensions are L = 0.5 m, W = 0.4 m, and the theoretical point cloud count Ntheory = 0.5 × 0.4 × 204,800 = 40,960 points. The actual precise point cloud count obtained after step S4 on the upper surface of carton A is 42,800 points, a ratio of approximately 42,800 / 40,960 ≈ 1.045 = 104.5%, which is greater than 50%, therefore it is determined to be unobstructed. Similarly, the actual point cloud count ratios of other cartons are all greater than 50%, and all are determined to be unobstructed.

[0169] S56. Based on the basic geometric features and the point clouds of each height layer, extract complex stacking features, including suspended / overlapping features, tilted features and multi-layered interlacing features.

[0170] Specifically, the quantitative calculation methods for each complex stacking feature are as follows: the suspended / overlapping feature is quantified by analyzing the number of support point clouds under the carton and the projected overlap length; the tilting feature is quantified by analyzing the normal vector of the carton and the bottom contact support area; the multi-layer staggered feature is quantified by analyzing the layer number of the carton, the interlayer projection overlap relationship and the load-bearing strength of the lower layer carton.

[0171] S561, the suspended / overlapping feature is quantified by analyzing the number of support point clouds under the carton and the projected overlap length.

[0172] For the upper cardboard box, its suspension status is determined by analyzing whether there are supporting points underneath. The calculation parameters require the precise dimensions, center coordinates, and length direction of the cardboard box from step S53. The specific calculation steps are as follows:

[0173] 1. Determine the projection area of ​​the cardboard box on the horizontal plane. Query the point cloud located within this projection area and whose height is lower than the current cardboard box. Determine the projection area of ​​the cardboard box to be measured on the horizontal plane (XY plane), denoted as Rproj. The projection area is defined by the circumscribed rectangle of the cardboard box's smallest directed bounding box on the XY plane. The projection area can be represented as:

[0174] Rproj=[x_min,x_max]×[y_min,y_max]

[0175] Where x_min, x_max, y_min, and y_max are the minimum and maximum values ​​of the smallest directed bounding box of the carton on the X-axis and Y-axis, respectively.

[0176] 2. Perform connectivity analysis on the lower point cloud that meets the height condition to obtain support point clusters. The number of support points is the number of clusters.

[0177] Query the point clouds of other layers (i.e., the lower point clouds) located within the projection area and whose height is lower than the current cardboard box, and denote them as Pbelow. For each point in Pbelow, calculate the difference Δh between its height and the current bottom height of the cardboard box (obtained by subtracting half the thickness from the center height of the cardboard box). If Δh is less than the thickness t of the current cardboard box (i.e., the vertical distance between the support point and the bottom of the cardboard box is less than one box thickness), then the point is considered to be a potential support point.

[0178] 3. Calculate the projected length of each support area along the current length of the carton, and take the maximum projected length as the overlap length.

[0179] To identify independent support regions, a connected component analysis (e.g., using DBSCAN clustering with a neighborhood radius of 3cm) is performed on the point cloud that meets the above height conditions, resulting in multiple support point clusters, each representing an independent support region. The number of support points is the number of clusters, Nsupport. For each support region, its projected length, Lovelap, along the current length direction of the carton (i.e., the main direction of the carton) is calculated, and the largest projected length among all support regions is taken as the overlap length. The formula for calculating the overlap length is:

[0180] Loverlap=max_i(proj_length(support_i,direction))

[0181] Where proj_length(support_i,direction) is the projected length of the i-th support region along the length direction of the carton.

[0182] The output of this step includes: the number of support points Nsupport, the overlap length Loverlap, and quantitative data on the distribution of support areas.

[0183] It is important to note that the overhang / overlap feature only applies to upper-layer cartons (i.e., cartons with other cartons below them and not entirely on the lower support surface). For middle and lower-layer cartons, since they are located at the bottom or have complete lower support, there is no risk of overhang or overlap, so this calculation is not performed, and the output is empty. An empty output only indicates that the feature is not applicable to the current carton, not that the carton is unextractable; the feasibility of extracting the carton will be determined by the subsequent decision module based on other features (such as basic geometry, occlusion, multi-layer overlap, etc.).

[0184] In this embodiment:

[0185] Cardboard box A: Projected area X∈[1.0m,1.5m], Y∈[0.6m,1.0m], points in the point cloud of the lower middle cardboard box C that meet the conditions are obtained into 2 independent clusters through connected component analysis, Nsupport=2, and the maximum overlap length Lovelap=36cm;

[0186] Cardboard box B: Projection area X∈[1.0m,1.5m], Y∈[1.02m,1.42m], the points in the point cloud of the lower middle cardboard box D that meet the conditions are analyzed to obtain 2 independent clusters, Nsupport=2, and the maximum overlap length Loverlap=38cm;

[0187] If the middle layer cartons C and D, and the lower layer cartons E, F, and G are located at the bottom layer or have no cartons below them, this judgment is not performed, and the output is empty.

[0188] S562, the tilt feature is quantified by analyzing the normal vector of the carton and the bottom contact support area.

[0189] For a cardboard box detected as tilted (in this embodiment, the attitude angle is greater than 5°), its tilt degree, direction, and stability are quantitatively described by analyzing its point cloud normal vector distribution and bottom support. The calculation parameters require the upper surface normal vector from step S53; the specific calculation steps are as follows:

[0190] 1. Calculate the attitude angle of the carton based on the direction of the normal vector of the upper surface.

[0191] Based on the normal vector n=(nx,ny,nz) of the upper surface of the carton obtained through principal component analysis (PCA) in step 53, the tilt angle of the carton is calculated. The tilt of the carton can be decomposed into tilt angles about the X-axis and about the Y-axis:

[0192] The tilt angle about the X-axis is Φx = arctan(ny / nz), which represents the tilt angle of the carton in the Y direction;

[0193] The tilt angle about the Y-axis is Φy = arctan(nx / nz), which represents the tilt angle of the carton in the X direction.

[0194] Total attitude angle Φ = √(Φx) 2 +Φy 2 The direction of inclination is determined by the projection direction of the normal vector onto the horizontal plane, that is, the direction of the vector (nx, ny).

[0195] 2. Project the point cloud of the carton vertically onto its bottom plane to obtain the lower support point cloud in the bottom projection area. Perform convex hull calculation on the support point cloud to obtain the actual contact support area at the bottom of the carton, and calculate the contact support area.

[0196] To evaluate the extraction stability of a tilted cardboard box, it is necessary to determine the contact support range between its bottom and the underlying support. The point cloud of the cardboard box is projected vertically onto its bottom plane (determined by subtracting half the thickness from the center height of the box), obtaining the bottom projection area Rbottom. Simultaneously, the lower support point cloud Psupport, located within this projection area and near the bottom of the box (height difference less than 2 cm), is obtained. Convex hull calculation is performed on Psupport to obtain the minimum convex polygon of the support point cloud; this minimum convex polygon represents the actual contact support area at the bottom of the box. The contact support area Asupport of this minimum convex polygon and its extension range in the length and width directions of the cardboard box are calculated.

[0197] Calculate the contact support area Asupport of the smallest convex polygon and its extension along the length and width directions of the carton. The ratio of the contact support area to the theoretical bottom area of ​​the carton is:

[0198] Support area ratio = Asupport / (L × W)

[0199] The output of this step includes: attitude angle Φ, tilt direction, contact support area Asupport, and quantitative data of the support area ratio.

[0200] In this embodiment:

[0201] Cardboard box A: Normal vector n=(0.23,0,0.97), Φy=18°, Φx=0°, total attitude angle 18°, tilt direction along the positive X-axis, bottom contact support area Asupport≈0.18m² 2 The supporting area ratio is 0.18 / (0.5×0.4) = 90%.

[0202] Cardboard box B: Normal vector n=(0.27,0,0.96), Φy=15.7°, total attitude angle 15.7°, tilt direction along the positive X-axis, bottom contact support area Asupport≈0.19m² 2 The supporting area ratio is 0.19 / (0.5×0.4) = 95%.

[0203] The cardboard box's C, D, E, F, and G attitude angles are 0°, and the output attitude angle is 0°.

[0204] S563, the multi-layer interlacing characteristics are quantified by analyzing the layer number of the carton, the overlapping relationship of the interlayer projection, and the load-bearing strength of the lower layer of the carton.

[0205] For each carton in the multi-layer stacking scenario, quantify the complexity of the multi-layer stacking by analyzing its layer number, the projection relationship with adjacent cartons on the same layer, and the bearing capacity of the cartons on the lower layer. Its calculation parameters need to come from the center height and thickness (exact carton dimensions and center coordinates) in step S53. The specific calculation steps are as follows:

[0206] 1. Determine the layer number of each carton based on the trough position of the height distribution histogram.

[0207] Based on the set of layer height segmentation thresholds Hthresholds determined by the troughs of the height histogram in step S2, compare the center height zc of the point cloud of each carton with the thresholds to determine its layer number Li (1 is the bottom layer, increasing sequentially). Layer number division rule:

[0208] Layer number = 1 + count(number of thresholds < z_c in Hthresholds)

[0209] 2. Analyze the overlap degree between the projection area of the current carton on the horizontal plane and the projection areas of the adjacent cartons on the layer directly below. If part of it exceeds the projection areas of all the cartons below, calculate the proportion of the exceeded area and record the numbers of the overlapping cartons.

[0210] For cartons that are not on the bottom layer (Li > 1), analyze the inter-layer interference relationship with the adjacent cartons on the layer directly below. Analyze the overlap degree between the projection area Rproj of the current carton on the horizontal plane (XY plane) and the projection areas of all the cartons on the layer directly below. If the projection area of the current carton is completely within the projection area of a certain carton below (i.e., the projection area is completely covered), it is determined as "no interference"; if part of it exceeds the projection areas of all the cartons below, it is determined as "there is overhanging interference", and calculate the proportion of the exceeded area:

[0211] αoverhang = Aout / Atotal

[0212] where Aout is the area of the exceeded part and Atotal is the total area of the bottom of the carton; if the projection area of the current carton overlaps with multiple cartons below, it is necessary to further analyze the support distribution and record the numbers of the overlapping cartons and their respective overlapping areas.

[0213] 3. Measure the vertical distance between layers, that is, the vertical distance between the bottom of the upper carton and the top of the lower carton.

[0214] Measure the vertical distance between layers dz, that is, the vertical distance between the bottom of the upper carton and the top of the lower carton. The calculation formula is as follows:

[0215] dz = z_upper_bottom - z_lower_top

[0216] Among them, z_upper_bottom=z_upper_center-H_upper / 2, z_lower_top= z_lower_center+H_lower / 2.

[0217] The output of this step includes: carton layer number Li, excess area ratio αoverhang, list of overlapping carton numbers, and vertical spacing between layers dz.

[0218] In this embodiment:

[0219] Layer numbering: Based on height thresholds of 0.15m and 0.30m, the center heights of cartons A and B are 0.38m and 0.41m respectively, and the layer number is 3 (upper layer); the center height of cartons C and D is 0.22m, and the layer number is 2 (middle layer); the center height of cartons E, F, and G is 0.05m, and the layer number is 1 (lower layer).

[0220] Carton A: The projection area is completely located within the projection area of ​​carton C, αoverhang=0, the overlapping carton number is {C}, and the interlayer spacing d_z=0.025m.

[0221] Carton B: The projection area is completely located within the projection area of ​​carton D, αoverhang=0, the overlapping carton number is {D}, and the interlayer spacing d_z=0.055m.

[0222] Carton C: The projected area overlaps with the lower cartons E and F. The overlapping cartons are numbered {E, F}, and the interlayer spacing d_z = 0.045m.

[0223] Carton D: The projection area is completely located within the projection area of ​​carton G, αoverhang=0, the overlapping carton number is {G}, and the interlayer spacing d_z=0.045m.

[0224] Cartons E, F, G: Layer number 1, no cartons below, this output is empty.

[0225] S6. Based on the grasping-related features, form a feature vector for each carton.

[0226] Specifically, the basic geometric features extracted in step S53, the damage features extracted in step S54, the occlusion relationships extracted in step S55, and the complex stacking features (including overhang / overlap features, tilt features, and multi-layer interlacing features) extracted in step S56 are summarized and organized into feature vectors for each carton according to the following fields:

[0227] Basic geometric features: dimensions (L, W, H), center coordinates (x_c, y_c, z_c), attitude angles (Φx, Φy, Φz);

[0228] Damage characteristics: mask of the damaged area, damaged area, and maximum scratch length;

[0229] Occlusion relationship: occlusion ratio;

[0230] Suspended / Overlap Features: Number of support points Nsupport, overlap length Loverlap (only applicable to the upper carton, the middle and lower layers are empty).

[0231] Tilt characteristics: attitude angle Φ, tilt direction, contact support area A_support, support area ratio;

[0232] Multi-layered staggered features: layer number Li, excess area ratio αoverhang, overlapping carton number, and interlayer spacing dz.

[0233] It is important to emphasize that the feature data output in this step is based on the scene point cloud collected at the current moment, reflecting the real-time characteristics of each carton in the current stacking state. After the extraction is completed, if the scene point cloud changes (such as some cartons being removed), steps S1 to S5 must be re-executed to update the feature data of the remaining cartons, ensuring that subsequent decisions are based on the latest scene information.

[0234] The complete feature data output in this embodiment is as follows:

[0235] Cardboard box A (top-tilted cardboard box):

[0236] Basic geometric features: Dimensions 0.5m × 0.4m × 0.12m, center coordinates (1.25m, 0.80m, 0.38m), attitude angles Φx=0°, Φy=18°, Φz=0°;

[0237] Damage characteristics: No damage;

[0238] Occlusion ratio: 104.5%;

[0239] Suspended / overlapping characteristics: Number of support points Nsupport=2, overlap length Loverlap=0.36m;

[0240] Tilt characteristics: attitude angle 18°, tilt direction along the positive X-axis, contact support area Asupport = 0.18m² 2 The supporting area accounts for 90%;

[0241] Multi-layer staggered features: layer number Li=3, overhang area ratio αoverhang=0, overlapping carton number {C}, interlayer spacing dz=0.025m.

[0242] Cardboard box B (top-tilted cardboard box):

[0243] Basic geometric features: Dimensions 0.5m × 0.4m × 0.12m, center coordinates (1.25m, 1.22m, 0.41m), attitude angles Φx = 0°, Φy = 15.7°, Φz = 0°;

[0244] Damage characteristics: No damage;

[0245] Occlusion ratio: 102%;

[0246] Suspended / overlapping characteristics: Number of support points Nsupport=2, overlap length Loverlap=0.38m;

[0247] Tilt characteristics: attitude angle 15.7°, tilt direction along the positive X-axis, contact support area Asupport = 0.19m². 2 The supporting area accounts for 95%;

[0248] Multi-layer staggered features: layer number Li=3, overhang area ratio αoverhang=0, overlapping carton number {D}, interlayer spacing dz=0.055m.

[0249] The feature data of cartons C and D (middle-layer regularly stacked cartons) and cartons E, F, and G (lower-layer regularly stacked cartons) are similar to those of carton A, but their orientation angles are all 0°, with no damage or obstruction, and the suspended / overlapping features are not applicable (output is empty). Specific dimensions, center coordinates, and other data for each carton are omitted.

[0250] The feature data extracted by this method reflects the scene state at a specific moment. In actual extraction operations, as the robotic arm successively extracts cardboard boxes, the stacked scene continuously changes: extracted cardboard boxes are removed from the point cloud, previously occluded boxes are exposed, previously supported boxes may lose their support, and the interlayer relationships also change accordingly. Therefore, after each extraction action, the system must re-execute this method to re-extract the features of the remaining cardboard boxes based on the updated point cloud data to ensure the accuracy of subsequent decisions. This dynamic update mechanism is the key to ensuring the real-time performance and accuracy of features in this invention, and it is also the foundation of the subsequent closed-loop optimization control system.

[0251] The method was tested and verified in the scenario described in this embodiment, and the results are as follows: Point cloud segmentation accuracy: The success rate of extracting the precise upper surface point cloud of 7 cartons is 100%. After removing the warped points, the flatness of the upper surface point cloud is improved, and the normal vector calculation error is ≤0.5°.

[0252] Accuracy of quantification of complex features: The quantification results of suspended / overlapping features (number of support points, overlap length), tilted features (attitude angle, support area ratio), and multi-layer staggered features (layer number, excess area ratio, inter-layer spacing) are compared with manual measurements, and the error is within an acceptable range;

[0253] Scene coverage capability: It can effectively handle single and mixed complex scenes such as suspended, tilted (≤45°), and multi-layered overlapping.

[0254] To verify the enhanced effect of optional constraints, an additional experiment was performed under the same stacking scenario (three layers of cardboard boxes, top layer tilted, middle layer regular, bottom layer regular), introducing known specifications. The known specifications were set as: rated length = 0.5m, rated width = 0.4m, rated height = 0.12m, and the flatness parameter corresponded to the new boxes. Experimental results show that:

[0255] Layering accuracy: After introducing the rated height constraint, the layer height segmentation threshold is more accurate, and the accuracy of determining the layer assignment of edge cartons is improved compared with the case without constraint;

[0256] Clustering accuracy: By introducing the constraints of rated length and rated width, the slight adhesion between the middle and upper layer point clouds is effectively avoided and the clustering accuracy is further improved.

[0257] Warp rejection: After introducing flatness constraints, the curvature threshold is set adaptively, which is more effective in rejecting cartons with slight surface wrinkles.

[0258] Therefore, known specifications, as optional enhancement constraints, can further improve the recognition accuracy and robustness of the system in complex scenarios.

[0259] Example 2

[0260] This embodiment provides a feature extraction system for multi-layer stacked cardboard boxes based on 3D vision. This system is deployed on a central control system to implement a feature extraction method for multi-layer stacked cardboard boxes based on 3D vision. The system includes the following software modules deployed on the central control system:

[0261] 1. Data acquisition module, used to acquire raw point cloud data and two-dimensional RGB images of the stacked scene.

[0262] The data acquisition module receives the raw point cloud P_raw and the synchronized two-dimensional RGB image I_rgb acquired by the 3D structured light main camera, and establishes the pixel correspondence between the point cloud and the image through a pre-calibrated extrinsic parameter matrix for use by subsequent modules.

[0263] As an optional implementation, while acquiring the raw point cloud data and 2D RGB image of the stacked scene, the known rated specifications of the cardboard box to be identified are obtained. The known rated specifications include at least one of the following parameters: rated length, rated width, rated height, and flatness. The purpose of each parameter is as follows:

[0264] When constructing the height distribution histogram, the width of the height interval is adaptively determined by combining the known rated height;

[0265] When dividing the sorted point cloud into different height layers, the known rated height constraint layer height segmentation threshold is combined;

[0266] When clustering the sorted point cloud at each layer, the clustering results are constrained by the known rated length and rated width.

[0267] When removing warped abnormal points, the curvature threshold is adaptively set in conjunction with the known flatness parameters.

[0268] 2. Point cloud preprocessing module, used to preprocess the raw point cloud, remove outliers, sort by Z-axis height, and construct a height distribution histogram, including:

[0269] Statistical filtering: For each point, if the number of points in its neighborhood with radius r is less than a preset threshold (e.g., 10), it is identified as an outlier and removed, resulting in the filtered point cloud P_filtered.

[0270] Z-axis sorting: Sort P_filtered by Z-axis coordinate values ​​from smallest to largest to obtain an ordered point set P_sorted.

[0271] Histogram construction: Count the number of points in each height interval with a preset interval width (e.g., 5mm) to generate a height distribution histogram, and determine the set of floor height segmentation thresholds H_thresholds based on the trough positions of the histogram.

[0272] As an optional implementation, if a known rated height is imported into the system configuration, the module can adaptively determine the height interval width based on the rated height (for example, the interval width is set to the smaller value between the rated height and 10, and compared with a fixed value of 5mm to take the smaller value), so that the position of the histogram trough more accurately reflects the real inter-layer boundary.

[0273] 3. A point cloud hierarchical clustering module is used to divide the sorted point cloud into different height layers based on the trough positions of the height distribution histogram, and to cluster the point cloud in each height layer to identify the initial point cloud clusters representing individual cardboard boxes, including:

[0274] Height layer partitioning: Using the layer height threshold H_thresholds output by the point cloud preprocessing module, the ordered point set P_sorted is partitioned into height layers L_j (j=1,2,…,k+1) corresponding to different stacking layers.

[0275] Clustering identification: For the point cloud L_j at each height level, a clustering algorithm based on Euclidean distance (such as DBSCAN) is used with the neighborhood radius ε=3cm and the minimum number of neighborhood points MinPts=50 as parameters to identify the initial point cloud clusters representing independent cardboard boxes, and to calculate the minimum directed bounding box (OBB) and normal vector of each cluster.

[0276] As an optional implementation, if the system configuration includes a known rated height, the module can use this rated height for threshold verification and correction when determining the layer height segmentation threshold (e.g., the difference between adjacent layer heights should be greater than half and less than twice the rated height; otherwise, they should be merged or adjusted), avoiding incorrect layering due to noise or missing local point clouds. If the system configuration includes a known rated length and width, the module can use these rated length and width to impose size constraints on the clustering results during clustering identification (e.g., removing invalid clusters whose minimum directed bounding box size deviates from the rated specification by more than ±20%), effectively preventing incorrect segmentation due to cardboard box adhesion or point cloud noise.

[0277] 4. Point cloud refinement module: This module removes warped outliers from each initial point cloud cluster through curvature analysis and optimizes the carton boundaries through normal vector consistency analysis, resulting in a precise top surface point cloud for each carton. This includes:

[0278] Curvature analysis: Fit a quadratic surface to each point in the cluster and its neighborhood (radius 1cm), calculate the average curvature k, and if k>0.3, it is identified as a warped point and removed.

[0279] Normal vector consistency analysis: Calculate the angle between the normal vector of each point and the average normal vector of its neighboring points. If the angle is greater than 15°, the point is discarded.

[0280] Principal Component Analysis (PCA): Principal component analysis is performed on the precise upper surface point cloud to obtain the length, width, thickness direction, and precise spatial orientation of the carton.

[0281] As an optional implementation, if a known flatness parameter is imported into the system configuration, the module can adaptively set the curvature threshold according to the flatness parameter when removing warp points through curvature analysis (e.g., a threshold of 0.3 for new boxes with good flatness, and an appropriate reduction for old boxes that may be deformed), making the removal of warp points more accurate and adapting to different carton surface conditions.

[0282] 5. Feature extraction module, used to quantify and extract relevant features based on the precise upper surface point cloud of each carton and the synchronously acquired two-dimensional RGB image, including:

[0283] 1) The registration and segmentation submodule projects the precise upper surface point cloud onto the two-dimensional image plane to generate a mask_i corresponding to each carton; the GrabCut algorithm is used to accurately segment the mask area, and the boundary of the precise upper surface point cloud is corrected by back projection.

[0284] 2) Basic geometric feature extraction submodule: Perform principal component analysis on the corrected point cloud to extract the dimensions (L, W, H), center coordinates (x_c, y_c, z_c), and attitude angles (Φx, Φy, Φz) of the carton.

[0285] 3) Damage feature extraction submodule: Input the RGB image region corresponding to the cardboard box into the pre-trained deep learning semantic segmentation model (such as U-Net), identify the damaged area, and output the binary mask, area and maximum scratch length of the damaged area.

[0286] 4) Occlusion Relationship Quantification Submodule: Calculate the theoretical number of points N_theory = L×W×ρ based on the precise dimensions of the carton and the point cloud density ρ. Compare it with the actual number of points N_actual, output the occlusion ratio = N_actual / N_theory, and determine "no occlusion" (ratio ≥ 50%) or "partial occlusion" (ratio < 50%).

[0287] 5) Suspended / Overlap Feature Extraction Submodule: For the upper cardboard box, analyze the number of support points and the projected overlap length below it, and output the number of support points N_support and the overlap length L_overlap.

[0288] 6) Tilt Feature Extraction Submodule: Calculates the attitude angle Φ and tilt direction based on the cardboard box normal vector, calculates the bottom contact support area A_support and support area ratio through the convex hull algorithm, and outputs the attitude angle, tilt direction, contact support area and support area ratio.

[0289] 7) Multi-layer interlacing feature extraction submodule: Determine the carton layer number L_i based on the layer height threshold, analyze the interlayer projection overlap relationship, calculate the excess area ratio α_overhang, record the overlapping carton numbers, measure the interlayer vertical spacing d_z, and output the layer number, excess area ratio, overlapping carton number, and interlayer spacing.

[0290] 6. Feature Output Module: This module structures and organizes all the feature data extracted by quantization to form a feature vector for each carton. The feature vector includes basic geometric features, damage features, occlusion relationships, overhang / overlap features, tilt features, and multi-layered interlacing features.

[0291] When the scene changes after the extraction is completed, the external system calls the data acquisition module, point cloud preprocessing module, point cloud hierarchical clustering module, point cloud refinement module, and feature extraction module again to perform a new round of feature extraction on the point cloud and image data in the new scene. The feature output module outputs the updated feature vector again.

[0292] When the system is working, the data acquisition module first acquires the raw point cloud P_raw and the synchronous two-dimensional RGB image I_rgb collected by the 3D vision system.

[0293] Subsequently, the point cloud preprocessing module performs statistical filtering on the original point cloud to remove outliers, sorts the points by Z-axis height, constructs a height distribution histogram, and outputs the layer height segmentation threshold H_thresholds and the sorted point cloud P_sorted.

[0294] The point cloud hierarchical clustering module divides the sorted point cloud into different height layers based on the layer height threshold, performs DBSCAN clustering on the point cloud of each layer, identifies the initial point cloud clusters representing independent cardboard boxes, and calculates the minimum directed bounding box and normal vector of each cluster.

[0295] The point cloud refinement module performs curvature analysis to remove warped points and normal vector consistency analysis to optimize the boundary for each initial point cloud cluster. It also obtains the precise upper surface point cloud and spatial orientation of each carton through principal component analysis.

[0296] The feature extraction module performs multi-dimensional feature quantization based on accurate upper surface point clouds and synchronized RGB images: the registration and segmentation submodule projects the point cloud onto the image plane and corrects the boundaries; the basic geometric feature extraction submodule extracts dimensions, center coordinates, and attitude angles; the damage feature extraction submodule identifies damaged areas using a deep learning model; the occlusion relationship quantification submodule calculates the occlusion ratio; the suspended / overlapping feature extraction submodule analyzes the number of support points and overlap length; the tilt feature extraction submodule calculates the attitude angle and support area; and the multi-layer staggered feature extraction submodule determines the layer number, excess area ratio, overlap relationship, and inter-layer spacing. Each submodule processes in parallel without interfering with the others.

[0297] Finally, the feature output module organizes all the quantified features into a structured feature vector representing the grasping-related features of each carton, and outputs it to an external system (such as an extraction decision system).

[0298] As an optional implementation, if known rated specifications (including rated length, rated width, rated height and flatness parameters) are imported into the system configuration, the point cloud preprocessing module, point cloud hierarchical clustering module and point cloud refinement module will combine these specifications to make adaptive parameter adjustments and result constraints, further improving the accuracy of hierarchical, clustering and warp removal in complex scenarios with dense stacking and similar sizes.

[0299] When the scene changes after the extraction is completed, the external system re-triggers the above process to perform a new round of feature extraction on the point cloud and image data in the new scene, ensuring that the output features always reflect the current scene state.

[0300] Example 3

[0301] This embodiment provides an electronic device, the electronic device comprising:

[0302] A memory for storing computer programs, the memory including non-volatile memory and random access memory, the non-volatile memory for storing system programs and the random access memory for caching runtime data;

[0303] The processor is used to execute the computer program stored in the memory to implement the steps of the feature extraction method for multi-layer stacked cartons based on 3D vision as described in Embodiment 1.

[0304] For details on the specific implementation of each step and related explanations, please refer to Example 1, which will not be repeated here.

[0305] The processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices.

[0306] Example 4

[0307] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of a 3D vision-based multi-layer stacked carton feature extraction method as described in Embodiment 1.

[0308] For details on the specific implementation of each step of this method and related explanations, please refer to Example 1, which will not be elaborated here.

[0309] The computer-readable storage medium can be any tangible medium that contains or stores program instructions, such as: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0310] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for feature extraction of multi-layer stacked cardboard boxes based on 3D vision, characterized in that, include: Acquire raw point cloud data and 2D RGB images of the stacked scene; Obtain the known rated specifications of the carton to be identified, wherein the known rated specifications include at least one of the rated length, rated width, rated height and flatness parameters; The original point cloud data is preprocessed to obtain a sorted point cloud. Based on the sorted point cloud, the number of points in each height interval is counted and the height interval width is adaptively determined by combining the known rated height to construct a height distribution histogram. Based on the trough positions of the height distribution histogram and the known rated height constraint layer height segmentation threshold, the sorted point cloud is divided into different height layers to obtain the point cloud of each height layer. Then, the sorted point cloud of each layer is clustered to identify the initial point cloud clusters representing individual cardboard boxes. When clustering the sorted point cloud of each layer, the clustering results are combined with the known rated length and rated width constraints. For each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud for each carton. When removing warped abnormal points, the curvature threshold is adaptively set in combination with the known flatness parameters. Based on the precise point cloud of the upper surface of each carton and the synchronously acquired two-dimensional RGB image, relevant features are quantitatively extracted and captured. Based on the grasping-related features, a feature vector is formed for each carton.

2. The method according to claim 1, characterized in that, For each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud for each carton. Specifically, this includes: For each point in the cluster and its neighborhood, fit a quadratic surface and calculate the average curvature of the point. If the curvature is greater than a preset threshold, it is identified as a warped point and removed, resulting in a point cloud after removing warped points. Perform a normal vector consistency analysis on the point cloud after removing warped points, calculate the angle between the normal vector of each point and the average normal vector of its neighboring points, and remove points if the angle is greater than 15°, thus obtaining the accurate upper surface point cloud of each carton. Principal component analysis was performed on the precise upper surface point cloud to obtain the length direction, width direction, and upper surface normal vector direction of the carton.

3. The method according to claim 1, characterized in that, The extraction of relevant features includes: Based on the precise point cloud of the upper surface, basic geometric features are extracted through principal component analysis. The basic geometric features include: precise dimensions of the carton, center coordinates, attitude angle and tilt angle, as well as the length direction, width direction and the direction of the normal vector of the upper surface. Based on the RGB image region corresponding to the cardboard box, the damaged area is identified and the damage features are extracted, including the damaged area and the scratch length. By calculating the ratio of the actual point cloud number on the upper surface of the cardboard box to its theoretical point cloud number, it is determined whether the box is unobstructed or partially obstructed, and the obstruction ratio is output. Based on the basic geometric features and point cloud layering results, complex stacking features are extracted, including suspended / overlapping features, tilted features, and multi-layered interlaced features.

4. The method according to claim 3, characterized in that, The suspended / overlapping features are quantified by analyzing the number of support point clouds beneath the cardboard box and the projected overlap length, specifically including: Determine the projection area of ​​the cardboard box on the horizontal plane, and query the point cloud below the current cardboard box that is located within the projection area and whose height is lower than the current cardboard box; Connectivity analysis is performed on the lower point cloud that meets the height condition to obtain support point clusters. The number of support points is the number of clusters. Calculate the projected length of each support area along the current length of the carton, and take the maximum projected length as the overlap length.

5. The method according to claim 3, characterized in that, The tilting characteristic is quantified by analyzing the normal vector of the carton and the bottom contact support area, specifically including: Based on the principal component analysis results of the point cloud on the upper surface of the cardboard box, the normal vector of the upper surface of the cardboard box is calculated, and the tilt angle around the X-axis and the tilt angle around the Y-axis are calculated according to the normal vector to obtain the total attitude angle; Project the point cloud of the carton vertically onto its bottom plane to obtain the lower support point cloud within the bottom projection area. Perform convex hull calculation on the support point cloud to obtain the actual contact support area at the bottom of the carton, and calculate the contact support area.

6. The method according to claim 3, characterized in that, The multi-layered interlacing feature is quantified by analyzing the layer number of the carton, the overlapping relationship of the interlayer projections, and the load-bearing strength of the lower layer of the carton, specifically including: The layer number of each carton is determined based on the trough locations of the height distribution histogram; Analyze the overlap between the projection area of ​​the current cardboard box on the horizontal plane and the projection area of ​​the adjacent cardboard box directly below. If it partially exceeds the projection area of ​​all the cardboard boxes below, calculate the proportion of the excess area and record the overlapping cardboard box number. Measure the vertical spacing between layers, that is, the vertical distance between the bottom of the upper carton and the top of the lower carton.

7. A feature extraction system for multi-layer stacked cardboard boxes based on 3D vision, characterized in that, include: The data acquisition module is used to acquire raw point cloud data and two-dimensional RGB images of the stacked scene; The point cloud preprocessing module is used to preprocess the original point cloud, remove outliers, sort them by Z-axis height, and construct a height distribution histogram. The point cloud hierarchical clustering module is used to divide the sorted point cloud into different height layers according to the trough position of the height distribution histogram, and to cluster the point cloud in each height layer to identify the initial point cloud clusters representing individual cardboard boxes. The point cloud refinement module is used to remove warped outliers from each initial point cloud cluster through curvature analysis and optimize the carton boundary through normal vector consistency analysis to obtain the accurate top surface point cloud of each carton. The feature extraction module is used to quantify and extract relevant features based on the precise upper surface point cloud of each carton and the synchronously acquired two-dimensional RGB image. The feature output module is used to structure and organize all the feature data extracted by quantization to form the feature vector of each carton.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images