A method and system for feature extraction of multi-layer stacked cartons based on 3D vision

Abstract

This invention discloses a method and system for feature extraction of multi-layer stacked cardboard boxes based on 3D vision. Addressing the problem that existing technologies cannot accurately quantify surface anomalies and multi-dimensional stacking features of cardboard boxes in complex stacking scenarios, this invention acquires the original point cloud and RGB images of the stacking scene; after preprocessing, a height distribution histogram is constructed and clustered to obtain initial point cloud clusters; warped anomalies are eliminated through curvature analysis and normal vector consistency to obtain accurate upper surface point clouds; principal component analysis is used to obtain the length, width, and normal vector direction of the cardboard box; basic geometric features, damage features, occlusion relationships, overhang / overlap features, tilt features, and multi-layer interlacing features are extracted by combining RGB images; finally, the feature vector of each cardboard box is output in a structured manner. This invention achieves refined point cloud processing, effectively eliminates warped points, systematically quantifies complex stacking features, and integrates multimodal data, significantly improving the ability to identify and extract features from cardboard boxes in complex logistics scenarios.

Claims

1. A method for feature extraction of multi-layer stacked cardboard boxes based on 3D vision, characterized in that, include: Acquire raw point cloud data and 2D RGB images of the stacked scene; The original point cloud data is preprocessed to obtain a sorted point cloud. Based on the sorted point cloud, the number of points in each height interval is counted to construct a height distribution histogram. The sorted point cloud is divided into different height layers according to the trough positions of the height distribution histogram, and the point cloud of each height layer is obtained. Then, the sorted point cloud of each layer is clustered to identify the initial point cloud clusters representing individual cardboard boxes. For each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud for each carton. Based on the precise point cloud of the upper surface of each carton and the synchronously acquired two-dimensional RGB image, relevant features are quantitatively extracted and captured. Based on the grasping-related features, a feature vector is formed for each carton.

2. The method according to claim 1, characterized in that, Also includes: Obtain the known rated specifications of the carton to be identified, wherein the known rated specifications include at least one of the rated length, rated width, rated height and flatness parameters; When constructing the height distribution histogram, the width of the height interval is adaptively determined by combining the known rated height; When dividing the sorted point cloud into different height layers, the known rated height constraint layer height segmentation threshold is combined; When clustering the sorted point cloud at each layer, the clustering results are constrained by the known rated length and rated width. When removing warped abnormal points, the curvature threshold is adaptively set in conjunction with the known flatness parameters.

3. The method according to claim 1, characterized in that, For each initial point cloud cluster, warped outliers are removed, and the carton boundaries are analyzed and optimized to obtain the accurate top surface point cloud for each carton. Specifically, this includes: For each point in the cluster and its neighborhood, fit a quadratic surface and calculate the average curvature of the point. If the curvature is greater than a preset threshold, it is identified as a warped point and removed, resulting in a point cloud after removing warped points. Perform a normal vector consistency analysis on the point cloud after removing warped points, calculate the angle between the normal vector of each point and the average normal vector of its neighboring points, and remove points if the angle is greater than 15°, thus obtaining the accurate upper surface point cloud of each carton. Principal component analysis was performed on the precise upper surface point cloud to obtain the length direction, width direction, and upper surface normal vector direction of the carton.

4. The method according to claim 1, characterized in that, The extraction of relevant features includes: Based on the precise point cloud of the upper surface, basic geometric features are extracted through principal component analysis. The basic geometric features include: precise dimensions of the carton, center coordinates, attitude angle and tilt angle, as well as the length direction, width direction and the direction of the normal vector of the upper surface. Based on the RGB image region corresponding to the cardboard box, the damaged area is identified and the damage features are extracted, including the damaged area and the scratch length. By calculating the ratio of the actual point cloud number on the upper surface of the cardboard box to its theoretical point cloud number, it is determined whether the box is unobstructed or partially obstructed, and the obstruction ratio is output. Based on the basic geometric features and point cloud layering results, complex stacking features are extracted, including suspended / overlapping features, tilted features, and multi-layered interlaced features.

5. The method according to claim 4, characterized in that, The suspended / overlapping features are quantified by analyzing the number of support point clouds beneath the cardboard box and the projected overlap length, specifically including: Determine the projection area of ​​the cardboard box on the horizontal plane, and query the point cloud below the current cardboard box that is located within the projection area and whose height is lower than the current cardboard box; Connectivity analysis is performed on the lower point cloud that meets the height condition to obtain support point clusters. The number of support points is the number of clusters. Calculate the projected length of each support area along the current length of the carton, and take the maximum projected length as the overlap length.

6. The method according to claim 4, characterized in that, The tilting characteristic is quantified by analyzing the normal vector of the carton and the bottom contact support area, specifically including: Based on the principal component analysis results of the point cloud on the upper surface of the cardboard box, the normal vector of the upper surface of the cardboard box is calculated, and the tilt angle around the X-axis and the tilt angle around the Y-axis are calculated according to the normal vector to obtain the total attitude angle; Project the point cloud of the carton vertically onto its bottom plane to obtain the lower support point cloud within the bottom projection area. Perform convex hull calculation on the support point cloud to obtain the actual contact support area at the bottom of the carton, and calculate the contact support area.

7. The method according to claim 4, characterized in that, The multi-layered interlacing feature is quantified by analyzing the layer number of the carton, the overlapping relationship of the interlayer projections, and the load-bearing strength of the lower layer of the carton, specifically including: The layer number of each carton is determined based on the trough locations of the height distribution histogram; Analyze the overlap between the projection area of ​​the current cardboard box on the horizontal plane and the projection area of ​​the adjacent cardboard box directly below. If it partially exceeds the projection area of ​​all the cardboard boxes below, calculate the proportion of the excess area and record the overlapping cardboard box number. Measure the vertical spacing between layers, that is, the vertical distance between the bottom of the upper carton and the top of the lower carton.

8. A feature extraction system for multi-layer stacked cardboard boxes based on 3D vision, characterized in that, include: The data acquisition module is used to acquire raw point cloud data and two-dimensional RGB images of the stacked scene; The point cloud preprocessing module is used to preprocess the original point cloud, remove outliers, sort them by Z-axis height, and construct a height distribution histogram. The point cloud hierarchical clustering module is used to divide the sorted point cloud into different height layers according to the trough position of the height distribution histogram, and to cluster the point cloud in each height layer to identify the initial point cloud clusters representing individual cardboard boxes. The point cloud refinement module is used to remove warped outliers from each initial point cloud cluster through curvature analysis and optimize the carton boundary through normal vector consistency analysis to obtain the accurate top surface point cloud of each carton. The feature extraction module is used to quantify and extract relevant features based on the precise upper surface point cloud of each carton and the synchronously acquired two-dimensional RGB image. The feature output module is used to structure and organize all the feature data extracted by quantization to form the feature vector of each carton.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images