Parcel specification detection method based on shape fitting

Abstract

The invention discloses a parcel specification detection method based on shape fitting, and relates to the technical field of logistics, and the method comprises the steps: carrying out the target detection of an RGB image of a parcel detection region through employing a target detection model, determining a parcel region where a to-be-detected parcel is located, extracting the depth data of the parcel region, and converting the depth data into parcel point cloud data, then randomly selecting a plurality of point clouds from the parcel point cloud data to respectively fit a plurality of reference geometric shapes, and evaluating the matching degree between other parcel point cloud data and each fitted reference geometric shape to obtain the confidence coefficient of each fitted reference geometric shape; and finally, calculating according to the reference geometrical shape with the maximum confidence coefficient to obtain specification parameters of the parcel to be detected. According to the method, through combination of an image algorithm and a point cloud processing technology, robust processing can be carried out on point cloud data with different qualities, so that various standard and special-shaped parcel specifications can be quickly and accurately identified, and the sorting efficiency and the application range of a logistics automation system can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of logistics technology, and in particular to a package specification detection method based on shape fitting. Background Art

[0002] In modern logistics systems, the automated sorting and processing of parcels is the key to improving logistics efficiency, and the accuracy and efficiency of parcel detection directly affect the subsequent sorting and processing effects. Currently, two-dimensional image detection technology is commonly used for parcel detection, but most parcels are three-dimensional in shape, and it is difficult for two-dimensional image detection technology to accurately and completely capture the three-dimensional shape characteristics of the parcel. This leads to the existing two-dimensional image detection technology has great limitations, and can only have a good detection effect on standard parcels of standard size and standard shape. However, in actual logistics scenarios, in addition to standard parcels, there are many irregularly shaped parcels or special-shaped parcels with sizes beyond the conventional range. Two-dimensional image detection technology often cannot accurately identify these special-shaped parcels and cannot meet the needs of modern logistics for high efficiency and accuracy. Summary of the invention

[0003] In view of the above problems and technical requirements, this application proposes a package specification detection method based on shape fitting. The technical solution of this application is as follows:

[0004] A package specification detection method based on shape fitting, the package specification detection method comprising:

[0005] Use RGBD intelligent depth camera to obtain RGB image and depth data of the package detection area;

[0006] The target detection model is used to perform target detection on the RGB image to determine the parcel area where the parcel to be detected is located, and then the depth data of the parcel area is extracted and converted to obtain the parcel point cloud data;

[0007] A plurality of point clouds are randomly selected from the parcel point cloud data as sample points, and a plurality of reference geometric shapes are respectively fitted based on the selected sample points;

[0008] Evaluate the degree of matching between other parcel point cloud data and each fitted reference geometry to obtain the confidence of each fitted reference geometry; the higher the degree of matching between other parcel point cloud data and the reference geometry, the higher the confidence;

[0009] Based on the package point cloud data, the specification parameters of the package to be inspected are calculated according to the reference geometry with the highest confidence.

[0010] A further technical solution is that the confidence level of each reference geometric shape obtained by fitting includes:

[0011] All point clouds that meet the geometric constraints of the reference geometric shape are selected from the parcel point cloud data to form a point cloud set P ψ ;

[0012] Point cloud collection P ψ The point clouds in the cluster are clustered based on the location, and the category with the largest number of point clouds is extracted to obtain the largest connected point cloud set P max ;

[0013] According to the maximum connected point cloud set P max The number of included point clouds determines the confidence level.

[0014] A further technical solution is that the reference geometric shape includes a plane shape, a cylindrical shape and a spherical shape, and the geometric constraints for detecting whether any point cloud in the package point cloud data conforms to the reference geometric shape include:

[0015] For the fitted plane shape, when the angle between the point cloud normal vector of the point cloud and the plane normal vector of the fitted plane shape is less than the angle threshold β1, it is determined that the point cloud meets the geometric constraints of the plane shape, otherwise it is determined that the point cloud does not meet the geometric constraints of the plane shape;

[0016] For the fitted cylindrical shape, determine the intersection of the projection plane perpendicular to the central axis of the fitted cylindrical shape and the central axis where the point cloud is located. When the angle between the vector from the intersection to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape is less than the cross-sectional radius of the cylinder, determine that the point cloud meets the geometric constraints of the cylindrical shape, otherwise determine that the point cloud does not meet the geometric constraints of the cylindrical shape;

[0017] For the fitted spherical shape, when the angle between the vector from the center of the fitted spherical shape to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted spherical shape is less than the radius of the sphere, it is determined that the point cloud meets the geometric constraints of the spherical shape; otherwise, it is determined that the point cloud does not meet the geometric constraints of the spherical shape.

[0018] A further technical solution is that the reference geometric shape includes a plane shape, and fitting the plane shape based on the selected sample points includes:

[0019] Randomly select three non-collinear point clouds from the parcel point cloud data as sample points p1, p2 and p3;

[0020] Use vector cross product to calculate the plane normal vector of the candidate plane formed by sample points p1, p2 and p3 in, is the vector from sample point p1 to sample point p2, is the vector from sample point p1 to sample point p3;

[0021] When the angles between the point cloud normal vectors of the sample points p1, p2 and p3 and the plane normal vector n of the candidate plane are all less than the angle threshold β1, the candidate plane formed by the sample points p1, p2 and p3 is used as the fitted plane shape, otherwise the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as the sample points p1, p2 and p3 is re-executed.

[0022] A further technical solution is that the reference geometric shape includes a cylindrical shape, and the cylindrical shape is fitted based on the selected sample points and the geometric characteristic parameters of the fitted cylindrical shape are determined to include:

[0023] Randomly select two point clouds from the parcel point cloud data as sample points p4 and p5;

[0024] Calculate the vector product of the point cloud normal vector n4 of the sample point p4 and the point cloud normal vector n5 of the sample point p5 to obtain the normal vector of the candidate central axis ν, and determine the projection plane Pv perpendicular to the normal vector of the candidate central axis ν;

[0025] Determine the projection line L'4 of the line L4 where the point cloud normal vector n4 of the sample point p4 lies on the projection plane Pv, determine the projection line L'5 of the line L5 where the point cloud normal vector n5 of the sample point p5 lies on the projection plane Pv, and determine the intersection point c of the projection line L'4 and the projection line L'5, and take the line along the normal vector of the candidate central axis ν and passing through the intersection point c as the candidate central axis ν;

[0026] Determine the projection point p'4 of the sample point p4 in the projection plane Pv, determine the projection point p'5 of the sample point p5 in the projection plane Pv, and take the average of the Euclidean distances between the intersection point c and the projection point p'4 and the projection point p'5 as the candidate cross-section radius of the cylinder;

[0027] When the difference between the Euclidean distance between the projection point p'4 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the angle between the vector from the intersection point c to the projection point p'4 and the point cloud normal vector n4 of the sample point p4 is less than the angle threshold β2, and the angle between the vector from the intersection point c to the projection point p'5 and the point cloud normal vector n5 of the sample point p5 is less than the angle threshold β2, the cylindrical shape determined by the candidate central axis and the candidate cylinder cross section radius is used as the fitted cylindrical shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 is re-executed.

[0028] A further technical solution is that the reference geometric shape includes a spherical shape, and the spherical shape is fitted based on the selected sample points and the geometric characteristic parameters of the fitted spherical shape are determined to include:

[0029] Randomly select two point clouds from the package point cloud data as sample points p6 and p7, and calculate the straight line L6 passing through the sample point p6 and along the direction of the point cloud normal vector n6 of the sample point p6, and calculate the straight line L7 passing through the sample point p7 and along the direction of the point cloud normal vector n7 of the sample point p7;

[0030] Calculate the shortest line segment between straight line L6 and straight line L7, and use the midpoint of the shortest line segment as the candidate sphere center;

[0031] The average of the Euclidean distances between the candidate sphere center and the sample point p6 and the sample point p7 is taken as the candidate sphere radius;

[0032] When the difference between the Euclidean distance between the candidate sphere center and the sample point p6 and the radius of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center and the sample point p7 and the radius of the candidate sphere is less than the distance threshold λ2, and the angle between the vector from the candidate sphere center to the sample point p6 and the point cloud normal vector n6 of the sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center to the sample point p7 and the point cloud normal vector n7 of the sample point p7 is less than the angle threshold β3, the spherical shape determined according to the candidate sphere center and the candidate sphere radius is used as the fitted spherical shape, otherwise the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 is re-executed.

[0033] A further technical solution is to calculate the specification parameters of the package to be detected based on the package point cloud data according to the reference geometric shape with the highest confidence, including:

[0034] When the reference geometric shape with the highest confidence is a plane shape, determine the maximum connected point cloud set P max The distribution area of ​​the point cloud in the fitted plane shape and the distribution area in the direction of the plane normal vector of the plane shape are obtained to obtain three-dimensional size information, and the specification parameters of the package to be detected include: the package shape is a cube, and the package size is the calculated three-dimensional size information;

[0035] When the reference geometric shape with the highest confidence is a cylinder, determine the maximum connected point cloud set P max The distribution area of ​​the point cloud in the image along the central axis of the fitted cylindrical shape is used to determine the length of the cylinder, and the specification parameters of the package to be detected include: the package shape is a cylindrical shape, the bottom radius of the package is the cross-sectional radius of the cylindrical shape fitted, and the package length is the calculated cylindrical length;

[0036] When the reference geometric shape with the highest confidence is a spherical shape, the specification parameters of the package to be detected include: the package shape is a spherical shape, and the package radius is the spherical radius of the spherical shape obtained by fitting.

[0037] Its further technical solution is that the converted package point cloud data includes:

[0038] The internal parameters of the RGBD intelligent depth camera are used to convert the depth data of the wrapped area into three-dimensional point cloud data and perform equidistant sampling to obtain the initial point cloud data;

[0039] The point clouds whose neighborhood point clouds are less than the number threshold in the sampled initial point cloud data are filtered out to complete the radius filtering, and the neighborhood point clouds of each point cloud include all the point clouds within the filtering radius of the point cloud;

[0040] For any point cloud in the initial point cloud data that has completed the radius filtering, the average value μ and the standard deviation σ of the Euclidean distance between the point cloud and its nearest K point clouds are calculated respectively, and the distance threshold max=μ+ε*σ is obtained. When the average value μ of the Euclidean distance between the point cloud and its nearest K point clouds is greater than max, the point cloud is filtered out, otherwise the point cloud is retained, and ε is the threshold coefficient;

[0041] After traversing all point clouds in the initial point cloud data after radius filtering, the wrapped point cloud data is obtained.

[0042] A further technical solution is to extract the depth data of the package area and convert it into the package point cloud data, including:

[0043] Extract the depth data of the wrapped area, filter out the depth data that is much higher than the hanging height of the RGBD smart depth camera, and perform morphological processing on the edges of the remaining depth data to obtain the depth data after removing noise in the wrapped area.

[0044] A statistical analysis is performed on the depth data after noise removal in the package area, and the height estimation value of the package to be detected is obtained according to the reference plane position of the calibrated package detection area. The depth data whose deviation from the height estimation value reaches the deviation threshold in the depth data after noise removal in the package area is filtered out to obtain the processed depth data of the package area, and the processed depth data of the package area is converted into the package point cloud data.

[0045] Its further technical solution is that the target detection model used is designed based on the PP-PicoDet series target detection model. The target detection model uses a lightweight LCNet neural network as a feature extraction layer, uses the PicoHeadV2 backbone network as the target detection head structure and combines the LCPAN module;

[0046] During the model training process, data augmentation techniques and batch processing strategies of random batch resizing and normalization are used to expand the training sample set, and a momentum optimizer combined with cosine decay and linear warm-up strategies is used to adjust the learning rate.

[0047] The beneficial technical effects of this application are:

[0048] The present application discloses a package specification detection method based on shape fitting. The method uses RGB images and depth data combined with a target detection model to extract package point cloud data, and then performs shape fitting construction on the package point cloud data according to different reference geometric shapes and performs a matching degree evaluation, so as to finally select the reference geometric shape with the highest confidence to determine the package shape and obtain the specification parameters. The method can robustly process various point cloud data of different qualities. By combining image algorithms and point cloud processing technology, it can quickly and accurately identify various standard and special-shaped package specifications, significantly improving the sorting efficiency and scope of application of logistics automation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a method flow chart of a package specification detection method according to an embodiment of the present application.

[0050] Figure 2 It is a schematic diagram of constructing a candidate cylindrical shape using two selected sample points.

[0051] Figure 3 It is a schematic diagram of constructing a spherical shape using two selected sample points. DETAILED DESCRIPTION

[0052] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0053] This application discloses a package specification detection method based on shape fitting, please refer to Figure 1 As shown in the flowchart, the package specification detection method includes:

[0054] Step 1: Use the RGBD smart depth camera to obtain the RGB image and depth data of the parcel detection area. The RGBD smart depth camera is suspended above the parcel detection area and vertically downward toward the parcel detection area to obtain the RGB image and depth data. In the actual application scenario, the parcel specification detection operation is performed during the parcel sorting process, so the RGBD smart depth camera is suspended above the sorting belt and obtains the RGB image and depth data at the sorting belt.

[0055] The field of view of an RGBD smart depth camera is usually large, so the entire field of view of the RGBD smart depth camera is usually not used as the package detection area. Instead, the package detection area, that is, the area where the package will appear, is pre-calibrated, and then the RGB image and depth data of the area of ​​interest, that is, the package detection area, are intercepted from the RGB image and depth data directly obtained by the RGBD smart depth camera, thereby filtering out background data and reducing the amount of subsequent data processing.

[0056] Step 2: Use the target detection model to perform target detection on the RGB image of the package detection area to determine the package area where the package to be detected is located.

[0057] The target detection model is pre-trained. The target detection model can be used to detect and locate the package in the RGB image of the package detection area, so as to locate the target detection frame of the package to be detected and thus determine the package area where the package to be detected is located.

[0058] The target detection model used in this step is designed based on the PP-PicoDet series target detection model. This model is the latest lightweight model that not only has excellent detection performance, but also can be efficiently deployed on mobile devices. This target detection model uses a lightweight LCNet neural network as the feature extraction layer. Its features include a scaling factor of 0.35 and feature maps of three different scales. It optimizes computational efficiency by adjusting the depth and width of the network. This target detection model uses the PicoHeadV2 backbone network as the target detection head structure. It is an efficient feature extraction structure with a convolutional feature extraction module called PicoFeat at its core. This module is capable of processing input feature maps with 96 channels and processes them through two convolutional layers while keeping the number of channels of the output feature map unchanged. This target detection model also combines the LCPAN module as a feature extraction module, which can output feature maps of 96 channels, providing rich feature representation for subsequent target detection tasks. In order to enhance the expressiveness of features, batch normalization and squeeze-and-excitation (SE) channel attention mechanisms are used. In addition, the module also supports multi-scale feature fusion, and effectively integrates features of different scales through the Feature Pyramid Network (FPN). Category prediction and bounding box regression share the same feature representation. This design aims to reduce the number of model parameters while maintaining high performance.

[0059] When training the target detection model, a certain amount of package sample data is first collected to construct a training sample set. The collected package sample data includes standard packages and special-shaped packages of various shapes and sizes. Then, data enhancement technology and batch processing strategies of random adjustment of image size and normalization are used to expand the training sample set. The data enhancement technologies used include random cropping, random flipping and color distortion. Each batch is set to 12 samples, and data shuffling and discarding the last incomplete batch are enabled to enhance the robustness of the model to different image changes. In addition, 6 working threads are set to process data in parallel to improve training efficiency. In the specific implementation, in the setting of training parameters, the initial basic learning rate is set to 0.01, and adjusted through two scheduling strategies: first, the linear warm-up strategy, the first 500 steps of training gradually increase the learning rate from 0.1 times the basic learning rate to full speed to stabilize the performance in the early stage of training; followed by the cosine decay strategy, which simulates the decay trend of the cosine function in the next training cycle to gradually reduce the learning rate, helping the model to fine-tune in the later stage of training and avoid overfitting. The optimizer selected is the Momentum Optimizer, whose momentum parameter is 0.9, which helps to accelerate convergence and reduce oscillations during training. In addition, in order to control the complexity of the model and prevent overfitting, L2 regularization is introduced with a regularization factor of 0.00004. By imposing a penalty on the sum of squares of weight parameters, the model is prompted to learn a more concise representation. The training process is set to 600 epochs to ensure that the model has enough time to learn and optimize its performance. During the evaluation and testing phases, the model ensures the accuracy of the evaluation results through data processing processes such as image decoding, scaling, normalization, and batch filling.

[0060] Step 3, then extract the depth data of the package area and convert it into the package point cloud data.

[0061] After extracting the target detection frame of the package to be detected in step 2 to determine the package area where the package to be detected is located, the depth data in the package area can be extracted, and then combined with the internal parameter conversion of the RGBD smart depth camera to obtain the package point cloud data.

[0062] (1) In order to improve the efficiency of subsequent data processing and reduce the interference caused by noise data, in one embodiment, the depth data in the package area is first pre-processed and then converted into package point cloud data. The pre-processing of the depth data in the package area includes:

[0063] The depth data of the wrapped area is extracted, and the depth data that is much greater than the hanging height of the RGBD intelligent depth camera is filtered out, and the edges of the remaining depth data are morphologically processed to obtain the depth data in the wrapped area after noise is removed.

[0064] Then, a statistical analysis is performed on the depth data after noise removal in the package area, and the height estimation value of the package to be detected is obtained according to the reference plane position of the calibrated package detection area. The depth data whose degree of deviation from the height estimation value in the depth data after noise removal in the package area reaches the deviation threshold is filtered out to obtain the processed depth data of the package area, and the processed depth data of the package area is converted into the package point cloud data.

[0065] The reference plane position of the package detection area is pre-calibrated by using an RGBD smart depth camera to obtain depth data when no package is set in the package detection area. Then, the internal parameters of the RGBD smart depth camera are used to convert the depth data when no package is set in the package detection area into three-dimensional point cloud data, which is then fitted through the least squares algorithm to obtain the reference plane position of the package detection area.

[0066] (2) In order to further improve the efficiency of subsequent data processing and reduce the interference caused by noise data, in one embodiment, the three-dimensional point cloud data obtained by converting the depth data of the package area using the internal parameters of the RGBD smart depth camera is not directly used as the package point cloud data, but further post-processing operations are also performed:

[0067] After converting the depth data of the enclosed area into three-dimensional point cloud data using the internal parameters of the RGBD smart depth camera, equidistant sampling is first performed to obtain the initial point cloud data. This operation helps reduce the amount of calculation, and the sampling distance can be customized.

[0068] Then, radius filtering is performed. Specifically, the point clouds whose neighborhood point clouds in the sampled initial point cloud data are less than the number threshold are filtered out. The radius filtering is completed. The neighborhood point clouds of each point cloud include all point clouds within the filtering radius of the point cloud. In an example, the filtering radius is set to 3cm and the number threshold is 5. If the number of other point clouds within the 3cm radius of a point cloud is less than 5, the point will be filtered out.

[0069] Further perform statistical outlier filtering, specifically: for any point cloud in the initial point cloud data that completes the radius filtering, respectively calculate the average value μ and the standard deviation σ of the Euclidean distance between the point cloud and its nearest K point clouds, and obtain the distance threshold max=μ+ε*σ. When the average value μ of the Euclidean distance between the point cloud and its nearest K point clouds is greater than max, filter out the point cloud, otherwise keep the point cloud, ε is the threshold coefficient. In one example, the threshold coefficient ε=2.5, and K is 8.

[0070] After traversing all the point clouds in the initial point cloud data of the radius filter, the parcel point cloud data is obtained. The above radius filtering and statistical outlier filtering can identify and remove noise or abnormal points in the data set. Through two-step filtering, higher-quality parcel point cloud data can be obtained, which is conducive to improving the accuracy and efficiency of subsequent processing tasks.

[0071] Step 4: randomly select multiple point clouds from the package point cloud data as sample points, and fit multiple reference geometric shapes based on the selected sample points.

[0072] In practice, most packages are rectangular in shape. According to actual application scenarios, common special-shaped packages mainly include cylindrical barrel packages and spherical packages. Therefore, in one embodiment, these package shapes are mainly detected in a targeted manner. Therefore, the reference geometric shapes in this embodiment include plane shapes, cylindrical shapes, and spherical shapes. The methods for fitting these different reference geometric shapes based on the selected sample points are different, which are introduced as follows:

[0073] (1) Fitting the plane shape based on the selected sample points

[0074] First, three non-collinear point clouds are randomly selected from the parcel point cloud data as sample points p1, p2, and p3;

[0075] Use vector cross product to calculate the plane normal vector of the candidate plane formed by sample points p1, p2 and p3 in, is the vector from sample point p1 to sample point p2, is the vector from sample point p1 to sample point p3.

[0076] When the angle between the point cloud normal vector of the sample point p1 and the plane normal vector n of the candidate plane is less than the angle threshold β1, and the angle between the point cloud normal vector of the sample point p2 and the plane normal vector n of the candidate plane is less than the angle threshold β1, and the angle between the point cloud normal vector of the sample point p3 and the plane normal vector n of the candidate plane is less than the angle threshold β1, it is determined that the sample points p1, p2 and p3 all meet the direction constraints of the constructed candidate plane, and the candidate plane formed by the sample points p1, p2 and p3 is used as the fitted plane shape. Otherwise, re-execute the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as sample points p1, p2 and p3. The angle threshold β1 is set by default.

[0077] (2) Fitting the cylindrical shape based on the selected sample points

[0078] Please combine Figure 2 , randomly select two point clouds from the package point cloud data as sample points p4 and p5.

[0079] First, the vector product of the point cloud normal vector n4 of the sample point p4 and the point cloud normal vector n5 of the sample point p5 is calculated to obtain the normal vector, and then the projection plane Pv perpendicular to the normal vector is determined. The straight line L4 where the point cloud normal vector n4 of the sample point p4 is located is projected onto the projection plane Pv to obtain the projection straight line L'4, and the straight line L5 where the point cloud normal vector n5 of the sample point p5 is located is projected onto the projection plane Pv to obtain the projection straight line L'5. The intersection point c of the projection straight line L'4 and the projection straight line L'5 on the projection plane Pv is determined, and the straight line passing through the intersection point c and along the normal vector is the candidate central axis ν.

[0080] Determine the projection point p'4 of the sample point p4 on the projection plane Pv, and determine the projection point p'5 of the sample point p5 on the projection plane Pv, and then calculate the average of the Euclidean distances between the intersection point c and the projection point p'4 and the projection point p'5 as the candidate cylinder cross-section radius r. Then, according to the candidate central axis ν and the candidate cylinder cross-section radius r, a candidate cylinder shape with an infinite length can be determined, such as Figure 2 Shown by dotted line.

[0081] When the difference between the Euclidean distance between the projection point p'4 and the intersection point c (that is, the Euclidean distance between the sample point p4 and the candidate central axis ν) and the radius r of the cross-section of the candidate cylinder is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c (that is, the Euclidean distance between the sample point p5 and the candidate central axis ν) and the radius r of the cross-section of the candidate cylinder is less than the distance threshold λ1, it is determined that the sample point p4 and the sample point p5 are both near the surface of the constructed candidate cylindrical shape, and thus it is determined that the sample point p4 and the sample point p5 both meet the distance constraint. The distance threshold λ1 is set by default.

[0082] When the angle between the vector from the intersection point c to the projection point p'4 and the point cloud normal vector n4 of the sample point p4 is less than the angle threshold β2, and the angle between the vector from the intersection point c to the projection point p'5 and the point cloud normal vector n5 of the sample point p5 is less than the angle threshold β2, it can be determined that both the sample point p4 and the sample point p5 meet the direction constraint. The angle threshold β2 is customizable.

[0083] When it is determined that both sample point p4 and sample point p5 satisfy the distance constraint and direction constraint respectively, it indicates that the candidate cylindrical shape is reasonable, and the cylindrical shape determined by the candidate central axis ν and the candidate cylindrical cross-section radius r is used as the final fitted cylindrical shape. Otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 is re-executed.

[0084] (3) Fitting the spherical shape based on the selected sample points

[0085] Please combine Figure 3, randomly select two point clouds from the package point cloud data as sample points p6 and p7, and calculate the straight line L6 passing through the sample point p6 and along the direction of the point cloud normal vector n6 of the sample point p6, and calculate the straight line L7 passing through the sample point p7 and along the direction of the point cloud normal vector n7 of the sample point p7.

[0086] Calculate the shortest line segment H6H7 between the straight line L6 and the straight line L7, and use the midpoint of the shortest line segment H6H7 as the candidate sphere center O. The shortest line segment H6H7 is perpendicular to the straight line L6 with its foot being H6, and the shortest line segment H6H7 is perpendicular to the straight line L7 with its foot being H7.

[0087] The average of the Euclidean distances between the candidate sphere center O and the sample point p6 and the sample point p7 is taken as the candidate sphere radius R. A candidate sphere can be constructed based on the candidate sphere center O and the candidate sphere radius R.

[0088] When the difference between the Euclidean distance between the candidate sphere center O and the sample point p6 and the radius R of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center O and the sample point p7 and the radius R of the candidate sphere is less than the distance threshold λ2. It is determined that the sample points p6 and p7 are both near the surface of the constructed candidate sphere, and thus it is determined that the sample points p6 and p7 both meet the distance constraint. The distance threshold λ2 can be customized.

[0089] When the angle between the vector from the candidate sphere center O to the sample point p6 and the point cloud normal vector n6 of the sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center O to the sample point p7 and the point cloud normal vector n7 of the sample point p7 is less than the angle threshold β3, it can be determined that both the sample point p6 and the sample point p7 meet the direction constraint. The angle threshold β3 is custom set.

[0090] When it is determined that both sample point p6 and sample point p7 satisfy the distance constraint and direction constraint respectively, it means that the candidate sphere is reasonable. Then the sphere shape currently determined according to the candidate sphere center O and the candidate sphere radius R is used as the final fitted sphere shape. Otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 is re-executed.

[0091] Step 5, evaluate the degree of match between the other parcel point cloud data and each fitted reference geometry, and obtain the confidence of each fitted reference geometry. The higher the degree of match between the other parcel point cloud data and the reference geometry, the higher the confidence, indicating that the parcel point cloud data is more consistent with the characteristics of the reference geometry. The method for evaluating the degree of match between the other parcel point cloud data and each reference geometry is similar, mainly including the following steps to evaluate from multiple dimensions, for any reference geometry:

[0092] (1) All point clouds that meet the geometric constraints of the reference geometric shape are selected from the parcel point cloud data to form a point cloud set P ψ The specific screening methods under the three different reference geometric shapes are different, which are introduced as follows:

[0093] For the fitted plane shape, when the angle between the point cloud normal vector of any point cloud in the wrapped point cloud data and the plane normal vector of the fitted plane shape is less than the angle threshold β1, it is determined that the point cloud meets the geometric constraints of the plane shape, otherwise it is determined that the point cloud does not meet the geometric constraints of the plane shape, so that the point cloud set P that meets the geometric constraints of the plane shape can be screened out. ψ .

[0094] For the fitted cylindrical shape, for any point cloud in the wrapped point cloud data, first determine the projection plane where the point cloud is located that is perpendicular to the central axis of the fitted cylindrical shape, and then determine the intersection of the projection plane and the central axis. When the angle between the vector from the intersection to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape (the distance is the distance from the intersection of the projection plane where the point cloud is located and the central axis to the point cloud) is less than the cross-sectional radius of the cylinder, it is determined that the point cloud meets the geometric constraints of the cylindrical shape, otherwise it is determined that the point cloud does not meet the geometric constraints of the cylindrical shape, so that a point cloud set P that meets the geometric constraints of the cylindrical shape can be screened out. ψ .

[0095] For the fitted spherical shape, for any point cloud in the wrapped point cloud data, when the angle between the vector from the center of the fitted spherical shape to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted spherical shape is less than the radius of the fitted sphere, it is determined that the point cloud meets the geometric constraints of the spherical shape, otherwise it is determined that the point cloud does not meet the geometric constraints of the spherical shape, so that the point cloud set P that meets the geometric constraints of the spherical shape can be screened out. ψ .

[0096] (2) Point cloud set P ψ The point clouds in the cluster are clustered based on the location, and the category with the largest number of point clouds is extracted to obtain the largest connected point cloud set P max .

[0097] (3) According to the maximum connected point cloud set P max The number of point clouds included determines the confidence, that is, only the point cloud set P with geometric constraints is considered in the end ψ The point clouds that form the largest connected components of the shape are selected to ensure the integrity and continuity of the shape. The calculation method of different point cloud quantities and confidence levels can be customized.

[0098] Step 6: Calculate the specification parameters of the package to be inspected based on the package point cloud data according to the reference geometry with the highest confidence.

[0099] When the reference geometric shape with the highest confidence is a plane shape, it can be determined that the package shape of the package to be detected is a rectangle. Then the size of the package to be detected can be further determined, including: determining the maximum connected point cloud set P max The distribution area of ​​the point cloud in the fitted plane shape and the distribution area along the plane normal vector direction of the plane shape are obtained to obtain the three-dimensional size information as the package size. The specific method of calculating the three-dimensional size information according to the distribution situation can refer to the existing practice, which will not be repeated here.

[0100] When the reference geometric shape with the highest confidence is a cylinder, it can be determined that the package shape of the package to be detected is a cylinder. The cross-sectional dimensions of the cylinder can be determined, but the length of the cylinder cannot be determined. In this case, the maximum connected point cloud set P can be further determined. max The distribution area of ​​the point cloud in the figure along the central axis of the fitted cylindrical shape is used to determine the length of the cylinder, and then the cross-sectional radius of the cylinder whose bottom radius is the fitted cylindrical shape and the length of the cylinder is the calculated length of the cylinder. The specific method of calculating the length of the cylinder according to the distribution situation can refer to the existing practice and will not be repeated here.

[0101] When the reference geometric shape with the highest confidence is a spherical shape, it can be determined that the package shape of the package to be detected is a spherical shape, and the package radius can also be determined to be the spherical radius of the fitted spherical shape.

[0102] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A package specification detection method based on shape fitting, characterized in that: The package specification detection method comprises: Use RGBD intelligent depth camera to obtain RGB image and depth data of the package detection area; The target detection model is used to perform target detection on the RGB image to determine the parcel area where the parcel to be detected is located, and then the depth data of the parcel area is extracted and converted to obtain the parcel point cloud data; A plurality of point clouds are randomly selected from the parcel point cloud data as sample points, and a plurality of reference geometric shapes are respectively fitted based on the selected sample points; Evaluate the degree of matching between other parcel point cloud data and each fitted reference geometry to obtain the confidence of each fitted reference geometry; the higher the degree of matching between other parcel point cloud data and the reference geometry, the higher the confidence; The specification parameters of the package to be detected are calculated based on the package point cloud data according to the reference geometric shape with the highest confidence.

2. The package specification detection method according to claim 1, characterized in that: The confidence level for each reference geometry that is fitted includes: All point clouds that meet the geometric constraints of the reference geometric shape are selected from the parcel point cloud data to form a point cloud set P ψ ; Point cloud collection P ψ The point clouds in the cluster are clustered based on the location, and the category with the largest number of point clouds is extracted to obtain the largest connected point cloud set P max ; According to the maximum connected point cloud set P max The number of included point clouds determines the confidence level.

3. The package specification detection method according to claim 2, characterized in that: The reference geometric shapes include plane shapes, cylindrical shapes, and spherical shapes. The geometric constraints for detecting whether any point cloud in the wrapped point cloud data conforms to the reference geometric shapes include: For the fitted plane shape, when the angle between the point cloud normal vector of the point cloud and the plane normal vector of the fitted plane shape is less than the angle threshold β1, it is determined that the point cloud meets the geometric constraints of the plane shape; otherwise, it is determined that the point cloud does not meet the geometric constraints of the plane shape; For the fitted cylindrical shape, determine the intersection of the projection plane perpendicular to the central axis of the fitted cylindrical shape and the central axis where the point cloud is located; when the angle between the vector from the intersection to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape is less than the cross-sectional radius of the cylinder, determine that the point cloud meets the geometric constraints of the cylindrical shape; otherwise, determine that the point cloud does not meet the geometric constraints of the cylindrical shape; For the fitted spherical shape, when the angle between the vector from the center of the fitted spherical shape to the point cloud and the point cloud normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted spherical shape is less than the radius of the sphere, it is determined that the point cloud meets the geometric constraints of the spherical shape; otherwise, it is determined that the point cloud does not meet the geometric constraints of the spherical shape.

4. The package specification detection method according to claim 1, characterized in that: The reference geometry includes a plane shape, and the plane shape fitted based on the selected sample points includes: Randomly select three non-collinear point clouds from the parcel point cloud data as sample points p1, p2 and p3; Use vector cross product to calculate the plane normal vector of the candidate plane formed by sample points p1, p2 and p3 in, is the vector from sample point p1 to sample point p2, is the vector from sample point p1 to sample point p3; When the angles between the point cloud normal vectors of the sample points p1, p2 and p3 and the plane normal vector n of the candidate plane are all less than the angle threshold β1, the candidate plane formed by the sample points p1, p2 and p3 is used as the fitted plane shape, otherwise the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as the sample points p1, p2 and p3 is re-executed.

5. The package specification detection method according to claim 1, characterized in that: The reference geometric shape includes a cylindrical shape. The cylindrical shape is fitted based on the selected sample points and the geometric characteristic parameters of the fitted cylindrical shape are determined to include: Randomly select two point clouds from the parcel point cloud data as sample points p4 and p5; Calculate the vector product of the point cloud normal vector n4 of the sample point p4 and the point cloud normal vector n5 of the sample point p5 to obtain the normal vector of the candidate central axis ν, and determine the projection plane Pv perpendicular to the normal vector of the candidate central axis ν; Determine the projection line L'4 of the line L4 where the point cloud normal vector n4 of the sample point p4 lies on the projection plane Pv, determine the projection line L'5 of the line L5 where the point cloud normal vector n5 of the sample point p5 lies on the projection plane Pv, and determine the intersection point c of the projection line L'4 and the projection line L'5, and take the line along the normal vector of the candidate central axis ν and passing through the intersection point c as the candidate central axis ν; Determine the projection point p'4 of the sample point p4 in the projection plane Pv, determine the projection point p'5 of the sample point p5 in the projection plane Pv, and take the average of the Euclidean distances between the intersection point c and the projection point p'4 and the projection point p'5 as the candidate cross-section radius of the cylinder; When the difference between the Euclidean distance between the projection point p'4 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the angle between the vector from the intersection point c to the projection point p'4 and the point cloud normal vector n4 of the sample point p4 is less than the angle threshold β2, and the angle between the vector from the intersection point c to the projection point p'5 and the point cloud normal vector n5 of the sample point p5 is less than the angle threshold β2, the cylindrical shape determined according to the candidate central axis and the candidate cylinder cross section radius is used as the fitted cylindrical shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 is re-executed.

6. The package specification detection method according to claim 1, characterized in that: The reference geometric shape includes a spherical shape. The spherical shape is fitted based on the selected sample points and the geometric characteristic parameters of the fitted spherical shape are determined to include: Randomly select two point clouds from the package point cloud data as sample points p6 and p7, and calculate the straight line L6 passing through the sample point p6 and along the direction of the point cloud normal vector n6 of the sample point p6, and calculate the straight line L7 passing through the sample point p7 and along the direction of the point cloud normal vector n7 of the sample point p7; Calculate the shortest line segment between the straight line L6 and the straight line L7, and use the midpoint of the shortest line segment as the candidate sphere center; The average of the Euclidean distances between the candidate sphere center and the sample point p6 and the sample point p7 is taken as the candidate sphere radius; When the difference between the Euclidean distance between the candidate sphere center and the sample point p6 and the radius of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center and the sample point p7 and the radius of the candidate sphere is less than the distance threshold λ2, and the angle between the vector from the candidate sphere center to the sample point p6 and the point cloud normal vector n6 of the sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center to the sample point p7 and the point cloud normal vector n7 of the sample point p7 is less than the angle threshold β3, the spherical shape determined according to the candidate sphere center and the candidate sphere radius is used as the fitted spherical shape, otherwise the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 is re-executed.

7. The package specification detection method according to claim 3, characterized in that: The specification parameters of the package to be detected are calculated based on the package point cloud data according to the reference geometric shape with the highest confidence, including: When the reference geometric shape with the highest confidence is a plane shape, determine the maximum connected point cloud set P max The distribution area of ​​the point cloud in the plane shape obtained by fitting and the distribution area in the direction of the plane normal vector of the plane shape are obtained to obtain three-dimensional size information, and the specification parameters of the package to be detected include: the package shape is a cube, and the package size is the calculated three-dimensional size information; When the reference geometric shape with the highest confidence is a cylinder, determine the maximum connected point cloud set P max The distribution area of ​​the point cloud in the image along the central axis of the fitted cylindrical shape is used to determine the length of the cylinder, and the specification parameters of the package to be detected include: the package shape is a cylindrical shape, the bottom radius of the package is the cross-sectional radius of the cylinder of the fitted cylindrical shape, and the package length is the calculated cylindrical length; When the reference geometric shape with the highest confidence is a spherical shape, the specification parameters of the package to be detected include: the package shape is a spherical shape, and the package radius is the spherical radius of the spherical shape obtained by fitting.

8. The package specification detection method according to claim 1, characterized in that: The converted package point cloud data includes: The internal parameters of the RGBD intelligent depth camera are used to convert the depth data of the wrapped area into three-dimensional point cloud data and perform equidistant sampling to obtain the initial point cloud data; The point clouds whose neighborhood point clouds in the sampled initial point cloud data are less than the number threshold are filtered out to complete the radius filtering, and the neighborhood point clouds of each point cloud include all the point clouds within the filtering radius of the point cloud; For any point cloud in the initial point cloud data that has completed the radius filtering, the average value μ and the standard deviation σ of the Euclidean distance between the point cloud and its nearest K point clouds are calculated respectively, and the distance threshold max=μ+ε*σ is obtained. When the average value μ of the Euclidean distance between the point cloud and its nearest K point clouds is greater than max, the point cloud is filtered out, otherwise the point cloud is retained, and ε is the threshold coefficient; After traversing all point clouds in the initial point cloud data after radius filtering, the wrapped point cloud data is obtained.

9. The package specification detection method according to claim 8, characterized in that: The depth data of the parcel area is extracted and converted into the parcel point cloud data including: Extract the depth data of the wrapped area, filter out the depth data that is much higher than the hanging height of the RGBD smart depth camera, and perform morphological processing on the edges of the remaining depth data to obtain the depth data after removing noise in the wrapped area. A statistical analysis is performed on the depth data after noise removal in the package area, and a height estimation value of the package to be detected is obtained according to the reference plane position of the calibrated package detection area. The depth data whose degree of deviation from the height estimation value in the depth data after noise removal in the package area reaches a deviation threshold is filtered out to obtain the processed depth data of the package area, and then the processed depth data of the package area is converted into the package point cloud data.

10. The package specification detection method according to claim 1, characterized in that: The target detection model used is designed based on the PP-PicoDet series target detection model. The target detection model uses a lightweight LCNet neural network as a feature extraction layer, a PicoHeadV2 backbone network as a target detection head structure, and combines an LCPAN module. During the model training process, data augmentation techniques and batch processing strategies of random batch resizing and normalization are used to expand the training sample set, and a momentum optimizer combined with cosine decay and linear warm-up strategies is used to adjust the learning rate.

Images