A method of measuring the volume of a logistics package

Abstract

The present application relates to the technical field of machine vision measurement, and discloses a logistics parcel volume measurement method, which comprises the following steps: firstly, a measurement reference plane is constructed; then, whether there is a parcel is judged after excluding arm interference; in the case that there is a parcel, the parcel is distinguished as a thin parcel or a non-thin parcel; in the case that the parcel is a non-thin parcel, whether the parcel is a regular parcel or an irregular parcel is judged; finally, volume measurement is carried out according to the classification. Compared with the prior art, the present application overcomes the problems of the prior art, such as interference of light texture, difficulty in thin parcel detection and lack of unified identification framework, realizes unified identification of regular parcels, irregular parcels and thin parcels, volume measurement by classification, and improves the sorting efficiency and adaptability of the logistics automation system.

Description

Technical Field

[0001] This invention relates to the field of machine vision measurement technology, and more specifically, to a method for measuring the volume of logistics parcels. Background Technology

[0002] With the development of 3D vision perception technology and the reduction of hardware costs, various non-contact measurement technologies have been gradually introduced into logistics automation systems to achieve automatic acquisition of package volume. Currently, the industry mainly uses sensors such as structured light, line laser scanning, or measuring light curtains to collect package size information. Among them, line laser scanning and measuring light curtain solutions are mostly geared towards dynamic transportation scenarios, usually relying on conveyor belt coordination, installation calibration, and spatial layout design, and the system deployment has certain requirements for site conditions. In contrast, structured light depth cameras can quickly acquire package surface depth information under single-frame or few-frame conditions, with advantages such as fast measurement speed and relatively simple system structure, making them very suitable for logistics operation scenarios with limited space and static package placement. However, in actual logistics scenarios, package types are complex and diverse, covering various types such as regular cardboard boxes, soft bags, irregularly shaped items, and thin flat packages, and various types of packages have significant differences in shape contours, surface flatness, and height characteristics. Existing volume measurement algorithms often assume regular boxes as idealized scenarios and rely on a single geometric model or fixed measurement process. In practical applications, these algorithms are prone to problems such as unstable contour extraction, boundary fitting deviations, missing depth information, and decreased measurement accuracy when dealing with various types of logistics packages. Furthermore, traditional two-dimensional image-based methods for logistics package recognition are highly susceptible to the influence of package surface patterns, similar colors between the package and the weighing platform, and changes in ambient lighting, resulting in poor robustness. Moreover, existing depth detection methods are insensitive to the height signal of thin packages, making reliable detection difficult. Additionally, existing methods typically design independent recognition strategies for the three types of packages mentioned above, lacking a unified recognition framework. Therefore, researching a volume visual measurement system for multiple types of logistics packages has significant engineering practical implications for improving generalization ability and accuracy in practical measurement scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a method for measuring the volume of logistics parcels, which solves the above-mentioned technical problems.

[0004] A method for measuring the volume of logistics parcels, used to measure the volume of parcels of different shapes, includes the following steps:

[0005] S1: System Input: Acquire images of the measurement area through a 3D vision sensor, and upload the depth map and color map of the image to the measurement system;

[0006] S2: Image preprocessing: Filter the depth map, correct the distortion of the color image, and align the depth map and color image to RGB-D.

[0007] S3: Measurement reference plane construction: Set up a plane fitting ROI and a measurement ROI. Based on the depth image information in the plane fitting ROI under no-load conditions, construct a measurement reference plane and perform coordinate correction on the point cloud data to unify the point cloud to the world coordinate system with the measurement reference plane as the reference.

[0008] S4: Package volume measurement:

[0009] S41: Detect arm interference within the measured ROI to identify whether there is operator hand interference;

[0010] If yes, then pause the measurement; if no, proceed to step S42.

[0011] S42: Identify whether there are packages within the measured ROI;

[0012] If not, the process ends; if yes, proceed to step S43.

[0013] S43: Identify whether the package is a thin or non-thin package;

[0014] If it is a thin package, proceed to step S431: calculate the volume of the thin package using the first calculation method;

[0015] If it is not a thin package, proceed to step S432: identify regular packages and irregular packages;

[0016] If it is a regular package, proceed to step S4321: calculate the volume of the regular package using the second calculation method;

[0017] If the package is irregular, proceed to step S4322: calculate the volume of the irregular package using the third calculation method.

[0018] According to one embodiment of the present invention, the construction of the measurement reference plane in step S3 specifically includes the following steps:

[0019] S31: Obtain the depth map within the ROI under no-load conditions and back-project the depth map into a 3D point cloud;

[0020] S32: Use the RANSAC algorithm to extract in-plane points from the 3D point cloud and obtain a candidate in-plane point set;

[0021] S33: Principal component analysis (PCA) is used to refine the plane parameters of the candidate point set in the plane. The centroid and covariance matrix of the candidate point set in the plane are calculated. The covariance matrix is ​​decomposed into eigenvalues. The eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the measurement platform plane. The plane equation parameters are determined by combining the centroid.

[0022] S34: Using the Z-axis direction of the measurement coordinate system of the 3D vision sensor as the target direction, construct a rotation matrix based on the plane normal vector of the measurement platform obtained in step S33 and the Rodriguez rotation formula.

[0023] S35: Calculate the translation vector between the original point cloud centroid and the point cloud centroid after rotation by the rotation matrix. The rotation matrix and translation vector form a rigid body transformation matrix to perform spatial correction on the original point cloud, aligning the plane normal vector of the measurement platform with the Z-axis direction of the measurement coordinate system of the 3D vision sensor, and unifying the point cloud to the world coordinate system with the measurement reference plane as the reference.

[0024] According to one embodiment of the present invention, in step S41, identifying whether there is interference from the operator's hands specifically includes:

[0025] S411: Obtain the depth map within the measured ROI and identify pixels higher than the measurement reference plane as candidate target regions;

[0026] S412: Perform connected component analysis on the candidate target region to obtain the bounding rectangle and geometric features of each connected component;

[0027] S413: Calculate the following three criteria for each connected component:

[0028] a. Boundary proximity criterion: Calculate the minimum distance between the bounding rectangle of the connected region and the boundary of the measured ROI. If the minimum distance is less than the preset edge distance threshold, the boundary proximity feature is satisfied.

[0029] b. Aspect Ratio Criterion: Calculate the ratio of the major axis to the minor axis of the smallest circumscribed rectangle of the connected region. If the ratio exceeds the preset aspect ratio threshold, it is determined that the aspect ratio is abnormal.

[0030] c. Height distribution criterion: Calculate the dispersion of the height distribution of pixels within the connected domain. If the dispersion exceeds the preset height dispersion threshold, it is determined that the height distribution is abnormal.

[0031] S414: If any connected component satisfies at least two of the three criteria a, b, and c, then it is determined that there is arm interference in the current measurement area.

[0032] According to one embodiment of the present invention, step S42, identifying whether there is a package, specifically includes the following steps:

[0033] S421: Height Difference Map Construction: Obtaining a Baseline Depth Map under Unloaded Conditions Compared with the current depth map Calculate the height difference map of the current depth map relative to the reference depth map. ,in Represents the pixel coordinates of the image;

[0034] S422: Candidate Region Extraction: Set a minimum height threshold within the measured ROI Extract candidate pixels from the height difference map whose height values ​​are greater than a preset minimum height threshold to form a candidate mask. :

[0035] ,in, For height difference map The height difference at the location;

[0036] S423: Local One-Sample Z-Test: Count the total number of valid pixels N in the candidate mask and calculate the average height of all valid pixels within the candidate mask. : ;

[0037] S424: Constructing a one-sample Z-test statistic: Obtaining the standard deviation of the height noise of the reference plane under no-load conditions. Calculate the standard error Construct the one-sample Z-test statistic. ;

[0038] S425: If the single-sample Z-statistic is greater than the preset Z-test threshold, then it is determined that there is a package within the measured ROI; otherwise, it is determined that there is no package.

[0039] According to one embodiment of the present invention, step S43, identifying whether the package is a thin package or a non-thin package, specifically includes:

[0040] S431: Obtain the height values ​​of all valid pixels within the package area and form a height set;

[0041] S432: Extract the maximum height value from the height set. ;

[0042] S433: If If the thickness is less than or equal to the preset thin-packet thickness threshold h, the package is determined to be a thin package; if If the thickness exceeds the preset thin package thickness threshold h, the package is determined to be a non-thin package.

[0043] According to one embodiment of the present invention, the step S432 of identifying regular and irregular packages specifically includes:

[0044] S4321: Calculate the planar variance of the top surface of the non-thin package. When the planar variance is less than or equal to the preset planar variance threshold, the top surface of the non-thin package is determined to have flatness.

[0045] S4322: Calculate the area shrinkage of the top surface of the package. If the area shrinkage is less than the preset area shrinkage threshold, the package is determined to have regularity in its outline.

[0046] S4323: If a package satisfies both the flatness of its top surface and the regularity of its outline, it is determined to be a regular package; if neither criterion is satisfied, it is determined to be an irregular package.

[0047] According to one embodiment of the present invention, in step S4321, the calculation of the plane variance specifically includes:

[0048] S43211: Perform least-squares plane fitting on the effective pixels in the non-thin-bag target region and solve for the plane coefficients;

[0049] S43212: Calculate the residual of each pixel relative to the fitted plane;

[0050] S43212: Calculate the variance of the residuals as the plane variance.

[0051] According to one embodiment of the present invention, in step S4322, the calculation of the area shrinkage specifically includes:

[0052] Extract the effective pixel mask M based on the height threshold;

[0053] Apply a square structuring element with side length k to the mask M. Perform erosion calculations to obtain the eroded mask. : ; Calculate the area retention rate : Calculate the area shrinkage : .

[0054] According to one embodiment of the present invention, the volume calculation of the thin package using the first calculation method in step S431 specifically includes:

[0055] S4311: Acquire a single frame of unwrapped color image of the measurement platform as a background reference image, convert it to the CIELAB color space, and obtain the background Lab image. ;

[0056] S4312: Acquire the current frame color image containing the thin packet, convert it to the CIELAB color space, and obtain the current Lab image. ;in: ,in, Lab space image serving as background reference. This is the Lab space image of the current frame. This is a color space conversion from RGB to Lab.

[0057] S4313: Calculate the channel-wise difference image between the current Lab image and the background Lab image. ;in, This is the brightness difference value. The difference between the red and green axes is the red-green axis difference value. The difference between the yellow and blue opposing axes is represented by T, which is the transpose sign.

[0058] S4314: Weighted fusion of the difference images yields a grayscale difference image: ;

[0059] in ;in, , , These are the weighting coefficients for the three Lab channels, respectively. ;

[0060] S4315: Threshold segmentation is performed on the grayscale difference image to obtain the foreground mask. Connectivity analysis is performed on the foreground mask, and the connected component with the largest area is retained as the thin-packed target region.

[0061] S4316: Using a thin-bag target region mask, extract the corresponding thin-bag depth information from the RGB-D aligned depth map and back-project it into a 3D point cloud;

[0062] S4317: Transform the 3D point cloud into the world coordinate system through a spatial rigid body transformation, and project it onto the XY reference plane to obtain a 2D projected point set;

[0063] S4318: Calculate the convex hull for the two-dimensional projection point set, and use the rotating caliper algorithm to find the minimum area bounding rectangle. Use the long side and short side of the minimum area bounding rectangle as the length and width of the thin hull, respectively.

[0064] S4319: Calculate the height of the package and the volume of the package based on its length, width, and height.

[0065] The logistics parcel volume measurement method of this embodiment, in step S4321, specifically includes calculating the volume of a regular parcel using the second calculation method:

[0066] S43211: Extract depth pixels from the mask of the regular wrapping area and backproject them into a 3D point cloud. Transform it into the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the regular wrapping.

[0067] S43212: Morphological erosion of the mask covering the area to cause the edge area to shrink inward;

[0068] S43213: The RANSAC method is used to perform plane fitting on the point cloud of the eroded top surface to obtain the plane parameters of the top surface;

[0069] S43214: Calculate the distance from each point in the top surface point cloud of the rule-wrapped area to the top surface plane, identify points whose distance is greater than the preset outlier threshold as outliers and remove them, and obtain the effective point cloud and the filtered wrapping mask image.

[0070] S43215: Extract the maximum outer contour of the filtered wrapping mask image, convert the contour pixels to the world coordinate system and project them onto the XY reference plane to form a two-dimensional contour point set;

[0071] S43216: Find the minimum directed bounding rectangle for the two-dimensional contour point set, and use the long side and short side of the minimum directed bounding rectangle as the length and width of the regular wrapping, respectively.

[0072] S43217: Calculate the height of the rule package and the volume of the rule package based on its length, width, and height.

[0073] According to one embodiment of the present invention, the volume calculation of the irregular package using the third calculation method in step S4322 specifically includes:

[0074] S43221: Extract depth pixels from the mask of the irregular package area and backproject them into a 3D point cloud. Transform the point cloud to the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the irregular package.

[0075] S43222: Project the point cloud of the irregularly wrapped top surface onto the XY reference plane to obtain a two-dimensional projection point set;

[0076] S43223: Calculate the convex hull for the two-dimensional projection point set. Use the rotating caliper algorithm to traverse each edge of the convex hull. Use the direction of each edge as the main axis direction of the candidate bounding rectangle. Calculate the area of ​​the corresponding minimum bounding rectangle one by one. Select the candidate rectangle with the smallest area as the minimum bounding rectangle.

[0077] S43224: Use the long and short sides of the smallest bounding rectangle as the length and width of the irregular wrapping, respectively;

[0078] S43225: Calculate the height of an irregular package and its volume based on its length, width, and height.

[0079] According to one embodiment of the present invention, the calculation steps for the package height include:

[0080] Obtain the set of valid point clouds within the package area any one of them To the measurement reference plane height The calculation formula is: ;

[0081] Traverse all valid point clouds to obtain the height set. Arrange the height sets in ascending order to obtain an ordered height sequence. ;

[0082] The 98th percentile of the ordered height sequence was selected as the estimated value of the package height. ,Right now:

[0083] Where n is the total number of valid point clouds, This indicates rounding down to the nearest integer.

[0084] Compared with the prior art, the logistics parcel volume measurement method of the present invention has the following advantages:

[0085] The present invention provides a method for measuring the volume of logistics parcels. First, a measurement reference plane is constructed. Then, after eliminating arm interference, it is determined whether a parcel exists. If a parcel exists, it is distinguished as a thin parcel or a non-thin parcel. If the parcel is non-thin, it is determined whether the surrounding parcel is regular or irregular. Finally, the volume is measured according to the classification. This method overcomes the problems of existing methods, such as interference from lighting and texture, difficulty in detecting thin parcels, and lack of a unified recognition framework. It achieves unified recognition and classified volume measurement of regular, irregular, and thin parcels, thereby improving the sorting efficiency and adaptability of logistics automation systems. Attached Figure Description

[0086] Figure 1 This is a flowchart illustrating the measurement process of the logistics parcel volume measurement method of the present invention.

[0087] Figure 2 Comparison images before and after calibration of the reference plane;

[0088] Figure 3 Example image of arm interference detection results;

[0089] Figure 4 Flowchart for package identification;

[0090] Figure 5 Here is a flowchart of the algorithm for measuring the volume of thin packages;

[0091] Figure 6 Flowchart for thin-package measurement;

[0092] Figure 7 Here is a flowchart of the algorithm for measuring the volume of a regular package;

[0093] Figure 8Flowchart for extracting the outline and calculating the length and width of a rule-wrapped object;

[0094] Figure 9 Flowchart of the algorithm for measuring the volume of irregular packages;

[0095] Figure 10 Flowchart for extracting the outline and calculating the length and width of irregular packages;

[0096] The implementation and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0097] The following drawings will disclose several embodiments of the present invention. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential. Furthermore, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0098] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0099] Furthermore, in this invention, the use of terms such as "first" and "second" is for descriptive purposes only and does not specifically refer to any order or sequence, nor is it intended to limit the invention. They are merely used to distinguish components or operations described using the same technical terms, and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but only if they are feasible for those skilled in the art. If a combination of technical solutions is contradictory or impossible to implement, such a combination should be considered nonexistent and not within the scope of protection claimed by this invention.

[0100] To further understand the content, features, and effects of this invention, the following embodiments are provided, and detailed descriptions are given below in conjunction with the accompanying drawings:

[0101] This invention's measurement method is applied to the scenario of measuring the volume of logistics parcels. The measurement height distance is generally between 0.7m and 1.5m, requiring processing of solid-color parcel surfaces and demanding millimeter-level depth accuracy. Therefore, the measurement system of this invention selects a depth camera as the 3D vision sensor. The depth camera selected in this invention employs active binocular infrared speckle structured light technology. Its core hardware components include two infrared cameras, an infrared speckle projector, and a color camera. It adopts the active binocular principle, is equipped with a 940nm infrared laser, and supports depth map, color map, and infrared map output. It has advantages such as high depth accuracy and strong resistance to ambient light interference, and can meet the measurement requirements of logistics parcel volume measurement scenarios. The depth camera is installed directly above the measurement platform, with its optical axis approximately perpendicular to the plane of the measurement platform, at a height of approximately 1m. The depth camera simultaneously acquires depth maps and color maps of the measurement area and transmits them to the processor via a network port.

[0102] The following details the specific steps of implementing the logistics parcel volume measurement method of the present invention using the measurement system of the present invention. The logistics parcel volume measurement method of the present invention includes the following steps:

[0103] S1: System Input: Acquire images of the measurement area through a 3D vision sensor, and upload the depth map and color map of the image to the measurement system;

[0104] The process of acquiring depth information using a 3D vision sensor can be summarized in the following three steps:

[0105] 1. Infrared speckle pattern projection:

[0106] An infrared laser projects a pseudo-random speckle pattern onto a target scene using a diffractive optical element (DOE). This speckle pattern possesses a high degree of randomness and uniqueness, enabling different regions to form discriminative texture features. The actively projected speckle pattern adds rich texture to the scene surface, effectively solving the problem that passive binocular vision cannot reliably perform stereo matching in texture-poor areas.

[0107] 2. Binocular stereo matching:

[0108] Two infrared cameras captured left and right infrared images, respectively, with speckle patterns superimposed. Since the speckle pattern is unique at different locations, a stereo matching algorithm can efficiently and accurately establish pixel-level correspondences between the left and right images. Let the horizontal coordinates of a pixel P in the left image on the left camera's image plane be... The horizontal coordinates of its corresponding point on the right camera image plane are: Then the disparity d at that point is defined as: The magnitude of the parallax value d is inversely proportional to the distance from the target point to the camera. The closer an object is to the camera, the greater the displacement difference between the left and right images, and the larger the parallax value; conversely, the farther away an object is, the smaller the parallax value.

[0109] 3. Depth calculation:

[0110] After the binocular camera has been calibrated, its focal length and baseline distance are known. Based on the principle of binocular stereo vision triangulation, the depth Z of the target point can be calculated using the following formula: Where f is the camera focal length in pixels; B is the binocular baseline length in millimeters; d is the disparity value in pixels; and Z is the depth value in millimeters. By calculating the disparity and depth values ​​for each pixel in the image, a depth map with the same resolution as the infrared image can be generated. Each pixel in the depth map stores the distance from the corresponding scene point to the camera.

[0111] S2: Image preprocessing: Filter the depth map, correct the distortion of the color image, and align the depth map and color image to RGB-D.

[0112] In other words, the raw data output by the depth camera needs to be preprocessed before entering the measurement algorithm to remove noise, eliminate outliers, and correct imaging distortion, thereby improving the accuracy and robustness of subsequent algorithms. The image preprocessing module mainly filters the depth map and corrects distortion in the color image to eliminate radial and tangential distortion. The depth map and color image are RGB-D aligned using the depth camera's intrinsic and extrinsic parameters to provide consistent image data for subsequent processing.

[0113] This invention employs a consistency-based filtering method to preprocess the depth map. The core idea of ​​this method is to effectively identify and remove isolated outlier points by statistically analyzing the proportion of pixels with significant depth differences within a local window. Within the local neighborhood of the depth map, if the depth value of a central pixel differs excessively from the depth values ​​of most of its valid neighboring pixels, this point is considered an outlier. Therefore, when the proportion of neighboring pixels whose depth value exceeds a specific difference threshold is greater than a set invalidity threshold, this pixel is determined to be outlier noise and marked as invalid.

[0114] The specific algorithm steps are as follows:

[0115] 1. Input the original depth map Filter window radius Depth difference threshold Invalid ratio threshold .

[0116] 2. For each valid pixel in the depth map (Right now Extracting data centered on a size of [missing information]. neighborhood window .

[0117] 3. Calculate the neighborhood window Depth difference between all valid pixels and the center pixel : ,in For neighborhood windows The i-th valid pixel Depth value at that location, For neighborhood windows The pixel coordinates of the i-th valid pixel.

[0118] 4. Statistical analysis shows that the depth difference within the neighborhood window exceeds a threshold. Number of pixels and the total number of effective pixels :

[0119] ,

[0120] .

[0121] 5. Calculate the proportion of depth inconsistencies within a local window. : .

[0122] 6. If the invalid ratio threshold is exceeded If the pixel is not a noise point, it is marked as a noise point and invalidated; otherwise, its original depth value is retained. : .

[0123] The color image output of a 3D vision sensor is the raw image without distortion correction. Distortion can cause straight lines to bend and shapes to deform. If the color image is used directly for RGB-D alignment or wrapping segmentation without processing, the accuracy of segmentation and contour extraction will be affected. Therefore, distortion correction of the color image is necessary.

[0124] The goal of distortion correction is to map pixels in a distorted image back to their correct positions in a distortion-free image. Since the distortion model is a forward mapping from distortion-free coordinates to distorted coordinates, directly reversing this process is difficult. In practical engineering, a reverse mapping method is typically used, where the coordinates of each pixel in the distortion-free image are... Calculate the coordinates in the corresponding distorted image. Then, values ​​are taken from the distorted image.

[0125] The specific algorithm steps are as follows:

[0126] 1. Convert the pixel coordinates of the distortion-free image Convert to normalized coordinates :

[0127] ;in, The horizontal pixel coordinates of the distortion-free image. The vertical pixel coordinates of the distortion-free image;

[0128] 2. Calculate the distorted normalized coordinates based on the distortion model. : ,

[0129] ,

[0130] ;

[0131] 3. Convert the distorted normalized coordinates back to pixel coordinates. : ;

[0132] 4. In the original distorted image, with Bilinear interpolation is performed on the coordinates to obtain values ​​as a distortion-free image. The pixel value at that location. This is the internal reference of the color camera; These are distortion coefficients, all read from the camera firmware.

[0133] The color camera and depth sensor in a depth camera often differ in their spatial installation location and imaging parameters, and each has its own independent imaging intrinsic parameters. This results in a one-to-one correspondence between the acquired color image and the depth map in pixel coordinates. To achieve the fusion of color and depth information, it is necessary to spatially align the RGB image and the depth map and unify their coordinate representation.

[0134] The basic idea of ​​RGB-D alignment is to use the calibration parameters between the color camera and the depth camera to project the 3D points generated in the depth map onto the imaging plane of the color camera, thus establishing a correspondence between depth information and color pixels.

[0135] Let the coordinates of a 3D point in the depth camera coordinate system be... Using the known extrinsic parameter transformation relationships, it can be transformed to the color camera coordinate system: ;

[0136] in, and Let represent the rotation matrix and translation vector between the depth camera and the color camera, respectively. Then, using the intrinsic parameter model of the color camera, the 3D point is projected onto the color image plane to obtain its pixel coordinates in the color image. After RGB-D alignment, the color pixels and depth data achieve accurate point-to-point correspondence.

[0137] S3: Measurement reference plane construction: Set up a plane fitting ROI and a measurement ROI. Based on the depth image information in the plane fitting ROI under no-load conditions, construct a measurement reference plane and perform coordinate correction on the point cloud data to unify the point cloud to a world coordinate system with the measurement reference plane as the reference.

[0138] Here, the plane fitting ROI is used to determine "where to take points to fit the measurement reference plane," that is, to perform plane fitting on the corresponding 3D point cloud within this image frame to obtain the plane equation parameter measurement; the measurement ROI is used to determine "in which range of the image to detect packages during subsequent volume measurement," that is, during the measurement stage, only within this frame are packages detected and contours extracted. In the embodiments of the present invention, the measurement reference plane is a plane equation in 3D space, used as a height reference benchmark, and the measurement ROI is a rectangular window in a 2D image, used as the data processing range. The purpose of constructing the reference plane is to eliminate the systematic errors introduced by camera mounting tilt and provide a unified geometric reference benchmark for subsequent height calculation.

[0139] In the field of computer vision, cameras typically employ an ideal pinhole imaging model to describe the imaging process from three-dimensional space to a two-dimensional image plane. This pinhole imaging model usually requires establishing multiple coordinate systems to describe the projection process of points in space, including the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system. Among these:

[0140] World coordinate system ( ): Describes the true geometric position of the target in three-dimensional space. Its axes represent spatial distances in millimeters (mm). The origin and coordinate axis directions can be set according to the specific application scenario.

[0141] Camera coordinate system ( ): with camera optical center With the origin as the point, The axis is aligned with the camera's optical axis. and The world coordinate system and camera coordinate system represent the horizontal and vertical directions of the image, respectively, forming a right-handed coordinate system. All physical quantities on each axis of this coordinate system are spatial distances, measured in millimeters (mm). The spatial relationship between the world coordinate system and the camera coordinate system can be described by rigid body transformation, and the transformation relationship can be expressed as: ;

[0142] in for Rotation matrix, for The translation vector and the translation vector together constitute the extrinsic parameters of the camera.

[0143] Image coordinate system ( The image coordinate system is located on the imaging plane, usually with the image center as the origin, and its coordinate axes are oriented in the same direction as those in the camera coordinate system. , The orientation remains consistent. This coordinate system is used to represent the position on the normalized imaging plane. Under the ideal pinhole imaging model, the projection relationship from the camera coordinate system to the image coordinate system can be expressed as: At this point, x and y are normalized coordinates.

[0144] Pixel coordinate system ( ): Describes the position of pixels in an image, with its origin usually located at the top left corner of the image, and its horizontal direction being... The axis, vertical direction is The axis is in pixels. The transformation relationship between the image coordinate system and the pixel coordinate system can be represented by the camera intrinsic parameter matrix: ;in, The focal lengths are the horizontal and vertical focal lengths. Using the coordinates of the main point, a complete mapping relationship from three-dimensional spatial points to two-dimensional image pixels can be established based on the above relationship.

[0145] In an embodiment of the present invention, the construction of the measurement reference plane specifically includes the following steps:

[0146] S31: Obtain the depth map within the plane-fitted ROI under no-load conditions, and back-project the depth map into a 3D point cloud;

[0147] Each pixel in the depth map contains not only its two-dimensional position in the image plane, but also its depth value along the optical axis (Z-axis) in the camera coordinate system. Based on the pinhole camera model, the pixels in the depth map can be back-projected into three-dimensional space to reconstruct the corresponding three-dimensional point cloud.

[0148] Let the coordinates of a pixel in the depth map be... Its corresponding depth value is Z. First, based on the intrinsic parameters of the depth camera's color sensor... Converting these coordinates to normalized image coordinates, the relationship can be expressed as: By combining the projection relationship from the camera coordinate system to the image coordinate system and the transformation relationship between the image coordinate system and the pixel coordinate system, and by combining the normalized coordinates with the depth value, the three-dimensional coordinates of the point in the camera coordinate system can be obtained. : As can be seen from the above relationship, each pixel with a valid depth value in the depth map can be mapped to a three-dimensional point in the camera coordinate system through back projection, and the set of all three-dimensional points constitutes the point cloud data.

[0149] S32: The RANSAC algorithm is used to extract in-plane points from the three-dimensional point cloud to obtain a candidate in-plane point set;

[0150] Let the point cloud set of the measurement area be... The general form of the plane equation is: ;in, Let be a plane normal vector, satisfying , This represents the signed distance from the plane to the origin. The goal of plane fitting is to estimate the parameters that best represent the plane of the measurement platform from point cloud data containing noise and outliers. .

[0151] In each iteration, the RANSAC algorithm randomly selects three non-collinear points, calculates the corresponding planar model, and counts the number of points in the point cloud whose distance to the plane is less than a given threshold as the evaluation metric. After K iterations, the model with the largest number of interior points is selected as the optimal fitting result, thus obtaining a candidate set of interior points in the plane.

[0152] The specific steps are as follows:

[0153] 1. From point clouds Three points were randomly selected from the middle. ;

[0154] 2. Calculate the plane parameters determined by these three points. ;

[0155] 3. Calculate the distance from all points in point cloud p to the plane. ;

[0156] 4. Set distance threshold Statistical satisfaction The points are designated as interior points, and the number of interior points is recorded.

[0157] 5. Repeat steps 1-4 K times, select the model with the most interior points as the best fit, and obtain the corresponding set of interior points. .

[0158] By iterating through the above process, the RANSAC algorithm can separate the set of interior points of the measurement platform plane even when there are a large number of non-planar points, providing a reliable data foundation for subsequent accurate fitting. Compared with the least squares method, the RANSAC method has stronger robustness to outliers and is suitable for the initial modeling of the measurement platform plane.

[0159] However, RANSAC determines candidate planes based on only three randomly sampled points in each iteration, and the resulting model parameters do not fully utilize the information from all interior points, thus limiting its fitting accuracy. To further improve the fitting accuracy of plane parameters, this paper introduces Principal Component Analysis (PCA) to optimize the fitting of plane parameters after obtaining the set of interior points using the RANSAC method.

[0160] S33: Principal component analysis (PCA) is used to refine the plane parameters of the candidate point set in the plane. The centroid and covariance matrix of the candidate point set in the plane are calculated. Eigenvalue decomposition is performed on the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the measurement platform plane. The plane equation parameters are determined in combination with the centroid.

[0161] The specific steps are as follows:

[0162] S331: Calculate the centroids of interior points: For the set of interior points selected by RANSAC... Calculate its centroid: Where M is the number of interior points. Let k be the interior point;

[0163] S332: Construct the covariance matrix: Calculate the deviation vector of each inlier relative to the centroid. Construct a 3×3 covariance matrix: ;

[0164] Here, matrix C is a real symmetric positive semi-definite matrix that describes the distribution characteristics of the interior point set in each direction in three-dimensional space. for The transpose of .

[0165] S333: Perform eigenvalue decomposition on the covariance matrix C: ,in, Let be an eigenvalue matrix, and , ;

[0166] The corresponding eigenvector matrix; minimum eigenvalue Corresponding feature vector This is the direction with the minimum variance of the inlier data distribution, which is also the direction of the normal vector of the best-fit plane.

[0167] S334: Confirm plane parameters: Take the eigenvector corresponding to the smallest eigenvalue. As the refined plane normal vector, its direction is restricted to the positive Z-axis, and after normalization, it becomes the unit normal. : ,

[0168] The intercept parameter D of the plane equation is determined by the relationship between the normal vector and the centroid:

[0169] By combining the centroid to determine the plane equation parameters, the plane equation of the measurement platform is obtained. , Let x be the component of the normal vector. Let y be the component of the normal vector. Let z be the normal vector z component, and let D be the constant term of the plane equation.

[0170] Through the above process, the robustness of RANSAC is first used to screen out a reliable set of interior points from the original point cloud containing noise and outliers. Then, PCA is used to perform global optimal fitting on the interior point set to obtain a plane parameter fitting result with higher accuracy and stronger stability.

[0171] S34: The Z-axis direction of the measurement coordinate system of the 3D vision sensor (i.e., ( ) is the target direction, based on the normal vector of the measurement platform plane obtained in step S33. The rotation matrix is ​​constructed based on the Rodriguez rotation formula.

[0172] In real-world environments, the mounting pose of depth cameras is rarely perfectly ideal. The optical axis of the camera and the normal to the measurement platform plane form a certain angle, causing the XOY plane in the depth camera coordinate system to be non-parallel to the measurement platform plane. Without compensation for this tilt angle, the height information extracted directly from the point cloud data will contain systematic biases introduced by the plane tilt, thus affecting the accuracy of subsequent package size measurements and volume estimations. To eliminate these geometric biases, this embodiment of the invention, based on a plane correction framework, employs a rigid body transformation method based on normal vector alignment. By constructing a composite transformation matrix of rotation and translation, the measurement reference plane is corrected to a pose parallel to the camera imaging plane, thereby providing a unified and stable geometric reference coordinate system for subsequent volume measurement algorithms for various packages.

[0173] In embodiments of the present invention, the spatial correction of the measurement platform plane can be viewed as a rotation alignment problem from the plane normal vector to the target reference normal vector; therefore, the Rodriguez rotation formula is used to obtain the rotation matrix. When the unit vector of the rotation axis is known... With rotation angle The rotation matrix R can be expressed as:

[0174] Where I is a 3 × 3 identity matrix, and K is an antisymmetric matrix constructed from the unit vector k of the rotation axis: ;

[0175] In the camera coordinate system, the unit normal vector of the imaging plane is defined as... That is, it coincides with the Z-axis direction of the camera coordinate system. The unit normal vector of the plane obtained by the plane fitting method of the measurement platform is... When the camera is mounted at an angle, and They are usually not collinear, so the angle between them is... It can be determined by the vector dot product relationship:

[0176] ;

[0177] Will Rotate to Alignment and rotation operations occur within the plane spanned by the two vectors, where the normal vector of the plane is the direction of the rotation axis. and By cross product ,vector for and Normalize the normal vector of the plane formed by the plane, and let

[0178] , Then the unit rotation axis vector .

[0179] Rotation operation is performed by a rotating axis With rotation angle It is uniquely determined that the Rodriguez rotation formula is satisfied.

[0180] Given the input requirements, the rotation matrix... It can be represented as:

[0181] .

[0182] by Represents the original point cloud coordinates, in... Let represent the point cloud coordinates after preliminary correction. The preliminary correction process can be achieved using a rotation matrix. Left-multiplying the original point cloud coordinates: ;

[0183] After this step, the normal vector of the point cloud's measurement reference plane has been rotated to be aligned with... Alignment eliminates the skewness of the reference plane geometrically. However, since the rotation center is located at the origin of the coordinate system by default, rather than the centroid of the point cloud, the rotation operation also introduces spatial displacement of the entire point cloud, which needs to be corrected by further translation compensation.

[0184] S35: Calculate the translation vector between the original point cloud centroid and the point cloud centroid after rotation by the rotation matrix. The rotation matrix and the translation vector form a rigid body transformation matrix to perform spatial correction on the original point cloud, so that the plane normal vector of the measurement platform is aligned with the Z-axis direction of the measurement coordinate system of the 3D vision sensor, and the point cloud is unified to the world coordinate system with the measurement reference plane as the reference.

[0185] When a rotation matrix R is applied to a point cloud with the origin of the coordinate system as the center of rotation, if the centroid of the point cloud is not at the origin, the entire point cloud will experience spatial drift after rotation. To prevent spatial drift caused by the rotation operation, let the original centroid of the point cloud be... : The centroid of the point cloud after preliminary correction is : ,

[0186] The point cloud after preliminary planar correction not only changes the plane angle, but also shifts the entire point cloud. To correct only the angle of the point cloud's plane tilt without changing its position, the position of the point cloud after preliminary planar correction needs to be translated from its centroid to the centroid of the original point cloud. The translation vector is... Defined as: ,

[0187] Calculate the translation vector between the original point cloud centroid and the point cloud centroid after rotation using the rotation matrix. The rigid body transformation matrix M is composed of the rotation matrix R and the translation vector t. Applying this rigid body transformation to the original point cloud aligns the normal vector of the measurement platform plane with the Z-axis of the world coordinate system, thus unifying the point cloud to a world coordinate system referenced to the measurement reference plane. Figure 2 As shown, the reference plane in the point cloud before correction is significantly tilted due to camera installation error, while the reference plane tends to be horizontal after correction.

[0188] S4: Package volume measurement:

[0189] S41: Detect arm interference within the measured ROI to identify whether there is operator hand interference;

[0190] During the volume measurement of logistics parcels, the operator's hand or arm may enter the measurement area. If not identified and removed, the system will misidentify the arm as part of the parcel. This invention constructs a discrimination method based on multi-feature constraints to detect arm interference. The specific methods for identifying whether operator hand interference exists include:

[0191] S411: Obtain the depth map within the measured ROI and identify pixels higher than the measurement reference plane as candidate target regions;

[0192] S412: Perform connected component analysis on the candidate target region to obtain the bounding rectangle and geometric features of each connected component; label each connected component based on the eight-neighbor connectivity to obtain the bounding rectangle, minimum bounding rotation rectangle and geometric features of each connected component;

[0193] S413: Calculate the following three criteria for each connected component:

[0194] a. Boundary proximity criterion: Calculate the minimum distance between the bounding rectangle of the connected region and the boundary of the measured ROI. If the minimum distance is less than the preset edge distance threshold, the boundary proximity feature is satisfied.

[0195] In practice, operators typically place the package onto the platform from one side of the measurement area. Therefore, when the arm enters the measurement area, it often comes into contact with or is very close to the boundary of the measurement area. Normal packages are usually placed in the central area of ​​the platform, maintaining a certain distance from the boundary of the measurement area. Therefore, when the minimum distance between the bounding rectangle of the candidate connected component and the boundary of the ROI is less than a threshold... When this is the case, the target can be considered to have edge contact characteristics.

[0196] b. Aspect Ratio Criterion: Calculate the ratio of the major axis to the minor axis of the smallest circumscribed rectangle of the connected region. If the ratio exceeds the preset aspect ratio threshold, it is determined that the aspect ratio is abnormal.

[0197] Arm regions in images typically exhibit an elongated structure, with a significantly higher aspect ratio for their minimum bounding rectangle of rotation compared to regular wrapping regions. Regular wrapping regions, on the other hand, are usually blocky structures with relatively limited aspect ratios. The minimum bounding rectangle of rotation for each sub-connected region is calculated, and its major axis length is obtained. and minor axis length , aspect ratio Exceeding the threshold The sub-connected domain is marked as a suspected arm.

[0198] c. Height distribution criterion: Calculate the dispersion of the height distribution of pixels within the connected domain. If the dispersion exceeds the preset height dispersion threshold, it is determined that the height distribution is abnormal.

[0199] Since packages typically have a relatively flat upper surface, their height distribution is relatively uniform; however, when an arm enters the measurement area, there is often a significant height change along its length, resulting in a large dispersion in the height distribution within the area. For example, if the variance of the height distribution of pixels within a connected region exceeds a preset height variance threshold, it is determined to meet the abnormal height distribution characteristics.

[0200] S414: If any connected component satisfies at least two of the three criteria a, b, and c, then it is determined that there is arm interference in the current measurement area.

[0201] If so, then pause the measurement;

[0202] If not, proceed to step S42;

[0203] like Figure 3 As shown, the red box marks the detected arm interference area, and the system pauses the measurement; the green box marks the normal wrapped area, and the system continues the measurement.

[0204] S42: Identify whether there is a package within the measured ROI;

[0205] After confirming the absence of arm interference, it is necessary to determine whether the package to be tested exists within the measurement ROI. This invention employs a package identification method based on a local one-sample Z-test, and the identification of whether a package exists specifically includes the following steps:

[0206] S421: Height Difference Map Construction: Obtaining a Baseline Depth Map under Unloaded Conditions Compared with the current depth map Calculate the height difference map of the current depth map relative to the reference depth map.

[0207] ,in This represents the image pixel coordinates; in the camera coordinate system, the farther the target is from the camera, the greater the depth value. The reference depth map in the unloaded state, serving as the background, is usually farther from the camera than the wrapped surface. Therefore, the wrapped surface area... ;

[0208] S422: Candidate Region Extraction: To reduce interference from irrelevant regions, embodiments of the present invention perform detection only within the measurement region ROI. Let the height difference map within the ROI be... To eliminate minor noise disturbances, a minimum height threshold is set within the measured ROI. Extract candidate pixels from the height difference map whose height values ​​are greater than a preset minimum height threshold to form a candidate mask. : ,in, For height difference map The height difference at the location;

[0209] S423: Local One-Sample Z-Test: Count the total number of valid pixels N in the candidate mask and calculate the average height of all valid pixels within the candidate mask. : ;

[0210] S424: Constructing a one-sample Z-test statistic: Obtaining the standard deviation of the height noise of the reference plane under no-load conditions. Calculate the standard error Construct the one-sample Z-test statistic: ;

[0211] S425: If the single-sample Z statistic is greater than the preset Z test threshold, it is determined that there is a package within the measured ROI; otherwise, it is determined that there is no package. If not, the process ends; if so, proceed to step S43.

[0212] The statistic Z reflects the degree to which the average height of the candidate region deviates from the zero mean. The statistic increases rapidly with the increase of the effective pixel count N. Therefore, even if the target height is only a few millimeters, it can still be stably identified as long as it forms a continuous region of a certain area in the image. This paper sets a statistic threshold Z > 3.0 to indicate that the measurement platform is enclosed.

[0213] Compared to traditional detection methods that rely on grayscale, color, or texture features, the proposed method directly utilizes depth geometry information to describe the spatial protrusions on the measurement platform. It is insensitive to changes in illumination and exhibits good robustness. Furthermore, this method uses a single-sample Z-test based on the region mean, rather than a single-point threshold, which effectively suppresses random depth noise interference.

[0214] After completing the package presence detection, it is necessary to further identify the specific type of the package. This invention constructs a hierarchical classification and identification process:

[0215] S43: Identify whether the package is a thin package or a non-thin package;

[0216] After completing the package existence detection, a threshold h is used to distinguish between thin packages and non-thin packages. Specifically, identifying a package as a thin package or a non-thin package includes:

[0217] S431: Obtain the height values ​​of all valid pixels within the package area, i.e., the vertical distance of each point to the measurement reference plane, forming a height set;

[0218] S432: Extract the maximum height value from the height set. ;

[0219] S433: If If the thickness is less than or equal to the preset thin-packet thickness threshold h, the package is determined to be a thin package; if If the thickness exceeds the preset thin package thickness threshold h, the package is determined to be a non-thin package.

[0220] If it is a thin package, proceed to step S431: calculate the volume of the thin package using the first calculation method;

[0221] If it is not a thin package, proceed to step S432: identify regular packages and irregular packages;

[0222] If it is a regular package, proceed to step S4321: calculate the volume of the regular package using the second calculation method;

[0223] If the package is irregular, proceed to step S4322: calculate the volume of the irregular package using the third calculation method.

[0224] For non-thin packages, this invention proposes a dual-criteria identification method that combines planar variance and area shrinkage, judging from two aspects: top surface flatness and contour regularity, in order to distinguish between regular and irregular packages.

[0225] Step S432, which identifies regular and irregular packages, specifically includes:

[0226] S4321: Calculate the planar variance of the top surface of the non-thin package. When the planar variance is less than or equal to the preset planar variance threshold, the top surface of the non-thin package is determined to have flatness.

[0227] For the height map after region filtering, this paper uses a height threshold H to pre-filter the target region to remove most planar noise. For regular packages, their top surface can usually be approximated as a plane; while for soft packages, bulges, or irregularly shaped parts, their top surface will show more obvious local undulations. This paper performs least-squares plane fitting on the effective pixels within the ROI and uses the variance of the fitting residual as a measure of the flatness of the top surface, denoted as the plane variance.

[0228] According to one embodiment of the present invention, in step S4321, the calculation of the plane variance specifically includes:

[0229] S43211: Perform least-squares plane fitting on the effective pixels in the non-thin-bag target region and solve for the plane coefficients;

[0230] set up For pixel coordinates, This corresponds to the height value. The coefficients are solved using least-squares plane fitting. , so that: ;

[0231] S43212: Calculate the residual of each pixel relative to the fitted plane;

[0232] S43212: Calculate the variance of the residuals as the plane variance.

[0233] Let the fitted plane be at point The estimated value at that location is residual Then the plane variance is defined as: ;

[0234] For a regular wrapper with a flat top surface, the residual only originates from sensor noise, therefore The residuals are relatively small. For irregular packages with bulges, depressions, or slope variations on the top surface, the residuals increase significantly. Larger. When Not exceeding the threshold If so, the top surface is determined to be flat.

[0235] S4322: Calculate the area shrinkage of the top surface of the package. If the area shrinkage is less than the preset area shrinkage threshold, the package is determined to have regularity in its outline.

[0236] The flatness of the top surface alone is insufficient to fully describe the regularity of a package. For example, a partially flattened soft package may exhibit a small planar variance on its top surface, but its outer contour may still show an irregular shape. To further describe the regularity of the contour boundary, this paper introduces area shrinkage as a second criterion and measures the area change of the target region after boundary shrinkage through morphological erosion operations.

[0237] The calculation of area shrinkage specifically includes:

[0238] Based on height threshold Extract effective pixel mask ;

[0239] For the mask Apply side length is square structural element Perform erosion calculations to obtain the eroded mask. , where ⊖ represents the morphological erosion operation.

[0240] Calculate the area retention rate : ; Calculate the area shrinkage : .

[0241] For regular wrappers with square outlines and smooth boundaries, the proportion of edge pixels is relatively low, resulting in less area loss after erosion, thus maintaining a high area retention rate. Larger area shrinkage The area loss is relatively small for irregularly shaped wrappers with obvious jagged edges and complex shapes; however, for such wrappers, the proportion of edge pixels is relatively high, resulting in a larger area loss after erosion. Smaller Larger. When the area shrinkage is significant. Below the threshold When the target outline is regular, it can be determined that the target outline has regularity.

[0242] S4323: If a package satisfies both the flatness of its top surface and the regularity of its outline, it is determined to be a regular package; if neither criterion is satisfied, it is determined to be an irregular package.

[0243] Thin packages, typically no more than 20mm in height, have insignificant height differences between the foreground and background in depth maps, making traditional depth segmentation difficult to extract their regions and posing a significant measurement challenge. Relying solely on depth information makes stable segmentation difficult. In logistics visual inspection tasks, the commonly used RGB color space exhibits high correlation between its channels and is extremely sensitive to changes in illumination. Due to uneven light distribution in actual measurement scenarios, shadows easily form around the package and its surroundings. Directly performing background subtraction in the RGB space can easily lead to missegmentation. In contrast, the CIELAB (Lab) color space is a physiological characteristic model closer to human visual perception. Its core advantage lies in separating luminance information (L channel) from color information (a and b channels). Utilizing this characteristic, the interference of illumination and shadows on image segmentation can be effectively reduced by adjusting the weights of each channel. This invention employs a strategy of compensating for insufficient depth information in color image segmentation, using a first calculation method to calculate the volume of the thin package, specifically including:

[0244] S4311: Acquire a single frame of unwrapped color image of the measurement platform as a background reference image, convert it to the CIELAB color space, and obtain the background Lab image. ;

[0245] S4312: Acquire the current frame color image containing the thin packet, convert it to the CIELAB color space, and obtain the current Lab image. ;in: ;

[0246] S4313: Calculate the channel-wise difference image between the current Lab image and the background Lab image:

[0247] ;in, This is the brightness difference value. The difference between the red and green axes is the red-green axis difference value. The values ​​represent the yellow-blue opposite axis difference, and T is the transpose sign. Simultaneously, the color image and depth map are RGB-D aligned, with the aligned depth map using the color camera as the reference coordinate system.

[0248] S4314: Weighted fusion of the difference images to obtain a grayscale difference image: Measurement ROI cropping is performed on the difference images to reduce interference from non-measurement areas; the multi-channel difference images are then fused into a grayscale image.

[0249] ; , , These are the weighting coefficients for the three Lab channels, respectively. To improve computational efficiency, the influence of wrapping shadows on segmentation is reduced by lowering the weights. In this paper, α=0.15, β=0.425, and γ=0.425.

[0250] S4315: Perform thresholding on the grayscale difference image to obtain the foreground mask. To improve segmentation quality, morphological opening and closing operations are performed on the mask to eliminate noise and fill small holes, ultimately resulting in a smooth and continuous mask. For mask Perform connected component analysis to identify all connected regions and their areas, and retain the connected region with the largest area as the target wrapping region. And remove other interfering areas.

[0251] S4316: Utilizing a thin-packed target area mask Extract the corresponding thin bag depth information from the RGB-D aligned depth map. And back-projected into a three-dimensional point cloud;

[0252] Using the bounding region mask in the obtained color image and aligned depth map The formula is obtained through the color 3D camera: Point clouds in a color coordinate system can be obtained. Since the system's spatial correction model is established in the depth coordinate system, it is necessary to utilize the extrinsic parameter relationship between the color camera and the depth camera to perform a rigid transformation on the point cloud, converting it to the depth coordinate system to obtain the point cloud in the depth coordinate system. ;in: ;

[0253] S4317: Transform the 3D point cloud into the world coordinate system using a rigid body transformation and project it onto the XY reference plane to obtain a 2D projected point set; using the formula: The point cloud is corrected and projected onto the XY reference plane to obtain a two-dimensional point set. .

[0254] S4318: Calculate the convex hull for the two-dimensional projection point set, and use the rotating caliper algorithm to find the minimum area bounding rectangle. Use the long side and short side of the minimum area bounding rectangle as the length and width of the thin hull, respectively.

[0255] Generating the convex hull can reduce the computational cost of subsequent searches because the edges of the minimum bounding rectangle must be collinear with some edge of the convex hull. First, the set of points projected onto the reference plane... Preprocessing is performed. Since the minimum bounding rectangle is determined only by its outer boundary, and the internal points do not contribute to the result, only its outer boundary needs to be extracted. Let the convex hull have... vertices And arranged in counter-clockwise order: .

[0256] Obtaining the convex hull Then, the rotating caliper algorithm is used to traverse each edge of the convex hull. Using the direction of that side as the principal axis of the candidate bounding rectangle, calculate the corresponding minimum enclosing rectangle one by one and record its area. The specific steps are as follows:

[0257] Step 1: Initialization and Edge Traversal. Traverse each edge of the convex hull. The direction unit vector of the current side is used as the principal axis direction of the candidate rectangle. Its normal vector is denoted as .

[0258] Step 2: Projection and Span Calculation. Project all vertices of the convex hull onto the principal axis direction. With normal vector direction Calculate the maximum span of the projection values ​​of each vertex in two directions, which is the length of the two sides of the current candidate rectangle.

[0259] Step 3: Area Update and Optimization. Calculate the area of ​​the current candidate rectangle while iterating through all rectangles. During the process of stripping edges, the minimum area record and its corresponding rectangle parameters are continuously maintained and updated.

[0260] After the above traversal and optimization, the candidate rectangle with the smallest area is selected as the final result. The lengths of the longer and shorter sides of this smallest bounding rectangle are the length and width dimensions of the irregular enclosure: ,in and These represent the projected spans in the directions corresponding to achieving the minimum area. This is the optimal edge index.

[0261] S4319: Calculate the height of the thin package and its volume based on its length, width, and height. Since the height calculation method for thin and non-thin packages is the same, it will be detailed below.

[0262] Considering the good planar characteristics of the top surface of the regular wrapping area, this embodiment first performs planar fitting on the point cloud of the top surface of the wrapping area, and removes outliers based on the distance of the points to the fitted plane. Points more than 10mm away from the plane are considered outliers and removed, thus retaining the more reliable point cloud data on the top surface. Then, the maximum outer contour is extracted from the denoised wrapping area, and the contour points are back-projected onto the XY plane for length and width measurements.

[0263] According to one embodiment of the present invention, the volume calculation of the regular package using the second calculation method in step S4321 specifically includes:

[0264] S43211: Extract depth pixels from the mask of the regular wrapping area and backproject them into a 3D point cloud. Transform it into the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the regular wrapping.

[0265] Based on the wrapping area mask obtained in step S422 Extract the corresponding pixels from the depth map and use the formula: The point cloud is converted into a three-dimensional point cloud. Then, the spatial rigid body transformation matrix and formula obtained by plane fitting and correction are used to finally correct the point cloud coordinate formula. The point cloud is then transformed to the measurement reference plane coordinate system to obtain the point cloud of the regular wrapped top surface.

[0266] S43212: Morphological erosion is performed on the mask of the wrapped area to shrink the edge area inward, thereby reducing the influence of unstable edge points on the plane fitting results;

[0267] S43213: The RANSAC method is used to perform plane fitting on the point cloud of the eroded top surface to obtain the plane parameters of the top surface;

[0268] S43214: Calculate the distance from each point in the top surface point cloud to the top surface plane, and assign points with distances greater than a threshold. Points that are identified as outliers and removed are used in this paper. After this processing step, a regularly wrapped effective point cloud with significantly reduced edge noise can be obtained. At the same time, The pixel coordinates corresponding to each point are set to valid in the mask image, while the rest are set to zero, resulting in the filtered wrapping mask image. .

[0269] S43215: Extract the maximum outer contour of the filtered wrapping mask image, convert the contour pixels to the world coordinate system and project them onto the XY reference plane to form a two-dimensional contour point set;

[0270] Extracting the mask image The largest outer contour in The contour pixels are transformed to the measurement reference plane coordinate system using the back projection formula and transformation matrix. The XY components of each point are then taken to form a two-dimensional contour point set. .

[0271] S43216: Find the minimum directed bounding rectangle for the two-dimensional contour point set, and use the long side and short side of the minimum directed bounding rectangle as the length and width of the regular wrapping, respectively; Find the smallest directed bounding rectangle, and use its long side and short side as estimates of the length and width of the bounding rectangle, respectively, to obtain the length and width of the target.

[0272] S43217: Calculate the height of the rule package and the volume of the rule package based on its length, width, and height.

[0273] When a package is identified as irregular, it indicates poor flatness of its top surface and a significant deviation from a rectangular structure when viewed from above. In this case, measurement methods based on plane fitting and regular contour extraction are no longer applicable. This invention proposes a method for measuring the minimum bounding rectangle size based on a two-dimensional convex hull and a rotating caliper. The package point cloud is projected onto the XY reference plane to form a two-dimensional point set. The outer contour is then extracted using the convex hull, and the bounding rectangle with the smallest area along the convex hull's edge direction is searched. The lengths of the two sides of this rectangle are used as the estimated length and width of the package.

[0274] In an embodiment of the present invention, step S4322, which involves calculating the volume of the irregular package using a third calculation method, specifically includes:

[0275] S43221: Extract depth pixels from the mask of the irregular package area and backproject them into a 3D point cloud. Transform the point cloud to the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the irregular package.

[0276] S43222: Project the point cloud of the irregularly wrapped top surface onto the XY reference plane to obtain a two-dimensional projected point set; extract depth pixels based on the wrapping mask and backproject them into a three-dimensional point cloud, then transform it to the measurement reference coordinate system through a spatial rigid body transformation, and finally project it onto the XY reference plane to obtain the target point cloud XY projected point set. .

[0277] S43223: Calculate the convex hull for the two-dimensional projection point set. Use the rotating caliper algorithm to traverse each edge of the convex hull. Use the direction of each edge as the main axis direction of the candidate bounding rectangle. Calculate the area of ​​the corresponding minimum bounding rectangle one by one. Select the candidate rectangle with the smallest area as the minimum bounding rectangle.

[0278] Generating the convex hull can reduce the computational cost of subsequent searches because the edges of the minimum bounding rectangle must be collinear with some edge of the convex hull. First, the set of points projected onto the reference plane... Preprocessing is performed. Since the minimum bounding rectangle is determined only by its outer boundary, and the internal points do not contribute to the result, only its outer boundary needs to be extracted. Let the convex hull have... vertices And arranged in counter-clockwise order: .

[0279] Obtaining the convex hull Then, the rotating caliper algorithm is used to traverse each edge of the convex hull. Using the direction of that side as the principal axis of the candidate bounding rectangle, calculate the corresponding minimum enclosing rectangle one by one and record its area. The specific steps are as follows:

[0280] Step 1: Initialization and Edge Traversal. Traverse each edge of the convex hull. The direction unit vector of the current side is used as the principal axis direction of the candidate rectangle. Its normal vector is denoted as .

[0281] Step 2: Projection and Span Calculation. Project all vertices of the convex hull onto the principal axis direction. With normal vector direction Calculate the maximum span of the projection values ​​of each vertex in two directions, which is the length of the two sides of the current candidate rectangle.

[0282] Step 3: Area Update and Optimization. Calculate the area of ​​the current candidate rectangle while iterating through all rectangles. During the process of stripping edges, the minimum area record and its corresponding rectangle parameters are continuously maintained and updated.

[0283] After the above traversal and optimization, the candidate rectangle with the smallest area is selected as the final result. The lengths of the longer and shorter sides of this smallest bounding rectangle are the length and width dimensions of the irregular enclosure: ,in and These represent the projected spans in the directions corresponding to achieving the minimum area. This is the optimal edge index.

[0284] S43224: Use the long and short sides of the smallest bounding rectangle as the length and width of the irregular wrapping, respectively;

[0285] S43225: Calculate the height of an irregular package and its volume based on its length, width, and height.

[0286] In an embodiment of the present invention, the calculation step of the package height includes:

[0287] Obtain the set of valid point clouds within the package area any one of them To the measurement reference plane The formula for calculating the height is: ;

[0288] Traverse all valid point clouds to obtain the height set. Arrange the height sets in ascending order to obtain an ordered height sequence. ;

[0289] The 98th percentile of the ordered height sequence was selected as the estimated value of the package height. ,Right now: Where n is the total number of valid point clouds, This indicates rounding down. By using 98% as an empirical parameter, extreme values ​​caused by outliers can be effectively filtered out, achieving highly stable measurements.

[0290] After obtaining a stable and reliable package height Then, combining the obtained package length and width, the package volume is calculated. Considering the actual needs of logistics sorting and warehouse planning, the space occupied by a package is usually measured by its minimum circumscribed cuboid. Therefore, this paper uses the length obtained from the aforementioned algorithm to calculate the volume. ,Width and height The final calculated volume of the package can be expressed as: .

[0291] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for measuring the volume of logistics parcels, used to measure the volume of parcels of different shapes, characterized in that: The volume measurement method includes the following steps: S1: System Input: Acquire images of the measurement area through a 3D vision sensor, and upload the depth map and color map of the image to the measurement system; S2: Image preprocessing: Filter the depth map, correct the distortion of the color image, and align the depth map and color image to RGB-D. S3: Measurement reference plane construction: Set up a plane fitting ROI and a measurement ROI. Based on the depth image information in the plane fitting ROI under no-load conditions, construct a measurement reference plane and perform coordinate correction on the point cloud data to unify the point cloud to the world coordinate system with the measurement reference plane as the reference. S4: Package volume measurement: S41: Detect arm interference within the measured ROI to identify whether there is operator hand interference; If yes, then pause the measurement; if no, proceed to step S42. S42: Identify whether there is a package within the measured ROI; If not, the process ends; if yes, proceed to step S43. S43: Identify whether the package is a thin or non-thin package; If it is a thin package, the volume of the thin package is calculated using the first calculation method. The specific calculation steps include: S4311: Acquire a single frame of unwrapped color image of the measurement platform as a background reference image, convert it to the CIELAB color space, and obtain the background Lab image. ; S4312: Acquire the current frame color image containing the thin packet, convert it to the CIELAB color space, and obtain the current Lab image. ;in: ,in, Lab space image serving as background reference. This is the Lab color space image of the current frame; RGB2Lab is the color space conversion from RGB to Lab. S4313: Calculate the channel-wise difference image between the current Lab image and the background Lab image. ;in, This is the brightness difference value. The difference between the red and green axes is the red-green axis difference value. The difference between the yellow and blue opposing axes is represented by T, which is the transpose sign. S4314: Perform weighted fusion on the difference images to obtain a grayscale difference image: ,in, , , These are the weighting coefficients for the three Lab channels, respectively. ; S4315: Threshold segmentation is performed on the grayscale difference image to obtain a foreground mask. Connectivity analysis is performed on the foreground mask, and the connected component with the largest area is retained as the thin-packed target region. S4316: Using the thin-packed target region mask, extract the corresponding thin-packed depth information from the RGB-D aligned depth map and back-project it into a three-dimensional point cloud; S4317: Transform the three-dimensional point cloud into the world coordinate system through a spatial rigid body transformation, and project it onto the XY reference plane to obtain a two-dimensional projection point set; S4318: Calculate the convex hull of the two-dimensional projection point set, and use the rotating caliper algorithm to find the minimum area bounding rectangle. Use the long side and short side of the minimum area bounding rectangle as the length and width of the thin hull, respectively. S4319: Calculate the height of the thin package and calculate the volume of the thin package based on the length, width and height; If the package is not thin, then distinguish between regular and irregular packages; If it is a regular package, the volume of the regular package is calculated using the second calculation method. The specific calculation steps include: S43211: Extract depth pixels from the mask of the wrapped area according to the rule and back-project them into a three-dimensional point cloud. Transform the point cloud into the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the wrapped area. S43212: Perform morphological erosion on the mask of the encapsulated area to shrink the edge area inward; S43213: The RANSAC method is used to perform plane fitting on the point cloud of the eroded top surface to obtain the plane parameters of the top surface; S43214: Calculate the distance from each point in the top surface point cloud of the rule-wrapped top surface to the top surface plane, and identify points whose distance is greater than the preset outlier threshold as outliers and remove them to obtain the effective point cloud and the filtered wrapping mask image. S43215: Extract the maximum outer contour from the filtered wrapping mask image, convert the contour pixels to the world coordinate system and project them onto the XY reference plane to form a two-dimensional contour point set; S43216: Find the minimum directed bounding rectangle for the two-dimensional contour point set, and use the long side and short side of the minimum directed bounding rectangle as the length and width of the regular wrapping, respectively. S43217: Calculate the height of the regular package and calculate the volume of the regular package based on the length, width and height; If the package is irregular, its volume is calculated using a third calculation method. The specific calculation steps include: S43221: Extract depth pixels from the mask of the irregular package area and back-project them into a three-dimensional point cloud. Transform the point cloud into the world coordinate system through spatial rigid body transformation to obtain the point cloud of the top surface of the irregular package. S43222: Project the point cloud of the irregularly wrapped top surface onto the XY reference plane to obtain a two-dimensional projection point set; S43223: Calculate the convex hull of the two-dimensional projection point set, and use the rotating caliper algorithm to traverse each edge of the convex hull. Take the direction of each edge as the main axis direction of the candidate bounding rectangle, calculate the area of ​​the corresponding minimum bounding rectangle one by one, and select the candidate rectangle with the smallest area as the minimum bounding rectangle. S43224: The long side and short side of the minimum bounding rectangle are respectively used as the length and width of the irregular wrapping; S43225: Calculate the height of the irregular package and calculate the volume of the irregular package based on the length, width and height.

2. The method for measuring the volume of a logistics parcel according to claim 1, characterized in that, Step S3, the construction of the measurement reference plane, specifically includes the following steps: S31: Obtain the depth map within the plane-fitted ROI under no-load conditions, and back-project the depth map into a 3D point cloud; S32: The RANSAC algorithm is used to extract in-plane points from the three-dimensional point cloud to obtain a candidate in-plane point set; S33: The principal component analysis method is used to refine the plane parameters of the candidate point set in the plane, the centroid and covariance matrix of the candidate point set in the plane are calculated, the covariance matrix is ​​decomposed into eigenvalues, the eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the measurement platform plane, and the plane equation parameters are determined in combination with the centroid. S34: Using the Z-axis direction of the measurement coordinate system of the 3D vision sensor as the target direction, construct a rotation matrix based on the plane normal vector of the measurement platform obtained in step S33 and the Rodriguez rotation formula. S35: Calculate the translation vector between the original point cloud centroid and the point cloud centroid after rotation by the rotation matrix. The rotation matrix and the translation vector form a rigid body transformation matrix to perform spatial correction on the original point cloud, so that the plane normal vector of the measurement platform is aligned with the Z-axis direction of the measurement coordinate system of the 3D vision sensor, and the point cloud is unified to the world coordinate system with the measurement reference plane as the reference.

3. The method for measuring the volume of a logistics parcel according to claim 1, characterized in that, In step S41, identifying whether there is interference from the operator's hands specifically includes: S411: Obtain the depth map within the measured ROI, and determine the pixels higher than the measured reference plane as candidate target regions; S412: Perform connected component analysis on the candidate target region to obtain the bounding rectangle and geometric features of each connected component; S413: Calculate the following three criteria for each connected component: a. Boundary proximity criterion: Calculate the minimum distance between the circumscribed rectangle of the connected domain and the boundary of the measured ROI. If the minimum distance is less than a preset edge distance threshold, the boundary proximity feature is satisfied. b. Aspect Ratio Criterion: Calculate the ratio of the major axis to the minor axis of the smallest circumscribed rectangle of the connected region. If the ratio exceeds a preset aspect ratio threshold, it is determined that the aspect ratio is abnormal. c. Height distribution criterion: Calculate the dispersion of the height distribution of pixels within the connected domain. If the dispersion exceeds the preset height dispersion threshold, it is determined that the height distribution is abnormal. S414: If any of the connected components simultaneously satisfies at least two of the three criteria a, b, and c, then it is determined that there is arm interference in the current measurement area.

4. The method for measuring the volume of a logistics parcel according to claim 1, characterized in that, Step S42, identifying whether there is a package, specifically includes the following steps: S421: Height Difference Map Construction: Obtaining a Baseline Depth Map under Unloaded Conditions Compared with the current depth map Calculate the height difference map of the current depth map relative to the reference depth map. ,in Represents the pixel coordinates of the image; S422: Candidate Region Extraction: Set a minimum height threshold within the measured ROI. Candidate pixels with height values ​​greater than a preset minimum height threshold are extracted from the height difference map to form a candidate mask. : ;in, For height difference map The height difference at the location; S423: Local One-Sample Z-Test: Count the total number N of valid pixels in the candidate mask and calculate the average height of all valid pixels in the candidate mask. : ; S424: Construct a one-sample Z-test statistic: Obtain the standard deviation of the height noise of the measurement reference plane under no-load conditions. Calculate the standard error Construct the one-sample Z-test statistic. ; S425: If the single-sample Z statistic is greater than the preset Z test threshold, then it is determined that there is a package within the measured ROI; otherwise, it is determined that there is no package.

5. The method for measuring the volume of a logistics parcel according to claim 1, characterized in that, In step S43, identifying whether the package is a thin package or a non-thin package specifically includes: S431: Obtain the height values ​​of all valid pixels within the package area and form a height set; S432: Extract the maximum height value from the height set. ; S433: If If the thickness of the package is less than or equal to a preset thin-package thickness threshold h, the package is determined to be a thin-package; if If the thickness of the package is greater than the preset thin package thickness threshold h, the package is determined to be a non-thin package.

6. The method for measuring the volume of a logistics parcel according to claim 1, characterized in that, The identification of regular and irregular packages specifically includes: S4321: Calculate the planar variance of the top surface of the non-thin package. When the planar variance is less than or equal to a preset planar variance threshold, it is determined that the top surface of the non-thin package has flatness. S4322: Calculate the area shrinkage of the top surface of the package. If the area shrinkage is less than a preset area shrinkage threshold, then the outline of the package is determined to be regular. S4323: If the package simultaneously meets both the criteria of top surface flatness and contour regularity, it is determined to be a regular package; if either criterion is not met, it is determined to be an irregular package.

7. The method for measuring the volume of a logistics parcel according to claim 6, characterized in that, In step S4321, the calculation of the plane variance specifically includes: S43211: Perform least-squares plane fitting on the effective pixels in the non-thin-bag target region and solve for the plane coefficients; S43212: Calculate the residual of each pixel relative to the fitted plane; S43213: Calculate the variance of the residuals as the plane variance.

8. The method for measuring the volume of a logistics parcel according to claim 6, characterized in that, In step S4322, the calculation of the area shrinkage specifically includes: Extract the effective pixel mask M based on the height threshold; Apply a square structural element with side length k to the mask M. Perform erosion calculations to obtain the eroded mask. : ; Calculate the area retention rate : ; Calculate the area shrinkage : .

9. The method for measuring the volume of a logistics parcel according to any one of claims 1 to 8, characterized in that, The steps for calculating the height of the package include: Obtain the set of valid point clouds within the package area any one of them To the measurement reference plane height The calculation formula is: ; Traverse all valid point clouds to obtain the height set. Arrange the height set in ascending order to obtain an ordered height sequence. ; The 98th percentile of the ordered height sequence was selected as the estimated package height. ,Right now: Where n is the total number of valid point clouds, This indicates rounding down to the nearest integer.

Images