Multi-box volume measurement method and system
By processing RGB and depth maps, the volume of multiple boxes is calculated using a box segmentation model and background plane equations. This solves the problem of poor multi-target measurement results in traditional methods and improves sorting and transportation efficiency.
Patent Information
- Application Number
- CN202510878493.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-11-21
AI Technical Summary
Traditional volume measurement methods are ineffective for measuring multiple targets, leading to low efficiency and high transportation costs in the sorting and transportation process.
By acquiring pixel-aligned RGB and depth maps, multiple boxes are segmented using a pre-trained box segmentation model. Connected components are found, and the center point coordinates and depth values of the boxes are calculated. The height and minimum bounding rectangle of the boxes are obtained by combining the background plane equation, and the length and width of the boxes are calculated to finally determine the volume of the boxes.
This technology enables simultaneous measurement of multiple containers, improving the efficiency of volume measurement and thus enhancing the efficiency of cargo transportation.
Smart Images
Figure CN120997278A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of volume measurement, more particularly, to a multi-box volume measurement method and system. BACKGROUND
[0002] In express sorting and transportation, the volume of goods needs to be measured to improve transportation efficiency and reduce transportation cost. The traditional volume measurement method only relies on point cloud data, and the volume measurement effect is poor for multiple targets. Due to the poor effect of the traditional volume measurement method in measuring the volume of goods, the efficiency of the sorting and transportation process is low, thereby causing problems such as high transportation cost. SUMMARY
[0003] In view of the above problems, the present application aims to provide a multi-box volume measurement method and system to solve the problems of poor effect of the traditional volume measurement method in measuring the volume of goods, low efficiency of the sorting and transportation process, and high transportation cost.
[0004] In one aspect, the present application provides a multi-box volume measurement method, comprising:
[0005] acquiring a pixel-aligned RGB image and a depth image;
[0006] performing background removal on the acquired depth image to obtain a foreground image; and
[0007] segmenting multiple boxes in the RGB image by using a pre-trained box segmentation model to segment each box;
[0008] finding a connected domain of each box on the foreground image, and calculating a center point coordinate of a top surface of each box and a depth value of the top surface of each box according to the connected domain;
[0009] processing each connected domain by using a pre-acquired background plane equation to obtain a height of each box; and
[0010] calculating a minimum circumscribed rectangle of the connected domain, and obtaining a length and a width of each box according to the minimum circumscribed rectangle, the center point coordinate of the top surface of each box, and the depth value of the top surface of each box;
[0011] determining a volume of each box according to the height of each box, the length of each box, and the width of each box.
[0012] Optionally, the step of training the box segmentation model comprises:
[0013] downloading a public data set of boxes from a network; and
[0014] The downloaded box public data set and the collected box data set are taken as a multi-box data set;
[0015] A box segmentation model is trained through the multi-box data set.
[0016] Optionally, the step of obtaining the background plane equation comprises:
[0017] A background depth map of a scene to be detected is collected;
[0018] The background depth map is converted into a world coordinate system by using camera intrinsic parameters, and a background plane equation is fitted by using a least square method, wherein a formula of the background plane equation is A*x+B*y+C*z+D=0, and A, B, C and D are constants and A, B and C are not zero at the same time.
[0019] Optionally, the step of finding a connected domain of each box on the foreground map comprises:
[0020] A ROI region of each box is extracted according to a mask map of each box on the foreground map, wherein the mask map of each box is obtained after each box is segmented.
[0021] A connected domain of each box in the ROI region is found by using a region growing method.
[0022] Optionally, the step of finding a connected domain of each box in the ROI region by using the region growing method comprises:
[0023] A point set Z satisfying a condition 0<Depth- i <threshold3 is screened out in an n*n region centered on a center point of the ROI region, wherein n is a natural number, threshold3=-D, D is a constant of the plane equation A*x+B*y+C*z+D=0, and -D is equal to a depth value of the ground;
[0024] A number of occurrences of each depth value in the point set Z is counted, and the number of occurrences of each depth value is sorted in descending order, and a point corresponding to a depth value with a cumulative number greater than a threshold value threshold2 is taken as a seed point.
[0025] All seed points are traversed, a difference between a depth value of the seed point and a depth value of a point in an eight-neighbor domain of the seed point is calculated, if the difference is less than a threshold value threshold4, the point in the eight-neighbor domain of the seed point is taken as a seed point to continue participating in operation, until all seed points are traversed, and the traversed seed points are connected to form a connected domain of each box.
[0026] Optionally, the method further comprises:
[0027] The connected domain is taken as a box top surface.
[0028] The coordinates of the center point of the connected domain are calculated, and the coordinates of the center point of the connected domain are taken as the coordinates of the center point of the box top surface.
[0029] The average of the depth values of all points in the connected domain is calculated, and the average of the depth values of all points in the connected domain is taken as the depth value of the box top surface.
[0030] Optionally, the method further comprises:
[0031] The coordinates of each point in the connected domain are converted into coordinates in the world coordinate system, and the distance of each point to the ground is calculated.
[0032] The average of the distances of all points in the connected domain to the ground is calculated, and the average is taken as the height of the box.
[0033] Optionally, the formula used to calculate the distance of each point to the ground is:
[0034]
[0035] Wherein, A, B, C, D are constants in the background plane equation A*x+B*y+C*z+D=0, (x w , y w , z w ) are the coordinates of the point in the world coordinate system.
[0036] Optionally, the formula used to calculate the minimum circumscribed rectangle of the connected domain is:
[0037] Width=width*cebterValue / f x
[0038] Height=height*centerValue / f x
[0039] boxLength=max(Width,Height)
[0040] boxWidth=min(Width,Height)
[0041] Wherein, Width and Height are the actual width and height of the minimum circumscribed rectangle, respectively.
[0042] width, height are pixel width, pixel height of the minimum bounding rectangle respectively;
[0043] centerValue is the distance from the box to the camera;
[0044] f x is the focal length of the camera, boxLength is the length of the box, and boxWidth is the width of the box.
[0045] In another aspect, the present application also provides a multi-box volume measurement system, comprising:
[0046] an image acquisition unit, configured to acquire a pixel-aligned RGB image and a depth image;
[0047] a foreground image acquisition unit, configured to remove the background from the acquired depth image to acquire a foreground image;
[0048] a box segmentation unit, configured to segment a plurality of boxes in the RGB image by using a pre-trained box segmentation model to segment each box;
[0049] a connected domain finding unit, configured to find a connected domain of each box on the foreground image, and calculate a center point coordinate of a top surface of each box and a depth value of the top surface of the box according to the connected domain;
[0050] a box height acquisition unit, configured to process each connected domain by using a pre-acquired background plane equation to acquire a height of each box;
[0051] a box length and width acquisition unit, configured to calculate a minimum bounding rectangle of the connected domain, and acquire a length and a width of each box according to the minimum bounding rectangle, the center point coordinate of the top surface of the box, and the depth value of the top surface of the box;
[0052] a box volume determination unit, configured to determine a volume of each box according to the height of each box, the length of each box, and the width of each box.
[0053] As can be seen from the above technical solution, the multi-box volume measurement method and system provided by the present application first segment the boxes in the RGB image by using a pre-trained box segmentation model, wherein the number of boxes in the RGB image is not limited, so that a plurality of boxes in the RGB image can be segmented at one time when the boxes are segmented, then the length, width and height of each box are determined by finding the connected domain of each box, so as to calculate the volumes of the plurality of boxes, and the image does not need to be repeatedly acquired multiple times in the process of calculating the volumes, thereby improving the efficiency of volume measurement and further improving the efficiency of cargo transportation.
[0054] To the accomplishment of the foregoing and related ends, one or more aspects of the application comprise the features hereinafter fully described. The following description and the annexed drawings set forth in detail certain illustrative aspects of the application. These aspects are indicative, however, of but a few of the various ways in which the principles of the application can be employed. Other aspects and advantages of the application will be made apparent to those of ordinary skill in the art upon reading the following description and the appended claims. The application is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF DRAWINGS
[0055] Other objects and advantages of the application will become apparent to those skilled in the art from the following description, taken in connection with the accompanying drawings, wherein, by way of illustration and example:
[0056] Figure 1 Flow chart of multi-box volume measurement method according to embodiments of the application;
[0057] Figure 2 Schematic diagram of multi-box scenario according to embodiments of the application;
[0058] Figure 3 Multi-carton instance segmentation result diagram according to embodiments of the application;
[0059] Figure 4 World coordinate system and distance calculation schematic diagram according to embodiments of the application;
[0060] Figure 5 Minimum circumscribed rectangle schematic diagram according to embodiments of the application;
[0061] Figure 6 Actual length and pixel length relationship diagram of an object according to embodiments of the application;
[0062] Figure 7 Logical structure block diagram of multi-box volume measurement system according to embodiments of the application.
[0063] The same numbers in different drawings identify the same elements or features. DETAILED DESCRIPTION
[0064] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It can be evident, however, that the embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more embodiments.
[0065] In the present application, all the embodiments mentioned in the text and the preferred implementation methods can be combined with each other to form new technical solutions if not specifically stated.
[0066] The traditional volume measurement method has poor measurement effect on cargo volume, resulting in low efficiency of sorting and transportation process, high transportation cost and other problems.
[0067] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0068] To illustrate the multi-box volume measurement method provided by the present application, Figure 1 The multi-box volume measurement method according to the embodiment of the present application is shown.
[0069] As Figure 1 shown, the multi-box volume measurement method provided by the present embodiment includes:
[0070] S1: acquiring a pixel-aligned RGB image and a depth image;
[0071] S21: removing the background of the acquired depth image to obtain a foreground image; and S22: segmenting the multiple boxes in the RGB image by using a pre-trained box segmentation model to segment each box;
[0072] S3: finding the connected domain of each box on the foreground image, and calculating the center point coordinates of the top surface of each box and the depth value of the top surface of each box according to the connected domain;
[0073] S41: processing each connected domain by using a pre-acquired background plane equation to obtain the height of each box; and
[0074] S42: calculating the minimum circumscribed rectangle of the connected domain, and obtaining the length and width of each box according to the minimum circumscribed rectangle, the center point coordinates of the top surface of each box and the depth value of the top surface of each box;
[0075] S5: determining the volume of each box according to the height of each box, the length of each box and the width of each box.
[0076] In the present application, multi-target segmentation is performed based on the box segmentation model to obtain the mask image of each target, and then the depth value of the depth image is processed to obtain the height, length and width of the target.
[0077] As Figure 2 shown, a depth camera is installed horizontally above the multi-box area to be monitored to capture the cargo image (depth image, RGB image or IR image). A scene background depth image without boxes is first acquired, and the background is calibrated by using the depth image. The subsequent images to be measured are all removed from the background image to obtain a foreground image. The image is segmented by using deep learning to segment each box, and the length, width and height of the box are calculated according to each box and the corresponding foreground image.
[0078] In the present application, the box is not limited to the paper box, plastic box, foam box and other common material box, as long as the shape of the regular cube can use the present solution, here only take the paper box as an example, the specific implementation of the multi-box volume measurement is described. The depth camera is horizontally installed above the measurement area, and the measurement area photographed by the camera is a horizontal plane.
[0079] In the embodiment of the present application, the step of training the box segmentation model comprises: obtaining multi-box data, and training the pre-constructed box segmentation model through the multi-box data set. The multi-box data comprises a public data set of the box and a data set of the collected box, and the pre-constructed box segmentation model is trained based on deep learning. Specifically, as an example, the public data set of the box can be searched and downloaded from the network, the image acquisition environment is built to collect the data set of the box with different sizes and different placement positions, and the box segmentation model is trained by using the collected data set. The image type here can be RGB or IR image or depth map, which depends on the accuracy and performance of the segmentation model, and in this embodiment, the remaining steps are described taking RGB image as an example.
[0080] In the embodiment of the present application, the process of obtaining the background plane equation is that: the measurement area is not placed with the box, only a background depth map is collected, the background depth map is converted to the world coordinate system by using the camera internal parameter, and the plane equation is fitted by using the least square method to obtain the formula A*x+B*y+C*z+D=0 of the background plane equation, wherein A, B, C and D are constants, and A, B and C are not zero at the same time; this background plane equation can be considered as the ground plane equation.
[0081] In S1, the RGB image and the depth map are collected, and the depth map and the RGB image are pixel-aligned.
[0082] In S21, the background is removed from the collected depth map to obtain a foreground image. When the background is removed, the point of the current depth map is considered as a background point if the distance between the point and the point of the background depth map is less than a certain threshold threshold1, the background point depth value is set to 0, and some invalid 0xFFFF points are also set to 0, thereby obtaining the foreground image.
[0083] In S22, the pre-trained box segmentation model is used to segment the RGB image to obtain a single paper box, and a mask image of the single paper box is obtained (as shown in Figure 3
[0084] In S3, for each paper box, the ROI region of the single paper box is extracted on the foreground image according to the mask image, and the connected domain of the ROI region is found by using the region growing method. The connected domain refers to a region composed of pixel points with the same pixel value and adjacent positions in the image.
[0085] When finding the connected domain by the region growing method, a seed point needs to be determined first: in an n*n region (n is a natural number, preferably 20) centered on the center point of the ROI region, a point set Z satisfying the condition 0<Depth i i ] is sorted in descending order, and points corresponding to depth values with a cumulative proportion greater than a threshold threshold2 are counted as seed points.
[0086] It should be noted that the depth value of the top surface of the box accounts for the largest proportion, and a part of the noise points are filtered out. The threshold threshold3 in the screening condition is -D, D is a constant of the plane equation A*x+B*y+C*z+D=0, generally negative, and the values of A, B and C are small, and -D is approximately equal to the depth value of the ground.
[0087] Then, the growth criterion: all seed points are traversed, and the difference between the depth value of the seed point and the depth value of the point in the eight-neighborhood of the seed point is judged. If the difference is less than a threshold threshold4, the point in the eight-neighborhood of the seed point is recorded as a seed point to continue participating in the operation.
[0088] Finally, the growth stopping condition: all seed points are traversed, and the traversed seed points are connected to find the connected domain of each box.
[0089] In the present application, after the connected domain (objectRegion) is found, the center point coordinates of the connected domain are calculated as the center point coordinates of the top surface of the box, and the average depth value of all points in the connected domain is calculated as the depth value of the top surface of the box.
[0090] In S41, the background plane equation is the ground plane equation. As shown in Figure 4 , first, the coordinates of each point in the connected domain are converted into coordinates in the world coordinate system, and the distance of each point to the ground is calculated:
[0091]
[0092] where A, B, C and D are constants in the ground plane equation A*x+B*y+C*z+D=0, (x w ,y w ,z w ) is the coordinate of the point in the world coordinate system, and the average distance of all points in the connected domain is taken as the height of the box.
[0093] In S42, the minimum circumscribed rectangle of the connected domain is calculated Figure 5 The length and width of the box are calculated using the minimum circumscribed rectangle and the depth value of the top surface of the box.
[0094] In Figure 6 In the embodiment shown, f is the focal length of the camera, d is the actual distance from the camera to the target box, H is the actual length of the target box, and h is the pixel length of the box in the image. According to similar triangles, we can deduce that:
[0095] H = h * d / f
[0096] That is, the actual length of the object can be calculated using the pixel length in the image, the focal length of the camera, and the distance from the object to the camera: actual length = pixel length in image * distance from object to camera / focal length
[0097] The formula corresponding to the present application is:
[0098] Width = width * centerValue / f x
[0099] Height = height * centerValue / f x
[0100] boxLength = max(Width, Height)
[0101] boxWidth = min(Width, Height)
[0102] where Weith and height are the actual width and height of the minimum circumscribed rectangle, width and height are the pixel width and pixel height of the minimum circumscribed rectangle, centerValue is the calculated distance from the box to the camera, f x is the focal length of the camera in the x-axis direction. In the camera intrinsic parameters, f x is the length of the focal length in the x-axis direction, and f y is the length of the focal length in the y-axis direction. In general, they are equal, i.e., the pixel is a square. In the camera intrinsic parameters used in the present application, they are very similar, and are represented by the focal length f x of the x-axis direction. The width and height of the minimum circumscribed rectangle are not distinguished between long and short, and after the actual length is calculated, the long side is taken as the length of the box and the short side is taken as the width of the box.
[0103] In S5, the volume of the final box is V = boxLength (length of the box) * boxWidth (width of the box) * boxHeight (height of the box).
[0104] From the above S1 to S5, it can be seen that the traditional volume measurement generally requires the box to be placed in the middle of the image, and there is only one box in one image, and only one box can be measured at a time. In the present application, the box in the RGB image is segmented by using the pre-trained box segmentation model, wherein the number of boxes in the RGB image is not limited, so that multiple boxes in the RGB image can be segmented at a time when the box segmentation is performed. The length, width and height of each box are determined by finding the connected domain of each box, so as to calculate the volume of multiple boxes, and the image does not need to be repeatedly collected multiple times during the volume calculation, thereby improving the efficiency of volume measurement and further improving the efficiency of cargo transportation.
[0105] Corresponding to the above method, the present application also provides a multi-box volume measurement system, Figure 7 The logic structure of the multi-box volume measurement system according to the embodiment of the present application is shown.
[0106] As Figure 7 shown, the multi-box volume measurement system provided by the present application comprises: an image acquisition unit 710 for acquiring a pixel-aligned RGB image and a depth image; a foreground image acquisition unit 721 for removing the background of the acquired depth image to obtain a foreground image; a box segmentation unit 722 for segmenting multiple boxes in the RGB image by using a pre-trained box segmentation model to segment each box; a connected domain finding unit 730 for finding the connected domain of each box on the foreground image, and calculating the center point coordinates of the top surface of each box and the depth value of the top surface of each box; a box height acquisition unit 741 for processing each connected domain by using a pre-acquired background plane equation to obtain the height of each box; a box length and width acquisition unit 742 for calculating the minimum circumscribed rectangle of the connected domain, and obtaining the length and width of each box according to the minimum circumscribed rectangle, the center point coordinates of the top surface of the box and the depth value of the top surface of the box; and a box volume determination unit 750 for determining the volume of each box according to the height of each box, the length of each box and the width of each box.
[0107] For the embodiment of the multi-box volume measurement system provided by the present application, since it is basically similar to the embodiment of the multi-box volume measurement method, the related parts are described in the method embodiment part, which will not be repeated here.
[0108] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods, which are all included in the protection scope of the present application.
Claims
1. A multi-box volume measurement method, characterized by, The method comprises: collecting a pixel-aligned RGB image and a depth image; performing background removal on the collected depth image to obtain a foreground image; and segmenting a plurality of boxes in the RGB image by using a pre-trained box segmentation model to segment each box; finding a connected domain of each box on the foreground image, and calculating a center point coordinate of a top surface of each box and a depth value of the top surface of each box according to the connected domain; processing each connected domain by using a pre-obtained background plane equation to obtain a height of each box; and calculating a minimum bounding rectangle of the connected domain, and obtaining a length and a width of each box according to the minimum bounding rectangle, the center point coordinate of the top surface of the box, and the depth value of the top surface of the box; determining a volume of each box according to the height of each box, the length of each box, and the width of each box.
2. The multi-box volume measurement method of claim 1, wherein, The step of training the box segmentation model comprises: downloading a public dataset of boxes from a network; and collecting a dataset of boxes of different sizes and different positions from a built image collection environment; using the downloaded public dataset of boxes and the collected dataset of boxes as a multi-box dataset; training a pre-constructed box segmentation model by using the multi-box dataset.
3. The multi-box volume measurement method of claim 1, wherein, The step of obtaining the background plane equation comprises: collecting a background depth image of a scene to be detected; converting the background depth image to a world coordinate system by using a camera intrinsic parameter, and fitting a background plane equation by using a least square method, wherein a formula of the background plane equation is A*x+B*y+C*z+D=0, wherein A, B, C, and D are constants, and A, B, and C are not zero at the same time.
4. The multi-box volume measurement method of claim 3, wherein, The step of finding a connected domain of each box on the foreground image comprises: extracting a ROI region of each box on the foreground image according to a mask image of each box, wherein the mask image of each box is obtained after segmenting each box; finding a connected domain of each box in the ROI region by using a region growing method.
5. The multi-box volume measurement method of claim 4, wherein, The step of finding a connected domain of each box in the ROI region by using the region growing method comprises: In an n*n region centered on the center point of the ROI region, a point set Z satisfying the condition 0<Depth- i < threshold3 is screened out, where n is a natural number, threshold3 = -D, D is a constant of the plane equation A*x+B*y+C*z+D=0, and -D is equal to the depth value of the ground; counting a number of times each depth value in the point set Z appears, and sorting the number of times each depth value appears from large to small, and taking a point corresponding to a depth value with a cumulative number greater than a threshold value threshold2 as a seed point; iterating all seed points, calculating a difference value between a depth value of the seed point and a depth value of a point in an eight-neighbor domain of the seed point, and if the difference value is less than a threshold value threshold4, taking the point in the eight-neighbor domain of the seed point as a seed point to continue participating in operation, until all seed points are iterated, and connecting the seed points that have been iterated to form a connected domain of each box.
6. The multi-box volume measurement method of claim 5, wherein, The step of calculating a center point coordinate of a top surface of each box and a depth value of the top surface of each box according to the connected domain comprises: taking the connected domain as a top surface of a box; calculating a coordinate of a center point of the connected domain, and taking the coordinate of the center point of the connected domain as the center point coordinate of the top surface of the box; calculating a mean value of depth values of all points in the connected domain, and taking the mean value of the depth values of all points in the connected domain as the depth value of the top surface of the box.
7. The multi-box volume measurement method of claim 6, wherein, The height of each box is obtained by processing each connected domain through the pre-acquired background plane equation, including: Converting the coordinates of each point in the connected domain into coordinates in the world coordinate system, and calculating the distance of each point to the ground; Calculating the average of the distances of all points in the connected domain to the ground, and taking the average as the height of the box.
8. The multi-box volume measurement method of claim 7, wherein, The formula used to calculate the distance of each point to the ground is: where A, B, C, D are constants in the background plane equation A*x+B*y+C*z+D=0, (x w , y w , z w ) are coordinates of the point in the world coordinate system.
9. The multi-box volume measurement method of claim 8, wherein, The formula used to calculate the minimum circumscribed rectangle of the connected domain is: Width = width * centerNalue / f x Height = height * centerValue / f x boxLength = max(Width, Height) boxWidth = min(Width, Height) Where Width and Height are the actual width and height of the minimum circumscribed rectangle; width and height are the pixel width and pixel height of the minimum circumscribed rectangle; centerValue is the distance of the box to the camera; f x For the camera focal length, boxLength is the length of the box, and boxWidth is the width of the box.
10. A multi-box volume measurement system, characterized by, It includes: An image acquisition unit for acquiring pixel-aligned RGB images and depth maps; A foreground image acquisition unit for removing the background from the acquired depth map to obtain a foreground image; A box segmentation unit for segmenting multiple boxes in the RGB image through a pre-trained box segmentation model to segment each box; A connected domain finding unit for finding the connected domain of each box on the foreground image and calculating the center point coordinates of the top surface of each box and the depth value of the top surface of each box; A box height acquisition unit for processing each connected domain through a pre-acquired background plane equation to obtain the height of each box; A box length and width acquisition unit for calculating the minimum circumscribed rectangle of the connected domain, and obtaining the length and width of each box according to the minimum circumscribed rectangle, the center point coordinates of the top surface of the box, and the depth value of the top surface of the box; A box volume determination unit for determining the volume of each box according to the height of each box, the length of each box, and the width of each box.