A cargo volume measurement method based on three-dimensional point cloud data

Abstract

The application discloses a kind of cargo volume measurement methods based on three-dimensional point cloud data, it is related to image processing technical field, the method based on the color image and depth image of the goods and the background of being placed on the goods table obtains initial goods point cloud set and initial goods table point cloud set, initial goods point cloud set and initial goods table point cloud set are rotated to the overhead perspective, then carry out point cloud repair, and then, the bottom edge point cloud set is extracted from the repaired goods point cloud set and the point cloud rotation matrix is obtained to carry out point cloud rotation and obtain target point cloud set, the method is repaired and rotation transformation to three-dimensional point cloud by the relationship between goods and goods table, in the case where there is three-dimensional point cloud missing in single-view shooting scene, the accuracy of the bounding box of the target point cloud set obtained is also higher, so that the volume of goods can be accurately calculated, the method is simple and the accuracy of the result obtained is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a cargo volume measurement method based on three-dimensional point cloud data. BACKGROUND

[0002] In the field of warehousing and logistics, it is often necessary to plan a transportation scheme and determine a service price according to the volume of the cargo. The traditional manual measurement of the volume of the cargo is low in efficiency and low in accuracy. With the development of image processing technology, a method of measuring the volume of the cargo by capturing a picture of the cargo by a camera to obtain three-dimensional point cloud data is widely applied.

[0003] However, in the existing method of measuring the volume of the cargo by using image processing technology, the picture of the cargo captured by the camera at a single position is single in perspective, and the obtained three-dimensional point cloud data is incomplete and not complete, which leads to low accuracy of the measured volume of the cargo. Although the picture of the cargo captured by the camera at multiple different positions can improve the accuracy to a certain extent by comprehensively using multiple perspective pictures, this will lead to high structural complexity and high data processing difficulty. SUMMARY

[0004] The present application relates to the technical field of image processing, and particularly relates to a cargo volume measurement method based on three-dimensional point cloud data.

[0005] A cargo volume measurement method based on three-dimensional point cloud data, the method comprising:

[0006] obtaining a color image and a depth image of a cargo placed on a placement table and a background where the cargo is located, and obtaining an initial cargo point cloud set and an initial placement table point cloud set based on the obtained color image and depth image, the initial cargo point cloud set comprising three-dimensional point clouds in a region where the cargo is located, and the initial placement table point cloud set comprising three-dimensional point clouds in a region where a placement table of the cargo background is located;

[0007] rotating the initial cargo point cloud set to a vertical downward overhead perspective to obtain a rotated cargo point cloud set, and rotating the initial placement table point cloud set to the vertical downward overhead perspective to obtain a rotated placement table point cloud set;

[0008] performing point cloud repair on the rotated cargo point cloud set according to the rotated placement table point cloud set to obtain a repaired cargo point cloud set;

[0009] extracting a bottom edge point cloud set from the repaired cargo point cloud set, the bottom edge point cloud set comprising three-dimensional point clouds of bottom edges located on the same side of the cargo;

[0010] A point cloud rotation matrix is obtained based on the bottom edge point cloud set, and the three-dimensional point cloud in the repaired cargo point cloud set is rotated according to the point cloud rotation matrix to obtain a target point cloud set.

[0011] A bounding box of the target point cloud set is constructed, and the volume of the cargo is calculated based on the constructed bounding box.

[0012] The beneficial technical effects of the present application are:

[0013] The present application discloses a cargo volume measurement method based on three-dimensional point cloud data. The method repairs and rotates the three-dimensional point cloud based on the relationship between the cargo and the storage platform. In the case of three-dimensional point cloud missing in single-view shooting scene, the bounding box of the cargo can be obtained more accurately, so that the volume of the cargo can be calculated accurately, and the method is simple and the result is accurate. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a method flowchart of a cargo volume measurement method in an embodiment of the present application.

[0015] Figure 2 FIG. 2 is a method flowchart of a cargo volume measurement method in another embodiment of the present application.

[0016] Figure 3 FIG. 3 is a plane distribution diagram of three-dimensional point clouds in an initial point cloud set obtained in an example.

[0017] Figure 4 FIG. 4 is a plane distribution diagram of three-dimensional point clouds in an initial storage platform point cloud set obtained in the embodiment shown in FIG. 1. Figure 3

[0018] FIG. 5 is a schematic diagram of a unit normal vector of the initial storage platform point cloud set, a unit direction vector of the Z direction of the world coordinate system, and the included angle between the two vectors in the example shown in FIG. 1. Figure 5 Figure 4

[0019] Figure 6 FIG. 7 is a plane distribution diagram of three-dimensional point clouds in a rotated cargo point cloud set obtained in the example shown in FIG. 1. Figure 3

[0020] FIG. 8 is a plane distribution diagram of three-dimensional point clouds in a rotated storage platform point cloud set obtained in the example shown in FIG. 1. Figure 7 Figure 3

[0021] Figure 8 ​​​​​​Fig. 1 is a schematic diagram of the distribution of three-dimensional point clouds in the world coordinate system in the bottom edge point cloud set extracted from the repaired goods point cloud set, and a schematic diagram of the distribution of three-dimensional point clouds in the world coordinate system after projecting the three-dimensional point clouds onto the XOY plane.

[0022] Figure 9 Fig. 2 is a schematic diagram of the distribution of three-dimensional point clouds in the world coordinate system in the target point cloud set and a schematic diagram of the bounding box constructed. DETAILED DESCRIPTION

[0023] The specific embodiments of the present application will be further described below with reference to the accompanying drawings.

[0024] The present application discloses a goods volume measurement method based on three-dimensional point cloud data, please refer to Figure 1 the flowchart shown in the figure, the method comprises the following steps:

[0025] Step 110, color images and depth images of goods placed on a goods table and a background thereof are acquired, and initial goods point cloud set and initial goods table point cloud set are obtained based on the acquired color images and depth images.

[0026] The initial goods point cloud set includes three-dimensional point clouds in a region where the goods are located, and the initial goods table point cloud set includes three-dimensional point clouds in a region where the goods table of the goods background is located.

[0027] Step 120, the initial goods point cloud set is rotated to a vertical downward overhead view to obtain a rotated goods point cloud set, and the initial goods table point cloud set is rotated to a vertical downward overhead view to obtain a rotated goods table point cloud set.

[0028] Step 130, point cloud repair is performed on the rotated goods point cloud set according to the rotated goods table point cloud set to obtain a repaired goods point cloud set.

[0029] Step 140, a bottom edge point cloud set is extracted from the repaired goods point cloud set, and the bottom edge point cloud set includes three-dimensional point clouds located at the bottom edge of the same side of the goods.

[0030] Step 150, a point cloud rotation matrix is obtained based on the bottom edge point cloud set, and the three-dimensional point clouds in the repaired goods point cloud set are rotated according to the point cloud rotation matrix to obtain a target point cloud set.

[0031] Step 160, a bounding box of the target point cloud set is constructed, and the volume of the goods is calculated based on the constructed bounding box.

[0032] Based on the cargo volume measurement method provided in the embodiment, the three-dimensional point cloud is repaired and rotationally transformed based on the relationship between the cargo and the storage table. In the case of missing three-dimensional point cloud in the single-view shooting scene, the bounding box of the cargo can also be accurately obtained, so that the volume of the cargo can be accurately calculated.

[0033] In another embodiment, referring to the flowchart shown in Figure 2 The cargo volume measurement method based on three-dimensional point cloud data provided in the embodiment includes the following steps:

[0034] In step 201, a color camera in the camera device acquires a color image within the field of view, and a depth camera in the camera device acquires a depth image.

[0035] The camera device is suspended above the storage table and faces the cargo, and the field of view of the camera device covers the upper surface of the cargo, at least one side of the cargo, and the storage table in the background of the cargo. In this application, the shooting angle of the camera device is generally fixed after the camera device is erected, that is, the color image and the depth image acquired in this step are shot by the camera device at a fixed position.

[0036] In step 202, the acquired color image and depth image are aligned to obtain a pose image, and point cloud calculation is performed on the pose image, so that the coordinates of the three-dimensional point cloud corresponding to each pixel in the pose image in the world coordinate system OXYZ can be calculated, and an initial point cloud set is obtained. The initial point cloud set includes all three-dimensional point clouds in the cargo and the storage table region in the background. The alignment of the color image and the depth image and the point cloud calculation are common methods in image processing technology, which will not be described in detail in this embodiment.

[0037] For example, in one instance, the plane diagram of the three-dimensional point cloud in the acquired initial point cloud set is as shown in Figure 3 As shown in Figure 3 It can be seen that due to the camera device shooting, there is no three-dimensional point cloud in part of the region.

[0038] In step 203, the initial point cloud set is divided into an initial cargo point cloud set and an initial storage table point cloud set based on the RANSAC algorithm. It includes: using the RANSAC algorithm to perform plane segmentation on the initial point cloud set to obtain a plane point cloud set, until the maximum value Z max and the minimum value Z min of the Z-axis coordinates of all three-dimensional point clouds in the plane point cloud set in the world coordinate system satisfy min{|dis max -Z max |,|dis min -Z min |}≤diff and |Zmax -Z min When |diff| is less than or equal to a threshold value, the three-dimensional point clouds in the set of planar point clouds obtained by the segmentation are determined as the initial set of point clouds of the placement table, and the three-dimensional point clouds in the initial set of point clouds except the initial set of point clouds of the placement table are determined as the initial set of point clouds of the goods.

[0039] dis max is the maximum value of the Z-axis coordinates of all the three-dimensional point clouds in the initial set of point clouds in the world coordinate system, and dis min is the minimum value of the Z-axis coordinates of all the three-dimensional point clouds in the initial set of point clouds in the world coordinate system. min{|dis max -Z max |,|dis min -Z min |} represents the minimum value of |dis max -Z max | and |dis min -Z min |, and diff is a deviation threshold value. |dis max -Z max | represents the absolute value of the difference between dis max and Z max . |dis min -Z min | represents the absolute value of the difference between dis min and Z min . |Z max -Z min | represents the absolute value of the difference between Z max and Z min .

[0040] In the example shown in FIG. 18, a planar schematic diagram of the three-dimensional point clouds in the initial set of point clouds of the placement table extracted by this step is shown in FIG. 19. Figure 3 Figure 4

[0041] Step 204, respectively rotate the initial set of point clouds of the goods and the initial set of point clouds of the placement table to a vertical downward overhead view to obtain a rotated set of point clouds of the goods and a rotated set of point clouds of the placement table. The view rotation matrix R1 is determined as follows:

[0042]

[0043] wherein, is the unit normal vector of the plane on which all the three-dimensional point clouds in the initial set of point clouds of the placement table lie. The vector is the unit direction vector of the Z direction of the world coordinate system, is the unit direction vector of the Y direction of the world coordinate system, and is the unit direction vector of the X direction of the world coordinate system. ​​​The included angle, vector sum vector The unit normal vector of the plane

[0044] Figure 5 This diagram illustrates the distribution of the 3D point cloud in the world coordinate system of the initial platform point cloud set. The unit normal vector of the resulting planar point cloud set is...

[0045] Then, the 3D point cloud in the initial cargo point cloud set is rotated according to the view rotation matrix R1 to obtain the rotated cargo point cloud set. Similarly, the 3D point cloud in the initial platform point cloud set is rotated according to the view rotation matrix R1 to obtain the rotated platform point cloud set. In other words, any 3D point cloud with coordinates (x, y, z) is rotated to coordinates R1·(x, y, z).

[0046] exist Figure 3 , Figure 4 and Figure 5 In the example shown, the planar distribution diagram of the 3D point cloud in the rotated cargo point cloud set is as follows: Figure 6 As shown, the planar distribution diagram of the 3D point cloud in the point cloud set after rotation of the stage is as follows. Figure 7 As shown.

[0047] Step 205: Repair the point cloud set of the rotated cargo based on the point cloud set of the rotated platform to obtain the repaired point cloud set of the cargo.

[0048] Due to the way the goods are placed on the platform and the shooting angle of the camera, the point cloud set of the goods after rotation will generally be missing some three-dimensional point clouds, especially the three-dimensional point clouds on the bottom surface and the sides that cannot be captured by the camera. These missing three-dimensional point clouds are the main reason why the bounding box is not accurately established in traditional volume measurement methods.

[0049] In this application, the average Z-axis coordinate of all 3D point clouds in the world coordinate system of the point cloud set after rotation of the stage is first determined. mean Then, add the cargo point cloud set after rotation to the set with coordinates (X) in the world coordinate system. min Y max Z mean ) and (X max Y max Z mean The 3D point cloud of the cargo was obtained by repairing the cargo point cloud set. The two added 3D point clouds were placed in key locations, which can effectively make up for the problem of missing 3D point clouds on some surfaces. Although the number of 3D point clouds added for repair is not large, it can significantly improve the accuracy of subsequent bounding box construction.

[0050] wherein X max is the maximum value of the X-axis coordinate in the world coordinate system of all three-dimensional point clouds in the rotated goods point cloud set, X min is the minimum value of the X-axis coordinate in the world coordinate system of all three-dimensional point clouds in the rotated goods point cloud set, Y max is the maximum value of the Y-axis coordinate in the world coordinate system of all three-dimensional point clouds in the rotated goods point cloud set, Y max is the side of the goods away from the camera device.

[0051] Step 206, a bottom edge point cloud set is extracted from the repaired goods point cloud set. It includes: determining the minimum value Z min of the Z-axis coordinate in the world coordinate system of all three-dimensional point clouds in the repaired goods point cloud set, and then extracting the three-dimensional point clouds in the repaired goods point cloud set whose Z-axis coordinate in the world coordinate system is less than or equal to Z min +th, to construct a bottom edge point cloud set. Th is a threshold value, which can be set by the user.

[0052] Step 207, the Z-axis coordinate in the world coordinate system of the three-dimensional point clouds in the bottom edge point cloud set is set to 0, so that the three-dimensional point clouds in the bottom edge point cloud set are projected to the XOY plane of the world coordinate system.

[0053] In the examples shown in Figure 3 , Figure 4 and Figure 5 , the distribution of the three-dimensional point clouds in the bottom edge point cloud set extracted from the repaired goods point cloud set in the world coordinate system, and the distribution of these three-dimensional point clouds in the world coordinate system after being projected to the XOY plane are shown in Figure 8 .

[0054] Step 208, based on the SVD singular value decomposition algorithm, the three-dimensional point clouds in the bottom edge point cloud set projected to the XOY plane of the world coordinate system are linearly fitted, and a bottom edge fitting straight line is fitted.

[0055] Step 209, based on the bottom edge fitting straight line, a point cloud rotation matrix is obtained, including determining the point cloud rotation matrix R2 as:

[0056]

[0057] wherein, the vector is the unit direction vector of the bottom edge fitting straight line, the vector is the unit direction vector of the X direction of the world coordinate system, and θ is the included angle between the vector and the vector , the vector and the vector Unit normal vector of the plane Vector And vector And the angle θ between them are shown in Figure 8 .

[0058] Step 210, according to the point cloud rotation matrix, the three-dimensional point cloud in the repaired cargo point cloud set is rotated to obtain the target point cloud set, and the obtained target point cloud set not only rotates and adjusts, but also completes the point cloud repair.

[0059] Step 211, then build the bounding box of the target point cloud set, and calculate the volume of the cargo based on the bounding box obtained. In one embodiment, the AABB bounding box is calculated for the three-dimensional point cloud in the target point cloud set, which is more accurate than using the OBB bounding box, and the eight vertices of the obtained AABB bounding box are (x min ,y min ,z min ), (x min ,y min ,z max ), (x max ,y min ,z min ), (x max ,y min ,z max ), (x min ,y max ,z min ), (x min ,y max ,z max ), (x max ,y max ,z min ) and (x max ,y max ,z max ). Please refer to the example schematic diagram shown in Figure 9 (x min ,y max ,z min ) and (x max ,y max ,z min ) are obtained based on the two three-dimensional point clouds added in step 205, and from the example, it can be seen that the two three-dimensional point clouds added for repair are important for the construction of the bounding box, and directly affect the accuracy of the bounding box. After obtaining the individual bounding box, the cargo volume can be calculated according to the conventional method.

[0060] Wherein, x min is the minimum value of the X-axis coordinate of all three-dimensional point clouds in the target point cloud set in the world coordinate system, y minis the minimum value of the Y-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, z min is the minimum value of the Z-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system; x max is the maximum value of the X-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, y max is the maximum value of the Y-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, z max is the maximum value of the Z-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system.

[0061] The above only describes the preferred embodiments of the present application, and the present application is not limited to the above embodiments. It can be understood that other improvements and changes directly derived or thought of by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A cargo volume measurement method based on three-dimensional point cloud data, characterized by, The method comprises: acquiring a color image and a depth image of goods and a background on which the goods are placed on a placement table, and obtaining an initial goods point cloud set and an initial placement table point cloud set based on the acquired color image and depth image, the initial goods point cloud set comprising a three-dimensional point cloud in a region where the goods are located, and the initial placement table point cloud set comprising a three-dimensional point cloud in a region where a placement table of the goods background is located; rotating the initial goods point cloud set to a vertical downward overhead view to obtain a rotated goods point cloud set, and rotating the initial placement table point cloud set to a vertical downward overhead view to obtain a rotated placement table point cloud set; performing point cloud repair on the rotated goods point cloud set according to the rotated placement table point cloud set to obtain a repaired goods point cloud set; extracting a bottom edge point cloud set from the repaired goods point cloud set, the bottom edge point cloud set comprising a three-dimensional point cloud of a bottom edge located on the same side of the goods; obtaining a point cloud rotation matrix based on the bottom edge point cloud set, and rotating the three-dimensional point cloud in the repaired goods point cloud set according to the point cloud rotation matrix to obtain a target point cloud set; constructing a bounding box of the target point cloud set, and calculating a volume of the goods based on the constructed bounding box.

2. The method of claim 1, wherein, The method for performing point cloud repair on the rotated goods point cloud set according to the rotated placement table point cloud set to obtain a repaired goods point cloud set comprises: determining an average of Z-axis coordinates of all three-dimensional point clouds in the rotated post-object stage point cloud set in the world coordinate system ; Add the coordinates in the world coordinate system to the rotated cargo point cloud set. as well as The three-dimensional point cloud is used to obtain the repaired cargo point cloud set; wherein, It is the maximum value of the X-axis coordinates of all 3D point clouds in the world coordinate system of the rotated cargo point cloud set. It is the minimum value of the X-axis coordinate of all 3D point clouds in the world coordinate system of the rotated cargo point cloud set. It is the maximum value of the Y-axis coordinates of all three-dimensional point clouds in the world coordinate system of the rotated cargo point cloud set. 3.The method of claim 1, wherein, The method for extracting a bottom edge point cloud set from the repaired goods point cloud set comprises: determining a minimum value of Z-axis coordinates of all three-dimensional point clouds in the world coordinate system in the repaired goods point cloud set ; extracting a three-dimensional point cloud whose Z-axis coordinate in a world coordinate system is less than or equal to a threshold value from the repaired goods point cloud set, to obtain the bottom edge point cloud set, is a threshold value. 4.The method of claim 1, wherein, The method for obtaining a point cloud rotation matrix based on the bottom edge point cloud set comprises: projecting the three-dimensional point cloud in the bottom edge point cloud set to an XOY plane of a world coordinate system by setting the Z-axis coordinate of the three-dimensional point cloud in the bottom edge point cloud set in the world coordinate system to 0; performing straight line fitting on the three-dimensional point cloud in the bottom edge point cloud set projected to the XOY plane of the world coordinate system based on an SVD singular value decomposition algorithm, and fitting to obtain a bottom edge fitting straight line; obtaining the point cloud rotation matrix based on the bottom edge fitting straight line. 5.The method of claim 4, wherein, The fitting straight line based on the bottom edge obtains the point cloud rotation matrix, including determining the point cloud rotation matrix is: ； in, ,vector It is the unit direction vector of the fitted straight line at the bottom edge, vector It is the unit direction vector in the X direction of the world coordinate system. It is a vector sum vector The included angle, vector sum vector The unit normal vector of the plane . 6.The method of claim 1, wherein, The method for acquiring the initial goods point cloud set and the initial placement table point cloud set comprises: obtaining an initial point cloud set based on the acquired color image and depth image, the initial point cloud set comprising all three-dimensional point clouds in a region of the goods and a placement table of the background thereof; dividing the initial point cloud set into the initial goods point cloud set and the initial placement table point cloud set based on an RANSAC algorithm.

7. The method of claim 6, wherein, The method for dividing the initial point cloud set into the initial goods point cloud set and the initial placement table point cloud set comprises: determining a maximum value of Z-axis coordinates of all three-dimensional point clouds in the initial point cloud set in the world coordinate system and a minimum value of Z-axis coordinates ; The RANSAC algorithm is used for recycling to perform plane segmentation on the initial point cloud set to obtain a plane point cloud set, until the maximum value of the Z-axis coordinates of all three-dimensional point clouds in the plane point cloud set obtained by segmentation in the world coordinate system and the minimum value of the Z-axis coordinates satisfy and When the three-dimensional point clouds in the plane point cloud set obtained by segmentation constitute the initial placement table point cloud set, and the three-dimensional point clouds in the initial point cloud set other than the three-dimensional point clouds in the initial placement table point cloud set constitute the initial cargo point cloud set. wherein, represents the minimum value of and , is a deviation threshold value, represents the minimum value of and , represents the minimum value of and , represents the minimum value of and . 8.The method of claim 1, wherein, The method for rotating the initial goods point cloud set and the initial placement table point cloud set comprises: Determining a view angle rotation matrix is: ； wherein is a unit normal vector of a plane on which all three-dimensional point clouds in the initial set of point clouds lie, vector is a unit normal vector of a plane on which all three-dimensional point clouds in the initial set of point clouds lie, vector is a unit direction vector of the Z direction of the world coordinate system, is an angle between vector and vector , vector is a unit normal vector of a plane on which all three-dimensional point clouds in the initial set of point clouds lie, vector is a unit normal vector of a plane on which all three-dimensional point clouds in the initial set of point clouds lie, vector ; According to the view angle rotation matrix Rotating the three-dimensional point clouds in the initial set of cargo point clouds to obtain a set of rotated cargo point clouds; According to the view angle rotation matrix Rotating the three-dimensional point clouds in the initial set of product table point clouds to obtain the set of rotated product table point clouds. 9.The method of claim 1, wherein, The method for constructing a bounding box of the target point cloud set comprises: An AABB bounding box is calculated for the three-dimensional point cloud in the target point cloud set, and the eight vertices of the AABB bounding box are , , , , , , and ; wherein, is a minimum value of X-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, is a minimum value of Y-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, is a minimum value of Z-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system; is a maximum value of X-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, is a maximum value of Y-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system, is a maximum value of Z-axis coordinates of all three-dimensional point clouds in the target point cloud set in the world coordinate system. 10.The method of claim 6, wherein, The method for obtaining an initial point cloud set based on the acquired color image and depth image comprises: acquiring the color image within a field of view range by a color camera in a camera device, and acquiring the depth image by a depth camera in the camera device, the camera device being suspended above the goods shelf and facing the goods, a field of view range of the camera device covering an upper surface of the goods, at least one side surface of the goods, and the goods shelf; aligning the acquired color image and the acquired depth image to obtain a pose image, and performing point cloud calculation on the pose image to obtain the initial point cloud set.

Images