A method for measuring the volume of cubic objects based on depth images
Through the depth image-based method, segmenting and fitting images of cubic objects, detecting and clustering edge lines, extracting corner points, and calculating the volume of cubic objects, the limitations and time-consuming problems of cubic objects in the logistics industry are solved, and the data are improved. The accuracy and stability of the measurement are obtained.
Patent Information
- Application Number
- CN201910990407.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-17
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has limitations and time-consuming problems in the volume measurement of cubic object-like packages, goods, and containers in the logistics industry, and it is difficult to accurately measure in scenarios where packages are densely packed, and the calculation accuracy is affected by external environmental noise interference.
Using the cubic object volume measurement method based on depth images, by acquiring depth images, segmenting images to distinguish the upper surface and vertical planes, detect edge lines, cluster and merge edge lines, extract corner points, pair corner points, fit three-dimensional planes, and eliminate noise Point, calculate the length, width and height of the cubic object to calculate the volume.
It improves the accuracy and stability of cubic object volume measurement, reduces the calculation amount, enhances the stability of the algorithm, and can quickly and accurately measure the volume in densely packed scenarios, reducing the impact of noise interference.
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target recognition of depth images, and in particular to a method for measuring the volume of a cubic object based on a depth image. Background Art
[0002] Depth images, also known as distance images, refer to images that use the distance from the image collector to each point in the scene as the pixel value. They directly reflect the geometry of the visible surface of the target. Therefore, depth images are widely used, especially in the logistics industry. At present, the throughput of domestic logistics, port shipping, and airport shipping is very large and increasing. The volume measurement of parcels, goods, containers, etc. at various sorting centers is relatively limited and time-consuming, which reduces transportation efficiency.
[0003] In the prior art, most packages are in the shape of cubic objects. The volume measurement of packages is mainly achieved by comparing the depth images of the conveyor belt before and after loading, extracting the length and width of the package from the rectangular area where the package is located, and obtaining the height of the package using the difference feature to calculate the volume of the package. This method is limited to specific scenarios and has a small application scope, and cannot meet the needs of densely stacked packages in the actual logistics industry. In addition, the prior art is difficult to accurately detect multiple faces of cubic objects, and is affected by external environmental noise, which affects the calculation accuracy. There is basically no distinction between the planes of the cubic object, which can easily lead to misjudgment of the length, width and height of the rectangular object, resulting in large errors in volume calculation. At present, the volume measurement of large goods and containers is also basically done by manual measurement. Summary of the invention
[0004] In order to solve the problem of limitation and time-consuming volume measurement of cubic object-shaped packages, goods, containers and the like in the prior art, the present invention proposes a cubic object volume measurement method based on depth image.
[0005] The technical solution of the present invention is achieved in this way:
[0006] A method for measuring the volume of a cubic object based on a depth image, comprising:
[0007] S1. Obtain the depth image of the target, and obtain the gradient maps of the same size in the X and Y directions according to the depth image; S2. According to the gradient map, the pixels in it are divided into three categories: horizontal plane, left vertical plane and right vertical plane, and the edge point map of the depth image is obtained; S3. According to the classification results, the gradient map and the edge point map, the segmented images of the upper surface, left vertical plane and right vertical plane of the target in the depth image are obtained; S4. Edge line detection is performed on the edge point map, and all detected edge lines are clustered and merged to obtain a set of collinear edge line groups; S5. Corner points are extracted according to the longest edge line of each group of collinear edge line groups; S6. The extracted corner points are pairing; S7, determine the upper surface and overall contour of the cubic object according to the acquired quadrilateral and upper surface segmentation image; S8, fit the plane in the three-dimensional space according to the upper surface of the cubic object, eliminate the noise points on each surface by repeated fitting, obtain the ideal point set corresponding to the upper surface, perform contour detection according to the ideal point set, and obtain the contour edge line of the upper surface; S9, determine the upper surface quadrilateral according to the contour edge line of the upper surface; S10, calculate the length and width of the cubic object according to the upper surface quadrilateral, fit the three-dimensional plane where the ground plane is located, calculate the height of the cubic object according to the distance between the upper surface and the ground plane, and calculate the volume of the cubic object.
[0008] Preferably, the specific method of obtaining the segmented images of the upper surface, the left vertical surface and the right vertical surface of the target in the depth image according to the classification result, the gradient map and the edge point map in S3 is:
[0009] According to the classification results, binary images of the horizontal plane, left vertical plane and right vertical plane with the same size as the gradient map are obtained, and the upper surface segmentation image with the same size as the gradient map is defined, and the initial value of the pixel point is set to 0;
[0010] Traverse the pixels in the first column of the binary image of the horizontal plane from bottom to top, obtain the credibility value of each pixel according to the depth map, give a credibility threshold, if the credibility value of the pixel is less than the given credibility threshold, the state of the pixel is unknown, and process the next pixel; otherwise, if the corresponding pixel value is 255, the state of the pixel is horizontal, if the corresponding pixel is 0, the state of the pixel is vertical, if the number of pixels with a pixel value of 0 is greater than the given height threshold, then the pixels with a pixel value of 255 that appear later are mapped to the upper surface segmentation image and their pixel values are updated to 255, and the row number of the pixel where the pixel state changes each time is recorded. After traversing all rows of the first column, set the pixel values of all pixels in the first column from the recorded row number to the first row corresponding to the position on the upper surface segmentation image to 0, traverse all rows and columns of the parallel surface image in the same way, obtain the initial upper surface segmentation image, repair the obtained initial upper surface segmentation image, and update the upper surface segmentation image;
[0011] The edge points in the edge point map are respectively mapped one by one to the updated upper surface segmentation image, the binary map of the left vertical surface, and the binary map of the right vertical surface. The pixel values of the horizontal non-edge points in the updated upper surface segmentation image are updated to 0, the pixel values of the non-edge points in the binary map of the left vertical surface with a pixel value of 255 are updated to 0, and the pixel values of the non-edge points in the binary map of the right vertical surface with a pixel value of 255 are updated to 0, so as to obtain the final upper surface segmentation image, the binary map of the left vertical surface, and the binary map of the right vertical surface. The binary map of the left vertical surface and the binary map of the right vertical surface are respectively recorded as the segmentation image of the left vertical surface and the segmentation image of the right vertical surface.
[0012] Preferably, the specific method of obtaining the edge point map of the depth image is:
[0013] Calculate the average gradient Yp of all pixels in the vertical plane after classification, the average gradient Yq of all pixels in the horizontal plane, the average gradient Xp of all pixels in the left vertical plane, and the average gradient Xq of all pixels in the right vertical plane; set the positive threshold in the Y direction, the negative threshold in the Y direction, the positive threshold in the X direction, and the negative threshold in the X direction according to Yp, Yq, Xp, and Xq; define an image with the same size as the gradient map, and set the initial value of the pixel in the image to 0; traverse all the pixels in the gradient map in the X direction and the gradient map in the Y direction, and for For each pixel point, if the absolute value of the gradient value in the Y direction is greater than the absolute value of the gradient value in the X direction, and is less than the negative threshold in the Y direction or greater than or equal to the positive threshold in the Y direction, the pixel point is an edge point, and the pixel point is mapped to the defined image and its pixel value is updated to 255; if the absolute value of the gradient value in the Y direction is less than the absolute value of the gradient value in the X direction, and the absolute value of the gradient value in the X direction is less than or equal to the negative threshold in the X direction or greater than or equal to the positive threshold in the X direction, the pixel point is an edge point, and the pixel point is mapped to the defined image and its pixel value is updated to 255; an edge point map is obtained.
[0014] Preferably, edge line detection is performed on the edge point graph, and all detected edge lines are clustered and merged to obtain a collinear edge line group set in the following specific manner:
[0015] Perform edge line detection on the edge point graph, obtain all edge lines and sort them into edge line sets from long to short, define an empty collinear edge line group set, the collinear edge line group set contains multiple groups of edge lines, the edge lines in each group are collinear, and the edge lines between two groups are not collinear. First, add the longest edge line to one of the collinear edge line group sets, for each A edge line in the edge line set, determine whether it is collinear with the longest B edge line in each group of the collinear edge line group set, and the judgment conditions are as follows:
[0016] C1. Calculate the angle difference between edge line A and edge line B, and determine whether the angle difference is less than or equal to a given angle difference threshold value e;
[0017] C2. If the angle difference is less than or equal to the angle difference threshold e, calculate the distances T1 and T2 from the start point of edge line B to the start point and end point of edge line A, and T3 and T4 from the end point of edge line B to the start point and end point of edge line A. Given the point distance threshold G, determine whether the minimum distance among the distances T1, T2, T3 and T4 is less than or equal to the point distance threshold G.
[0018] C3. If min(min(T1, T2), min(T3, T4))≤G, calculate the length L of edge line A and the length H of edge line B, and determine whether the maximum distance among distances T1, T2, T3 and T4 is less than G+H+L;
[0019] C4. If max(max(T1, T2), max(T3, T4))<(G+H+L), calculate the distances S1 and S2 from the start and end points of edge line B to edge line A, and the distances S3 and S4 from the start and end points of edge line A to edge line B. Given a distance threshold J, if the maximum distance between T1 and T2 is less than L+G, determine whether S1 is less than or equal to J; if the maximum distance between T3 and T4 is less than L+G, determine whether S2 is less than or equal to J; if the maximum distance between T1 and T3 is less than L+G, determine whether S3 is less than or equal to J; if the maximum distance between T2 and T4 is less than L+G, determine whether S4 is less than or equal to J; if any maximum distance is greater than or equal to L+G, directly execute the next step;
[0020] C5. If the condition in C4 is met, the edge line A and the edge line B are collinear, and the edge line A is added to the group of the collinear edge line group set, and the edge line of the group is updated;
[0021] During the judgment process, if any condition is not met, directly process the longest edge line of the next group in the collinear edge line group set, until all groups in the collinear edge line group set are processed, and each group of edge lines is represented by its longest edge line. If edge line A is not collinear with the longest edge lines of all groups in the collinear edge line group set, directly add edge line A to the new group in the collinear edge line group set, and update the number of groups in the collinear edge line group set; after edge line A and edge line B are collinear, determine whether edge line A and edge line B meet the merging condition. If T1≤H, T2≤H, T3≤H and T4≤H, edge line A and edge line B do not meet the merging condition, otherwise, merge edge line A and edge line B, update the starting point and end point of edge line B, and update the collinear edge line group set;
[0022] The next edge line in the edge line set is processed in the same way as edge line A, and the collinear edge line group set is continuously updated. The edge lines of each group in the collinear edge line group set are sorted by length to obtain the final collinear edge line group set.
[0023] Preferably, the specific method of extracting corner points according to the longest edge line of each group of collinear edge line groups is:
[0024] C1. Get the longest edge line of each group of collinear edge line groups, record it as the seed edge line set, sort all the edge lines of the seed edge line set from long to short, traverse all the edge lines of the seed edge line set, and start from the longest C edge line to calculate the angle a between it and any other D edge line, where a ranges from 0 to 180 degrees. Given a curvature threshold b, determine whether min (a, 180-a) is greater than b;
[0025] C2. If min(a, 180-a)>b, calculate the intersection point W(Wx, Wy) of the edge line C and the edge line D, and determine whether Wx or Wy does not exceed the image range;
[0026] C3. If Wx or Wy does not exceed the image range, determine whether the intersection W is a corner point. The condition for determining whether the intersection W is a corner point is as follows: first calculate the distances d1 and d2 from the two endpoints of the C edge line to the intersection W, and define the minimum distance min_d1. If the intersection W is between the two endpoints of the C edge line, then min_d1=0, otherwise min_d1=min(d1, d2). If the minimum distance min_d1 is not 0 or min(d1, d2) is less than the given arm length threshold Z1, then calculate the distances d3 and d4 from the two endpoints of the D edge line to the intersection W, and define the minimum distance min_d2. If the intersection W is between the two endpoints of the D edge line, then min_d2=0, otherwise min_d2=min(d3, d4). If the minimum distance min_d2 is not 0 or min(d3, d4) is less than the given arm length threshold Z2, determine whether min(d1, d2) and min(d3, d4) are less than the given intersection radius threshold r.
[0027] C4. If the condition of C3 is met, given the threshold value v of the gap between the arms of the quadrilateral, determine whether max(min(d1, d2), min(d3, d4)) is less than or equal to v. If so, the intersection point W is the corner point of the quadrilateral.
[0028] The conditions C1 to C4 are judged in sequence. If any condition is not met, the next edge line is processed;
[0029] The first corner point obtained is added to the corner point set, and for the remaining corner points obtained, the distance L between them and all corner points in the corner point set is calculated. i, i is a variable, judging whether it is similar to all corner points in the corner point set, if L i <r, and is not similar to the other corner points, add the corner point to the corner point set;
[0030] The intersection point W corresponds to two arms, the first arm is the line connecting it with the end point of the C edge line, and the second arm is the line connecting it with the end point of the D edge line. The end point of the C edge line is the larger corresponding endpoint of d1 and d2 in the C edge line, and the end point of the D edge line is the larger corresponding endpoint of d3 and d4 in the D edge line. The same method is used to process any other two edge lines in the seed edge line group set to obtain all corner points, and the two arms of each corner point are expanded according to the seed edge line set to obtain the final corner point set, thereby realizing corner point extraction.
[0031] Preferably, the specific method of pairing the extracted corner points is:
[0032] C1. Pair all corner points in the corner point set. If the coordinates of the two paired corner points are different, continue to determine whether the two corner points share an arm. Mark the two arms of one corner point as arm11 and arm12, and the two arms of the other corner point as arm21 and arm22. First determine whether arm11 and arm21 are the same arm of the two corner points.
[0033] C2. The method of judging whether arm11 and arm21 are the same arm is as follows: if the distance between the two corner points is greater than one-fourth of the minimum value of arm11 and arm21, first calculate the absolute value x1 of the angle difference between arm11 and the line connecting the two corner points, and the absolute value x2 of the angle difference between arm21 and the line connecting the two corner points. The value range of x1 and x2 is [0, 180]. Given an angle similarity threshold f, if both x1 and x2 are less than the angle similarity threshold f, given a corner point distance threshold u and a length ratio threshold ratio, if the sum of the lengths of arm11 and arm21 is greater than or equal to ratio times the distance between the two corner points, and greater than or equal to the difference between the distance between the two corner points and u, then arm11 and arm21 are judged to be the same arm and the corner point pairing is successful.
[0034] Otherwise, the same method is used to determine whether the other two arms of the two corner points are the same arm. If neither is the case, the corner point pairing fails, otherwise the corner point pairing succeeds.
[0035] In the same way, the remaining corner points are paired in pairs to obtain all successfully paired corner point combinations. The same arm of each corner point combination is the shared arm, and the other two arms are open arms. The angles u1 and u2 between the two open arms and the shared arm in each corner point combination are calculated. The value range of the angles u1 and u2 is [0, 180]. The corner point combinations whose angles u1 or u2 are less than the curvature threshold b are eliminated; for the remaining corner point combinations, determine whether their two open arms are on the same side of the shared arm, and eliminate the corner point combinations whose two open arms are not on the same side of the shared arm. Finally, determine a quadrilateral based on the currently paired corner point combination; process all corner point combinations in the same way to obtain all quadrilaterals.
[0036] Preferably, the specific method of obtaining the contour edge line of the upper surface in S8 is:
[0037] C1. Traverse all the pixel points on the upper surface of each detected cubic object, and obtain the world coordinate points of all the pixel points on the upper surface in the three-dimensional space according to the transformation relationship between the image coordinate system and the world coordinate system;
[0038] C2, traverse all world coordinate points on the current upper surface, give a point count threshold, and determine whether all points are greater than the given point count threshold;
[0039] C3. If the number of points is greater than the point threshold, calculate the mean point of all points and perform normalization. All coordinate points on the upper surface of each cubic object are located in the same plane in the three-dimensional space, and the mean point is located on the plane. Calculate the normal vector of the plane in the three-dimensional space where the mean point is located, and obtain the initial fitting three-dimensional plane equation based on the normal vector and the mean point.
[0040] C4, calculate the distances from all world coordinate points on the upper surface to the first fitted three-dimensional plane, give a distance threshold, remove all points whose distances are greater than the given distance threshold, obtain an ideal point set, perform fitting again based on the ideal point set, and obtain a second fitted three-dimensional plane equation;
[0041] C5. Repeat the above steps to perform multiple repetitive fittings. After each fitting, calculate the distances from all points in the ideal point set to the fitted three-dimensional plane, remove all points whose distances are greater than a given distance threshold, and update the ideal point set, which is recorded as the upper ideal point set.
[0042] C6. Obtain the plane coordinate points on the two-dimensional plane corresponding to all world coordinate points of the ideal point set on the current upper surface, and perform contour detection based on them, obtain all contours, calculate the overlapping area between the circumscribed rectangle of each contour and the circumscribed rectangle of the current upper surface, and find the contour with the largest overlapping area that is greater than a given threshold. This contour is the final contour of the upper surface, calculate the convex hull of the upper surface based on the contour, obtain the corresponding fitting polygon based on its convex hull, obtain all edge lines of the fitting polygon of the upper surface and sort them from long to short, and these edge lines are the contour edge lines of the upper surface.
[0043] Preferably, the specific method of determining the upper surface quadrilateral according to the contour edge line of the upper surface in S9 is:
[0044] Given a parallel angle difference threshold j, determine whether the angle difference between any two contour edge lines is less than j. If this condition is met, the current two contour edge lines are in the same group. Group all contour edge lines. If the total number of groups after grouping is greater than or equal to 2, calculate the total length of the contour edge lines of each group and sort them from large to small. The two groups with the largest total length are recorded as group 1 and group 2. Group 1 contains a group of approximately parallel sides of the upper surface quadrilateral, and group 2 contains another group of approximately parallel sides of the upper surface quadrilateral. If the total number of groups after grouping is less than 2, process the next upper surface.
[0045] Calculate the distance difference between any two contour edge lines in group 1 and group 2 respectively, give a parallel distance difference threshold, and divide the contour edge lines in group 1 and group 2 whose distance difference is less than the parallel distance difference threshold into the same group respectively. If the total number of groups after division in group 1 is greater than or equal to 2, calculate the total length of the edge lines of each group in group 1 and sort them from large to small, and record the two groups with the largest total length as group 11 and group 12. If the total number of groups after division in group 2 is greater than or equal to 2, calculate the total length of the edge lines of each group in group 2 and sort them from large to small, and record the two groups with the largest total length as group 21 and group 22. If the total number of groups after division in group 1 or group 2 is less than 2, process the next upper surface.
[0046] Traverse all pixel points of the upper surface contour, calculate the distance from each pixel point to all contour edge lines of group 11, group 12, group 21 and group 22 respectively, and obtain the contour edge line with the minimum distance in each group and less than a given threshold, which is the four sides of the upper surface.
[0047] Preferably, the specific method of calculating the volume of the cubic object in S10 is:
[0048] For any side of the upper surface quadrilateral, record it as side 1, and record the side that is approximately parallel to side 1 as side 2. Calculate the distance from all edge points of side 1 to the vertical plane where side 2 is located. At the same time, calculate the distance from all edge points of side 2 to the vertical plane where side 1 is located. Calculate the average value of all distances as the length of the cubic object corresponding to the current upper surface.
[0049] The other two approximately parallel sides of the upper surface quadrilateral are respectively recorded as side 3 and side 4, and the distances from all edge points of side 3 to the vertical plane where side 4 is located are calculated. At the same time, the distances from all edge points of side 4 to the vertical plane where side 3 is located are calculated, and the average value of all distances is calculated as the width of the cubic object corresponding to the current upper surface;
[0050] Calculate the distances from all points of the upper surface quadrilateral to the fitted three-dimensional plane of the ground plane and find the average. The average is the height of the cubic object. Calculate the volume of the cubic object based on the length, width and height of the cubic object.
[0051] Preferably, the method for obtaining the fitted three-dimensional plane of the ground plane is as follows: define a ground plane image of the same size as the gradient image, set the initial value of the pixel point to 0; traverse the binary image of the horizontal plane and the upper surface segmentation image, update the pixel point value on the ground plane image corresponding to the pixel point value of 255 in the binary image of the horizontal plane and the pixel point value of 0 in the upper surface segmentation image to 255, and obtain the ground plane image; calculate the world coordinate points of all pixel points with a pixel point value of 255 on the ground plane image in the three-dimensional space, calculate the mean point of all pixel points and perform normalization processing, all coordinate points and the mean point of the ground plane image are on the same plane in the three-dimensional space, and calculate Calculate the normal vector of the plane in the three-dimensional space where it is located, obtain the fitted three-dimensional plane equation of the ground plane according to the mean point and the normal vector of the ground plane, calculate the distance from the pixel point with a pixel value of 255 on the ground plane image to the fitted three-dimensional plane equation after fitting, eliminate the pixel points whose distance is greater than a given threshold, obtain the ideal point set belonging to the ground plane point, fit the three-dimensional plane equation of the ground plane again according to the ideal point set, calculate the distance from all pixel points of the ideal point set of the ground plane point to the fitted three-dimensional plane equation, eliminate the pixel points whose distance is greater than a given threshold, update the ideal point set of the ground plane point, repeat the fitting multiple times to eliminate the noise points and obtain the final fitted three-dimensional plane equation of the ground plane.
[0052] The beneficial effects of the present invention are as follows: the cubic object volume measurement method based on the depth image of the present invention, on the basis of quickly distinguishing the surface of the cubic object by segmenting the image, clusters and merges all the detected edge lines, obtains all the colinear edge line groups, selects the longest edge line of each group to obtain the corner point, reduces the amount of calculation, and improves the stability of the detection; then the corner points are paired to obtain all possible quadrilateral detections, and then all the upper surface contours are obtained by detecting the upper surface segmentation image, and the upper surface contours are matched with all the detected quadrilaterals to obtain the upper surface of the cubic object and the overall contour of the cubic object, and the plane of the cubic object in the three-dimensional space is fitted according to the upper surface of the cubic object, and the points with a large distance to the fitted three-dimensional plane are eliminated after each fitting, and the points are eliminated after repeated fitting. The noise points on each surface are removed to increase the accuracy of fitting and improve the stability of the algorithm. All possible contour boundary lines of the upper surface quadrilateral are obtained according to the fitted upper surface, and they are grouped according to the angle and the distance between the boundary lines. The four sides of the quadrilateral that can form the upper surface must contain two groups of approximately parallel sides, and the angles between the two groups of sides are different. For each group of approximately parallel sides, there must be a certain distance between the sides. This feature is used to obtain the longest boundary line that can form the upper surface quadrilateral among all the contour boundary lines of the upper surface, and the upper surface quadrilateral is finally determined. The length and width of the cubic object are obtained according to the two groups of sides of the upper surface, and the distance from each boundary point to the ground plane is calculated according to the four sides of the upper surface, the height of the cubic object is obtained, and the volume of the cubic object is calculated. The pixel classification method is applicable to any object in any scene. In actual application in volume measurement, the cubic object contains at least a visible upper surface and a vertical plane in the image, and the plane where the target is located in the depth image is the lowest plane or directly on the ground.
[0053] The cubic object volume measurement method of the present invention is applied to the volume measurement of objects such as parcels, goods, and containers. It avoids comparing the depth images of the conveyor belt before and after loading, but directly identifies the cubic object, and finally obtains the length, width, and height of the cubic object to calculate its volume, thereby solving the limitation and time-consuming problem of parcel volume measurement, helping the logistics industry to improve parcel sorting efficiency, and having important practical application significance. The cubic object volume measurement method of the present invention can also be applied in the field of image recognition. DETAILED DESCRIPTION
[0054] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] Embodiment: A method for measuring the volume of a cubic object based on a depth image, S1, obtaining a depth image of a target, and obtaining gradient maps of the same size in the X direction and the Y direction according to the depth image; S2, dividing the pixel points therein into three categories of horizontal plane, left vertical plane and right vertical plane according to the gradient map, and obtaining an edge point map of the depth image; S3, obtaining a segmented image of the upper surface, left vertical plane and right vertical plane of the target in the depth image according to the classification result, the gradient map and the edge point map; S4, performing edge line detection on the edge point map, clustering and merging all detected edge lines, and obtaining a set of collinear edge line groups; S5, extracting corner points according to all edge lines of the collinear edge line set; S6 , pair the extracted corner points; S7, determine the upper surface and overall contour of the cubic object based on the acquired quadrilateral and upper surface segmentation image; S8, fit the plane of the cubic object in the three-dimensional space according to the upper surface, eliminate the noise points on each surface by repeated fitting, obtain the ideal point set corresponding to the upper surface, perform contour detection based on the ideal point set, and obtain the contour edge line of the upper surface; S9, determine the upper surface quadrilateral according to the contour edge line of the upper surface; S10, calculate the length and width of the cubic object according to the upper surface quadrilateral, fit the three-dimensional plane where the ground plane is located, calculate the height of the cubic object according to the distance between the upper surface and the ground plane, and calculate the volume of the cubic object.
[0056] Depth images can be acquired using depth cameras. Current depth cameras can be divided into three types based on their working principles: TOF (Time Of Flight) depth cameras, binocular stereo depth cameras, and structured light depth cameras. In actual use, you can select a suitable depth camera to acquire depth images based on specific needs. The ground plane refers to the lowest plane in the depth image. When actually used in volume measurement, the plane where the cubic object is placed is the lowest plane in the depth image.
[0057] The specific method of dividing the pixels in S2 into three categories according to the gradient map: horizontal plane, left vertical plane and right vertical plane: traverse each pixel value DX (r, c) in the gradient map in the X direction, and each pixel value DY (r, c) in the gradient map in the Y direction, where r = 0...m, c = 0...n, m represents the number of rows in the gradient map, and n represents the number of columns in the gradient map; given a positive segmentation threshold p and a negative segmentation threshold q, the pixel points corresponding to DY (r, c) greater than or equal to p are located in the vertical plane, and the pixel points corresponding to DY (r, c) less than or equal to q are located in the horizontal plane; for all points on the vertical plane, the pixel points corresponding to DX (r, c) less than or equal to q are located in the left vertical plane, and the pixel points corresponding to DX (r, c) greater than or equal to p are located in the right vertical plane.
[0058] The specific method of obtaining the edge point map of the depth image in S2 is as follows: calculate the gradient average value Yp in the Y direction of all pixels of the vertical plane after classification, the gradient average value Yq in the Y direction of all pixels of the horizontal plane, the gradient average value Xp in the X direction of all pixels of the left vertical plane, and the gradient average value Xq in the X direction of all pixels of the right vertical plane; set the positive threshold in the Y direction, the negative threshold in the Y direction, the positive threshold in the X direction, and the negative threshold in the X direction according to Yp, Yq, Xp, and Xq; define an image with the same size as the gradient map, and set the initial value of the pixel in the image to 0; traverse all the gradient maps in the X direction and the gradient maps in the Y direction. Pixel point, for each pixel point, if the absolute value of its gradient value in the Y direction is greater than the absolute value of the gradient value in the X direction, and is less than the negative threshold in the Y direction or greater than or equal to the positive threshold in the Y direction, the pixel point is an edge point, and the pixel point is matched to the defined image and its pixel value is updated to 255; if the absolute value of its gradient value in the Y direction is less than the absolute value of the gradient value in the X direction, and the absolute value of its gradient value in the X direction is less than or equal to the negative threshold in the X direction or greater than or equal to the positive threshold in the X direction, the pixel point is an edge point, and the pixel point is matched to the defined image and its pixel value is updated to 255; an edge point map is obtained, and the edge point map is a binary map.
[0059] The specific method of obtaining the segmented images of the upper surface, left vertical surface and right vertical surface of the target in the depth image according to the classification results, the gradient map and the edge point map in S3 is as follows: according to the classification results, the binary images of the horizontal plane, the left vertical plane and the right vertical plane of the same size as the gradient map are obtained, and the upper surface segmented image of the same size as the gradient map is defined, and the initial value of the pixel is set to 0; the pixels in the first column of the binary image of the horizontal plane are traversed from bottom to top, and the credibility value of each pixel is obtained according to the depth map, and a credibility threshold is given. If the credibility value of the pixel is less than the given credibility threshold, the state of the pixel is unknown, and the next pixel is processed; otherwise, if the corresponding pixel value is 255, the state of the pixel is horizontal, if the corresponding pixel is 0, the state of the pixel is vertical, and if the number of pixels with a pixel value of 0 is greater than the given height threshold, the pixels with a pixel value of 255 that appear later are matched on the upper surface segmented image and their pixel values are updated to 255, and each change in the pixel state is recorded. After traversing all the rows of the first column, the pixel values of all the pixels in the first column from the recorded row number to the first row corresponding to the position on the upper surface segmentation image are set to 0. In the same way, all the rows and columns of the parallel surface image are traversed to obtain the initial upper surface segmentation image; the initial upper surface segmentation image is repaired to update the upper surface segmentation image; the edge points in the edge point map are respectively mapped one by one to the updated upper surface segmentation image, the binary image of the left vertical surface, and the binary image of the right vertical surface. In the binary image of the upper surface, the pixel values of the horizontal non-edge points in the updated upper surface segmentation image are updated to 0, the pixel values of the non-edge points in the binary image of the left vertical surface with a pixel value of 255 are updated to 0, and the pixel values of the non-edge points in the binary image of the right vertical surface with a pixel value of 255 are updated to 0, so as to obtain the final upper surface segmentation image, the binary image of the left vertical surface and the binary image of the right vertical surface, and record the binary image of the left vertical surface and the binary image of the right vertical surface as the segmentation image of the left vertical surface and the segmentation image of the right vertical surface, respectively.
[0060] During the calculation process, due to noise interference, individual points on the upper surface may be misjudged as points on the vertical surface, resulting in discontinuity of the upper surface points. In order to improve the stability of the algorithm, the upper surface segmentation image is patched. The specific method of patching the initial upper surface segmentation image is as follows: given an image slice size threshold t, define an upper surface patch image with the same size as the upper surface segmentation image, set the initial value of the pixel to 0, update the pixel value of the upper surface patch image from the tth column to the total number of columns minus the tth column according to the upper surface segmentation image, traverse all rows and 1 to 2*t columns of the upper surface segmentation image, update each pixel value of the tth column of each row of the upper surface patch image to the sum of the pixel values of the 1 to 2*t columns of the current row of the upper surface segmentation image, and update the upper surface patch image; traverse all rows and t+tth columns of the upper surface segmentation image. 1 column to the total number of columns minus t columns, update each pixel value of the upper surface patch image to the pixel value of the upper surface patch image in the previous column of the current row and current column plus the pixel value of the upper surface segmentation image in the current row and current column plus t columns, and then minus the pixel value of the upper surface segmentation image in the current row and current column minus t columns, to obtain the final upper surface patch image; traverse all pixels of the upper surface patch image, give a pixel value sum threshold, if the pixel value of the current pixel is greater than the given pixel value sum threshold, then update its corresponding pixel value on the upper surface segmentation image to 255, otherwise update it to 0, and update the upper surface segmentation image.
[0061] Record the row number of each pixel point where the pixel state changes, obtain the depth image of the target in S1, the target can be completely seen in the depth image, traverse each column of pixels from bottom to top, if the vertical and horizontal surfaces of the cubic object appear successively, then the pixel category will definitely change again, and set the pixel values of all pixels in the current column from the row number corresponding to the position where the pixel state changed for the last time to the first row to 0 (the point value of the recorded row number to the first row may be 0 or 255, and is uniformly set to 0); otherwise, it is very likely that the upper surface of the current cubic object does not appear completely in the picture, and such upper surface points are eliminated, and their corresponding pixel values on the upper surface segmentation image are set to 0.
[0062] In S4, edge line detection is performed on the edge point image, and all detected edge lines are clustered and merged to obtain a collinear edge line group set. The specific method is as follows: edge line detection is performed on the edge point image, all edge lines are obtained and sorted into an edge line set from long to short, and an empty collinear edge line group set is defined. The collinear edge line group set contains multiple groups of edge lines. The edge lines in each group are collinear, and the edge lines between two groups are not collinear. First, the longest edge line is added to one group of the collinear edge line group set. For each A edge line in the edge line set, it is determined whether it is collinear with the longest B edge line in each group of the collinear edge line group set. The judgment conditions are as follows: C1. Calculate the angle difference between edge line A and edge line B, and judge Is the angle difference less than or equal to the given angle difference threshold e? If the angle difference is greater than e, edge line A and edge line B are definitely not collinear, because two collinear edge lines should be approximately parallel and have a relatively small angle difference. C2. If the angle difference is less than or equal to e, calculate the distances T1 and T2 from the starting point of edge line B to the starting point and end point of edge line A, and T3 and T4 from the end point of edge line B to the starting point and end point of edge line A. Given the point distance threshold G, is the minimum distance among distances T1, T2, T3 and T4 less than or equal to the point distance threshold G? This part is to detect whether the minimum distance between the two endpoints of edge line A and edge line B is less than the given point distance threshold G. If the minimum distance is greater than the given point distance threshold, it means that this The two edge lines are far apart, min(min(T1, T2), min(T3, T4))>G, then edge lines A and B are definitely not collinear; C3. If min(min(T1, T2), min(T3, T4))≤G, calculate the length L of edge line A and the length H of edge line B, and determine whether the maximum distance among distances T1, T2, T3 and T4 is less than G+H+L, max(max(T1, T2), max(T3, T4))≥(G+H+L), edge lines A and B are not collinear; C4. If max(max(T1, T2), max(T3, T4))<(G+H+L), calculate the distance from the start and end points of edge line B to edge line A. From S1 and S2, the distances S3 and S4 from the start and end points of edge line A to edge line B, given a distance threshold J, if the maximum distance between T1 and T2 is less than L+G, determine whether S1 is less than or equal to J; if the maximum distance between T3 and T4 is less than L+G, determine whether S2 is less than or equal to J; if the maximum distance between T1 and T3 is less than L+G, determine whether S3 is less than or equal to J; if the maximum distance between T2 and T4 is less than L+G, determine whether S4 is less than or equal to J; if any maximum distance is greater than or equal to L+G, directly execute the next step; C5, if the conditions in C4 are met, edge line A and edge line B are collinear, add edge line A to the group of the collinear edge line group set, and update the edge line of the group;In the above sequential judgment process from C1 to C6, if any condition is not satisfied, directly process the longest edge line of the next group in the collinear edge line group set, until all groups in the collinear edge line group set are processed, and each group of edge lines is represented by its longest edge line. If edge line A is not collinear with the longest edge lines of all groups in the collinear edge line group set, directly add edge line A to the new group in the collinear edge line group set, and update the number of groups in the collinear edge line group set; after edge line A and edge line B are collinear, determine whether edge line A and edge line B meet the merging condition. If T1≤H, T2≤H, T3≤H and T4≤H, edge line A and edge line B do not meet the merging condition. Otherwise, merge edge line A and edge line B, update the starting point and end point of edge line B, and update the collinear edge line group set; process the next edge line in the edge line set in the same way as processing edge line A, continuously update the collinear edge line group set, sort the edge lines in the collinear edge line group set by length, and obtain the final collinear edge line group set. ;
[0063] The main reason for performing collinear clustering and merging of edge lines here is that in the process of edge line detection, the originally continuous edge lines are often disconnected due to noise interference. Originally there was only one edge line, but multiple nearly parallel edge lines with close distances but different lengths may be detected during detection, which will reduce the stability and accuracy of the algorithm. The main purpose of performing collinear clustering and merging is to divide multiple line segments that may belong to the same line into the same group and merge them, merge multiple disconnected collinear edge lines, obtain the longest edge line of each group, and use the longest edge line to represent all edge lines of the group. On the one hand, it reduces the amount of calculation, and on the other hand, it improves the stability of the algorithm.
[0064] Merge edge line A and edge line B, and update the starting and ending points of edge line B as follows: edge line B is the longest edge line in its group. If edge line A and edge line B are collinear and meet the merging conditions, update the starting and ending points of edge line B according to edge line A. If the x-coordinates of the two end points of the edge line are different, define the left end point of the edge line as the starting point and the right end point as the end point. If the x-coordinates of the two end points are the same, define the upper end point as the starting point and the lower end point as the end point. The judgment method for updating the end point of edge line B is as follows: C1, if T1 >H and T2 > H, on this basis, T3 > T4, then the end point of edge line B is updated to the starting point of edge line A. If the condition of T3 > T4 is not met, the end point of edge line B is updated to the end point of edge line A; C2, if the condition in C1 is not met, T1 >H and T2 > H, if T1 > H but T2 <= H, then the end point of edge line B is updated to the starting point of edge line A; C3, if the conditions in C1 and C2 are not met, if T2 >H but T1<= H, then the end point of edge line B is updated to the end point of edge line A. The judgment method for updating the starting point of edge line B is: C1, if T3>H and T4>H, on this basis, T1>T2, then the starting point of edge line B is updated to the starting point of edge line A. If the condition of T1>T2 is not met, the starting point of edge line B is updated to the end point of edge line A; C2, if the condition in C1 is not met, T3>H and T4>H, if T3>H but T4<=H, then the starting point of edge line B is updated to the starting point of edge line A; C3, if the conditions in C1 and C2 are not met, if T4>H but T3<=H, then the starting point of edge line B is updated to the end point of edge line A.
[0065] The specific method of extracting corner points according to the longest edge line of each group of collinear edge line groups in S5 is as follows: C1. Obtain the longest edge line of each group of collinear edge line groups, record it as the seed edge line set, sort all edge lines of the seed edge line set from long to short, traverse all edge lines of the seed edge line set, and start from the longest C edge line to calculate the angle a between it and any other D edge line, where a ranges from 0 to 180 degrees. Given a curvature threshold b, determine whether min (a, 180-a) is greater than b; C2. If min (a, 180-a)> b, calculate the intersection W of the C edge line and the D edge line (Wx, Wy), determine whether Wx or Wy does not exceed the image range; if Wx or Wy exceeds the image range, start processing the next edge line; C3, if Wx or Wy does not exceed the image range, determine whether the intersection W is a corner point. The condition for determining whether the intersection W is a corner point is: first calculate the distances d1 and d2 from the two endpoints of the C edge line to the intersection W, and define the minimum distance min_d1. If the intersection W is between the two endpoints of the C edge line, then min_d1=0, otherwise min_d1=min(d1, d2), If the minimum distance min_d1 is not 0 or min(d1, d2) is less than the given arm length threshold Z1, then calculate the distances d3 and d4 from the two endpoints of the edge line D to the intersection point W, and define the minimum distance min_d2. If the intersection point W is between the two endpoints of the edge line D, then min_d2=0, otherwise min_d2=min(d3, d4). If the minimum distance min_d2 is not 0 or min(d3, d4) is less than the given arm length threshold Z2, determine min(d1, d2) and min(d3, d4) Is it less than the given intersection radius threshold r? C4. If the condition of C3 is met, given the quadrilateral arm gap threshold v, determine whether max(min(d1, d2), min(d3, d4)) is less than or equal to v. If not, process the next edge line. If satisfied, the intersection W is the corner point of the quadrilateral. The conditions C1 to C4 are judged in turn. If any condition is not met, process the next edge line. Add the first corner point obtained to the corner point set. For the remaining corner points obtained, calculate their distance L to all corner points in the corner point set. i , i is a variable, judging whether it is similar to all corner points in the corner point set, if L i<r, and is not similar to the rest of the corner points, add the corner point to the corner point set; the intersection point W corresponds to two arms, the first arm is the line connecting it with the end point of the C edge line, the second arm is the line connecting it with the end point of the D edge line, the end point of the C edge line is the larger corresponding endpoint of d1 and d2 in the C edge line, and the end point of the D edge line is the larger corresponding endpoint of d3 and d4 in the D edge line; process any other two edge lines in the seed edge line group set in the same way to obtain all the corner points, expand the two arms of each corner point according to the seed edge line set, obtain the final corner point set, and realize corner point extraction.
[0066] The specific method of extending the two arms of each corner point according to the seed edge line set is:
[0067] Traverse all edge lines of the seed edge line set, and for each corner point, calculate the distance from it to the current seed edge line. If the distance is less than a given threshold, determine whether the seed edge line can extend the two arms of the current corner point. First, determine whether the seed edge line can extend arm 1 of the current corner point. The specific method is:
[0068] C1. Calculate the distance from the current corner point to the starting point and the end point of the seed edge line, recorded as dist1 and dist2, and determine whether the current corner point is between the two end points of the seed edge line;
[0069] C2. If the current corner point is between the two endpoints of the seed edge line, the line segment from the current corner point to the starting point of the seed edge line is recorded as M, the starting point of line segment M is the current corner point, the end point is the starting point of the seed edge line, the line segment where arm 1 of the current corner point is located is recorded as N, the starting point of line segment N is the current corner point, calculate the angle angle1 of line segment M and the angle angle2 of line segment N, and determine whether the absolute value of the difference between angle1 and angle2 is less than the given angle threshold;
[0070] C3. If the absolute value of the difference between angle1 and angle2 is less than the given angle threshold, calculate the distance rM from the end point of line M to the current corner point, calculate the length rN of line segment N, and determine whether the difference between rM and rN is less than the given length difference threshold;
[0071] C4, if the difference between rM and rN is less than the given length difference threshold, calculate the distance from the end point of line segment N to the current seed edge line. If the distance is less than the given point-to-line distance threshold and dist1 is greater than rN, the current seed edge line can extend the arm 1 of the current corner point. The end point of the extended arm 1 of the current corner point is the starting point of the seed edge line, and the length of the extended arm 1 is dist1.
[0072] C5. If any of the conditions from C2 to C4 is not met, it indicates that the current seed edge line cannot extend the arm 1 of the current corner point. Then the same method is used to determine whether the current seed edge line can extend the arm 2 of the current corner point. If it can be extended, the end point of the extended arm 2 is the starting point of the seed edge line, and the length of the extended arm 2 is dist1. If it cannot be extended, directly execute the next step;
[0073] C6, update the end point of line segment M in C2 to the end point of the current seed edge line, update line segment N to the line segment where arm 2 of the current corner point is located, the starting point of line segment N is the current corner point, dist1 is updated to dist2, and the same method is used to determine again whether the current seed edge line can extend arm 1 and arm 2 of the current corner point to achieve the extension of the two arms of the current corner point;
[0074] C7. At this time, if the condition in C1 is not met, determine whether dist1 is less than or equal to dist2;
[0075] C8, if dist1 is less than or equal to dist2, update the starting point of line segment M in C2 to the current seed edge
[0076] The starting point and end point of the edge line are updated to the end point of the current edge line, and the line segment N is the line segment where the arm 1 or arm 2 of the current corner point is located. The same method is used to determine whether the current seed edge line can extend the arm 1 and arm 2 of the current corner point to achieve the extension of the two arms of the current corner point;
[0077] C9. If dist1 is greater than dist2, update the starting point of the line segment M in C2 to the end point of the current seed edge line, and the end point is updated to the starting point of the current edge line. The line segment N is the line segment where arm 1 or arm 2 of the current corner point is located. Dist1 is updated to dist2. The same method is used to determine whether the current seed edge line can extend arm 1 and arm 2 of the current corner point to achieve the extension of the two arms of the current corner point.
[0078] The specific method of pairing the extracted corner points in S6 is as follows: C1. Pair all corner points in the corner point set in pairs. If the coordinates of the two paired corner points are different, continue to determine whether the two corner points share an arm. Mark the two arms of one corner point as arm11 and arm12, and the two arms of the other corner point as arm21 and arm22. First determine whether arm11 and arm21 are the same arm of the two corner points; C2. Determine whether arm11 and arm21 are the same arm as follows: if the distance between the two corner points is less than one-fourth of the minimum value of arm11 and arm21, then arm11 and arm21 are not are the same arm; if the distance between the two corner points is greater than one-fourth of the minimum value of arm11 and arm21, first calculate the absolute value x1 of the angle difference between arm11 and the line connecting the two corner points, and the absolute value x2 of the angle difference between arm21 and the line connecting the two corner points. The value range of x1 and x2 is [0, 180]. Given the angle similarity threshold f, if both x1 and x2 are less than the angle similarity threshold f, given the corner point distance threshold u and the length ratio threshold ratio, if the sum of the lengths of arm11 and arm21 is greater than or equal to ratio times the distance between the two corner points, and greater than or equal to the difference between the distance between the two corner points and u, then arm1 is determined to be If arm1 and arm21 are the same arm, the corner point pairing is successful; otherwise, the same method is used to determine whether the other two arms of the two corner points are the same arm. If neither is the case, the corner point pairing fails, otherwise the corner point pairing is successful; the same method is used to pair the remaining corner points in pairs to obtain all successfully paired corner point combinations. The same arm of each corner point combination is a shared arm, and the other two arms are open arms. The angles u1 and u2 between the two open arms and the shared arm in each corner point combination are calculated. The value range of the angles u1 and u2 is [0, 180]. The corner point combinations whose angles u1 or u2 are less than the curvature threshold b are eliminated; for the remaining corner point combinations, determine whether their two open arms are the same arm. Whether the arms are on the same side of the shared arm, calculate the center point of the shared arm of the corner point combination, take the center point as the starting point, calculate the unit normal vector of the shared arm, calculate the vector v1 where the line connecting the end point of the open arm of the first corner point and the center point is located, and the vector v2 where the line connecting the end point of the open arm of the second corner point and the center point is located. If the inner products of vectors v1 and v2 with the unit normal vector of the shared arm are one positive and one negative, the two open arms are not on the same side of the shared arm. The corner point combination whose two open arms are not on the same side of the shared arm is eliminated. Finally, a quadrilateral is determined based on the currently paired corner point combination; all corner point combinations are processed in the same way to obtain all quadrilaterals.
[0079] The specific method to determine whether two corner points are similar is: calculate the absolute values a11, a12, a21, and a22 of the difference between the angle of each open arm of one corner point and the angle of each open arm of the other corner point, and the absolute value of the angle difference ranges from [0,180]. Select min(a11, a12), min(a21, a22). For a given angle similarity threshold f, if max(min(a11, a12), min(a21, a22)) < f, the two corner points are similar.
[0080] The similarity of two corner points means that the opening arms corresponding to the two corner points are approximately parallel. For example: if the two corner points contain a horizontal opening arm and a vertical opening arm respectively, the horizontal opening arms of the two corner points are approximately parallel, and the angle difference is very small. The two vertical opening arms are approximately parallel, and the angle difference is also very small. Therefore, the minimum value of min(a11, a12) and min(a21, a22) should also be less than the given similarity angle threshold, otherwise the two corner points are not similar.
[0081] The specific method of determining the upper surface and the overall contour of the cubic object according to the obtained quadrilateral and the upper surface segmentation image in S7 is as follows: C1, firstly process the obtained quadrilateral, if the end point distance of the two arms of the matching quadrilateral is less than the given distance ratio threshold times, the quadrilateral does not meet the requirements, and process the next quadrilateral, obtain all the quadrilaterals that meet the requirements, and obtain the contour and the circumscribed rectangle of each quadrilateral, and calculate its convex hull according to the contour coordinate points;
[0082] C2. Perform contour detection based on the upper surface segmentation image, obtain all contours, calculate their convex hulls based on the contour coordinate points, obtain the smallest rotated rectangle containing each contour and the circumscribed rectangle containing the rotated rectangle, calculate the area of each contour, and determine whether the area of the contour is greater than a given area threshold;
[0083] C3. If the area of the contour is greater than the area threshold, match the current contour with a qualified quadrilateral, calculate the minimum distances h1 and h2 from the two corner points of the matching quadrilateral to the convex hull of the current contour, and determine whether the two corner points of the matching quadrilateral are outside the convex hull of the matching quadrilateral and whether h1 and h2 are less than or equal to the given distance threshold;
[0084] C4. If the two corner points of the matching quadrilateral are outside the convex hull of the matching quadrilateral and h1 and h2 are both less than or equal to the distance threshold, determine whether the center point of the rotated rectangle outside the current contour is inside the convex hull of the matching quadrilateral;
[0085] C5. If the center point of the rotated rectangle outside the current contour is located inside the convex hull of the matching quadrilateral, calculate the absolute value of the minimum distance from the rotated rectangle to the convex hull of the matching quadrilateral, and satisfy the conditions that the value is less than or equal to the length and width of the rotated rectangle, and the center point is located on the lower side of any arm of the matching quadrilateral, calculate the overlapping area of the circumscribed rectangle of the matching quadrilateral and the circumscribed rectangle of the current contour, if the overlapping area is greater than a given ratio threshold, the current contour and the matching quadrilateral are preliminarily matched successfully, and the matching quadrilateral is recorded as the preliminarily matched quadrilateral;
[0086] C6. Normalize the distance from the center point of the rotated rectangle of the current contour to the three arms of the preliminary matching quadrilateral. If the normalized value is within the theoretical range and the sum of h1 and h2 is less than or equal to a given threshold, the preliminary matching quadrilateral is recorded as a valid quadrilateral.
[0087] C7. Determine whether the two open arms of the valid quadrilateral are facing downward, and record the valid quadrilateral with two open arms facing downward as a seed quadrilateral; if the next seed quadrilateral is obtained, calculate whether the sum of the minimum distances from the two corner points of the seed quadrilateral to its contour is less than the sum of the minimum distances from the two corner points of the previous seed quadrilateral to its contour, if so, replace the previous seed quadrilateral with the current seed quadrilateral, otherwise remove the current seed quadrilateral, and continue processing until the final seed quadrilateral of the current contour is obtained;
[0088] C8, determine whether the final seed quadrilateral of the current contour has shared corner points with the valid quadrilateral;
[0089] C9. If there is a shared corner point, determine whether the line segments where the shared arms and open arms of the valid quadrilateral are located overlap with the line segments where the shared arms of the seed quadrilateral are located. If there is an overlap, determine whether the number of overlaps between the line segments where the shared arms and open arms of the valid quadrilateral are located and the line segments where the open arms of the seed quadrilateral are located is greater than or equal to 1. If so, obtain a cubic object contour with the three arms of the seed quadrilateral as the edges of the upper surface and the arms of the valid quadrilateral other than the overlapping arms as the edges of the lateral vertical surfaces, obtain all cubic object contours, and record them as a cubic object contour set;
[0090] C10, obtaining the coordinates of all points of the cubic object contour according to the endpoints of each edge of the cubic object contour, where the cubic object contour uses the seed quadrilateral as the upper surface;
[0091] C11. Traverse the cubic object contour set, calculate the y coordinate Py of the center point of each cubic object contour and the y coordinate ty of the center point of its upper surface, calculate the area P.area of each cubic object contour and the area t.area of its upper surface. If ty>Py or t.area>P.area, remove the cubic object contour and obtain the final cubic object contour set.
[0092] The specific method for obtaining the contour edge line of the upper surface in S8 is as follows: C1, traverse all pixel points on the upper surface of each detected cubic object, and obtain the world coordinate points of all pixel points on the upper surface in the three-dimensional space according to the transformation relationship between the image coordinate system and the world coordinate system; C2, traverse all world coordinate points of the current upper surface, give a point count threshold, and determine whether all the points are greater than the given point count threshold; C3, if the point count is greater than the point count threshold, calculate the mean point of all points and perform normalization processing, all coordinate points on the upper surface of each cubic object are located in the same plane in the three-dimensional space, and the mean point is located on the plane, calculate the normal vector of the plane in the three-dimensional space where the mean point is located, and obtain its initial fitting three-dimensional plane equation according to the normal vector and the mean point; C4, calculate the distance from all world coordinate points on the upper surface to the initial fitting three-dimensional plane, give a distance threshold, eliminate all points whose distance is greater than the given distance threshold, and obtain the rational point. C5, repeat the above steps to perform multiple fittings, calculate the distances from all points in the ideal point set to the fitted three-dimensional plane after each fitting, eliminate all points whose distances are greater than a given distance threshold, update the ideal point set, and record it as the upper ideal point set; C6, obtain the plane coordinate points on the two-dimensional plane corresponding to all world coordinate points of the upper ideal point set of the current upper surface, and perform contour detection based on them to obtain all contours, calculate the overlapping area between the circumscribed rectangle of each contour and the circumscribed rectangle of the current upper surface, and find the contour with the largest overlapping area and greater than a given threshold. This contour is the final contour of the upper surface, calculate the convex hull of the upper surface based on the contour, obtain its corresponding fitting polygon based on its convex hull, obtain all edge lines of the fitting polygon of the upper surface and sort them from long to short, and these edge lines are the contour edge lines of the upper surface.
[0093] The specific method for determining the upper surface quadrilateral according to the contour edge lines of the upper surface in S9 is as follows: given a parallel angle difference threshold j, determine whether the angle difference between any two contour edge lines is less than j. If this condition is met, the current two contour edge lines are in the same group. Group all contour edge lines. If the total number of groups after grouping is greater than or equal to 2, calculate the total length of the contour edge lines of each group and sort them from large to small. The two groups with the largest total length are recorded as Group 1 and Group 2. Group 1 contains a group of approximately parallel sides of the upper surface quadrilateral, and Group 2 contains another group of approximately parallel sides of the upper surface quadrilateral. If the total number of groups after grouping is less than 2, process the next upper surface; calculate the distance difference between any two contour edge lines in Group 1 and Group 2 respectively. Given a parallel distance difference threshold, sort the distance lines in Group 1 and Group 2 respectively. The contour edge lines whose difference is less than the parallel distance difference threshold are divided into the same group. If the total number of groups after division in group 1 is greater than or equal to 2, the total length of the edge lines of each group in group 1 is calculated and sorted from large to small, and the two groups with the largest total length are recorded as group 11 and group 12. If the total number of groups after division in group 2 is greater than or equal to 2, the total length of the edge lines of each group in group 2 is calculated and sorted from large to small, and the two groups with the largest total length are recorded as group 21 and group 22. If the total number of groups after division in group 1 or group 2 is less than 2, process the next upper surface; traverse all pixel points of the upper surface contour, calculate the distance from each pixel point to all contour edge lines of group 11, group 12, group 21 and group 22 respectively, and obtain the contour edge lines with the smallest distance in each group and less than the given threshold, which are the four sides of the upper surface.
[0094] The specific method for calculating the volume of the cubic object in S10 is as follows: for any side of the upper surface quadrilateral, record it as side 1, and record the side that is approximately parallel to side 1 as side 2, calculate the distances from all edge points of side 1 to the vertical plane where side 2 is located, and at the same time calculate the distances from all edge points of side 2 to the vertical plane where side 1 is located, and calculate the average value of all distances as the length of the cubic object corresponding to the current upper surface; record the other two approximately parallel sides of the upper surface quadrilateral as sides 3 and 4 respectively, calculate the distances from all edge points of side 3 to the vertical plane where side 4 is located, and at the same time calculate the distances from all edge points of side 4 to the vertical plane where side 3 is located, and calculate the average value of all distances as the width of the cubic object corresponding to the current upper surface; calculate the distances from all points of the upper surface quadrilateral to the fitted three-dimensional plane of the ground plane and calculate the average value, which is the height of the cubic object, and calculate the volume of the cubic object according to the length, width and height of the cubic object.
[0095] The range of approximate parallelism is the parallel angle difference threshold. If the angle difference between edge 1 and edge 2 is less than the given parallel angle difference threshold, then the two edges are approximately parallel; if edge 1 and edge 2 are completely parallel, the angles of the two edges should be equal, and approximate parallelism means that the angle difference between the two edges is less than the given parallel angle difference threshold. Since the two originally parallel edges of each face of the cubic object may not be completely parallel after projection, this method helps improve the stability of the algorithm. The method for obtaining the fitted three-dimensional plane of the ground plane is: define a ground plane image of the same size as the gradient image, set the initial value of the pixel to 0; traverse the binary image of the horizontal plane and the upper surface segmentation image, and update the pixel value of the pixel point on the ground plane image corresponding to the pixel point value of 255 in the binary image of the horizontal plane and the pixel point value of 0 in the upper surface segmentation image to 255, and obtain the ground plane image; calculate the world coordinate points of all pixel points with a value of 255 on the ground plane image in three-dimensional space, calculate the mean point of all pixel points and normalize them, all coordinate points and mean points of the ground plane image are on the same plane in three-dimensional space, and calculate their The normal vector of the plane in the three-dimensional space where the ground plane is located is obtained, and the fitting three-dimensional plane equation of the ground plane is obtained according to the mean point and the normal vector of the ground plane. After fitting, the distance from the pixel point with a pixel value of 255 on the ground plane image to the fitting three-dimensional plane equation is calculated, and the pixel points with a distance greater than a given threshold are eliminated to obtain the ideal point set belonging to the ground plane point. The three-dimensional plane equation of the ground plane is fitted again according to the ideal point set, and the distance from all pixel points of the ideal point set of the ground plane point to the fitting three-dimensional plane equation is calculated, and the pixel points with a distance greater than a given threshold are eliminated. The ideal point set of the ground plane point is updated, and the final fitting three-dimensional plane equation of the ground plane is obtained after repeated fitting and elimination of noise points.
[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A cubic object volume measurement method based on depth image, It is characterized in that include: S1. Obtain the depth image of the target, and obtain the gradient maps of the same size in the X and Y directions according to the depth image; S2. According to the gradient map, the pixels in it are divided into three categories: horizontal plane, left vertical plane and right vertical plane, and the edge point map of the depth image is obtained; S3. According to the classification results, the gradient map and the edge point map, the segmented images of the upper surface, left vertical plane and right vertical plane of the target in the depth image are obtained; S4. Edge line detection is performed on the edge point map, and all detected edge lines are clustered and merged to obtain a set of collinear edge line groups; S5. Corner points are extracted according to the longest edge line of each group of collinear edge line groups; S6. The extracted corner points are pairing; S7, determine the upper surface and overall contour of the cubic object according to the obtained quadrilateral and upper surface segmentation image; S8, fit the plane of the cubic object in the three-dimensional space according to the upper surface, remove the noise points on each surface by repeated fitting, obtain the ideal point set corresponding to the upper surface, perform contour detection according to the ideal point set, and obtain the contour edge line of the upper surface; S9, determine the upper surface quadrilateral according to the contour edge line of the upper surface; S10, calculate the length and width of the cubic object according to the upper surface quadrilateral, fit the three-dimensional plane where the ground plane is located, calculate the height of the cubic object according to the distance between the upper surface and the ground plane, and calculate the volume of the cubic object; The specific method of obtaining the segmented images of the upper surface, left vertical surface and right vertical surface of the target in the depth image according to the classification results, gradient map and edge point map in S3 is: According to the classification results, binary images of the horizontal plane, left vertical plane and right vertical plane with the same size as the gradient map are obtained, and the upper surface segmentation image with the same size as the gradient map is defined, and the initial value of the pixel point is set to 0; Traverse the pixels in the first column of the binary image of the horizontal plane from bottom to top, obtain the credibility value of each pixel according to the depth map, give a credibility threshold, if the credibility value of the pixel is less than the given credibility threshold, the state of the pixel is unknown, and process the next pixel; otherwise, if the corresponding pixel value is 255, the state of the pixel is horizontal, if the corresponding pixel is 0, the state of the pixel is vertical, if the number of pixels with a pixel value of 0 is greater than the given height threshold, then the pixels with a pixel value of 255 that appear later are mapped to the upper surface segmentation image and their pixel values are updated to 255, and the row number of the pixel where the pixel state changes each time is recorded. After traversing all rows of the first column, set the pixel values of all pixels in the first column from the recorded row number to the first row corresponding to the position on the upper surface segmentation image to 0, traverse all rows and columns of the parallel surface image in the same way, obtain the initial upper surface segmentation image, repair the obtained initial upper surface segmentation image, and update the upper surface segmentation image; The edge points in the edge point map are respectively matched one by one in the updated upper surface segmentation image, the binary map of the left vertical surface, and the binary map of the right vertical surface, the pixel values of the horizontal non-edge points in the updated upper surface segmentation image are updated to 0, the pixel values of the non-edge points in the binary map of the left vertical surface with a pixel value of 255 are updated to 0, and the pixel values of the non-edge points in the binary map of the right vertical surface with a pixel value of 255 are updated to 0, and the final upper surface segmentation image, the binary map of the left vertical surface, and the binary map of the right vertical surface are obtained, and the binary map of the left vertical surface and the binary map of the right vertical surface are recorded as the segmentation image of the left vertical surface and the segmentation image of the right vertical surface, respectively; The specific method of obtaining the edge point map of the depth image is: Calculate the average gradient Yp of all pixels in the vertical plane after classification, the average gradient Yq of all pixels in the horizontal plane, the average gradient Xp of all pixels in the left vertical plane, and the average gradient Xq of all pixels in the right vertical plane; set the positive threshold in the Y direction, the negative threshold in the Y direction, the positive threshold in the X direction, and the negative threshold in the X direction according to Yp, Yq, Xp, and Xq; define an image with the same size as the gradient map, and set the initial value of the pixel in the image to 0; traverse all the pixels in the gradient map in the X direction and the gradient map in the Y direction, and for For each pixel point, if the absolute value of the gradient value in the Y direction is greater than the absolute value of the gradient value in the X direction, and is less than the negative threshold in the Y direction or greater than or equal to the positive threshold in the Y direction, the pixel point is an edge point, and the pixel point is mapped to the defined image and its pixel value is updated to 255; if the absolute value of the gradient value in the Y direction is less than the absolute value of the gradient value in the X direction, and the absolute value of the gradient value in the X direction is less than or equal to the negative threshold in the X direction or greater than or equal to the positive threshold in the X direction, the pixel point is an edge point, and the pixel point is mapped to the defined image and its pixel value is updated to 255; an edge point map is obtained; The specific method of performing edge line detection on the edge point graph, clustering and merging all detected edge lines, and obtaining a set of collinear edge line groups is as follows: Perform edge line detection on the edge point graph, obtain all edge lines and sort them into edge line sets from long to short, define an empty collinear edge line group set, the collinear edge line group set contains multiple groups of edge lines, the edge lines in each group are collinear, and the edge lines between two groups are not collinear. First, add the longest edge line to one of the collinear edge line group sets, for each A edge line in the edge line set, determine whether it is collinear with the longest B edge line in each group of the collinear edge line group set, and the judgment conditions are as follows: C1. Calculate the angle difference between edge line A and edge line B, and determine whether the angle difference is less than or equal to a given angle difference threshold value e; C2. If the angle difference is less than or equal to the angle difference threshold e, calculate the distances T1 and T2 from the start point of edge line B to the start point and end point of edge line A, and T3 and T4 from the end point of edge line B to the start point and end point of edge line A. Given the point distance threshold G, determine whether the minimum distance among the distances T1, T2, T3 and T4 is less than or equal to the point distance threshold G. C3. If min(min(T1, T2), min(T3, T4))≤G, calculate the length L of edge line A and the length H of edge line B, and determine whether the maximum distance among distances T1, T2, T3 and T4 is less than G+H+L; C4. If max(max(T1, T2), max(T3, T4))<(G+H+L), calculate the distances S1 and S2 from the start and end points of edge line B to edge line A, and the distances S3 and S4 from the start and end points of edge line A to edge line B. Given a distance threshold J, if the maximum distance between T1 and T2 is less than L+G, determine whether S1 is less than or equal to J; if the maximum distance between T3 and T4 is less than L+G, determine whether S2 is less than or equal to J; if the maximum distance between T1 and T3 is less than L+G, determine whether S3 is less than or equal to J; if the maximum distance between T2 and T4 is less than L+G, determine whether S4 is less than or equal to J; if any maximum distance is greater than or equal to L+G, directly execute the next step; C5. If the condition in C4 is met, the edge line A and the edge line B are collinear, and the edge line A is added to the group of the collinear edge line group set, and the edge line of the group is updated; During the judgment process, if any condition is not met, directly process the longest edge line of the next group in the collinear edge line group set, until all groups in the collinear edge line group set are processed, and each group of edge lines is represented by its longest edge line. If edge line A is not collinear with the longest edge lines of all groups in the collinear edge line group set, directly add edge line A to the new group in the collinear edge line group set, and update the number of groups in the collinear edge line group set; after edge line A and edge line B are collinear, determine whether edge line A and edge line B meet the merging condition. If T1≤H, T2≤H, T3≤H and T4≤H, edge line A and edge line B do not meet the merging condition, otherwise, merge edge line A and edge line B, update the starting point and end point of edge line B, and update the collinear edge line group set; Process the next edge line in the edge line set in the same way as processing edge line A, continuously update the collinear edge line group set, sort the edge lines of each group in the collinear edge line group set by length, and obtain the final collinear edge line group set; The specific method of extracting corner points based on the longest edge line of each group of collinear edge line groups is: C1. Get the longest edge line of each group of collinear edge line groups, record it as the seed edge line set, sort all the edge lines of the seed edge line set from long to short, traverse all the edge lines of the seed edge line set, and start from the longest C edge line to calculate the angle a between it and any other D edge line, where a ranges from 0 to 180 degrees. Given a curvature threshold b, determine whether min (a, 180-a) is greater than b; C2. If min(a, 180-a)>b, calculate the intersection point W(Wx, Wy) of the edge line C and the edge line D, and determine whether Wx or Wy does not exceed the image range; C3. If Wx or Wy does not exceed the image range, determine whether the intersection W is a corner point. The condition for determining whether the intersection W is a corner point is as follows: first calculate the distances d1 and d2 from the two endpoints of the C edge line to the intersection W, and define the minimum distance min_d1. If the intersection W is between the two endpoints of the C edge line, then min_d1=0, otherwise min_d1=min(d1, d2). If the minimum distance min_d1 is not 0 or min(d1, d2) is less than the given arm length threshold Z1, then calculate the distances d3 and d4 from the two endpoints of the D edge line to the intersection W, and define the minimum distance min_d2. If the intersection W is between the two endpoints of the D edge line, then min_d2=0, otherwise min_d2=min(d3, d4). If the minimum distance min_d2 is not 0 or min(d3, d4) is less than the given arm length threshold Z2, determine whether min(d1, d2) and min(d3, d4) are less than the given intersection radius threshold r. C4. If the condition of C3 is met, given the threshold value v of the gap between the arms of the quadrilateral, determine whether max(min(d1, d2), min(d3, d4)) is less than or equal to v. If so, the intersection point W is the corner point of the quadrilateral. The conditions C1 to C4 are judged in sequence. If any condition is not met, the next edge line is processed; The first corner point obtained is added to the corner point set. For the remaining corner points obtained, the distance Li between them and all the corner points in the corner point set is calculated, where i is a variable. It is determined whether they are similar to all the corner points in the corner point set. If Li < r and they are not similar to the remaining corner points, the corner point is added to the corner point set. The intersection point W corresponds to two arms, the first arm is the line connecting it with the end point of the C edge line, and the second arm is the line connecting it with the end point of the D edge line. The end point of the C edge line is the larger corresponding end point of d1 and d2 in the C edge line, and the end point of the D edge line is the larger corresponding end point of d3 and d4 in the D edge line. The same method is used to process any two other edge lines in the seed edge line group set to obtain all corner points, and the two arms of each corner point are expanded according to the seed edge line set to obtain the final corner point set, thereby realizing corner point extraction. The specific method of pairing the extracted corner points is: C1. Pair all corner points in the corner point set. If the coordinates of the two paired corner points are different, continue to determine whether the two corner points share an arm. Mark the two arms of one corner point as arm11 and arm12, and the two arms of the other corner point as arm21 and arm22. First determine whether arm11 and arm21 are the same arm of the two corner points. C2. The method of judging whether arm11 and arm21 are the same arm is as follows: if the distance between the two corner points is greater than one-fourth of the minimum value of arm11 and arm21, first calculate the absolute value x1 of the angle difference between arm11 and the line connecting the two corner points, and the absolute value x2 of the angle difference between arm21 and the line connecting the two corner points. The value range of x1 and x2 is [0, 180]. Given an angle similarity threshold f, if both x1 and x2 are less than the angle similarity threshold f, given a corner point distance threshold u and a length ratio threshold ratio, if the sum of the lengths of arm11 and arm21 is greater than or equal to ratio times the distance between the two corner points, and greater than or equal to the difference between the distance between the two corner points and u, then arm11 and arm21 are judged to be the same arm and the corner point pairing is successful. Otherwise, the same method is used to determine whether the other two arms of the two corner points are the same arm. If neither is the case, the corner point pairing fails, otherwise the corner point pairing succeeds. In the same way, the remaining corner points are paired in pairs to obtain all successfully paired corner point combinations. The same arm of each corner point combination is the shared arm, and the other two arms are open arms. The angles u1 and u2 between the two open arms and the shared arm in each corner point combination are calculated. The value range of the angles u1 and u2 is [0, 180]. The corner point combinations whose angles u1 or u2 are less than the curvature threshold b are eliminated; for the remaining corner point combinations, determine whether their two open arms are on the same side of the shared arm, and eliminate the corner point combinations whose two open arms are not on the same side of the shared arm. Finally, determine a quadrilateral based on the currently paired corner point combination; process all corner point combinations in the same way to obtain all quadrilaterals.
2. The method for measuring the volume of a cubic object based on a depth image according to claim 1, It is characterized in that The specific method of obtaining the contour edge line of the upper surface in S8 is: C1. Traverse all the pixel points on the upper surface of each detected cubic object, and obtain the world coordinate points of all the pixel points on the upper surface in the three-dimensional space according to the transformation relationship between the image coordinate system and the world coordinate system; C2, traverse all world coordinate points on the current upper surface, give a point count threshold, and determine whether all points are greater than the given point count threshold; C3. If the number of points is greater than the point threshold, calculate the mean point of all points and perform normalization. All coordinate points on the upper surface of each cubic object are located in the same plane in the three-dimensional space, and the mean point is located on the plane. Calculate the normal vector of the plane in the three-dimensional space where the mean point is located, and obtain the initial fitting three-dimensional plane equation based on the normal vector and the mean point. C4, calculate the distances from all world coordinate points on the upper surface to the first fitted three-dimensional plane, give a distance threshold, remove all points whose distances are greater than the given distance threshold, obtain an ideal point set, perform fitting again based on the ideal point set, and obtain a second fitted three-dimensional plane equation; C5. Repeat the above steps to perform multiple repetitive fittings. After each fitting, calculate the distances from all points in the ideal point set to the fitted three-dimensional plane, remove all points whose distances are greater than a given distance threshold, and update the ideal point set, which is recorded as the upper ideal point set. C6. Obtain the plane coordinate points on the two-dimensional plane corresponding to all world coordinate points of the ideal point set on the current upper surface, and perform contour detection based on them, obtain all contours, calculate the overlapping area between the circumscribed rectangle of each contour and the circumscribed rectangle of the current upper surface, and find the contour with the largest overlapping area that is greater than a given threshold. This contour is the final contour of the upper surface, calculate the convex hull of the upper surface based on the contour, obtain the corresponding fitting polygon based on its convex hull, obtain all edge lines of the fitting polygon of the upper surface and sort them from long to short, and these edge lines are the contour edge lines of the upper surface.
3. The method for measuring the volume of a cubic object based on a depth image according to claim 2, It is characterized in that The specific method of determining the upper surface quadrilateral according to the contour edge line of the upper surface in S9 is: Given a parallel angle difference threshold j, determine whether the angle difference between any two contour edge lines is less than j. If this condition is met, the current two contour edge lines are in the same group. Group all contour edge lines. If the total number of groups after grouping is greater than or equal to 2, calculate the total length of the contour edge lines of each group and sort them from large to small. The two groups with the largest total length are recorded as group 1 and group 2. Group 1 contains a group of approximately parallel sides of the upper surface quadrilateral, and group 2 contains another group of approximately parallel sides of the upper surface quadrilateral. If the total number of groups after grouping is less than 2, process the next upper surface. Calculate the distance difference between any two contour edge lines in group 1 and group 2 respectively, give a parallel distance difference threshold, and divide the contour edge lines in group 1 and group 2 whose distance difference is less than the parallel distance difference threshold into the same group respectively. If the total number of groups after division in group 1 is greater than or equal to 2, calculate the total length of the edge lines of each group in group 1 and sort them from large to small, and record the two groups with the largest total length as group 11 and group 12. If the total number of groups after division in group 2 is greater than or equal to 2, calculate the total length of the edge lines of each group in group 2 and sort them from large to small, and record the two groups with the largest total length as group 21 and group 22. If the total number of groups after division in group 1 or group 2 is less than 2, process the next upper surface. Traverse all pixel points of the upper surface contour, calculate the distance from each pixel point to all contour edge lines of group 11, group 12, group 21 and group 22 respectively, and obtain the contour edge line with the minimum distance in each group and less than a given threshold, which is the four sides of the upper surface.
4. The method for measuring the volume of a cubic object based on a depth image according to claim 2, It is characterized in that The specific method for calculating the volume of a cubic object in S10 is: For any side of the upper surface quadrilateral, record it as side 1, and record the side that is approximately parallel to side 1 as side 2. Calculate the distance from all edge points of side 1 to the vertical plane where side 2 is located. At the same time, calculate the distance from all edge points of side 2 to the vertical plane where side 1 is located. Calculate the average value of all distances as the length of the cubic object corresponding to the current upper surface. The other two approximately parallel sides of the upper surface quadrilateral are respectively recorded as side 3 and side 4, and the distances from all edge points of side 3 to the vertical plane where side 4 is located are calculated. At the same time, the distances from all edge points of side 4 to the vertical plane where side 3 is located are calculated, and the average value of all distances is calculated as the width of the cubic object corresponding to the current upper surface; Calculate the distances from all points of the upper surface quadrilateral to the fitted three-dimensional plane of the ground plane and find the average. The average is the height of the cubic object. Calculate the volume of the cubic object based on the length, width and height of the cubic object.
5. The method for measuring the volume of a cubic object based on a depth image according to claim 4, It is characterized in that The method of obtaining the fitted three-dimensional plane of the ground plane is as follows: define a ground plane image of the same size as the gradient image, set the initial value of the pixel to 0; traverse the binary image of the horizontal plane and the upper surface segmentation image, update the pixel value of the pixel point on the ground plane image corresponding to the pixel point value of 255 in the binary image of the horizontal plane and the pixel point value of 0 in the upper surface segmentation image to 255, and obtain the ground plane image; calculate the world coordinate points of all pixel points with a pixel value of 255 on the ground plane image in the three-dimensional space, calculate the mean point of all pixel points and normalize them, all coordinate points and the mean point of the ground plane image are on the same plane in the three-dimensional space, and calculate their The normal vector of the plane in the three-dimensional space where the ground plane is located is obtained, and the fitting three-dimensional plane equation of the ground plane is obtained according to the mean point and the normal vector of the ground plane. After fitting, the distance from the pixel point with a pixel value of 255 on the ground plane image to the fitting three-dimensional plane equation is calculated, and the pixel points with a distance greater than a given threshold are eliminated to obtain the ideal point set belonging to the ground plane point. The three-dimensional plane equation of the ground plane is fitted again according to the ideal point set, and the distance from all pixel points of the ideal point set of the ground plane point to the fitting three-dimensional plane equation is calculated, and the pixel points with a distance greater than a given threshold are eliminated. The ideal point set of the ground plane point is updated, and the final fitting three-dimensional plane equation of the ground plane is obtained after repeated fitting and elimination of noise points.
