Method and System for Obtaining Loading Volume Based on 3D Reconstruction of Material Piles before and after Loading

Abstract

The method of the present invention includes: collecting binocular images of the surface of the material pile before loading; detecting, describing, matching, and purifying feature points of the binocular images, calculating sparse matching feature points and disparities, and obtaining triangular mapping parameters; constructing a sparse three-dimensional point cloud based on the sparse feature points and disparities; constructing a dense three-dimensional point cloud model of the surface of the material pile before loading based on the sparse three-dimensional point cloud and triangular mapping parameters; collecting binocular images of the surface of the material pile after the construction machinery has completed loading, and constructing a dense three-dimensional point cloud model of the surface of the material pile after loading; performing point cloud downsampling and point cloud registration on the dense three-dimensional point cloud models of the surfaces of the material piles before and after loading; performing fine segmentation on the registered point cloud model according to the boundary of the loading area to obtain the point cloud model of the actual loading area, and obtaining the loading volume. The present invention can quickly and accurately estimate the volume of the material loaded by the construction machinery, which is of great significance for real-time evaluation of the project progress and the operation efficiency of the operators.

Description

Technical Field

[0001] The present invention relates to the technical fields of binocular stereo vision three-dimensional reconstruction and earthmoving machinery loading volume estimation, and particularly relates to a method and system for obtaining the loading volume of earthmoving machinery based on three-dimensional reconstruction of the material pile before and after loading. Background Technique

[0002] As a mechanized operation process, earthmoving operations of construction machinery require productivity planning, project progress assessment, and project labor fund disbursement. The key lies in estimating the quantity of earthmoving operation materials through different technologies and methods, and using this as an important parameter for evaluating the performance and operation efficiency of construction machinery. The measurement of earthmoving operation volume can be carried out through integral measurement of the material pile or single-shovel loading measurement. Total station robot scanners and photogrammetry technologies based on UAV aerial photography, as an effective three-dimensional modeling method, are widely used in the integral measurement of earthwork volume. However, the integral measurement method is suitable for planning the overall productivity of earthwork projects, but it is difficult to evaluate the real-time operation efficiency during the operation process. Therefore, single-shovel loading measurement is required. For single-shovel loading measurement, the existing technologies mainly focus on constructing a three-dimensional point cloud model of the bucket and the materials inside the bucket, calculating the volume of the materials inside the bucket, and evaluating the full-bucket rate and loading operation efficiency based on this. However, the buckets of different earthmoving machines have different specifications and sizes. This method of estimating the loading volume based on the three-dimensional point cloud model of the bucket has the disadvantages of strong dependence on the bucket model and poor generality. Therefore, it is of great significance to study a method for measuring the loading volume of earthmoving machinery based on three-dimensional reconstruction of the material pile to evaluate the continuous project progress and the operation efficiency of operators.

[0003] For the method of obtaining the volume information of an object through non-contact measurement, a three-dimensional laser scanning system and a vision sensor are usually used for three-dimensional reconstruction of the object surface. Since the amount of point cloud data generated when scanning an object using a laser scanning system is much larger than that of a vision sensor, post-processing the three-dimensional point cloud and performing three-dimensional reconstruction will be a more time-consuming task, making it difficult to meet the requirement of real-time update of object volume information perception. In addition, the three-dimensional laser scanning system is sensitive to shape and size, unable to obtain color information, and expensive, which to a certain extent limits its application in earthwork projects. Binocular stereo vision uses two cameras to simulate the human eye to photograph the object to be measured from different positions, finds the corresponding matching points of the same spatial point in the left and right camera images according to the image features, and calculates and solves the spatial coordinate points using the parallax principle and spatial correspondence relationship. This method can quickly obtain dense depth information, and has low cost and light weight, and is suitable for three-dimensional reconstruction of the material pile of earthmoving operations.

[0004] Chinese Invention Patent 202010538172.X proposes a method for calculating the earthwork volume of a foundation pit based on digital information technology. This method realizes accurate and rapid calculation of the earthwork volume of the foundation pit based on the oblique photogrammetry technology of UAV aerial photography. Chinese Invention Patent 202010134566.9 proposes a method and system for automatically extracting and volume measuring a material pile based on 3D point cloud. This method uses a laser scanner to obtain the 3D point cloud data of the material pile scene and calculates the volume of the entire material pile. Chinese Invention Patent 201910972066.X proposes a method for measuring the volume of an irregular stockpile based on a binocular camera. This method uses the semi-global stereo matching algorithm (SGBM) to obtain the disparity map of the overall stockpile image. SGBM determines the cost value between corresponding matching points by comparing pixel intensities. The description performance of this pixel-block-based descriptor is relatively weak, affecting the accuracy of volume estimation and consuming a lot of computing time. The above methods are applicable to the integral measurement of earthwork volume, but it is difficult to be applicable to the estimation of the single-shovel loading volume of construction machinery and the real-time evaluation of operation efficiency. Chinese Invention Patent 201911240082.6 proposes a method and a measuring device for real-time measuring the earthwork volume of an excavator bucket based on structured light, and Chinese Invention Patent 202111578028.X proposes a method for measuring the volume of bulk materials based on image features and 3D point cloud technology. The above two methods both belong to the volume estimation methods based on 3D modeling of the materials in the bucket, and it is difficult to be applicable to the volume measurement of materials in buckets with different specifications and sizes. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method and a system for obtaining the loading volume based on 3D reconstruction of the material pile before and after loading, which can quickly and accurately estimate the volume of the materials loaded by construction machinery and is of great significance for real-time evaluation of the project progress and the operation efficiency of operators.

[0006] On the one hand, a method for obtaining the loading volume based on 3D reconstruction of the material pile before and after loading includes:

[0007] Step (1), collecting binocular images of the surface of the material pile before loading;

[0008] Step (2): For the binocular images of the surface of the material pile before loading, use the SuperPoint algorithm for feature point detection and feature point description, and use the SuperGlue algorithm and the random sample consensus algorithm for feature point matching and purification. Calculate the sparse matching feature points and disparities, and obtain the corresponding triangular mapping parameters through Delaunay triangulation. Based on the sparse matching feature points and disparities obtained by matching, construct a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle. According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, construct a maximum a posteriori probability model to estimate the optimal disparity values of the remaining pixel points, and construct a dense three-dimensional point cloud model of the surface of the material pile before loading.

[0009] Step (3): Collect binocular images of the surface of the material pile after the construction machinery has completed loading.

[0010] Step (4): For the binocular images of the surface of the material pile after loading, use the SuperPoint algorithm for feature point detection and feature point description, and use the SuperGlue algorithm and the random sample consensus algorithm for feature point matching and purification. Calculate the sparse matching feature points and disparities, and obtain the corresponding triangular mapping parameters through Delaunay triangulation. Based on the sparse matching feature points and disparities obtained by matching, construct a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle. According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, construct a maximum a posteriori probability model to estimate the optimal disparity values of the remaining pixel points, and construct a dense three-dimensional point cloud model of the surface of the material pile after loading.

[0011] Step (5): Use the voxel grid method to perform point cloud downsampling on the dense three-dimensional point cloud model of the surface of the material pile before loading and the dense three-dimensional point cloud model of the surface of the material pile after loading respectively. Coarsely segment the downsampled point clouds in the area expanded a certain distance outward from the boundary of the loading area, and use the ICP algorithm to perform point cloud registration on the coarsely segmented point cloud model before loading and the point cloud model after loading. Refinely segment the registered point cloud model according to the boundary of the loading area to obtain the actual loading area point cloud model, and use the Alpha shape algorithm of Delaunay triangulation to estimate the loading volume.

[0012] Preferably, before collecting the binocular images of the surface of the material pile before loading, it also includes:

[0013] Calibrate and correct the images of the binocular stereo camera installed on the construction machinery.

[0014] Preferably, the calibration of the binocular stereo camera specifically includes:

[0015] The internal and external parameters of the camera are obtained by the Zhang Zhengyou calibration method. The internal parameters of the camera include the projection position coordinates (u0, v0) of the optical axis of the camera lens in the pixel coordinate system and the camera focal length f. The external parameters of the camera include the rotation matrix R and the translation matrix T. The conversion between the image pixel coordinate system and the world coordinate system is realized through the internal and external parameters of the camera.

[0016] Preferably, image correction is performed on the binocular stereo camera, specifically including:

[0017] So that a point in the left image can find a corresponding point in the right image along the same horizontal epipolar line.

[0018] Preferably, the SuperPoint algorithm is used for feature point detection and feature point description, and the SuperGlue algorithm and the random sample consensus algorithm are used for feature point matching and purification. The sparse matching feature points and disparities are calculated, and the corresponding triangular mapping parameters are obtained through Delaunay triangulation, specifically including:

[0019] Step (2.1), using the pre-annotated set shape data set as supervised data, pre-training the basic detection network with VGG16 as the basic detector, extracting the corner points of the left image and the right image respectively, and performing key point annotation; constructing the key point loss function and the descriptor loss function, and jointly training to obtain the SuperPoint detection network;

[0020] Step (2.2), obtaining the descriptors and coordinate positions of the corresponding feature points of the left image and the right image of the material pile according to the SuperPoint detection network respectively, and jointly obtaining the feature description vectors D (l) and D (r) ; inputting the two groups of vectors into the attention map neural network and the optimal matching layer of SuperGlue, iteratively solving the assignment optimization problem through the Sinkhorn algorithm to obtain the optimal assignment matrix; then using the feature point pairs corresponding to the horizontal and vertical coordinates of the maximum value in each column of the assignment matrix as the matching point pairs, using the random sample consensus algorithm to find the affine transformation matrix, and performing the corresponding affine transformation on the right image of the material pile to complete the matching and purification of the left image and the right image, and obtaining the sparse matching points;

[0021] Step (2.3), obtaining the corresponding triangular mapping parameters through Delaunay triangulation according to the sparse matching points The calculation is as follows:

[0022]

[0023] Where n represents the pixels included in the left image The serial number of the triangle; Solving the linear equation for the three vertices of each triangle can obtain the plane parameters of the triangle (a i , b i , c i ); u n represents the abscissa of the image pixel; v n represents the ordinate of the image pixel.

[0024] Preferably, according to the sparse feature points and disparities obtained by matching, a sparse three-dimensional point cloud is constructed based on the binocular camera stereo imaging principle; According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, a maximum a posteriori probability model is constructed to estimate the optimal disparity value of the remaining pixel points, and a dense three-dimensional point cloud model of the surface of the material pile before shoveling or a dense three-dimensional point cloud model of the surface of the material pile after shoveling is constructed. Specifically, it includes:

[0025] Step (3.1), the sparse three-dimensional point cloud constructed based on the binocular camera stereo imaging principle is calculated by the following formula:

[0026]

[0027] Among them, (X c , Y c , Z c ) is a point in the three-dimensional scene, and (u, v) is a pixel point on the two-dimensional image; f x represents the transformation of the camera focal length f to the pixel metric in the x direction; f y represents the transformation of the camera focal length f to the pixel metric in the y direction; B represents the baseline length of the binocular camera; u (l) represents the abscissa of the left image pixel point; u (r) represents the abscissa of the right image pixel point; u0 represents the abscissa of the projection position of the camera lens optical axis in the pixel coordinate system; v0 represents the ordinate of the projection position of the camera lens optical axis in the pixel coordinate system;

[0028] Step (3.2), in the method of constructing a maximum a posteriori probability model to estimate the optimal disparity value of the remaining pixel points, the constructed probability estimation model is shown in Equation (3):

[0029]

[0030] Among them, represents the estimated probability value; argmax represents finding the parameter set function;

[0031] (|) represents finding the conditional probability; S = (s1, s2,..., s M Z ) represents the sparse three-dimensional point cloud constructed previously, and each point cloud has three-dimensional coordinates (X c , Y c , Zc ), and each point is represented by s m =(u m ,v m ,d m ), d m For point (u m ,v m ) corresponding to the disparity; u m Indicates the horizontal coordinate of each point in the image pixel; v m Indicates the vertical coordinate of each point in the image pixel; Represents All pixels in the right image with the same horizontal epipolar line; a point in the left image Parallax d n Considered as a random variable to be solved, the posterior probability is expressed as the product of the prior probability and the likelihood probability, as shown in formula (4):

[0032]

[0033] Assume that the prior probability is proportional to the Gaussian distribution, as shown in formula (5):

[0034]

[0035] in, Represents the pixel points consisting of a sparse three-dimensional point cloud S triangular mapping; ∝ means proportional to; exp means the exponential function with the natural constant e as the base; σ means the standard deviation; μ means the mean; assuming that the likelihood probability can be expressed as a Laplace distribution, as shown in formula (6):

[0036]

[0037] in, and Respectively represent the feature description vectors of the nth pixel point in the left image and the nth pixel point in the right image; ‖.‖ represents the norm function; since the left and right images obtained by the binocular camera have been rectified in advance, the corresponding points of the left and right images must appear on the same horizontal epipolar line, so this restriction is guaranteed by the if condition in formula (6); the likelihood probability model is further derived from formula (6) as shown in formula (7):

[0038]

[0039] In step (3.3), after obtaining the disparity values of all pixels in the image, a dense three-dimensional point cloud model of the material pile surface before shoveling or a dense three-dimensional point cloud model of the material pile surface after shoveling is calculated according to formula (2).

[0040] Preferably, step (5) specifically includes:

[0041] Step (5.1), first, the voxelized grid method is used to downsample the three-dimensional point clouds on the surfaces of the material piles before and after shoveling simultaneously; after the downsampling is completed, then a certain area is extended outward from the boundary of the shoveling area, and the three-dimensional point clouds on the surfaces of the material piles before and after shoveling are roughly segmented respectively;

[0042] Step (5.2), taking the three-dimensional point cloud on the surface of the material pile before shoveling after rough segmentation as the reference model, and the three-dimensional point cloud on the surface of the material pile after shoveling after rough segmentation as the registration model, and using the ICP algorithm for registration to make the poses of the three-dimensional point clouds on the surfaces of the material piles before and after shoveling consistent;

[0043] Step (5.3), simultaneously performing fine segmentation on the registered three-dimensional point clouds on the surfaces of the material piles before and after shoveling according to the boundary limit of the shoveling area to obtain the point cloud model of the actual shoveling area;

[0044] Step (5.4), according to the obtained point cloud model of the actual shoveling area, performing Delaunay triangulation on the point cloud model, parameterizing the point cloud to fit the point cloud contour envelope, and using the Alpha shape algorithm to estimate the shoveling volume.

[0045] Preferably, the surface of the material pile before shoveling refers to the surface of the material pile before the initial shoveling or single shoveling of the material pile, including the surface contour information of the material pile; the surface of the material pile after shoveling refers to the surface of the material pile after a single shoveling of the material pile, including the surface contour information of the material pile.

[0046] On the other hand, a shoveling volume acquisition system based on three-dimensional reconstruction of the material piles before and after shoveling includes:

[0047] A material pile image acquisition module for respectively performing binocular image acquisition on the surfaces of the material piles before and after shoveling;

[0048] A triangular mapping parameter acquisition module for respectively performing feature point detection and feature point description on the binocular images of the surfaces of the material piles before and after shoveling by using the SuperPoint algorithm, and performing feature point matching and purification by using the SuperGlue algorithm and the random sample consensus algorithm, calculating the sparse matching feature points and the disparity, and obtaining the corresponding triangular mapping parameters through Delaunay triangulation;

[0049] The dense three-dimensional point cloud model construction module is used to construct a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle according to the matched sparse feature points and disparities; according to the obtained sparse three-dimensional point cloud and triangular mapping parameters, construct a maximum a posteriori probability model to estimate the optimal disparity values of the remaining pixel points, and respectively construct a dense three-dimensional point cloud model of the surface of the material pile before shoveling and a dense three-dimensional point cloud model of the surface of the material pile after shoveling;

[0050] The shoveling volume acquisition module is used to perform point cloud downsampling on the dense three-dimensional point cloud model of the surface of the material pile before shoveling and the dense three-dimensional point cloud model of the surface of the material pile after shoveling respectively by using the voxel grid method; perform rough segmentation on the downsampled point cloud at a certain distance area extended outward from the boundary of the shoveling area, and perform point cloud registration on the pre-shoveling point cloud model and the post-shoveling point cloud model after rough segmentation by using the ICP algorithm; perform fine segmentation on the registered point cloud model according to the boundary of the shoveling area to obtain the actual shoveling area point cloud model, and estimate the shoveling volume by using the Alpha shape algorithm of Delaunay triangulation.

[0051] The present invention has the following innovative points and significant advantages compared with the existing technologies:

[0052] (1) By adopting the image feature point detection algorithm SuperPoint and the feature point description algorithm SuperGlue based on the deep neural network, the present invention accurately and quickly extracts rich feature information of the material pile;

[0053] (2) The present invention proposes to construct a probability model based on Bayesian estimation to obtain a dense disparity map, using the matched sparse point cloud and disparity as prior information, reducing the search range of the disparity of the remaining pixel points, improving the calculation efficiency while ensuring sufficient matching accuracy;

[0054] (3) The present invention proposes a volume estimation method based on the three-dimensional point cloud of the surface of the material pile before and after shoveling, including: point cloud downsampling, point cloud rough segmentation, point cloud registration, point cloud fine segmentation and volume calculation, etc., reducing the calculation cost of the point cloud model while having high calculation accuracy. Description of the Drawings

[0055] Figure 1 It is a flow chart of the shoveling volume acquisition method based on the three-dimensional reconstruction of the material pile before and after shoveling of the present invention;

[0056] Figure 2 It is a flow chart of the reconstruction of the dense three-dimensional point cloud on the surface of the material pile before and after shoveling of the present invention;

[0057] Figure 3 It is a schematic diagram of point cloud registration and segmentation considering the shoveling boundary limitation of the present invention;

[0058] Figure 4This is the structural block diagram of the system for obtaining the loading volume based on the three-dimensional reconstruction of the material pile before and after loading in the present invention. Detailed implementation manners

[0059] The following further explains this aspect in conjunction with the drawings and embodiments.

[0060] Refer to Figure 1 As shown, a method for obtaining the loading volume based on the three-dimensional reconstruction of the material pile before and after loading in the present invention includes the following steps:

[0061] Step (1): Calibrate and correct the images of the binocular stereo camera, and install the binocular camera at a high position on the construction machinery to collect binocular images of the surface of the material pile before loading.

[0062] In this embodiment, calibrating the binocular stereo camera means obtaining the internal parameters and external parameters of the camera through the Zhang-Zhengyou calibration method. The internal parameters of the camera are related to the optical characteristics of the camera, such as the projection position coordinates (u0, v0) of the optical axis of the camera lens in the pixel coordinate system and the focal length f of the camera. The external parameters of the camera include the rotation matrix R and the translation matrix T. Through the internal and external parameters, the conversion between the image pixel coordinate system and the world coordinate system can be realized.

[0063] Correcting the images of the binocular stereo camera can ensure that a point in the left image can find a corresponding point in the right image along the same horizontal epipolar line.

[0064] Specifically, install the binocular camera at a high position on the construction machinery to ensure that the binocular camera can capture the complete binocular images of the surface of the material pile during the entire loading operation without being blocked by the task object.

[0065] In this embodiment, the surface of the material pile before loading refers to the surface of the material pile before the initial loading or single loading of the material pile, which contains the surface contour information of the material pile.

[0066] The material pile refers to the common operation object of construction machinery, such as earth-rock material piles of fine sand, cinder, native soil, ore, etc.

[0067] Step (2): Detect and describe the feature points of the binocular images of the surface of the material pile before loading using the SuperPoint algorithm, then use the SuperGlue algorithm and the Random Sample Consensus (RANSAC) algorithm for feature point matching and purification, calculate the sparse matching feature points and disparities, and obtain the corresponding triangular mapping parameters through Delaunay triangulation.

[0068] Specifically, the pre-annotated set shape dataset is used as supervised data, and the VGG16-like network is used as the basic detector to pre-train the basic detection network. The corner points of the left and right images (i.e., the left image and the right image) are extracted respectively for key point annotation. The key point loss function and the descriptor loss function are constructed, and the SuperPoint detection network is obtained through joint training.

[0069] Specifically, according to the SuperPoint detection network, the descriptors and coordinate positions of the corresponding key points of the left and right images of the material pile are obtained respectively, and the left and right image feature description vectors D (l) and D (r) are obtained. The two groups of vectors are input into the attention graph neural network and the optimal matching layer of SuperGlue, and the assignment optimization problem is iteratively solved through the Sinkhorn algorithm to obtain the optimal assignment matrix. Then, the feature point pairs corresponding to the horizontal and vertical coordinates of the maximum value in each column of the assignment matrix are used as the matching point pairs, and the random sample consensus algorithm (RANSAC) is used to obtain the affine transformation matrix, and the corresponding affine transformation is performed on the right image of the material pile to complete the matching and purification of the left and right images and obtain sparse matching points.

[0070] Specifically, according to the sparse matching points, the corresponding triangular mapping parameters are obtained through Delaunay triangulation The calculation is as follows:

[0071]

[0072] where n represents the triangle number containing the pixels of the left image . Solving the linear equation for the three vertices of each triangle can obtain the triangle plane parameters (a i , b i , c i ).

[0073] Step (3), as shown in Figure 2 , according to the sparse feature points and disparities obtained by matching, a sparse three-dimensional point cloud is constructed based on the binocular camera stereo imaging principle. According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, the optimal disparity value of the remaining pixel points is estimated by constructing a maximum a posteriori probability model, and a dense three-dimensional point cloud model of the surface of the material pile before shoveling is constructed. The specific steps are as follows:

[0074] Step (3.1), a sparse three-dimensional point cloud is constructed based on the binocular camera stereo imaging principle, and is calculated by the following formula:

[0075]

[0076] where (X c , Y c , Z c) is a point in the three-dimensional scene, (u,v) is a pixel point on the two-dimensional image, and f x =f / d x , f y =f / d y , respectively, represents the conversion of the camera's focal length f into pixel measurements in the x and y directions. B represents the baseline length of the binocular camera. u (l) 、u (r) Represent the horizontal coordinates of the pixels in the left and right images respectively.

[0077] In step (3.2), the optimal disparity value of the remaining pixels is estimated by constructing a maximum a posteriori probability model. The constructed probability estimation model is shown in formula (3).

[0078]

[0079] Where S=(s1,s2,……,s M ) represents the sparse three-dimensional point cloud constructed previously, and each point is represented by s m =(u m ,v m ,d m ), d m For point (u m ,v m ) corresponds to the parallax. Represents All pixels in the right image have the same horizontal epipolar line. Parallax d n Assuming it is a random variable to be solved, the posterior probability can be expressed as the product of the prior probability and the likelihood probability, as shown in formula (4).

[0080]

[0081] Assume that the prior probability is proportional to the Gaussian distribution, as shown in formula (5).

[0082]

[0083] in, Represents the pixel points consisting of a sparse three-dimensional point cloud S Assume that the likelihood probability can be expressed as a Laplace distribution, as shown in formula (6).

[0084]

[0085] in, and Feature description vectors representing the nth pixel of the left image and the nth pixel of the right image respectively. Since the left and right images obtained by the binocular camera have been pre-calibrated, corresponding points in the left and right images will necessarily appear on the same horizontal epipolar line. Therefore, the if condition in Equation (6) ensures this constraint. Therefore, the likelihood probability model is further derived from Equation (6) as shown in Equation (7).

[0086]

[0087] Step (3.3), after obtaining the disparity values of all pixels in the image, a dense three-dimensional point cloud model of the surface of the material pile before shoveling is calculated according to Equation (2).

[0088] Step (4), when the construction machinery has completed shoveling, binocular images of the surface of the material pile after shoveling are collected. Referring to Step (2) and Step (3), a dense three-dimensional point cloud model of the surface of the material pile after shoveling is constructed.

[0089] Specifically, the surface of the material pile after shoveling refers to the surface of the material pile after a single shoveling operation on the material pile, which contains the surface contour information of the material pile.

[0090] Step (5), for the dense three-dimensional point cloud models of the surfaces of the material pile before and after shoveling, the voxel grid method is used to downsample the point clouds respectively. Coarse segmentation of the downsampled point clouds is performed in a region extended a certain distance outward from the boundary of the shoveling area, and the ICP algorithm is used to register the point cloud models of the material pile before and after shoveling after coarse segmentation. Fine segmentation of the registered point cloud model is performed according to the boundary of the shoveling area to obtain the point cloud model of the actual shoveling area, and the Alpha shape algorithm of Delaunay triangulation is used to estimate the shoveling volume, and its principle is as Figure 3 shown.

[0091] Specifically, the steps are as follows:

[0092] Step (5.1), first, the voxel grid method is used to downsample the three-dimensional point clouds of the surfaces of the material pile before and after shoveling simultaneously. After the downsampling is completed, coarse segmentation of the three-dimensional point clouds of the surfaces of the material pile before and after shoveling is performed simultaneously in a region extended a certain distance outward from the boundary of the shoveling area. Specifically, the certain distance region needs to ensure that the point cloud model after coarse segmentation can contain all shoveling areas. For example, the length of the certain distance can be set to 0.1 - 0.5 times the minimum distance from the boundary of the shoveling area to the outer boundary.

[0093] Step (5.2), taking the three-dimensional point cloud of the surface of the material pile before shoveling after coarse segmentation as the reference model and the three-dimensional point cloud of the surface of the material pile after shoveling after coarse segmentation as the registration model, the ICP algorithm is used for registration to make the poses of the three-dimensional point clouds of the surfaces of the material pile before and after shoveling have high consistency.

[0094] Step (5.3): According to the boundary limit of the loading area, simultaneously perform fine segmentation on the three-dimensional point cloud of the material pile surface before and after loading after registration to obtain the point cloud model of the actual loading area.

[0095] Step (5.4): According to the obtained point cloud model of the loading area, perform Delaunay triangulation on the point cloud model, parameterize the point cloud for fitting to obtain the point cloud contour envelope, and use the Alpha shape algorithm to estimate the loading volume.

[0096] The loading volume estimation method based on three-dimensional reconstruction of the material pile before and after loading of the present invention has the following beneficial effects:

[0097] (1) By adopting the image feature point detection algorithm SuperPoint and the feature point description algorithm SuperGlue based on a deep neural network, rich feature information of the material pile can be accurately and quickly extracted;

[0098] (2) A probability model is constructed based on Bayesian estimation to obtain a dense disparity map. The sparse point cloud and disparity obtained by matching are used as prior information to reduce the search range of the disparity of the remaining pixel points, improving the calculation efficiency while ensuring sufficient matching accuracy;

[0099] (3) A volume estimation method based on the three-dimensional point cloud of the material pile surface before and after loading is proposed, including five stages: point cloud downsampling, point cloud rough segmentation, point cloud registration, point cloud fine segmentation, and volume calculation, reducing the calculation cost of the point cloud model while having high calculation accuracy.

[0100] See Figure 4 As shown, on the other hand of the present invention, a loading volume acquisition system based on three-dimensional reconstruction of the material pile before and after loading is also disclosed, including:

[0101] The material pile image acquisition module 401 is used to respectively perform binocular image acquisition on the surface of the material pile before loading and the surface of the material pile after loading;

[0102] The triangular mapping parameter acquisition module 402 is used to respectively perform feature point detection and feature point description on the binocular images of the surface of the material pile before loading and the binocular images of the surface of the material pile after loading by using the SuperPoint algorithm, and perform feature point matching and purification by using the SuperGlue algorithm and the random sample consensus algorithm, calculate the sparse matching feature points and the disparity, and obtain the corresponding triangular mapping parameters through Delaunay triangulation;

[0103] The dense three-dimensional point cloud model construction module 403 is used to construct a sparse three-dimensional point cloud based on the sparse feature points and disparities obtained by matching according to the binocular camera stereo imaging principle; according to the obtained sparse three-dimensional point cloud and the triangular mapping parameters, construct a maximum a posteriori probability model to estimate the optimal disparity values of the remaining pixel points, and respectively construct a dense three-dimensional point cloud model of the surface of the material pile before shoveling and a dense three-dimensional point cloud model of the surface of the material pile after shoveling;

[0104] The shoveling volume acquisition module 404 is used to perform point cloud downsampling on the dense three-dimensional point cloud model of the surface of the material pile before shoveling and the dense three-dimensional point cloud model of the surface of the material pile after shoveling respectively by using the voxel grid method; perform rough segmentation on the downsampled point cloud at a certain distance outward from the boundary of the shoveling area, and perform point cloud registration on the pre-shoveling point cloud model and the post-shoveling point cloud model after rough segmentation by using the ICP algorithm; perform fine segmentation on the registered point cloud model according to the boundary of the shoveling area to obtain the point cloud model of the actual shoveling area, and estimate the shoveling volume by using the Alpha shape algorithm of Delaunay triangulation.

[0105] The specific implementation of a shoveling volume acquisition system based on three-dimensional reconstruction of the material pile before and after shoveling is the same as a shoveling volume acquisition method based on three-dimensional reconstruction of the material pile before and after shoveling, and will not be repeated in this embodiment.

[0106] The above embodiments are only for illustrating the technical concept and features of the present invention, and the purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and cannot be used to limit the protection scope of the present invention. All equivalent changes or modifications made according to the spirit of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading, characterized in that Including: Step (1), performing binocular image acquisition on the surface of the material pile before loading; Step (2), for the binocular image of the surface of the material pile before loading, using the SuperPoint algorithm for feature point detection and feature point description, and using the SuperGlue algorithm and the random sample consensus algorithm for feature point matching and purification, calculating the sparse matching feature points and disparity, and obtaining the corresponding triangular mapping parameters through Delaunay triangulation; based on the sparse feature points and disparity obtained by matching, constructing a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle; According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, constructing a maximum a posteriori probability model to estimate the optimal disparity value of the remaining pixel points, and constructing a dense three-dimensional point cloud model of the surface of the material pile before loading; Step (3), performing binocular image acquisition on the surface of the material pile after the construction machinery has completed loading; Step (4), for the binocular image of the surface of the material pile after loading, using the SuperPoint algorithm for feature point detection and feature point description, and using the SuperGlue algorithm and the random sample consensus algorithm for feature point matching and purification, calculating the sparse matching feature points and disparity, and obtaining the corresponding triangular mapping parameters through Delaunay triangulation; based on the sparse feature points and disparity obtained by matching, constructing a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle; According to the obtained sparse three-dimensional point cloud and triangular mapping parameters, constructing a maximum a posteriori probability model to estimate the optimal disparity value of the remaining pixel points, and constructing a dense three-dimensional point cloud model of the surface of the material pile after loading; Step (5), using the voxel grid method to perform point cloud downsampling on the dense three-dimensional point cloud model of the surface of the material pile before loading and the dense three-dimensional point cloud model of the surface of the material pile after loading respectively; performing rough segmentation on the downsampled point cloud at a certain distance outward from the boundary of the loading area, and using the ICP algorithm to perform point cloud registration on the point cloud model before loading and the point cloud model after loading after rough segmentation; performing fine segmentation on the registered point cloud model according to the boundary of the loading area to obtain the actual loading area point cloud model, and using the Alpha shape algorithm of Delaunay triangulation to estimate the loading volume; Using the SuperPoint algorithm for feature point detection and feature point description, and using the SuperGlue algorithm and the random sample consensus algorithm for feature point matching and purification, calculating the sparse matching feature points and disparity, and obtaining the corresponding triangular mapping parameters, specifically including: Step (2.1), using the pre-annotated set shape data set as supervised data, pre-training a basic detection network with VGG16 as the basic detector, extracting the corner points of the left image and the right image respectively, and performing key point annotation; constructing a key point loss function and a descriptor loss function, and jointly training to obtain a SuperPoint detection network; Step (2.2): According to the SuperPoint detection network, the descriptors and coordinate positions of the feature points corresponding to the left and right images of the material pile are obtained respectively, and the feature description vectors D of the left and right images are obtained jointly. (l) and D (r) The two sets of vectors are input into SuperGlue's attention graph neural network and optimal matching layer. The Sinkhorn algorithm is used to iteratively solve the allocation optimization problem and obtain the optimal allocation matrix. The feature point pairs corresponding to the horizontal and vertical coordinates of the maximum value in each column of the allocation matrix are then used as matching point pairs. The random sampling consensus algorithm is used to calculate the affine transformation matrix. The corresponding affine transformation is performed on the right image of the material pile to complete the matching and purification of the left and right images and obtain sparse matching points. Step (2.3), based on the sparse matching points, obtain the corresponding triangular mapping parameters through Delaunay triangulation The calculation is as follows: Among them, n represents the serial number of the triangle containing the left image pixel ; Solving the linear equation for the three vertices of each triangle can obtain the triangle plane parameters (a i , b i , c i ); u n represents the abscissa of the image pixel; v n represents the ordinate of the image pixel.

2. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 1, wherein, Before performing binocular image acquisition on the surface of the material pile before loading, it also includes: Calibrating and image correcting the binocular stereo camera installed on the construction machinery.

3. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 2, wherein, Calibrate the binocular stereo camera, specifically including: Obtain the internal and external parameters of the camera through the Zhang Zhengyou calibration method. The internal parameters of the camera include the projection position coordinates (u0, v0) of the camera lens optical axis in the pixel coordinate system and the camera focal length f. The external parameters of the camera include the rotation matrix R and the translation matrix T. The conversion between the image pixel coordinate system and the world coordinate system is realized through the internal and external parameters of the camera.

4. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 2, wherein Perform image correction on the binocular stereo camera, specifically including: Make a point in the left image find its corresponding point in the right image along the same horizontal epipolar line.

5. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 4, wherein Based on the sparse feature points and disparities obtained by matching, construct a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle; according to the obtained sparse three-dimensional point cloud and the triangulation mapping parameters, construct a maximum a posteriori probability model to estimate the optimal disparity value of the remaining pixel points, and construct a dense three-dimensional point cloud model of the surface of the material pile before shoveling or a dense three-dimensional point cloud model of the surface of the material pile after shoveling. Specifically including: Step (3.1), the sparse three-dimensional point cloud constructed based on the binocular camera stereo imaging principle is calculated by the following formula: Among them, (X c , Y c , Z c ) is a point in the three-dimensional scene, and (u, v) is a pixel point on the two-dimensional image; f x represents transforming the focal length f of the camera into the pixel metric in the x direction; f y represents transforming the focal length f of the camera into the pixel metric in the y direction; B represents the baseline length of the binocular camera; u (l) represents the abscissa of the pixel point in the left image; u (r) represents the abscissa of the pixel point in the right image; u0 represents the abscissa of the projection position of the optical axis of the camera lens in the pixel coordinate system; v0 represents the ordinate of the projection position of the optical axis of the camera lens in the pixel coordinate system; Step (3.2), in the method of constructing a maximum a posteriori probability model to estimate the optimal disparity value of the remaining pixel points, the constructed probability estimation model is shown in Equation (3): Among them, represents the estimated probability value; argmax represents the function of finding the parameter set; P(|) represents finding the conditional probability; S = (s1, s2,... s M ) represents the sparse three-dimensional point cloud constructed previously, and each point cloud has three-dimensional coordinates (X c , Y c , Z c ), and each point is represented as s m = (v m , u m , d m ), where d m is the disparity corresponding to the point (u m , u m ); u m represents the abscissa in the image pixel corresponding to each point; v m represents the ordinate in the image pixel corresponding to each point; represents all pixel points in the right image that have the same horizontal epipolar line as ; regarding the disparity d of a certain point n in the left image as the random variable to be solved, the posterior probability is expressed as the product of the prior probability and the likelihood probability, as shown in Equation (4): The prior probability is proportional to the Gaussian distribution, as shown in Equation (5): Among them, represents a triangular mapping composed of the sparse three-dimensional point cloud S and containing pixel points ; ∝ represents being proportional to; exp represents the exponential function with the natural constant e as the base; σ represents the standard deviation; μ represents the mean; the likelihood probability is expressed as a Laplace distribution, as shown in Equation (6): Among them, and respectively represent the feature description vectors of the nth pixel point in the left image and the nth pixel point in the right image; ‖.‖ represents the norm function; since the left and right images obtained by the binocular camera have been pre-calibrated, the corresponding points in the left image and the right image will necessarily appear on the same horizontal epipolar line, so the limiting condition is ensured by the if condition in Equation (6); further derivation of the likelihood probability model for Equation (6) is shown in Equation (7): Step (3.3), after obtaining the disparity values of all pixel points in the image, then calculate the dense three-dimensional point cloud model of the surface of the material pile before shoveling or the dense three-dimensional point cloud model of the surface of the material pile after shoveling according to Equation (2).

6. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 1, wherein Step (5) specifically includes: Step (5.1), first use the voxel grid method to downsample the three-dimensional point clouds of the surfaces of the material piles before and after shoveling at the same time; after completing the downsampling, then expand a certain area outward from the boundary of the shoveling area, and perform rough segmentation on the three-dimensional point clouds of the surfaces of the material piles before and after shoveling respectively. Step (5.2), take the three-dimensional point cloud of the surface of the material pile before shoveling after rough segmentation as the reference model, and the three-dimensional point cloud of the surface of the material pile after shoveling after rough segmentation as the registration model, and use the ICP algorithm for registration to make the poses of the three-dimensional point clouds of the surfaces of the material piles before and after shoveling consistent. Step (5.3), according to the boundary limit of the shoveling area, perform fine segmentation on the registered three-dimensional point clouds of the surfaces of the material piles before and after shoveling at the same time to obtain the point cloud model of the actual shoveling area. Step (5.4), according to the obtained point cloud model of the actual shoveling area, perform Delaunay triangulation on the point cloud model, parameterize the point cloud to fit the point cloud contour envelope, and use the Alpha shape algorithm to estimate the shoveling volume.

7. The method for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading according to claim 1, wherein The surface of the material pile before shoveling refers to the surface of the material pile before the first or single shoveling of the material pile, including the surface contour information of the material pile; the surface of the material pile after shoveling refers to the surface of the material pile after a single shoveling of the material pile, including the surface contour information of the material pile.

8. A system for obtaining the loading volume based on the three-dimensional reconstruction of the material piles before and after loading, characterized in that, Based on the method according to any one of claims 1 to 7, the system includes: The material pile image acquisition module is used to perform binocular image acquisition on the surface of the material pile before and after loading respectively; The triangular mapping parameter acquisition module is used to perform feature point detection and feature point description on the binocular images of the surface of the material pile before and after loading respectively using the SuperPoint algorithm, and perform feature point matching and purification using the SuperGlue algorithm and the random sample consensus algorithm, calculate the sparse matching feature points and disparities, and obtain the corresponding triangular mapping parameters through Delaunay triangulation; The dense three-dimensional point cloud model construction module is used to construct a sparse three-dimensional point cloud based on the binocular camera stereo imaging principle according to the matched sparse feature points and disparities; construct a maximum a posteriori probability model to estimate the optimal disparity values of the remaining pixel points according to the obtained sparse three-dimensional point cloud and triangular mapping parameters, and construct a dense three-dimensional point cloud model of the surface of the material pile before loading and a dense three-dimensional point cloud model of the surface of the material pile after loading respectively; The loading volume acquisition module is used to perform point cloud downsampling on the dense three-dimensional point cloud models of the surface of the material pile before and after loading respectively using the voxelized grid method; perform rough segmentation on the downsampled point cloud at a certain distance outside the boundary of the loading area, and perform point cloud registration on the point cloud models before and after loading after rough segmentation using the ICP algorithm; perform fine segmentation on the registered point cloud model according to the boundary of the loading area to obtain the actual loading area point cloud model, and estimate the loading volume using the Alpha shape algorithm of Delaunay triangulation.

Images