Real-time measurement method of large irregular stockpile volume based on binocular stereo matching

Abstract

A real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching includes the following steps: S1 using multiple binocular cameras to acquire images; S2 robust stereo matching based on deep learning; S3 matching confidence assessment; S4 point cloud generation and registration; S5 stockpiling 3D model reconstruction; S6 stockpiling volume calculation. This measurement method can effectively resist the influence of complex outdoor lighting on measurement accuracy, avoid or reduce the influence of stockpiling occlusion, and is used for fast and accurate measurement of large stockpiles with complex surface textures, thereby meeting the actual needs of efficient stockpiling supervision, and has the advantages of low cost, high accuracy, efficient calculation, and real-time response.

Description

[Technical field]

[0001] The invention relates to a measurement method, in particular to a real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching. [Background technology]

[0002] In engineering projects and port freight, it is necessary to measure the volume of stockpiles (such as coal piles, gravel, etc.). Accurate measurement of the volume of stockpiles is very important for port operation management and cost control. The current methods for measuring the volume of stockpiles have limitations, including the following methods:

[0003] (1) Direct weighing method, which uses a large electronic scale to directly measure the weight of the pile and calculates the volume based on the density parameter of the material. This method is simple to operate, but requires an expensive large electronic scale and cannot measure the volume of a single pile of material.

[0004] (2) UAV aerial survey modeling method, which uses drones to take low-altitude aerial photos to obtain images of the stockpile, and then uses the principle of photogrammetry to build a three-dimensional point cloud model to calculate the volume of the stockpile. This method can measure the volume of each stockpile individually, but it requires drone equipment, which is labor-intensive and computationally complex. It takes hours or even days from flight photography to generating a three-dimensional model, which cannot meet the port's demand for real-time monitoring of the stockpile volume.

[0005] (3) Laser scanning 3D reconstruction method, which uses a ground laser scanner to perform 3D scanning on the pile, obtain point cloud data, and then generate a 3D model for calculation. This method has high accuracy, but the equipment is expensive and the scanning range is limited.

[0006] (4) Depth camera volume measurement method. Existing depth camera-based volume measurement methods usually use binocular camera stereo vision to obtain the image of the target object, calculate the parallax information, and construct a three-dimensional scene. Patent announcement number: CN110738618B discloses a method for measuring the volume of irregular piles based on binocular cameras. This method uses a special binocular camera arrangement position to avoid the problem of occlusion between piles. However, this solution requires the camera to be placed at a position perpendicular to the pile, which is difficult to achieve in practical applications and cannot adapt to complex on-site environments. Another method is to use a laser group to project a laser grid to increase the surface texture features of the pile to improve the matching accuracy of the binocular camera. For example, patent announcement number: CN112561983A discloses a device and method for measuring the volume of irregular piles with weak surface textures. This method has problems in outdoor strong light environments. The laser grid is easily interfered by other strong light sources and cannot effectively improve the matching and reconstruction effects of the binocular camera. If a higher-power laser is used, it may cause harm to the health of the staff and also face safety issues.

[0007] It can be seen that the existing volume measurement technology based on depth cameras, whether from the specific arrangement of binocular cameras or the use of auxiliary means such as laser structured light, is difficult to adapt to the influence of lighting in complex outdoor scenes, and thus cannot guarantee the accuracy, efficiency and real-time performance of the measurement. [Summary of the invention]

[0008] In order to solve the above problems, the present invention provides a real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching, which has low cost, high accuracy, efficient calculation and real-time response. The measurement method can effectively resist the influence of complex outdoor lighting on the measurement accuracy, avoid or reduce the influence of stockpile occlusion, and is used for fast and accurate measurement of large stockpiles with complex surface textures, thereby meeting the actual needs of efficient supervision of stockpiles.

[0009] The present invention is implemented by the following technical scheme, and provides a real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching, comprising the following steps:

[0010] S1 uses multiple sets of binocular cameras to acquire images;

[0011] S2 Robust stereo matching based on deep learning;

[0012] S3 matching confidence evaluation;

[0013] S4 point cloud generation and registration;

[0014] S5 3D model reconstruction of stockpile;

[0015] S6 calculates the volume of the stockpile.

[0016] In particular, the S1 is implemented according to the following scheme:

[0017] Multiple groups of binocular cameras are used to shoot the pile of materials to be tested from different angles to obtain multi-view image information; the binocular cameras are two closely arranged monocular cameras. When in use, the cameras on both sides synchronously obtain left and right view images under the same field of view.

[0018] In particular, in S1, the plurality of binocular cameras are arranged in an array at a high position around the pile of materials, and each camera is aimed at the pile of materials to capture images in a bird's-eye or tilted shooting state.

[0019] In particular, the binocular cameras in S1 are all calibrated and work synchronously after calibration; the cameras realize image distortion correction, and the internal parameters and the external parameters between the binocular system are accurately determined.

[0020] In particular, the S2 is implemented in accordance with the following scheme:

[0021] S21 uses neural networks to extract image features, directly learns the cost calculation and optimization strategy in the matching process, and obtains the disparity map of the left and right images through the cost space constructed by stereo matching;

[0022] S22 builds a large-scale customized training dataset, which is collected by using both binocular cameras and lidar at the actual stockpiling site, and contains rich training samples of various types of stockpiles.

[0023] In particular, the S3 is implemented in accordance with the following scheme:

[0024] Using information entropy theory, the matching credibility is directly regressed from the probability distribution. The formula is as follows:

[0025]

[0026] In formula (1), M(x, y) is the uncertainty of the disparity map, C(x, y, d) is the cost space for matching each pixel, and D corresponds to the maximum disparity pre-set by the network.

[0027] In particular, the S4 is implemented according to the following scheme:

[0028] S41 disparity map post-processing, post-processing the original disparity map generated by stereo matching, using bilateral filtering and median filtering to optimize the smoothness and integrity of the disparity map, fill the small area holes in the disparity map, and output the optimized high-quality disparity results, laying the foundation for accurate point cloud generation;

[0029] S42 multi-view point cloud generation, based on the calibration parameters of the binocular system, based on the disparity information of the stereo matching point pairs, generates a three-dimensional point cloud through triangulation principle;

[0030] S43 camera pose determination, by adding multiple calibration objects with known three-dimensional coordinates in the scene, which appear in the images of each binocular camera respectively, and using the PnP solution algorithm to determine the pose of the binocular camera based on the mapping relationship between the image coordinates of the calibration objects and the world coordinates;

[0031] S44 point cloud global registration: according to the position information of the binocular cameras, the 3D point clouds of multiple binocular cameras are converted to the same 3D coordinate system to achieve rough registration of the pile point cloud;

[0032] S45 precision registration uses a registration algorithm to perform detailed point cloud registration, so that point clouds from different perspectives accurately overlap to obtain a three-dimensional point cloud of the scene.

[0033] In particular, the S5 is implemented in accordance with the following scheme:

[0034] S51 performs noise reduction processing on the point cloud to filter out the point cloud noise;

[0035] S52 point cloud segmentation removes irrelevant point clouds and extracts complete stockpile point clouds.

[0036] In particular, the point cloud segmentation in S52 is specifically implemented according to the following scheme:

[0037] S521 Clustering based on point cloud;

[0038] S522 Point cloud segmentation method based on deep learning;

[0039] S523 uses deep learning methods to directly segment the stockpiling area on the binocular camera image pair, extract the corresponding area on the disparity map, convert it into a point cloud, and align it.

[0040] In particular, the S6 is implemented in accordance with the following scheme:

[0041] S61 performs voxel processing on the three-dimensional grid model of the pile, that is, evenly divides the model space into small cubic blocks, each of which is a voxel;

[0042] S62 calculates the side length of the voxel, which can be set according to the model size and the required calculation accuracy, and is assumed to be set to a;

[0043] S63 determines, for each voxel, whether it is inside the stockpile model;

[0044] S64 For voxels inside the model, their volumes can be directly accumulated, and the voxel volume is a^3. For voxels on the surface of the model, the proportion of the model volume actually contained in the voxel is calculated and then multiplied by the voxel volume accumulation;

[0045] S65 repeats the process to calculate the sum of the model volumes corresponding to all voxels.

[0046] Compared with the prior art, the present invention provides a real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching. The method can effectively resist the influence of complex outdoor lighting on the measurement accuracy, avoid or reduce the influence of stockpile occlusion, and is suitable for fast and accurate measurement of large stockpiles with complex surface textures, thereby meeting the actual needs of efficient supervision of stockpiles. The method has the advantages of low cost, high accuracy, efficient calculation, and real-time response.

Brief Description of the Drawings

[0047] Figure 1 is a flow chart of the present invention;

[0048] Figure 2 (a) is a hardware schematic diagram of the present invention for collecting images, wherein (b) is a left image collected by a binocular camera in an embodiment, and (c) is a right image collected by a binocular camera in an embodiment;

[0049] Figure 3A calibration board used for multi-camera extrinsic calibration in an embodiment of the present invention;

[0050] Figure 4 (a) is a disparity map predicted by a deep learning method in an embodiment of the present invention, (b) is a disparity map predicted by an SGBM method in an embodiment, and (c) is a disparity map predicted by a BM method in an embodiment.

[0051] Figure 5 (a) is the uncertainty of the disparity map estimated based on information entropy in the embodiment, and the black part is the area with low confidence, and (b) is the disparity map obtained after confidence filtering in the embodiment;

[0052] Figure 6 (a) is a multi-view point cloud generated by two binocular cameras in an embodiment of the present invention, (b) is a point cloud after coarse registration in an embodiment, and (c) is a point cloud after fine registration in an embodiment;

[0053] Figure 7 This is the stockpile point cloud extracted in the embodiment of the present invention. [Specific implementation method]

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] See also Figure 1 The present invention provides a real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching, which specifically comprises the following steps:

[0056] 1. Multiple binocular cameras acquire images

[0057] This step uses multiple sets of binocular cameras to shoot the pile of materials to be tested from different angles to obtain multi-view image information. In the present invention, the binocular camera includes two closely arranged monocular cameras, and the cameras on both sides synchronously obtain left and right view images under the same field of view. This embodiment uses multiple sets of binocular cameras, and the array is arranged at a high place around the pile of materials. Each camera is aimed at the pile of materials to capture images in a top-down or tilted state.

[0058] In the present invention, the use of multiple sets of binocular cameras can solve the problem of occlusion in single-view stereo vision, and the mutual occlusion caused by the structure of the pile itself can be minimized by comprehensive multi-view imaging. The camera position only needs to ensure that the main area of ​​each pile surface can be imaged by the camera's photosensitive surface. Considering the convenience of on-site work, industrial-grade cameras installed on floor stands are used.

[0059] It should be noted that all the binocular cameras used in this system have been calibrated, and the camera intrinsic parameters and the extrinsic parameters between the binocular system have been accurately determined, which can lay the foundation for the accuracy of subsequent image matching and three-dimensional reconstruction algorithms.

[0060] All cameras work synchronously after fixed calibration, and output image streams in real time for subsequent algorithm processing to meet the needs of dynamic monitoring of stockpile volume.

[0061] In this step, the method for calibrating the internal and external parameters of the binocular camera is binocular stereo vision system calibration, and stereo correction is performed on the collected image according to the parameters obtained by binocular calibration.

[0062] In the embodiment of the present invention, two binocular cameras are used for scene reconstruction. Each binocular camera is equipped with an industrial camera lens, has a 90° field of view, is installed on a 2.5-meter-high bracket, and is distributed on the left and right sides of the object to be measured. Figure 2 (a) and (b) are the left and right images taken by the same binocular camera.

[0063] After installing and fixing multiple binocular cameras, before performing volume measurement, it is necessary to obtain the approximate position and posture of each binocular camera. The specific method is as follows: Place a binocular camera in the scene. Figure 3 The chessboard calibration plate of known size (1.4m×1.2m) is shown, so that the calibration plate is included in the left camera image of each binocular camera. Use the findChessboardCorners function in OpenCV to detect the chessboard corners in the image. Since the actual coordinates of the calibration plate are known, the solvePnP algorithm is used to calibrate the external parameters of each binocular camera in turn.

[0064] 2. Robust Stereo Matching Based on Deep Learning

[0065] Stereo matching principle: The basic principle of stereo matching is to find corresponding points between binocular image pairs, restore the three-dimensional coordinate information of the pixels according to the disparity of the pixels in the left and right views, and finally reconstruct the three-dimensional scene. The problem that this method needs to solve is how to improve the robustness of the matching algorithm in scenes with relatively weak textures such as the pile surface.

[0066] Stereo matching method based on deep learning: Considering the above problems, this method adopts the most advanced stereo matching technology based on deep learning, such as CREStereo, Raft-Stereo, etc. This method uses neural networks to extract image features and directly learn the cost calculation and optimization strategy in the matching process. Compared with traditional algorithms, deep learning methods are more effective in processing difficult texture areas.

[0067] Customized training data and model optimization: In order to make the deep learning model better adapt to the visual scene of the pile, a large-scale customized training data set is constructed. The data set is collected by using both a binocular camera and a lidar at the actual pile site, and contains rich training samples of various types of piles. The model is trained on this data set and achieves better matching accuracy and robustness than the general pre-trained model.

[0068] Based on the proposed method, experiments were conducted to Figure 2 The left and right images of the binocular camera in are used as the neural network input, and the disparity map of the input scene is obtained by different methods, including the deep learning method after customized data training, the SGBM (Semi-Global Block Matching) method, and the BM (Block Matching) method to predict the disparity map, such as Figure 4 According to the experimental results, it can be seen that the traditional methods SGBM and BM methods have a large number of empty areas, while the deep learning-based method can completely estimate the scene disparity map and has stronger robustness.

[0069] 3. Matching confidence evaluation

[0070] The use of deep learning methods has greatly improved the robustness of matching, but there may still be mismatches in some local areas due to occlusion and other reasons, which will cause distortion in subsequent 3D reconstruction. To avoid this problem, a matching confidence evaluation module is designed based on stereo matching.

[0071] Based on the deep learning stereo matching framework, a matching credibility evaluation module is introduced to achieve accurate mismatch detection and control. The matching network includes core processes such as feature extraction, correlation learning and cost aggregation. Based on this perspective, the information entropy theory is used to directly regress the matching credibility from the probability distribution. The formula is as follows:

[0072]

[0073] Where M(x,y) is the uncertainty of the disparity map, C(x,y,d) is the cost space for matching each pixel, and D corresponds to the maximum disparity set by the network. Since the information entropy calculation is differentiable, the joint optimization training of matching and matching credibility evaluation is achieved. Finally, the credibility result accurately indicates the location of the error in the disparity map, which can be used to improve the matching quality. Figure 5 As shown in (a), the disparity map estimated based on information entropy has uncertainty, and the black part is the area with low confidence. Therefore, in order to solve this problem, the present invention adopts the above method to obtain the following after confidence filtering: Figure 5 (b).

[0074] 4 Point cloud generation and registration

[0075] (1) Disparity map post-processing: Post-process the original disparity map generated by stereo matching, and use algorithms such as bilateral filtering and median filtering to further optimize the smoothness and integrity of the disparity map. Fill in small area holes that may exist in the disparity map, and output optimized high-quality disparity results, laying the foundation for accurate point cloud generation.

[0076] (2) Multi-view point cloud generation: Based on the calibration parameters of the binocular system and the disparity information of the stereo matching point pairs, a 3D point cloud is generated through triangulation. Figure 6 (a) shows the point cloud. In the present invention, the point clouds at different camera positions reflect independent field of view coverage.

[0077] (3) Camera pose determination: This method adds multiple calibration objects (such as calibration plates) with known three-dimensional coordinates in the scene, which appear in the images of each binocular camera respectively. Based on the mapping relationship between the image coordinates of the calibration objects and the world coordinates, the PnP solution algorithm is used to determine the poses of multiple binocular cameras.

[0078] (4) Global registration of point clouds: According to the position information of the binocular cameras, the 3D point clouds of multiple binocular cameras are converted to the same 3D coordinate system. After the rough registration of the pile point clouds, the following is obtained: Figure 6 (b) Point cloud shown.

[0079] (5) Precise registration: Use a registration algorithm such as ICP to perform detailed point cloud registration so that point clouds from different perspectives accurately overlap and obtain a three-dimensional point cloud of the scene. Figure 6 (c) Point cloud shown.

[0080] In this embodiment, after obtaining the depth map of the binocular camera, the opencv function medianBlur is used to perform median filtering, and then the disparity map is converted into a depth map by the following formula:

[0081]

[0082] Among them, Z is the depth value of the pixel in the target depth image, f is the focal length of the lens, B is the baseline length of the binocular camera, and D is the disparity value corresponding to the pixel.

[0083] According to the multi-camera extrinsic calibration results in the previous step, the point clouds under the two perspectives are converted to the same coordinate system, and the rough registration effect is obtained as follows Figure 6 (b) The color ICP algorithm is used to further optimize the registration of the point cloud, fuse the point cloud information from different perspectives, and finally obtain the 3D point cloud of the object under test.

[0084] 5. Pile point cloud extraction

[0085] (1) Perform denoising on the point cloud to filter out the point cloud noise.

[0086] (2) Point cloud segmentation: remove irrelevant point clouds and extract the complete pile point cloud, such as Figure 7 shown.

[0087] In this step, the point cloud segmentation method described in step (2) can be: 1) clustering based on point cloud, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering. DBSCAN clustering is a density-based spatial clustering algorithm. Its main idea is that for each point, if the density (i.e., the number of neighborhood points) within the specified radius exceeds the minimum number of points, the point is considered to be a core point; those points within the neighborhood of the core point are considered to be directly density-reachable points of the core point; points that do not belong to any cluster are considered to be noise points or outliers. The advantage of DBSCAN is that it does not require the specification of the number of clusters, can handle clusters of any shape well, and identify noise points. In this embodiment, the cluster_dbscan function in the Open3D library is used, where the search radius eps = 0.01 and the minimum number of points required for clustering min_points = 100; 2) Alternatively, a point cloud segmentation method based on deep learning is used. In this embodiment, the PointNet++ method is used to segment the sand pile point cloud. This type of method can learn the global and local features of the point cloud end-to-end and perform semantic segmentation. Compared with traditional methods, deep learning methods have stronger generalization capabilities for complex scenes. ; 3) Alternatively, in this embodiment, a deep learning-based image semantic segmentation method, such as SAM (Segment Anything Model), is used to directly perform pixel-level semantic segmentation on the binocular camera image and extract the target sand pile area. Then, according to the disparity information of the image, it is converted and registered into a 3D point cloud for further analysis.

[0088] 6. Pile volume calculation

[0089] (1) The three-dimensional grid model of the pile is voxelized, that is, the model space is evenly divided into small cubic blocks, each of which is a voxel.

[0090] (2) The side length of the calculated voxel can be set according to the model size and the required calculation accuracy. In this embodiment, it is set to 0.1 meter.

[0091] (3) For each voxel, determine whether it is inside the pile model. In this embodiment, ray detection is used for determination: a detection ray parallel to the coordinate axis is emitted from the center point of each voxel, and the intersection points of the ray and the sand pile surface model are counted. If the number of intersection points is an odd number, the voxel is inside; if it is an even number, it is outside.

[0092] (4) For voxels inside the model, their volumes can be directly accumulated; for voxels on the surface of the model, the proportion of the model volume actually contained in the voxel can be calculated and then multiplied by the voxel volume accumulation

[0093] (5) Repeat this process to calculate the sum of the model volumes corresponding to all voxels.

Claims

1. A real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching, characterized in that: The following steps are involved: S1 uses multiple sets of binocular cameras to acquire images; S1 is specifically implemented according to the following scheme: Multiple sets of binocular cameras are used to shoot the pile to be tested from different angles to obtain multi-view image information; the binocular cameras are two closely arranged monocular cameras. When in use, the cameras on both sides synchronously obtain left and right view images under the same field of view; S2 is a robust stereo matching based on deep learning; S2 is specifically implemented according to the following scheme: S21 uses neural networks to extract image features, directly learns the cost calculation and optimization strategy in the matching process, and obtains the disparity map of the left and right images through the cost space constructed by stereo matching; S22 builds a large-scale customized training data set. The data set is collected by using both binocular cameras and lidar at the actual stockpiling site, and contains rich training samples of various types of stockpiles. S3: Matching confidence evaluation; S3 is specifically implemented according to the following scheme: Using information entropy theory, the matching credibility is directly regressed from the probability distribution. The formula is as follows: In formula (1), M(x, y) is the uncertainty of the disparity map, C(x, y, d) is the cost space for matching each pixel, and D corresponds to the maximum disparity pre-set by the network; S4 point cloud generation and registration; S4 is specifically implemented according to the following scheme: S41 disparity map post-processing, post-processing the original disparity map generated by stereo matching, using bilateral filtering and median filtering to optimize the smoothness and integrity of the disparity map, fill the small area holes in the disparity map, and output the optimized high-quality disparity results, laying the foundation for accurate point cloud generation; S42 multi-view point cloud generation, based on the calibration parameters of the binocular system, based on the disparity information of the stereo matching point pairs, generates a three-dimensional point cloud through triangulation principle; S43 camera pose determination, by adding multiple calibration objects with known three-dimensional coordinates in the scene, which appear in the images of each binocular camera respectively, and using the PnP solution algorithm to determine the pose of the binocular camera based on the mapping relationship between the image coordinates of the calibration objects and the world coordinates; S44 point cloud global registration: according to the position information of the binocular cameras, the 3D point clouds of multiple binocular cameras are converted to the same 3D coordinate system to achieve rough registration of the pile point cloud; S45 precision registration uses registration algorithms to perform detailed point cloud registration, so that point clouds from different perspectives accurately overlap to obtain the three-dimensional point cloud of the scene; S5: Reconstruction of the three-dimensional model of the pile; S5 is specifically implemented according to the following scheme: S51 performs noise reduction processing on the point cloud to filter out the point cloud noise; S52 point cloud segmentation, remove irrelevant point clouds, and extract complete stockpile point clouds; the point cloud segmentation in S52 is specifically implemented according to the following scheme: S521 Clustering based on point cloud; S522 Point cloud segmentation method based on deep learning; S523 uses deep learning methods to directly segment the stockpiling area on the binocular camera image pair, extract the corresponding area on the disparity map, convert it into a point cloud, and align it; S6 calculates the volume of the stockpile.

2. The real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching according to claim 1 is characterized in that: In S1, the multiple groups of binocular cameras are arranged in an array at a high position around the pile of materials, and each camera is aimed at the pile of materials to collect images in a bird's-eye or tilted shooting state.

3. The real-time measurement method of large irregular stockpile volume based on binocular stereo matching according to claim 1 is characterized in that: The binocular cameras in S1 are all calibrated and work synchronously after calibration; the cameras realize image distortion correction, and the internal parameters and the external parameters between the binocular system are accurately determined.

4. The real-time measurement method for the volume of large irregular stockpiles based on binocular stereo matching according to claim 1 is characterized in that: The S6 is specifically implemented according to the following scheme: S61 performs voxel processing on the three-dimensional grid model of the pile, that is, evenly divides the model space into small cubic blocks, each of which is a voxel; S62 calculates the side length of the voxel, which can be set according to the model size and the required calculation accuracy, and is assumed to be set to a; S63 determines, for each voxel, whether it is inside the stockpile model; S64 For voxels inside the model, their volumes can be directly accumulated, and the voxel volume is a^3; for voxels on the surface of the model, the proportion of the model volume actually contained in the voxel is calculated and then multiplied by the voxel volume accumulation; S65 repeats the process to calculate the sum of the model volumes corresponding to all voxels.

Images