Mine three-dimensional reconstruction system and method based on improved semi-global matching algorithm

Abstract

The invention relates to a mine three-dimensional reconstruction system and method based on an improved semi-global matching algorithm, and belongs to the technical field of image processing. According to the invention, equipment such as the binocular camera, the multispectral camera, the structured light transmission instrument and the IMU are integrated on the mine inspection vehicle, so that acquisition and fusion processing of multi-modal data are realized. The system performs three-dimensional reconstruction by using an improved semi-global matching algorithm, and the reconstruction precision and efficiency in a complex mine environment are improved. Finally, through a visual display module and a model control module, visual and comprehensive three-dimensional reconstruction result display and interactive operation functions are provided for a user. According to the method, the problems of uneven illumination, texture loss, shielding and the like existing in the underground coal mine in a traditional three-dimensional mode are solved, and a more accurate mine three-dimensional reconstruction model is provided.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a mine three-dimensional reconstruction system and method based on an improved semi-global matching algorithm. BACKGROUND

[0002] In the field of mine exploitation and management, three-dimensional reconstruction technology is playing an increasingly important role and showing broad application prospects.

[0003] In the field of mine geological exploration and resource assessment, traditional methods often have difficulty in intuitively and accurately presenting the geological structure and resource distribution of the mine. Three-dimensional reconstruction technology can clearly display key information such as the geological structure, rock layer distribution, and fault location of the mine by converting geological data into an intuitive three-dimensional model. This not only provides more comprehensive data support for geological exploration personnel, but also makes resource reserve calculation more accurate, helps to develop more scientific mining plans, reduces resource waste, and improves mining efficiency.

[0004] Mine safety is always a top priority, but the complex and variable mine environment poses a great challenge to safety monitoring and disaster warning. Traditional monitoring methods often have problems such as incomplete information acquisition and untimely warning. The introduction of three-dimensional reconstruction technology makes it possible to monitor the deformation and displacement of the roadway in real time. By constructing a three-dimensional model of the mine and combining sensors and monitoring systems, environmental parameters and equipment status within the mine can be monitored in real time, and potential safety hazards can be detected in a timely manner. At the same time, disaster simulation and warning using three-dimensional models can help develop emergency plans in advance, reduce the risk and loss of disasters, and provide strong support for mine safety production.

[0005] In the planning and design stage of the mine, traditional two-dimensional drawings cannot fully and accurately reflect the actual situation of the mine, resulting in unreasonable mining plans and equipment layout. Three-dimensional reconstruction technology provides accurate geological and mine structure information to help engineers make more scientific planning and design. Based on the three-dimensional model, mining sequence, roadway layout, and equipment selection and layout can be optimized to ensure efficient operation of equipment, reduce equipment failure and downtime, and improve overall operational efficiency of the mine.

[0006] In the aspect of mine operation and management, traditional inspection and maintenance methods are often inefficient and difficult to fully grasp the running status of equipment. Three-dimensional reconstruction technology combined with personnel positioning technology can monitor the distribution of personnel within the mine in real time to ensure personnel safety. At the same time, through the intuitive display of equipment position and status in the three-dimensional model, maintenance personnel can more conveniently perform inspection and maintenance work, improve equipment utilization and life, and reduce equipment replacement and maintenance costs.

[0007] However, due to the problems of uneven illumination, texture loss and occlusion in the mine environment, the accuracy of the existing stereo matching and three-dimensional reconstruction method is affected. SUMMARY

[0008] Therefore, the purpose of the present application is to provide a mine three-dimensional reconstruction system and method based on an improved semi-global matching algorithm.

[0009] To achieve the above purpose, the present application provides the following technical scheme:

[0010] A mine three-dimensional reconstruction system based on an improved semi-global matching algorithm, the system comprises a data acquisition layer, a preprocessing layer, a three-dimensional reconstruction layer and an application layer, wherein the data acquisition layer sets up a mine inspection vehicle, and a binocular camera, a multispectral camera, a structured light transmitter and an inertial measurement unit IMU are arranged on each mine inspection vehicle;

[0011] The data acquisition layer continuously acquires left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine through the above-mentioned devices;

[0012] The data preprocessing layer performs corresponding preprocessing operations on the acquired left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data;

[0013] The three-dimensional reconstruction layer dynamically adjusts the path weight according to the IMU data and aggregates the cost, then fuses the multi-modal cost to determine the final disparity, and then calculates the depth map based on the disparity and the camera parameters, and then converts the depth map into point cloud in the camera coordinate system, and finally performs denoising, hole filling and surface reconstruction to obtain the three-dimensional reconstruction result;

[0014] The application layer sets up a visual display module and a model control module for the operating user, the visual display module displays the three-dimensional reconstruction result; and the model control module can perform local adjustment of the model.

[0015] Further, the three-dimensional reconstruction layer comprises a cost calculation module, a cost aggregation module, a disparity calculation module, a depth conversion module, a point cloud generation module and a point cloud optimization module, the cost calculation module calculates the corresponding cost under each modal data respectively, the cost aggregation module dynamically adjusts the path weight according to the IMU data and aggregates the cost; the disparity calculation module fuses the multi-modal cost to determine the final disparity; the depth conversion module calculates the depth map based on the disparity and the camera parameters; the point cloud generation module converts the depth map into point cloud in the camera coordinate system; and the point cloud optimization module performs denoising, hole filling and surface reconstruction.

[0016] Further, in the cost calculation module, multi-modal cost calculation is performed:

[0017] The visible light cost is calculated by using the improved Census transform and gradient fusion method:

[0018] C rgb ((u,v),(u,v) d )=λ1·C census ((u,v),(u,v) d )+λ2·C grad ((u,v),(u,v) d )

[0019] wherein C rgb ((u,v),(u,v) d ) represents the cost function between the pixel (u,v) and its corresponding pixel (u,v) d with the displacement d in the visible light image; λ1, λ2 are weight coefficients for balancing the relative importance between the Census transform cost and the gradient fusion cost; C census ((u,v),(u,v) d ) represents the Census transform cost, which is calculated by comparing the order difference of pixel intensity in the neighborhood window of the pixel (u,v) and (u,v) d ; C grad ((u,v),(u,v) d ) is the gradient fusion cost, which calculates the gradient difference between the pixel (u,v) and (u,v) d by using the Scharr operator;

[0020] The band difference cost of the near-infrared band is calculated:

[0021]

[0022] wherein C nir ((u,v),(u,v) d ) represents the cost function between the pixel (u,v) and its corresponding pixel (u,v) d with the displacement d in the near-infrared band; and respectively represent the intensity values of the pixel (u,v) and (u,v) d in the near-infrared band in the left and right images; ||·||1 represents the L1 norm;

[0023] The structured light cost is calculated by taking the decoded speckle displacement as a depth prior constraint:

[0024]

[0025] C struct ((u,v),(u,v) d) represents the structured light cost function, which uses the decoded speckle displacement as a depth prior constraint; β is a weight coefficient for adjusting the importance of the structured light cost in the overall cost function; d struct represents the speckle displacement decoded by the structured light; σ 2 is a variance parameter for controlling the smoothness of the exponential function and affecting the calculation of the structured light cost.

[0026] Further, in the cost aggregation module, dynamic path weight adjustment is performed, and on the basis of several path aggregations, the weight is adjusted according to the motion data of the inertial measurement unit, and the formula is represented as:

[0027]

[0028] wherein, W base is a basic path weight, ΔW high_dyn is a path weight increment in a high dynamic scene, and ΔW cross_band is a cross-band path weight increment in a weak texture region, wherein the high dynamic scene refers to a scene with an angular velocity greater than a preset value;

[0029] Then, for each pixel position (u, v) and disparity d, the aggregated cost is:

[0030]

[0031] wherein, C((u,v),(u,v) d ,p) is the original cost of the pixel (u, v) and disparity (u, v) d on the path p.

[0032] Further, in the disparity calculation module, initial disparity selection is performed, and the disparity that minimizes the aggregated cost is selected as the initial disparity:

[0033]

[0034] Then, multi-modal cost fusion disparity determination is performed, and the weighted results of the visible light, multispectral and structured light cost bodies are fused to obtain the final disparity:

[0035] C total (u,v,d)=w rgb ·C rgb ((u,v),(u,v) d )+w nir ·C nir ((u,v),(u,v) d )+w struct ·C struct ((u,v),(u,v) d )

[0036]

[0037] wherein w rgb ,w nir ,w struct are confidence weights for each modality.

[0038] Further, in the depth conversion module, the relationship between the disparity d and the depth z is expressed as:

[0039]

[0040] wherein f is the camera focal length, and b is the binocular camera baseline distance;

[0041] Then the depth value z(u, v) of each pixel is combined into a depth map D(u, v).

[0042] Further, in the point cloud generation module, first, point cloud generation under the camera coordinate system is performed:

[0043] According to the depth map and the camera intrinsic matrix K, the pixel coordinates (u, v) are converted into three-dimensional points (X, Y, Z) under the camera coordinate system:

[0044]

[0045] wherein the camera intrinsic matrix K is:

[0046]

[0047] K -1 is its inverse matrix; after collecting binocular images of multiple views, respectively performing the above processing to obtain point clouds of each view, through feature matching and registration algorithm, the point clouds of different views are aligned to the same coordinate system, then merged to obtain a complete three-dimensional point cloud model.

[0048] Further, the data acquisition layer continuously collects left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine through the binocular camera, multi-spectral camera, structured light transmitter and inertial measurement unit device on the mine inspection vehicle.

[0049] The data preprocessing layer includes a binocular camera data preprocessing module, a multi-spectral camera data preprocessing module, a structured light transmitter data preprocessing module and an IMU data preprocessing module, which respectively preprocess left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data.

[0050] Further, the multi-dimensional visual presentation can be performed in the visual display module, including overall panoramic display, hierarchical visual display, and dynamic data visual display; the visual display module can perform information labeling and querying, including key information labeling and interactive querying; the visual display module can also perform historical data comparison and playback, allowing the user to select three-dimensional reconstruction models at different time points for comparison to observe the changes of the mine at different stages; the data playback is supported for real-time collected data and model change process;

[0051] In the model control module, at least the scaling and focusing of the local model can be performed; the model control module can also perform model cutting and profile analysis, including arbitrary cutting and profile analysis operations.

[0052] On the other hand, a method performed by the aforementioned mine three-dimensional reconstruction system based on the improved semi-global matching algorithm is also proposed, which comprises the following steps:

[0053] S1: Multi-modal data acquisition based on the inspection vehicle and the binocular camera, the multi-spectral camera, the structured light transmitter and the IMU;

[0054] S2: Different ways are adopted for data preprocessing for different modal data;

[0055] S3: According to the preprocessed data, the disparity cost corresponding to the left and right stereo image pairs, the multi-band radiation image and the structured light encoding image is calculated respectively

[0056] S4: Depth conversion and point cloud generation are performed, and the generated point cloud is optimized to obtain a three-dimensional reconstruction result.

[0057] The beneficial effects of the present application are as follows:

[0058] The present application significantly improves the accuracy and robustness of mine three-dimensional reconstruction through multi-sensor fusion and improved algorithm design. In the complex and variable mine environment, the problems of uneven illumination, texture loss and occlusion can be effectively overcome, and high-precision three-dimensional reconstruction can be realized.

[0059] The present application significantly improves the accuracy of three-dimensional reconstruction by using multi-modal data fusion and improved semi-global matching algorithm, and can accurately restore the complex structure and details of the mine.

[0060] The present application can adapt to complex environments such as uneven illumination and texture loss in the mine by dynamically adjusting the path weight and multi-modal cost fusion, ensuring the stability and reliability of the reconstruction result.

[0061] The present application adopts integrated design, realizes synchronous acquisition and processing of multi-modal data, improves the reconstruction efficiency, and meets the needs of real-time monitoring and rapid response.

[0062] The visual display module of the present application provides a multi-dimensional, interactive three-dimensional reconstruction result display method, which facilitates the user to intuitively understand the mine structure, and supports local model adjustment and information query, thereby improving the user experience and work efficiency.

[0063] Other advantages, objects, and features of the application will be set forth in part in the following specification taken in conjunction with the accompanying drawings, and in part will become apparent to those skilled in the art from a consideration of the following specification and drawings. The objects and other advantages of the application will be realized and attained by means of the instrumentalities and combinations pointed out in the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0065] Fig. 1 The figure is a schematic diagram of the architecture of the mine three-dimensional reconstruction system based on the improved semi-global matching algorithm under the embodiment of the present application.

[0066] Fig. 2 The figure is a schematic diagram of the process of the mine three-dimensional reconstruction method based on the improved semi-global matching algorithm under the embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described below through specific and concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and each detail in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the figures provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0068] The figures are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation on the present application; in order to better illustrate the embodiments of the present application, some components in the figures may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable for those skilled in the art that some known structures and their descriptions in the figures may be omitted.

[0069] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back", etc. are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0070] Please refer to Figs. 1-2 , a mine three-dimensional reconstruction system and method based on an improved semi-global matching algorithm.

[0071] Embodiment 1

[0072] The present embodiment 1 first introduces in detail a mine three-dimensional reconstruction system based on an improved semi-global matching algorithm.

[0073] In the present embodiment, the system adopts a framework combining multi-sensor fusion and improved SGM algorithm, as shown in Fig. 1 , which includes a data acquisition layer, a preprocessing layer, a three-dimensional reconstruction layer and an application layer, wherein the data acquisition layer sets up mine inspection vehicles, and each mine inspection vehicle is provided with a binocular camera, a multispectral camera, a structured light transmitter and an inertial measurement unit IMU.

[0074] The data acquisition layer continuously acquires left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine through the above-mentioned devices;

[0075] The data preprocessing layer respectively performs corresponding preprocessing operations on the acquired left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data;

[0076] The three-dimensional reconstruction layer dynamically adjusts the path weight according to the IMU data and aggregates the cost, then fuses the multi-modal cost to determine the final disparity, and then calculates the depth map based on the disparity and the camera parameters, and then converts the depth map into point cloud in the camera coordinate system, and finally performs denoising, hole filling and surface reconstruction to obtain the three-dimensional reconstruction result;

[0077] The application layer sets up a visual display module and a model control module for the operating user, the visual display module displays the three-dimensional reconstruction result; the model control module can perform local adjustment of the model.

[0078] In the data acquisition work of the mine three-dimensional reconstruction, the data acquisition layer equipment of the embodiment is reasonably configured and cooperates in order. Specifically, the binocular camera selects an explosion-proof industrial binocular camera, such as Stereolabs ZED 2i, which is installed on the top of the mine inspection vehicle through a support rod. At the same time, the multispectral camera and the binocular camera are rigidly connected at the top of the binocular camera, the structured light projector is arranged at the middle position of the binocular camera, and the IMU is rigidly connected with the shell of the binocular camera, so as to ensure the stability of the relative positions among the devices. The baseline distance of the binocular camera is set to 15-30 cm, the pitch angle is adjustable in the range of ±30°, and the binocular camera synchronously collects RGB images with a resolution of 1280×720 at a frame rate of 10 fps during operation. The exposure time can be automatically adjusted in the range of 1-100 ms according to the lighting conditions. The stereo image pair can provide a basis for subsequent SGM algorithm for parallax calculation and initial depth estimation.

[0079] The multispectral camera is rigidly connected with the binocular camera, for example, a FLIR Blackfly S BFS-U3-51S5C model is adopted, and a 400-1700 nm filter wheel is configured. When facing the texture missing area, it will switch to the near-infrared wave band (900-1000 nm), and the visible light image is collected at a frequency of 5 Hz, and the multi-band radiation image (5-8 bands) can enhance the material characteristics of the rock wall, equipment and other materials, and assist the matching work of the weak texture area.

[0080] The structured light projector, such as Laser Components LD-808-1000G, is installed at the middle position of the binocular camera, with a projection angle of 60°×45°. When the illumination is lower than 50 lux, it will be activated to generate a pseudo-random speckle pattern with a wavelength of 808 nm. The structured light coding image collected can solve the problem of absolute scale ambiguity and improve the matching accuracy in dark areas.

[0081] The IMU is directly fixed to the shell of the binocular camera, such as TDK ICM-42688-P model, which shares a rigid reference with the camera and outputs angular velocity and acceleration data at a high frequency of 1000 Hz. These six-degree-of-freedom pose data can provide motion priors for subsequent work, such as occlusion compensation and time synchronization. In terms of cooperative work, all devices are synchronized through PTPv2 protocol, with a trigger error control within 1 ms. When the acquisition vehicle travels at a constant speed of 0.5 m / s, the overlap rate of adjacent frames is not less than 60%, thereby ensuring the quality and integrity of data acquisition.

[0082] In the embodiment, the data preprocessing layer includes a binocular camera data preprocessing module, a multispectral camera data preprocessing module, a structured light projector data preprocessing module, and an IMU data preprocessing module.

[0083] For binocular camera data, there are uneven illumination, dynamic blur, low texture area matching difficulty and other situations. The binocular camera data preprocessing module first balances the contrast of light and dark areas through adaptive histogram equalization (CLAHE) block processing, and uses a nonlinear brightness compression algorithm to process the overexposed area to achieve image enhancement; combined with the IMU angular velocity data to estimate the motion blur kernel, use Wiener filter deconvolution to remove blur, and segment and optimize the ROI of dynamic object area, and finally perform stereo rectification based on the calibration parameters to ensure that the left and right image row alignment error meets the standard.

[0084] The multispectral camera data has problems of registration error between wavebands and noise interference. The multispectral camera data preprocessing module uses dark current subtraction and flat field correction for radiation correction, and normalizes the radiation of each waveband. Based on SIFT feature matching, sub-pixel level waveband alignment is realized, LoG operator texture enhancement is used for near-infrared waveband, and visible light and near-infrared waveband are fused through PCA to generate a 4-channel enhanced image.

[0085] The structured light data is affected by dust scattering and uneven projection. A U-Net structured speckle decoding network based on deep learning is used to resist dust interference and mark invalid projection areas. The decoded displacement is converted into an initial disparity map as the input cost of the SGM algorithm, and combined with the IMU pose data to remove false matching points. The IMU data has problems of zero drift and cumulative error. Mahony complementary filtering is used to fuse accelerometer and gyroscope data and output stable attitude, and the accelerometer data is filtered by moving average. According to the high-frequency pose data of the IMU, the binocular image is pixel-level motion compensated, and the IMU data and the image frame are strictly synchronized through bidirectional timestamp interpolation. In addition, joint preprocessing of multiple data sources is required, and a unified space-time coordinate system is established to map multispectral data, structured light depth, and IMU pose to binocular camera frames. The invalid area of structured light and the disparity void are fused to generate an occlusion mask image. The IMU motion estimation and optical flow results are compared and identified to identify and mark moving objects as temporary occlusion areas.

[0086] In this embodiment, the three-dimensional reconstruction layer includes a cost calculation module, a cost aggregation module, a disparity calculation module, a depth conversion module, a point cloud generation module, and a point cloud optimization module. The cost calculation module calculates the corresponding cost under each modality data. The cost aggregation module dynamically adjusts the path weight according to the IMU data and aggregates the cost. The disparity calculation module fuses multiple modal costs to determine the final disparity. The depth conversion module calculates the depth map based on the disparity and camera parameters. The point cloud generation module converts the depth map into a point cloud in the camera coordinate system. The point cloud optimization module performs denoising, hole filling and surface reconstruction. These modules work together to achieve high-precision three-dimensional reconstruction of complex scenes such as mines.

[0087] In the cost calculation module, first, the multi-modal cost calculation is improved, including:

[0088] Visible light cost: Using an improved Census transform + gradient fusion:

[0089] C rgb ((u,v),(u,v) d )=λ1·C census ((u,v),(u,v) d )+λ2·C grad ((u,v),(u,v) d )

[0090] The Census window has been adjusted from 9x7 to a dynamic window (5x5 to 15x15), and the Scharr operator has been changed to enhance edge response based on texture richness adaptive gradient calculation. rgb ((u,v),(u,v) d This represents the pixel (u,v) and its corresponding pixel (u,v) at a displacement of d in a visible light image. d The cost function between the Census transform cost and the gradient fusion cost. λ1 and λ2 are weighting coefficients used to balance the relative importance of the Census transform cost and the gradient fusion cost. census ((u,v),(u,v) d The ) represents the cost of the Census transform, calculated by comparing pixel (u,v) with (u,v). d The difference in pixel intensity order within the neighborhood window is used for calculation. grad ((u,v),(u,v) d The gradient fusion cost is calculated using the Scharr operator to compute the values ​​of pixels (u,v) and (u,v). d The gradient difference between them enhances the edge response.

[0091] Multispectral cost: Increased inter-band differences in the near-infrared (NIR) band.

[0092]

[0093] Among them, C nir ((u,v),(u,v) d The symbol () represents the pixel (u,v) and its corresponding pixel (u,v) with displacement d in the near-infrared (NIR) band. d The cost function between them. and Let (u,v) and (u,v) represent pixels (u,v) in the left and right images, respectively. d Intensity value in the near-infrared band. ||·||1 represents the L1 norm.

[0094] Structured light cost: Using the decoded speckle displacement as a depth prior constraint:

[0095]

[0096] C struct ((u,v),(u,v) d ) represents the structured light cost function, which uses the decoded speckle displacement as a depth prior constraint. β is the weight coefficient, which is used to adjust the importance of the structured light cost in the overall cost function; d struct represents the speckle displacement decoded by the structured light. σ 2 is the variance parameter, which controls the smoothness of the exponential function and affects the calculation of the structured light cost.

[0097] In the cost aggregation module, first, dynamic path weight adjustment is performed, and on the basis of 8-path aggregation, the weight is adjusted according to the IMU motion data, and the formula is represented as:

[0098]

[0099] wherein, W base is the basic path weight, ΔW high_dyn is the path weight increment in the high dynamic scene, and ΔW cross_band is the cross-band path weight increment in the weak texture area, wherein, in the embodiment, the high dynamic scene refers to a scene with an angular velocity greater than 0.3 rad / s.

[0100] Then, for each pixel position (u, v) and disparity d, the aggregated cost is:

[0101]

[0102] wherein, C((u,v),(u,v) d ,p) is the original cost of the pixel (u, v) and disparity (u, v) d on the path p;

[0103] In the disparity calculation module, first, the initial disparity selection is performed, and the disparity that makes the aggregated cost minimum is selected as the initial disparity:

[0104]

[0105] Then, the multi-modal cost fusion disparity determination is performed, and the weighted results of the visible light, multispectral and structured light cost bodies are fused to obtain the final disparity:

[0106] C total (u,v,d)=w rgb ·C rgb ((u,v),(u,v) d )+w nir ·C nir ((u,v),(u,v)d )+w struct ·C struct ((u,v),(u,v) d )

[0107]

[0108] where w rgb ,w nir ,w struct are confidence weights for each modality.

[0109] In the depth conversion module, the relationship between the disparity d and the depth z is expressed as:

[0110]

[0111] where f is the camera focal length (in pixel units) and b is the baseline distance of the binocular camera (in physical units, such as centimeters).

[0112] Then the depth value z(u,v) of each pixel is combined into a depth map D(u,v):

[0113] D(u,v)=z(u,v)

[0114] In the point cloud generation module, it includes:

[0115] Point cloud generation in the camera coordinate system: according to the depth map and the camera intrinsic matrix K, the pixel coordinates (u,v) are converted into three-dimensional points (X,Y,Z) in the camera coordinate system:

[0116]

[0117] where the camera intrinsic matrix K is:

[0118]

[0119] K -1 is its inverse matrix.

[0120] After collecting binocular images from multiple viewpoints and processing each viewpoint to obtain point clouds, the point clouds from different viewpoints are aligned to the same coordinate system through feature matching and registration algorithms (such as ICP algorithm), and then merged to obtain a more complete three-dimensional point cloud model.

[0121] In the point cloud optimization module, it includes:

[0122] Noise removal: statistical filtering, radius filtering and other methods are used to remove noise points. For example, statistical filtering calculates the average distance of each point to its neighborhood points, and removes points with a distance exceeding a certain threshold as noise points.

[0123] Hole filling: There may be some hole regions in the depth map, resulting in missing parts in the corresponding point cloud. Methods based on multi-source data fusion can be used for hole filling, such as the multispectral texture-guided repair mentioned earlier, structure light data interpolation, and temporal frame propagation.

[0124] Surface reconstruction: In order to get a smoother and more realistic three-dimensional model, surface reconstruction can be performed on the optimized point cloud. Common surface reconstruction methods include Poisson reconstruction, greedy projection triangulation, etc. For example, Poisson reconstruction converts point cloud data into a continuous surface model by solving a Poisson equation.

[0125] According to the reconstructed three-dimensional model, visualization and model control are performed in the application layer.

[0126] Specifically, in the visualization module, multi-dimensional visualization can be performed, including:

[0127] Overall panoramic display: The visualization module can present the three-dimensional reconstruction results of the mine in panoramic mode, allowing users to intuitively see the overall situation of the entire mine, including the direction and layout of the roadway, and the connection of each area. Through scaling, rotating, and other operations, users can observe the mine from different angles, as if they were on-site.

[0128] Layered visualization: In order to more clearly display different structures and information inside the mine, the module supports layered visualization. For example, geological layers, roadway layers, and equipment layers can be displayed separately, and users can select to view the superimposed information of one or more layers as needed. In the geological layer, different rock layer distribution, fault location, etc. can be highlighted; in the roadway layer, the cross-sectional shape of the roadway, support conditions, etc. are displayed; in the equipment layer, the specific location and running state of various equipment are displayed.

[0129] Dynamic data visualization: In addition to static three-dimensional model display, the module can superimpose real-time collected mine environment parameters (such as temperature, humidity, gas concentration, etc.) and equipment operation data (such as equipment speed, power, etc.) in the form of dynamic charts or heat maps on the three-dimensional model. In this way, users can real-time understand the environmental conditions and equipment operation conditions at each location in the mine, and timely discover abnormal data points.

[0130] In the visualization module, information labeling and querying can also be performed, including:

[0131] Key information labeling: Important geological structures, equipment, safety areas, etc. are labeled on the three-dimensional model. Labeling information can include name, specification, parameter, etc. detailed content. For example, for a coal mining machine, its model, production capacity, current working state, etc. information is labeled; for a fault, its trend, dip angle, throw, etc. geological parameters are labeled.

[0132] Interactive query: Users can click on the annotation points or specific areas on the three-dimensional model to pop up detailed information windows and query related detailed data and materials. At the same time, the module supports keyword search function, users can input geological structure name, equipment number and other keywords, quickly locate to the corresponding position and view related information.

[0133] The visualization display module of the embodiment can also perform historical data comparison and playback:

[0134] Historical model comparison: The visualization display module allows users to select three-dimensional reconstruction models at different time points for comparison to observe the changes of the mine at different stages. For example, compare the changes of geological structure before and after mining, the deformation of the roadway, etc. Through color coding or difference marking, etc., the differences between the models are highlighted to help users analyze the dynamic change process of the mine.

[0135] Data playback: For real-time collected data and model change process, the module supports data playback function. Users can select a specific time period to playback the mine environment parameters, equipment running status and three-dimensional model changes in the form of animation, so as to more deeply analyze the operation rules and potential problems of the mine.

[0136] In the model control module, it can at least scale and focus on the local model. The model control module allows users to perform fine scaling operations on the local area of the three-dimensional model. For example, when a user wants to view the details of a geological structure or equipment in detail, the user can zoom in on the area through the mouse wheel or specific zoom buttons to observe the features more clearly. Focusing function: Users can select a specific target (such as a device, a geological anomaly point), the module will automatically focus on the target at the center of the view and zoom in appropriately, while adjusting the viewing angle to allow users to view the details of the target from all angles.

[0137] In the model control module, model cutting and profile analysis can also be performed:

[0138] Arbitrary cutting: In order to deeply understand the structure and situation inside the mine, the module supports arbitrary direction cutting operation on the three-dimensional model. Users can draw a cutting plane or select a preset cutting direction (such as horizontal, vertical, oblique, etc.) to cut the model into different parts and view the detailed information inside the model, such as the support structure inside the roadway, the internal features of the geological layer, etc.

[0139] Section analysis: After cutting the model, the module can generate the corresponding section drawing and analyze the section drawing. For example, mark the thickness, dip angle and other information of the geological layer on the section drawing, calculate the cross-sectional area of the roadway and other parameters, and provide more detailed data support for geological exploration and mining planning.

[0140] Embodiment 2

[0141] The embodiment provides an improved semi-global matching algorithm-based mine three-dimensional reconstruction method based on the improved semi-global matching algorithm-based mine three-dimensional reconstruction system. Fig. 2 As shown in the figure, it comprises:

[0142] S1: Multi-modal data acquisition based on inspection vehicle and binocular camera, multi-spectral camera, structured light transmitter and IMU;

[0143] In the mine three-dimensional reconstruction work, the first step is data acquisition. The specific operation is to equip the mine inspection vehicle with a binocular camera, a multi-spectral camera, a structured light transmitter and an IMU (inertial measurement unit). During the travel of the inspection vehicle in the mine, these devices work continuously, respectively collecting left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine. During the acquisition process, the stability of the devices should be ensured to ensure that the collected data is complete and accurate, and the data is preliminarily time-stamped to lay a foundation for subsequent data synchronization processing.

[0144] S2: Different ways of data preprocessing are adopted for different modal data;

[0145] The collected various data needs to be preprocessed before being used for subsequent three-dimensional reconstruction.

[0146] For left and right stereo image pairs, first, denoising algorithms such as Gaussian filtering and median filtering are used for denoising processing to remove noise interference in the image, and then the binocular camera is calibrated to obtain the internal and external parameters of the camera. These parameters are used to correct the image so that the corresponding epipolar lines are parallel, facilitating subsequent stereo matching.

[0147] Multi-band radiation images need to be radiometrically corrected to eliminate the influence of factors such as non-uniform sensor response and atmospheric attenuation on image radiation values, so as to truly reflect the radiation characteristics of ground objects. Then, according to the requirements of mine three-dimensional reconstruction, methods such as principal component analysis (PCA) and minimum noise separation (MNF) are used for band selection and fusion to highlight useful geological information.

[0148] The structured light encoding image needs to be decoded first to extract the encoding information for subsequent depth calculation. Then, filtering is performed to remove noise and interference, and image enhancement algorithms such as histogram equalization are used to enhance the contrast of the image and improve the recognition accuracy of the structured light.

[0149] The six-degree-of-freedom pose data needs to be filtered and denoised, and algorithms such as Kalman filtering and complementary filtering can be used to remove the influence of sensor noise and errors. At the same time, the pre-processed pose data is time-synchronized with the image data to ensure the consistency of the data in subsequent processing.

[0150] S3: Calculate the disparity cost corresponding to the left and right stereo image pair, multi-band radiation image and structured light encoding image according to the pre-processed data respectively;

[0151] After completing the data preprocessing, enter the three-dimensional reconstruction phase. First, obtain the motion information of the mine inspection vehicle from the IMU data, including attitude changes and moving speeds, etc. Use these information to dynamically adjust the path weight in the semi-global matching (SGM) algorithm, so that the algorithm adapts to the complex changes of the mine environment. For example, increase the path weight when the motion speed is fast to reduce the calculation amount, and adjust the path direction weight when the attitude changes greatly to improve the matching accuracy. After adjusting the path weight, aggregate the cost on different paths to obtain the initial disparity cost matrix.

[0152] For the left and right stereo image pair, a cost calculation method based on gray similarity is used, the multi-band radiation image combines the radiation characteristics for cost calculation, and the structured light encoding image calculates the cost according to the decoding information. Then a weighted fusion method is used to fuse the disparity costs of different modalities, and different weights are assigned according to the reliability and importance of different modal data in the mine environment. Through the fused cost matrix, the optimal path search method of the semi-global matching algorithm is used to determine the final disparity map.

[0153] S4: Depth conversion and point cloud generation are performed, and the generated point cloud is optimized to obtain the three-dimensional reconstruction result.

[0154] According to the baseline distance, focal length and other parameters of the binocular camera and the final disparity map, the depth value of each pixel point is calculated using the triangulation principle to obtain the depth map, and the depth map is filtered to remove outliers and noise. Then, according to the internal and external parameters of the camera, each pixel point in the depth map is converted to a three-dimensional coordinate point in the camera coordinate system to obtain point cloud data. The point cloud data is denoised to remove isolated points and error points, and statistical filtering, radius filtering and other methods can be used.

[0155] Finally, the point cloud data is further denoised by using methods such as voxel grid filtering and moving least square method, so that the point cloud data is smoother and more accurate. For the hollow area in the point cloud data, an interpolation-based method is used for filling, such as radial basis function interpolation, Kriging interpolation, etc. The point cloud data after denoising and hole filling is converted into a three-dimensional surface model by using Poisson reconstruction, Delaunay triangulation and other algorithms, and the final mine three-dimensional reconstruction result is obtained.

[0156] Example 3

[0157] This example introduces the practical application according to the system of example 1 and the method of example 2.

[0158] In this example, the following contents are included in detail:

[0159] Visible light cost calculation: assuming that the neighborhood window of pixel p is 5x5, the intensity order difference of p and the corresponding pixel with displacement d=3 is compared, and C census =12 (difference bits).

[0160] The gradient difference is calculated by using Scharr operator, and C grad =8.

[0161] Substitute the formula: C rgb (p,3) = 0.6x12 + 0.4x8 = 10.4.

[0162] Near-infrared band cost calculation: assuming that I left (p) = 150, I right (p+3) = 140, then:

[0163] C nir (p,3) = |150-140| = 10.

[0164] Structured light cost calculation: β is a weight coefficient (such as 0.8), σ is a variance parameter (such as 1.5), assuming that d struct =3, and the current disparity d=3, then:

[0165] C struct (p,3) = 0.8·exp(-2x1.52(3-3) 2 ) = 0.8.

[0166] Objective: dynamically adjust path weight according to IMU data and aggregate cost

[0167] In the path weight adjustment formula, W base is the basic path weight (such as 0.25), ΔW dynamic is the high dynamic scene increment (such as +0.1 when the angular velocity is greater than 0.3 rad / s), and ΔWt extureIncrement weak texture area (e.g. +0.05). High dynamic scene (angular velocity = 0.4 rad / s) and weak texture area:

[0168] W path = 0.25 + 0.1 + 0.05 = 0.4

[0169] C agg (p, 3) = 8 * 0.4 * 10 = 32

[0170] In the process of multi-modal cost fusion, w rgb = 0.5, w nir = 0.3, w struct = 0.2.

[0171] Calculate the cost of each modality at d = 3: C rgb = 10.4, C nir = 10, C struct = 0.8. Weighted sum:

[0172] 0.5 * 10.4 + 0.3 * 10 + 0.2 * 0.8 = 8.36

[0173] In contrast to other disparities (e.g. weighted sum = 8.5 at d = 4), the final disparity d final = 3.

[0174] Convert the depth map to a point cloud in the camera coordinate system, where f is the camera focal length (e.g. 500 pixels), B is the baseline distance (e.g. 0.2 m), and d is the disparity (e.g. 3). Calculate the depth: Z = 500 * 0.2 / 3 ≈ 33.33 m

[0175] Point cloud coordinate calculation:

[0176] Convert pixel coordinates (u, v) to three-dimensional points (X, Y, Z) in the camera coordinate system:

[0177] X = f (u - c x ) * Z, Y = f (v - c y ) * Z

[0178] Assuming the principal point (c x , c y ) = (320, 240) and pixel coordinates (u, v) = (330, 250), then: X = 500 * (330 - 320) * 33.33 ≈ 0.67 m, Y = 500 * (250 - 240) * 33.33 ≈ 0.67 m.

[0179] This embodiment verifies the robustness and precision of the improved semi-global matching algorithm in the complex environment of a mine through specific numerical calculations, providing an operable quantitative method for three-dimensional reconstruction.

[0180] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A mine three-dimensional reconstruction system based on an improved semi-global matching algorithm, characterized in that: The system comprises a data acquisition layer, a preprocessing layer, a three-dimensional reconstruction layer and an application layer, wherein the data acquisition layer is provided with mine inspection vehicles, each of which is provided with a binocular camera, a multispectral camera, a structured light transmitter and an inertial measurement unit (IMU); The data acquisition layer continuously acquires left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine through the above-mentioned devices; The data preprocessing layer performs corresponding preprocessing operations on the acquired left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data; The three-dimensional reconstruction layer dynamically adjusts path weights and aggregates costs according to the IMU data, fuses multi-modal costs to determine a final disparity, then calculates a depth map based on the disparity and camera parameters, converts the depth map into a point cloud in the camera coordinate system, and finally performs denoising, hole filling and surface reconstruction to obtain a three-dimensional reconstruction result; The application layer is provided with a visual display module and a model control module for an operating user, the visual display module displays the three-dimensional reconstruction result, and the model control module can perform local adjustment of the model.

2. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 1, characterized in that: The three-dimensional reconstruction layer comprises a cost calculation module, a cost aggregation module, a disparity calculation module, a depth conversion module, a point cloud generation module and a point cloud optimization module, the cost calculation module calculates corresponding costs under each modal data, the cost aggregation module dynamically adjusts path weights and aggregates costs according to the IMU data, the disparity calculation module fuses multi-modal costs to determine a final disparity, the depth conversion module calculates a depth map based on the disparity and camera parameters, the point cloud generation module converts the depth map into a point cloud in the camera coordinate system, and the point cloud optimization module performs denoising, hole filling and surface reconstruction.

3. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 2, characterized in that: In the cost calculation module, multi-modal cost calculation is performed: An improved Census transform and gradient fusion method is used to calculate the visible light cost: C rgb ((u,v),(u,v) d ) = λ1 · C census ((u,v),(u,v) d ) + λ2 · C grad ((u,v),(u,v) d ) wherein C rgb ((u, v), (u, v) d ) denotes a cost function between pixel (u, v) and its corresponding pixel (u, v) d with displacement d in the visible light image; λ1, λ2 are weight coefficients to balance the relative importance between the Census transform cost and the gradient fusion cost; C census ((u, v), (u, v) d ) denotes the Census transform cost, which is calculated by comparing the order difference of pixel intensity within the neighborhood window of pixel (u, v) and (u, v) d ; C grad ((u, v), (u, v) d ) is the gradient fusion cost, which calculates the gradient difference between pixel (u, v) and (u, v) d using the Scharr operator. The inter-band difference cost of the near-infrared band is calculated: where C nir (u,v) and (u,v) d denote the intensity values of pixel (u,v) and (u,v) d in the near-infrared band, respectively; and ||·||1 denotes the L1 norm. and denote the intensity values of pixel (u,v) and (u,v) d in the left and right images, respectively; and ||·||1 denotes the L1 norm. The structured light cost is calculated, and the decoded speckle displacement is used as a depth prior constraint: C struct ((u,v),(u,v) d ) represents the structured light cost function, which uses the decoded speckle displacement as a depth prior constraint; β is the weight coefficient, which is used to adjust the importance of the structured light cost in the overall cost function; d struct represents the speckle displacement decoded by the structured light; σ 2 is the variance parameter, which controls the smoothness of the exponential function and affects the calculation of the structured light cost.

4. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 3, characterized in that: In the cost aggregation module, dynamic path weight adjustment is performed, and the weight is adjusted according to the motion data of the inertial measurement unit on the basis of aggregation of a plurality of paths, and the formula is represented as: wherein, W base is a basic path weight, ΔW high_dyn is a path weight increment under a high dynamic scene, ΔW cross_band is a cross-band path weight increment under a weak texture region, wherein the high dynamic scene refers to a scene in which an angular velocity is greater than a preset value. Then, for each pixel position (u, v) and disparity d, the aggregated cost is: where C((u, v), (u, v) d , p) is the original cost of pixel (u, v) and disparity (u, v) d on path p.

5. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 4, characterized in that: In the disparity calculation module, initial disparity selection is performed, and the disparity that minimizes the aggregated cost is selected as the initial disparity: Then, multi-modal cost fusion disparity determination is performed, and the weighted results of the visible light, multispectral and structured light cost bodies are fused to obtain the final disparity: C total (u,v,d) = w rgb ·C rgb ((u,v),(u,v) d ) + w nir ·C nir ((u,v),(u,v) d ) + w struct ·C struct ((u,v),(u,v) d ) where w rgb ,w nir ,w struct are confidence weights for each modality.

6. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 5, characterized in that: In the depth conversion module, the relationship between the disparity d and the depth z is represented as: Wherein, f is the camera focal length, and b is the binocular camera baseline distance; Then, the depth value z (u, v) of each pixel is combined into a depth map D (u, v).

7. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 6, characterized in that: In the point cloud generation module, first, point cloud generation in the camera coordinate system is performed: According to the depth map and the camera intrinsic matrix K, the pixel coordinates (u, v) are converted into three-dimensional points (X, Y, Z) in the camera coordinate system: Wherein, the camera intrinsic matrix K is: K -1 is its inverse matrix; after collecting binocular images of multiple perspectives, respectively performing the above processing to obtain point clouds of each perspective, aligning the point clouds of different perspectives to the same coordinate system through feature matching and registration algorithm, and then merging to obtain a complete three-dimensional point cloud model.

8. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 1, characterized in that: The data acquisition layer continuously collects left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data of the mine through the binocular camera, multi-spectral camera, structured light transmitter and inertial measurement unit device on the mine inspection vehicle. The data preprocessing layer includes a binocular camera data preprocessing module, a multi-spectral camera data preprocessing module, a structured light transmitter data preprocessing module and an IMU data preprocessing module, which respectively preprocess left and right stereo image pairs, multi-band radiation images, structured light encoding images and six-degree-of-freedom pose data.

9. The mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to claim 1, characterized in that: The visualization display module can perform multi-dimensional visualization, including overall panoramic display, layered visualization display and dynamic data visualization display. The visualization display module can perform information labeling and querying, including key information labeling and interactive querying. The visualization display module can also perform historical data comparison and playback, allowing users to select three-dimensional reconstruction models at different time points for comparison to observe the changes of the mine at different stages. The visualization display module supports data playback for real-time collected data and model change processes. In the model control module, at least the scaling and focusing of the local model can be performed. The model control module can also perform model cutting and profile analysis, including arbitrary cutting and profile analysis operations.

10. A method performed by the mine three-dimensional reconstruction system based on the improved semi-global matching algorithm according to any one of the preceding claims 1-9, characterized in that: The method comprises the following steps: S1: Multi-modal data acquisition based on the inspection vehicle and the binocular camera, multi-spectral camera, structured light transmitter and IMU; S2: Different ways of data preprocessing are adopted for different modal data; S3: According to the preprocessed data, the parallax cost corresponding to the left and right stereo image pairs, multi-band radiation images and structured light encoding images is calculated respectively S4: Depth conversion and point cloud generation are performed, and the generated point cloud is optimized to obtain a three-dimensional reconstruction result.

Images