Three-dimensional mine rapid modeling method and related equipment

Abstract

The invention belongs to the technical field of mine modeling, and discloses a three-dimensional mine rapid modeling method and related equipment, three-dimensional modeling is carried out on the interior of a mine based on a mobile device provided with ultrasonic scanning equipment and depth camera detection equipment, and by obtaining ultrasonic scanning data and depth camera shooting data, the three-dimensional mine rapid modeling method is obtained. According to the method, the three-dimensional space block library and the equipment model library are combined to realize rapid modeling of the mine space and the internal equipment, so that the problems of low modeling efficiency and poor precision in the prior art are solved, and three-dimensional modeling can be rapidly and accurately performed on the mine.

Description

Technical Field

[0001] The present application relates to the technical field of mine modeling, and in particular to a three-dimensional mine rapid modeling method and related equipment. Background Art

[0002] In real life, based on the needs of coal mining, it is usually necessary to detect the spatial distribution map inside the coal mine. In the process of spatial modeling of the mine, laser radar, visual sensors, etc. are often used to collect spatial information, and then the space inside the coal mine is constructed in three dimensions. However, this method has some significant problems and limitations in the actual application process.

[0003] First, most existing 3D reconstruction technologies are based on LiDAR and visual sensors. Although LiDAR can quickly obtain spatial information, its accuracy is easily affected by complex environmental factors in coal mines, such as dust, floating small particles, and water vapor. These factors interfere with the propagation and reflection of laser signals, resulting in reduced accuracy of collected data, thus affecting the final 3D reconstruction effect.

[0004] Secondly, most of the visual sensors currently used are planar two-dimensional image sensors. This type of sensor can only obtain projection information within a range, that is, a two-dimensional image, but cannot characterize the depth information of the characteristic objects in the projection part within the range. This limitation makes it difficult to obtain three-dimensional spatial information, which in turn affects the accuracy and integrity of three-dimensional reconstruction.

[0005] In addition, existing 3D reconstruction technology usually performs object surface shape fitting based on the point cloud data obtained by the sensor to achieve object modeling. Although this method can achieve 3D reconstruction, the modeling efficiency is low. In a complex mine environment, a large amount of point cloud data needs to be processed, which is not only time-consuming, but also requires high computing resources, which is not conducive to rapid modeling and real-time updating.

[0006] More importantly, the existing 3D reconstruction strategies for coal mines are mostly simple splicing of existing sensor data, but unable to deeply fuse the data. This simple data processing method leads to poor reconstruction results and cannot accurately reflect the complex structure and equipment distribution inside the mine. At the same time, this method can only construct static 3D scenes and cannot adapt to dynamic changes in the mine environment, such as equipment movement, changes in spatial structure, etc.

[0007] Therefore, how to quickly reconstruct the scene in the coal mine in three dimensions while ensuring a low error rate for local details is an urgent problem to be solved. The industry urgently needs a method that can quickly, accurately and dynamically perform three-dimensional modeling of mines to meet the needs of mine safety monitoring, equipment management and space planning.

[0008] In view of the above problems, the existing technology needs to be improved urgently. Summary of the invention

[0009] The purpose of the present application is to provide a three-dimensional mine rapid modeling method and related equipment, which can quickly and accurately perform three-dimensional modeling of the mine.

[0010] In a first aspect, the present application provides a method for rapid three-dimensional mine modeling, which performs three-dimensional modeling of the interior of a mine based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device, including the following steps: A1. Obtaining ultrasonic scanning data and depth camera shooting data collected by the mobile device while moving along the mine; A2. According to the ultrasonic scanning data, a corresponding reference three-dimensional space block is matched from a three-dimensional space block library as an optional three-dimensional space block for splicing to achieve modeling of the mine space; A3. Perform device positioning and device identification based on the depth camera data; A4. According to the equipment identification and positioning results, the corresponding equipment model is extracted from the equipment model library and placed at the corresponding position to realize the three-dimensional modeling of the interior of the mine.

[0011] Three-dimensional modeling of the interior of the mine is carried out based on a mobile device equipped with ultrasonic scanning equipment and depth camera detection equipment. By acquiring ultrasonic scanning data and depth camera shooting data, combined with a three-dimensional space block library and an equipment model library, rapid modeling of the mine space and internal equipment is achieved, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art, and being able to quickly and accurately carry out three-dimensional modeling of the mine.

[0012] Preferably, the three-dimensional mine rapid modeling method further comprises the steps of: A5. After completing the three-dimensional modeling of the interior of the mine, obtain the ultrasonic scanning data and depth camera shooting data collected in real time by the mobile device during inspection in the mine, so as to dynamically update the three-dimensional modeling results.

[0013] This method can realize the dynamic update of the mine model, timely reflect the changes in the internal environment of the mine, and maintain the accuracy and timeliness of the three-dimensional model.

[0014] Preferably, in step A1, ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the S-shaped route are obtained; In step A5, ultrasonic scanning data and depth camera shooting data collected in real time when the mobile device inspects along the S-shaped route are obtained to dynamically update the three-dimensional modeling results.

[0015] The mobile device moves along an S-shaped route to collect ultrasonic scanning data and depth camera shooting data, which can make the collected data cover the mine space more comprehensively and reduce the blind spots of data collection.

[0016] Preferably, step A2 comprises: Calibrate the position of the ultrasonic scanning data according to the position of the mobile device at the time of collecting the ultrasonic scanning data; The ultrasonic scanning data is recognized by using a pre-trained three-dimensional space block recognition model to match a corresponding reference three-dimensional space block from a three-dimensional space block library as a matching three-dimensional space block; The selected three-dimensional space blocks are spliced ​​according to the position calibration result of the ultrasonic scanning data.

[0017] Preferably, the reference three-dimensional space block includes a plurality of sub-space blocks; the three-dimensional space block library records the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block; Step A3 includes: A301. Positioning the captured device according to the depth camera data and the position of the mobile device at the time of collecting the depth camera data; A302. Determine the subspace block where each device is located according to the positioning result of each device; A303. According to the subspace block and the corresponding optional three-dimensional space block in which each device is located, the corresponding optimal recognition algorithm is obtained from the three-dimensional space block library; A304. Classify and identify each device using the corresponding optimal identification algorithm.

[0018] Preferably, the step A1 further includes the following steps: A01. Acquire multiple historical equipment data of each subspace block of each reference three-dimensional space block; the historical equipment data includes historical equipment images and equipment type annotation information; A02. For each of the subspace blocks, a plurality of different recognition algorithms are used to classify and recognize the historical equipment images in the corresponding historical equipment data, and the recognition results are compared with the corresponding equipment type labeling information to calculate the recognition accuracy of the various recognition algorithms for the subspace blocks; A03. Use the recognition algorithm with the highest recognition accuracy as the optimal recognition algorithm corresponding to the subspace block, form an optimal recognition algorithm query table and record it in the three-dimensional space block library.

[0019] Preferably, the subspace blocks are divided into important subspace blocks and unimportant subspace blocks; In step A3, only the devices located in the important subspace blocks are classified and identified; In step A4, for devices located in important subspace blocks, the corresponding device models are extracted from the device model library and placed at the corresponding positions in the corresponding optional three-dimensional space blocks; for devices located in non-important subspace blocks, a simplified model with a preset shape is placed at the corresponding positions in the corresponding optional three-dimensional space blocks.

[0020] In a second aspect, the present application provides a three-dimensional mine rapid modeling device, which performs three-dimensional modeling of the internal space of the mine based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device, including: A first acquisition module is used to acquire ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine; A space modeling module, used to match corresponding reference three-dimensional space blocks from a three-dimensional space block library according to the ultrasonic scanning data as optional three-dimensional space blocks for splicing, so as to realize modeling of the mine space; A positioning and identification module, used for performing device positioning and device identification based on the data captured by the depth camera; The equipment modeling module is used to extract the corresponding equipment model from the equipment model library according to the equipment identification results and equipment positioning results, and place it at the corresponding position in the corresponding optional three-dimensional space block to realize three-dimensional modeling of the interior of the mine.

[0021] In a third aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the three-dimensional mine rapid modeling method described above.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the three-dimensional mine rapid modeling method as described above are executed.

[0023] Beneficial effects: The three-dimensional mine rapid modeling method and related equipment provided in the present application perform three-dimensional modeling of the interior of the mine based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device. By acquiring ultrasonic scanning data and depth camera shooting data, combined with a three-dimensional space block library and an equipment model library, rapid modeling of the mine space and internal equipment is achieved, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art, and being able to quickly and accurately perform three-dimensional modeling of the mine. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a three-dimensional mine rapid modeling method provided in an embodiment of the present application.

[0025] Figure 2A schematic diagram of the structure of a three-dimensional mine rapid modeling device provided in an embodiment of the present application.

[0026] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0027] Figure 4 Schematic diagram of mine tunnels and selected three-dimensional space blocks.

[0028] Explanation of reference numerals: 1. First acquisition module; 2. Space modeling module; 3. Positioning and identification module; 4. Equipment modeling module; 301. Processor; 302. Memory; 303. Communication bus. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0031] Three-dimensional modeling of the interior of a mine is an important part of coal mining and safety management. Traditional three-dimensional reconstruction technology mainly relies on laser radar and visual sensors to obtain spatial information. However, these methods face many challenges in practical applications. Although laser radar can quickly obtain spatial information, its accuracy is easily affected by environmental factors such as dust, floating small particles and water vapor in the mine. Ordinary visual sensors are mostly two-dimensional image sensors, which can only obtain planar projection information and cannot accurately represent the depth information of objects. In addition, existing three-dimensional reconstruction technology usually requires point-by-point fitting of point cloud data, and the modeling efficiency is low. More importantly, the current three-dimensional reconstruction strategy of coal mines often simply splices sensor data and lacks deep fusion, resulting in poor reconstruction results and can only construct static three-dimensional scenes. These problems seriously restrict the accuracy, efficiency and practicality of three-dimensional modeling of the interior space of mines.

[0032] Specifically, in the underground mining area of ​​a large coal mine, the mine environment is complex and changeable, including multiple branch passages, coal mining faces and various types of equipment. Traditional 3D modeling methods face severe challenges in this environment. For example, when using LiDAR scanning, due to the large amount of dust and water vapor generated during the coal mining process, the laser signal is prone to scattering and attenuation, resulting in a large amount of noise and voids in the acquired point cloud data. The two-dimensional images taken with ordinary cameras cannot accurately reflect the degree of curvature of the tunnel and the spatial position relationship of the equipment. In addition, the lighting conditions in the mine are usually poor, which further reduces the image quality. In the data processing stage, due to the lack of effective data fusion methods, the data collected by different sensors are difficult to coordinate, resulting in a large number of errors and incoherent areas in the reconstructed model. More importantly, it is difficult for traditional methods to update the model in real time and cannot reflect the dynamic changes in the mine environment, such as equipment movement or changes in tunnel morphology.

[0033] If these technical problems cannot be effectively solved, it will have a serious impact on mine management and safe production. First, inaccurate 3D models may lead to misjudgment of the spatial structure of the mine, affecting mining planning and safety assessment. Secondly, inefficient modeling processes will prolong the time for mine detection and assessment and reduce production efficiency. Furthermore, static models cannot reflect real-time changes in the mine environment, which may lead to the failure to timely discover and deal with safety hazards. For example, in an emergency, if the 3D model that rescuers rely on deviates greatly from the actual situation, it may seriously affect the rescue effect. In addition, inaccurate equipment positioning information may cause the failure of remote control and automation systems, increasing production risks. Therefore, developing a new method that can quickly, accurately and dynamically perform 3D modeling inside mines is of great significance for improving mine management efficiency, ensuring production safety and optimizing resource utilization.

[0034] To do this, please refer to Figure 1 The present application provides a method for rapid three-dimensional mine modeling, which is based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device to perform three-dimensional modeling of the interior of the mine, including the steps of: A1. Obtaining ultrasonic scanning data and depth camera shooting data collected by the mobile device while moving along the mine; A2. According to the ultrasonic scanning data, the corresponding reference three-dimensional space blocks are matched from the three-dimensional space block library as the optional three-dimensional space blocks for splicing to achieve the modeling of the mine space; A3. Device positioning and device identification based on the depth camera data; A4. According to the equipment identification results and equipment positioning results, the corresponding equipment model is extracted from the equipment model library and placed in the corresponding position (i.e. the position corresponding to the equipment positioning result) to realize the three-dimensional modeling of the interior of the mine.

[0035] Three-dimensional modeling of the interior of the mine is carried out based on a mobile device equipped with ultrasonic scanning equipment and depth camera detection equipment. By acquiring ultrasonic scanning data and depth camera shooting data, combined with a three-dimensional space block library and an equipment model library, rapid modeling of the mine space and internal equipment is achieved, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art, and being able to quickly and accurately carry out three-dimensional modeling of the mine.

[0036] The core innovation of this application is to combine ultrasonic scanning and depth camera technology, and use preset 3D space blocks and equipment model libraries to achieve rapid 3D modeling inside the mine. This method overcomes the accuracy problems of traditional LiDAR in complex environments, while providing richer 3D information. The preset 3D space blocks and equipment model library greatly improve the modeling efficiency and avoid the tedious process of point-by-point reconstruction.

[0037] The working principle of this application can be described in detail as follows: First, the mobile device moves in the mine, and collects data using ultrasonic scanning equipment and depth camera detection equipment. The ultrasonic scanning equipment acquires spatial information of the surrounding environment by emitting ultrasonic waves and receiving reflected waves, forming ultrasonic scanning data. The depth camera detection equipment captures the three-dimensional information of the scene through structured light or time-of-flight technology, generating depth camera shooting data.

[0038] Next, spatial modeling is performed based on the ultrasonic scanning data. The system will match the acquired ultrasonic scanning data with the reference 3D spatial blocks pre-stored in the 3D spatial block library. After matching the most similar reference 3D spatial block, it will be used as the optional 3D spatial block. Then, the system will splice these optional 3D spatial blocks according to the spatial relationship of the ultrasonic scanning data, thereby achieving overall modeling of the mine space.

[0039] At the same time, the system uses the data captured by the depth camera to locate and identify the device. The three-dimensional information provided by the depth camera enables the spatial position of the device to be accurately determined. Through image processing and machine learning algorithms, the system can identify various types of devices in the captured image.

[0040] Finally, based on the equipment identification and positioning results, the system extracts the corresponding 3D equipment models from the equipment model library. These models will be placed in the corresponding positions in the previously established mine space model, thus completing the detailed 3D modeling of the mine interior.

[0041] This method chooses ultrasonic scanning instead of LiDAR because ultrasonic waves have advantages in penetrating dust and water vapor, and can obtain more accurate spatial information in complex mine environments. The introduction of depth cameras makes up for the inability of ordinary two-dimensional image sensors to obtain depth information. The preset three-dimensional space blocks and equipment model library greatly improve the modeling efficiency and avoid the tedious process of point-by-point reconstruction in traditional methods.

[0042] The mobile device may be a robot, an AGV or other wheeled or tracked mobile platforms.

[0043] The ultrasonic scanning device may be implemented by, but is not limited to, an ultrasonic sensor array.

[0044] The depth camera detection device may be implemented by, but is not limited to, a depth camera based on structured light or time-of-flight (ToF) technology.

[0045] The three-dimensional space block library stores pre-established reference three-dimensional space blocks of various typical mine space structures, and the three-dimensional space block library can be implemented by a computer storage system.

[0046] Among them, the equipment model library stores pre-established standard three-dimensional models (i.e., equipment models) of various typical mining equipment (such as for coal mines, typical mining equipment includes coal scrapers, conveyor belts, and transport vehicles, etc.). The equipment model library can be implemented using a computer storage system.

[0047] In some preferred embodiments, the three-dimensional mine rapid modeling method further includes the steps of: A5. After completing the 3D modeling of the interior of the mine, obtain the ultrasonic scanning data and depth camera shooting data collected in real time by the mobile device during inspection in the mine, and use it to dynamically update the 3D modeling results.

[0048] This method can realize the dynamic update of the mine model, timely reflect the changes in the internal environment of the mine, and maintain the accuracy and timeliness of the three-dimensional model.

[0049] Specifically, the initial three-dimensional modeling of the interior of the mine is completed through steps A1 to A4; then the mobile device conducts inspections (the inspections can be conducted according to the preset inspection route or according to actual needs), and real-time ultrasonic scanning data and depth camera shooting data are collected during the inspection process. These data contain the latest spatial information and equipment information inside the mine; then, the system compares and fuses the new data collected during the inspection with the existing three-dimensional model. If it is found that the mine environment has changed, such as the addition of new equipment, changes in spatial structure (for example, the real-time recognition result of the optional three-dimensional space block is different from the last recognition result), changes in equipment position (for example, the position deviation between the real-time positioning result of the equipment position and the last positioning result exceeds the preset deviation threshold, which can be set according to actual needs), etc., the system will update the three-dimensional model of the interior of the mine accordingly (refer to steps A2 to A4 when updating, and update the corresponding optional three-dimensional space block and / or the equipment model in the corresponding optional three-dimensional space block).

[0050] This dynamic update mechanism works well with the initial 3D modeling approach. The initial modeling provides the base model, while the dynamic update ensures the continued accuracy of the model so that managers can understand the mine status in a timely manner. This combination not only solves the problem that the static model cannot reflect environmental changes, but also improves the efficiency and accuracy of the overall modeling.

[0051] In some preferred embodiments, in step A1, ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the S-shaped route are obtained; In step A5, ultrasonic scanning data and depth camera shooting data collected in real time when the mobile device inspects along the S-shaped route are obtained to dynamically update the three-dimensional modeling results.

[0052] The mobile device moves along an S-shaped route to collect ultrasonic scanning data and depth camera shooting data, which can make the collected data cover the mine space more comprehensively and reduce the blind spots of data collection.

[0053] When the mobile device moves or inspects along an S-shaped route, it can move along a preset path, or dynamically plan a path based on the surrounding environment perceived in real time, or the operator can guide the movement of the mobile device through remote control.

[0054] Among them, ultrasonic scanning data and depth camera shooting data are generally collected in a synchronous collection method (that is, the ultrasonic scanning equipment and the depth camera work at the same time to obtain data in real time), but can also be collected in an alternating collection method (that is, after the mobile device moves a certain distance, ultrasonic scanning is performed first, and then depth camera shooting is performed).

[0055] Among them, the inspection frequency and inspection time of the mobile device can be flexibly set according to actual needs. For example, the inspection can be carried out in a fixed period and fixed time manner (such as inspection at 12:00 every day), or in an event-triggered manner (such as inspection when a preset event is detected in the mine, and the preset event can be set according to actual needs), or in a combination of the above two methods (i.e., inspection is first carried out in a fixed period and fixed time manner, and once a preset event is detected in the mine, inspection is immediately carried out).

[0056] In some embodiments, step A2 comprises: A201. Calibrate the position of the ultrasonic scanning data according to the position of the mobile device at the time of acquisition of the ultrasonic scanning data; A202. Using a pre-trained three-dimensional space block recognition model to recognize the ultrasound scan data, to match the corresponding reference three-dimensional space block from the three-dimensional space block library as the optional three-dimensional space block; A203. Splice the selected three-dimensional space blocks according to the position calibration results of the ultrasonic scanning data.

[0057] Among them, in step A201, an inertial measurement unit (IMU) or a global positioning system (GPS) can be used to obtain the precise position of the mobile device at the time of collecting the ultrasonic scanning data. The position information in the ultrasonic scanning data is the position information in the ultrasonic scanning device coordinate system. It is necessary to convert the position information in the ultrasonic scanning data into the position information in the mobile device coordinate system according to the conversion relationship between the ultrasonic scanning device coordinate system and the mobile device coordinate system (which can be calibrated in advance), and then convert the position information in the mobile device coordinate system into the position information in the reference coordinate system (a fixed coordinate system, such as a geodetic coordinate system) according to the position of the mobile device, so as to complete the position calibration of the ultrasonic scanning data.

[0058] Among them, the pre-trained 3D space block recognition model can be a neural network model based on deep learning, such as a convolutional neural network (CNN) model or a point cloud processing network (such as PointNet) model. The training data of the 3D space block recognition model can include various typical mine spatial structures, such as tunnels, intersections, support structures, etc. In practical applications, the 3D space block recognition model can be fine-tuned according to the specific mine environment to improve the recognition accuracy. During the recognition process, the 3D space block recognition model will match the ultrasonic scanning data with the preset spatial structure features to determine the 3D space block type that the current scanning data is most likely to correspond to.

[0059] Among them, a 3D space block library can be pre-established, including standard 3D models (i.e. reference 3D space blocks) of various typical mine space structures, and the reference 3D space blocks of various mine space structures can be set according to the actual space structure. Figure 4 In the figure, for the rectangular-shaped tunnel a, the three-dimensional space block b can be a rectangular-shaped one.

[0060] Among them, in step A203, the splicing process needs to consider the connection relationship between the selected three-dimensional space blocks to ensure the continuity and rationality of the overall structure. A graph optimization algorithm can be used to optimize the relative positions of the selected three-dimensional space blocks to minimize the splicing error. At the same time, the original information of the ultrasonic scanning data can be used to make local adjustments to the spliced ​​model to better fit the actual environment.

[0061] This method greatly improves the efficiency and accuracy of 3D modeling by combining position calibration and pre-trained model recognition. Compared with traditional methods, this method can complete the 3D modeling of mine space more quickly while ensuring the accuracy of the modeling results. Especially in complex mine environments, the use of pre-trained models can effectively identify various spatial structures, while position calibration ensures the accuracy of spatial block splicing.

[0062] In some preferred embodiments, the reference three-dimensional space block includes a plurality of sub-space blocks (so that the optional three-dimensional space block also has a plurality of sub-space blocks); the three-dimensional space block library records the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block; Step A3 includes: A301. Positioning the captured device based on the depth camera data and the position of the mobile device at the time of collection of the depth camera data; A302. Determine the subspace block where each device is located according to the positioning results of each device; A303. According to the subspace block of each device and the corresponding optional three-dimensional space block, the corresponding optimal recognition algorithm is obtained from the three-dimensional space block library; A304. Classify and identify each device using the corresponding optimal identification algorithm.

[0063] By dividing a reference three-dimensional space block into a plurality of subspace blocks and predetermining an optimal recognition algorithm for each subspace block, the accuracy and efficiency of device recognition are improved.

[0064] The reference three-dimensional space block is divided into multiple subspace blocks, which can be divided equally (such as Figure 4In the above example, the size of each subspace block c is the same); more detailed recognition processing can be performed according to the characteristics of different areas in the mine. For example, the reference three-dimensional space block can be divided into subspace blocks of different sizes according to factors such as the lighting conditions and equipment density in different areas of the mine. In areas with dense equipment, smaller subspace blocks can be divided to improve recognition accuracy; while in areas with sparse equipment, larger subspace blocks can be divided to improve processing efficiency.

[0065] The optimal recognition algorithm corresponding to each subspace block of each reference three-dimensional space block can be predetermined by experiment and recorded in the three-dimensional space block library.

[0066] Therefore, before step A1, the following steps are also included: A01. Acquire multiple historical equipment data of each subspace block of each reference three-dimensional space block; historical equipment data includes historical equipment images and equipment type annotation information; A02. For each subspace block, use a variety of different recognition algorithms (for example, including convolutional neural network (CNN) algorithm, support vector machine (SVM) algorithm and random forest algorithm, etc., but not limited thereto) to classify and recognize the historical equipment images in the corresponding historical equipment data, and compare the recognition results with the corresponding equipment type labeling information to calculate the recognition accuracy of various recognition algorithms for the subspace block; A03. The recognition algorithm with the highest recognition accuracy is used as the optimal recognition algorithm for the corresponding subspace block, and an optimal recognition algorithm query table is formed and recorded in the three-dimensional space block library.

[0067] Among them, in step A301, the present application uses the depth camera shooting data and the mobile device position information to locate the device to ensure the accuracy of the device position during the identification process. The depth camera can provide the three-dimensional spatial information of the device, and combined with the position information of the mobile device, the position of the device in the mine space can be accurately located. This positioning method is more accurate than the traditional two-dimensional image positioning, and can effectively avoid positioning errors caused by overlapping or occlusion of the equipment. Among them, the three-dimensional spatial information of the device in the depth camera shooting data is the position information in the depth camera detection device coordinate system. It is necessary to first convert the position information of the device in the depth camera detection device coordinate system into the position information in the mobile device coordinate system according to the conversion relationship between the depth camera detection device coordinate system and the mobile device coordinate system (which can be pre-calibrated), and then convert the position information of the device in the mobile device coordinate system into the position information in the reference coordinate system according to the position of the mobile device at the time of collecting the depth camera shooting data, so as to complete the positioning of the device.

[0068] Among them, in step A302, the subspace block where the device is located is determined according to the device positioning result, in preparation for the subsequent selection of the optimal recognition algorithm. This step associates the device with the pre-divided subspace block, laying the foundation for selecting the most suitable recognition algorithm. In this way, the device recognition problem can be transformed into an recognition problem for a specific subspace block, thereby improving the pertinence and efficiency of recognition.

[0069] In step A303, after determining the subspace block where the device is located, the corresponding optimal recognition algorithm is queried from the three-dimensional space block library according to the subspace block where the device is located and the corresponding optional three-dimensional space block. This query method can quickly locate the recognition algorithm that best suits the current device and environment, avoiding the problem of reduced recognition efficiency and accuracy that may be caused by using a general algorithm.

[0070] In step A304, the optimal recognition algorithm obtained by the query is used to classify and recognize each device to improve the accuracy and efficiency of recognition. Since the recognition algorithm optimized for a specific subspace block is used, it can better adapt to the characteristics of the area, such as lighting conditions, device type distribution, etc., thereby improving the accuracy of recognition. At the same time, since unnecessary algorithm attempts and comparisons are avoided, the efficiency of recognition is also improved.

[0071] This application can dynamically select the most suitable recognition algorithm according to the specific location of the equipment, thereby ensuring the recognition accuracy while improving the recognition efficiency. This method is particularly suitable for scenarios such as mines where the environment is complex, the equipment types are diverse, and the distribution areas of various types of equipment are relatively fixed (for example, the conveyor belt is usually set in the middle of the bottom of the mine, and the coal scraper is usually set on the left and right sides of the mine). It can effectively solve the problem of low recognition accuracy and efficiency of traditional methods in complex environments.

[0072] Specifically, in step A3, all equipment in the mine can be identified, and in step A4, according to the equipment identification results and equipment positioning results of all equipment, the corresponding equipment models are extracted from the equipment model library and placed in the corresponding positions.

[0073] This approach can produce the most detailed modeling results so that managers can understand the most accurate space and equipment distribution.

[0074] In some other embodiments, the subspace blocks are divided into important subspace blocks and unimportant subspace blocks; In step A3, only the devices located in the important subspace blocks are classified and identified; for example, in step A304, only the devices located in the important subspace blocks are classified and identified using the corresponding optimal identification algorithm; In step A4, for the equipment located in the important subspace block, the corresponding equipment model is extracted from the equipment model library and placed in the corresponding position in the corresponding optional three-dimensional space block; for the equipment located in the non-important subspace block, simplified modeling is performed.

[0075] This method effectively solves the problem of how to improve the modeling accuracy of important areas while taking into account the modeling efficiency of non-important areas in the 3D mine modeling process by classifying the subspace blocks according to their importance and adopting differentiated equipment identification and modeling strategies.

[0076] There are many ways to classify subspace blocks. For example, subspace blocks can be classified according to their locations. Subspace blocks close to main channels, key equipment, or important operating areas can be marked as important subspace blocks. Another method is to classify according to the density or complexity of the equipment in the subspace blocks. Areas with high equipment density or high complexity can be regarded as important subspace blocks. In addition, the importance of subspace blocks can also be determined based on factors such as safety and production efficiency. By classifying and identifying only the equipment in important subspace blocks, the overall identification workload can be greatly reduced.

[0077] For the equipment in the important subspace blocks, the accurate equipment models are extracted from the equipment model library; while for the equipment in the unimportant subspace blocks, simplified modeling is performed. This strategy can improve the overall modeling efficiency while ensuring the modeling accuracy of important areas, and can also reduce the storage space requirements of the 3D model inside the mine.

[0078] There are many ways to implement the simplified modeling process. For example, do not set up a three-dimensional model for the equipment located in the non-important subspace block, or mark the non-important subspace block with the equipment (for example, fill the corresponding non-important subspace block with a preset color to achieve marking), or, for the equipment located in the non-important subspace block, place a simplified model of a preset shape at the corresponding position in the corresponding optional three-dimensional space block. The simplified model can be a simple geometric shape, such as a cube, sphere or cylinder, to roughly represent the volume and position of the equipment. Before placing the simplified model, the size of the simplified model can be adjusted according to the size of the corresponding equipment, so that the simplified model is the smallest model that envelops the equipment, thereby better representing the volume of the equipment.

[0079] From the above, it can be seen that the three-dimensional mine rapid modeling method obtains ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine; according to the ultrasonic scanning data, the corresponding reference three-dimensional space blocks are matched from the three-dimensional space block library as optional three-dimensional space blocks for splicing to achieve modeling of the mine space; equipment positioning and equipment identification are performed according to the depth camera shooting data; according to the equipment identification results and equipment positioning results, the corresponding equipment model is extracted from the equipment model library and placed in the corresponding position to achieve three-dimensional modeling of the inside of the mine; thereby, the mine can be three-dimensionally modeled quickly and accurately.

[0080] refer to Figure 2 The present application provides a three-dimensional mine rapid modeling device, which performs three-dimensional modeling of the internal space of the mine based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device, including: A first acquisition module 1 is used to acquire ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine (refer to step A1 for the specific process); The space modeling module 2 is used to match the corresponding reference three-dimensional space block from the three-dimensional space block library according to the ultrasonic scanning data as the optional three-dimensional space block for splicing, so as to realize the modeling of the mine space (for the specific process, refer to step A2); Positioning and identification module 3, used to locate and identify the device according to the data captured by the depth camera (refer to step A3 for the specific process); The equipment modeling module 4 is used to extract the corresponding equipment model from the equipment model library according to the equipment identification result and the equipment positioning result, and place it at the corresponding position in the corresponding optional three-dimensional space block to realize the three-dimensional modeling of the interior of the mine (refer to step A4 for the specific process).

[0081] In some embodiments, the three-dimensional mine rapid modeling device further includes: The dynamic update module is used to obtain the ultrasonic scanning data and depth camera shooting data collected in real time by the mobile device during inspection in the mine after completing the three-dimensional modeling of the interior of the mine, so as to dynamically update the three-dimensional modeling results (refer to step A5 for the specific process).

[0082] In some embodiments, the 3D mine rapid modeling device further includes a first building module, which is used to execute: A01. Acquire multiple historical equipment data of each subspace block of each reference three-dimensional space block; the historical equipment data includes historical equipment images and equipment type annotation information; A02. For each subspace block, use a variety of different recognition algorithms (for example, including convolutional neural network (CNN) algorithm, support vector machine (SVM) algorithm and random forest algorithm, etc., but not limited thereto) to classify and recognize the historical equipment images in the corresponding historical equipment data, and compare the recognition results with the corresponding equipment type labeling information to calculate the recognition accuracy of various recognition algorithms for the subspace block; A03. The recognition algorithm with the highest recognition accuracy is used as the optimal recognition algorithm for the corresponding subspace block, and an optimal recognition algorithm query table is formed and recorded in the three-dimensional space block library.

[0083] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores a computer program executable by the processor 301. When the electronic device is running, the processor 301 executes the computer program to execute the three-dimensional mine rapid modeling method in any optional implementation of the above-mentioned embodiment to achieve the following functions: obtaining ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine; according to the ultrasonic scanning data, matching corresponding reference three-dimensional space blocks from a three-dimensional space block library as optional three-dimensional space blocks for splicing to achieve modeling of the mine space; performing equipment positioning and equipment identification according to the depth camera shooting data; according to the equipment identification results and the equipment positioning results, extracting the corresponding equipment model from the equipment model library and placing it in the corresponding position to achieve three-dimensional modeling of the inside of the mine.

[0084] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for rapid three-dimensional mine modeling in any optional implementation of the above-mentioned embodiment is executed to achieve the following functions: acquiring ultrasonic scanning data and depth camera shooting data collected when a mobile device moves along the mine; matching corresponding reference three-dimensional space blocks from a three-dimensional space block library according to the ultrasonic scanning data as optional three-dimensional space blocks for splicing to achieve modeling of the mine space; performing equipment positioning and equipment identification according to the depth camera shooting data; extracting corresponding equipment models from the equipment model library and placing them in corresponding positions according to the equipment identification results and equipment positioning results to achieve three-dimensional modeling of the interior of the mine.

[0085] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0086] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0087] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0089] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0090] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional mine rapid modeling method, characterized in that: The three-dimensional modeling of the interior of the mine is performed based on a mobile device equipped with an ultrasonic scanning device and a depth camera detection device, including the following steps: A1. Obtaining ultrasonic scanning data and depth camera shooting data collected by the mobile device while moving along the mine; A2. According to the ultrasonic scanning data, a corresponding reference three-dimensional space block is matched from a three-dimensional space block library as an optional three-dimensional space block for splicing to achieve modeling of the mine space; A3. Perform device positioning and device identification based on the depth camera data; A4. According to the equipment identification and positioning results, the corresponding equipment model is extracted from the equipment model library and placed at the corresponding position to realize the three-dimensional modeling of the interior of the mine.

2. The three-dimensional mine rapid modeling method according to claim 1, characterized in that: Also includes the steps: A5. After completing the 3D modeling of the interior of the mine, obtain the ultrasonic scanning data and depth camera shooting data collected in real time by the mobile device during inspection in the mine, and use it to dynamically update the 3D modeling results.

3. The three-dimensional mine rapid modeling method according to claim 2 is characterized in that: In step A1, ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the S-shaped route are obtained; In step A5, ultrasonic scanning data and depth camera shooting data collected in real time when the mobile device inspects along the S-shaped route are obtained to dynamically update the three-dimensional modeling results.

4. The three-dimensional mine rapid modeling method according to claim 1, characterized in that: Step A2 includes: Calibrate the position of the ultrasonic scanning data according to the position of the mobile device at the time of collecting the ultrasonic scanning data; The ultrasonic scanning data is recognized by using a pre-trained three-dimensional space block recognition model to match a corresponding reference three-dimensional space block from a three-dimensional space block library as a matching three-dimensional space block; The selected three-dimensional space blocks are spliced ​​according to the position calibration result of the ultrasonic scanning data.

5. The three-dimensional mine rapid modeling method according to claim 1, characterized in that: The reference three-dimensional space block includes a plurality of sub-space blocks; the three-dimensional space block library records the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block; Step A3 includes: A301. Positioning the captured device according to the depth camera data and the position of the mobile device at the time of collecting the depth camera data; A302. Determine the subspace block where each device is located according to the positioning result of each device; A303. According to the subspace block and the corresponding optional three-dimensional space block in which each device is located, the corresponding optimal recognition algorithm is obtained from the three-dimensional space block library; A304. Classify and identify each device using the corresponding optimal identification algorithm.

6. The three-dimensional mine rapid modeling method according to claim 5, characterized in that: Before step A1, the following steps are also included: A01. Acquire multiple historical equipment data of each subspace block of each reference three-dimensional space block; the historical equipment data includes historical equipment images and equipment type annotation information; A02. For each of the subspace blocks, a plurality of different recognition algorithms are used to classify and recognize the historical equipment images in the corresponding historical equipment data, and the recognition results are compared with the corresponding equipment type labeling information to calculate the recognition accuracy of the various recognition algorithms for the subspace blocks; A03. Use the recognition algorithm with the highest recognition accuracy as the optimal recognition algorithm corresponding to the subspace block, form an optimal recognition algorithm query table and record it in the three-dimensional space block library.

7. The three-dimensional mine rapid modeling method according to claim 5, characterized in that: The subspace blocks are divided into important subspace blocks and non-important subspace blocks; In step A3, only the devices located in the important subspace blocks are classified and identified; In step A4, for devices located in important subspace blocks, the corresponding device models are extracted from the device model library and placed at the corresponding positions in the corresponding optional three-dimensional space blocks; for devices located in non-important subspace blocks, a simplified model with a preset shape is placed at the corresponding positions in the corresponding optional three-dimensional space blocks.

8. A three-dimensional mine rapid modeling device, characterized in that: The three-dimensional modeling of the internal space of the mine is carried out based on a mobile device equipped with ultrasonic scanning equipment and depth camera detection equipment, including: A first acquisition module is used to acquire ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine; A space modeling module, used to match corresponding reference three-dimensional space blocks from a three-dimensional space block library according to the ultrasonic scanning data as optional three-dimensional space blocks for splicing, so as to realize modeling of the mine space; A positioning and identification module, used for performing device positioning and device identification based on the data captured by the depth camera; The equipment modeling module is used to extract the corresponding equipment model from the equipment model library according to the equipment identification results and equipment positioning results, and place it at the corresponding position in the corresponding optional three-dimensional space block to realize three-dimensional modeling of the interior of the mine.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the three-dimensional mine rapid modeling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional mine rapid modeling method as described in any one of claims 1 to 7 are executed.

Images