A three-dimensional mine rapid modeling method and related device

Abstract

The application belongs to the technical field of mine modeling, and discloses a three-dimensional mine rapid modeling method and related equipment. The interior of a mine is modeled in three dimensions based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device. By acquiring ultrasonic scanning data and depth camera shooting data, and in combination with a three-dimensional space block library and a device model library, rapid modeling of the space and internal devices of the mine is realized, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art, and enabling rapid and accurate three-dimensional modeling of the mine.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mine modeling, in particular to a three-dimensional mine rapid modeling method and related equipment. BACKGROUND

[0002] In real life, based on the demand for coal mining, it is often necessary to detect the spatial distribution map in the coal mine. In the process of spatial modeling of the mine, laser radar, visual sensor and the like are often used for spatial information collection, and then the space inside the coal mine is three-dimensionally constructed. However, this method has some significant problems and limitations in actual application.

[0003] Firstly, the existing three-dimensional reconstruction technology is mostly based on laser radar and visual sensor for three-dimensional reconstruction. Although laser radar can quickly obtain spatial information, its accuracy is easily affected by complex environmental factors in the coal mine, such as dust, floating small particles and water vapor, etc. These factors will interfere with the propagation and reflection of laser signals, resulting in a decrease in the accuracy of the collected data, thereby affecting the final three-dimensional reconstruction effect.

[0004] Secondly, the visual sensor currently used is mostly a two-dimensional image sensor at the plane level. This kind of sensor can only obtain the projection information in a range, i.e. two-dimensional image, and cannot represent the depth information of the feature objects in the projection part of the range. This limitation makes it difficult to obtain three-dimensional spatial information, thereby affecting the accuracy and integrity of three-dimensional reconstruction.

[0005] In addition, the existing three-dimensional reconstruction technology usually performs object surface shape fitting according to the point cloud data obtained by the sensor, thereby realizing object modeling. Although this method can realize three-dimensional reconstruction, the modeling efficiency is low. In a complex mine environment, a large amount of point cloud data needs to be processed, which not only takes a long time, but also has a high requirement for computing resources, which is not conducive to rapid modeling and real-time updating.

[0006] More importantly, the existing three-dimensional reconstruction strategy of the coal mine mostly simply splices the existing sensor data, and cannot deeply fuse the data. This simple data processing method leads to poor reconstruction effect, and cannot accurately reflect the complex structure and equipment distribution inside the mine. At the same time, this method can only construct a static three-dimensional scene, and cannot adapt to the dynamic changes of the mine environment, such as equipment movement, spatial structure change, etc.

[0007] Therefore, how to quickly three-dimensionally reconstruct the scene inside the coal mine while ensuring the low error rate of local details is a problem to be solved. The industry urgently needs a method that can quickly, accurately and dynamically model the mine in three dimensions to meet the needs of mine safety monitoring, equipment management and spatial planning, etc.

[0008] For the above problems, the prior art needs to be improved. SUMMARY

[0009] The purpose of the present application is to provide a three-dimensional mine rapid modeling method and related equipment, which can quickly and accurately model the mine in three dimensions.

[0010] In the first aspect, the present application provides a three-dimensional mine rapid modeling method, which models the inside of a mine in three dimensions based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device, comprising the steps of:

[0011] A1. Obtain the ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the mine;

[0012] A2. According to the ultrasonic scanning data, match the corresponding reference three-dimensional space block from the three-dimensional space block library as the selected three-dimensional space block for splicing to realize the modeling of the mine space;

[0013] A3. According to the depth camera shooting data, device positioning and device recognition are performed;

[0014] A4. According to the device recognition result and the device positioning result, extract the corresponding device model from the device model library and place it in the corresponding position to realize the three-dimensional modeling of the inside of the mine.

[0015] Based on the mobile device provided with the ultrasonic scanning device and the depth camera detection device, the inside of the mine is modeled in three dimensions. By obtaining the ultrasonic scanning data and the depth camera shooting data, combining the three-dimensional space block library and the device model library, the rapid modeling of the mine space and the internal devices is realized, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art, and enabling the three-dimensional modeling of the mine to be quickly and accurately performed.

[0016] Preferably, the three-dimensional mine rapid modeling method further comprises the step of:

[0017] A5. After completing the three-dimensional modeling of the inside of the mine, obtain the ultrasonic scanning data and depth camera shooting data collected in real time by the mobile device when inspecting in the mine, to dynamically update the three-dimensional modeling result.

[0018] This method can realize dynamic updating of the mine model, timely reflect changes in the internal environment of the mine, and maintain the accuracy and timeliness of the three-dimensional model.

[0019] Preferably, in step A1, the ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the S-shaped route are obtained;

[0020] In step A5, the ultrasonic scanning data and the depth camera shooting data collected in real time when the mobile device inspects along the S-shaped route are used to dynamically update the three-dimensional modeling result.

[0021] The mobile device moves along the S-shaped route to collect the ultrasonic scanning data and the depth camera shooting data, so that the collected data can more comprehensively cover the mine space and reduce the blind area of data collection.

[0022] Preferably, step A2 comprises:

[0023] According to the position of the mobile device at the collection time of the ultrasonic scanning data, the ultrasonic scanning data is positionally calibrated;

[0024] The ultrasonic scanning data is identified by using a pre-trained three-dimensional space block identification model, so as to match a corresponding reference three-dimensional space block from a three-dimensional space block library as a selected three-dimensional space block;

[0025] According to the position calibration result of the ultrasonic scanning data, the selected three-dimensional space block is spliced.

[0026] Preferably, the reference three-dimensional space block comprises a plurality of sub-space blocks; and the three-dimensional space block library records an optimal identification algorithm corresponding to each sub-space block of each reference three-dimensional space block;

[0027] Step A3 comprises:

[0028] A301. According to the depth camera shooting data and the position of the mobile device at the collection time of the depth camera shooting data, the equipment photographed is positioned;

[0029] A302. According to the positioning result of each equipment, the sub-space block in which each equipment is located is determined;

[0030] A303. According to the sub-space block in which each equipment is located and the corresponding selected three-dimensional space block, the corresponding optimal identification algorithm is queried from the three-dimensional space block library;

[0031] A304. Each equipment is classified and identified by using the corresponding optimal identification algorithm.

[0032] Preferably, step A1 further comprises a step before step A1:

[0033] A01. A plurality of historical equipment data of each sub-space block of each reference three-dimensional space block are obtained; the historical equipment data comprises historical equipment images and equipment type annotation information;

[0034] A02. For each of the sub-space blocks, a plurality of different recognition algorithms are respectively used to classify and recognize historical device images in the historical device data, and the recognition results are compared with the corresponding device type label information to calculate the recognition accuracy of the various recognition algorithms for the sub-space blocks;

[0035] A03. The recognition algorithm with the highest recognition accuracy is used as the optimal recognition algorithm for the corresponding sub-space block, and an optimal recognition algorithm query table is formed and recorded in the three-dimensional space block library.

[0036] Preferably, the sub-space blocks are divided into important sub-space blocks and non-important sub-space blocks.

[0037] In step A3, only the devices located in the important sub-space blocks are classified and recognized.

[0038] In step A4, for the devices located in the important sub-space blocks, the corresponding device models are extracted from the device model library and placed in the corresponding positions in the corresponding selected three-dimensional space blocks; for the devices located in the non-important sub-space blocks, a simplified model with a preset shape is placed in the corresponding positions in the corresponding selected three-dimensional space blocks.

[0039] In a second aspect, the present application provides a three-dimensional mine rapid modeling device, which is based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device to perform three-dimensional modeling on the internal space of a mine, and includes:

[0040] A first acquisition module is configured to acquire ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the mine;

[0041] A space modeling module is configured to match corresponding reference three-dimensional space blocks from a three-dimensional space block library as selected three-dimensional space blocks for splicing according to the ultrasonic scanning data, so as to realize modeling of the space of the mine;

[0042] A positioning and recognition module is configured to perform device positioning and device recognition according to the depth camera shooting data;

[0043] A device modeling module is configured to extract corresponding device models from a device model library and place them in corresponding positions in corresponding selected three-dimensional space blocks according to the device recognition results and the device positioning results, so as to realize three-dimensional modeling of the internal space of the mine.

[0044] In a third aspect, the present application provides an electronic device including a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the steps of the three-dimensional mine rapid modeling method described above are performed.

[0045] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, performs the steps of the three-dimensional mine rapid modeling method as described above.

[0046] Beneficial effects: The three-dimensional mine rapid modeling method and related device provided by the present application perform three-dimensional modeling on the inside of a mine based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device, obtain ultrasonic scanning data and depth camera shooting data, and combine a three-dimensional space block library and a device model library to achieve rapid modeling of the space and internal devices of the mine, thereby solving the problems of low modeling efficiency and poor accuracy in the prior art and enabling rapid and accurate three-dimensional modeling of the mine. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart of the three-dimensional mine rapid modeling method provided by the embodiments of the present application.

[0048] Figure 2 The structural schematic diagram of the three-dimensional mine rapid modeling device provided by the embodiments of the present application.

[0049] Figure 3 The structural schematic diagram of the electronic device provided by the embodiments of the present application.

[0050] Figure 4 The schematic diagram of a mine roadway and selected three-dimensional space blocks.

[0051] Label explanation: 1, first obtaining module; 2, space modeling module; 3, positioning and identification module; 4, device modeling module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0053] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings, and that, once an item is defined in one drawing, it should not be further defined and explained in subsequent drawings. Also, in the description of the present application, the terms "first", "second", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0054] Three-dimensional modeling of mine interiors is an important step in coal mining and safety management. Traditional three-dimensional reconstruction techniques mainly rely on laser radars and visual sensors to obtain spatial information. However, these methods face many challenges in practical applications. Laser radars can quickly obtain spatial information, but their accuracy is easily affected by environmental factors such as dust, floating small particles, and water vapor in the mine. Ordinary visual sensors are mostly two-dimensional image sensors, which can only obtain planar projection information and cannot accurately represent the depth information of objects. In addition, existing three-dimensional reconstruction techniques usually require point-by-point fitting of point cloud data, which is low in modeling efficiency. More importantly, current three-dimensional reconstruction strategies for coal mines often simply splice sensor data, lack depth fusion, resulting in poor reconstruction results and only static three-dimensional scenes can be constructed. These problems seriously restrict the accuracy, efficiency, and practicality of three-dimensional modeling of mine interiors.

[0055] Specifically, in a large coal mine's underground mining area, the mine environment is complex and variable, containing multiple branch channels, coal mining faces, and various equipment. Traditional three-dimensional modeling methods face serious challenges in such an environment. For example, when using laser radars to scan, due to the large amount of dust and water vapor generated during coal mining, laser signals are easily scattered and attenuated, resulting in a large amount of noise and holes in the obtained point cloud data. Two-dimensional images taken by ordinary cameras cannot accurately reflect the degree of curvature of the tunnel and the spatial position relationship of the equipment. In addition, the lighting conditions in the mine are usually poor, further reducing the image quality. In the data processing stage, due to the lack of effective data fusion methods, the data collected by different sensors are difficult to coordinate, causing a large number of errors and discontinuous areas in the reconstructed model. More importantly, traditional methods are difficult to update the model in real time, and cannot reflect the dynamic changes of the mine environment, such as equipment movement or tunnel shape change.

[0056] If these technical problems cannot be effectively solved, it will have a serious impact on mine management and safety production. First, inaccurate three-dimensional models may lead to incorrect judgments about the spatial structure of the mine, affecting mining planning and safety assessment. Second, the inefficient modeling process will prolong the time of mine detection and assessment, reducing production efficiency. Third, static models cannot reflect real-time changes in the mine environment, which may lead to safety hazards that cannot be discovered and addressed in a timely manner. For example, in an emergency, if the three-dimensional model relied on by rescue personnel has a large deviation from the actual situation, it may seriously affect the rescue effect. In addition, inaccurate equipment positioning information may cause the failure of remote control and automation systems, increasing the risk of production. Therefore, developing a new method that can quickly, accurately and dynamically model the three-dimensional interior of a mine is of great significance for improving mine management efficiency, ensuring production safety and optimizing resource utilization.

[0057] To this end, please refer to Figure 1 The application provides a three-dimensional mine rapid modeling method, which is based on a mobile device provided with ultrasonic scanning equipment and depth camera detection equipment to model the interior of a mine in three dimensions, including the following steps:

[0058] A1. Obtain ultrasonic scanning data and depth camera shooting data collected when the mobile device moves along the mine;

[0059] A2. According to the ultrasonic scanning data, match the corresponding reference three-dimensional space block from the three-dimensional space block library as the selected three-dimensional space block for splicing to realize the modeling of the mine space;

[0060] A3. According to the depth camera shooting data, perform equipment positioning and equipment identification;

[0061] A4. According to the equipment identification result and the equipment positioning result, extract the corresponding equipment model from the equipment model library and place it at the corresponding position (i.e. the position corresponding to the equipment positioning result) to realize the three-dimensional modeling of the interior of the mine.

[0062] Based on the mobile device provided with ultrasonic scanning equipment and depth camera detection equipment, the three-dimensional modeling of the interior of the mine is realized by obtaining ultrasonic scanning data and depth camera shooting data, combining the three-dimensional space block library and the equipment model library, which realizes the rapid modeling of the mine space and the interior equipment, thereby solving the problem of low modeling efficiency and poor accuracy in the prior art, and quickly and accurately modeling the mine in three dimensions.

[0063] The core innovation of the present application is to combine ultrasonic scanning and depth camera technology, use a pre-set three-dimensional space block and a device model library to achieve rapid three-dimensional modeling of the inside of a mine. This method overcomes the precision problem of traditional laser radar in complex environments, while providing more rich three-dimensional information. Through the pre-set three-dimensional space block and the device model library, the modeling efficiency is greatly improved, avoiding the tedious process of point-by-point reconstruction.

[0064] The working principle of the present application can be described in detail as follows:

[0065] First, the mobile device moves inside the mine while collecting data using ultrasonic scanning equipment and depth camera detection equipment. The ultrasonic scanning equipment obtains spatial information of the surrounding environment by emitting ultrasonic waves and receiving reflected waves, forming ultrasonic scanning data. The depth camera detection equipment captures three-dimensional information of the scene through structured light or time-of-flight technology, generating depth camera shooting data.

[0066] Next, spatial modeling is performed according to the ultrasonic scanning data. The system matches the obtained ultrasonic scanning data with the pre-stored reference three-dimensional space blocks in the three-dimensional space block library. After matching the most similar reference three-dimensional space blocks, they are selected as three-dimensional space blocks. Then, the system will splice these selected three-dimensional space blocks according to the spatial relationship of the ultrasonic scanning data, thereby realizing the overall modeling of the mine space.

[0067] At the same time, the system uses depth camera shooting data for device positioning and recognition. The three-dimensional information provided by the depth camera allows the spatial position of the device to be accurately determined. Through image processing and machine learning algorithms, the system can identify various types of devices in the shooting picture.

[0068] Finally, according to the device recognition and positioning results, the system extracts the corresponding three-dimensional device model from the device model library. These models will be placed in the corresponding position in the previously established mine space model, thereby completing the detailed three-dimensional modeling of the inside of the mine.

[0069] The reason why this method chooses ultrasonic scanning instead of laser radar is that ultrasonic waves have an advantage in penetrating dust and water vapor, and can obtain more accurate spatial information in complex mine environments. The introduction of the depth camera makes up for the lack of depth information that ordinary two-dimensional image sensors cannot obtain. The pre-set three-dimensional space block and the device model library greatly improve the modeling efficiency, avoiding the tedious process of point-by-point reconstruction in traditional methods.

[0070] Among them, the mobile device can adopt a robot, AGV or other wheeled or tracked mobile platform.

[0071] Among them, the ultrasonic scanning equipment can but not limited to adopt an ultrasonic sensor array to achieve.

[0072] The depth camera detection device can be implemented by a depth camera based on structured light or time-of-flight (ToF) technology, but is not limited thereto.

[0073] The three-dimensional space block library stores reference three-dimensional space blocks of various typical mine space structures established in advance, and can be implemented by a computer storage system.

[0074] The device model library stores standard three-dimensional models (i.e., device models) of various typical mine devices (such as a coal mine, including a coal scraper, a conveyor belt, and a transport vehicle) established in advance, and can be implemented by a computer storage system.

[0075] In some preferred embodiments, the three-dimensional mine rapid modeling method further comprises the steps of:

[0076] A5. After completing the three-dimensional modeling of the mine interior, real-time ultrasonic scanning data and depth camera shooting data collected by the mobile device during the inspection in the mine are obtained to dynamically update the three-dimensional modeling result.

[0077] This method can dynamically update the mine model and timely reflect the changes in the mine interior environment, maintaining the accuracy and timeliness of the three-dimensional model.

[0078] Specifically, the initial three-dimensional modeling of the mine interior is first completed through steps A1-A4; then the mobile device performs inspection (which can be performed according to a preset inspection route or according to actual needs), and real-time ultrasonic scanning data and depth camera shooting data are collected during the inspection, which contain the latest spatial information and device information in the mine interior; then, the system compares and fuses the new data collected during the inspection with the existing three-dimensional model. If changes in the mine environment are found, such as the addition of devices, changes in spatial structure (for example, the real-time recognition result of the selected three-dimensional space block is different from the last recognition result), changes in device location (for example, the position deviation between the real-time positioning result of the device location and the last positioning result exceeds a preset deviation threshold, which can be set according to actual needs), etc., the system will update the three-dimensional model of the mine interior accordingly (when updating, refer to steps A2-A4 to update the selected three-dimensional space block and / or the device model in the selected three-dimensional space block).

[0079] This dynamic updating mechanism works well with the initial three-dimensional modeling method. The initial modeling provides a basic model, and the dynamic updating ensures the continuous accuracy of the model, so that the management personnel can timely understand the mine conditions. This combination not only solves the problem that a static model cannot reflect environmental changes, but also improves the efficiency and accuracy of the overall modeling.

[0080] In some preferred embodiments, in step A1, the ultrasonic scanning data and the depth camera shooting data collected when the mobile device moves along the S-shaped route are acquired;

[0081] In step A5, the ultrasonic scanning data and the depth camera shooting data collected in real time when the mobile device inspects along the S-shaped route are acquired to dynamically update the three-dimensional modeling result.

[0082] The movement of the mobile device along the S-shaped route to collect the ultrasonic scanning data and the depth camera shooting data can make the collected data more comprehensively cover the mine space and reduce the blind area of data collection.

[0083] When the mobile device moves along the S-shaped route or inspects along the S-shaped route, the mobile device can move along a preset path, dynamically plan a path according to a real-time perceived surrounding environment, or be guided to move by an operator through a remote control mode.

[0084] The ultrasonic scanning data and the depth camera shooting data are generally collected in a synchronous collection mode (i.e., the ultrasonic scanning device and the depth camera work simultaneously to acquire data in real time), but can also be collected in an alternating collection mode (i.e., the mobile device moves a certain distance, then performs ultrasonic scanning, and then performs depth camera shooting).

[0085] The inspection frequency and the inspection time of the mobile device during inspection can be flexibly set according to actual needs. For example, the inspection can be performed in a fixed period and a fixed time (e.g., inspection is performed at 12:00 every day), the inspection can be performed in an event-triggered manner (e.g., inspection is performed when a preset event occurs in the mine, and the preset event can be set according to actual needs), or the inspection can be performed in combination with the two aforementioned manners (i.e., the inspection is first performed in a fixed period and a fixed time, and once a preset event occurs in the mine, the inspection is immediately performed).

[0086] In some embodiments, step A2 includes:

[0087] A201. Positionally calibrating the ultrasonic scanning data according to the position of the mobile device at the collection time of the ultrasonic scanning data;

[0088] A202. Identifying the ultrasonic scanning data by using a pre-trained three-dimensional space block identification model to match a corresponding reference three-dimensional space block from a three-dimensional space block library as a selected three-dimensional space block;

[0089] A203. Splicing the selected three-dimensional space block according to the positionally calibrated result of the ultrasonic scanning data.

[0090] In step A201, an inertial measurement unit (IMU) or global positioning system (GPS) can be used to obtain the precise position of the mobile device at the time of ultrasound scan data collection. The position information in the ultrasound scan data is in the ultrasound scan device coordinate system, which needs to be converted to the mobile device coordinate system according to the conversion relationship between the ultrasound scan device coordinate system and the mobile device coordinate system (which can be pre-calibrated), and then converted to the position information in the reference coordinate system (a fixed coordinate system, such as the geodetic coordinate system) according to the position of the mobile device, completing the position calibration of the ultrasound scan data.

[0091] The pre-trained three-dimensional space block recognition model can be a neural network model based on deep learning, such as a convolutional neural network (CNN) model or a point cloud processing network (such as PointNet) model. The training data of the three-dimensional space block recognition model can include various typical mine space structures, such as roadways, intersections, support structures, etc. In actual application, the three-dimensional space block recognition model can be fine-tuned according to the specific mine environment to improve the accuracy of recognition. In the recognition process, the three-dimensional space block recognition model matches the ultrasound scan data with the pre-set spatial structure features to determine the most likely three-dimensional space block type corresponding to the current scan data.

[0092] The three-dimensional space block library can be pre-established and contains standard three-dimensional models of various typical mine space structures (i.e. reference three-dimensional space blocks). The reference three-dimensional space blocks of various mine space structures can be set according to the actual space structure. For example Figure 4 For a cuboid-shaped roadway a, the selected three-dimensional space block b can be a cuboid.

[0093] In step A203, the connection relationship between the selected three-dimensional space blocks needs to be considered in the splicing process to ensure the continuity and rationality of the overall structure. A graph optimization algorithm can be used to optimize the relative positions of the selected three-dimensional space blocks to minimize the splicing error. At the same time, the original information of the ultrasound scan data can be used to make local adjustments to the spliced model to better fit the actual environment.

[0094] This method greatly improves the efficiency and accuracy of three-dimensional modeling by combining position calibration and pre-trained model recognition. Compared with traditional methods, this method can complete three-dimensional modeling of mine space more quickly while ensuring the accuracy of the modeling results. Especially in complex mine environments, the use of pre-trained models can effectively identify various space structures, and position calibration ensures the accuracy of space block splicing.

[0095] In some preferred embodiments, each reference three-dimensional space block comprises a plurality of sub-space blocks (so that the selected three-dimensional space block also has a plurality of sub-space blocks); and the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block is recorded in the three-dimensional space block library.

[0096] Step A3 comprises:

[0097] A301. Positioning each device according to the depth camera shooting data and the position of the mobile device at the time of collecting the depth camera shooting data;

[0098] A302. Determining the sub-space block in which each device is located according to the positioning result of each device;

[0099] A303. Querying the corresponding optimal recognition algorithm from the three-dimensional space block library according to the sub-space block in which each device is located and the corresponding selected three-dimensional space block;

[0100] A304. Classifying and recognizing each device by using the corresponding optimal recognition algorithm.

[0101] By dividing the reference three-dimensional space block into a plurality of sub-space blocks and pre-determining the optimal recognition algorithm for each sub-space block, the accuracy and efficiency of device recognition are improved.

[0102] In the method, the reference three-dimensional space block is divided into a plurality of sub-space blocks, which can be divided equally (for example, the size of each sub-space block is the same as that of the reference three-dimensional space block); Figure 4 In the method, the reference three-dimensional space block is divided into a plurality of sub-space blocks, which can be divided equally (for example, the size of each sub-space block is the same as that of the reference three-dimensional space block);

[0103] In the method, the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block can be pre-determined by trial and recorded in the three-dimensional space block library.

[0104] Therefore, before step A1, the method further comprises the following step:

[0105] A01. Obtaining a plurality of historical device data of each sub-space block of each reference three-dimensional space block; the historical device data comprises historical device images and device type annotation information;

[0106] A02. For each subspace block, a plurality of different recognition algorithms (for example, including convolutional neural network (CNN) algorithm, support vector machine (SVM) algorithm and random forest algorithm, etc., but not limited to) are respectively adopted to classify and recognize the historical equipment images in the corresponding historical equipment data, and the recognition results are compared with the corresponding equipment type labeling information to calculate the recognition accuracy of various recognition algorithms for the subspace block;

[0107] A03. The recognition algorithm with the highest recognition accuracy is taken as the optimal recognition algorithm for the corresponding subspace block, and an optimal recognition algorithm query table is formed and recorded in the three-dimensional space block library.

[0108] In step A301, the application uses depth camera shooting data and mobile device location information to locate the equipment, ensuring the accuracy of the equipment location in the recognition process. The depth camera can provide three-dimensional space information of the equipment, combined with the location information of the mobile device, the position of the equipment in the mine space can be accurately located. This positioning method is more accurate than traditional two-dimensional image positioning, which can effectively avoid positioning errors caused by equipment overlap or shielding. The three-dimensional space information of the equipment in the depth camera shooting data is the position information in the depth camera detection equipment coordinate system. The position information of the equipment in the depth camera detection equipment coordinate system needs to be converted into the position information in the mobile device coordinate system according to the conversion relationship (which can be calibrated in advance) between the depth camera detection equipment coordinate system and the mobile device coordinate system, and then the position information of the equipment in the mobile device coordinate system is converted into the position information in the reference coordinate system according to the position of the mobile device at the time of collecting the depth camera shooting data, and the positioning of the equipment is completed.

[0109] In step A302, the subspace block where the equipment is located is determined according to the equipment positioning result, which prepares for the subsequent selection of the optimal recognition algorithm. This step associates the equipment with the pre-divided subspace block, laying the foundation for selecting the most suitable recognition algorithm. In this way, the equipment recognition problem can be transformed into a recognition problem for a specific subspace block, thereby improving the pertinence and efficiency of recognition.

[0110] In step A303, after determining the subspace block where the equipment is located, the corresponding optimal recognition algorithm is queried from the three-dimensional space block library according to the subspace block where the equipment is located and the corresponding selected three-dimensional space block. This query method can quickly locate the most suitable recognition algorithm for the current equipment and environment, avoiding the problem of reduced recognition efficiency and accuracy caused by using general algorithms.

[0111] In step A304, the optimal recognition algorithm obtained by the query is used to classify and recognize each device, improving the accuracy and efficiency of recognition. Since the recognition algorithm optimized for a specific subspace block is used, it can better adapt to the characteristics of the region, such as lighting conditions, device type distribution, etc., thereby improving the accuracy of recognition. At the same time, since unnecessary algorithm attempts and comparisons are avoided, the efficiency of recognition is also improved.

[0112] The present application can dynamically select the most suitable recognition algorithm according to the specific location of the device, thereby ensuring the accuracy of recognition while improving the efficiency of recognition. This method is particularly suitable for complex environments such as mines, where there are many types of devices and the distribution of each type of device is relatively fixed (e.g., conveyor belts are usually set in the middle of the mine floor, and coal scraping machines are usually set on the left and right sides of the mine). This method can effectively solve the problem of low recognition accuracy and efficiency in complex environments.

[0113] Specifically, in step A3, all devices in the mine can be recognized, and in step A4, the corresponding device model is extracted from the device model library according to the device recognition result and the device positioning result of all devices and placed in the corresponding position.

[0114] This way can get the most detailed modeling result, so that the management personnel can understand the most accurate spatial and device distribution situation.

[0115] In other embodiments, the subspace blocks are divided into important subspace blocks and non-important subspace blocks.

[0116] In step A3, only the devices located in the important subspace blocks are classified and recognized; for example, in step A304, only the corresponding optimal recognition algorithm is used to classify and recognize the devices located in the important subspace blocks.

[0117] In step A4, for the devices located in the important subspace blocks, the corresponding device model is extracted from the device model library and placed in the corresponding position in the corresponding selected three-dimensional space block; for the devices located in the non-important subspace blocks, simplified modeling processing is performed.

[0118] This method classifies the importance of subspace blocks and adopts differentiated device recognition and modeling strategies, effectively solving the problem of how to improve the modeling accuracy of important areas while considering the modeling efficiency of non-important areas in the three-dimensional mine modeling process.

[0119] Among them, various methods can be used to classify the subspace blocks. For example, classification can be performed according to the location of the subspace block. Subspace blocks close to main channels, key equipment or important work areas can be marked as important subspace blocks. Another method is to classify according to the density or complexity of the equipment in the subspace block. Areas with high equipment density or complexity can be considered important subspace blocks. In addition, the importance of the subspace block can also be determined according to factors such as safety and production efficiency. By classifying and identifying only the equipment in the important subspace blocks, the overall identification workload can be greatly reduced.

[0120] For equipment in important subspace blocks, accurate equipment models are extracted from the equipment model library; for equipment in non-important subspace blocks, simplified modeling is performed. This strategy can ensure the modeling accuracy of important areas while improving the overall modeling efficiency, and can also reduce the storage space requirements of the three-dimensional model inside the mine.

[0121] Among them, there are various ways to implement simplified modeling. For example, no three-dimensional model is set for the equipment located in the non-important subspace block, or the non-important subspace block with equipment is marked (for example, the corresponding non-important subspace block is filled with a pre-set color to achieve marking), or a pre-set shape of a simplified model is placed in the corresponding position inside the corresponding selected three-dimensional space block for the equipment located in the non-important subspace block. The simplified model can be a simple geometric shape such as a cube, sphere or cylinder to roughly represent the volume and position of the equipment. Among them, before placing the simplified model, the size of the simplified model can be adjusted according to the size of the corresponding equipment, so that the simplified model is the smallest model that envelopes the equipment, thereby better representing the volume of the equipment.

[0122] As can be seen from the above, the three-dimensional mine rapid modeling method acquires ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the mine; according to the ultrasonic scanning data, the corresponding reference three-dimensional space block is matched from the three-dimensional space block library as a selected three-dimensional space block for splicing to realize modeling of the mine space; according to the depth camera shooting data, equipment positioning and equipment identification are performed; according to the equipment identification result and the equipment positioning result, the corresponding equipment model is extracted from the equipment model library and placed in the corresponding position to realize three-dimensional modeling of the inside of the mine; thereby the three-dimensional modeling of the mine can be quickly and accurately performed.

[0123] Reference Figure 2 The present application provides a three-dimensional mine rapid modeling device for three-dimensional modeling of the inside of a mine based on a mobile device provided with ultrasonic scanning equipment and depth camera detection equipment, comprising:

[0124] The first acquisition module 1 is configured to acquire ultrasonic scanning data and depth camera shooting data collected when the mobile device moves in the mine (for a detailed process, refer to step A1);

[0125] The space modeling module 2 is configured to match corresponding reference three-dimensional space blocks from the three-dimensional space block library according to the ultrasonic scanning data, to splice the selected three-dimensional space blocks, so as to realize the modeling of the mine space (for a detailed process, refer to step A2);

[0126] The positioning and recognition module 3 is configured to perform device positioning and device recognition according to the depth camera shooting data (for a detailed process, refer to step A3);

[0127] The device modeling module 4 is configured to place the corresponding device model extracted from the device model library according to the device recognition result and the device positioning result in the corresponding position in the selected three-dimensional space block, to realize the three-dimensional modeling of the interior of the mine (for a detailed process, refer to step A4).

[0128] In some embodiments, the three-dimensional mine rapid modeling device further comprises:

[0129] The dynamic updating module is configured to acquire ultrasonic scanning data and depth camera shooting data collected in real time when the mobile device inspects in the mine after completing the three-dimensional modeling of the interior of the mine, to dynamically update the three-dimensional modeling result (for a detailed process, refer to step A5).

[0130] In some embodiments, the three-dimensional mine rapid modeling device further comprises a first construction module, which is configured to perform:

[0131] A01. Acquire a plurality of historical device data of each sub-space block of each reference three-dimensional space block; the historical device data includes historical device images and device type annotation information;

[0132] A02. For each sub-space block, respectively adopt a plurality of different recognition algorithms (for example, including a convolutional neural network (CNN) algorithm, a support vector machine (SVM) algorithm, and a random forest algorithm, but not limited thereto) to classify and recognize the historical device images in the corresponding historical device data, and compare the recognition results with the corresponding device type annotation information, to calculate the recognition accuracy of each recognition algorithm for the sub-space block;

[0133] A03. Take the recognition algorithm with the highest recognition accuracy as the optimal recognition algorithm for the corresponding sub-space block, to form an optimal recognition algorithm query table and record in the three-dimensional space block library.

[0134] Please refer to Figure 3 , Figure 3A structural schematic diagram of an electronic device provided by the embodiment of the present application, the present application provides an electronic device, comprising: a processor 301 and a memory 302, the processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanism (not marked), the memory 302 stores a computer program executable by the processor 301, when the electronic device runs, the processor 301 executes the computer program to execute the three-dimensional mine rapid modeling method in any optional implementation manner of the above-mentioned embodiment, to realize the following functions: acquiring ultrasonic scanning data and depth camera shooting data collected when a mobile device moves along a mine; according to the ultrasonic scanning data, matching the corresponding reference three-dimensional space block from the three-dimensional space block library as the selected three-dimensional space block for splicing, to realize the modeling of the mine space; according to the depth camera shooting data, device positioning and device identification are carried out; according to the device identification result and the device positioning result, the corresponding device model is extracted from the device model library and placed in the corresponding position, to realize the three-dimensional modeling of the inside of the mine.

[0135] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the three-dimensional mine rapid modeling method in any optional implementation manner of the above-mentioned embodiment is executed, to realize the following functions: acquiring ultrasonic scanning data and depth camera shooting data collected when a mobile device moves along a mine; according to the ultrasonic scanning data, matching the corresponding reference three-dimensional space block from the three-dimensional space block library as the selected three-dimensional space block for splicing, to realize the modeling of the mine space; according to the depth camera shooting data, device positioning and device identification are carried out; according to the device identification result and the device positioning result, the corresponding device model is extracted from the device model library and placed in the corresponding position, to realize the three-dimensional modeling of the inside of the mine.

[0136] Wherein, the computer readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0137] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely exemplary, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0138] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, and can be located in one position, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0139] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0140] In this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0141] The above description is merely exemplary of the application, and is not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A three-dimensional mine rapid modeling method, characterized in that, Based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device, a mine interior is three-dimensionally modeled, including steps of: A1. acquiring ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the mine; A2. according to the ultrasonic scanning data, matching corresponding reference three-dimensional space blocks from a three-dimensional space block library as selected three-dimensional space blocks for splicing to realize modeling of the mine space; A3. according to the depth camera shooting data, device positioning and device recognition are performed; A4. according to the device recognition result and the device positioning result, corresponding device models are extracted from a device model library and placed at corresponding positions to realize three-dimensional modeling of the mine interior; The reference three-dimensional space block includes a plurality of sub-space blocks; the three-dimensional space block library records the optimal recognition algorithm corresponding to each sub-space block of each reference three-dimensional space block; Step A3 includes: A301. according to the depth camera shooting data and the position of the mobile device at the time of collecting the depth camera shooting data, the photographed device is positioned; A302. according to the positioning result of each device, the sub-space block in which each device is located is determined; A303. according to the sub-space block in which each device is located and the corresponding selected three-dimensional space block, the corresponding optimal recognition algorithm is queried from the three-dimensional space block library; A304. each device is classified and recognized by using the corresponding optimal recognition algorithm.

2. The three-dimensional mine rapid modeling method according to claim 1, characterized in that, Further comprising the step of: A5. after completing the three-dimensional modeling of the mine interior, real-time acquisition of ultrasonic scanning data and depth camera shooting data collected by the mobile device when patrolling in the mine is performed to dynamically update the three-dimensional modeling result.

3. The three-dimensional mine rapid modeling method according to claim 2, characterized in that, In step A1, the ultrasonic scanning data and the depth camera shooting data collected by the mobile device when moving along an S-shaped route are acquired; In step A5, real-time acquisition of ultrasonic scanning data and depth camera shooting data collected by the mobile device when patrolling along an S-shaped route is performed to dynamically update the three-dimensional modeling result.

4. The three-dimensional mine rapid modeling method according to claim 1, characterized in that, Step A2 includes: According to the position of the mobile device at the time of collecting the ultrasonic scanning data, the ultrasonic scanning data is positionally calibrated; The ultrasonic scanning data is recognized by using a pre-trained three-dimensional space block recognition model to match corresponding reference three-dimensional space blocks from a three-dimensional space block library as selected three-dimensional space blocks; According to the position calibration result of the ultrasonic scanning data, the selected three-dimensional space blocks are spliced.

5. The three-dimensional mine rapid modeling method according to claim 1, characterized in that, Before step A1, further comprising the steps of: A01. acquiring a plurality of historical device data of each sub-space block of each reference three-dimensional space block; the historical device data includes historical device images and device type annotation information; A02. for each sub-space block, a plurality of different recognition algorithms are respectively used to classify and recognize historical device images in the corresponding historical device data, and the recognition results are compared with the corresponding device type annotation information to calculate the recognition accuracy of each recognition algorithm for the sub-space block; A03. the recognition algorithm with the highest recognition accuracy is used as the optimal recognition algorithm corresponding to the sub-space block, and an optimal recognition algorithm query table is formed and recorded in the three-dimensional space block library.

6. The three-dimensional mine rapid modeling method according to claim 1, characterized in that, The sub-space blocks are divided into important sub-space blocks and non-important sub-space blocks; In step A3, only the devices located in the important sub-space blocks are classified and identified; In step A4, for the devices located in the important sub-space blocks, corresponding device models are extracted from a device model library and placed in corresponding positions in the corresponding selected three-dimensional space blocks; for the devices located in the non-important sub-space blocks, a simplified model with a preset shape is placed in the corresponding positions in the corresponding selected three-dimensional space blocks.

7. A three-dimensional mine rapid modeling device, characterized in that, A three-dimensional modeling method for a mine interior space based on a mobile device provided with an ultrasonic scanning device and a depth camera detection device, comprising: A first acquisition module for acquiring ultrasonic scanning data and depth camera shooting data collected by the mobile device when moving along the mine; A space modeling module for matching corresponding reference three-dimensional space blocks from a three-dimensional space block library as selected three-dimensional space blocks for splicing according to the ultrasonic scanning data, so as to realize modeling of the mine space; A positioning and identification module for device positioning and device identification according to the depth camera shooting data; A device modeling module for extracting corresponding device models from a device model library and placing them in corresponding positions in the corresponding selected three-dimensional space blocks according to the device identification results and the device positioning results, so as to realize three-dimensional modeling of the mine interior; The reference three-dimensional space blocks comprise a plurality of sub-space blocks; the three-dimensional space block library records optimal identification algorithms corresponding to each sub-space block of each reference three-dimensional space block; The positioning and identification module performs the following steps when performing device positioning and device identification according to the depth camera shooting data: A301. Positioning the photographed devices according to the depth camera shooting data and the position of the mobile device at the time of collection of the depth camera shooting data; A302. Determining the sub-space blocks in which each device is located according to the positioning results of each device; A303. Querying the corresponding optimal identification algorithms from the three-dimensional space block library according to the sub-space blocks in which each device is located and the corresponding selected three-dimensional space blocks; A304. Classifying and identifying each device by using the corresponding optimal identification algorithms.

8. An electronic device, comprising: A processor and a memory, the memory storing a computer program executable by the processor, and the processor executing the computer program to run the steps in the three-dimensional mine rapid modeling method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to run the steps in the three-dimensional mine rapid modeling method of any one of claims 1-6.

Images