A method, apparatus, equipment and storage medium for constructing downhole maps

Abstract

This invention relates to the field of intelligent mining technology, specifically to a method, apparatus, device, and storage medium for constructing underground maps. The method includes: adaptively preprocessing multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data; obtaining a real-time local point cloud map corresponding to the underground roadway by combining key feature points and inertial navigation feature data; processing the real-time local point cloud map based on depth images and semantic segmentation results to obtain a real-time local point cloud map with semantic labels; tightly coupling and fusing the multi-source feature data to obtain real-time local fused data of the underground roadway; optimizing the real-time local point cloud map with semantic labels to obtain a real-time high-precision local map; and fusing and optimizing the real-time high-precision local maps corresponding to different time series to obtain a global map of the underground roadway with high robustness, thereby ensuring the continuous operation capability of tunneling machines in extremely harsh underground roadways.

Description

Technical Field

[0001] This invention relates to the field of intelligent mining technology, and in particular to a method, apparatus, equipment, and storage medium for constructing underground maps for underground environments such as coal mines and metal mines. Background Technology

[0002] Underground safety production heavily relies on real-time and accurate perception of complex underground environments. Traditional map-building methods suffer from the following limitations: static nature and blind spots, manifested in: reliance on pre-installed fixed sensor networks with limited coverage, inability to perceive dynamic changes in undeployed areas, and numerous blind spots; sensor vulnerability, manifested in: single or a few types of sensors are prone to failure under extreme underground conditions such as dust, humidity, darkness, and electromagnetic interference, resulting in poor system robustness; data isolation, manifested in: data from different sensors are typically displayed in isolation, failing to be deeply coupled with spatial location, and unable to form a situation map for automated machine decision-making; and high bandwidth dependence, manifested in: mobile inspection equipment needs to transmit raw sensor data, such as point clouds, video, cylinder, and rotation data, back to the central server, placing extremely high demands on the bandwidth of the underground communication network and making real-time updates difficult.

[0003] Therefore, there is an urgent need for a solution that can adapt to the harsh underground environment, achieve intelligent fusion of multi-source information, and generate truly dynamic maps. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for constructing downhole maps, aiming to achieve highly robust and accurate construction of downhole maps.

[0005] Firstly, this application provides a method for constructing a downhole map, including:

[0006] Acquire multi-source data collected by a multi-source heterogeneous sensor cluster; the multi-source heterogeneous sensor cluster is deployed on a tunneling machine operating in an underground roadway; the multi-source heterogeneous sensor cluster includes lidar, millimeter-wave radar, thermal imaging camera, binocular camera and IMU module; the multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data and tunneling machine inertial navigation data;

[0007] Adaptive preprocessing of multi-source data is performed according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, which is then uploaded to the edge computing system. The multi-source feature data includes key feature points extracted from LiDAR point cloud data, depth images and semantic segmentation results extracted from binocular vision data, inertial navigation feature data extracted from inertial navigation data, temperature distribution summary information extracted from thermal imaging image data, and moving target trajectory information of obstacles. The moving target trajectory information of obstacles is obtained from millimeter-wave radar point cloud data.

[0008] Key feature points and inertial navigation feature data are used in real time for localization and map building on an edge computing system using a SLAM framework to obtain a real-time local point cloud map of the underground roadway.

[0009] Based on the depth image and semantic segmentation results, the real-time local point cloud map is processed on the edge computing system to obtain a real-time local point cloud map with semantic labels, and the multi-source feature data and the real-time local point cloud map with semantic labels are sent to the central processing server.

[0010] Based on adaptive weights, multi-source feature data are tightly coupled and fused on the central processing server to obtain real-time local fusion data of underground roadways;

[0011] Based on real-time local fusion data from underground tunnels, the real-time local point cloud map with semantic labels is optimized on the central processing server to obtain a real-time high-precision local map with time stamps and semantic labels.

[0012] Real-time local high-precision maps corresponding to different time series are fused and optimized on the central processing server to obtain a global map of the underground roadways.

[0013] Optionally, the edge computing system includes a first edge computing unit, a second edge computing unit, and a third edge computing unit; it performs adaptive preprocessing on multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and uploads the multi-source feature data to the edge computing system, including:

[0014] For lidar point cloud data in multi-source data, a voxel grid filter is used to downsample the original lidar point cloud acquired by lidar to obtain downsampled lidar point cloud data.

[0015] Key feature points are extracted from the downsampled lidar point cloud data based on the curvature feature extraction algorithm. The key feature points include edge points corresponding to point clouds with curvature values ​​greater than a first preset curvature value, and planar points corresponding to point clouds with curvature values ​​less than a second preset curvature value.

[0016] Based on the number of key feature points, ideal point cloud density, and lidar scanning area, the point cloud density confidence level corresponding to lidar point cloud data is obtained.

[0017] The final confidence level of the lidar point cloud data is obtained based on the confidence level of the point cloud density, the confidence level of the lidar's health status, and the confidence level of the lidar's effective return rate. The confidence level of the lidar's effective return rate is based on the total number of emitted laser beams and the number of effectively returned point clouds. The confidence level of the lidar's health status is calculated based on the lidar's self-test parameters.

[0018] Based on the final confidence level corresponding to the lidar point cloud data, determine whether to upload the key feature points to the first edge computing unit;

[0019] And / or, for millimeter-wave radar point cloud data in multi-source data, a clustering algorithm is used to cluster the original millimeter-wave radar point cloud acquired by the millimeter-wave radar to obtain clustered point cloud clusters, so as to determine obstacles based on each point cloud cluster;

[0020] Based on a multi-target tracking algorithm, the system performs inter-frame correlation and state prediction for each obstacle to obtain a list of moving target trajectories for each obstacle. The list of moving target trajectories includes the position information, velocity information, and direction of motion information of the obstacle.

[0021] For each obstacle, the predicted position of the obstacle in the next moment is determined based on the list of moving target trajectories corresponding to the obstacle and the position information of the obstacle in the current frame.

[0022] Based on the predicted position of the obstacle in the next moment, the trajectory information of the moving target corresponding to the obstacle is obtained;

[0023] For each obstacle, the association consistency confidence of the obstacle is obtained based on the association success rate of the obstacle between consecutive frames;

[0024] For each obstacle, the final confidence level of the millimeter-wave radar point cloud data corresponding to the obstacle is obtained based on the correlation consistency confidence level and the signal-to-noise ratio confidence level; the signal-to-noise ratio confidence level is calculated based on the effective signal power and noise power of the millimeter-wave radar.

[0025] Based on the final confidence level of the millimeter-wave radar point cloud data corresponding to each obstacle, determine whether to upload the trajectory information of the moving target corresponding to the obstacle to the first edge computing unit.

[0026] And / or, for thermal imaging image data in multi-source data, a heat source detection algorithm based on Otsu threshold segmentation is used to extract temperature anomaly regions in the thermal imaging image data, and obtain temperature distribution summary information corresponding to each temperature anomaly region; the temperature distribution summary information includes the highest temperature, average temperature and temperature distribution histogram.

[0027] The final confidence level of the thermal imaging image data is obtained based on the difference between the target ambient temperature and the current ambient temperature.

[0028] Based on the final confidence level corresponding to the thermal imaging image data, determine whether to upload the temperature distribution summary information corresponding to each temperature anomaly area to the second edge computing unit.

[0029] And / or, for binocular visual data in multi-source data, calculate the binocular visual data based on the SGM stereo matching algorithm to generate left and right eye disparity maps;

[0030] The left and right disparity maps are sequentially subjected to left-right consistency checks and weighted least squares optimization to remove mismatched points in the left and right disparity maps, thus obtaining the final left and right disparity maps.

[0031] Based on the left and right eye disparity maps, the depth image and semantic segmentation results in the binocular coordinate system are obtained;

[0032] Based on the image sharpness confidence and illumination intensity confidence of the left and right eye disparity maps, the final confidence level corresponding to the binocular vision data is obtained;

[0033] Based on the final confidence level corresponding to the binocular vision data, determine whether to upload the depth image and semantic segmentation results to the third edge computing unit.

[0034] Optionally, key feature points and inertial navigation feature data are used in real-time for localization and map building on an edge computing system using a SLAM framework to obtain a real-time local point cloud map corresponding to the underground roadway, including:

[0035] Based on the preset neighborhood radius, the lidar feature points corresponding to the real-time frame are extracted to obtain the real-time ISS feature points;

[0036] For each real-time ISS feature point, the SHOT descriptor corresponding to the real-time ISS feature point is calculated according to the preset feature dimension.

[0037] The similarity between the SHOT descriptor corresponding to the real-time ISS feature point of the current frame and the SHOT descriptor corresponding to the real-time ISS feature point of the previous frame is calculated to establish the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM front end.

[0038] By using the SLAM backend to construct a point cloud map based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM frontend, as well as the inertial navigation feature data, a real-time local point cloud map of the underground roadway corresponding to the current location of the tunneling machine is obtained.

[0039] Optionally, based on the depth image and semantic segmentation results, the real-time local point cloud map is processed on an edge computing system to obtain a real-time local point cloud map with semantic labels, including:

[0040] Based on the semantic segmentation results and the depth information in the depth image, a set of target objects with semantic labels is generated;

[0041] Based on semantic segmentation results and depth images, dense point clouds are generated. Coordinate transformation and point cloud fusion are then performed on the dense point clouds to obtain a semantic point cloud map corresponding to the underground roadway.

[0042] Based on the semantic point cloud map corresponding to the underground tunnel, the location information of each semantic tag on the semantic point cloud map is obtained;

[0043] The position information of each semantic label on the semantic point cloud map is projected onto the real-time local point cloud map through coordinate transformation to obtain a real-time local point cloud map with semantic labels.

[0044] Optionally, based on adaptive weights, multi-source feature data are tightly coupled and fused on a central processing server to obtain real-time local fused data of the underground roadway, including:

[0045] Spatiotemporal synchronization is performed on each source feature data in the multi-source feature data to obtain spatiotemporally synchronized multi-source feature data, and a time stamp is added to the spatiotemporally synchronized multi-source feature data.

[0046] Each source feature data in the spatiotemporally synchronized multi-source feature data is evaluated to determine the fusion weight corresponding to each source feature data.

[0047] Based on the fusion weights corresponding to each source feature data, multi-source feature data are tightly coupled and fused to obtain real-time local fused data with time stamps.

[0048] Optionally, based on real-time local fusion data from underground roadways, the real-time local point cloud map with semantic labels is optimized on a central processing server to obtain a real-time high-precision local map with time stamps and semantic labels, including:

[0049] Voxel grid filtering is used to sample real-time local fusion data to obtain real-time local fusion data with uniform density; the real-time local fusion data includes key feature points, as well as the corresponding 3D coordinates, semantic labels and confidence scores, and timestamps of the key feature points;

[0050] Real-time local fusion data is input into the laser-inertial SLAM algorithm to optimize the real-time local point cloud map, resulting in a real-time high-precision local map with time stamps and semantic labels.

[0051] Optionally, the real-time local high-precision maps corresponding to different time series are fused and optimized on the central processing server to obtain a global map of the underground roadways, including:

[0052] Cross-modal semantic alignment is performed on real-time local high-precision maps corresponding to different time series to obtain real-time local high-precision maps in a unified semantic space.

[0053] Based on the inertial navigation feature data of the tunneling machine and the corresponding time stamps of each real-time local high-precision map, the real-time local high-precision maps under a unified semantic space are fused to obtain a global map of the underground roadway.

[0054] Secondly, this application provides an apparatus for constructing downhole maps, comprising:

[0055] The acquisition module is used to acquire multi-source data collected by the multi-source heterogeneous sensor cluster. The multi-source heterogeneous sensor cluster is deployed on the tunneling machine operating in the underground roadway. The multi-source heterogeneous sensor cluster includes lidar, millimeter-wave radar, thermal imaging camera, binocular camera and IMU module. The multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data and inertial navigation data of the tunneling machine.

[0056] An adaptive preprocessing module is used to adaptively preprocess multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and then upload the multi-source feature data to the edge computing system. The multi-source feature data includes key feature points extracted from LiDAR point cloud data, depth images and semantic segmentation results extracted from binocular vision data, inertial navigation feature data extracted from inertial navigation data, temperature distribution summary information extracted from thermal imaging image data, and moving target trajectory information of obstacles. The moving target trajectory information of obstacles is obtained from millimeter-wave radar point cloud data.

[0057] The real-time local point cloud map generation module is used to perform real-time localization and map construction on the edge computing system using the SLAM framework, based on key feature points and inertial navigation feature data, to obtain the real-time local point cloud map corresponding to the underground roadway.

[0058] The real-time local point cloud map optimization module is used to process the real-time local point cloud map on the edge computing system based on the depth image and semantic segmentation results, to obtain a real-time local point cloud map with semantic labels, and send the multi-source feature data and the real-time local point cloud map with semantic labels to the central processing server.

[0059] The real-time local fusion data calculation module is used to perform tight coupling fusion of multi-source feature data on the central processing server based on adaptive weights to obtain real-time local fusion data of underground roadways.

[0060] The real-time local high-precision map generation module is used to optimize the real-time local point cloud map with semantic tags on the central processing server based on the real-time local fusion data of the underground roadway, so as to obtain a real-time local high-precision map with time stamp and semantic tags.

[0061] The global map generation module is used to fuse and optimize real-time local high-precision maps corresponding to different time series on the central processing server to obtain a global map of the underground roadway.

[0062] Thirdly, an electronic device is provided, including a memory and a processor, the memory for storing processor-executable instructions; the processor is configured to execute the executable instructions in the memory to implement the steps of the method as described in the first aspect or its various implementations.

[0063] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect or its various implementations.

[0064] The method for constructing underground maps provided in this application adaptively preprocesses multi-source data collected by a multi-source heterogeneous sensor cluster to obtain low-bandwidth multi-source feature data. This multi-source feature data is then tightly coupled and fused to obtain real-time local fusion data of the underground roadway. Even when some sensors in the multi-source heterogeneous sensor cluster are interfered with, the real-time local fusion data of the underground roadway can be used to optimize the real-time local point cloud map with semantic labels, still resulting in a real-time high-precision local map with time stamps and semantic labels. Finally, the real-time high-precision local maps corresponding to different time sequences are fused and optimized on a central processing server to obtain a highly robust global map of the underground roadway, ensuring the continuous operation capability of tunneling machines in extremely harsh underground roadways. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a structural schematic diagram of the application scenario provided in the embodiments of this application;

[0067] Figure 2 This is a flowchart illustrating the method for constructing a downhole map provided in an embodiment of this application;

[0068] Figure 3 This is a schematic diagram of the structure of the downhole map construction device provided in the embodiments of this application;

[0069] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0072] It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to:

[0073] In some possible ways, Figure 1 An application scenario diagram provided for an embodiment of this application, such as Figure 1As shown, this application scenario may include electronic device 110 and network device 120. Electronic device 110 can establish a connection with network device 120 through a wired network or a wireless network.

[0074] For example, electronic device 110 may be a desktop computer, laptop computer, tablet computer, etc., but is not limited thereto. Network device 120 may be a terminal device or a server, but is not limited thereto. In one embodiment of this application, electronic device 110 may send a request message to network device 120, which may be used to request the acquisition of multi-source data. Further, electronic device 110 may receive a response message sent by network device 120, which includes the acquisition of multi-source data.

[0075] also, Figure 1 An electronic device and a network device are given as examples, but in practice, other numbers of electronic devices and network devices may be included, and this application does not limit this.

[0076] In other possible implementations, the technical solution of this application may also be executed by the aforementioned electronic device 110, or by the aforementioned network device 120, and this application does not impose any restrictions on this.

[0077] After introducing the application scenarios of the embodiments of this application, the technical solution of this application will be described in detail below:

[0078] Figure 2 A flowchart illustrating a method for constructing a downhole map, as provided in this application embodiment, is available. This method can be implemented by, for example... Figure 1 The electronic device 110 shown performs, but is not limited to, its functions. For example... Figure 2 As shown, the method may include the following steps:

[0079] S210: Acquire multi-source data collected by a multi-source heterogeneous sensor cluster.

[0080] The multi-source heterogeneous sensor cluster is deployed on the tunneling machine operating in the underground roadway. The multi-source heterogeneous sensor cluster includes lidar, millimeter-wave radar, thermal imaging cameras, binocular cameras, and an IMU module. The multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data, and the tunneling machine's inertial navigation data.

[0081] It should be noted that the multi-source data is obtained by a cluster of heterogeneous sensors traveling with the tunneling machine in the underground tunnel. For example, when the tunneling machine reaches position A, the cluster of heterogeneous sensors collects data at position A. Therefore, the obtained multi-source data is the data corresponding to position A. Based on this multi-source data, local data of the underground tunnel corresponding to position A can be obtained. For example, based on lidar point cloud data, a local point cloud map of the underground tunnel can be obtained; based on millimeter-wave radar point cloud data, obstacles located near the tunneling machine can be detected.

[0082] Here, when acquiring multi-source data collected by a multi-source heterogeneous sensor cluster, the system monitors whether the lidar, millimeter-wave radar, thermal imaging camera, binocular camera, and IMU module are time-synchronized when acquiring data. When the time deviation exceeds the threshold of 2ms, the system automatically triggers a timestamp to recalibrate the multi-source heterogeneous sensor cluster, so that the time deviation of the lidar, millimeter-wave radar, thermal imaging camera, binocular camera, and IMU module when acquiring data is within a preset range.

[0083] S220. Perform adaptive preprocessing on the multi-source data according to the preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and upload the multi-source feature data to the edge computing system.

[0084] The multi-source feature data includes key feature points extracted from lidar point cloud data, depth images and semantic segmentation results extracted from binocular vision data, inertial navigation feature data extracted from inertial navigation data, temperature distribution summary information extracted from thermal imaging image data, and moving target trajectory information of obstacles; the moving target trajectory information of obstacles is obtained from millimeter-wave radar point cloud data.

[0085] Here, the adaptive bandwidth allocation strategy adopted when uploading multi-source feature data to the edge computing system is as follows: the amount of data uploaded is dynamically adjusted according to the communication link quality between the multi-source heterogeneous sensor cluster and the edge computing system. When bandwidth is limited, key feature points and security data corresponding to key feature points, such as semantic segmentation results corresponding to key feature points, are prioritized. Then, the Modbus protocol is used to serialize other data in the multi-source feature data to achieve a compression ratio of 40%-60%, so that each source feature data in the multi-source feature data is uploaded in the serialized order, ensuring that high-priority data in the high-source feature data is transmitted first.

[0086] S230. The key feature points and inertial navigation feature data are used to perform real-time localization and map construction on the edge computing system through the SLAM framework to obtain a real-time local point cloud map corresponding to the underground roadway.

[0087] Here, by using the SLAM framework to perform real-time localization and map building on an edge computing system using key feature points and inertial navigation feature data, laser-inertial mapping of real-time local point cloud maps can be achieved.

[0088] In this step, based on key feature points and inertial navigation feature data (IMU pre-integration data), a tightly coupled laser-inertial fusion SLAM framework is used for real-time localization and map construction to obtain a real-time local point cloud map corresponding to the underground roadway.

[0089] S240. Based on the depth image and semantic segmentation results, the real-time local point cloud map is processed on the edge computing system to obtain a real-time local point cloud map with semantic labels, and the multi-source feature data and the real-time local point cloud map with semantic labels are sent to the central processing server.

[0090] The real-time local point cloud map with semantic labels obtained in this step can provide multi-view environmental representation for tight coupling fusion in subsequent steps, so that the real-time local high-precision map obtained in subsequent steps has high accuracy.

[0091] Multi-source feature data and real-time local point cloud maps with semantic labels are uploaded to the central processing server via the underground industrial ring network communication protocol. The central server receives data packets from the edge computing system and fixed sensing nodes, and is able to perform cross-modal semantic alignment and fusion of multi-source feature data.

[0092] S250: Based on adaptive weights, multi-source feature data are tightly coupled and fused on the central processing server to obtain real-time local fusion data of underground roadways.

[0093] Before performing tight coupling fusion of multi-source feature data on the central processing server, each source feature data in the multi-source feature data is subjected to spatiotemporal synchronization processing.

[0094] When performing tight-coupled fusion of multi-source feature data on the central processing server, the process includes: establishing a semantic association model based on spatial features; when semantic conflicts occur, adopting an arbitration mechanism based on confidence weighting and spatial context verification to generate optimal semantic labels in order to obtain the final real-time local fusion data.

[0095] Here, when performing tight-coupled fusion of multi-source feature data on the central processing server, it may also include: establishing a state prediction model based on IMU pre-integration; treating key feature points, depth images and semantic segmentation results, inertial navigation feature data, temperature distribution summary information, and moving target trajectory information as different observation items; and adaptively adjusting the noise covariance matrix of each observation item. To achieve weight allocation, where, The value of ε is determined by a combination of factory parameter calibration and field measurements for each sensor; ε is set to 0.01 to prevent division by zero errors.

[0096] S260. Based on the real-time local fusion data of the underground roadway, the real-time local point cloud map with semantic labels is optimized on the central processing server to obtain a real-time local high-precision map with time stamps and semantic labels.

[0097] Here, the graph optimization framework GTSAM can be used to jointly optimize real-time local point cloud maps with semantic labels using high-confidence semantic landmarks as strong constraints, generating real-time local high-precision maps with time stamps and semantic labels.

[0098] S270. The real-time local high-precision maps corresponding to different time series are fused and optimized on the central processing server to obtain a global map of the underground roadway.

[0099] During the movement of the tunneling machine in the underground roadway, it can continuously perform fusion optimization based on the real-time local high-precision maps corresponding to different time series. That is, it can fuse and optimize the currently obtained real-time local high-precision map and the historical local high-precision map to obtain the global map of the underground roadway.

[0100] The local map is uploaded to the central processing server via the underground industrial ring network communication protocol. The central server receives data packets from multiple edge computing units and fixed sensing nodes, and performs cross-modal semantic alignment and fusion on the multi-source heterogeneous sensing data. This includes: establishing a semantic association model based on spatial features; and when semantic conflicts occur, adopting an arbitration mechanism based on confidence weighting and spatial context verification to generate optimal semantic labels. Using the graph optimization framework GTSAM, with high-confidence semantic landmarks as strong constraints, the trajectory and global map are jointly optimized to generate a globally consistent semantic map.

[0101] Here, when the central processing server performs fusion optimization on the real-time local high-precision map, it can compress and serialize the real-time local high-precision map, and then use a compression algorithm to convert the real-time local high-precision map into data packets, with the compression rate controlled below 30%. Here, the transmission protocol adopts a priority-based QoS mechanism to ensure that semantic segmentation results and key feature point data are transmitted first, and other data are transmitted second. In addition, the central processing server establishes a data verification mechanism to ensure the integrity and orderliness of data transmission through CRC32 check and data packet sequence number detection.

[0102] Using the above method, multi-source data collected by a multi-source heterogeneous sensor cluster is adaptively preprocessed to obtain low-bandwidth multi-source feature data. This multi-source feature data is then tightly coupled and fused to obtain real-time local fusion data of the underground roadway. This allows for the optimization of a real-time local point cloud map with semantic labels using real-time local fusion data of the underground roadway, even when some sensors in the multi-source heterogeneous sensor cluster are interfered with. This still yields a real-time high-precision local map with time stamps and semantic labels. Finally, the real-time high-precision local maps corresponding to different time sequences are fused and optimized on a central processing server to obtain a global map of the underground roadway with high robustness, thus ensuring the continuous operation capability of the tunneling machine in extremely harsh underground roadways.

[0103] In some possible implementations, the edge computing system includes a first edge computing unit, a second edge computing unit, and a third edge computing unit; adaptive preprocessing of multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and uploading the multi-source feature data to the edge computing system, may include the following steps:

[0104] S310. For the lidar point cloud data in the multi-source data, a voxel grid filter is used to downsample the original lidar point cloud acquired by the lidar to obtain the downsampled lidar point cloud data.

[0105] S320: Extract key feature points from downsampled lidar point cloud data based on curvature feature extraction algorithm.

[0106] Among them, key feature points include edge points corresponding to point clouds with curvature values ​​greater than a first preset curvature value, and planar points corresponding to point clouds with curvature values ​​less than a second preset curvature value.

[0107] S330. Based on the number of key feature points, ideal point cloud density, and LiDAR scanning area, the point cloud density confidence level corresponding to the LiDAR point cloud data is obtained.

[0108] Here, the formula for calculating the confidence level of point cloud density corresponding to LiDAR point cloud data is as follows:

[0109]

[0110] in, The confidence level of the point cloud density corresponding to the lidar point cloud data; The ideal point cloud density is set to 2000 points / square meter; The area scanned by the lidar; This represents the number of key feature points.

[0111] S340. Based on the confidence level of the point cloud density corresponding to the lidar point cloud data, the confidence level of the lidar's health status, and the confidence level of the lidar's effective return rate, the final confidence level corresponding to the lidar point cloud data is obtained.

[0112] Among them, the confidence level of the effective return rate of the lidar is obtained based on the total number of emitted laser beams and the number of effective returned point clouds; the confidence level of the lidar's health status is calculated based on the corresponding self-test parameters of the lidar.

[0113] Here, the formula for calculating the confidence level of the effective return rate of lidar is as follows:

[0114]

[0115] in, Confidence level for the effective return rate of lidar; This represents the effective return point cloud count of the lidar; This represents the total number of laser beams emitted by the lidar.

[0116] The formula for calculating the final confidence level of LiDAR point cloud data is as follows:

[0117]

[0118] in, The final confidence level corresponding to the lidar point cloud data; The confidence level of the point cloud density corresponding to the lidar point cloud data; Confidence level of the health status of the lidar; The confidence level of the effective return rate of the lidar.

[0119] S350. Based on the final confidence level corresponding to the lidar point cloud data, determine whether to upload the key feature points to the first edge computing unit.

[0120] Here, the decision to upload key feature points to the first edge computing unit can be made based on the final confidence level of the LiDAR point cloud data. For example, if the final confidence level of the LiDAR point cloud data is greater than a first threshold, the key feature points can be uploaded to the first edge computing unit.

[0121] S360. For millimeter-wave radar point cloud data in multi-source data, a clustering algorithm is used to cluster the original millimeter-wave radar point cloud collected by the millimeter-wave radar to obtain clustered point cloud clusters, so as to determine obstacles based on each point cloud cluster.

[0122] Here, the DBSCAN clustering algorithm is used to cluster the original millimeter-wave radar point cloud acquired by the millimeter-wave radar, where the clustering parameters eps=0.3 and min_samples=3.

[0123] S370. Based on the multi-target tracking algorithm, the association and state prediction of each obstacle are performed between consecutive frames to obtain a list of moving target trajectories corresponding to each obstacle.

[0124] The list of moving target trajectories includes the location, speed, and direction of movement information of the obstacles.

[0125] Here, a multi-target tracking algorithm based on Kalman filtering is used to perform inter-frame correlation and state prediction of each obstacle, generating a list of moving target trajectories corresponding to each obstacle.

[0126] S380. For each obstacle, based on the list of moving target trajectories corresponding to the obstacle and the obstacle's position information in the current frame, determine the predicted position of the obstacle in the next moment.

[0127] S390. Based on the predicted position of the obstacle at the next moment, obtain the trajectory information of the moving target corresponding to the obstacle.

[0128] S3100. For each obstacle, obtain the association consistency confidence of the obstacle based on the association success rate of the obstacle between consecutive frames.

[0129] S3110. For each obstacle, based on the association consistency confidence and signal-to-noise ratio confidence, obtain the final confidence of the millimeter-wave radar point cloud data corresponding to the obstacle.

[0130] The signal-to-noise ratio confidence level is calculated based on the effective signal power and noise power of the millimeter-wave radar.

[0131] Here, the formula for calculating the signal-to-noise ratio confidence level is as follows:

[0132] ;

[0133] in, For signal-to-noise ratio confidence level; Effective signal power; This represents noise power.

[0134] The formula for calculating the final confidence level of millimeter-wave radar point cloud data is as follows:

[0135] ;

[0136] in, The final confidence level corresponding to the millimeter-wave radar point cloud data; For signal-to-noise ratio confidence level; This represents the confidence level for association consistency.

[0137] S3120: Based on the final confidence level of the millimeter-wave radar point cloud data corresponding to each obstacle, determine whether to upload the trajectory information of the moving target corresponding to the obstacle to the first edge computing unit.

[0138] Here, the decision on whether to upload key feature points to the first edge computing unit can be made based on the final confidence level corresponding to the millimeter-wave radar point cloud data. For example, if the final confidence level corresponding to the millimeter-wave radar point cloud data is greater than the second threshold, it is determined that the trajectory information of the moving target corresponding to the obstacle should be uploaded to the first edge computing unit.

[0139] S3130. For thermal imaging image data in multi-source data, the heat source detection algorithm based on Otsu threshold segmentation is used to extract temperature anomaly regions in the thermal imaging image data and obtain temperature distribution summary information corresponding to each temperature anomaly region.

[0140] The temperature distribution summary information includes the highest temperature, average temperature, and temperature distribution histogram.

[0141] S3140. Based on the difference between the target ambient temperature and the current ambient temperature, obtain the final confidence level corresponding to the thermal imaging image data.

[0142] The formula for calculating the final confidence level corresponding to the thermal imaging image data is as follows:

[0143] ;

[0144] in, ΔT represents the final confidence level corresponding to the thermal imaging image data; ΔT is the difference between the target ambient temperature and the current ambient temperature, and the unit of ΔT is degrees Celsius.

[0145] S3150. Based on the final confidence level corresponding to the thermal imaging image data, determine whether to upload the temperature distribution summary information corresponding to each temperature anomaly area to the second edge computing unit.

[0146] Here, the decision on whether to upload key feature points to the second edge computing unit can be made based on the final confidence level corresponding to the thermal imaging image data. For example, if the final confidence level corresponding to the thermal imaging image data is greater than the third threshold, the decision is made to upload the temperature distribution summary information corresponding to each temperature anomaly region to the second edge computing unit.

[0147] S3160. For binocular visual data in multi-source data, calculate the binocular visual data based on the SGM stereo matching algorithm to generate left and right eye disparity maps.

[0148] S3170. Perform left-right consistency checks and weighted least squares optimization on the left and right eye disparity maps in sequence to remove mismatched points in the left and right eye disparity maps and obtain the final left and right eye disparity maps.

[0149] S3180. Based on the left and right eye disparity maps, obtain the depth image and semantic segmentation results in the binocular coordinate system.

[0150] S3190. Based on the image sharpness confidence and illumination intensity confidence of the left and right eye disparity maps, obtain the final confidence level corresponding to the binocular vision data.

[0151] Because the final confidence level of binocular vision data is positively correlated with image texture richness and illumination uniformity, and negatively correlated with the area of ​​repeating texture regions and the degree of illumination abrupt change. Among these, the image sharpness confidence level... The confidence level of illumination intensity can be calculated based on the Brenner gradient algorithm. It can be calculated from the average brightness of the image.

[0152] The formula for calculating the final confidence level for binocular vision data is as follows:

[0153] ;

[0154] in, The final confidence level corresponding to the binocular vision data; Image sharpness confidence level; represents the confidence level of light intensity.

[0155] S3200: Based on the final confidence level corresponding to the binocular vision data, determine whether to upload the depth image and semantic segmentation results to the third edge computing unit.

[0156] Here, the decision to upload key feature points to the third edge computing unit can be made based on the final confidence level of the binocular vision data. For example, if the final confidence level of the binocular vision data is greater than the fourth threshold, the decision is made to upload the depth image and semantic segmentation results to the third edge computing unit.

[0157] By employing the above method, differentiated adaptive preprocessing strategies are designed for different types of multi-source data, achieving accurate dimensionality reduction and feature extraction, and significantly reducing bandwidth consumption. The use of a modular architecture in an edge computing system enables the classified uploading and parallel processing of multi-source feature data, improving the real-time performance of data processing and the rationality of computing power allocation. It also achieves distributed preprocessing of multi-source data and modular processing of edge computing, reducing the computing load on the central processing server and network transmission pressure, thus improving the overall operating efficiency of the monitoring system. Furthermore, the combination of adaptive preprocessing and confidence screening of multi-source data enhances the robustness and environmental adaptability of the final multi-source feature data.

[0158] In some possible implementations, key feature points and inertial navigation feature data are used in real-time for localization and map building on an edge computing system using a SLAM framework to obtain a real-time local point cloud map corresponding to the underground roadway. This may include the following steps:

[0159] S410. Extract the LiDAR feature points corresponding to the real-time frame according to the preset neighborhood radius to obtain the real-time ISS feature points.

[0160] Preferably, the SLAM process employs a tightly coupled laser-inertial fusion framework.

[0161] The front end uses a feature matching method based on ISS feature points and SHOT descriptors, while the back end uses a joint optimization algorithm based on pose graph optimization. When constructing the local point cloud map, voxel grid filtering downsampling is used to maintain uniform map density. At the same time, the visual semantic segmentation results are projected onto the point cloud map through an extrinsic parameter matrix, and each feature point is assigned a semantic label.

[0162] Preferably, the local situation map generation includes:

[0163] Local map data is stored using an octree structure; each map element contains the following attributes: 1) 3D coordinates; 2) semantic label and confidence level; 3) timestamp; 4) dynamic flag; 5) environmental parameters; an incremental map update strategy is adopted, updating only the changed areas to reduce computational overhead.

[0164] Preferably, it includes an anomaly handling mechanism: the confidence level of a certain sensor at time t. Below the threshold When the system automatically reduces the weight of the sensor to zero and triggers a fault diagnosis procedure, the system automatically switches to the fusion mode of lidar-IMU-millimeter-wave radar when all visual sensors fail to ensure basic navigation functions.

[0165] S420. For each real-time ISS feature point, calculate the SHOT descriptor corresponding to the real-time ISS feature point according to the preset feature dimension.

[0166] S430. Calculate the similarity between the SHOT descriptor corresponding to the real-time ISS feature point of the current frame and the SHOT descriptor corresponding to the real-time ISS feature point of the previous frame, and establish the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM front end.

[0167] When using a feature matching method based on real-time ISS feature points and SHOT descriptors in the SLAM front-end, the preset neighborhood radius can be 0.3 and the preset feature dimension can be 352.

[0168] S440. Using the SLAM backend, a point cloud map is constructed based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM frontend, as well as the inertial navigation feature data, to obtain a real-time local point cloud map of the underground roadway corresponding to the current location of the tunneling machine.

[0169] In the SLAM backend, a joint optimization algorithm based on pose graph optimization is adopted to construct a point cloud map based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM frontend, as well as the inertial navigation feature data. The optimization frequency is 2.

[0170] Using the above method, the ISS feature points corresponding to the lidar feature points, SHOT descriptors, and inertial navigation feature data can be tightly coupled through the SLAM front end. The SLAM back end constructs a point cloud map based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM front end, as well as the inertial navigation feature data. This allows for the real-time output of a real-time local point cloud map of the underground roadway corresponding to the current location of the tunneling machine.

[0171] In some possible implementations, processing the real-time local point cloud map on an edge computing system based on the depth image and semantic segmentation results to obtain a real-time local point cloud map with semantic labels may include the following steps:

[0172] S510: Based on the semantic segmentation results and the depth information in the depth image, generate a set of target objects with semantic labels.

[0173] Here, based on the semantic segmentation results, depth images and their corresponding depth information, the coordinate transformation matrix is ​​used to transform them from the camera coordinate system to a unified carrier coordinate system, generating a set of target objects with semantic labels.

[0174] Specifically, the binocular vision data is read, and depth images in the binocular coordinate system are constructed respectively. Based on the depth images in the binocular coordinate system, the spatial coordinates of the target object in the binocular coordinate system are calculated. Then, the target object is identified by the target detection method, such as: large rocks, pedestrians, ladders, anchor rods, drilling equipment, etc.

[0175] Taking the left-side binocular visual data acquired by the left-side binocular camera as an example, the semantic segmentation results and depth information from the depth image are integrated into a vector. ,in , This represents the x and y coordinates of the center of the detection anchor frame for the k-th target object. This represents the Z-axis coordinate of the k-th target object in the binocular spatial coordinate system, which is the depth value of the target object's detection anchor box center in the depth image;

[0176] Calculate the spatial coordinates of the point cloud corresponding to each target object:

[0177] The pixel coordinates of the sidewall, bottom, and top plate regions of the scene were identified using semantic segmentation methods, as follows:

[0178] :

[0179] : ,

[0180] : ,

[0181] : ,

[0182] : ,

[0183] : ,in This represents the point cloud coordinate vector of the left sidewall point cloud in the left binocular coordinate system. This represents the point cloud coordinate vector of the left base plate in the left binocular coordinate system. This represents the point cloud coordinate vector of the left top plate in the left binocular coordinate system. This represents the point cloud coordinate vector of the right sidewall in the right-side binocular coordinate system. This represents the point cloud coordinate vector of the right-side base plate in the right-side binocular coordinate system. This represents the point cloud coordinate vector of the right top plate in the right binocular coordinate system.

[0184] The depth information in the depth image is obtained by median filtering of the disparity map generated by the SGBM algorithm.

[0185] S520. Based on the semantic segmentation results and depth images, a dense point cloud is generated. The dense point cloud is then subjected to coordinate transformation and point cloud fusion processing in sequence to obtain a semantic point cloud map corresponding to the underground roadway.

[0186] Here, the semantic segmentation results can include masks for the sidewalls, bottom plate, and top plate regions. These semantic segmentation results are obtained by inputting binocular vision data into the DeepLabV3 semantic segmentation model.

[0187] It should be noted that the semantic point cloud map obtained here does not include a local semantic map with global pose optimization and reference to the instantaneous carrier coordinate system, and the semantic point cloud map contains the precise location of the target object and the geometry of the alley.

[0188] When performing coordinate transformation on dense point clouds, the extrinsic parameter matrix from the binocular camera to the IMU coordinate system obtained by hand-eye calibration is used. Point cloud fusion adopts the nearest neighbor deduplication algorithm based on voxel grid, and the voxel size is set to 0.03.

[0189] S530. Based on the semantic point cloud map corresponding to the underground roadway, obtain the location information of each semantic label on the semantic point cloud map.

[0190] S540. Project the position information of each semantic label on the semantic point cloud map onto the real-time local point cloud map through coordinate transformation to obtain a real-time local point cloud map with semantic labels.

[0191] Using the above method, the resulting real-time local point cloud map with semantic labels can provide multi-view environmental representation for subsequent tightly coupled fusion, so that the final global map of the underground roadway has high accuracy.

[0192] In some possible implementations, based on adaptive weights, multi-source feature data is tightly coupled and fused on a central processing server to obtain real-time local fused data of the underground roadway. This may include the following steps:

[0193] S610. Perform spatiotemporal synchronization on each source feature data in the multi-source feature data to obtain spatiotemporally synchronized multi-source feature data, and add a time stamp to the spatiotemporally synchronized multi-source feature data.

[0194] Here, each source feature data in the multi-source feature data is unified to a common time reference with the IMU as the core through hardware trigger signals and a timestamp synchronization mechanism based on a precise time protocol; the external parameter matrix of each source feature data relative to the IMU is accurately obtained by hand-eye calibration method, and each source feature data in all the multi-source feature data is transformed to a unified carrier coordinate system through coordinate transformation.

[0195] S620. Evaluate each source feature data in the multi-source feature data after spatiotemporal synchronization, and determine the fusion weight corresponding to each source feature data.

[0196] S630. Based on the fusion weights corresponding to each source feature data, perform tight coupling fusion on the multi-source feature data to obtain real-time local fusion data with time stamps.

[0197] Among them, each source feature data in the spatiotemporally synchronized multi-source feature data can be evaluated based on a multi-factor evaluation model to obtain the fusion weight corresponding to each source feature data.

[0198] Specifically, the multi-factor assessment model adopts a three-layer architecture: the first layer is sensor-level confidence assessment, outputting the real-time confidence scores of each sensor. The second layer is historical health status assessment, using an exponentially weighted moving average model. Where λ=0.9; the third layer is the task-related weight allocation, which dynamically adjusts the weight coefficients of each sensor according to the current navigation task requirements of the tunneling machine.

[0199] Here, an extended Kalman filter framework can be used to tightly couple and fuse multi-source feature data.

[0200] It should be noted that when performing tightly coupled fusion of multi-source feature data, this includes: dynamically adjusting the noise covariance matrix of each sensor measurement model in a multi-source heterogeneous sensor cluster within a SLAM framework based on error state Kalman filtering (ESKF). To achieve fusion weight allocation, where the noise covariance matrix... The calculation formula is as follows:

[0201]

[0202] in, This is the fundamental measurement noise matrix of one sensor in a multi-source heterogeneous sensor cluster under optimal conditions. ε is a very small positive number to prevent division by zero errors. This represents the overall confidence weight. Therefore, it can be seen that the higher the fusion weight, the greater the corresponding confidence level. The smaller the value, the more the filtering algorithm will trust the sensor's observations at that moment. Here, the sensors in the multi-source heterogeneous sensor cluster include LiDAR, millimeter-wave radar, thermal imaging cameras, binocular cameras, and IMU modules.

[0203] Furthermore, the comprehensive confidence weight calculation process includes: for each sensor i at each time t, its comprehensive confidence weight... Based on its immediate confidence level Historical health status (A factor that decays smoothly) and its criticality in the current task. (Preset weights) jointly determine:

[0204] Where α, β, γ are adjustable hyperparameters that satisfy α + β + γ = 1. Based on the sensor's historical data over a period of time Update.

[0205] For example, when the ambient light dims, the final confidence level corresponding to the binocular vision data... During a sharp decline, Decrease As the filter size increases, it naturally reduces the contribution of visual observation, while relying more on data from lidar and millimeter-wave radar, which are less affected by light.

[0206] The above method can improve the spatiotemporal consistency of multi-source data and eliminate fusion errors caused by data heterogeneity. At the same time, it can also realize adaptive dynamic adjustment of fusion weights, improving the reliability and robustness of fused data. In addition, by combining tightly coupled fusion with adaptive weights, it can also ensure the high accuracy and real-time performance of fused data, so that the final real-time local fused data can be adapted to the complex and harsh downhole application environment, and has strong engineering practicality.

[0207] In some possible implementations, based on real-time local fusion data of underground roadways, the real-time local point cloud map with semantic labels is optimized on a central processing server to obtain a real-time high-precision local map with time stamps and semantic labels. This may include the following steps:

[0208] S710. Voxel grid filtering is used to sample real-time local fusion data to obtain real-time local fusion data with uniform density.

[0209] The real-time local fusion data includes key feature points, their corresponding 3D coordinates, semantic labels and confidence levels, and timestamps.

[0210] Here, when sampling real-time local fusion data using voxel grid filtering, the voxel size is set to 0.05 to maintain uniform map density and reduce the computational load on real-time local fusion data.

[0211] S720: Input the real-time local fusion data into the laser-inertial SLAM algorithm to optimize the real-time local point cloud map and obtain a real-time high-precision local map with time stamps and semantic labels.

[0212] Using the above method, pose estimation and key feature points from real-time local fusion data are input into a tightly coupled laser-inertial SLAM algorithm to generate a real-time dense or semi-dense local point cloud map. Simultaneously, the semantic segmentation results are projected onto the real-time local point cloud map through coordinate transformation, assigning semantic labels to points or regions in the map, forming a real-time high-precision local map with temporal stamps and semantic labels, thus improving the accuracy and robustness of the local map.

[0213] In some possible implementations, fusing and optimizing real-time local high-precision maps corresponding to different time series on a central processing server to obtain a global map of the underground roadway may include the following steps:

[0214] S810. Perform cross-modal semantic alignment processing on real-time local high-precision maps corresponding to different time series to obtain real-time local high-precision maps under a unified semantic space.

[0215] S820. Based on the inertial navigation feature data of the tunneling machine and the time stamps corresponding to each real-time local high-precision map, the real-time local high-precision maps under the unified semantic space are fused to obtain the global map of the underground roadway.

[0216] By employing the above method, cross-modal semantic alignment is used to eliminate the semantic heterogeneity of real-time local high-precision maps corresponding to different time series, thereby achieving semantic unification of the global map. By combining inertial navigation feature data and time stamps, accurate spatiotemporal registration of local maps can be achieved, improving the stitching accuracy and coherence of the global map. At the same time, by dynamically fusing local high-precision maps from different time series, real-time updates and dynamic improvements of the global map can be achieved, thereby enhancing the reliability and anti-interference capability of the global map and adapting it to the complex and harsh application environment downhole.

[0217] Please see Figure 3 This is a schematic diagram of the structure of the downhole map construction device provided in the embodiments of this application. A second aspect of this application provides a downhole map construction device, applied to the downhole map construction device provided in the first aspect. The device includes:

[0218] The acquisition module 910 is used to acquire multi-source data collected by the multi-source heterogeneous sensor cluster. The multi-source heterogeneous sensor cluster is deployed on a tunneling machine operating in an underground roadway. The multi-source heterogeneous sensor cluster includes lidar, millimeter-wave radar, thermal imaging camera, binocular camera, and IMU module. The multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data, and inertial navigation data of the tunneling machine.

[0219] The adaptive preprocessing module 920 is used to adaptively preprocess multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and then upload the multi-source feature data to the edge computing system. The multi-source feature data includes key feature points extracted from lidar point cloud data, depth images and semantic segmentation results extracted from binocular vision data, inertial navigation feature data extracted from inertial navigation data, temperature distribution summary information extracted from thermal imaging image data, and moving target trajectory information of obstacles. The moving target trajectory information of obstacles is obtained from millimeter-wave radar point cloud data.

[0220] The real-time local point cloud map generation module 930 is used to perform real-time local point cloud map generation on the edge computing system using the SLAM framework to generate key feature points and inertial navigation feature data, thereby obtaining the real-time local point cloud map corresponding to the underground roadway.

[0221] The real-time local point cloud map optimization module 940 is used to process the real-time local point cloud map on the edge computing system based on the depth image and semantic segmentation results, to obtain a real-time local point cloud map with semantic labels, and to send the multi-source feature data and the real-time local point cloud map with semantic labels to the central processing server.

[0222] The real-time local fusion data calculation module 950 is used to perform tight coupling fusion of multi-source feature data on the central processing server based on adaptive weights to obtain real-time local fusion data of underground roadways.

[0223] The real-time local high-precision map generation module 960 is used to optimize the real-time local point cloud map with semantic tags on the central processing server based on the real-time local fusion data of the underground roadway, so as to obtain a real-time local high-precision map with time stamp and semantic tags.

[0224] The global map generation module 970 is used to fuse and optimize real-time local high-precision maps corresponding to different time series on the central processing server to obtain a global map of the underground roadway.

[0225] Optionally, the edge computing system includes a first edge computing unit, a second edge computing unit, and a third edge computing unit; the adaptive preprocessing module 920 includes:

[0226] The downsampling unit is used to downsample the original lidar point cloud data acquired by lidar using a voxel grid filter to obtain downsampled lidar point cloud data from multi-source data.

[0227] The key feature point extraction unit is used to extract key feature points from the downsampled lidar point cloud data based on the curvature feature extraction algorithm. The key feature points include edge points corresponding to point clouds with curvature values ​​greater than a first preset curvature value, and planar points corresponding to point clouds with curvature values ​​less than a second preset curvature value.

[0228] The point cloud density confidence calculation unit is used to obtain the point cloud density confidence of the lidar point cloud data based on the number of key feature points, the ideal point cloud density, and the lidar scanning area.

[0229] The first final confidence calculation unit is used to obtain the final confidence level corresponding to the lidar point cloud data based on the point cloud density confidence level, the lidar health status confidence level, and the lidar effective return rate confidence level. The lidar effective return rate confidence level is obtained based on the total number of emitted laser beams and the number of effectively returned point clouds. The lidar health status confidence level is calculated based on the lidar's corresponding self-test parameters.

[0230] The first data uploading unit is used to determine whether to upload key feature points to the first edge computing unit based on the final confidence level corresponding to the lidar point cloud data.

[0231] And / or, an obstacle determination unit, used to cluster the original millimeter-wave radar point cloud data collected by millimeter-wave radar using a clustering algorithm to obtain clustered point cloud clusters, so as to determine obstacles based on each point cloud cluster;

[0232] The moving target trajectory list generation unit is used to perform inter-frame correlation and state prediction of each obstacle based on a multi-target tracking algorithm to obtain a moving target trajectory list corresponding to each obstacle; the moving target trajectory list includes the position information, velocity information and movement direction information of the obstacle;

[0233] The predicted position determination unit is used to determine the predicted position of each obstacle in the next moment based on the list of moving target trajectories corresponding to the obstacle and the position information of the obstacle in the current frame.

[0234] The moving target trajectory information determination unit is used to obtain the moving target trajectory information corresponding to the obstacle based on the predicted position of the obstacle in the next moment;

[0235] The association consistency confidence calculation unit is used to obtain the association consistency confidence of each obstacle based on the association success rate of the obstacle between consecutive frames.

[0236] The second final confidence calculation unit is used to obtain the final confidence of the millimeter-wave radar point cloud data corresponding to each obstacle based on the correlation consistency confidence and the signal-to-noise ratio confidence; the signal-to-noise ratio confidence is calculated based on the effective signal power and noise power of the millimeter-wave radar.

[0237] The second data uploading unit is used to determine whether to upload the trajectory information of the moving target corresponding to the obstacle to the first edge computing unit based on the final confidence level of the millimeter-wave radar point cloud data corresponding to each obstacle.

[0238] And / or, a temperature distribution summary information determination unit is used to extract temperature anomaly regions from thermal imaging image data in multi-source data based on the Otsu threshold segmentation heat source detection algorithm, and obtain temperature distribution summary information corresponding to each temperature anomaly region; the temperature distribution summary information includes the highest temperature, average temperature and temperature distribution histogram.

[0239] The third final confidence calculation unit is used to obtain the final confidence level corresponding to the thermal imaging image data based on the difference between the target ambient temperature and the current ambient temperature.

[0240] The third data uploading unit is used to determine whether to upload the temperature distribution summary information corresponding to each temperature anomaly area to the second edge computing unit based on the final confidence level corresponding to the thermal imaging image data.

[0241] And / or, a left and right eye disparity map generation unit, used to calculate the left and right eye disparity maps for binocular visual data in multi-source data based on the SGM stereo matching algorithm;

[0242] The left and right eye disparity map optimization unit is used to perform left and right consistency checks and weighted least squares optimization on the left and right eye disparity maps in sequence to remove mismatched points in the left and right eye disparity maps and obtain the final left and right eye disparity maps.

[0243] The extraction unit is used to obtain the depth image and semantic segmentation results in the binocular coordinate system based on the left and right eye disparity maps;

[0244] The fourth final confidence calculation unit is used to obtain the final confidence of the binocular vision data based on the image sharpness confidence and illumination intensity confidence of the left and right eye disparity maps.

[0245] The fourth data uploading unit is used to determine whether to upload the depth image and semantic segmentation results to the third edge computing unit based on the final confidence level corresponding to the binocular vision data.

[0246] Optionally, the real-time local point cloud map generation module 930 includes:

[0247] The real-time ISS feature point extraction unit is used to extract the lidar feature points corresponding to the real-time frame according to the preset neighborhood radius to obtain the real-time ISS feature points.

[0248] The SHOT descriptor calculation unit is used to calculate the SHOT descriptor corresponding to each real-time ISS feature point according to a preset feature dimension.

[0249] The correspondence establishment unit is used to calculate the similarity between the SHOT descriptor corresponding to the real-time ISS feature point of the current frame and the SHOT descriptor corresponding to the real-time ISS feature point of the previous frame, and establish the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM front end.

[0250] The real-time local point cloud map construction unit is used to construct a point cloud map by utilizing the SLAM backend based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM frontend, as well as the inertial navigation feature data, to obtain a real-time local point cloud map of the underground roadway corresponding to the current location of the tunneling machine.

[0251] Optionally, the real-time local point cloud map optimization module 940 includes:

[0252] The target object set generation unit is used to generate a set of target objects with semantic labels based on the semantic segmentation results and depth information in the depth image;

[0253] The semantic point cloud map acquisition unit is used to generate dense point clouds based on semantic segmentation results and depth images, and to perform coordinate transformation and point cloud fusion processing on the dense point clouds in sequence to obtain the semantic point cloud map corresponding to the underground roadway.

[0254] The location information acquisition unit is used to obtain the location information of each semantic tag on the semantic point cloud map based on the semantic point cloud map corresponding to the underground roadway;

[0255] The real-time local point cloud map acquisition unit is used to project the position information of each semantic label on the semantic point cloud map onto the real-time local point cloud map through coordinate transformation, so as to obtain a real-time local point cloud map with semantic labels.

[0256] Optionally, the real-time local fusion data computing module 950 includes:

[0257] The spatiotemporal synchronization unit is used to perform spatiotemporal synchronization on each source feature data in the multi-source feature data to obtain spatiotemporally synchronized multi-source feature data, and add a time stamp to the spatiotemporally synchronized multi-source feature data.

[0258] The fusion weight determination unit is used to evaluate each source feature data in the multi-source feature data after spatiotemporal synchronization and determine the fusion weight corresponding to each source feature data.

[0259] The tightly coupled fusion unit is used to perform tightly coupled fusion of multi-source feature data based on the fusion weight corresponding to each source feature data, so as to obtain real-time local fusion data with time stamp.

[0260] Optionally, the real-time local high-precision map generation module 960 includes:

[0261] The sampling processing unit is used to sample the real-time local fusion data using voxel grid filtering to obtain real-time local fusion data with uniform density. The real-time local fusion data includes key feature points, as well as the corresponding three-dimensional coordinates, semantic labels and confidence scores, and timestamps of the key feature points.

[0262] The map optimization unit is used to input real-time local fusion data into the laser-inertial SLAM algorithm to optimize the real-time local point cloud map and obtain a real-time high-precision local map with time stamps and semantic labels.

[0263] Optionally, the global map generation module 970 includes:

[0264] The alignment processing unit is used to perform cross-modal semantic alignment processing on real-time local high-precision maps corresponding to different time series to obtain real-time local high-precision maps in a unified semantic space.

[0265] The global map generation unit is used to fuse various real-time local high-precision maps in a unified semantic space based on the inertial navigation feature data of the tunneling machine and the time stamps corresponding to each real-time local high-precision map, so as to obtain a global map of the underground roadway.

[0266] This disclosure provides an embodiment of an electronic device. Optionally, the electronic device includes a memory for storing processor-executable instructions; a processor configured to execute the executable instructions in the memory to implement the steps of the electronic device control method provided in this disclosure.

[0267] Figure 4 This is a schematic block diagram of an electronic device 110 according to an embodiment of the present invention.

[0268] like Figure 4 As shown, the electronic device 110 may include an electronic device, and the electronic device 110 may further include:

[0269] The system includes a memory 1101 and a processor 1102. The memory 1101 stores computer programs and transfers the program code to the processor 1102. In other words, the processor 1102 can retrieve and run the computer programs from the memory 1101 to implement the methods described in the embodiments of the present invention.

[0270] For example, the processor 1102 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0271] In some embodiments of the present invention, the electronic device 110 may include, but is not limited to:

[0272] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0273] In some embodiments of the present invention, the memory 1101 includes, but is not limited to:

[0274] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0275] In some embodiments of the present invention, the computer program may be divided into one or more modules, which are stored in the memory 1101 and executed by the processor 1102 to perform the method provided by the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the controller.

[0276] like Figure 4 As shown, the electronic device 110 may further include:

[0277] Transceiver 1103, which can be connected to processor 1102 or memory 1101.

[0278] The processor 1102 can control the transceiver 1103 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 1103 may include a transmitter and a receiver. The transceiver 1103 may further include antennas, and the number of antennas may be one or more.

[0279] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0280] The present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, one embodiment of the present invention also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0281] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Video Disc (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)).

[0282] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0283] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0284] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0285] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing an underground map, characterized in that, include: The system acquires multi-source data collected by a multi-source heterogeneous sensor cluster, which is deployed on a tunneling machine operating in an underground roadway. The multi-source heterogeneous sensor cluster includes a lidar, a millimeter-wave radar, a thermal imaging camera, a binocular camera, and an IMU module. The multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data, and inertial navigation data of the tunneling machine. Adaptive preprocessing is performed on multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, which is then uploaded to the edge computing system. The multi-source feature data includes key feature points extracted from the lidar point cloud data, depth images and semantic segmentation results extracted from the binocular vision data, inertial navigation feature data extracted from the inertial navigation data, temperature distribution summary information extracted from the thermal imaging image data, and moving target trajectory information of obstacles. The moving target trajectory information of obstacles is obtained based on the millimeter-wave radar point cloud data. The key feature points and the inertial navigation feature data are used in real time for localization and map construction on the edge computing system through the SLAM framework to obtain a real-time local point cloud map corresponding to the underground roadway. Based on the depth image and semantic segmentation results, the real-time local point cloud map is processed on the edge computing system to obtain a real-time local point cloud map with semantic labels, and the multi-source feature data and the real-time local point cloud map with semantic labels are sent to the central processing server. Based on adaptive weights, the multi-source feature data is tightly coupled and fused on the central processing server to obtain real-time local fusion data of the underground roadway. Based on the real-time local fusion data of the underground roadway, the real-time local point cloud map with semantic tags is optimized on the central processing server to obtain a real-time local high-precision map with time stamps and semantic tags. The real-time local high-precision maps corresponding to different time series are fused and optimized on the central processing server to obtain a global map of the underground roadway; The step of performing tight-coupled fusion of the multi-source feature data on the central processing server based on adaptive weights to obtain real-time local fusion data of the underground roadway includes: Spatiotemporal synchronization is performed on each source feature data in the multi-source feature data to obtain spatiotemporally synchronized multi-source feature data, and a time stamp is added to the spatiotemporally synchronized multi-source feature data. Each source feature data in the multi-source feature data after spatiotemporal synchronization is evaluated to determine the fusion weight corresponding to each source feature data. Based on the fusion weight corresponding to each source feature data, the multi-source feature data are tightly coupled and fused to obtain real-time local fused data with time stamps; The step of optimizing the real-time local point cloud map with semantic labels on the central processing server based on the real-time local fusion data of the underground roadway to obtain a real-time high-precision local map with time stamps and semantic labels includes: The real-time local fusion data is sampled using voxel grid filtering to obtain real-time local fusion data with uniform density; the real-time local fusion data includes key feature points, as well as the corresponding three-dimensional coordinates, semantic labels and confidence scores, and timestamps of the key feature points; The real-time local fusion data is input into the laser-inertial SLAM algorithm to optimize the real-time local point cloud map, resulting in a real-time high-precision local map with time stamps and semantic labels.

2. The method according to claim 1, characterized in that, The edge computing system includes a first edge computing unit, a second edge computing unit, and a third edge computing unit; the step of adaptively preprocessing multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and uploading the multi-source feature data to the edge computing system, includes: For the lidar point cloud data in the multi-source data, a voxel grid filter is used to downsample the original lidar point cloud acquired by the lidar to obtain downsampled lidar point cloud data. The key feature points in the downsampled lidar point cloud data are extracted based on the curvature feature extraction algorithm. The key feature points include edge points corresponding to point clouds with curvature values ​​greater than a first preset curvature value, and planar points corresponding to point clouds with curvature values ​​less than a second preset curvature value. Based on the number of key feature points, the ideal point cloud density, and the lidar scanning area, the point cloud density confidence level corresponding to the lidar point cloud data is obtained. The final confidence level of the lidar point cloud data is obtained based on the point cloud density confidence level, the lidar health status confidence level, and the lidar effective return rate confidence level. The lidar effective return rate confidence level is obtained based on the total number of emitted laser beams and the number of effectively returned point clouds. The lidar health status confidence level is calculated based on the lidar's self-test parameters. Based on the final confidence level corresponding to the lidar point cloud data, determine whether to upload the key feature points to the first edge computing unit; And / or, for the millimeter-wave radar point cloud data in the multi-source data, a clustering algorithm is used to cluster the original millimeter-wave radar point cloud collected by the millimeter-wave radar to obtain clustered point cloud clusters, so as to determine obstacles based on each point cloud cluster; Based on a multi-target tracking algorithm, the obstacles are correlated and their states are predicted between consecutive frames to obtain a list of moving target trajectories corresponding to each obstacle; the list of moving target trajectories includes the position information, speed information and direction of movement information of the obstacle; For each obstacle, the predicted position of the obstacle at the next moment is determined based on the list of moving target trajectories corresponding to the obstacle and the position information of the obstacle in the current frame. Based on the predicted position of the obstacle at the next moment, the trajectory information of the moving target corresponding to the obstacle is obtained; For each obstacle, the association consistency confidence of the obstacle is obtained based on the association success rate of the obstacle between consecutive frames; For each obstacle, the final confidence level of the millimeter-wave radar point cloud data corresponding to the obstacle is obtained based on the association consistency confidence level and the signal-to-noise ratio confidence level; the signal-to-noise ratio confidence level is calculated based on the effective signal power and noise power corresponding to the millimeter-wave radar. Based on the final confidence level of the millimeter-wave radar point cloud data corresponding to each obstacle, determine whether to upload the trajectory information of the moving target corresponding to the obstacle to the first edge computing unit. And / or, for the thermal imaging image data in the multi-source data, based on the Otsu threshold segmentation heat source detection algorithm, extract the temperature anomaly regions in the thermal imaging image data to obtain temperature distribution summary information corresponding to each of the temperature anomaly regions; the temperature distribution summary information includes the highest temperature, the average temperature, and the temperature distribution histogram. The final confidence level of the thermal imaging image data is obtained based on the difference between the target ambient temperature and the current ambient temperature. Based on the final confidence level corresponding to the thermal imaging image data, determine whether to upload the temperature distribution summary information corresponding to each of the temperature anomaly regions to the second edge computing unit. And / or, for the binocular visual data in the multi-source data, calculate the binocular visual data based on the SGM stereo matching algorithm to generate left and right eye disparity maps; The left and right eye disparity maps are sequentially subjected to left-right consistency checks and weighted least squares optimization to remove mismatched points in the left and right eye disparity maps, thus obtaining the final left and right eye disparity maps. Based on the left and right eye disparity maps, the depth image and semantic segmentation results in the binocular coordinate system are obtained; Based on the image sharpness confidence and illumination intensity confidence of the left and right eye disparity maps, the final confidence level corresponding to the binocular vision data is obtained; Based on the final confidence level corresponding to the binocular vision data, determine whether to upload the depth image and semantic segmentation results to the third edge computing unit.

3. The method according to claim 2, characterized in that, The step of performing real-time local point cloud map of the underground roadway by using the key feature points and inertial navigation feature data on the edge computing system through a SLAM framework for localization and map construction includes: Based on a preset neighborhood radius, the laser radar feature points corresponding to the real-time frame are extracted to obtain real-time ISS feature points; For each of the real-time ISS feature points, the SHOT descriptor corresponding to the real-time ISS feature point is calculated according to the preset feature dimension. The similarity between the SHOT descriptor corresponding to the real-time ISS feature point of the current frame and the SHOT descriptor corresponding to the real-time ISS feature point of the previous frame is calculated to establish the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM front end. Using the SLAM backend, a point cloud map is constructed based on the correspondence between the real-time ISS feature points of the current frame and the previous frame in the SLAM frontend, as well as the inertial navigation feature data, to obtain a real-time local point cloud map of the underground roadway corresponding to the current location of the tunneling machine.

4. The method according to claim 2, characterized in that, The step of processing the real-time local point cloud map on the edge computing system based on the depth image and semantic segmentation results to obtain a real-time local point cloud map with semantic labels includes: Based on the semantic segmentation results and the depth information in the depth image, a set of target objects with semantic labels is generated; Based on the semantic segmentation results and the depth image, a dense point cloud is generated. The dense point cloud is then subjected to coordinate transformation and point cloud fusion processing in sequence to obtain a semantic point cloud map corresponding to the underground tunnel. Based on the semantic point cloud map corresponding to the underground roadway, the position information of each semantic tag on the semantic point cloud map is obtained; The position information of each semantic tag on the semantic point cloud map is projected onto the real-time local point cloud map through coordinate transformation to obtain a real-time local point cloud map with semantic tags.

5. The method according to claim 1, characterized in that, The process of fusing and optimizing real-time local high-precision maps corresponding to different time series on the central processing server to obtain a global map of the underground roadway includes: Cross-modal semantic alignment is performed on real-time local high-precision maps corresponding to different time series to obtain real-time local high-precision maps in a unified semantic space. Based on the inertial navigation feature data of the tunneling machine and the time stamps corresponding to each of the real-time local high-precision maps, the real-time local high-precision maps under a unified semantic space are fused to obtain a global map of the underground roadway.

6. A device for constructing downhole maps, characterized in that, include: The acquisition module is used to acquire multi-source data collected by a multi-source heterogeneous sensor cluster; The multi-source heterogeneous sensor cluster is deployed on a tunneling machine operating in an underground roadway; the multi-source heterogeneous sensor cluster includes lidar, millimeter-wave radar, thermal imaging camera, binocular camera, and IMU module; the multi-source data includes lidar point cloud data, millimeter-wave radar point cloud data, thermal imaging image data, binocular vision data, and inertial navigation data of the tunneling machine; An adaptive preprocessing module is used to adaptively preprocess multi-source data according to a preset bandwidth allocation strategy to obtain low-bandwidth multi-source feature data, and upload the multi-source feature data to the edge computing system. The multi-source feature data includes key feature points extracted from the lidar point cloud data, depth images and semantic segmentation results extracted from the binocular vision data, inertial navigation feature data extracted from the inertial navigation data, temperature distribution summary information extracted from the thermal imaging image data, and moving target trajectory information of obstacles. The moving target trajectory information of obstacles is obtained based on the millimeter-wave radar point cloud data. The real-time local point cloud map generation module is used to perform real-time local positioning and map construction on the edge computing system using the SLAM framework to generate the key feature points and the inertial navigation feature data, thereby obtaining the real-time local point cloud map corresponding to the underground roadway. The real-time local point cloud map optimization module is used to process the real-time local point cloud map on the edge computing system based on the depth image and semantic segmentation results to obtain a real-time local point cloud map with semantic labels, and send the multi-source feature data and the real-time local point cloud map with semantic labels to the central processing server. The real-time local fusion data calculation module is used to perform tight coupling fusion of the multi-source feature data on the central processing server based on adaptive weights to obtain real-time local fusion data of the underground roadway. The real-time local high-precision map generation module is used to optimize the real-time local point cloud map with semantic tags on the central processing server based on the real-time local fusion data of the underground roadway, so as to obtain a real-time local high-precision map with time stamp and semantic tags. The global map generation module is used to fuse and optimize real-time local high-precision maps corresponding to different time series on the central processing server to obtain a global map of the underground roadway. The real-time local fusion data calculation module includes: The spatiotemporal synchronization unit is used to perform spatiotemporal synchronization on each source feature data in the multi-source feature data to obtain spatiotemporally synchronized multi-source feature data, and to add a time stamp to the spatiotemporally synchronized multi-source feature data. The fusion weight determination unit is used to evaluate each source feature data in the multi-source feature data after spatiotemporal synchronization and determine the fusion weight corresponding to each source feature data. The tightly coupled fusion unit is used to perform tightly coupled fusion of the multi-source feature data based on the fusion weight corresponding to each source feature data to obtain real-time local fusion data with time stamps. The real-time local high-precision map generation module includes: The sampling processing unit is used to sample the real-time local fusion data using voxel grid filtering to obtain real-time local fusion data with uniform density; the real-time local fusion data includes key feature points, as well as the corresponding three-dimensional coordinates, semantic labels and confidence scores, and timestamps of the key feature points; The map optimization unit is used to input the real-time local fusion data into the laser-inertial SLAM algorithm to optimize the real-time local point cloud map and obtain a real-time local high-precision map with time stamps and semantic labels.

7. An electronic device, characterized in that, include: Memory is used to store processor-executable instructions; A processor is configured to execute executable instructions in the memory to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images