Well mining point cloud map construction method and device based on multi-sensor fusion, equipment and medium

Abstract

The invention discloses a method, a device, equipment and a medium for constructing a point cloud map of a well mine based on multi-sensor fusion, and the method comprises the steps: synchronously collecting underground environment data through a plurality of vehicle-mounted sensors, and the plurality of sensors comprise a laser radar, a camera, a GNSS receiver, an inertial measurement unit and a vehicle chassis encoder; calculating roll angle and pitch angle information through data provided by the GNSS receiver, and taking the roll angle and pitch angle information as absolute attitude constraints; generating an initial pose sequence based on the data of the laser radar, the camera, the inertial measurement unit and the vehicle chassis encoder; establishing a pose map containing a plurality of pose nodes in the initial pose sequence, adding an odometer constraint, a loopback constraint and an absolute pose constraint in the pose map, and carrying out global optimization on the pose map; and generating a dense point cloud map according to the optimized pose. The method is low in cost, high in data credibility and high in working stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of map construction, and in particular to a mine point cloud map construction method, device, equipment and medium based on multi-sensor fusion. BACKGROUND

[0002] A mine refers to a mine that mines coal, metal ore, etc. through an underground tunnel system. Compared with an open-pit mine, it has the characteristics of technical complexity and high safety requirements. Its mining and transportation is a complex and challenging process. The traditional mine operation mode requires a large number of workers to cooperate, and the operation efficiency is low. At the same time, the working environment for workers is relatively poor, and there is a certain safety risk. The application of unmanned driving technology can realize 24-hour uninterrupted operation of unmanned vehicles, improve mining and transportation efficiency, reduce human resource investment, and at the same time, can significantly reduce the risk of accidents caused by human factors and improve the safety of operation.

[0003] The mine point cloud map is a three-dimensional point cloud data set generated by scanning the internal environment of the mine using a laser radar and other sensors, which is used to accurately describe the spatial distribution of tunnels, terrain and facilities.

[0004] Due to the complex and variable environment of the mine operation area, including narrow tunnels, complex underground structures and possible signal interference, all of which bring great challenges to the production of point cloud maps, but the quality of the mine point cloud map directly affects the robustness of subsequent unmanned vehicle positioning, navigation and path planning, and it has become a key technology for the landing of unmanned driving technology in mine scenarios.

[0005] Chinese patent "CN116643289 Mine tunnel laser radar SLAM method constrained by composite conductor" proposes a method of laser radar SLAM mapping based on a reflector benchmark. This method needs to lay out survey stations in the environment in advance, and place reflectors after obtaining the coordinates of the survey stations. There are obstacles such as ore, equipment and pipelines in the mine, and there is geomagnetic interference. These obstacles and geomagnetic interference greatly affect the accuracy of the coordinates of the obtained survey stations, resulting in a large difference between the finally established point cloud map and the real world map; Chinese patent "CN114646937 Real-time fusion and mapping method of SLAM and UWB" proposes a SLAM mapping method based on IMU, laser radar and UWB equipment. This method needs to deploy a large number of UWB positioning base stations in the mine, which has a very high installation and maintenance cost and is not suitable for the application of point cloud maps in large-scale scenarios.

[0006] The Chinese patent "CN115290073 A mine underground non-structural feature SLAM method and system" proposes a SLAM positioning and mapping method based on IMU, lidar and camera. This method needs to extract feature points (corner points and plane points) in point cloud data. The mapping method of feature extraction ignores the subtle features in the environment, reduces the accuracy of mapping, and the extraction of features is also time-consuming, increasing the computational load. SUMMARY

[0007] The embodiments of the present application provide a mine point cloud map construction method, device, equipment and medium based on multi-sensor fusion, which is used to solve the technical problems of high installation and maintenance cost and low accuracy and precision in the prior art which needs to rely on more deployed measuring stations or positioning base stations.

[0008] The first aspect of the present application provides a mine point cloud map construction method based on multi-sensor fusion, comprising: Synchronously collecting underground environment data by a vehicle-mounted multi-sensor, wherein the multi-sensor comprises a lidar, a camera, a GNSS receiver, an inertial measurement unit and a vehicle chassis encoder; Calculating roll angle and pitch angle information through the data provided by the GNSS receiver, and taking the roll angle and pitch angle information as absolute attitude constraints; Generating an initial pose sequence based on the data of the lidar, the camera, the inertial measurement unit and the vehicle chassis encoder; Establishing a pose graph containing multiple pose nodes in the initial pose sequence, adding odometry constraints, loop constraints and absolute attitude constraints in the pose graph, and globally optimizing the pose graph; Generating a dense point cloud map according to the optimized pose.

[0009] Preferably, the generating of the initial pose sequence based on the data of the lidar, the camera, the inertial measurement unit and the vehicle chassis encoder comprises: Using error state iterative Kalman filtering to fuse the point cloud registration observation of the lidar, the visual observation of the camera, the inertial observation of the IMU and the speed observation of the vehicle chassis encoder, and taking the roll angle and pitch angle information of the GNSS receiver as an observation item for fusion, so as to output the initial pose sequence.

[0010] Preferably, the generating of the initial pose sequence based on the data of the lidar, the camera, the inertial measurement unit and the vehicle chassis encoder comprises: The point cloud data is organized by constructing a multi-resolution map structure, dynamic candidate points are identified by comparing the spatial occupancy state of the current frame point cloud with the local static map, and the free space information blocked by dynamic objects is recovered by combining a ray enhancement strategy to remove the dynamic obstacle point cloud.

[0011] Preferably, before generating the initial pose sequence based on the data of the lidar, camera, inertial measurement unit and vehicle chassis encoder, the method further comprises: The point cloud data is organized by constructing a multi-resolution map structure, dynamic candidate points are identified by comparing the spatial occupancy state of the current frame point cloud with the local static map, and the free space information blocked by dynamic objects is recovered by combining a ray enhancement strategy to remove the dynamic obstacle point cloud.

[0012] Preferably, the multi-resolution map structure comprises blocks, voxel units and sub-voxel units divided in sequence, wherein the size of the sub-voxel unit defines the final resolution of the map; the sizes of the structures at each level satisfy the relationship: ; Wherein, s is the size, d is the depth, i, j represent any two levels in the sub-voxel s, voxel v and block b respectively.

[0013] Preferably, a loopback constraint is added in the pose graph, comprising: The point cloud data is converted into a bird's eye view (BEV) image, a deep learning model is used to extract global descriptors of the BEV image to identify loopback frames, and a loopback pose is calculated by point cloud registration to form the loopback constraint.

[0014] Preferably, the initial pose sequence is generated based on the data of the lidar, camera, inertial measurement unit and vehicle chassis encoder, and further comprises: The data of the inertial measurement unit is pre-integrated, and the pre-integration is used to perform motion compensation on the original point cloud of the lidar.

[0015] In a second aspect of the present application, a mine point cloud map construction device based on multi-sensor fusion is provided, comprising: A data collection module is configured to synchronously collect underground environment data by a plurality of sensors carried by a vehicle, wherein the plurality of sensors comprise a lidar, a camera, a GNSS receiver, an inertial measurement unit and a vehicle chassis encoder; A GNSS data conversion module is configured to calculate roll angle and pitch angle information based on the data provided by the GNSS receiver, and use the roll angle and pitch angle information as absolute attitude constraints; An initial pose generation module is configured to generate an initial pose sequence based on the data of the lidar, camera, inertial measurement unit and vehicle chassis encoder; An optimization module is configured to establish a pose graph comprising a plurality of pose nodes in the initial pose sequence, add an odometry constraint, a loop constraint and an absolute attitude constraint in the pose graph, and globally optimize the pose graph; A point cloud generation module is configured to generate a dense point cloud map according to the optimized pose.

[0016] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to enable the electronic device to implement the method according to the first aspect of the present application.

[0017] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, wherein the computer program, when running on a computer, enables the computer to execute the method according to the first aspect of the present application. Advantages

[0018] The present application provides a multi-sensor fusion based mine point cloud map construction method, comprising: synchronously collecting underground environment data by a vehicle-mounted multi-sensor, wherein the multi-sensor comprises a laser radar, a camera, a GNSS receiver, an inertial measurement unit and a vehicle chassis encoder; calculating roll angle and pitch angle information based on data provided by the GNSS receiver, and taking the roll angle and pitch angle information as absolute attitude constraints; generating an initial pose sequence based on data of the laser radar, the camera, the inertial measurement unit and the vehicle chassis encoder; establishing a pose graph comprising a plurality of pose nodes in the initial pose sequence, adding an odometry constraint, a loop constraint and an absolute attitude constraint in the pose graph, and globally optimizing the pose graph; and generating a dense point cloud map according to the optimized pose.

[0019] The above-mentioned application only needs to use vehicle-mounted sensors, does not need to install and maintain equipment in a mine, and has low cost, without the need to install a uwb base station or arrange a reflector according to the prior art; a set of multi-sensor fusion mapping framework is designed, which fuses vehicle chassis, imu, camera, multi-laser radar and GNSS data to construct a dense point cloud map basically consistent with the real world; a multi-sensor fusion method is adopted to adaptively adjust the weight of each sensor, overcome the limitation of a single sensor, fully utilize the advantages of each sensor, and enable stable work in a mine with similar geometric features. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The accompanying drawings should not be construed as an inappropriate limitation on the present application. In the drawings: Figure 1 is a flowchart of a mine point cloud map construction method based on multi-sensor fusion according to an embodiment of the application.

[0021] Figure 2 is a point cloud image before dynamic obstacle removal in an embodiment of the application.

[0022] Figure 3 is a point cloud image after dynamic obstacle removal in an embodiment of the application.

[0023] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0026] As shown in Figure 1 is a schematic diagram of a mine point cloud map construction method based on multi-sensor fusion provided by an embodiment of the application, which comprises Figure 1 It can be seen that the mine point cloud map construction method based on multi-sensor fusion comprises: Step S102, synchronously collecting underground environment data by a vehicle-mounted multi-sensor, the multi-sensor comprising a laser radar, a camera, a GNSS receiver, an inertial measurement unit and a vehicle chassis encoder.

[0027] In this step, before the vehicle enters the underground operation area, the inter-sensor extrinsic parameters are obtained by completing the space-time joint calibration of all sensors at the entrance. After the vehicle starts, each sensor begins to synchronously collect data, and the camera exposure pulse is used as a hardware synchronization signal during the collection, and a unified timestamp is stamped on all data.

[0028] The laser radar is used to obtain the geometric point cloud information of the surrounding environment; the camera is used to obtain the visual texture information; the GNSS receiver is used to obtain the roll angle and pitch angle information; the inertial measurement unit is used to obtain the high-frequency angular velocity and acceleration information; and the chassis encoder is used to obtain the vehicle wheel speed information.

[0029] In some optional embodiments, after obtaining the multi-sensor data through step S102, the data is filtered according to the camera time to obtain all sensor data from the time of the previous frame image to the time of the current frame image. Among them, the laser radar point cloud data is scanned and reorganized according to the camera time, and the laser point cloud is rearranged. This can be applied to different types of laser radar sensors, including various solid-state laser radars and mechanical rotating laser radars.

[0030] Step S104, calculate the roll angle and pitch angle information through the data provided by the GNSS receiver, and use the roll angle and pitch angle information as absolute attitude constraints.

[0031] In this step, although the position signal of the GNSS receiver is unreliable underground, the attitude angle information is relatively stable within a certain time, so the attitude angle provided by the GNSS can be used as an absolute attitude constraint to constrain the global attitude in the entire map construction process, prevent divergence in height and angle direction, and ensure that the generated map is consistent with the attitude reference of the real world.

[0032] Step S106, generate an initial pose sequence based on the data of the laser radar, camera, inertial measurement unit and vehicle chassis encoder.

[0033] Specifically, the above step fuses the laser radar point cloud registration observation, the camera visual observation, the IMU inertial observation and the vehicle chassis encoder speed observation through error state iterative Kalman filtering, and fuses the roll angle and pitch angle information of the GNSS receiver as an observation item. The filtering process can adaptively adjust the fusion weight according to the uncertainty of each sensor observation. Through this step, an initial pose sequence with high local consistency but possible cumulative drift over time is output.

[0034] The laser radar point cloud registration observation is the relative pose change calculated by point cloud registration, the camera visual observation is the pose change calculated by projecting map points to the current image and calculating photometric error, the IMU inertial observation is the angular velocity and acceleration increment between adjacent frames obtained by pre-integration, and the chassis encoder speed observation is the vehicle wheel speed information obtained by the chassis encoder.

[0035] In this step, a multi-sensor tightly coupled odometer calculation framework is specifically implemented, and the above process specifically includes the following steps: Step S1061, data preprocessing and synchronization Take the current camera frame time as the reference, collect all IMU data, laser radar scanning data, chassis encoder pulse count and GNSS data within the time window from the previous frame to the current frame.

[0036] Step S1062, IMU pre-integration and point cloud distortion removal Pre-integration is performed on the IMU data within the time window to obtain the angular velocity and acceleration increments between two frames. Using this pre-integration result, motion compensation is applied to the original point cloud data with different timestamps acquired by the lidar within the time window, uniformly compensating it to the reference time of the current camera frame. This eliminates point cloud distortion caused by vehicle movement and avoids the impact of violent movement and rapid rotation on the quality of laser point clouds.

[0037] The point cloud compensation formula is:

[0038] in, The point cloud coordinates at time are The compensated point cloud coordinates are , The IMU pose at time t is , The IMU pose at time t is The extrinsic parameters of the lidar and IMU are .

[0039] Step S1063, Dynamic Obstacle Removal In underground mining areas, moving vehicles and pedestrians frequently pass through. On one hand, this noise can cause matching errors for radar and cameras, and in extreme cases, even lead to mapping failure. On the other hand, these noise residues in the point cloud map can negatively impact subsequent positioning, navigation, and path planning. To address the unique application scenario of mines and eliminate dynamic obstacles, this embodiment constructs a multi-resolution map structure. The current frame point cloud, obtained in the previous step after distortion correction, is inserted into this multi-resolution map structure. Simultaneously, the system maintains a local static background map built over the past 30 seconds, which also uses a multi-resolution structure. By comparing the occupancy status of both maps in the same voxel space and analyzing the rays from the sensor origin to each laser point, point clouds belonging to dynamic objects such as mine vehicles and personnel are removed, resulting in a clean static environment point cloud. This avoids the influence of dynamic objects on point cloud registration. Figure 2 and Figure 3 The image shows a comparison before and after obstacle removal. It can be seen that there were many obstacles in the alley before removal, while the alley was cleaner after removal.

[0040] This step involves organizing point cloud data by constructing a multi-resolution map structure, identifying dynamic candidate points by comparing the spatial occupancy status of the current frame point cloud with that of the local static map, and using a ray enhancement strategy to recover the free space information occluded by dynamic objects in order to eliminate dynamic obstacle point clouds.

[0041] In some preferred embodiments, the multi-resolution map structure includes uniformly dividing the global space into sections of size [missing information]. The blocks are further subdivided into blocks of size 1. The voxel unit, in the voxel unit space, each voxel unit is further subdivided into units of size . The sub-voxel units, i.e., the sequential division of blocks, voxel units, and sub-voxel units, where the size of the sub-voxel unit defines the final resolution of the map; the depths of the sub-voxel units, voxel units, and blocks are defined respectively. , and, The dimensions of each structural level satisfy the following relationship: ; Where s is the size, d is the depth, and i and j represent any two levels in sub-voxel s, voxel v, and block b, respectively.

[0042] Point cloud registration and pose calculation are performed using all laser point clouds to fully utilize the features of the environmental point cloud.

[0043] The image is then distorted again, the visual map points in the map are projected into the current frame, and the photometric error between the points and the pixels in the current image is calculated. The inter-frame pose change is calculated to obtain the camera's visual observation.

[0044] Step S1064: Tightly Coupled State Estimation A clean, static point cloud, along with the current frame image, IMU pre-integration, wheel speed increment, and GNSS attitude angle, are input into an error state iterative Kalman filter. The filter iteratively optimizes the optimal pose transformation from the previous frame to the current frame by minimizing the laser point cloud matching error, image photometric error, IMU pre-integration error, wheel speed error, and GNSS attitude angle error, and outputs the vehicle state (position, attitude, velocity, etc.) at the current moment. This process continues, forming an initial pose sequence.

[0045] Odometry diverges over time due to changes in altitude and angle, meaning the point clouds don't perfectly overlap when returning to the same historical location. Therefore, embodiments of the present invention further include the following steps: Step S108: Establish a pose graph containing multiple pose nodes in the initial pose sequence. Preferably, the pose graph is constructed using the poses of all key frames as nodes. Odometry constraints between adjacent key frames, loop edges between loop frames as loop constraints, and GNSS absolute attitude edges on each key frame as absolute attitude constraints are added to the pose graph to perform global optimization.

[0046] In this step, the odometer constraint constrains relative pose changes by connecting adjacent pose nodes.

[0047] In this step, the loop constraint can add edges between corresponding pose nodes when the current scene is detected to match the historical scene, thus constraining their relative poses.

[0048] In this step, the absolute attitude constraint is achieved by adding GNSS roll and pitch angle constraints to each pose node, serving as a global attitude reference. The optimization objective in the above process is to minimize the residuals of all constraint edges in the graph.

[0049] Input this graph into a nonlinear least squares problem constructed using the Ceres Solver library. The optimization objective function includes the residuals of all edges, with an adaptive weight adjustment mechanism applied to the odometry and GNSS attitude edges, and a Cauchy kernel function applied to the loop closure edge residuals. Through iterative solution, the globally optimal keyframe pose is obtained.

[0050] The algorithm iterates through all LiDAR frames, interpolates their corresponding timestamps to obtain the optimized global pose, and transforms the point cloud of each frame to the world coordinate system. Finally, all point clouds are merged into a single global map, and a voxel grid filter is used for downsampling to eliminate overlapping points, generating a final, globally consistent dense point cloud map of the mine.

[0051] Through the above optional embodiments and multiple iterations, a globally consistent and accurate pose sequence is obtained through optimization.

[0052] The above process specifically includes the following steps: Step S1081: Obtain the odometry pose generated in the previous step. and its corresponding point cloud data Where T and P are sets of pose and point cloud data; t and p are the pose and point cloud data for each keyframe. The pose and the number of point clouds are represented, with a one-to-one correspondence between the pose and the point cloud in each frame.

[0053] Step S1082: Convert each frame of point cloud p into a BEV image. Specifically, the BEV_Place++ method is used to extract descriptors, find similar loop closure frames, and calculate the initial pose to provide pose prior for point cloud registration. Specifically, a pre-trained BEV_Place++ neural network model is used to extract the global descriptor of the BEV image. When a new keyframe arrives, its descriptor is compared with the historical keyframe descriptor database for similarity. If a historical frame with a similarity exceeding a threshold is found, a loop closure is detected. Subsequently, the Generalized Iterative Closest Point (GICP) algorithm is used to perform fine registration of these two point cloud frames, calculating the precise relative pose transformation, which serves as the loop closure constraint. The aforementioned BEV_Place++ method is a deep learning method that does not require manually set features, has strong environmental universality, and can provide a good initial pose for point cloud registration.

[0054] Among them, bev pixel value Calculated as normalized point cloud density:

[0055] in, Indicates position The number of points in the grid, The normalization factor is set to the maximum point cloud density.

[0056] The BEV Place++ neural network model used in the above process has a core architecture based on existing publicly available model designs, and mainly includes the following publicly available components: Feature extraction backbone network: The ResNet-50 pre-trained on the large image dataset ImageNet is used as the basic feature extractor to extract dense two-dimensional feature maps from the input BEV image.

[0057] Global descriptor generation module: The core layer is NetVLAD (Net Vector of Locally Aggregated Descriptors). This layer clusters and aggregates the local feature descriptors output by the backbone network to generate a fixed-length global descriptor vector that is robust to changes in viewpoint and illumination.

[0058] The aforementioned core components, ResNet and NetVLAD, are well-known neural network modules in the field. Their specific structures and working principles can be found in the corresponding publicly available literature. K.He,

[0059] R.Arandjelovic, P.Gronat, A.Torri, T.Pajdla and J.Sivic, "NetVLAD: CNNArchitecture for Weakly Supervised Place Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp.5297-5307, doi: 10.1109 / CVPR.2016.572.

[0060] This invention directly uses the publicly available architecture for adaptive combination without making any improvements to the infrastructure itself.

[0061] Specific settings and parameters for model usage When using the pre-trained BEV P1ace++ model to extract global descriptors from BEV images, the following settings and operations must be performed to ensure the reproducibility of the results: (1) Input data normalization Input image size: Scaling or cropping the BEV image to be processed to H×W=256×256 pixels.

[0062] (2) Model loading and forward propagation Pre-trained weights: Weights from the BEV_Place++ model pre-trained on the publicly available BEV scene dataset nuScenes. This weight file is in pth format and is available through the public code repository GitHub.

[0063] Forward propagation: The normalized BEV image tensor is input into the loaded model, and the model directly outputs a global descriptor for the BEV image.

[0064] Training and fine-tuning Loss function: Triplet Margin Loss is used, and the boundary value of Triplet Loss is set to 0.5.

[0065] Optimizer and hyperparameters: The Adam optimizer was used with an initial learning rate of 1e-4, a batch size of 16, and 50 training epochs.

[0066] The learning rate is adjusted using a learning rate decay strategy, such as multiplying it by a factor of 0.1 when the loss on the validation set plateaus.

[0067] Step S1083: Use the generalized iterative nearest point method to perform point cloud matching and calculate the loop closure pose.

[0068] Step S1084: Build a pose optimization framework based on Ceres to solve nonlinear problems. Input odometry edge constraints, loop closure constraints, and absolute attitude constraints. Use residual iterative optimization to adaptively adjust the weights of each constraint in the optimization process; iterate the pose optimization multiple times. Add a Cauchy kernel function during the optimization process to suppress the influence of abnormal observation data such as erroneous loop matching, enhancing the robustness of the optimization algorithm.

[0069] Step S110: Generate a dense point cloud map based on the optimized pose.

[0070] In this step, the optimized pose is globally consistent. Each frame of LiDAR point cloud data is converted to a unified world coordinate system, stitched and fused, and finally a dense point cloud map of the mine is generated that is highly consistent with the real world, which is used for the positioning and navigation of unmanned vehicles.

[0071] The method provided in the embodiments of the present invention mainly generates a locally consistent odometry pose based on the collected multi-sensor data at the front end. Then, the back end calculates the loop closure pose based on the odometry pose from the previous step, and further optimizes the pose by adding loop closure constraints and GNSS angle constraints, thereby ensuring the global consistency of the point cloud map and making the final point cloud map consistent with the real world. The above method only uses vehicle-mounted sensors and does not require any additional infrastructure to be deployed underground. Through innovative multi-sensor tight coupling of the front-end odometry and back-end optimization that integrates global constraints, combined with efficient dynamic obstacle removal technology, it achieves low-cost, high-consistency, and highly robust underground dense point cloud map construction.

[0072] Based on the same inventive concept as the above-described method embodiments, this application also provides a device for constructing point cloud maps in underground mines based on multi-sensor fusion, comprising: The data collection module is used to synchronously collect downhole environmental data through multiple on-board sensors, including lidar, camera, GNSS receiver, inertial measurement unit, and vehicle chassis encoder. The GNSS data conversion module is used to calculate roll and pitch angle information using data provided by the GNSS receiver, and to use the roll and pitch angle information as absolute attitude constraints. The initial pose generation module is used to generate an initial pose sequence based on the data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder. The optimization module is used to establish a pose graph containing multiple pose nodes in the initial pose sequence, and to add odometry constraints, loop closure constraints and absolute attitude constraints to the pose graph to perform global optimization of the pose graph. The point cloud generation module is used to generate a dense point cloud map based on the optimized pose.

[0073] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0074] Based on the same inventive concept as the above method embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it enables the electronic device to implement the control method described in the above embodiments.

[0075] In one embodiment, the electronic device may be a server, and in this embodiment, the structure of the electronic device may be as follows: Figure 4 As shown, it includes a memory 2001, a communication module 2003, and one or more processors 2002.

[0076] The memory 2001 is used to store computer programs executed by the processor 2002. The memory 2001 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0077] Memory 2001 may be volatile memory, such as random-access memory (RAM); memory 2001 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 2001 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2001 may be a combination of the above-mentioned memories.

[0078] Processor 2002 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 2002 is used to implement the above-mentioned audio data processing method when calling computer programs stored in memory 2001.

[0079] The communication module 2003 is used to communicate with terminal devices and other servers.

[0080] This application embodiment does not limit the specific connection medium between the memory 2001, communication module 2003, and processor 2002. This application embodiment... Figure 4 The memory 2001 and the processor 2002 are connected via a bus 2004, which is in... Figure 3 The connections between other components are illustrated with arrows and are for illustrative purposes only, not as limiting information. The Bus 2004 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 4 The text uses only one arrow to describe it, but does not indicate that there is only one bus or one type of bus.

[0081] Based on the same inventive concept as the above-described method embodiments, embodiments of the present invention also provide a computer-readable storage medium for storing a computer program. When the computer program is run on a computer, it enables the electronic device to implement the control method described in the above embodiments. The computer-readable storage medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0082] Based on the same inventive concept as the above-described method embodiments, embodiments of the present invention also provide a computer program product, which includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the control methods described above according to various exemplary embodiments of this application. The program product may take the form of any combination of one or more readable media. These computer program commands can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the commands executed by the processor of the computer or other programmable data processing device generate a process for implementing... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A method for constructing point cloud maps in underground mines based on multi-sensor fusion, characterized in that, include: Downhole environmental data is collected synchronously by multiple sensors mounted on the vehicle, including lidar, camera, GNSS receiver, inertial measurement unit, and vehicle chassis encoder. The roll and pitch angles are calculated using data provided by the GNSS receiver, and the roll and pitch angles are used as absolute attitude constraints. Based on the data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder, an initial pose sequence is generated; A pose graph containing multiple pose nodes from the initial pose sequence is established, and odometry constraints, loop closure constraints, and absolute attitude constraints are added to the pose graph to perform global optimization. A dense point cloud map is generated based on the optimized pose.

2. The method according to claim 1, characterized in that, The generation of an initial pose sequence based on data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder includes: An error state iterative Kalman filter is used to fuse the point cloud registration observations of the lidar, the visual observations of the camera, the inertial observations of the IMU, and the speed observations of the vehicle chassis encoder. The roll angle and pitch angle information of the GNSS receiver are also fused as observation items to output the initial pose sequence.

3. The method according to claim 1, characterized in that, The generation of an initial pose sequence based on data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder includes: A multi-resolution map structure is constructed to organize point cloud data. Dynamic candidate points are identified by comparing the spatial occupancy status of the current frame point cloud with that of the local static map. A ray enhancement strategy is combined to restore the free space information occluded by dynamic objects, so as to eliminate dynamic obstacle point clouds.

4. The method according to claim 3, characterized in that, Before generating the initial pose sequence based on the data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder, the process further includes: A multi-resolution map structure is constructed to organize point cloud data. Dynamic candidate points are identified by comparing the spatial occupancy status of the current frame point cloud with that of the local static map. A ray enhancement strategy is combined to restore the free space information occluded by dynamic objects, so as to eliminate dynamic obstacle point clouds.

5. The method according to claim 3, characterized in that, The multi-resolution map structure includes sequentially divided blocks, voxel units, and sub-voxel units, where the size of the sub-voxel unit defines the final resolution of the map; the dimensions of each level of the structure satisfy the following relationship: ; Where s is the size, d is the depth, and i and j represent any two levels in sub-voxel s, voxel v, and block b, respectively.

6. The method according to claim 3, characterized in that, Adding closure constraints to the pose graph includes: Point cloud data is converted into bird's-eye view BEV images. A deep learning model is used to extract global descriptors from the BEV images to identify loop closure frames. The loop closure pose is calculated through point cloud registration to form the loop closure constraints.

7. The method according to claim 3, characterized in that, The step of generating an initial pose sequence based on data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder further includes: The data from the inertial measurement unit is pre-integrated, and the pre-integration is used to perform motion compensation on the original point cloud of the lidar.

8. A device for constructing point cloud maps in underground mines based on multi-sensor fusion, characterized in that, include: The data collection module is used to synchronously collect downhole environmental data through multiple on-board sensors, including lidar, camera, GNSS receiver, inertial measurement unit, and vehicle chassis encoder. The GNSS data conversion module is used to calculate roll and pitch angle information using data provided by the GNSS receiver, and to use the roll and pitch angle information as absolute attitude constraints. The initial pose generation module is used to generate an initial pose sequence based on the data from the lidar, camera, inertial measurement unit, and vehicle chassis encoder. The optimization module is used to establish a pose graph containing multiple pose nodes in the initial pose sequence, and to add odometry constraints, loop closure constraints and absolute attitude constraints to the pose graph to perform global optimization of the pose graph. The point cloud generation module is used to generate a dense point cloud map based on the optimized pose.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.

Images