Mine scene self-adaptive repositioning method based on multi-modal perception

Abstract

The invention relates to the technical field of mine environment positioning, in particular to a mine scene self-adaptive repositioning method based on multi-modal perception, which comprises the following steps of: S1, acquiring information, and acquiring visual images, spatial features, wireless signal features and inertial motion data of an internal environment of a mine by using a multi-modal sensor; s2, extracting information, namely extracting the feature data obtained in the step S1, forming a dynamic topological map, and marking temporary obstacle information at the same time; s3, information processing: adaptively adjusting data fusion weights of different sensors according to the information extracted in the step S2 based on a dynamic environment sensing result; and S4, information updating: combining particle filtering and a graph optimization algorithm to realize robust pose estimation and real-time map updating. According to the invention, the failure risk of a single sensor is reduced through multi-mode complementation, and the positioning error is reduced by more than 40%; and through semantic auxiliary matching, the relocation time consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of mine environment positioning, and more specifically, to a mine scene adaptive repositioning method based on multimodal perception. Background Art

[0002] Mine scene relocation refers to the technology and methods for high-precision positioning of personnel, equipment, etc. in a mine environment. Mine scene relocation has the following requirements: safe production needs. The mine operating environment is complex and safety issues are prominent. Accurate positioning systems are crucial to improving safe production levels and ensuring the safety of personnel lives.

[0003] In mine operations, due to the lack of GPS signals, uneven lighting, dust interference and frequent changes in dynamic obstacles, traditional positioning methods (such as single visual SLAM or inertial navigation) are prone to positioning drift or failure problems. In addition, existing algorithms have poor adaptability to dynamic environments and cannot correct positioning deviations in real time, resulting in increased cumulative errors.

[0004] Therefore, we propose an adaptive relocalization method for mine scenes based on multimodal perception to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a mine scene adaptive relocalization method based on multimodal perception to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: a mine scene adaptive relocalization method based on multimodal perception, comprising the following methods: Step S1: Acquire information, using multimodal sensors to collect visual images, spatial features, wireless signal features, and inertial motion data of the mine's internal environment; Step S2: extract information, extract the feature data obtained in step S1, and form a dynamic topological map, while marking temporary obstacle information; Step S3: Information processing, the information extracted in step S2 is used to adaptively adjust the data fusion weights of different sensors based on the dynamic environment perception results; Step S4: Information update, combining particle filtering and graph optimization algorithm to achieve robust pose estimation and real-time map update; Step S5: Repositioning. When an environmental change is detected or the positioning deviation exceeds a threshold, the weight adjustment algorithm based on transfer learning and the repositioning mechanism based on semantic feature matching are triggered to restore the system positioning capability.

[0007] In a preferred embodiment, the multimodal sensor comprises: A combination of at least two of the following: visible light camera, infrared camera, RGB-D camera, lidar, IMU, and millimeter-wave radar.

[0008] In a preferred embodiment, in the multimodal sensor: The visual sensor provides texture information, the lidar generates accurate three-dimensional structure, the IMU compensates for motion blur, and the millimeter-wave radar detects dynamic obstacles; A spatiotemporal synchronization algorithm is used to eliminate timestamp differences in multi-sensor data.

[0009] In a preferred embodiment, in step S2: the update frequency of the dynamic topology map is positively correlated with the speed of the mobile terminal.

[0010] In a preferred embodiment, in step S3: the adaptive weight adjustment mechanism includes: Dynamically assign sensor confidence based on ambient light intensity, point cloud density, and motion acceleration; The optimal weight combination in sensor degradation scenarios is predicted through an online learning model.

[0011] In a preferred embodiment, in step S4: the robust relocation mechanism includes the following contents: Establish a global map including obstacle labels and lane structure labels; When positioning is lost, the search range is quickly narrowed down through semantic matching, and the pose is restored in combination with geometric verification.

[0012] In a preferred embodiment, the relocation mechanism includes: Extract key semantic features of the scene based on convolutional neural network (CNN); Build a hierarchical matching strategy to prioritize searching for similar semantic areas in historical maps; The RANSAC algorithm is used to eliminate mismatched points and optimize the relocalization results.

[0013] In a preferred embodiment, the relocation mechanism further includes: Using the transfer learning algorithm, the positioning weight is dynamically adjusted according to the differences between historical positioning data and the current environment.

[0014] In a preferred embodiment, the relocalization mechanism based on the weight adjustment algorithm of transfer learning and the relocalization mechanism based on semantic feature matching are run synchronously.

[0015] Technical effects and advantages of the present invention: 1. Reduce the risk of single sensor failure through multimodal complementarity, and reduce positioning error by more than 40%; 2. Reduce relocation time through semantic-assisted matching, meeting the real-time requirements of underground equipment; 3. Through the relocalization mechanism, it can perceive environmental changes in real time and adjust the fusion strategy to adapt to interference from light, dust, and dynamic obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] Reference Figure 1 , a mine scene adaptive relocalization method based on multimodal perception, including the following methods: Step S1: Acquire information, using multimodal sensors to collect visual images, spatial features, wireless signal features, and inertial motion data of the mine's internal environment; Multimodal sensors include: A combination of at least two of the following: visible light camera, infrared camera, RGB-D camera, lidar, IMU, and millimeter-wave radar; In multimodal sensors: The visual sensor provides texture information, the lidar generates accurate three-dimensional structure, the IMU compensates for motion blur, and the millimeter-wave radar detects dynamic obstacles; A spatiotemporal synchronization algorithm is used to eliminate timestamp differences in multi-sensor data; The spatiotemporal synchronization algorithms include: Timestamp alignment mechanism: Each sensor (visible light camera, lidar, IMU, etc.) adopts hardware trigger synchronization mode and unifies the time base through an external clock source (such as the PTP protocol) to ensure that the data collection time error is ≤1ms.

[0019] For asynchronously collected data, an interpolation compensation algorithm is used: based on the sensor data of adjacent timestamps (such as IMU angular velocity and acceleration), data at synchronized time points are generated through linear interpolation or spline interpolation.

[0020] Spatial calibration and coordinate unification: A multi-sensor joint calibration model is established, and the transformation matrix (rotation matrix RR and translation vector tt) between the coordinate systems of each sensor is calculated by matching the calibration plate or feature points.

[0021] Dynamically compensate for coordinate system offsets caused by mechanical vibrations and use a Kalman filter to update the transformation matrix in real time.

[0022] Time and space synchronization frequency: The synchronization period is 10ms, which adapts to the rapid changes of dynamic obstacles in mine scenes.

[0023] For high-frequency sensors (such as IMU, 100Hz), a sliding window is used to cache data and output fused spatiotemporal aligned data packets according to the synchronization period.

[0024] Step S2: extract information, extract the feature data obtained in step S1, and form a dynamic topology map, while marking temporary obstacle information; in step S2: the update frequency of the dynamic topology map is positively correlated with the speed of the mobile terminal; Step S3: Information processing: The information extracted in step S2 is used to adaptively adjust the data fusion weights of different sensors based on the dynamic environment perception results. In step S3, the adaptive weight adjustment mechanism includes: Dynamically assign sensor confidence based on ambient light intensity, point cloud density, and motion acceleration; Predict the optimal weight combination in sensor degradation scenarios through online learning models; Step S4: Information update, combining particle filtering and graph optimization algorithm to achieve robust pose estimation and real-time map update; In step S4: the robust relocalization mechanism includes the following: Establish a global map including obstacle labels and lane structure labels; When positioning is lost, the search range is quickly narrowed down through semantic matching and the pose is restored in combination with geometric verification; The particle filter algorithm adopts an adaptive particle number adjustment mechanism, which is implemented as follows: Initial number of particles: set to Ninit=1000 according to the scene complexity to cover the uncertainty range of the initial pose; Convolutional Neural Network Framework: Dynamic adjustment rules: When the position reliability (calculated by the covariance matrix determinant) is below the threshold = 0.05, the number of particles increases to Nmax = 3000 to expand the search space; In the stable tracking phase, the number of particles is reduced to Nmin = 500 to reduce the computational overhead; Optimization method: Use KLD sampling to estimate the difference between the particle distribution and the true posterior distribution in real time, and dynamically adjust the number of particles to meet the preset accuracy; An annealing factor α = 0.98 is introduced to gradually narrow the particle distribution range and accelerate convergence; the upper limit of the number of particles is dynamically constrained according to the real-time computing power to avoid system delays.

[0025] Step S5: Re-positioning. When an environmental change is detected or the positioning deviation exceeds a threshold, a weight adjustment algorithm based on transfer learning and a re-positioning mechanism based on semantic feature matching are triggered to restore the system's positioning capability. Positioning deviation trigger threshold Typical range: 0.3m - 1.0m Setting basis: Positioning accuracy requirement: Mine safety regulations require that personnel positioning error ≤ 1 meter and equipment positioning error ≤ 0.5 meter.

[0026] Sensor error characteristics: UWB ranging error is about 0.1-0.3 meters, and IMU cumulative error is about 1-3 meters per minute.

[0027] Device positioning: Triggered when the deviation is >0.5 meters for 1 second continuously.

[0028] Signal strength fluctuation threshold: Typical range: 3dB-6dB (based on RSSI signal standard deviation) Setting basis: Wireless signals in mines are affected by multipath effects, and the RSSI fluctuation standard deviation can reach 5-10 dB, requiring filtering of instantaneous interference.

[0029] Motion speed threshold: Typical range: 0.5 m / s- 2 m / s; Setting basis: The maximum speed of the mine car is usually ≤3 m / s, and the walking speed of personnel is ≤1.5 m / s.

[0030] The relocation mechanisms include: Extract key semantic features of the scene based on convolutional neural network (CNN); Multi-branch feature extraction network: Input: multimodal data (RGB image, LiDAR point cloud, IMU sequence); Network structure: Vision branch: uses a lightweight ResNet-18 backbone, removes the last two fully connected layers, and outputs a 256-dimensional feature vector; Point cloud branch: Use the PointNet+ structure to perform hierarchical feature extraction on LiDAR point clouds and output 128-dimensional features; IMU branch: Modeling time series motion based on LSTM network, with hidden layer dimension of 64; Fusion layer: The three features are concatenated and input into the fully connected layer, which outputs the sensor weight vector.

[0031] Adaptive attention module: dynamically adjusts feature importance based on the state of the environment; Environmental context input: light intensity (0-1 normalized), dust concentration (0-1), motion acceleration (m / s2 ); Generate the attention mask A∈R3 through the multi-layer perceptron and perform Hadamard product with the sensor weight; Output the corrected weight: w′=w⊙A.

[0032] Relocalization semantic matching network: Input: local semantic features (such as lane outlines and equipment identification) collected in real time and the global map database; Network structure: Feature encoder: Use MobileNetV3 to extract image semantic descriptors with an output dimension of 128; Matching layer: Calculates the matching scores between local features and candidate regions in the global map based on cosine similarity; Geometric verification: PnP (Perspective-n-Point) algorithm combined with RANSAC is used to optimize the pose estimation results; Build a hierarchical matching strategy to prioritize searching for similar semantic areas in historical maps; Hierarchical structure design: First layer: semantic rough matching: Input: local semantic features collected in real time (lane outline, equipment identification code).

[0033] Matching method: Calculate the matching scores with candidate regions in the global map based on cosine similarity and select the top 50 candidate regions.

[0034] Acceleration strategy: Locality-sensitive hashing (LSH) is used to build a semantic feature index library, and the search time is ≤10ms.

[0035] Second layer: geometric precision matching: Input: candidate region after rough matching.

[0036] Matching method: Extract the geometric features of the candidate area (such as SIFT key points and LiDAR point cloud curvature) and perform point cloud registration using the ICP algorithm.

[0037] Verification conditions: A valid match is considered when the registration error is <0.1m and the inlier ratio is >70%.

[0038] Dynamic optimization mechanism: When the ambient lighting changes dramatically, the semantic layer weight is increased (accounting for 70%) and the geometric layer weight is weakened; In static scenes, geometric layer matching is prioritized (accounting for 80%) to improve positioning accuracy to the centimeter level.

[0039] The RANSAC algorithm is used to eliminate mismatched points and optimize the relocalization results; The relocation mechanism also includes: Using a transfer learning algorithm, the positioning weight is dynamically adjusted according to the difference between historical positioning data and the current environment. The relocation mechanism based on the transfer learning weight adjustment algorithm and the relocation mechanism based on semantic feature matching operate synchronously. Transfer learning algorithms also include: Transfer learning model architecture: A feature extractor is built based on a deep residual network (ResNet), pre-trained on a historical mine scene dataset (including positioning data under different lighting and dust concentrations).

[0040] Design a lightweight adaptation layer: Map the output features of the pre-trained model to the current environment domain, and use domain adversarial training (DANN) to reduce domain differences.

[0041] Dynamic weight adjustment mechanism: Input: Difference measurement (such as cosine similarity) between historical positioning data (feature vector Fhist) and current real-time data (Fcurr).

[0042] Output: Fusion weight adjustment coefficient Δw of each sensor (vision, LiDAR, IMU).

[0043] Algorithm flow: Calculate the difference matrix D = .

[0044] Through the online reinforcement learning model (Q-Learning), the weight distribution strategy is dynamically optimized according to the DD value, with the goal of minimizing the positioning error.

[0045] Transfer learning and relocalization synchronization: When a sudden change in the environment (such as a sudden increase in dust concentration) is detected, the transfer learning module is triggered, the pre-trained model parameters are loaded, and the weight adjustment is completed within 5ms.

[0046] It runs in parallel with the semantic matching mechanism, shares the feature encoding layer, and reduces computational redundancy.

[0047] The RANSAC algorithm also includes: Mismatched point elimination process: Step 1: Model hypothesis generation: Randomly select 4 pairs of matching points (based on semantic features extracted by CNN) and calculate the initial pose transformation matrix Tinit (using the PnP algorithm).

[0048] Step 2: Inlier screening: Set the reprojection error threshold ϵ = 2.5 pixels, calculate the projection error of all matching points under the model, and count the number of inliers.

[0049] Step 3: Iterative optimization: The maximum number of iterations Nmax = 200, dynamic adjustment: if the current inlier ratio is > 80%, terminate the iteration early.

[0050] Step 4: Model verification: Select the model with the highest inlier ratio, optimize Tfinal using the Levenberg-Marquardt algorithm, and output the accurate pose.

[0051] Parameter settings: Inlier threshold: dynamically adjusted according to the scene, relaxed to ϵ = 5.0 pixels in dust interference scenes.

[0052] Adaptive iteration strategy: Based on the covariance matrix of the particle filter, when the position confidence is low, increase Nmax to 500 times.

[0053] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A mine scene adaptive relocalization method based on multimodal perception, characterized by: Includes the following methods: Step S1: Acquire information, using multimodal sensors to collect visual images, spatial features, wireless signal features, and inertial motion data of the mine's internal environment; Step S2: extract information, extract the feature data obtained in step S1, and form a dynamic topological map, while marking temporary obstacle information; Step S3: Information processing, the information extracted in step S2 is used to adaptively adjust the data fusion weights of different sensors based on the dynamic environment perception results; Step S4: Information update, combining particle filtering and graph optimization algorithm to achieve robust pose estimation and real-time map update; Step S5: Repositioning. When an environmental change is detected or the positioning deviation exceeds a threshold, the weight adjustment algorithm based on transfer learning and the repositioning mechanism based on semantic feature matching are triggered to restore the system positioning capability.

2. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: Multimodal sensors include: A combination of at least two of the following: visible light camera, infrared camera, RGB-D camera, lidar, IMU, and millimeter-wave radar.

3. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: In multimodal sensors: The visual sensor provides texture information, the lidar generates accurate three-dimensional structure, the IMU compensates for motion blur, and the millimeter-wave radar detects dynamic obstacles; A spatiotemporal synchronization algorithm is used to eliminate timestamp differences in multi-sensor data.

4. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: In step S2: the update frequency of the dynamic topology map is positively correlated with the speed of the mobile terminal.

5. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: In step S3: the adaptive weight adjustment mechanism includes: Dynamically assign sensor confidence based on ambient light intensity, point cloud density, and motion acceleration; The optimal weight combination in sensor degradation scenarios is predicted through an online learning model.

6. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: In step S4: the robust relocation mechanism includes the following: Establish a global map including obstacle labels and lane structure labels; When positioning is lost, the search range is quickly narrowed down through semantic matching, and the pose is restored in combination with geometric verification.

7. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 1, characterized in that: The relocation mechanisms include: Extract key semantic features of the scene based on convolutional neural network (CNN); Build a hierarchical matching strategy to prioritize searching for similar semantic areas in historical maps; The RANSAC algorithm is used to eliminate mismatched points and optimize the relocalization results.

8. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 7, characterized in that: The relocation mechanism also includes: Using the transfer learning algorithm, the positioning weight is dynamically adjusted according to the differences between historical positioning data and the current environment.

9. The method for adaptive relocalization in mine scenes based on multimodal perception according to claim 8, characterized in that: The relocalization mechanism based on the weight adjustment algorithm of transfer learning and the relocalization mechanism based on semantic feature matching run synchronously.

Images