Underground robot multi-modal data fusion method and system

Abstract

The invention provides an underground robot multi-modal data fusion method and system, and the method comprises the steps: firstly collecting multi-dimensional data in an underground environment through a preset multi-modal sensor module, and carrying out the alignment of the multi-dimensional data through a preset data alignment module, so as to obtain the multi-modal data; the modal data is subjected to feature extraction through a preset feature extraction module to obtain multi-modal features, and then the multi-modal features are subjected to feature fusion to obtain underground fusion data, so that the robustness and feature expression ability of the multi-source sensing system in the underground complex environment are improved; therefore, the environment modeling and autonomous navigation capabilities of the robot in a complex, dynamic and low-visibility environment are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, particularly to the field of robot perception and information fusion technology, and more specifically, to a method and system for multimodal data fusion of underground robots. Background Technology

[0002] Robots, as autonomous mobile platforms, have become a research and application hotspot in the field of robotics. Especially in complex underground environments such as mines, tunnels, and underground utility tunnels, robot systems need to perform tasks such as data acquisition, scene modeling, and autonomous exploration in environments with insufficient lighting, complex structures, confined spaces, limited communication, and potential hazards. This presents significant challenges to robot systems. Typically, robot systems acquire specific data about their surroundings through various types of sensors and use corresponding data processing methods to convert this sensor data into representative information that the robot can use for analysis, thereby completing its perception of the surrounding environment. However, traditional single-sensor perception methods acquire limited information and lack sufficient resistance to interference in complex environments, making it difficult to meet the stability and robustness requirements of these special environments.

[0003] Multimodal sensor fusion relies on multimodal data fusion methods for perception. Currently, multimodal sensor fusion technology is widely used in the field of ground robotics to improve the perception capabilities of robot systems. For example, it combines data from multiple sensors such as vision, LiDAR, inertial measurement unit (IMU), and ultrasound, and uses techniques such as filtering, feature extraction, and deep learning to achieve accurate environmental modeling and real-time perception. However, these methods are mostly designed for well-lit, open, and highly structured ground environments, and will still face many challenges when applied to complex underground environments.

[0004] Underground environments typically suffer from poor lighting and low visibility, leading to a decline in the quality of information acquired by traditional visual sensors, or even their failure. Secondly, due to the complex and confined spatial structure, sensors that acquire surrounding depth information, such as lidar, depth cameras, and ultrasonic sensors, are severely interfered with, further increasing the difficulty of robot localization and navigation. Furthermore, inconsistencies in data acquisition frequencies and spatiotemporal misalignment between different sensors also make effective fusion of multimodal data challenging.

[0005] Therefore, there is an urgent need for a multimodal data fusion method for underground robots that is suitable for complex underground environments. This method can effectively integrate information from multiple sensors, improve perception accuracy and adaptability while ensuring system robustness, and thus provide reliable technical support for the autonomous operation of underground robots. Summary of the Invention

[0006] In view of the above problems, the purpose of this invention is to provide a multimodal data fusion method and system for underground robots, so as to solve the problems that due to the complex and narrow spatial structure, sensors such as lidar, depth cameras and ultrasonic sensors that acquire surrounding depth information are severely interfered with, which further increases the difficulty of robot localization and navigation. In addition, the problems of inconsistent data acquisition frequencies and spatiotemporal misalignment between different sensors also make it difficult to effectively fuse multimodal data.

[0007] This invention provides a multimodal data fusion method for underground robots, comprising:

[0008] Multi-dimensional data in the underground environment is collected by a preset multi-modal sensor module, and the multi-dimensional data is aligned by a preset data alignment module to obtain multi-modal data.

[0009] Multimodal features are obtained by extracting features from the modal data using a preset feature extraction module.

[0010] Feature fusion is performed on the multimodal features to obtain underground fused data.

[0011] Preferably, the multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot; wherein, the acquisition of multi-dimensional data in the underground environment includes:

[0012] The visual sensor acquires images of the underground environment to obtain visual images;

[0013] The lidar sensor emits laser pulses into the underground environment and receives the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment.

[0014] The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

[0015] Preferably, the multi-dimensional data is aligned using a preset data alignment module to obtain multimodal data, including:

[0016] A high-precision clock synchronization mechanism is adopted to perform time alignment on the visual image, the point cloud information, and the motion data based on the high stability clock inside the system to obtain time alignment data; wherein, a unified timestamp is pre-set in the visual sensor, lidar sensor, and inertial measurement unit;

[0017] Based on the sensor intrinsic and extrinsic parameters pre-calibrated for the vision sensor, lidar sensor, and inertial measurement unit, coordinate transformation is performed to map the time-aligned data to the body coordinate system of the underground robot, forming time- and space-aligned multimodal data. The multimodal data includes standard vision images, standard point cloud information, and standard motion data that correspond to the visual images, point cloud information, and motion data, and are time- and space-aligned with each other.

[0018] Preferably, the multimodal features include visual features, point cloud features, and motion features; wherein, the multimodal features are obtained by extracting features from the modal data using a preset feature extraction module, including:

[0019] Multi-scale feature extraction is performed on the standard visual image to obtain the visual features;

[0020] Feature extraction is performed on the standard point cloud information to obtain the point cloud features;

[0021] Feature extraction is performed on the standard motion data to obtain the motion features.

[0022] Preferably, multi-scale feature extraction is performed on the standard visual image to obtain the visual features, including:

[0023] The deep convolutional neural network in the feature extraction module is used as the backbone network to extract features from the standard visual image to obtain local texture features. The feature map of the local texture features is multiplied with a preset brightness enhancement weight matrix using a preset element-wise multiplication method to obtain enhanced local texture features. The enhanced local texture features are then transformed through the deep network of the deep convolutional neural network to obtain high-level semantic features.

[0024] The high-level semantic features are used as the visual features.

[0025] Prioritize feature extraction from the standard point cloud information to obtain the point cloud features, including:

[0026] The standard point cloud information is augmented and supplemented using a preset neighborhood point difference algorithm to obtain augmented point cloud information. A preset graph attention mechanism is used to establish point cloud topology relationships based on the augmented point cloud information to form complete point cloud information. The complete point cloud information is then processed by a point cloud processing neural network in the feature extraction module to obtain local geometric features.

[0027] The local geometric features are constructed in three dimensions using the deep part of the point cloud processing neural network to obtain a global three-dimensional structural feature vector;

[0028] The global three-dimensional structural feature vector is used as the point cloud feature.

[0029] Preferably, feature extraction is performed on the standard motion data to obtain the motion features, including:

[0030] The standard motion data is preprocessed and smoothed using a preset extended Kalman filter algorithm to obtain smoothed motion data.

[0031] The acceleration and angular velocity data in the smooth motion data are extracted using a temporal convolutional network or a long short-term memory network in the feature extraction module to obtain motion features; the motion features reflect the pose change trend information of the underground robot.

[0032] Preferably, feature fusion is performed on the multimodal features to obtain underground fused data, including:

[0033] Based on the motion features, the point cloud features are dynamically matched and enhanced to obtain enhanced point cloud features;

[0034] Visual features are used as the query input of the multi-head attention layer, and the enhanced point cloud features are used as the key-value pairs of the multi-head attention layer. Feature fusion is performed through the multi-scale feature pyramid fusion mechanism and the channel attention mechanism of the multi-head attention layer to obtain the enhanced feature output of the degraded features in low light and narrow environments.

[0035] The enhanced feature output is decoded to obtain underground fusion data.

[0036] This invention also provides a multimodal data fusion system for underground robots, enabling data acquisition and fusion in low-light and confined spaces using the multimodal data fusion method for underground robots described above, including:

[0037] Multimodal sensor modules are used to collect multi-dimensional data from underground environments;

[0038] A data alignment module is used to align the multi-dimensional data to obtain multimodal data;

[0039] The feature extraction module is used to extract features from the modal data to obtain multimodal features;

[0040] A multimodal fusion module is used to perform feature fusion on the multimodal features to obtain underground fusion data.

[0041] Preferably, the multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot; wherein,

[0042] The visual sensor is used to acquire images of the underground environment to obtain visual images;

[0043] The lidar sensor is used to emit laser pulses into the underground environment and receive the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment.

[0044] The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

[0045] As can be seen from the above technical solution, the multimodal data fusion method and system for underground robots provided by the present invention first collects multi-dimensional data in the underground environment through a preset multimodal sensor module, and then aligns the multi-dimensional data through a preset data alignment module to obtain multimodal data. Next, the modal data is used to extract features from the modal data through a preset feature extraction module to obtain multimodal features. Finally, the multimodal features are fused to obtain underground fused data, thereby improving the robustness and feature representation ability of the multi-source perception system in complex underground environments, and further improving the robot's environmental modeling and autonomous navigation capabilities in complex, dynamic, and low-visibility environments. Attached Figure Description

[0046] Other objects and results of the invention will become more apparent and readily understood by referring to the following description taken in conjunction with the accompanying drawings, and with a more complete understanding of the invention. In the drawings:

[0047] Figure 1 This is a flowchart of a multimodal data fusion method for underground robots according to an embodiment of the present invention;

[0048] Figure 2 A detailed flowchart of the multimodal data fusion method for underground robots according to an embodiment of the present invention;

[0049] Figure 3 This is a system block diagram of an underground robot multimodal data fusion system according to an embodiment of the present invention. Detailed Implementation

[0050] Underground environments typically suffer from poor lighting and low visibility, leading to a decline in the quality of information acquired by traditional visual sensors, or even their failure. Secondly, due to the complex and confined spatial structure, sensors that acquire surrounding depth information, such as lidar, depth cameras, and ultrasonic sensors, are severely interfered with, further increasing the difficulty of robot localization and navigation. Furthermore, inconsistencies in data acquisition frequencies and spatiotemporal misalignment between different sensors also make effective fusion of multimodal data challenging.

[0051] To address the aforementioned problems, this invention provides a method and system for multimodal data fusion of underground robots. The specific embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0052] To illustrate the multimodal data fusion method and system for underground robots provided by this invention, Figures 1-3 The embodiments of the present invention are illustrated by way of example.

[0053] The following description of exemplary embodiments is merely illustrative and is in no way intended to limit the invention or its application or use. Techniques and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques and equipment should be considered part of the specification.

[0054] like Figure 1 , Figure 2 As shown in the figure, this embodiment provides a multimodal data fusion method for underground robots, including:

[0055] S1: Collect multi-dimensional data in the underground environment through a preset multi-modal sensor module, and align the multi-dimensional data through a preset data alignment module to obtain multi-modal data;

[0056] S2: Extract features from the modal data using a preset feature extraction module to obtain multimodal features;

[0057] S3: Perform feature fusion on the multimodal features to obtain underground fusion data.

[0058] Step S1 involves collecting multi-dimensional data from the underground environment using a preset multi-modal sensor module, and aligning the multi-dimensional data using a preset data alignment module to obtain multi-modal data.

[0059] The multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot; it collects multi-dimensional data from the underground environment, including:

[0060] S111: Acquire visual images of the underground environment using the visual sensor;

[0061] S112: The laser radar sensor emits laser pulses into the underground environment and receives the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment;

[0062] S113: The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

[0063] Specifically, in this embodiment, a multimodal sensor module integrates multiple sensors, including a LiDAR sensor, a vision sensor, and an inertial measurement unit (IMU), on the robot platform to collect multimodal data in the underground environment. The LiDAR sensor, based on the time-of-flight (ToF) ranging principle, calculates the distance to target objects by emitting laser pulses and receiving reflected signals, thereby acquiring high-precision three-dimensional spatial structure data of the underground environment. Furthermore, this LiDAR sensor has a wide field of view and a high scanning frequency, enabling real-time generation of dense point clouds (point cloud information), providing a reliable basis for environmental modeling and obstacle detection. The vision sensor uses a high-resolution RGB camera to capture the color and texture features of the environment through optical imaging. Moreover, this vision sensor supports real-time image processing, combining deep learning algorithms to achieve target recognition and scene segmentation to form visual images, compensating for the shortcomings of LiDAR in acquiring semantic information. An inertial measurement unit (IMU) consists of a three-axis accelerometer and a gyroscope. It monitors the robot's motion state parameters, such as angular velocity and acceleration, in real time as motion data. The IMU provides short-term, high-precision pose estimation through integration. When fused with LiDAR and visual data, it can significantly improve the robot's positioning robustness in environments without GPS.

[0064] The multi-dimensional data is aligned using a preset data alignment module to obtain multimodal data, including:

[0065] S121: A high-precision clock synchronization mechanism is adopted to perform time alignment on the visual image, the point cloud information, and the motion data based on the high stability clock inside the system to obtain time alignment data; wherein, a unified timestamp is pre-set in the visual sensor, the lidar sensor, and the inertial measurement unit;

[0066] S122: Based on the sensor intrinsic and extrinsic parameters pre-calibrated for the vision sensor, lidar sensor, and inertial measurement unit, coordinate transformation is performed to map the time-aligned data to the body coordinate system of the underground robot to form time- and space-aligned multimodal data; wherein, the multimodal data includes standard vision images, standard point cloud information, and standard motion data that correspond to the vision image, the point cloud information, and the motion data, and are time- and space-aligned with each other.

[0067] Specifically, in one embodiment, a data alignment module is used to unify heterogeneous sensor data in the time and spatial domains. This method includes the following steps: Time synchronization: A high-precision clock synchronization mechanism is employed, based on a highly stable internal clock (such as an atomic clock or TCXO oscillator), to assign a unified timestamp to each sensor's data, ensuring time consistency across multiple data sources. The time synchronization accuracy is controlled at the microsecond level to meet real-time requirements. Spatial synchronization: Based on pre-calibrated sensor intrinsic parameters (such as focal length and distortion coefficient) and extrinsic parameters (such as installation position and attitude), a coordinate transformation matrix (such as a rigid body transformation matrix) is used to map the data collected by each sensor to the robot's body coordinate system. Specifically, this includes: LiDAR point cloud data being transformed to the robot's base coordinate system via an extrinsic parameter matrix; visual sensor image data undergoing distortion correction based on camera calibration parameters and spatially registered with LiDAR data using a perspective projection model; and IMU data being aligned with other sensor data after installation position compensation.

[0068] Step S2 is the process of extracting features from the modal data using a preset feature extraction module to obtain multimodal features; wherein, the multimodal features include visual features, point cloud features, and motion features; wherein, extracting features from the modal data using the preset feature extraction module to obtain multimodal features includes:

[0069] S21: Perform multi-scale feature extraction on the standard visual image to obtain the visual features;

[0070] S22: Perform feature extraction on the standard point cloud information to obtain the point cloud features;

[0071] S23: Perform feature extraction on the standard motion data to obtain the motion features.

[0072] Specifically, multi-scale feature extraction is performed on the standard visual image to obtain the visual features, including:

[0073] S211: The deep convolutional neural network in the feature extraction module is used as the backbone network to extract features from the standard visual image to obtain local texture features. The feature map of the local texture features is multiplied with the preset brightness enhancement weight matrix using a preset element-wise multiplication method to obtain enhanced local texture features. The enhanced local texture features are then transformed through the deep network of the deep convolutional neural network to obtain high-level semantic features.

[0074] S212: Use the high-level semantic features as the visual features.

[0075] Specifically, in this embodiment, for visual feature extraction under limited lighting conditions in underground environments, a deep convolutional neural network (such as ResNet, EfficientNet, etc.) is used to extract features from RGB images at multiple scales. Local texture features are captured through convolutional kernel operations, and a pre-trained brightness enhancement weight matrix is ​​used to perform element-wise multiplication with the feature maps of local texture features. This amplifies key features and suppresses the noise caused by low lighting conditions, strengthening the feature expression of key areas and improving visual features in underground low-light environments. The deep network further converts low-level visual features into high-level semantic features, achieving feature dimensionality upgrade from pixel-level to semantic-level.

[0076] Specifically, feature extraction is performed on the standard point cloud information to obtain the point cloud features, including:

[0077] S221: The standard point cloud information is augmented and supplemented based on a preset neighborhood point difference algorithm to obtain augmented point cloud information. A point cloud topology relationship is established based on the augmented point cloud information through a preset graph attention mechanism to form complete point cloud information. The complete point cloud information is then processed by a point cloud processing neural network in the feature extraction module to obtain local geometric features.

[0078] S222: The local geometric features are constructed in three dimensions using the deep part of the point cloud processing neural network to obtain a global three-dimensional structural feature vector.

[0079] S223: Use the global three-dimensional structural feature vector as the point cloud feature.

[0080] Specifically, in this embodiment, for point cloud feature extraction under conditions of narrow space, complex structure, and numerous obstructions in underground environments, a point cloud processing neural network (including but not limited to PointNet++, VoxelNet, or 3D sparse convolutional networks) can be used to learn geometric features from LiDAR point cloud data. First, the original point cloud is preprocessed using neighborhood point interpolation and density compensation strategies to alleviate the problem of missing points caused by uneven sampling or local occlusion, improving the integrity and continuity of the point set distribution. Local point set features (local geometric features) are extracted using a multilayer perceptron (MLP). Based on this, a graph attention mechanism is introduced to model the topological relationships between point clouds, strengthening the long-distance dependencies and semantic consistency between local features. Finally, operations such as max pooling are used to achieve global feature aggregation for 3D construction, outputting a rotation-invariant global 3D structural feature vector.

[0081] Feature extraction is performed on the standard motion data to obtain the motion features, including:

[0082] S231: The standard motion data is preprocessed and smoothed using a preset extended Kalman filter algorithm to obtain smoothed motion data;

[0083] S232: The acceleration and angular velocity data in the smooth motion data are extracted using the temporal convolutional network or long short-term memory network in the feature extraction module to obtain motion features; the motion features reflect the pose change trend information of the underground robot.

[0084] Specifically, in this embodiment, for motion feature extraction, the linear acceleration and angular velocity data acquired by the IMU are first processed using a sliding window, and motion pattern features are extracted using a Temporal Convolutional Network (TCN) or a Long Short-Term Memory Network (LSTM). This feature vector contains information on the robot's pose change trend, which is used for motion state prediction and dynamic compensation.

[0085] Step S3 is the process of performing feature fusion on the multimodal features to obtain subsurface fused data, wherein performing feature fusion on the multimodal features to obtain subsurface fused data includes:

[0086] S31: Dynamically match and enhance the point cloud features based on the motion features to obtain enhanced point cloud features;

[0087] S32: Using visual features as the query input of the multi-head attention layer, and the enhanced point cloud features as the key-value pairs of the multi-head attention layer, feature fusion is performed through the multi-scale feature pyramid fusion mechanism and channel attention mechanism of the multi-head attention layer to obtain the enhanced feature output of the degraded features in low light and narrow environments.

[0088] S33: Decode the enhanced feature output to obtain underground fusion data.

[0089] Before proceeding to step S3, the feature vectors of each modal feature in the multimodal feature set need to be mapped to a unified dimensional space using a dedicated encoder (fully connected network or projection matrix), and L2 normalization is applied to ensure feature scale consistency. This process achieves feature space alignment through learnable linear transformations, establishing a compatible mathematical foundation for subsequent cross-modal feature fusion.

[0090] Step S3 is completed by the multimodal fusion module, which employs a hierarchical interaction mechanism to achieve deep integration and collaborative enhancement of cross-modal features. First, this module utilizes motion features extracted by the IMU to construct a spatiotemporal compensation model. Through rigid body transformation, it achieves dynamic matching and enhancement of multi-frame LiDAR point cloud features, effectively alleviating the point cloud sparsity problem in underground environments. Second, it employs a bimodal interaction network based on an attention mechanism, using visual features as query input and LiDAR features as key-value pairs. A multi-head attention layer dynamically calculates the feature correlation between modalities and generates an adaptive weight mask. Simultaneously, it combines multi-scale feature pyramid fusion and channel attention mechanisms to enhance degraded features in complex environments such as low light and narrow spaces. Finally, the fused result is output, and the built-in environmental state perception unit monitors parameters such as light intensity and spatial density in real time, dynamically adjusting the fusion strategy. Under low light conditions, LiDAR features are prioritized, while in narrow space scenarios, high-resolution visual features are emphasized, forming a closed-loop optimization mechanism. This fusion method significantly improves the robustness and feature representation capability of multi-source sensing systems in complex underground environments.

[0091] This improves the robot's perception accuracy and stability in complex underground environments; supports collaborative work of multiple sensors, enhancing the system's fault tolerance and environmental adaptability; introduces a dynamic weight adjustment mechanism, which can flexibly adjust the contribution of different types of data according to environmental changes; and adopts an efficient fusion algorithm, taking into account both system real-time performance and computational resource limitations.

[0092] In summary, a more specific and complete embodiment requires a multimodal sensor module, which includes: a visual sensor, a lidar sensor, and an inertial measurement unit (IMU) for collecting data related to the underground environment; a data alignment module, comprising temporal alignment and spatial alignment. The temporal alignment part ensures data consistency in the temporal dimension by allocating a unified timestamp, while the spatial alignment part projects heterogeneous data onto a unified coordinate system using sensor intrinsic parameters and extrinsic parameters between sensors to achieve spatial consistency; a feature extraction module extracts features from data of different modalities using different methods; and a multimodal fusion module first fuses motion features and point cloud features extracted from the IMU to compensate for the sparseness of point clouds in underground space, and then further interacts and fuses the enhanced point cloud features and visual features to improve the expressive power of the features and finally obtain the result of multimodal data fusion.

[0093] First, a hardware platform is built, deploying multi-source sensors on the robot platform. Specifically, this can include an integrated mechanical multi-line LiDAR sensor, a vision sensor, and an inertial measurement unit. Each sensor is responsible for collecting data on the spatial structure, visual information, and motion status of the environment. ROS (Robot Operating System) is used as the system development platform to achieve modular functional deployment.

[0094] The sensor data acquisition and synchronization steps are implemented based on the ROS framework. Dedicated driver nodes are used to acquire multi-source data in real time from LiDAR, vision sensors, and inertial measurement units. A high-precision time synchronization module is employed in the time synchronization model to achieve microsecond-level timestamp alignment, as shown in Equation 1.

[0095] t sync =t hw +Δt cal +ε (1)

[0096] Where t hw For hardware timestamps, Δt cal To calibrate the compensation amount, ε~N(0,σ) 2 The synchronization error is represented by the data. After time alignment, the extrinsic parameters between sensors are calibrated using a multi-sensor joint calibration tool. The spatial coordinate system is unified by solving for the optimal rotation matrix R and translation vector t. Taking the joint calibration of LiDAR and camera as an example, based on a calibration board, the camera and multi-line LiDAR are jointly calibrated. Calibration board features are extracted from the image and point cloud respectively, and the rotation matrix and translation vector are solved using an optimization algorithm. Point cloud data P can be obtained from the multi-line LiDAR. l =(x l ,y l ,z l The point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system using the following formula:

[0097] P c =R lc P l +t lc (2)

[0098] Where R lc It is a rotation matrix, and t is... lc Translation vector.

[0099] In this specific embodiment, in order to achieve effective fusion of data from multiple sensors in a complex underground environment, it is first necessary to extract representative features from the raw data. These features not only need to capture key environmental information, but also need to have good comparability and compatibility to facilitate subsequent multimodal data fusion.

[0100] Then feature extraction is performed, which includes:

[0101] Image Feature (Visual Feature) Extraction: For data from visual sensors, we employed a deep convolutional neural network, ResNet-18, as the backbone network to extract high-level semantic features. ResNet-18 is a classic deep learning model that effectively solves the gradient vanishing problem commonly encountered during deep network training by stacking multiple residual blocks to construct a deep network structure. In this process, the input image undergoes a series of convolutions, pooling, and non-linear activation operations, ultimately outputting a fixed-length feature vector containing key information about the image, such as object boundaries, texture patterns, and the overall layout of the scene. Specifically, addressing the common problems of low light, weak image contrast, and severe noise interference in underground environments, this invention further introduces a feature enhancement strategy based on element-wise multiplication. Specifically, after the backbone network extracts preliminary local texture features, the feature map is multiplied element-wise with a preset brightness enhancement weight matrix to amplify the feature response of key areas and suppress the influence of background noise. The brightness enhancement weight matrix can be obtained through offline training or dynamically adjusted according to the current light intensity to adapt to lighting changes in different underground scenes. Let I represent the local texture feature map obtained by the backbone network from the input visual image, with dimensions H×W×C, where H and W are the height and width of the feature map, respectively, and C is the number of channels. Define a brightness enhancement matrix M with the same dimensions as I. Each element m in this matrix... hwc This represents the enhancement coefficient at the corresponding position, used to adjust the intensity at that location in the feature map:

[0102] I'(h,w,c)=I(h,w,c)×m hwc

[0103] Where h∈[0,H), w∈[0,W), and c∈[0,C) represent the position indices in feature map I, and the brightness enhancement weight matrix M can be obtained through offline training. This enhancement mechanism not only effectively alleviates the problem of decreased image feature representation capability under low illumination conditions, but also improves the feature alignment accuracy and semantic consistency in the subsequent multimodal fusion process. Finally, the enhanced image features are output as a fixed-length high-level semantic feature vector, serving as an important component of multimodal data fusion.

[0104] Point Cloud Feature Extraction: For 3D point cloud data acquired by LiDAR, this specific embodiment uses the PointNet++ network to extract geometric features. PointNet++ is an improved version of PointNet, specifically designed to handle irregularly distributed 3D point cloud data. It refines the feature representation within local regions through hierarchical sampling and grouping strategies, thereby better capturing fine-grained structural information in the point cloud. Considering the common problems of narrow spaces, complex structures, and numerous obstructions in underground environments, which often result in sparse, unevenly distributed, and locally missing point cloud data acquired by LiDAR, this specific embodiment introduces the following key technologies based on the standard PointNet++ structure: neighborhood point interpolation and density compensation strategies. In the early stages of point cloud feature extraction, the original point cloud P is processed using a neighborhood point interpolation algorithm based on K-nearest neighbors (KNN) to generate synthetic points in sparse regions to supplement missing geometric information. Let p... i The geometric mean of the K nearest neighbors of ∈P is The coordinates of the composite point p′i are calculated as follows:

[0105]

[0106] Wherein, weight w j It can be determined using the inverse proportional distance method:

[0107]

[0108] Where d(·,·) represents the Euclidean distance between two points, and Υ is an adjustable parameter, usually taking a positive value; the graph attention mechanism is embedded to further enhance the model's ability to understand the topological relationships of the point cloud. This invention introduces a graph attention mechanism into the local feature aggregation layer of PointNet++. For each point i and its neighborhood point set... The attention coefficient is calculated as follows:

[0109] a ij =softmax j (LeakyReLU(a T [f i ||f j ]))

[0110] in, This represents the eigenvector of point i. Let be a learnable parameter vector, and || denote the vector concatenation operation. The final aggregated feature representation is as follows:

[0111]

[0112] in, It is a learnable linear transformation matrix, and the activation function is ReLU. Finally, after the above enhancement processing, the point cloud processing neural network outputs a set of global 3D structural feature vectors with rotation invariance and spatial robustness, which serve as important inputs for subsequent multimodal fusion modules, significantly improving the system's environmental perception and modeling capabilities in narrow, low-visibility underground environments.

[0113] In this way, we can obtain a set of feature vectors that describe the spatial distribution characteristics of point clouds, which is crucial for understanding the spatial structure of the surrounding environment.

[0114] IMU Feature (Motion Feature) Extraction: The Inertial Measurement Unit (IMU) provides information about the robot's own motion state, including parameters such as acceleration and angular velocity. To utilize this information, we need to extract meaningful time-series features. Specifically, the robot's posture change trajectory can be calculated from the IMU data and converted into a form suitable for fusion with other sensor data. For example, the raw IMU data can be preprocessed and smoothed using complementary filters or extended Kalman filters (EKF) algorithms, and then key features reflecting the robot's motion trends can be extracted.

[0115] Feature vector mapping: Since features extracted from different types of sensor data may have different dimensions and representations, it is necessary to map all feature vectors to the same dimensional space before multimodal data fusion. This is typically accomplished through a fully connected layer that takes feature vectors from each modality as input and transforms them into a unified dimensional feature representation through linear transformation and a non-linear activation function. The purpose of this is to ensure that data from all modalities can be processed and compared in a consistent manner during the subsequent fusion process.

[0116] Once the feature extraction and mapping steps are completed, the next step is to effectively integrate these data from different modalities to form a comprehensive understanding of the current environmental state.

[0117] In this specific embodiment, the multimodal fusion module employs a Transformer-based fusion network: that is, a fusion network based on the Transformer architecture is constructed to achieve this goal. While the Transformer was originally designed for natural language processing tasks, its powerful self-attention mechanism is equally suitable for handling multimodal data fusion problems. This network includes a multi-head attention mechanism module, which can automatically learn the important relationships between different modalities; in addition, there is a positional encoding module to preserve the order information of the input data, which is particularly important for understanding dynamically changing environments.

[0118] Input and Output: The input to this fusion network is a set of feature vectors generated by each modality feature extraction module, while its output is an environmental state feature vector that integrates all input information. This output vector not only contains rich environmental information but also reflects the inherent relationships between various sensor data in the current environment.

[0119] Model Training and Inference: During the training phase, supervised learning methods can be used to optimize the overall system performance. The loss function consists of two parts: classification loss and regression loss. The classification loss evaluates the model's accuracy for a specific task (such as obstacle recognition); the regression loss measures the difference between the model's predictions and the true values. By minimizing these two types of loss functions, the model can gradually learn how to accurately fuse multimodal data and make reasonable decisions. Flexibility in the Inference Phase: In practical applications, some sensors may temporarily fail due to environmental factors (such as low visibility affecting visual sensors). Therefore, during the inference phase, the system can dynamically adjust the selection of input modalities based on currently available sensor data, ensuring that a certain level of perception is maintained even when some sensors are unavailable. This multi-layered, modular feature extraction and fusion strategy not only improves the robot's perception accuracy and robustness in complex underground environments but also leaves ample room for future expansion and optimization.

[0120] Therefore, the multimodal data fusion method for underground robots in this embodiment can improve the robot's perception accuracy and stability in complex underground environments. Furthermore, this method supports the collaborative work of multiple sensors, which can enhance the system's fault tolerance and environmental adaptability. The introduction of a dynamic weight adjustment mechanism can flexibly adjust the contribution of different types of data according to environmental changes. The adoption of an efficient fusion algorithm takes into account both system real-time performance and computational resource limitations.

[0121] like Figure 3 As shown, the present invention also provides a multimodal data fusion system 100 for underground robots, used to implement the multimodal data fusion method for underground robots as described above, for data acquisition and fusion in low light and confined spaces, including:

[0122] The multimodal sensor module 110 is used to collect multidimensional data in the underground environment;

[0123] Data alignment module 120 is used to align the multi-dimensional data to obtain multimodal data;

[0124] Feature extraction module 130 is used to extract features from the modal data to obtain multimodal features;

[0125] The multimodal fusion module 140 is used to perform feature fusion on the multimodal features to obtain underground fusion data.

[0126] The multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot;

[0127] The visual sensor is used to acquire images of the underground environment to obtain visual images;

[0128] The lidar sensor is used to emit laser pulses into the underground environment and receive the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment.

[0129] The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

[0130] For a more specific implementation of the underground robot multimodal data fusion system, please refer to the embodiments of the underground robot multimodal data fusion method described above, which will not be repeated here.

[0131] As described above, the underground robot multimodal data fusion system provided by the present invention collects multi-dimensional data in the underground environment through a multimodal sensor module, aligns the multi-dimensional data through a data alignment module to obtain multimodal data, extracts features from the modal data through a feature extraction module to obtain multimodal features, and then fuses the multimodal features to obtain underground fused data. This improves the robustness and feature representation capability of the multi-source perception system in complex underground environments, thereby enhancing the robot's environmental modeling and autonomous navigation capabilities in complex, dynamic, and low-visibility environments.

[0132] The method and system for multimodal data fusion of underground robots according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the method and system for multimodal data fusion of underground robots proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the contents of the appended claims.

Claims

1. A method for multimodal data fusion of underground robots, characterized in that, include: Multi-dimensional data in the underground environment is collected by a preset multi-modal sensor module, and the multi-dimensional data is aligned by a preset data alignment module to obtain multi-modal data. Multimodal features are obtained by extracting features from the modal data using a preset feature extraction module. Feature fusion is performed on the multimodal features to obtain underground fused data.

2. The multimodal data fusion method for underground robots as described in claim 1, characterized in that, The multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot; wherein, it collects multi-dimensional data from the underground environment, including: The visual sensor acquires images of the underground environment to obtain visual images; The lidar sensor emits laser pulses into the underground environment and receives the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment. The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

3. The multimodal data fusion method for underground robots as described in claim 2, wherein the multi-dimensional data is aligned using a preset data alignment module to obtain multimodal data, comprising: A high-precision clock synchronization mechanism is adopted to perform time alignment on the visual image, the point cloud information, and the motion data based on the high stability clock inside the system to obtain time alignment data; wherein, a unified timestamp is pre-set in the visual sensor, lidar sensor, and inertial measurement unit; Based on the sensor intrinsic and extrinsic parameters pre-calibrated for the vision sensor, lidar sensor, and inertial measurement unit, coordinate transformation is performed to map the time-aligned data to the body coordinate system of the underground robot, forming time- and space-aligned multimodal data. The multimodal data includes standard vision images, standard point cloud information, and standard motion data that correspond to the visual images, point cloud information, and motion data, and are time- and space-aligned with each other.

4. The multimodal data fusion method for underground robots as described in claim 3, characterized in that, The multimodal features include visual features, point cloud features, and motion features; wherein, the multimodal features are obtained by extracting features from the modal data through a preset feature extraction module, including: Multi-scale feature extraction is performed on the standard visual image to obtain the visual features; Feature extraction is performed on the standard point cloud information to obtain the point cloud features; Feature extraction is performed on the standard motion data to obtain the motion features.

5. The multimodal data fusion method for underground robots as described in claim 4, characterized in that, Multi-scale feature extraction is performed on the standard visual image to obtain the visual features, including: The deep convolutional neural network in the feature extraction module is used as the backbone network to extract features from the standard visual image to obtain local texture features. The feature map of the local texture features is multiplied with a preset brightness enhancement weight matrix using a preset element-wise multiplication method to obtain enhanced local texture features. The enhanced local texture features are then transformed through the deep network of the deep convolutional neural network to obtain high-level semantic features. The high-level semantic features are used as the visual features.

6. The multimodal data fusion method for underground robots as described in claim 4, characterized in that, Feature extraction is performed on the standard point cloud information to obtain the point cloud features, including: The standard point cloud information is augmented and supplemented using a preset neighborhood point difference algorithm to obtain augmented point cloud information. A preset graph attention mechanism is used to establish point cloud topology relationships based on the augmented point cloud information to form complete point cloud information. The complete point cloud information is then processed by a point cloud processing neural network in the feature extraction module to obtain local geometric features. The local geometric features are constructed in three dimensions using the deep part of the point cloud processing neural network to obtain a global three-dimensional structural feature vector; The global three-dimensional structural feature vector is used as the point cloud feature.

7. The multimodal data fusion method for underground robots as described in claim 4, characterized in that, Feature extraction is performed on the standard motion data to obtain the motion features, including: The standard motion data is preprocessed and smoothed using a preset extended Kalman filter algorithm to obtain smoothed motion data. The acceleration and angular velocity data in the smooth motion data are extracted using a temporal convolutional network or a long short-term memory network in the feature extraction module to obtain motion features; the motion features reflect the pose change trend information of the underground robot.

8. The multimodal data fusion method for underground robots as described in claim 7, characterized in that, Feature fusion of the multimodal features to obtain subsurface fused data includes: Based on the motion features, the point cloud features are dynamically matched and enhanced to obtain enhanced point cloud features; The visual features are used as the query input of the multi-head attention layer, and the enhanced point cloud features are used as the key-value pairs of the multi-head attention layer. The multi-scale feature pyramid fusion mechanism and the channel attention mechanism of the multi-head attention layer are used to perform feature fusion to obtain the enhanced feature output of the degraded features in low light and narrow environments. The enhanced feature output is decoded to obtain underground fusion data.

9. A multimodal data fusion system for underground robots, used for data acquisition and fusion in low-light, confined spaces using the multimodal data fusion method for underground robots as described in any one of claims 1-8, comprising: Multimodal sensor modules are used to collect multi-dimensional data from underground environments; A data alignment module is used to align the multi-dimensional data to obtain multimodal data; The feature extraction module is used to extract features from the modal data to obtain multimodal features; A multimodal fusion module is used to perform feature fusion on the multimodal features to obtain underground fusion data.

10. The underground robot multimodal data fusion system as described in claim 9, characterized in that, The multimodal sensor module includes at least a vision sensor, a lidar sensor, and an inertial measurement unit integrated on the underground robot; wherein, The visual sensor is used to acquire images of the underground environment to obtain visual images; The lidar sensor is used to emit laser pulses into the underground environment and receive the reflected signals of the laser pulses to form point cloud information about the three-dimensional spatial structure of the underground environment. The inertial measurement unit is used to monitor the pose of the underground robot to generate motion data.

Images