A Deep Learning-Based Multi-Sensor Fusion SLAM Method and System

Abstract

This invention provides a deep learning-based multi-source sensor fusion SLAM method and system, belonging to the field of SLAM technology. The method is applied to a system including a feature processing module, a quality prediction module, an anomaly detection module, an adaptive fusion module, and a SLAM output module. Specifically, it involves: extracting features from heterogeneous raw data from multiple sensors; predicting the data quality of the multiple sensors using a sensor quality prediction network; detecting abnormal data using a cross-modal anomaly detection network; combining the prediction and detection results; calculating the dynamic fusion weights of the multiple sensors and optimizing the state estimation; and building an environmental map based on the optimized state estimation. This invention reduces the impact of sensor data quality fluctuations and anomaly interference on the positioning results through quality prediction and cross-modal anomaly detection, and adapts to changes in the dynamic and complex environment by combining adaptive dynamic fusion weights, ensuring positioning accuracy and thus improving the accuracy of the constructed environmental map.

Description

Technical Field

[0001] This invention relates to the field of SLAM technology, and in particular to a multi-source sensor fusion SLAM method and system based on deep learning. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology is a core support for autonomous navigation systems such as autonomous driving, drone inspection, and mobile robots to achieve autonomous movement. Its performance directly determines the reliability of autonomous navigation systems in complex environments. With the development of multi-sensor fusion technology, visual-inertial-GNSS fusion SLAM systems, by integrating the advantages of visual sensors, inertial measurement units, and global navigation satellite systems, can achieve high-precision positioning and mapping in different scenarios, and have gradually become the mainstream SLAM solution used in autonomous navigation systems.

[0003] However, existing visual-inertial GNSS fusion SLAM systems often rely on residual calculations to infer sensor quality after data usage, failing to anticipate changes in sensor performance. Corrections are only made after low-quality data contaminates the system state estimate, leading to accumulated positioning errors. Furthermore, existing anomaly detection schemes designed for single sensors are unsuitable for multi-source sensor fusion SLAM systems, struggling to identify deep logical conflicts between multi-sensor data and easily causing distorted fusion results. Additionally, traditional fusion frameworks often employ fixed weights or weight allocation strategies based on simple statistical models, unable to adjust the trust level of each sensor in real-time according to the current environmental state, resulting in lower positioning accuracy in dynamic and complex environments.

[0004] It is evident that existing visual-inertial GNSS fusion SLAM systems are still affected by fluctuations in sensor data quality and abnormal interference during operation, and the fusion strategy is also difficult to adapt to dynamic and complex environments, resulting in low positioning accuracy. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing visual-inertial GNSS fusion SLAM systems for localization and mapping in dynamic and complex environments. These systems are still affected by fluctuations in sensor data quality and abnormal interference, and the fusion strategies are difficult to adapt to dynamic and complex environments, resulting in low positioning accuracy. This invention provides a deep learning-based multi-source sensor fusion SLAM method and system. It utilizes a sensor quality prediction network to output accurate evaluation results of multi-source sensor data quality in real time, enabling dynamic control of sensor data status. Simultaneously, it uses a cross-modal anomaly detection network to quickly identify abnormal data in multi-source sensors, reducing the impact of sensor data quality fluctuations and abnormal interference on the positioning results. Furthermore, it combines the evaluation results and the identification results to calculate the dynamic fusion weights of multi-source sensors to adapt to changes in dynamic and complex environments, improve the adaptability of the fusion process, ensure positioning accuracy in dynamic and complex environments, and thus improve the accuracy of the constructed environmental map.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] Deep learning-based multi-source sensor fusion SLAM methods include:

[0008] Acquire heterogeneous raw data from multiple source sensors and extract features from the heterogeneous raw data;

[0009] Based on the extracted feature data, the data quality of multi-source sensors is predicted through a sensor quality prediction network.

[0010] Based on the extracted feature data, abnormal data is detected through a cross-modal anomaly detection network;

[0011] By combining data quality prediction results and abnormal data detection results, the dynamic fusion weights of multi-source sensors are calculated;

[0012] The state estimation of SLAM is optimized based on dynamic fusion weights, and an environment map is built based on the optimized state estimation.

[0013] By using a sensor quality prediction network to predict the data quality of each sensor in advance, the accumulation of positioning errors can be avoided. Simultaneously, a cross-modal anomaly detection network quickly identifies anomalous data to prevent it from entering the fusion process, reducing the negative impact of data quality fluctuations and anomalies on positioning calculations. Based on this, dynamic fusion weights for multi-source sensors are calculated by combining data quality assessment results and anomaly identification results. This allows the fusion strategy to adapt to real-time changes in dynamic and complex environments, improving the adaptability of the fusion process to different environments. Furthermore, the state estimation of SLAM is optimized based on the dynamic fusion weights to ensure the accuracy of positioning results in various dynamic and complex scenarios, thereby ensuring the accuracy of the constructed environmental map and effectively avoiding map defects caused by positioning errors.

[0014] Furthermore, the prediction of data quality from multi-source sensors based on the extracted feature data via a sensor quality prediction network includes:

[0015] Based on a two-branch temporal modeling network with a Transformer neural network as the first branch and a long short-term memory network as the second branch, and a multilayer perceptron quality prediction head, a sensor quality prediction network is obtained.

[0016] Based on the sensor type, add the corresponding modality embedding to the extracted feature data and input it into the sensor quality prediction network;

[0017] Based on the dual-branch temporal modeling network, feature processing is performed on the feature data after adding modality embedding to obtain the first branch features and the second branch features.

[0018] Based on the gating mechanism, the features of the first branch and the features of the second branch are fused to obtain the fused features;

[0019] Based on the acquired fusion features, the multilayer perceptron quality prediction head outputs data quality scores for various sensors.

[0020] Furthermore, the detection of anomalous data based on the extracted feature data through a cross-modal anomaly detection network includes:

[0021] Based on the conditional variational autoencoder and graph convolutional network, a cross-modal anomaly detection network is obtained, and the extracted feature data is input into the cross-modal anomaly detection network.

[0022] The conditional variational autoencoder reconstructs each class of feature data through latent space mapping, obtains the corresponding reconstructed features, and outputs distribution anomaly scores.

[0023] The reconstruction error is calculated based on each type of feature data and the corresponding reconstruction feature, and then weighted and summed with the corresponding distribution anomaly score to obtain the anomaly score;

[0024] Abnormal data is filtered based on anomaly scores, and the anomaly types of the filtered abnormal data are classified using a graph convolutional network to obtain the anomaly type corresponding to each abnormal data.

[0025] Furthermore, the calculation of the dynamic fusion weights of the multi-source sensors by combining the data quality prediction results and the abnormal data detection results includes:

[0026] A heterogeneous sensor graph is constructed by taking the various types of sensors from multiple sources as nodes, the correlation between sensors as edges, and the cosine similarity between corresponding feature data as the weight of the edges.

[0027] Based on heterogeneous sensor graphs, the graph attention weights of each sensor node are obtained through a heterogeneous graph attention network;

[0028] The sensor quality score for each type of sensor at the current moment is obtained based on the data quality prediction results. The abnormal score of the corresponding feature data for each type of sensor is obtained based on the abnormal data detection results. The dynamic fusion weight of each type of sensor is obtained by combining the corresponding graph attention weight.

[0029] Furthermore, the heterogeneous raw data from the multi-source sensors includes visual data, inertial measurement data, and position data.

[0030] Furthermore, the feature extraction of the heterogeneous raw data includes:

[0031] Visual data is preprocessed and multi-dimensional feature maps are extracted through a pre-trained image classification network. Based on the feature pyramid network, feature maps of adjacent dimensions are fused layer by layer starting from the feature map with the highest dimension. Visual features are obtained through the fused feature maps.

[0032] The inertial measurement data and position data are extracted by a sliding window, and the inertial measurement input matrix and position input matrix are constructed according to the extracted time segments.

[0033] The inertial measurement input matrix is ​​input into a pre-trained one-dimensional convolutional neural network to obtain local features of inertial measurement.

[0034] The position input matrix is ​​fed into a pre-trained multilayer perceptron to obtain position features.

[0035] Furthermore, the step of optimizing the state estimation of SLAM based on dynamic fusion weights and establishing an environment map based on the optimized state estimation includes:

[0036] The calculated dynamic fusion weights of various sensors are applied to the corresponding error terms to establish a weighted total cost function;

[0037] The weighted total cost function is minimized by a nonlinear least squares algorithm, and the state parameters within a preset sliding window are iteratively optimized to obtain the optimal pose estimation results of the multi-source sensors at each time step within the preset sliding window.

[0038] Based on the optimal pose estimation results, the environmental features collected by multiple sensors are associated with a unified global coordinate system to obtain an environmental map.

[0039] Furthermore, in the process of optimizing the state estimation of SLAM based on dynamic fusion weights, the following is also performed:

[0040] Real-time calculation of performance monitoring metrics values; when the number of consecutive frames whose performance monitoring metrics values ​​meet the triggering conditions exceeds a preset threshold, meta-learning adaptation is triggered.

[0041] Based on the heterogeneous raw data from multiple sources of sensors after triggering meta-learning adaptation, the data quality prediction results, abnormal data detection results, and the state estimation error of the optimized state estimation, the corresponding model parameters are calculated by combining meta-learning adaptation to update feature extraction, data quality prediction, abnormal data detection, and dynamic fusion weights.

[0042] A deep learning-based multi-source sensor fusion SLAM system, used to perform the method described in any of the above, including:

[0043] The feature processing module is connected to the multi-source sensor and is used to receive heterogeneous raw data from the multi-source sensor and extract features from the heterogeneous raw data.

[0044] The quality prediction module, connected to the feature processing module, is used to predict the data quality of multi-source sensors based on the extracted feature data through the sensor quality prediction network.

[0045] An anomaly detection module, connected to the feature processing module, is used to detect anomalous data through a cross-modal anomaly detection network based on the extracted feature data.

[0046] The adaptive fusion module is connected to the quality prediction module and the anomaly detection module respectively, and is used to calculate the dynamic fusion weight of multi-source sensors by combining the data quality prediction results and the anomaly data detection results.

[0047] The SLAM output module, connected to the adaptive fusion module, is used to optimize the state estimation of SLAM according to the dynamic fusion weights and to build an environment map based on the optimized state estimation.

[0048] Furthermore, the multi-source sensor fusion SLAM system also includes:

[0049] The meta-learning adaptation module is connected to the feature processing module, the quality prediction module, the anomaly detection module, and the adaptive fusion module, respectively. It is used to calculate the corresponding model parameters based on the heterogeneous raw data from the multi-source sensors after triggering meta-learning adaptation, the data quality prediction results, the anomaly detection results, and the state estimation error of the optimized state estimation, combined with meta-learning adaptation to update feature extraction, data quality prediction, anomaly detection, and dynamic fusion weights.

[0050] The beneficial effects of this invention are:

[0051] (1) The sensor quality prediction network predicts the data quality of each sensor in advance to avoid the accumulation of positioning deviation. At the same time, the cross-modal anomaly detection network quickly identifies abnormal data to prevent abnormal data from entering the fusion process and reduce the negative impact of data quality fluctuations and abnormal interference from various sensors on positioning calculation. On this basis, the dynamic fusion weights of multi-source sensors are calculated by combining the data quality assessment results and the anomaly identification results, so that the fusion strategy can adapt to the scene changes of dynamic and complex environments in real time, improve the adaptability of the fusion process to different environments, and then optimize the state estimation of SLAM according to the dynamic fusion weights to ensure the accuracy of positioning results in various dynamic and complex scenarios, thereby ensuring the accuracy of the constructed environmental map and effectively avoiding map defects caused by positioning deviation.

[0052] (2) A dual-branch temporal modeling network based on Transformer and Long Short-Term Memory network is used to predict sensor data quality by combining modal embedding and gating fusion mechanism. It can efficiently capture the long temporal dependencies of feature data through Transformer and accurately mine local temporal features with the help of Long Short-Term Memory network. Then, the data quality score is output through the multilayer perceptron quality prediction head to improve the accuracy of data quality assessment of different modal sensors.

[0053] (3) Cross-modal anomaly detection is carried out by relying on the synergistic effect of conditional variational autoencoder and graph convolutional network. Conditional variational autoencoder can realize accurate reconstruction of various feature data through latent space mapping. Then, combined with the weighted calculation of distribution anomaly score and reconstruction error, it can efficiently screen anomaly data. Then, the graph convolutional network can further classify the anomaly data by type. While improving the accuracy of anomaly data identification, it can clarify the anomaly type, provide more comprehensive anomaly information, and effectively reduce the interference of anomaly data on the localization result.

[0054] (4) By constructing a heterogeneous sensor graph with sensors as nodes, correlation as edges and cosine similarity as weights, and then combining the heterogeneous graph attention network to obtain graph attention weights, and at the same time, the data quality score and anomaly score are fused to realize the calculation of dynamic fusion weights. This makes the calculation of dynamic fusion weights take into account both the intrinsic correlation between sensors and the real-time quality status and anomalies of each sensor, making the weight allocation more reasonable and adaptable, avoiding weight deviation caused by a single factor, and ensuring that multi-source sensor data can give full play to their respective advantages in the fusion process.

[0055] (5) For different sensor data, corresponding feature extraction algorithms are used to extract features to ensure the accuracy of the extracted feature data. After completing the corresponding dynamic fusion weight calculation, the state estimation is optimized by weighted total cost function and nonlinear least squares algorithm to improve the pose estimation accuracy of fused SLAM and reduce map defects caused by insufficient feature extraction or pose estimation deviation.

[0056] (6) A meta-learning adaptation mechanism is set up, which can dynamically update the model parameters of each module based on real-time data and error feedback, thereby enhancing the robustness and reliability of the fusion SLAM system in dynamic and complex scenarios. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of a process of the present invention;

[0058] Figure 2 This is a schematic diagram of a structure according to an embodiment of the present invention.

[0059] The module includes: 1. Feature processing module, 2. Quality prediction module, 3. Anomaly detection module, 4. Adaptive fusion module, 5. SLAM output module, 6. Meta-learning adaptation module, and 7. Multi-source sensor. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] Example:

[0062] Deep learning-based multi-source sensor fusion SLAM methods, such as Figure 1 As shown, it includes:

[0063] Acquire heterogeneous raw data from multiple source sensors and extract features from the heterogeneous raw data;

[0064] Based on the extracted feature data, the data quality of multi-source sensors is predicted through a sensor quality prediction network.

[0065] Based on the extracted feature data, abnormal data is detected through a cross-modal anomaly detection network;

[0066] By combining data quality prediction results and abnormal data detection results, the dynamic fusion weights of multi-source sensors are calculated;

[0067] The state estimation of SLAM is optimized based on dynamic fusion weights, and an environment map is built based on the optimized state estimation.

[0068] SLAM requires simultaneous localization and mapping, which necessitates data that reflects both the vehicle's motion and the environmental spatial characteristics. However, a single sensor cannot simultaneously meet both requirements. For example, visual cameras can capture environmental texture and semantics but are sensitive to changes in lighting and lack absolute motion references; inertial measurement units (IMUs) can output high-frequency motion parameters but suffer from cumulative drift; and global navigation satellite systems (GNSS) can provide absolute position but are susceptible to occlusion failures.

[0069] Therefore, this embodiment uses multiple source sensors to acquire complementary data to cover the data requirements of SLAM. Specifically, the multiple source sensors mainly include a visual camera, an inertial measurement unit (IMU), and a global navigation satellite system (GNSS).

[0070] The core of SLAM is to determine the relationship between the environment and the carrier through data association. However, different sensors operate on completely different principles and have completely different output formats. Visual cameras output a two-dimensional pixel matrix, inertial measurement units (IMUs) output one-dimensional time-series values, and global navigation satellite systems (GNSS) output discrete coordinates in a geodetic coordinate system. This heterogeneity makes it impossible to directly compare and fuse data from different sensors, hindering corresponding data association. Therefore, after acquiring heterogeneous raw data from multiple sensors, feature extraction must be performed on the heterogeneous raw data to provide reliable data for subsequent SLAM applications.

[0071] The heterogeneous raw data from the multi-source sensors includes visual data, inertial measurement data, and position data.

[0072] The feature extraction of heterogeneous raw data includes:

[0073] Visual data is preprocessed and multi-dimensional feature maps are extracted through a pre-trained image classification network. Based on the feature pyramid network, feature maps of adjacent dimensions are fused layer by layer starting from the feature map with the highest dimension. Visual features are obtained through the fused feature maps.

[0074] The inertial measurement data and position data are extracted by a sliding window, and the inertial measurement input matrix and position input matrix are constructed according to the extracted time segments.

[0075] The inertial measurement input matrix is ​​input into a pre-trained one-dimensional convolutional neural network to obtain local features of inertial measurement.

[0076] The position input matrix is ​​fed into a pre-trained multilayer perceptron to obtain position features.

[0077] Considering the different data formats of heterogeneous raw data, different data processing methods are used to carry out corresponding feature extraction.

[0078] The visual data acquired by the visual camera needs to be preprocessed. The visual data mainly consists of stereo image pairs acquired by the visual camera. First, distortion correction operations need to be performed according to the camera's intrinsic parameters to eliminate interference from lens optical distortion on feature matching. Then, the corresponding images are normalized to the pixel value range [0, 1], and finally, their resolution is unified to meet the input specifications of the subsequent image classification network.

[0079] In this embodiment, a pre-trained EfficientNet-B4 image classification network is used to perform feature processing on the pre-processed visual data. The expression for feature extraction using the EfficientNet-B4 image classification network is as follows:

[0080] ;

[0081] in, for A set of multi-dimensional features extracted from visual data at all times. for Visual data at any given moment The four elements in the text correspond to respectively The image classification network outputs a 4-layer feature map, where... This is for low-level detailed features such as edges and corners, with dimensions of 48×48×24. The texture features are of medium to shallow depth, with dimensions of 24×24×40. This represents the local structural features of the middle and upper levels, with dimensions of 12×12×112. It represents high-level scene semantic features with dimensions of 6×6×320.

[0082] When using the EfficientNet-B4 image classification network, the weights of its first 5 layers need to be frozen, and the weights of the last 3 layers need to be fine-tuned to adapt to SLAM scenarios.

[0083] After outputting four layers of feature maps, the feature pyramid network starts from the highest layer feature map, adjusts the number of channels through 1×1 convolution, and adds them element-wise with the next layer feature map upsampled to the corresponding dimension. The feature maps of adjacent dimensions are fused sequentially. Finally, global average pooling is performed on the fused four layers of feature maps, and each feature map is compressed and then stitched together to obtain the final visual features.

[0084] The expression for the feature pyramid network is:

[0085] ;

[0086] in, For the fusion Time of the first Layer feature map, For the original Time of the first Layer feature map, This is a 1×1 convolution operation function. This is the upsampling operation function.

[0087] For inertial measurement data acquired by the inertial measurement unit (IMU), time segments need to be extracted using a sliding window before preprocessing. The size of the sliding window can be set according to requirements; in this embodiment, the sliding window size is set to 100, corresponding to 0.5 seconds of data. The inertial measurement data mainly includes the acceleration and angular velocity data of the carrier. After extracting time segments using a sliding window and constructing the corresponding inertial measurement input matrix, the zero bias value is obtained through static calibration and subtracted from the original data to remove the IMU zero bias. Then, low-pass filtering is used to suppress the corresponding high-frequency vibration noise.

[0088] Then, the preprocessed inertial measurement input matrix is ​​input into a pre-trained one-dimensional convolutional neural network to extract the corresponding features and obtain local features of inertial measurement, so as to effectively capture short-term operating patterns in inertial measurement data, such as acceleration, turning, vibration and other information.

[0089] In this embodiment, the one-dimensional convolutional neural network adopts a 6-layer 1D-CNN network structure with convolutional kernel sizes of [7, 5, 5, 3, 3, 3] and channel numbers of [32, 64, 128, 128, 256, 256]. A max pooling layer is added after every two convolutional layers with a pooling kernel size of 2 and a stride of 2 to reduce dimensionality and retain key temporal features.

[0090] The expression for the one-dimensional convolutional temporal network to extract local features of inertial measurement is as follows:

[0091] ;

[0092] in, For extraction Local features of inertial measurement at time t, It is a one-dimensional convolutional neural network with 6 stacked layers. For Sliding window with time as the endpoint Acceleration data collected internally, For Sliding window with time as the endpoint Angular velocity data collected internally.

[0093] For position data acquired by the Global Navigation Satellite System (GNSS), time-series segments are also extracted using a sliding window and then preprocessed. The position data includes the carrier's three-dimensional position and uncertainty data. After extracting time-series segments using a sliding window and constructing a corresponding position input matrix, the position and uncertainty of adjacent frames are identified. When the position difference between adjacent frames exceeds three times the uncertainty, it is determined to be a jump point. Data from the previous frame is used for interpolation and replacement to eliminate GNSS jump points and ensure data continuity.

[0094] The preprocessed position input matrix is ​​then input into a pre-trained multilayer perceptron. In this embodiment, a three-layer multilayer perceptron is used, with the first layer having an output dimension of 64, the second layer having an output dimension of 128, and the third layer having an output dimension of 128. Each layer uses the GELU activation function, and LayerNorm (layer normalization) is added after each layer to normalize the features and accelerate training convergence. The final output is a position feature with a dimension of 128 to capture the position trend and reliability information of the position data.

[0095] The expression for extracting location features using the 3-layer multilayer perceptron is as follows:

[0096] ;

[0097] in, For extraction Locational characteristics at any given moment It is a multilayer perceptron with three stacked layers. For Sliding window with time as the endpoint Three-dimensional position data collected internally. For Sliding window with time as the endpoint Uncertainty data collected internally.

[0098] After completing the corresponding feature extraction, although the multi-source heterogeneous raw data has been transformed into structured, high-dimensional feature vectors, these features only contain information about the data itself and do not involve the judgment of data reliability. In fact, SLAM sensor data has inherent fluctuations in dynamic environments, and simple feature data cannot directly reflect the quality risks of sensor data.

[0099] Meanwhile, to avoid the cumulative bias caused by the quality of posterior residual judgment, a sensor quality prediction network is used to predict the data quality. The data quality is predicted before SLAM state estimation, and low-quality data weights are directly filtered or reduced to avoid subsequent errors.

[0100] The step of predicting the data quality of multi-source sensors based on the extracted feature data using a sensor quality prediction network includes:

[0101] Based on a two-branch temporal modeling network with a Transformer neural network as the first branch and a long short-term memory network as the second branch, and a multilayer perceptron quality prediction head, a sensor quality prediction network is obtained.

[0102] Based on the sensor type, add the corresponding modality embedding to the extracted feature data and input it into the sensor quality prediction network;

[0103] Based on the dual-branch temporal modeling network, feature processing is performed on the feature data after adding modality embedding to obtain the first branch features and the second branch features.

[0104] Based on the gating mechanism, the features of the first branch and the features of the second branch are fused to obtain the fused features;

[0105] Based on the acquired fusion features, the multilayer perceptron quality prediction head outputs data quality scores for various sensors.

[0106] The sensor quality prediction network in this embodiment mainly adopts a composite structure of dual-branch temporal modeling and a multilayer perceptron prediction head, which can simultaneously capture the temporal dependence and short-term dynamic changes of sensor features. Specifically, a Transformer neural network is used as the first branch of the dual-branch temporal modeling network, focusing on long-term temporal correlations, and a Long Short-Term Memory network is used as the second branch, focusing on short-term dynamic changes. Then, the fused temporal features output by the dual-branch temporal modeling network are mapped to the corresponding quality scores through the multilayer perceptron quality prediction head.

[0107] Since sensor quality prediction networks cannot distinguish the sources of heterogeneous features, modal embedding is added to the extracted feature data to obtain feature data with type representation, which can effectively avoid quality prediction bias caused by cross-modal feature confusion.

[0108] The features with added modality embedding are then arranged into a feature sequence according to time steps, and then input into a two-branch temporal modeling network for processing.

[0109] The first branch is a Transformer neural network. This branch effectively captures long-range dependencies in feature sequences through a multi-head self-attention mechanism. Specifically, it employs a 6-layer Transformer encoder, with each layer including multi-head self-attention and a feedforward network. Its expression is as follows:

[0110] ;

[0111] in, for The first branch feature at time step, For Transformer neural networks, For modal embedding Sliding window with time as the endpoint The first feature data collected inside, For modal embedding Sliding window with time as the endpoint The final feature data collected.

[0112] The second branch is a Long Short-Term Memory (LSTM) network, which can fully mine the forward and backward temporal information of the feature sequence through bidirectional recurrent processing. Specifically, it uses a 3-layer bidirectional LSTM with a hidden dimension of 256, and its expression is as follows:

[0113] ;

[0114] in, , for The second branch feature at time t, for The hidden state at all times for Cellular state at any given moment It is a long short-term memory network with a three-layer bidirectional LSTM structure.

[0115] For the first branch features and the second branch features output by the first branch and the second score, a gating mechanism is introduced to achieve adaptive combination.

[0116] Specifically, first calculate the corresponding gating coefficient. This is used to adjust the fusion weights of the two branch features, and its calculation expression is:

[0117] ;

[0118] in, It is a linear transformation matrix. The last time step feature output by the first branch. The hidden state output by the second branch. It is the Sigmoid activation function. This is the bias vector.

[0119] Then, calculate the fused temporal characteristics based on the gating coefficients. Its calculation expression is:

[0120] ;

[0121] in, For the fused temporal features, The gating coefficient, The last time step feature output by the first branch. This refers to the hidden state output by the second branch.

[0122] Finally, the prediction results are output through the multilayer perceptron quality prediction head. The multilayer perceptron quality prediction head adopts a 5-layer multilayer perceptron structure. The fused temporal features are used as input. After multilayer linear transformation and activation function processing, the multi-source sensor quality score for the next K steps can be output. This quality score can be activated by Sigmoid to ensure that the output is in the range of [0, 1]. 1 represents the best sensor data quality and 0 represents the worst quality. This completes the prediction of the future data quality of the multi-source sensors.

[0123] The expression for the multilayer perceptron quality prediction head is:

[0124] ;

[0125] in, for Sensor quality score of the K-step vision camera at Time Future. for The sensor mass fraction of the inertial measurement unit at a future time of K steps. for Sensor quality score of the K-step global satellite navigation system in the future This is a quality prediction head for a multilayer perceptron.

[0126] After completing the quality prediction, cross-modal anomaly detection is further initiated to address the multi-source data conflict problem that the quality prediction cannot cover, and to provide data support for subsequent fusion.

[0127] The step of detecting anomalous data through a cross-modal anomaly detection network based on extracted feature data includes:

[0128] Based on the conditional variational autoencoder and graph convolutional network, a cross-modal anomaly detection network is obtained, and the extracted feature data is input into the cross-modal anomaly detection network.

[0129] The conditional variational autoencoder reconstructs each class of feature data through latent space mapping, obtains the corresponding reconstructed features, and outputs distribution anomaly scores.

[0130] The reconstruction error is calculated based on each type of feature data and the corresponding reconstruction feature, and then weighted and summed with the corresponding distribution anomaly score to obtain the anomaly score;

[0131] Abnormal data is filtered based on anomaly scores, and the anomaly types of the filtered abnormal data are classified using a graph convolutional network to obtain the anomaly type corresponding to each abnormal data.

[0132] A cascaded structure of conditional variational autoencoder and graph convolutional network is adopted as a cross-modal anomaly detection network to detect anomalous data, and can simultaneously realize single-modal anomaly quantification and cross-modal anomaly type determination.

[0133] When performing anomaly detection, a latent space is first preset on the input multimodal feature data using a conditional variational autoencoder (CVA). The input consists of the extracted three types of feature data. The CVA specifically employs a 6-layer fully connected network structure, with the dimensions of each layer being [896, 512, 256, 128, 128, 64]. After multiple layers of linear transformation and activation processing, the mean and standard deviation of the output features are used to learn the latent probability distribution of normal multimodal feature data. The expression is as follows:

[0134] ;

[0135] in, for The mean of the characteristic data at time point, for Standard deviation of characteristic data at time point It is a conditional variational autoencoder.

[0136] Then, by using reparameterization techniques to generate latent variables, we can ensure the randomness of the latent variables and ensure that the gradient can be backpropagated through the mean and standard deviation, which can provide a data foundation for subsequent reconstruction and distribution difference calculation.

[0137] Then, the conditional variational autoencoder reconstructs the multimodal feature data based on the latent variables. The decoder receives the latent variables as input, performs reverse mapping through a fully connected network, and outputs reconstructed features with the same dimension as the original input.

[0138] ;

[0139] in, , and These are the reconstructed versions. Visual features at any moment, local features from inertial measurement, and positional features. For decoder, These are the potential variables generated.

[0140] Then, the reconstruction error is calculated based on the reconstructed feature data. Reconstruction error The calculation expression is:

[0141] .

[0142] The larger the reconstruction error, the more significant the deviation between the current input features and the normal distribution, and the higher the probability of anomalies.

[0143] Based on this, the anomaly score is calculated by combining the reconstruction error with the KL divergence, which is used to measure the difference between the feature distribution learned by the conditional variational autoencoder and the standard normal distribution. Taking into account both the feature reconstruction effect and the degree of distribution deviation, the higher the anomaly score, the more severe the anomaly of the current data.

[0144] The expression for calculating the abnormal score is:

[0145] ;

[0146] in, for Anomaly scores in feature data are collected continuously. and In this embodiment, the preset weighting coefficients are used. =0.7, =0.3, Let KL be the divergence function.

[0147] Finally, a four-layer graph convolutional network is introduced to classify anomaly types. Specifically, a heterogeneous sensor graph is first established using various types of sensors from multiple sources as nodes, the correlation between sensors as edges, and the cosine similarity between corresponding feature data as edge weights. This heterogeneous sensor graph is then input into the graph convolutional network, and multi-layer graph convolution operations are used to capture the dependencies between sensor nodes and anomaly propagation characteristics to determine the corresponding anomaly types. In this embodiment, 12 predefined anomaly types are specifically set, including visual occlusion, motion blur, sudden illumination changes, IMU saturation, vibration interference, magnetic field anomalies, GNSS multipath propagation, and signal occlusion.

[0148] After completing the corresponding data quality prediction and anomaly detection, dynamic fusion weights of multi-source sensors are set according to the corresponding results to ensure that high-quality, anomaly-free data dominates the state estimation, while suppressing the interference of low-quality or anomaly data.

[0149] Specifically, the calculation of dynamic fusion weights for multi-source sensors by combining data quality prediction results and abnormal data detection results includes:

[0150] A heterogeneous sensor graph is constructed by taking the various types of sensors from multiple sources as nodes, the correlation between sensors as edges, and the cosine similarity between corresponding feature data as the weight of the edges.

[0151] Based on heterogeneous sensor graphs, the graph attention weights of each sensor node are obtained through a heterogeneous graph attention network;

[0152] The sensor quality score for each type of sensor at the current moment is obtained based on the data quality prediction results. The abnormal score of the corresponding feature data for each type of sensor is obtained based on the abnormal data detection results. The dynamic fusion weight of each type of sensor is obtained by combining the corresponding graph attention weight.

[0153] By constructing a heterogeneous sensor graph to reflect the dependencies between sensor devices, and then using a heterogeneous graph attention network to learn the attention weights of sensor nodes, the graph attention network employs a four-layer heterogeneous graph attention layer structure. Each layer calculates the correlation strength between nodes through an attention mechanism. For any two nodes in the graph, the node features are first linearly transformed, then the transformed features are concatenated, and attention coefficients are calculated using the LeakyReLU activation function and attention parameters. Finally, after Softmax normalization, the attention weight of one node relative to the other node is obtained. This weight reflects the degree of influence of one node's features on the other node's features. Through multi-layer attention calculations, all attention weights obtained from previous calculations are aggregated to obtain the graph attention weights of each sensor node. ,in, It can be a visual camera, an inertial measurement unit (IMU), or a global navigation satellite system (GNSS).

[0154] The formula for calculating attention weights is:

[0155] ;

[0156] in, For nodes For nodes Attention weights For attention parameters, For nodes The set of neighboring nodes, For nodes One of the neighboring nodes, and They are nodes and nodes Feature data, The characteristic transformation matrix, For activation function, This is the softmax normalization function.

[0157] Then, based on the graph attention weights of each sensor node Combined with the sensor quality score at the current moment And the score representing the degree of anomaly of a single sensor, obtained by decomposing the anomaly score. Calculate the dynamic fusion weights. The expression for the dynamic fusion weights is:

[0158] ;

[0159] in, One type of sensor exist Dynamic fusion weights at any given time , and These are all balancing parameters used to adjust the contribution of each piece of information, taking values ​​of 0.2, 1.5, and 1 respectively. Modal embedding for visual cameras.

[0160] By dynamically fusing weights, the quality and abnormal state of the sensor itself can be reflected, as well as the correlation between sensors, thus achieving effective integration of multi-dimensional information.

[0161] Once the dynamic fusion weights are determined, the sensor weights can be adjusted to adapt the state estimation to environmental changes. Then, the optimized pose is used as a reference to achieve accurate map construction. The two support each other to form a complete SLAM function.

[0162] The step of optimizing the state estimation of SLAM based on dynamic fusion weights and building an environment map based on the optimized state estimation includes:

[0163] The calculated dynamic fusion weights of various sensors are applied to the corresponding error terms to establish a weighted total cost function;

[0164] The weighted total cost function is minimized by a nonlinear least squares algorithm, and the state parameters within a preset sliding window are iteratively optimized to obtain the optimal pose estimation results of the multi-source sensors at each time step within the preset sliding window.

[0165] Based on the optimal pose estimation results, the environmental features collected by multiple sensors are associated with a unified global coordinate system to obtain an environmental map.

[0166] The optimization of state estimation uses a preset sliding window as the basic optimization unit, which contains multi-source sensor data from several consecutive frames. During state estimation optimization, the calculated dynamic fusion weights for each type of sensor are first applied to their respective error terms to construct a weighted total cost function.

[0167] The expression for the constructed weighted total cost function is as follows:

[0168] ;

[0169] in, The total cost function value. For the preset sliding window, For the first Dynamic fusion weights of frame-based vision cameras For the first Dynamic fusion weights of frame inertial measurement units (IMUs) For the first Dynamic fusion weights of the Global Navigation Satellite System (GNSS). For the first The visual reprojection error term of a frame-based vision camera. For the first The IMU pre-integration error of the frame's inertial measurement unit (IMU). For the first GNSS position error of the Global Navigation Satellite System (GNSS) frame. These are prior constraints.

[0170] The cost function is then minimized using a nonlinear least squares method, iteratively optimizing the state parameters within a preset sliding window, including the pose of the visual camera or carrier device, and the zero-bias error of the inertial measurement unit (IMU), thereby outputting the optimal pose estimation results at each time point within the preset sliding window. During this optimization iteration process, when the data quality of a certain type of sensor deteriorates, its corresponding dynamic fusion weight decreases, reducing the proportion of its error term in the total cost function. This avoids low-quality data interfering with pose estimation, while the dynamic fusion weight of high-quality sensors increases, ensuring their dominant role in state estimation, thus achieving high-precision positioning in different environments.

[0171] An environment map is constructed based on the optimized state estimation to associate the environmental features observed by the sensors with a unified global coordinate system based on the precise pose.

[0172] To ensure that subsequent state estimation optimization can maintain accuracy in new environments, a meta-learning mechanism is introduced to continuously fine-tune the model parameters based on the feedback from state estimation optimization.

[0173] Specifically, in the process of optimizing the state estimation of SLAM based on dynamic fusion weights, the following also occurs:

[0174] Real-time calculation of performance monitoring metrics values; when the number of consecutive frames whose performance monitoring metrics values ​​meet the triggering conditions exceeds a preset threshold, meta-learning adaptation is triggered.

[0175] Based on the heterogeneous raw data from multiple sources of sensors after triggering meta-learning adaptation, the data quality prediction results, abnormal data detection results, and the state estimation error of the optimized state estimation, the corresponding model parameters are calculated by combining meta-learning adaptation to update feature extraction, data quality prediction, abnormal data detection, and dynamic fusion weights.

[0176] After the sliding window optimization is completed based on dynamic fusion weights, the absolute trajectory error, relative pose error, and anomaly detection F1 score are calculated in real time. If the number of frames in which the performance monitoring index value meets the trigger condition exceeds the preset threshold, it can be determined that a new environment has been entered or the sensor data mode has changed abruptly, triggering meta-learning adaptation to avoid the model continuously outputting low-precision results in abnormal states.

[0177] In the adaptation process, data from the new environment after triggering meta-learning adaptation is first collected, such as 20-30 frames of heterogeneous raw data from multi-source sensors, data quality prediction results, abnormal data detection results, and state estimation errors of optimized state estimation, which serve as samples for the inner loop of meta-learning.

[0178] Then, the inner loop update in the meta-learning adaptation is executed. The parameters calculated by the current sensor quality prediction network, cross-modal anomaly detection network, and dynamic fusion weights are used as initial values. The loss is calculated using new environment samples. Temporary parameters are obtained through 5 steps of gradient descent to quickly correct the prediction and detection bias of the model in the new environment.

[0179] Then, the outer loop update in the meta-learning adaptation is executed. During idle periods, such as the intervals between sensor data acquisitions, the adaptation results of multiple rounds of the inner loop are collected. After calculating the average loss, the meta-initialization parameters are updated with a smaller learning rate, solidifying the adaptation experience for the new environment and ensuring faster adaptation speed in similar scenarios in the future. At the same time, combined with the adaptive experience replay mechanism, historical high-value data with large prediction deviations and anomaly detection misjudgments are sampled according to the TD error priority and mixed with the new environment samples for model updates, improving adaptation efficiency.

[0180] Finally, the updated model parameters are applied to each stage of the model in real time, so that the constructed environmental map can be closer to the real environment.

[0181] Another aspect of this embodiment also provides a multi-source sensor fusion SLAM system based on deep learning, such as... Figure 2 As shown, it includes:

[0182] Feature processing module 1, connected to multi-source sensor 7, is used to receive heterogeneous raw data from multi-source sensor and extract features from the heterogeneous raw data;

[0183] Quality prediction module 2, connected to the feature processing module, is used to predict the data quality of multi-source sensors based on the extracted feature data through the sensor quality prediction network.

[0184] Anomaly detection module 3, connected to feature processing module, is used to detect anomalous data through cross-modal anomaly detection network based on the extracted feature data.

[0185] The adaptive fusion module 4 is connected to the quality prediction module and the anomaly detection module respectively, and is used to combine the data quality prediction results and the anomaly data detection results to calculate the dynamic fusion weight of the multi-source sensors.

[0186] SLAM output module 5, connected to the adaptive fusion module, is used to optimize the state estimation of SLAM according to the dynamic fusion weights and to build an environment map based on the optimized state estimation.

[0187] Also includes:

[0188] The meta-learning adaptation module 6 is connected to the feature processing module, the quality prediction module, the anomaly detection module, and the adaptive fusion module, respectively. It is used to calculate the corresponding model parameters based on the heterogeneous raw data of the multi-source sensors after triggering meta-learning adaptation, the data quality prediction results, the anomaly detection results, and the state estimation error of the optimized state estimation, combined with the meta-learning adaptation to update feature extraction, data quality prediction, anomaly detection, and dynamic fusion weights.

[0189] The feature processing module, quality prediction module, anomaly detection module, adaptive fusion module, SLAM output module, and meta-learning adaptation module are all data processing components with corresponding data processing and analysis capabilities, such as MCUs, microprocessors, and computers. They carry corresponding algorithms for feature extraction, sensor data quality prediction, anomaly detection, dynamic fusion weight calculation, state estimation optimization, environmental map construction, and model parameter optimization. Through mutual cooperation, they can achieve accurate positioning and mapping in dynamic and complex environments.

[0190] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Other variations and modifications are possible without departing from the technical solutions described in the claims.

Claims

1. A multi-source sensor fusion SLAM method based on deep learning, characterized in that, The method comprises the following steps: acquiring heterogeneous raw data of a multi-source sensor, and performing feature extraction on the heterogeneous raw data; based on the extracted feature data, predicting the data quality of the multi-source sensor through a sensor quality prediction network, wherein the sensor quality prediction network is acquired according to a double-branch time series modeling network taking a Transformer neural network as a first branch and a long short-term memory network as a second branch, and a multi-layer perception quality prediction head; according to the type of the sensor, adding corresponding modal embedding to the extracted feature data, and inputting the sensor quality prediction network; performing feature processing on the feature data after adding the modal embedding according to the double-branch time series modeling network, to acquire first-branch features and second-branch features; based on a gating mechanism, fusing the first-branch features and the second-branch features to acquire fused features; based on the acquired fused features, the multi-layer perception quality prediction head outputs data quality scores of various sensors; based on the extracted feature data, detecting abnormal data through a cross-modal anomaly detection network, wherein the cross-modal anomaly detection network is acquired according to a conditional variational autoencoder and a graph convolution network, and the extracted feature data is input into the cross-modal anomaly detection network; the conditional variational autoencoder maps each type of feature data through a latent space to reconstruct each type of feature data, acquires corresponding reconstructed features, and outputs a distribution anomaly score; the reconstruction error of each type of feature data and the corresponding reconstructed feature is calculated, and the corresponding distribution anomaly score is weighted and summed to acquire an anomaly score; abnormal data is screened according to the anomaly score, and the abnormal type of the screened abnormal data is classified through a graph convolution network to acquire the abnormal type corresponding to each abnormal data; combining the data quality prediction result and the abnormal data detection result, the dynamic fusion weight of the multi-source sensor is calculated, wherein a heterogeneous sensor graph is established by taking each type of sensor of the multi-source sensor as a node, the correlation between the sensors as an edge, and the cosine similarity between the corresponding feature data as the weight of the edge; based on the heterogeneous sensor graph, the graph attention weight of each sensor node is acquired through a heterogeneous graph attention network; the sensor quality score of each type of sensor at the current time is acquired according to the data quality prediction result, the anomaly score of the corresponding feature data of each type of sensor is acquired according to the abnormal data detection result, and the dynamic fusion weight of each type of sensor is acquired in combination with the corresponding graph attention weight; the state estimation of SLAM is optimized based on the dynamic fusion weight, and an environment map is established according to the optimized state estimation. 2.The deep learning based multi-source sensor fusion SLAM method of claim 1, wherein, The heterogeneous raw data of the multi-source sensor comprises visual data, inertial measurement data and position data. 3.The deep learning based multi-source sensor fusion SLAM method of claim 2, wherein, The feature extraction on the heterogeneous raw data comprises: performing preprocessing on the visual data, extracting a multi-dimensional feature map through a pre-trained image classification network, starting from the feature map with the highest dimension according to a feature pyramid network, layer by layer fusing the feature maps of adjacent dimensions, and acquiring visual features through the fused feature maps; time series segments of the inertial measurement data and the position data are respectively intercepted through a sliding window, and an inertial measurement input matrix and a position input matrix are respectively constructed according to the intercepted time series segments; Input the inertial measurement input matrix into a pre-trained one-dimensional convolutional neural network to obtain local features of the inertial measurement; Input the position input matrix into a pre-trained multi-layer perception to obtain position features. 4.The deep learning based multi-source sensor fusion SLAM method of claim 1, wherein, The state estimation of the SLAM is optimized based on the dynamic fusion weight, and an environment map is established according to the optimized state estimation, comprising: The dynamic fusion weight of each type of sensor calculated is respectively applied to the corresponding error term to establish a weighted total cost function; The weighted total cost function established is minimized by a nonlinear least squares algorithm to iteratively optimize the state parameters in the preset sliding window and obtain the optimal pose estimation result of the multi-source sensor at each time in the preset sliding window; Based on the optimal pose estimation result, the environment features collected by the multi-source sensor are associated to a unified global coordinate system to obtain an environment map. 5.The deep learning based multi-source sensor fusion SLAM method of claim 1, wherein, In the process of optimizing the state estimation of the SLAM based on the dynamic fusion weight, the following is also performed: Real-time performance monitoring index values are calculated, and when the number of frames in which the performance monitoring index values meet the triggering condition is continuously detected to exceed a preset threshold, meta-learning adaptation is triggered; Based on the heterogeneous raw data of the multi-source sensor after triggering the meta-learning adaptation, the data quality prediction result, the abnormal data detection result and the state estimation error of the optimized state estimation, the corresponding model parameters of the feature extraction, data quality prediction, abnormal data detection and dynamic fusion weight calculation are updated in combination with the meta-learning adaptation.

6. A deep learning based multi-source sensor fusion SLAM system for performing the method of any one of claims 1 to 5, characterized in that, Comprising: a feature processing module connected with the multi-source sensor, used for receiving the heterogeneous raw data of the multi-source sensor and performing feature extraction on the heterogeneous raw data; a quality prediction module connected with the feature processing module, used for predicting the data quality of the multi-source sensor through a sensor quality prediction network according to the extracted feature data; an abnormal detection module connected with the feature processing module, used for detecting abnormal data through a cross-modal anomaly detection network according to the extracted feature data; an adaptive fusion module connected with the quality prediction module and the abnormal detection module, respectively, used for calculating the dynamic fusion weight of the multi-source sensor in combination with the data quality prediction result and the abnormal data detection result; a SLAM output module connected with the adaptive fusion module, used for optimizing the state estimation of the SLAM according to the dynamic fusion weight, and establishing an environment map according to the optimized state estimation.

7. The deep learning based multi-source sensor fusion SLAM system of claim 6, wherein, Further comprising: a meta-learning adaptation module connected with the feature processing module, the quality prediction module, the abnormal detection module and the adaptive fusion module, respectively, used for updating the corresponding model parameters of the feature extraction, data quality prediction, abnormal data detection and dynamic fusion weight calculation in combination with the meta-learning adaptation according to the heterogeneous raw data of the multi-source sensor after triggering the meta-learning adaptation, the data quality prediction result, the abnormal data detection result and the state estimation error of the optimized state estimation.

Images