Multi-sensor fusion underwater SLAM system and method based on adaptive weight adjustment

Abstract

The invention discloses a multi-sensor fusion underwater SLAM system and method based on adaptive weight adjustment, and the system comprises a multi-modal sensor module, a data preprocessing and synchronization module, an adaptive weight adjustment module, a multi-sensor data fusion module, an SLAM calculation and map updating module, and an SLAMS result output module. And the adaptive weight adjustment module monitors the data quality of each sensor of the multi-modal sensor module in real time, and automatically adjusts the weight of each sensor according to the change of the underwater environment. The underwater positioning and navigation system can realize stable underwater positioning and navigation in an environment with insufficient light, complex obstacles or turbid water, and is especially suitable for tasks such as underwater exploration, detection and complex terrain modeling. Compared with the traditional SLAM technology, the method provided by the invention overcomes the limitation of a single data source by adjusting the weight of the sensor in real time, and significantly improves the adaptability and precision of the SLAM system in different underwater environments.

Description

Technical Field

[0001] The present invention relates to the field of underwater autonomous positioning and mapping (SLAM), in particular to an underwater SLAM optimization method based on multi-sensor fusion and adaptive weight adjustment. Background Art

[0002] With the increasing demand for underwater detection and monitoring tasks, it has become an important technical challenge for underwater robots and unmanned submersibles to achieve autonomous localization and mapping (SLAM) in complex environments. Traditional SLAM systems usually rely on a single sensor (such as sonar or visual sensors) to obtain environmental information, but the underwater environment is complex, and factors such as light changes, turbidity, and interference noise may cause data distortion or loss from a single sensor, thereby affecting the performance and reliability of the SLAM system. To this end, the use of multi-sensor fusion to enhance data stability and information richness has become the main method to improve the performance of underwater SLAM systems.

[0003] Existing multi-sensor fusion schemes usually combine sonar, IMU and visual sensors to provide more comprehensive environmental information. However, most of these methods adopt fixed weight allocation strategies, which are difficult to adapt to dynamic changes in the environment. For example, when light conditions are insufficient or the density of underwater plankton increases, the quality of visual data decreases, and the fixed weight scheme will reduce the performance of the overall SLAM system. In addition, the current technology lacks an adaptive weight adjustment mechanism in multi-sensor fusion, and cannot optimize the sensor weights according to real-time environmental changes, resulting in insufficient accuracy and robustness of the SLAM system in positioning and mapping in dynamic underwater environments. Summary of the invention

[0004] In order to solve the above-mentioned technical problems existing in the prior art, the present invention proposes an underwater SLAM system and method based on multi-sensor fusion and adaptive weight adjustment. The system integrates Ping360 sonar, IMU, binocular camera and depth sensor, and monitors the data quality of each sensor in real time through an adaptive algorithm, and dynamically adjusts the sensor weight according to environmental changes to ensure the accuracy and robustness of data fusion. The present invention can still achieve high-precision positioning and map construction in underwater scenes with limited vision or high noise. It is particularly suitable for underwater environments with complex structures and dense obstacles. It significantly improves the adaptability and accuracy of the SLAM system, provides high-precision positioning and mapping capabilities in complex underwater environments, and is widely used in underwater detection, structure detection, marine resource monitoring and other fields. Its specific technical solution is as follows: An underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment comprises a multi-modal sensor module, a data preprocessing and synchronization module, an adaptive weight adjustment module, a multi-sensor data fusion module, a SLAM calculation and map update module and a SLAMS result output module. The adaptive weight adjustment module monitors the data quality of each sensor of the multi-modal sensor module in real time and automatically adjusts the weight of each sensor according to changes in the underwater environment.

[0005] Furthermore, the multimodal sensor module integrates Ping360 sonar, IMU, binocular camera and depth sensor for real-time collection of underwater environment data; Ping360 sonar is used to detect the structural information of the surrounding environment and obtain the distance information of underwater obstacles through acoustic echo; IMU is used to collect the acceleration and angular velocity of the robot in real time, track the posture changes of the robot, and provide high-frequency posture data; binocular camera is used to obtain the robot's front view image and calculate the depth information through stereo vision algorithm to supplement the visual spatial understanding; depth sensor is used to detect the current depth of the robot and provide a height reference for the SLAM system. Each sensor can play its own unique detection advantages to adapt to the complex underwater light conditions, pressure changes at different depths and dense obstacle environments. Furthermore, the data preprocessing and synchronization module ensures the consistency of data collected by each sensor at the same time point through a hardware clock or software synchronization mechanism; performs denoising on the IMU data, and performs image enhancement or color correction on the visual data.

[0006] Further, the processing of the adaptive weight adjustment module includes a data quality detection unit, a deep learning optimization unit and a weight dynamic adjustment unit; The data quality monitoring unit is used to monitor the data quality of Ping360 sonar, IMU, binocular camera and depth sensor in real time; The deep learning optimization unit adopts an improved convolutional neural network structure to extract high-level semantic features of multimodal data layer by layer: The weight dynamic unit adjusts the weight of each sensor dynamically according to the detected data quality and environmental conditions.

[0007] This module optimizes multimodal data fusion by monitoring the quality of each sensor data and automatically adjusting the weight of each sensor according to different environmental conditions.

[0008] Furthermore, when the underwater light is insufficient, the data quality of the visual sensor decreases, and the weight dynamic unit automatically reduces the weight of the binocular camera while increasing the weight of the Ping360 sonar and IMU.

[0009] Furthermore, the weight dynamic unit monitors the output data quality of each sensor in real time and automatically adjusts the weights of different sensors.

[0010] Furthermore, the multimodal data fusion module adopts factor graph optimization, and the feature vector after dynamic weight adjustment is input into the factor graph optimization module. The factor graph model is used to treat the data of each sensor as different factor nodes, and a consistent pose estimation is generated through an optimization algorithm, including: Factor graph modeling: The data of Ping360 sonar, IMU, binocular camera and depth sensor are represented as factor nodes, and the observation value of each node is used as a constraint to build a complete factor graph structure; Factor graph optimization: An optimization algorithm is used to find the optimal solution in the factor graph to estimate the global pose of the robot.

[0011] Furthermore, the position information output by the multimodal data fusion module is transmitted to the SLAM calculation and map update module, and the SLAM calculation and map update module updates the underwater map of the system in real time according to the optimized position and environmental characteristics, including Map update process: Use the latest posture information of the SLAM system to integrate the newly observed environmental information into the existing map.

[0012] Environmental mapping: By integrating the data from sonar and visual sensors, a high-precision underwater environment map is formed. The SLAM calculation and map update module continuously receives the positioning results of the SLAM system and performs incremental updates on the existing map.

[0013] An underwater SLAM method based on multi-sensor fusion and adaptive weight adjustment comprises the following steps: Step 1: Collect underwater environment data from the Ping360 sonar, IMU, binocular camera, and depth sensor of the multimodal sensor module; Step 2: Time synchronize and preprocess the collected data to ensure the consistency of different data sources in time and space; Step 3: Monitor the quality of the synchronized data and dynamically adjust the weight distribution according to the real-time performance of each sensor data; Step 4: Fuse the data from each sensor through the factor graph optimization algorithm and deep learning model to generate the results of underwater positioning and map construction; Step 5: Output SLAM positioning and map construction data to update the position and path information of the underwater SLAM system in complex environments in real time.

[0014] Compared with the traditional underwater SLAM method, the present invention has the following advantages: Strong adaptability: Through the adaptive weight adjustment mechanism, the system can automatically optimize the sensor weights according to the dynamic changes of the underwater environment to achieve more robust positioning.

[0015] Multimodal data fusion: Combining data from sonar, IMU, vision, and depth sensors improves the positioning accuracy and map construction quality of the SLAM system in complex environments.

[0016] High robustness: The system can work effectively in complex environments such as insufficient light and noise interference, and is particularly suitable for scenarios such as underwater detection and environmental monitoring.

[0017] The present invention provides a SLAM solution with high precision and strong robustness in complex underwater environments through real-time data acquisition, adaptive weight adjustment and factor graph optimization fusion of multimodal sensors, providing reliable support for the autonomous navigation of underwater robots, autonomous submersibles and other equipment in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the architecture diagram of the underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment; Figure 2 is a schematic diagram of the multimodal sensor layout; Figure 3 is the adaptive weight adjustment flow chart; Figure 4 It is a schematic diagram of multi-sensor data fusion and factor graph optimization; Figure 5 It is a flowchart of SLAM calculation and map update; Figure 6 It is a schematic diagram of deep learning feature extraction and weight optimization. DETAILED DESCRIPTION

[0019] In order to better understand the purpose, structure and function of the present invention, the invention is further described in conjunction with the accompanying drawings.

[0020] like Figure 1 As shown, the underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment of the present invention includes a multimodal sensor module, a data preprocessing and synchronization module, an adaptive weight adjustment module, a multi-sensor data fusion module, a SLAM calculation and map update module and a SLAMS result output module.

[0021] like Figure 2As shown, the multimodal sensor module integrates Ping360 sonar, IMU, binocular camera and depth sensor for real-time collection of underwater environment data; Ping360 sonar is used to detect the structural information of the surrounding environment and obtain the distance information of underwater obstacles through acoustic echoes; IMU (inertial measurement unit) is used to collect the acceleration and angular velocity of the robot in real time, track the posture changes of the robot, and provide high-frequency posture data; the binocular camera is used to obtain the robot's front view image and calculate the depth information through the stereo vision algorithm to supplement the visual spatial understanding; the depth sensor is used to detect the current depth of the robot and provide a height reference for the SLAM system.

[0022] The data of each group of sensors is transmitted to the data preprocessing and synchronization module for synchronization and denoising to ensure time consistency and data cleanliness in the subsequent processing. Time synchronization: Through hardware clock or software synchronization mechanism, ensure that the data collected by each sensor is consistent at the same time point. Data preprocessing: De-noise the IMU data and perform image enhancement or color correction on the visual data to improve data quality and reliability.

[0023] After preprocessing, the data enters the adaptive weight adjustment module, and the processing of the adaptive weight adjustment module includes a data quality detection unit, a deep learning optimization unit and a weight dynamic adjustment unit.

[0024] The data quality monitoring unit is used to monitor the data quality of Ping360 sonar, IMU, binocular camera and depth sensor in real time The deep learning optimization unit adopts an improved convolutional neural network (CNN) structure, including multiple convolutional layers, pooling layers and fully connected layers, to extract high-level semantic features of multimodal data layer by layer: Convolutional layer: responsible for extracting local features such as edges and textures from the input sensor data, and scanning and processing the image and sonar data through convolution kernels of specific sizes.

[0025] Pooling layer: Reduces the resolution of the feature map through downsampling operations, reduces the amount of calculation, and extracts more representative features.

[0026] Fully connected layer: Integrate the feature maps after convolution and pooling to generate feature vectors, which represent the global features of multimodal sensor data.

[0027] The deep learning feature extraction module ultimately outputs a multimodal feature vector, providing high-level semantic information for subsequent dynamic adjustment of weights.

[0028] The weight dynamic unit adjusts the weight of each sensor dynamically according to the detected data quality and environmental conditions.

[0029] Weight adjustment: When the underwater light is insufficient, the data quality of the visual sensor decreases, and the weight of the binocular camera is automatically reduced, while the weight of the Ping360 sonar and IMU is increased to ensure the stability of the SLAM system in adverse environments.

[0030] Real-time response: It can monitor the output data quality of each sensor in real time and automatically adjust the weights of different sensors to adapt to dynamic changes in the environment.

[0031] The adaptive weight adjustment mechanism enables the SLAM system to always maintain high positioning and mapping accuracy in a changing underwater environment.

[0032] The multimodal data fusion module adopts factor graph optimization, and the feature vector after dynamic weight adjustment is input into the factor graph optimization module. This module uses the factor graph model to treat the data of each sensor as different factor nodes and generates consistent pose estimation through the optimization algorithm, including: Factor graph modeling: The data of Ping360 sonar, IMU, binocular camera and depth sensor are represented as factor nodes, and the observation value of each node is used as a constraint to build a complete factor graph structure; Factor graph optimization algorithm: The optimal solution is found in the factor graph through optimization algorithms (such as nonlinear optimization) to estimate the global position of the robot; the factor graph optimization algorithm can jointly consider the constraint information of each sensor to ensure the consistency and robustness of data fusion. The results of factor graph optimization directly affect the positioning accuracy and consistency of the SLAM system, and provide reliable data support for the map update module.

[0033] like Figure 6 As shown, the posture information output by the multimodal data fusion module is transmitted to the SLAM calculation and map update module. This module updates the underwater map of the system in real time according to the optimized posture and environmental characteristics: Map update process: Use the latest posture information of the SLAM system to integrate the newly observed environmental information into the existing map.

[0034] Environmental mapping: By integrating data from sonar and visual sensors, a high-precision underwater environment map is formed. The SLAM calculation and map update module continuously receives the positioning results of the SLAM system and incrementally updates the existing map to reflect the latest changes in the environment. The SLAM calculation and map update module enables the SLAM system to dynamically map the underwater environment.

[0035] The SLAMS result output module is used to output positioning and map data to underwater robots and unmanned submersibles, providing reliable support for the autonomous navigation of underwater robots, autonomous submersibles and other equipment in complex environments.

[0036] The present invention successfully realizes high-precision positioning and real-time map updating of underwater SLAM system through multimodal sensor fusion, deep learning feature extraction, adaptive weight adjustment and factor graph optimization. The system performs well in complex underwater environments such as insufficient light and turbid water, ensuring efficient navigation and stable operation of the robot. Through dynamic weight adjustment mechanism and factor graph optimization algorithm, the system can provide reliable positioning and map information in a changeable underwater environment.

Claims

1. An underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment, comprising a multi-modal sensor module, a data preprocessing and synchronization module, an adaptive weight adjustment module, a multi-sensor data fusion module, a SLAM calculation and map update module and a SLAMS result output module, characterized in that: The adaptive weight adjustment module monitors the data quality of each sensor of the multimodal sensor module in real time, and automatically adjusts the weight of each sensor according to changes in the underwater environment.

2. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 1, characterized in that: The multimodal sensor module integrates Ping360 sonar, IMU, binocular camera and depth sensor to collect underwater environment data in real time; The Ping360 sonar is used to detect the structural information of the surrounding environment and obtain the distance information of underwater obstacles through acoustic echoes; the IMU is used to collect the robot's acceleration and angular velocity in real time, track the robot's posture changes, and provide high-frequency posture data; the binocular camera is used to obtain the robot's front view image and calculate the depth information through a stereo vision algorithm to supplement the visual spatial understanding; the depth sensor is used to detect the robot's current depth and provide a height reference for the SLAM system.

3. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 1, characterized in that: The data preprocessing and synchronization module ensures the consistency of data collected by each sensor at the same time point through a hardware clock or software synchronization mechanism; performs denoising on the IMU data, and performs image enhancement or color correction on the visual data.

4. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 1, characterized in that: The processing of the adaptive weight adjustment module includes a data quality detection unit, a deep learning optimization unit and a weight dynamic adjustment unit; The data quality monitoring unit is used to monitor the data quality of Ping360 sonar, IMU, binocular camera and depth sensor in real time; The deep learning optimization unit adopts an improved convolutional neural network structure to extract high-level semantic features of multimodal data layer by layer: The weight dynamic unit adjusts the weight of each sensor dynamically according to the detected data quality and environmental conditions.

5. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 4, characterized in that: When the light underwater is insufficient, the data quality of the visual sensor decreases, and the weight dynamic unit automatically reduces the weight of the stereo camera and increases the weight of the Ping360 sonar and IMU.

6. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 4, characterized in that: The weight dynamic unit monitors the output data quality of each sensor in real time and automatically adjusts the weights of different sensors.

7. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 1, characterized in that: The multimodal data fusion module adopts factor graph optimization. The feature vector after dynamic weight adjustment is input into the factor graph optimization module. The factor graph model is used to treat the data of each sensor as different factor nodes, and a consistent pose estimation is generated through the optimization algorithm, including: Factor graph modeling: The data of Ping360 sonar, IMU, binocular camera and depth sensor are represented as factor nodes, and the observation value of each node is used as a constraint to build a complete factor graph structure; Factor Graph Optimization: An optimization algorithm is used to find the optimal solution in the factor graph to estimate the global pose of the robot.

8. The underwater SLAM system based on multi-sensor fusion and adaptive weight adjustment according to claim 1, characterized in that: The position information output by the multimodal data fusion module is transmitted to the SLAM calculation and map update module, and the SLAM calculation and map update module updates the underwater map of the system in real time according to the optimized position and environmental characteristics, including Map update process: using the latest posture information of the SLAM system, the newly observed environment information is integrated into the existing map; Environmental mapping: By integrating the data from sonar and visual sensors, a high-precision underwater environment map is formed. The SLAM calculation and map update module continuously receives the positioning results of the SLAM system and performs incremental updates on the existing map.

9. An underwater SLAM method based on multi-sensor fusion and adaptive weight adjustment, characterized in that The following steps are involved: Step 1: Collect underwater environment data from the Ping360 sonar, IMU, binocular camera, and depth sensor of the multimodal sensor module; Step 2: Time synchronize and preprocess the collected data to ensure the consistency of different data sources in time and space; Step 3: Monitor the quality of the synchronized data and dynamically adjust the weight distribution based on the real-time performance of each sensor data; Step 4: Fuse the data from each sensor through the factor graph optimization algorithm and deep learning model to generate the results of underwater positioning and map construction; Step 5: Output SLAM positioning and map construction data to update the position and path information of the underwater SLAM system in complex environments in real time.

Images