Map building and path planning method based on multi-mode SLAM

Abstract

The invention provides a map building and path planning method based on multi-modal SLAM, a robot system collects data through a multi-modal sensor, a data fusion module fuses the collected data by using an AI model, a map building module carries out real-time map building according to the fused data, and a path planning module carries out path planning according to the fused data. The global path planning module performs global path planning through an AI model, the actual path planning module outputs a local path by using the AI model, and the robot system dynamically adjusts and outputs an execution path according to the local path in combination with the latest data of the multi-modal sensor. The motion execution module controls the robot to move to a task position according to the output execution path, monitors the execution condition of the path in real time and feeds back the execution condition to the positioning and posture estimation module and the path planning module, and the positioning and posture estimation module updates the current positioning and adjusts the posture of the robot according to feedback data. And the path planning module finely adjusts the global path and the local path according to feedback data to improve the path planning quality.

Description

Technical Field

[0001] The invention belongs to the technical field of robots, and in particular relates to a multi-modal SLAM map construction and path planning method. Background Art

[0002] The positioning and navigation of mobile robots mainly rely on Simultaneous Localization and Mapping (SLAM) technology. By equipping with specific sensors, autonomous unmanned mobile devices can use the SLAM system to build a global map of unknown environments and achieve accurate positioning and navigation. SLAM technology has important applications in unmanned driving, drones and other fields. In SLAM, finding the correct representation of the corresponding environment and estimating the robot's position trajectory based on it plays an important role in both positioning and mapping.

[0003] With the development of science and technology, various SLAM technologies have gradually matured. Path planning is the core step of robot automatic walking to prevent the robot from hitting obstacles, but the existing robot system has poor path planning quality, which makes the robot unable to quickly reach the task location. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal SLAM map construction and path planning method to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for building a map and planning a path based on multimodal SLAM, wherein the robot system is started and the multimodal sensor is initialized, the system comprises a data fusion module, a map building module, a positioning and posture estimation module and a path planning module, the path planning module comprises a global path planning module, an actual path planning module, a motion execution module and an adaptive learning module;

[0006] S1: The robot system collects data through the multimodal sensor and processes the data to remove noise to obtain multimodal data;

[0007] S2: The data fusion module uses an AI model to fuse the multimodal data, extract and aggregate feature information of multiple sensors, and obtain fused data;

[0008] S3: The map construction module uses the fused data to perform real-time map construction, update and maintain the environment map to obtain a high-precision and high-detail environment map;

[0009] S4: The positioning and posture estimation module uses a multimodal SLAM algorithm to determine the current positioning and posture of the robot;

[0010] S5: The robot system determines the starting point and the end point of the walking path, as well as the goal and restriction conditions of the walking path planning, and the global path planning module performs global path planning through the AI ​​model based on the updated global map;

[0011] S6: The actual path planning module outputs a feasible local path through the AI ​​model according to the environment map constructed in real time by S3, recommends the robot's actions in the current environment, and then dynamically adjusts the output execution path according to the latest data of the multimodal sensor to adapt to real-time environmental changes;

[0012] S7: The motion execution module controls the robot chassis to move according to the path output by S6, monitors the execution of the path in real time, and feeds back to the positioning and posture estimation module and the path planning module. At the same time, the feedback data is stored in the database. The positioning and posture estimation module updates the current positioning and adjusts the robot posture according to the feedback data, and the path planning module fine-tunes the global path and the local path according to the feedback data.

[0013] S8: The adaptive learning module adaptively learns and optimizes the AI ​​model according to the S7 feedback data, and adjusts system parameters to improve the execution efficiency of future tasks.

[0014] Preferably, the multimodal sensor includes a camera, a laser radar and an inertial measurement unit, and the initialization of the multimodal sensor includes external parameters and internal parameters of the camera, the laser radar and the inertial measurement unit.

[0015] Preferably, the data processing step of S1 includes filtering out noise, removing outliers and synchronizing multimodal data timestamps.

[0016] Preferably, the AI ​​model is a Transformer-based AI model.

[0017] Preferably, the multimodal SLAM algorithm combines real-time data from vision, lidar and IMU for positioning and attitude estimation.

[0018] Preferably, the map construction module includes constructing a three-dimensional point cloud map and a topological map.

[0019] Preferably, the database is a cloud database, and the robot system is connected to the cloud database via a 5G network.

[0020] Preferably, the restriction conditions include energy consumption control conditions, task time-consuming conditions and safety control conditions.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The robot system of the present invention collects data through multimodal sensors. The data fusion module uses an AI model to fuse the collected data, extracts and aggregates the characteristic information of multiple sensors, and the map construction module constructs a real-time map based on the fused data, and updates the old map to improve the accuracy of the environmental map. The positioning and posture estimation module uses a multimodal SLAM algorithm to determine the current positioning and posture of the robot, so that the robot system can determine the starting point of the walking path. After the robot system confirms the end point, the global path planning module performs global path planning through the AI ​​model based on the updated map to determine the global path of the robot to reach the task location. The actual path planning module calculates the global path and the map according to the global path and the map. The construction module constructs the environment map in real time and uses the AI ​​model to output a feasible short-distance local path, which reduces computing power while providing high computing speed. The robot system dynamically adjusts the output execution path based on the local path and the latest data from the multimodal sensor. The motion execution module controls the robot to move to the task location according to the output execution path. At the same time, it monitors the execution of the path in real time and feeds back to the positioning and attitude estimation module and the path planning module. The positioning and attitude estimation module updates the current positioning and adjusts the robot's attitude based on the feedback data. The path planning module fine-tunes the global path and local path based on the feedback data to improve the quality of path planning and ensure that the robot reaches the task location quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Embodiment 1:

[0026] like Figure 1As shown, the present invention provides a method for building a map and planning a path based on multimodal SLAM. The robot system is started and the multimodal sensor is initialized. The system includes a data fusion module, a map building module, a positioning and posture estimation module and a path planning module. The path planning module includes a global path planning module, an actual path planning module, a motion execution module and an adaptive learning module. S1: The robot system collects data through a multimodal sensor and performs data processing to remove noise to obtain multimodal data. S2: The data fusion module uses an AI model to fuse the multimodal data, extracts and aggregates feature information of multiple sensors, and obtains fused data. S3: The map building module uses the fused data to build a real-time map, update and maintain the environmental map, so as to obtain a high-precision and high-detail environmental map. S4: The positioning and posture estimation module uses a multimodal SLAM algorithm to determine the current position and posture of the robot. S5: The robot system determines the starting point and end point of the walking path, and The goal and restriction of the walking path planning, the global path planning module performs global path planning through the AI ​​model based on the updated global map; S6: the actual path planning module uses the AI ​​model to output a feasible local path based on the environmental map constructed in real time by S3, recommends the robot's actions in the current environment, and then dynamically adjusts the output execution path according to the latest data of the multimodal sensor to adapt to real-time environmental changes; S7: the motion execution module controls the robot chassis to move according to the path output by S6, monitors the execution of the path in real time, and feeds back to the positioning and attitude estimation module and the path planning module. At the same time, the feedback data is stored in the database. The positioning and attitude estimation module updates the current positioning and adjusts the robot's attitude according to the feedback data, and the path planning module fine-tunes the global path and local path according to the feedback data; S8: the adaptive learning module adaptively learns and optimizes the AI ​​model according to the feedback data of S7, and adjusts the system parameters to improve the execution efficiency of future tasks. The multimodal sensor includes a camera, a lidar, and an inertial measurement unit. The initialization of the multimodal sensor includes the external and internal parameters of the camera, the lidar, and the inertial measurement unit. The data processing steps of S1 include filtering out noise, removing outliers, and synchronizing multimodal data timestamps. The AI ​​model is a Transformer-based AI model. The multimodal SLAM algorithm combines real-time data from vision, lidar, and IMU for positioning and posture estimation. The map construction module includes building a three-dimensional point cloud map and a topological map. The database is a cloud database, and the robot system communicates with the cloud database through a 5G network. The constraints include energy consumption control conditions, task time conditions, and safety control conditions.

[0027] Through the above technical scheme, the robot system of the present invention collects data through multimodal sensors, the data fusion module uses the AI ​​model to fuse the collected data, extracts and aggregates the characteristic information of multiple sensors, the map construction module constructs the map in real time according to the fused data, and updates the old map to improve the accuracy of the environmental map, the positioning and posture estimation module uses the multimodal SLAM algorithm to determine the current positioning and posture of the robot, so that the robot system can determine the starting point of the walking path, and after the robot system confirms the end point, the global path planning module performs global path planning through the AI ​​model based on the updated map to determine the global path of the robot to reach the task location, and the actual path planning module calculates the global path of the robot according to the global path. The path and map construction module constructs an environmental map in real time, and uses the AI ​​model to output a feasible short-distance local path, reducing computing power while providing high computing speed. The robot system dynamically adjusts the output execution path based on the local path and the latest data from the multimodal sensor. The motion execution module controls the robot to move to the task location according to the output execution path. At the same time, it monitors the execution of the path in real time and feeds back to the positioning and attitude estimation module and the path planning module. The positioning and attitude estimation module updates the current positioning and adjusts the robot's attitude according to the feedback data. The path planning module fine-tunes the global path and local path according to the feedback data to improve the quality of path planning and ensure that the robot reaches the task location quickly.

[0028] Embodiment 2:

[0029] like Figure 1 As shown, the present invention provides a method for building a map and planning a path based on multimodal SLAM. The method first starts the robot system and initializes the multimodal sensor. The system includes a data fusion module, a map building module, a positioning and posture estimation module, and a path planning module. The path planning module includes a global path planning module, an actual path planning module, a motion execution module, and an adaptive learning module.

[0030] In step S1, the robot system collects environmental data through multimodal sensors. These sensors include high-resolution cameras, ultrasonic sensors, 2D lidar, and 6-axis inertial measurement units (IMUs). Lidar scans the surrounding environment and obtains point cloud data; high-resolution cameras capture images; ultrasonic sensors emit ultrasonic waves and receive echoes; and IMUs provide acceleration and angular velocity data.

[0031] The system preprocesses the collected raw data, including applying Gaussian filtering to remove image noise, using the RANSAC algorithm to remove outliers in the lidar data, and performing zero-bias correction on the IMU data.

[0032] In step S2, the data fusion module uses a deep neural network to fuse the preprocessed multimodal data. The network uses an encoder-decoder structure to process images, point clouds, and IMU data separately, and then fuses the extracted features in the latent space. The network outputs a fused feature map that contains rich environmental geometry and semantic information.

[0033] In step S3, the map construction module uses the fused data to build a real-time map. The system adopts a SLAM framework based on graph optimization, takes the fused features as observations, and estimates the robot trajectory and the environment map simultaneously through factor graph optimization. The map representation adopts a hybrid structure, including sparse feature points, semi-dense depth maps, and semantic labels. The system continuously updates and refines the map through loop detection and global optimization.

[0034] In step S4, the localization and attitude estimation module uses a multimodal SLAM algorithm to determine the current position and attitude of the robot. The algorithm is based on a particle filter framework, matching the fused features with the map and combining the IMU data for prediction. The system maintains multiple hypotheses, continuously updates the particle weights through importance sampling and resampling, and finally outputs the optimal estimate.

[0035] In step S5, the robot system determines the starting and ending points of the walking path, as well as the goals and constraints of path planning. The global path planning module plans based on the updated global map through an AI algorithm enhanced by a neural network. The network learns the optimal heuristic function in a complex environment to improve search efficiency.

[0036] In step S6, the actual path planning module outputs a local path based on the local map built in real time using a policy network trained by reinforcement learning. The network considers the current state, target location, and local environment to generate a sequence of speed and steering instructions. The system dynamically adjusts the output path based on the latest sensor data to adapt to environmental changes.

[0037] In step S7, the motion execution module controls the robot chassis to move along the planned path. The system uses the model predictive control (MPC) algorithm to generate the optimal control input considering the robot dynamics model and path constraints. At the same time, the module monitors the path execution in real time, calculates the deviation between the actual trajectory and the planned path, and sends feedback data to the positioning and posture estimation module and the path planning module. The system updates the positioning results based on the feedback, adjusts the robot posture, and fine-tunes the global and local paths.

[0038] In step S8, the adaptive learning module optimizes the system AI model based on execution feedback data. The module uses a meta-learning algorithm to automatically adjust key parameters in SLAM and path planning, such as feature extraction threshold, loop detection threshold, path smoothing factor, etc., by analyzing historical task execution data. The system continuously improves its adaptability and efficiency in different environments through continuous learning.

[0039] Embodiment three:

[0040] like Figure 1 As shown, the present invention adopts three main sensors: a high-resolution camera, a 3D lidar and an inertial measurement unit (IMU). Other sensors, such as an ultrasonic radar, etc., can be added according to actual needs.

[0041] The camera's internal parameters include focal length, principal point coordinates, and distortion coefficients. The system uses a calibration plate to calibrate the camera and obtain an accurate internal parameter matrix. The external parameters describe the position and posture of the camera relative to the robot chassis coordinate system and are obtained through the hand-eye calibration method.

[0042] The laser radar is a multi-line 3D laser radar installed on the top of the robot. The internal parameters of the laser radar include the angle offset and distance offset of each laser beam. The external parameters describe the position and posture of the laser radar relative to the robot chassis coordinate system. The system scans the laser radar with a specific calibration object and then uses a nonlinear optimization algorithm to estimate the internal and external parameters.

[0043] The IMU is a 6-axis sensor that includes a 3-axis accelerometer and a 3-axis gyroscope. The IMU is installed close to the center of mass of the robot. The internal parameters of the IMU include the scale factor, zero bias and non-orthogonality parameters of each axis. The external parameters describe the position and posture of the IMU relative to the robot chassis coordinate system. The system calibrates the IMU internal parameters through static sampling and dynamic rotation tests. The time synchronization of the IMU and other sensors is achieved through hardware trigger signals.

[0044] During the system initialization phase, the pre-calibrated sensor parameters are first loaded. For the camera, the system checks whether the image resolution and frame rate meet the expectations and applies lens distortion correction. For the lidar, the system verifies the point cloud density and scanning frequency and transforms the point cloud to the robot coordinate system based on the external parameters. For the IMU, the system performs a fast calibration procedure, estimates the current zero bias, and applies temperature compensation.

[0045] The system also performs time synchronization and spatial registration between multiple sensors. Time synchronization is achieved through hardware triggering and software timestamp interpolation to ensure the time consistency of different sensor data. Spatial registration refines the relative position between each sensor through a joint optimization algorithm to improve fusion accuracy.

[0046] The initialization process also includes a sensor health check. The system analyzes the output of each sensor to detect outliers or data loss. For cameras, the system checks whether the image clarity and exposure are normal. For lidar, the system verifies whether there are large blind spots in the point cloud. For IMU, the system checks whether the acceleration and angular velocity readings are within reasonable ranges.

[0047] After initialization, the system enters normal working state and starts to continuously collect and process multimodal sensor data. During operation, the system continuously monitors the sensor status and dynamically adjusts parameters to adapt to environmental changes, such as camera exposure adjustment caused by light changes, or IMU zero bias correction caused by temperature changes.

[0048] Embodiment 4:

[0049] like Figure 1 As shown, based on the second embodiment, the data processing process in step S1 is further improved, including filtering out noise, removing outliers and synchronizing multimodal data timestamps.

[0050] For camera data, the system first applies a Gaussian filter to remove image noise. The Gaussian filter kernel size is 5x5, and the standard deviation σ=1.5. After filtering, the system uses the Canny edge detection algorithm to extract image edges with a threshold range of 50-150. To remove motion blur, the system applies a deconvolution algorithm, assuming that the motion blur kernel is linear and the length is dynamically adjusted based on the angular velocity measured by the IMU.

[0051] For the lidar data, the system first applies a distance filter to remove points that are beyond the valid range (0.1 meters to 50 meters). Then a statistical outlier removal filter (SOR) is used to identify and remove noise points. The SOR algorithm calculates the average distance of each point to its K nearest neighbors (K = 50), and marks points whose distance exceeds the global average distance plus or minus 1.5 times the standard deviation as outliers. The system also applies voxel filtering to downsample the point cloud with a voxel size of 0.05 meters to reduce the computational burden of subsequent processing.

[0052] IMU data processing first applies low-pass filtering to remove high-frequency noise. The system uses a Kalman filter to fuse accelerometer and gyroscope data to estimate attitude. To handle IMU zero bias drift, the system performs zero bias correction in a stationary state and dynamically adjusts the bias estimate.

[0053] Time synchronization is a key step in multimodal data processing. The system uses a timestamp-based software synchronization method. Each sensor data frame carries a high-precision timestamp. The system aligns all data to a unified time reference. For sensors with lower frame rates (such as cameras and lidar), the system uses linear interpolation to estimate the sensor state at intermediate moments.

[0054] Specifically, the system maintains a sliding time window. In each processing cycle, the system collects all sensor data that fall within the current window. For each sensor, if there are multiple data frames in the window, the frame with the timestamp closest to the center of the window is selected; if there are no data frames, the data of adjacent timestamps are interpolated and estimated.

[0055] When processing camera images, the system takes into account the time offset caused by exposure time. Assuming that the image timestamp corresponds to the end of exposure, the system adjusts the timestamp forward by half the exposure time to more accurately reflect the actual moment corresponding to the image content.

[0056] For LiDAR data, the system uses IMU data to dedistort the point cloud, taking into account the possibility that the robot may move during the laser scanning process. Specifically, the system uses the precise timestamp of each laser point and the IMU attitude data at the corresponding moment to calibrate all points to the coordinate system at the end of the scan.

[0057] After completing time synchronization, the system organizes the multimodal data into a unified data structure, including aligned images, point clouds, IMU measurements and their corresponding timestamps. This synchronized multimodal data packet serves as the input for subsequent SLAM and path planning algorithms.

[0058] The system also implements a data integrity check mechanism. If a sensor continues to lose data within a specified time window, the system will trigger a warning to indicate a possible sensor failure. At the same time, the system has data compensation capabilities. If a sensor is missing data for a short period of time, it can use other sensor data and historical information to make estimates to ensure the continuity of the algorithm.

[0059] Embodiment five:

[0060] like Figure 1 As shown, this embodiment elaborates on the application of the Transformer-based AI model in the multimodal SLAM map construction and path planning method. The AI ​​model is the core of the entire system and is responsible for the fusion, feature extraction and decision support of multimodal data.

[0061] The Transformer model has powerful feature extraction and long-distance dependency modeling capabilities, which makes it perform well in computer vision and robotics. In this method, an improved Transformer architecture is used to process heterogeneous data streams from visual cameras, lidars, and IMUs.

[0062] The input of the model includes visual image sequences, laser point cloud data, and IMU measurements. First, each modality data is subjected to preliminary feature extraction through an independent encoder network. The visual data is subjected to a convolutional neural network to extract spatial features, the point cloud data is subjected to a network such as PointNet++ to extract geometric features, and the IMU data is subjected to a recurrent neural network to capture temporal information.

[0063] The encoded features are divided into a series of "tokens" as input to the Transformer. The core of the model is the multi-head self-attention mechanism, which allows each token to interact with all other tokens, thereby capturing the complex relationships between different modalities, different time steps, and different spatial positions.

[0064] To handle multimodal data, the model introduces modality-specific attention masks and position encodings. This enables the model to distinguish information from different sources while maintaining spatiotemporal consistency. In addition, the cross-modal attention layer is specifically used to fuse information from different sensors, such as aligning visual features with corresponding point cloud features.

[0065] The decoder part of Transformer is customized according to the task requirements. For map building tasks, the decoder generates semantic segmentation maps and depth maps to assist in 3D reconstruction. For path planning tasks, the decoder outputs a probability map representing the environment's traversability and the optimal path.

[0066] The model training adopts a multi-task learning strategy to simultaneously optimize multiple objective functions related to map construction, localization, and path planning. It is pre-trained using large-scale simulation environments and real-world datasets, and then fine-tuned on a specific robot platform.

[0067] In the inference phase, the model processes real-time sensor data in a streaming manner. The attention mechanism allows the model to dynamically focus on the most relevant information for the current task, such as focusing on obstacles ahead during navigation. The model output is directly used to update the environment map and guide path planning decisions.

[0068] To improve real-time performance, the model uses techniques such as attention pruning and knowledge distillation. At the same time, model compression methods such as tensor core decomposition are used to reduce the model size so that it can run efficiently on the limited computing resources of the robot.

[0069] By applying Transformer to multimodal SLAM tasks, this method achieves deep fusion of sensor data and intelligent decision-making. Compared with traditional methods, the Transformer-based AI model shows stronger environmental understanding and adaptability, can effectively handle complex and dynamic real-world scenarios, and brings significant performance improvements to the robot navigation system.

[0070] Embodiment six:

[0071] like Figure 1 As shown, this embodiment describes in detail how the multimodal SLAM algorithm combines real-time data from vision, lidar, and IMU for accurate positioning and attitude estimation. By fusing information from multiple sensors, the algorithm overcomes the limitations of a single sensor and achieves stable and reliable positioning in complex environments.

[0072] The algorithm first preprocesses and synchronizes the data of each sensor. The visual data undergoes distortion correction and feature extraction to extract stable feature points such as ORB. The lidar point cloud is downsampled through voxel filtering to extract plane and edge features. The IMU data is corrected for zero bias and scale factor to remove the influence of gravity.

[0073] The visual SLAM part uses a keyframe-based sparse direct method. The algorithm maintains a local map containing recent keyframes and observed 3D feature points. When a new frame arrives, the pose is estimated by minimizing the photometric error. Feature matching and triangulation are performed simultaneously to dynamically update the map. Local bundle adjustment is performed periodically to optimize the camera pose and feature point positions.

[0074] The laser SLAM part uses an improved ICP (Iterative Closest Point) algorithm. Each scan is registered with the local map point cloud to estimate the laser radar pose transformation. The matching constraints of plane and edge features are introduced to improve the registration accuracy and efficiency. Loop detection and pose graph optimization are performed regularly to eliminate accumulated errors.

[0075] IMU provides high-frequency incremental estimates of pose through pre-integration theory. The IMU pre-integration results serve as constraints between vision and laser odometers, and are also used to predict the pose at the next moment, assist feature tracking and scan matching.

[0076] Multi-sensor fusion adopts a loosely coupled extended Kalman filter (EKF) framework. The state vector includes position, attitude, velocity and sensor parameters. The prediction step uses the IMU motion model for state propagation. The update step processes visual and laser observations separately to correct the predicted state. An adaptive mechanism is introduced to dynamically adjust the observation covariance and balance the reliability of different sensors.

[0077] The algorithm also includes robustness enhancement mechanisms. RANSAC and other methods are used to remove outliers in vision and laser matching. Fault detection logic is designed to automatically downgrade to dual-modal or single-modal SLAM when a single sensor fails. Dynamic objects are explicitly modeled and tracked to reduce their interference with positioning.

[0078] To handle large-scale environments, the algorithm adopts a hierarchical structure. It maintains an accurate local map for real-time positioning while maintaining a coarse-grained global map. The two-level maps achieve consistency through the pose graph. When performing loop detection, candidate regions are first searched in the global map, and then accurately matched at the local scale.

[0079] In terms of computational efficiency, the algorithm fully utilizes the parallelism of modern processors. Vision, laser, and IMU processing are performed in independent threads. A sliding window filter is used to limit the optimization scale. GPU acceleration is used for computationally intensive operations such as feature extraction and ICP registration.

[0080] This multimodal SLAM algorithm achieves centimeter-level positioning accuracy and reliable attitude estimation by deeply integrating vision, lidar and IMU data. It can adapt to challenging environments such as lighting changes and dynamic scenes, and provides stable posture input for subsequent map construction and path planning, which is the basis for the high-performance operation of the entire system.

[0081] Embodiment seven:

[0082] like Figure 1 As shown in the figure, the present invention further optimizes how the map construction module constructs a 3D point cloud map and a topological map at the same time, providing a comprehensive environment representation for robot navigation. This dual map representation combines geometric accuracy and structural abstraction, and can support multi-scale environment understanding and path planning.

[0083] The construction of 3D point cloud maps is based on multi-sensor fusion data. First, the original point cloud obtained by LiDAR scanning is downsampled and filtered to remove noise points and outliers. Then, the point cloud is colored using visual images to add semantic information. IMU data is used to compensate for motion distortion during the scanning process.

[0084] Point cloud registration uses an improved NDT (normal distribution transform) algorithm. Adjacent frame point clouds are aligned by minimizing the difference in probability distribution. Feature matching constraints based on curvature and normal consistency are introduced to improve registration accuracy. For large-scale environments, a hierarchical registration strategy is used, first performing coarse registration and then fine alignment.

[0085] To improve Figure 1 To ensure consistency, the system periodically performs loop detection and global optimization. Loop detection combines geometric and visual information, using 3D descriptors such as FPFH and scene recognition methods based on deep learning. After detecting a loop, the historical trajectory and map structure are adjusted through pose graph optimization.

[0086] Point cloud maps use an octree structure for efficient storage and indexing. In addition to storing occupancy probability, each voxel also contains attributes such as color and normal. The map handles dynamic changes through a probabilistic update mechanism, and can gradually "forget" disappeared objects. For large-scale environments, a block loading technique is used to retain only high-resolution representations of active areas.

[0087] Topological map construction begins with semantic segmentation of point cloud maps. A fully convolutional network is used to classify point clouds into semantic categories such as ground, wall, door, etc. Based on the segmentation results, key structural elements such as roads, intersections, rooms, etc. are extracted. These elements form nodes of the topological map.

[0088] The connection relationship between nodes is determined by analyzing the passability. A dilation algorithm is used to generate a free space map and identify connected areas. The connection between rooms is inferred by considering the location of doors. For outdoor environments, the road network is extracted as the main connection structure. Each edge is annotated with attributes such as distance and difficulty of passage.

[0089] The topology map is dynamically updated to adapt to changes in the environment. When new passages or obstacles are detected, edges are added or removed accordingly. Node properties such as occupancy status are also updated over time. The map also records the robot's visit frequency for heuristic search in subsequent path planning.

[0090] The two map representations are kept consistent through a unified coordinate system. Topological nodes have precise geometric correspondence in the point cloud map. This association allows seamless switching between different levels of abstraction, such as using the topological map for global navigation and referring to precise point cloud data for local path generation.

[0091] The map building process is incremental and online. New sensor data is continuously integrated into the existing map. A sliding window technique is used to limit the computational effort of the real-time optimization. A background process periodically performs global optimization, ensuring long-term consistency.

[0092] This dual map representation method overcomes the limitations of a single map type. The point cloud map provides highly accurate geometric information, supporting fine manipulation and obstacle avoidance. The topological map captures the high-level structure of the environment, which is conducive to fast path planning and semantic navigation. The collaborative use of the two maps significantly improves the robot's navigation ability and efficiency in complex environments.

[0093] Embodiment eight:

[0094] like Figure 1As shown, the robot system of the present invention adopts a distributed cloud-edge collaborative architecture to migrate data storage and some computing tasks to the cloud. The system establishes a high-speed, low-latency connection with the cloud database through the 5G network. The cloud database adopts a distributed NoSQL database (such as Apache Cassandra), which is highly scalable and fault-tolerant. The database stores the robot's historical operation data, environmental map information, AI model parameters, etc.

[0095] In actual applications, the robot first obtains the initial environment map and AI model parameters from the cloud database through the 5G network. During the task execution, the robot edge device is responsible for real-time data processing and decision-making, and uploads important data to the cloud. For example, newly constructed map fragments, detected special events, AI model gradient updates and other information will be periodically sent to the cloud database.

[0096] The cloud server uses more powerful computing resources to perform time-consuming offline tasks, such as global map fusion and optimization, large-scale data analysis, deep training of AI models, etc. These calculation results are regularly synchronized back to the robot to update and optimize its decision-making system.

[0097] The high bandwidth of 5G networks enables robots to quickly upload high-resolution sensor data, such as 3D point clouds or high-definition images. The low latency feature ensures that cloud processing results can be fed back to the robot in a timely manner, supporting near-real-time remote control and decision-making assistance.

[0098] In addition, the network slicing technology of 5G networks provides customized service quality assurance for robot systems. For example, high-priority slices are allocated for real-time control communications, and high-bandwidth slices are allocated for large-capacity data transmission, thereby meeting the needs of different application scenarios.

[0099] The cloud database not only serves a single robot, but also supports the coordination of multi-robot systems. Environmental information collected by different robots can be integrated in the cloud to build a more comprehensive shared map. Task allocation and path coordination between robots can also be globally optimized in the cloud.

[0100] To protect data security and privacy, the system uses end-to-end encryption technology to encrypt all transmitted data. The cloud database implements strict access control and data isolation policies to ensure that each robot can only access authorized data. At the same time, sensitive data (such as precise positioning information) can be selectively stored and processed only locally without uploading to the cloud.

[0101] By utilizing cloud databases and 5G networks, this system significantly improves data storage capacity, computing power, and information sharing efficiency, providing strong support for the robot's intelligent decision-making and collaborative operations.

[0102] Embodiment nine:

[0103] like Figure 1 As shown, the path planning module of the present invention comprehensively considers multiple constraints, including energy consumption control conditions, task time conditions and safety control conditions, to generate the optimal motion trajectory. These constraints are integrated into the optimization goal of path planning through a series of mathematical models and algorithms.

[0104] The energy consumption control condition mainly considers the energy consumption of the robot during the task execution. The system establishes an energy consumption model based on robot dynamics, which takes into account factors such as robot mass, speed, acceleration, and ground friction coefficient. For example, for a wheeled robot, the energy consumption model can be expressed as:

[0105] E = ∫(m·a+μ·m·g·v+k·v 2 )dt

[0106] Among them, E is the total energy consumption, m is the mass of the robot, a is the acceleration, μ is the friction coefficient, g is the acceleration due to gravity, v is the speed, and k is the air resistance coefficient. The path planning algorithm will try to minimize this energy consumption function while satisfying other constraints.

[0107] The task time requirement requires the robot to complete the task within the specified time. The system adopts the time optimal control theory and takes the task completion time as one of the optimization goals. In the specific implementation, the dynamic programming algorithm is used to calculate the shortest time path from the starting point to the end point and use it as a benchmark. Then, in the actual planning, a certain time margin is allowed (10%-20% of the benchmark time), and other factors such as energy consumption and safety are balanced within this range.

[0108] Safety control conditions are key constraints to ensure the safety of robot motion. The system defines a series of safety indicators, including the minimum distance to obstacles, the maximum allowed speed and acceleration, the turning radius limit, etc. These indicators are formulated as nonlinear constraints:

[0109] d(x,y)≥d min (Minimum obstacle distance constraint)

[0110] v≤v max (Maximum speed constraint)

[0111] |a|≤a max (Maximum acceleration constraint)

[0112] |κ|≤κ max (Maximum curvature constraint)

[0113] Among them, d(x,y) represents the distance from the point (x,y) on the path to the nearest obstacle, v and a represent the velocity and acceleration respectively, and κ represents the path curvature.

[0114] During path planning, these constraints are integrated into a multi-objective optimization problem:

[0115] minimize w1·E+w2·T+w3·S

[0116] subject to security constraints

[0117] Where E represents energy consumption, T represents task completion time, S represents the inverse of safety margin, and w1, w2, and w3 are weight coefficients. This optimization problem is solved by the improved Rapidly-exploring Random Tree (RRT*) algorithm. In the process of expanding the tree, the algorithm continuously evaluates the energy consumption, time, and safety of the newly generated paths, and only retains the paths that meet all constraints and have a smaller objective function value.

[0118] In order to adapt to different task requirements and environmental conditions, the system allows dynamic adjustment of the weights of various constraints. For example, the energy consumption weight can be reduced when the battery is sufficient, the time weight can be increased in emergency tasks, and the safety weight can be increased in complex environments. This dynamic adjustment mechanism is implemented through a reinforcement learning algorithm, and the system automatically selects the most suitable weight configuration based on the historical task execution results and the current status.

[0119] By comprehensively considering these constraints, the system can generate motion trajectories that are both energy-efficient and efficient as well as safe and reliable, significantly improving the robot's ability to perform tasks in complex environments.

[0120] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0121] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for building a multimodal SLAM map and planning a path, characterized in that: The robot system is started and the multimodal sensor is initialized. The system includes a data fusion module, a map construction module, a positioning and posture estimation module, and a path planning module. The path planning module includes a global path planning module, an actual path planning module, a motion execution module, and an adaptive learning module. S1: The robot system collects data through the multimodal sensor and processes the data to remove noise to obtain multimodal data; S2: The data fusion module uses an AI model to fuse the multimodal data, extract and aggregate feature information of multiple sensors, and obtain fused data; S3: The map construction module uses the fused data to perform real-time map construction, update and maintain the environment map to obtain a high-precision and high-detail environment map; S4: The positioning and posture estimation module uses a multimodal SLAM algorithm to determine the current positioning and posture of the robot; S5: The robot system determines the starting point and the end point of the walking path, as well as the goal and restriction conditions of the walking path planning. The global path planning is based on the updated global map and the global path planning is performed through the AI ​​model; S6: The actual path planning module outputs a feasible local path through the AI ​​model according to the environment map constructed in real time by S3, recommends the robot's actions in the current environment, and then dynamically adjusts the output execution path according to the latest data of the multimodal sensor to adapt to real-time environmental changes; S7: The motion execution module controls the robot chassis to move according to the path output by S6, monitors the execution of the path in real time, and feeds back to the positioning and posture estimation module and the path planning module. At the same time, the feedback data is stored in the database. The positioning and posture estimation module updates the current positioning and adjusts the robot posture according to the feedback data, and the path planning module fine-tunes the global path and the local path according to the feedback data. S8: The adaptive learning module adaptively learns and optimizes the AI ​​model according to the S7 feedback data, and adjusts system parameters to improve the execution efficiency of future tasks.

2. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The multimodal sensor includes a camera, a laser radar and an inertial measurement unit, and the initialization of the multimodal sensor includes external parameters and internal parameters of the camera, the laser radar and the inertial measurement unit.

3. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The data processing steps of S1 include filtering out noise, removing outliers and synchronizing multimodal data timestamps.

4. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The AI ​​model is a Transformer-based AI model.

5. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The multimodal SLAM algorithm combines real-time data from vision, lidar and IMU for positioning and attitude estimation.

6. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The map construction module includes constructing a three-dimensional point cloud map and a topological map.

7. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The database is a cloud database, and the robot system is connected to the cloud database via a 5G network.

8. A method for building a multimodal SLAM map and planning a path according to claim 1, characterized in that: The restriction conditions include energy consumption control conditions, task time-consuming conditions and safety control conditions.

Images