Semantic SLAM (Simultaneous Localization and Mapping) optimization method for fusing panoramic vision and laser radar
Through the semantic SLAM method of integrating panoramic vision and lidar, combined with deep learning models and optimization technology, the blind spots and error problems of environmental perception in the existing technology are solved, high-precision map construction and environmental adaptation are realized, and robot autonomous navigation capabilities are improved .
Patent Information
- Application Number
- CN202411086228.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-15
AI Technical Summary
The existing SLAM technology has the influence of light changes, dynamic obstacles and other factors in complex environments, resulting in blind spots and errors in image data, and the measurement error of lidar on the surface of highly reflective or absorbed materials, and lacks semantic understanding, resulting in limited map information and poor adaptability. It is difficult to update in real time, affecting the autonomous operation of the robot.
Using the semantic SLAM method of fusion of panoramic vision and lidar, through the steps of data acquisition, preprocessing, feature extraction and fusion, semantic analysis and map construction, map optimization and positioning correction, and learning and update, combined with deep learning models and optimization techniques, generate High-precision semantic map to adapt to environmental changes.
It significantly improves the positioning accuracy and map construction accuracy of the SLAM system, enhances the system's ability to adapt to environmental changes, provides high-precision navigation and decision-making support, and improves the robot's autonomous operation ability in complex environments.
Smart Images

Figure CN118962716A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of environmental perception technology, and in particular to a semantic SLAM optimization method integrating panoramic vision and laser radar. Background Art
[0002] With the rapid development of robotics and autonomous driving technologies, SLAM (Simultaneous Localization and Mapping) technology has been widely used in the fields of robot navigation, environmental perception, and autonomous driving. SLAM technology collects environmental data through sensors, builds environmental maps in real time, and realizes its own positioning in the environment, becoming a key technology for robots to achieve autonomous navigation and environmental interaction. Panoramic vision and LiDAR are two main means of environmental perception, with a wide field of view and high-precision distance measurement capabilities, respectively, which can provide rich environmental information for the SLAM system.
[0003] In the existing technology, panoramic vision SLAM and lidar SLAM have the following technical problems in practical applications. First, although panoramic vision SLAM can provide a wide field of view, it is easily affected by factors such as lighting changes and dynamic obstacles in complex environments, resulting in blind spots and errors in image data; although lidar SLAM has high-precision distance measurement capabilities, it is easy to produce measurement errors on the surface of highly reflective or absorbing materials. The use of panoramic vision or lidar alone cannot completely overcome these problems. Secondly, most of the existing SLAM technologies rely on traditional geometric feature extraction and matching methods, and lack semantic understanding of objects and structures in the environment, resulting in limited generated map information, which cannot meet the high-precision navigation and decision-making needs of robots in complex environments. Finally, the existing SLAM system has poor adaptability in dynamic environments, and it is difficult to update and optimize maps in real time, resulting in positioning errors and map failures in the system when the environment changes, affecting the robot's autonomous operation capabilities. Summary of the invention
[0004] Based on the above objectives, the present invention provides a semantic SLAM optimization method integrating panoramic vision and laser radar.
[0005] The semantic SLAM optimization method of panoramic vision and lidar fusion includes the following steps: S1, data collection: use panoramic camera and lidar to collect environmental data simultaneously, where the panoramic camera provides panoramic image data, and the lidar provides distance data and shape information data. S2, data preprocessing: denoising the panoramic image data collected by the panoramic camera, and filtering the distance data and shape information data collected by the lidar; S3, feature extraction and fusion: extract visual feature points from the preprocessed panoramic image data, extract spatial structure features from the distance data and shape information data, fuse the visual feature points with the spatial structure features, and construct a preliminary fusion feature map; S4, semantic parsing and map construction: Apply deep learning models to semantically annotate the fused feature map, identify objects and structures in the environment, and build a semantic map; S5, map optimization and positioning correction: Optimize the semantic map based on semantic information, adjust the erroneous marks and position deviations in the map, and perform dynamic positioning correction by combining the optimized semantic map with the real-time collected data to improve positioning accuracy; S6, learning update: continuously update the semantic map according to new environmental data, and optimize the deep learning model to adapt to new scenarios and environmental changes; S7, Output and Application: Generate an optimized semantic SLAM map and use this map for subsequent navigation and robot decision support, providing high-precision navigation functions and decision analysis.
[0006] Optionally, the S1 specifically includes: S11, hardware configuration: Use a camera that provides 360-degree panoramic images, select one or more laser radars, and configure them according to the required accuracy and range. The laser radar obtains the distance and shape information of the object by emitting laser pulses and measuring their reflection time, generating a high-precision 3D point cloud map; S12, Installation and Positioning: Mounting the panoramic camera and LiDAR on a backpack scanning device helps reduce data inconsistency caused by the physical separation between the sensors and ensures that the relative position between the panoramic camera and LiDAR is fixed and known; S13, time synchronization: Time synchronization of the panoramic camera and the lidar to ensure that the data collected by the two are aligned in time. This is achieved by using an accurate clock source such as a GPS clock to ensure that the data accurately matches; S14, data collection: operate the panoramic camera and lidar for data collection at the same time. The panoramic camera continuously captures panoramic images of the environment, and the lidar scans the surrounding environment and collects distance and shape information data. For mobile platforms (such as autonomous vehicles or mobile robots), ensure that the two sensors can adapt to different moving speeds and environmental conditions.
[0007] Optionally, the S2 specifically includes: S21, image data denoising: using a bilateral filtering algorithm to denoise the panoramic image data collected by the panoramic camera; S22, distance data filtering: using Gaussian filtering algorithm to filter the distance data; S23, shape information data filtering: using the moving least squares method to filter the shape information data.
[0008] Optionally, the S3 specifically includes: S31, visual feature extraction: using SIFT (Scale Invariant Feature Transform) algorithm to extract visual feature points from pre-processed panoramic image data; S32, spatial structure feature extraction: extract spatial structure features from distance data and shape information data using principal component analysis; S33, feature fusion: use ICP (iterative closest point) algorithm to fuse and align feature points; S34, fusion feature map construction: constructing the fused feature points into a fusion feature map.
[0009] Optionally, the S4 specifically includes: S41, data preparation: semantic labeling of the fused feature map, including roads, buildings, pedestrians and vehicles, to construct a labeled fused feature map dataset, and perform standardization and normalization processing; S42, model selection: select 3D U-Net model as the deep learning model; S43, model training: the fused feature map dataset is divided into a training set and a validation set, and the training set data is used to train the 3D U-Net model; S44, Model Application: Semantic annotation on the fused feature map to identify key objects and structures in the environment; S45, build detailed semantic map: integrate the semantically annotated fused feature map into a global semantic map.
[0010] Optionally, the step S5 optimizes the semantic map based on the semantic information to adjust the erroneous marks and position deviations in the map, specifically including: S51, incorrect label identification: based on known semantic information, verify the labels on the current semantic map; S52, position deviation detection: detecting position deviation by comparing the position of the same landmark in consecutive frames or in environmental data collected by the panoramic camera and the lidar, where if the same landmark shows spatial differences in consecutive frames or when compared with data collected by the panoramic camera or the lidar, there is position deviation; S53, Graph Optimization Technology: Use graph optimization technology to adjust and optimize the structure of the map.
[0011] Optionally, in S5, the optimized semantic map is combined with the real-time collected data to perform dynamic positioning correction to improve positioning accuracy, specifically including: S54, real-time data and map alignment: Use the real-time collected data for preliminary alignment. The initial position and posture can be provided by sensors such as GPS and IMU. The real-time collected data is finely aligned with the optimized semantic map using the ICP algorithm. S55, dynamic positioning correction: Fuse the real-time collected data with the optimized semantic map to generate the latest feature point set, and use the extended Kalman filter to perform dynamic positioning correction on the fused feature points.
[0012] Optionally, the S6 specifically includes: S61, data update: collect new environmental data in real time through panoramic cameras and lidar, and input the data into the deep learning model for update; S62, Model Optimization: Retrain and optimize the deep learning model based on new environmental data, and use the gradient descent algorithm to update the model parameters; S63, semantic map update: Update the semantic map in real time based on new data and optimized models to ensure the accuracy and timeliness of map information.
[0013] Optionally, the S7 specifically includes: S71, semantic SLAM map generation: Combine real-time data and optimized models to generate optimized semantic SLAM maps S72, Navigation and Decision Support: Using semantic SLAM maps for robot navigation and decision support, providing path planning, obstacle avoidance and target positioning functions; S73, user interface: display the optimized semantic SLAM map through a preset user interface, allowing users to interact and control, and provide map visualization and data export.
[0014] Beneficial effects of the present invention: The present invention significantly improves the positioning accuracy of the SLAM system and the accuracy of map construction by integrating panoramic vision and laser radar data, combining deep learning models and optimization technology. The panoramic image data provided by the panoramic camera and the distance data and shape information data provided by the laser radar complement each other and can operate stably in complex and changing environments, reducing the data blind spots and errors caused by a single sensor and improving the overall reliability.
[0015] The present invention uses a deep learning model to perform semantic analysis and annotation on the fused feature map, which can intelligently identify objects and structures in the environment and generate a high-precision semantic map. At the same time, the semantic map is dynamically optimized and positioned in combination with real-time data. The system can continuously learn and update, adapt to new environmental changes, and achieve efficient semantic SLAM optimization. This not only improves the accuracy of map construction, but also enhances the system's ability to adapt to environmental changes.
[0016] The optimized semantic SLAM map of the present invention provides accurate basic data for robot navigation and decision-making, can generate high-precision navigation paths in real time, avoid obstacles and locate targets, and significantly improve the robot's autonomous operation ability in complex environments. In addition, by displaying the optimized semantic SLAM map through the user interface, users can interact and control it, obtain intuitive visualization and detailed map data, provide effective support for robot decision-making, and improve practicality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention; Figure 2 Schematic diagram of S4 process according to an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.
[0020] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiments", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).
[0021] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.
[0022] like Figure 1-Figure 2 As shown in FIG. 1 , the semantic SLAM optimization method of panoramic vision and laser radar fusion includes the following steps: S1, data collection: use panoramic camera and lidar to collect environmental data simultaneously, where the panoramic camera provides panoramic image data, and the lidar provides distance data and shape information data. S2, data preprocessing: denoising the panoramic image data collected by the panoramic camera, and filtering the distance data and shape information data collected by the lidar; S3, feature extraction and fusion: extract visual feature points from the preprocessed panoramic image data, extract spatial structure features from the distance data and shape information data, fuse the visual feature points with the spatial structure features, and construct a preliminary fusion feature map; S4, semantic parsing and map construction: Apply deep learning models to semantically annotate the fused feature map, identify objects and structures in the environment, and build a semantic map; S5, map optimization and positioning correction: Optimize the semantic map based on semantic information, adjust the erroneous marks and position deviations in the map, and perform dynamic positioning correction by combining the optimized semantic map with the real-time collected data to improve positioning accuracy; S6, learning update: the semantic map is continuously updated according to new environmental data, and the deep learning model is optimized to adapt to new scenarios and environmental changes. Through the continuous learning mechanism, the system remains efficient and accurate in a constantly changing environment; S7, Output and Application: Generate an optimized semantic SLAM map and use this map for subsequent navigation and robot decision support, providing high-precision navigation functions and decision analysis.
[0023] S1 specifically includes: S11, hardware configuration: Use a camera that provides 360-degree panoramic images. This type of camera usually uses multiple lenses to capture images in different directions and stitches these images into a panoramic image through software algorithms. Ensure that the camera has sufficient resolution and dynamic range to adapt to various lighting conditions. Select one or more lidars and configure them according to the required accuracy and range. Lidars obtain the distance and shape information of objects by emitting laser pulses and measuring their reflection time to generate high-precision 3D point cloud images. S12, Installation and Positioning: Installing the panoramic camera and LiDAR on a backpack scanning device helps reduce data inconsistency caused by the physical separation between the sensors and ensures that the relative position between the panoramic camera and LiDAR is fixed and known, which is crucial for subsequent data fusion; S13, time synchronization: Time synchronization of panoramic cameras and lidar to ensure that the data collected by the two are aligned in time. This is usually achieved by using an accurate clock source such as a GPS clock to ensure that the data matches accurately; S14, data collection: operate the panoramic camera and lidar for data collection at the same time. The panoramic camera continuously captures panoramic images of the environment, and the lidar scans the surrounding environment and collects distance and shape information data. For mobile platforms (such as autonomous vehicles or mobile robots), ensure that the two sensors can adapt to different moving speeds and environmental conditions.
[0024] S2 specifically includes: S21, image data denoising: The panoramic image data collected by the panoramic camera is denoised using a bilateral filtering algorithm. Bilateral filtering is a nonlinear denoising technique that combines spatial proximity and pixel differences to effectively remove noise while maintaining edge clarity. The bilateral filtering algorithm is expressed as: For each pixel in the image , pixel value after bilateral filtering , calculated as: ; in, is the pixel value in the neighborhood, is a Gaussian function of intensity differences, measuring the difference between pixel values, is the Gaussian function of spatial distance, which is used to measure the spatial distance between pixels. is the normalization factor that ensures the filter response is equal to 1 and is expressed as: ; S22, distance data filtering: Gaussian filtering algorithm is used to filter the distance data. The Gaussian filtering algorithm can smooth the random noise in the distance measurement and retain important distance features, which is expressed as: For each point in the point cloud , its filtered position Calculated as: ; in, Indicate point The set of neighbor points in the point cloud, is the weight, which is determined by the Gaussian function and is calculated as: , Yes With its neighbors The Euclidean distance between is the standard deviation of the Gaussian kernel, controlling the degree of smoothing; S23, shape information data filtering: The shape information data is filtered using the moving least squares method. The moving least squares method improves the data quality by fitting the local surface, thereby better representing the geometric shape of the object, which is expressed as: For each point , first determine a local neighborhood , and then fit a smooth surface on this neighborhood , and recalculate The position is calculated as: ; in, is the new position after filtering, Yes and neighbors The distance between is the weight, usually also a Gaussian function based on distance, Indicate point The set of neighbor points in the point cloud, is through the fitting point and its neighboring points The resulting local surface.
[0025] S3 specifically includes: S31, visual feature extraction: Use SIFT (Scale Invariant Feature Transform) algorithm to extract visual feature points from the preprocessed panoramic image data. The specific steps are as follows: (1) Scale space extreme value detection: Detect key points at different scales using Gaussian difference pyramid, including: Scale space construction: construct a series of Gaussian blurred images , different Representing different scales, expressed as: ; in, is the Gaussian kernel, is the input image, * represents the convolution operation; Gaussian Difference Pyramid: Calculate the difference image between adjacent scales , expressed as: ; in, is a constant, usually taken as ; (2) Key point positioning: Accurately locate the extreme points in the Gaussian difference pyramid space and remove low-contrast points and edge response points, including: Sub-pixel precision key point positioning: Use Taylor expansion to interpolate at the extreme points to accurately locate the position and scale of the key points, expressed as: ; in, is the exact location of the key point, is the Hessian matrix, is the gradient of the Difference of Gaussian pyramid function; (3) Direction assignment: According to the gradient direction histogram of the area around the key point, one or more directions are assigned to each key point, including: Gradient calculation: ; ; Direction histogram: Calculate the gradient direction histogram in the neighborhood of the key point and select the peak direction as the main direction of the key point; Key point description: In the local area where the key point direction and scale remain unchanged, the gradient histogram is calculated as a descriptor, including: Local gradient histogram: Divide the key point neighborhood into For each sub-region, the gradient histogram in 8 directions is calculated, and a 128-dimensional feature descriptor is obtained in total; S32, spatial structure feature extraction: The principal component analysis method is used to extract spatial structure features from the distance data and shape information data. The steps are as follows: (1) Covariance matrix calculation: For point and the point set in its neighborhood, calculate the covariance matrix , calculated as: ; in, is a point in the neighborhood, is the average position of the points in the neighborhood, is the number of points in the neighborhood; (2) Eigenvalues and eigenvectors: Solving The eigenvalues and eigenvectors of the data, the eigenvectors determine the main direction of the data, and the eigenvalues give the variation in each direction; S33, feature fusion: Use the ICP (Iterative Closest Point) algorithm to fuse and align feature points. The ICP algorithm effectively combines data from different sensors and optimizes the registration by minimizing the distance between points. The specific steps are as follows: (1) Initial matching: for each visual feature point , find the nearest spatial structure feature point , expressed as: ; (2) Calculate rigid body transformation: Use the least squares method to calculate the set of visual feature points Register to the set of spatial structure feature points The rigid body transformation (rotation and pan ), expressed as: ; (3) Update matching: Update the position of the visual feature points based on the calculated rigid body transformation, perform nearest neighbor matching again, and iterate until convergence; S34, fusion feature map construction: construct the fused feature points into a fusion feature map, and use a graph optimization method to further optimize the fusion feature map to ensure the consistency and accuracy of the map; Through this series of detailed algorithm applications and steps, features can be effectively extracted and fused from panoramic camera and lidar data to build an accurate and practical fused feature map, providing strong data support for subsequent navigation and location recognition.
[0026] S4 specifically includes: S41, data preparation: semantic labeling of the fused feature map, including roads, buildings, pedestrians and vehicles, to construct a labeled fused feature map dataset, and perform standardization and normalization processing; S42, model selection: select 3D U-Net model as the deep learning model; The 3D U-Net model architecture includes: (1) Encoder part: a series of 3D convolutional layers and pooling layers to gradually extract high-level features of the input data. ReLU activation function is used after each convolutional layer, and the pooling layer reduces the spatial dimension. (2) Decoder: A series of upsampling layers and convolutional layers to gradually restore the spatial resolution of the feature map. The upsampling layer is connected to the feature map of the corresponding layer of the encoder (skip connection) to help restore high-resolution details. (3) Output layer: Use a 1x1x1 convolutional layer to map the feature map to the semantic label of each category; S43, model training: the fused feature map data set is divided into a training set and a validation set, and the training set data is used to train the 3D U-Net model. The training process specifically includes: (1) Input data: The training set data is input into the 3D U-Net model. The input data includes the spatial structure of the point cloud and the corresponding image features. (2) Feature extraction: The encoder part extracts high-level features of the fused feature map, and the decoder part gradually restores the spatial resolution of the feature map; (3) Semantic classification: The output layer classifies each voxel and generates a semantic label; (4) Loss function: The cross entropy loss function is used to evaluate the model performance. The cross entropy loss function is expressed as: ; in, is the sample size, is the number of categories, It is a sample The real label, is the probability predicted by the model; (5) Optimization algorithm: Optimize model parameters through back propagation and gradient descent. Common optimizers include Adam or SGD. (6) Training process: Iteratively train the model until the performance on the validation set reaches a satisfactory level; S44, Model Application: Semantic annotation on the fused feature map to identify key objects and structures in the environment, including: Model inference: Input the fused feature map into the trained 3D U-Net model for inference, and the model outputs the semantic label of each voxel; Semantic annotation result processing: Process the output semantic labels to generate a fused feature map of semantic annotation; Post-processing step: Post-process the semantic labels output by the model, such as using conditional random fields (CRF) to smooth and optimize the semantic annotation results, and identify key objects and structures in the environment, such as roads, buildings, pedestrians, vehicles, etc., according to predefined category labels; S45, build detailed semantic map: integrate the semantically annotated fused feature map into a global semantic map.
[0027] S5 optimizes the semantic map based on semantic information and adjusts the erroneous marks and position deviations in the map, including: S51, Error Marking Identification: Based on known semantic information, the markings on the current semantic map are verified. For example, if a marked area is identified as a "building", but the surrounding environment characteristics (such as other sensor data or time series data) indicate that the area is an open space, mark this area as a potential error; S52, position deviation detection: detecting position deviation by comparing the position of the same landmark in consecutive frames or in environmental data collected by the panoramic camera and the lidar, where if the same landmark shows spatial differences in consecutive frames or when compared with data collected by the panoramic camera or the lidar, there is position deviation; S53, graph optimization technology: Use graph optimization technology to adjust and optimize the structure of the map to ensure the global consistency of the map. The graph optimization formula is expressed as: Error function: ; in, is a set of constraints, representing all observations, It is a slave node To Node The measured value of Based on the node status and Calculated predicted measurements, is the Mahalanobis distance, where is the covariance matrix of the measurement noise.
[0028] S5 combines the optimized semantic map with the real-time collected data to perform dynamic positioning correction to improve positioning accuracy, including: S54, real-time data and map alignment: Use the real-time collected data for preliminary alignment. The initial position and posture can be provided by sensors such as GPS and IMU. The ICP (iterative closest point) algorithm is used to finely align the real-time collected data with the optimized semantic map, which is expressed as: ; in, Represents the feature points in real-time data, Represents the feature points in the map. and They represent the rotation matrix and translation vector respectively; S55, dynamic positioning correction: The real-time collected data is fused with the optimized semantic map to generate the latest feature point set, and the extended Kalman filter is used to perform dynamic positioning correction on the fused feature points, which is expressed as: ; in, represents the current state estimate, represents the Kalman gain, represents the observed value, represents the observation model; Through these steps, the semantic map can be effectively optimized based on semantic information, the erroneous marks and position deviations in the map can be adjusted, and the optimized semantic map can be combined with the real-time collected data for dynamic positioning correction, thereby improving the positioning accuracy.
[0029] S6 specifically includes: S61, data update: collect new environmental data in real time through panoramic cameras and lidar, and input the data into the deep learning model for update; S62, model optimization: Based on the new environmental data, the deep learning model is retrained and optimized, and the model parameters are updated using the gradient descent algorithm. The formula for gradient descent is: ; in, are model parameters, is the learning rate, is the loss function; S63, semantic map update: Update the semantic map in real time based on new data and optimized models to ensure the accuracy and timeliness of map information.
[0030] S7 specifically includes: S71, semantic SLAM map generation: Combine real-time data and optimized models to generate optimized semantic SLAM maps S72, Navigation and Decision Support: Use semantic SLAM maps for robot navigation and decision support, providing path planning, obstacle avoidance and target positioning functions; S73, user interface: display the optimized semantic SLAM map through a preset user interface, allowing users to interact and control, and provide map visualization and data export.
[0031] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.
[0032] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A semantic SLAM optimization method based on panoramic vision and lidar fusion, characterized in that: The following steps are involved: S1, data collection: use panoramic camera and lidar to collect environmental data at the same time. The panoramic camera provides panoramic image data, and the lidar provides distance data and shape information data. S2, data preprocessing: denoising the panoramic image data collected by the panoramic camera, and filtering the distance data and shape information data collected by the lidar; S3, feature extraction and fusion: extract visual feature points from the preprocessed panoramic image data, extract spatial structure features from the distance data and shape information data, fuse the visual feature points with the spatial structure features, and construct a preliminary fusion feature map; S4, semantic parsing and map construction: Apply deep learning models to semantically annotate the fused feature map, identify objects and structures in the environment, and build a semantic map; S5, map optimization and positioning correction: Optimize the semantic map based on semantic information, adjust the erroneous marks and position deviations in the map, and perform dynamic positioning correction by combining the optimized semantic map with the real-time collected data to improve positioning accuracy; S6, learning update: continuously update the semantic map according to new environmental data, and optimize the deep learning model to adapt to new scenarios and environmental changes; S7, Output and Application: Generate an optimized semantic SLAM map and use this map for subsequent navigation and robot decision support, providing navigation functions and decision analysis.
2. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 1 is characterized in that: The S1 specifically includes: S11, hardware configuration: use a camera that provides 360-degree panoramic images, select one or more lidars, and configure them according to the required accuracy and range; S12, installation and positioning: install the panoramic camera and the LiDAR on the same platform to ensure that the relative position between the panoramic camera and the LiDAR is fixed and known; S13, time synchronization: synchronize the panoramic camera and lidar to ensure that the data collected by the two are aligned in time; S14, data collection: operating the panoramic camera and the lidar to collect data simultaneously, the panoramic camera continuously captures panoramic images of the environment, and the lidar scans the surrounding environment to collect distance and shape information data.
3. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 2 is characterized in that: The S2 specifically includes: S21, image data denoising: using a bilateral filtering algorithm to denoise the panoramic image data collected by the panoramic camera; S22, distance data filtering: using Gaussian filtering algorithm to filter the distance data; S23, shape information data filtering: using the moving least squares method to filter the shape information data.
4. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 3 is characterized in that: The S3 specifically includes: S31, visual feature extraction: using SIFT algorithm to extract visual feature points from the preprocessed panoramic image data; S32, spatial structure feature extraction: extract spatial structure features from distance data and shape information data using principal component analysis; S33, feature fusion: use ICP algorithm to fuse and align feature points; S34, fusion feature map construction: constructing the fused feature points into a fusion feature map.
5. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 4 is characterized in that: The S4 specifically includes: S41, data preparation: semantic labeling of the fused feature map, including roads, buildings, pedestrians and vehicles, to construct a labeled fused feature map dataset, and perform standardization and normalization processing; S42, model selection: select 3D U-Net model as the deep learning model; S43, model training: the fused feature map dataset is divided into a training set and a validation set, and the training set data is used to train the 3DU-Net model; S44, model application: semantic annotation on the fused feature map to identify objects and structures in the environment; S45, build detailed semantic map: integrate the semantically annotated fused feature map into a global semantic map.
6. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 5 is characterized in that: In S5, the semantic map is optimized based on the semantic information to adjust the erroneous marks and position deviations in the map, specifically including: S51, incorrect label identification: based on known semantic information, verify the labels on the current semantic map; S52, position deviation detection: detecting position deviation by comparing the position of the same landmark in consecutive frames or in environmental data collected by the panoramic camera and the lidar, where if the same landmark shows spatial differences in consecutive frames or when compared with data collected by the panoramic camera or the lidar, there is position deviation; S53, Graph Optimization Technology: Use graph optimization technology to adjust and optimize the structure of the map.
7. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 6 is characterized in that: In S5, the optimized semantic map is combined with the real-time collected data to perform dynamic positioning correction to improve positioning accuracy, which specifically includes: S54, real-time data and map alignment: use the real-time collected data for preliminary alignment, and use the ICP algorithm to finely align the real-time collected data with the optimized semantic map; S55, dynamic positioning correction: Fuse the real-time collected data with the optimized semantic map to generate the latest feature point set, and use the extended Kalman filter to perform dynamic positioning correction on the fused feature points.
8. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 7 is characterized in that: The S6 specifically includes: S61, data update: collect new environmental data in real time through panoramic cameras and lidar, and input the data into the deep learning model for update; S62, Model Optimization: Retrain and optimize the deep learning model based on new environmental data, and use the gradient descent algorithm to update the model parameters; S63, semantic map update: update the semantic map in real time based on new data and optimized models.
9. The semantic SLAM optimization method of panoramic vision and laser radar fusion according to claim 8 is characterized in that: The S7 specifically includes: S71, semantic SLAM map generation: Combine real-time data and optimized models to generate optimized semantic SLAM maps S72, Navigation and Decision Support: Use semantic SLAM maps for robot navigation and decision support, providing path planning, obstacle avoidance and target positioning functions; S73, user interface: display the optimized semantic SLAM map through a preset user interface, allow users to interact and control, and provide map visualization and data export.
Citation Information
Cited By
Electric energy meter seal integrity detection method and system based on U-Net semantic segmentation
CN119359694A
Mobile robot navigation method and system based on deep learning
CN120143802A
Visual management method and system based on large model and AR live-action map
CN120372037A
Map dynamic construction method and system fused with laser vision
CN120506938A
Map dynamic construction method and system fusing laser vision
CN120506938B
