Inspection robot SLAM positioning mapping method based on deep learning multi-sensor fusion

Abstract

The invention discloses an inspection robot SLAM positioning mapping method based on deep learning multi-sensor fusion, and belongs to the technical field of synchronous positioning and mapping. The method comprises the following steps: collecting data of a laser radar, an IMU (Inertial Measurement Unit) and a visual camera, extracting inspection environment characteristics, and carrying out dynamic target detection and semantic segmentation to obtain dynamic target and static background semantic information; the information is combined with multi-sensor data by using a particle filter algorithm to realize robot positioning; an octree map is constructed, geometric constraint loopback detection and semantic information are combined to optimize the map, and a dynamic target state is updated; the limitation of a single sensor is overcome, and stable positioning can be achieved in a complex environment; static structure retention loss is introduced into the YOLOv8n model, and the static edge misjudgment dynamic probability is reduced; particle filtering and semantic consistency scoring are carried out, correction can be carried out during visual deficiency or dynamic interference, and robustness is improved; loopback detection uses a geometric and semantic fusion distance, the recall rate and accuracy are improved, multiple sensors are combined with deep learning to adapt to a complex environment, and routing inspection reliability is guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of Simultaneous Localization and Mapping (SLAM) technology, specifically relating to a SLAM localization and mapping method for inspection robots based on deep learning multi-sensor fusion. Background Technology

[0002] In today's era of rapid technological advancement, Simultaneous Localization and Mapping (SLAM) technology plays a crucial role in the field of inspection robots, especially in their application. It enables inspection robots to autonomously locate themselves in unknown environments and simultaneously build maps of their surroundings, laying the foundation for autonomous navigation and intelligent inspection functions. However, traditional SLAM systems face numerous challenges in real-world applications, limiting their performance and application scope. In complex real-world environments, such as frequently moving indoor spaces and high-traffic outdoor areas, numerous dynamic objects exist. Traditional SLAM systems struggle to effectively distinguish between dynamic objects and static environmental features, easily misclassifying dynamic object features as static ones during feature matching. In densely populated factory workshops, the frequent movement of workers and mobile devices leads to incorrect feature matching in traditional SLAM systems, causing deviations in localization and map building, severely impacting the positioning accuracy and mapping precision of inspection robots. These deviations accumulate over time, eventually causing the inspection robot to fail in localization and be unable to complete its inspection tasks.

[0003] Common sensors such as LiDAR, inertial measurement units (IMUs), and visual cameras each have significant limitations when providing data to SLAM systems. To compensate for the shortcomings of single sensors, multi-sensor fusion technology has been applied to SLAM systems. However, most existing multi-sensor fusion SLAM methods simply merge sensor data without fully exploring the complementarity between different sensor data. Existing mainstream SLAM solutions all have insurmountable limitations when facing complex inspection environments. Visual or visual-inertial solutions, represented by ORB-SLAM3 and VINS-Mono, while highly accurate, are extremely sensitive to dynamic objects and changes in lighting. LiDAR or laser-inertial solutions, represented by Cartographer, LOAM, and LIO-SAM, are stable in static environments but also cannot effectively handle dynamic interference, and most lack efficient global loop closure detection and semantic understanding capabilities. These shortcomings make them prone to positioning drift, inaccurate map building, and even system failure in real-world inspection scenarios such as factories and warehouses where personnel and vehicles move frequently. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a SLAM localization and mapping method for inspection robots based on deep learning and multi-sensor fusion, which is implemented through the following steps:

[0005] A deep learning-based multi-sensor fusion-based SLAM localization and mapping method for inspection robots, comprising the following steps:

[0006] S1: Collect data from lidar, IMU, and vision camera and preprocess them separately, while simultaneously extracting the features of the inspection environment;

[0007] S2: Based on the extracted inspection environment features, perform dynamic target detection and semantic segmentation to obtain dynamic target category, location and static background semantic information;

[0008] S3: Based on the dynamic target and static background, the robot's positioning is achieved by fusing lidar distance information, IMU attitude information and visual camera feature information using a particle filter algorithm.

[0009] S4: Construct an octree map, combine geometric constraint loop closure detection and semantic information to optimize the map, and update the dynamic target position and status in real time.

[0010] Preferably, in step S1, the lidar data preprocessing process includes:

[0011] Outlier removal using statistical filtering: Calculate the value of each point in the lidar point cloud. Distance to its neighborhood centroid Obtain the mean and standard deviation of the neighborhood distance of that point, and set a multiplier threshold. If the value is greater than the sum of the mean and a set multiple threshold multiplied by the standard deviation, then the decision point is... Identify and remove outliers;

[0012] Voxel filtering is used to reduce data density: the voxel grid size is set according to the LiDAR point cloud. The maximum and minimum values ​​of the axes divide the point cloud space into voxel grids. The centroid of the point set within each voxel is calculated, and the centroid is used as the representative point of the voxel to complete the data dimensionality reduction. The centroid is the average coordinate of all points within the voxel.

[0013] Preferably, in step S1, the IMU data preprocessing process includes:

[0014] Determine the initial attitude of the inspection robot: After denoising the IMU accelerometer data, calculate the initial pitch angle and initial roll angle based on the gravity direction vector;

[0015] Real-time update of inspection robot posture: At each sampling moment, the posture change is calculated by combining the angular velocity measured by the gyroscope, and the posture of the previous moment and the current posture change are merged according to the preset weighting coefficient to update the posture of the inspection robot.

[0016] Preferably, in step S1, the visual camera data preprocessing and inspection environment feature extraction process includes:

[0017] Denoising of visual camera images is achieved through bilateral filtering;

[0018] Distortion correction and environmental feature extraction are performed using Zhang's calibration method: Based on the pinhole camera model, the least squares method is used to solve the camera intrinsic parameter matrix, rotation matrix and translation vector, and the distortion model is combined to complete image correction and inspection environment feature extraction.

[0019] The pinhole camera model is as follows:

[0020] (10)

[0021] in, As a scaling factor, For the camera Focal length component of direction, The coordinates of the principal point of the image. Construct a rotation matrix, Image pixel coordinates, It is a translation vector.

[0022] Preferably, in step S2, the dynamic target detection process includes:

[0023] The YOLOv8n model was used as the detection model, and the training dataset was preprocessed to be suitable for the standard training format of the model.

[0024] Using the characteristics of the inspection environment as input and the dynamic target category and location as output, the model is trained using the stochastic gradient descent algorithm, specifically:

[0025] Set the initial learning rate, momentum, and number of training rounds, and iterate and optimize step by step according to the number of training rounds. After each preset number of training rounds, the learning rate decays to a preset percentage of the current value.

[0026] The joint loss function for multiple tasks is designed as follows:

[0027] (11)

[0028] in, These are the classification loss weight coefficients. To locate the loss weight coefficients, The confidence loss weighting coefficient is... The loss weight coefficients are retained for static structures. Cross-entropy classification loss, For CIoU positioning loss, For binary cross-entropy confidence loss, Loss is retained for static structures.

[0029] Preferably, in step S2, the semantic segmentation process includes:

[0030] The DeepLabv3+ network with an encoder-decoder architecture is used as the segmentation model. The encoder is used to extract deep semantic features of the image, and the decoder is used to restore pixel-level spatial resolution.

[0031] Using the characteristics of the inspection environment as input and static background semantic information as output, the model is trained using the stochastic gradient descent algorithm, specifically:

[0032] Set the learning rate and total number of training rounds, and iterate and optimize step by step according to the total number of training rounds;

[0033] The loss function is set as follows:

[0034] (13)

[0035] in, For cross-entropy loss, For Dice loss, the weighting coefficients , This is a semantic boundary consistency loss;

[0036] (14)

[0037] in, The total number of pixels involved in the calculation. This is the result of normalizing the grayscale gradient magnitude of the image. This is the geometric edge weight adjustment factor. For pixels Semantic prediction probability distribution at the location.

[0038] Preferably, in step S3, the process of locating the inspection robot based on the particle filter algorithm includes:

[0039] Based on the robot's pose prior information, initialize a preset number of particles, each particle representing a possible robot pose, assign an initial weight to each particle, and the sum of the initial weights of all particles is 1.

[0040] At each time step, based on the motion model of the inspection robot and the pose of each particle at the previous time step, the predicted pose of each particle at the current time step is calculated.

[0041] The LiDAR data, IMU data, visual camera data, and semantic information from step S2 are fused together, and a semantic consistency score is introduced to update the weight of each particle. After the update, the weights of all particles are normalized to ensure that the sum of the weights of all particles is 1.

[0042] The normalized weighted particles are resampled, and high-weighted particles are retained. The estimated pose of the inspection robot at the current moment is calculated by the particle weighted average, so as to realize the localization of the inspection robot.

[0043] The weight of each particle is updated as follows:

[0044] (15)

[0045] in, For multi-sensor likelihood, it reflects the degree of matching between particle pose and actual observation. For semantic weight coefficients, Score semantic consistency.

[0046] Preferably, in step S4, the process of constructing the octree map includes:

[0047] With a preset maximum recursion depth for the octree, a preset number of LiDAR point cloud 3D coordinates are selected as base point cloud elements, and the base point cloud is calculated in... The maximum and minimum values ​​along the three coordinate axes are used to construct a root node cube that can completely enclose the entire point cloud;

[0048] All LiDAR point clouds are incorporated into the root node cube. If the current cube contains multiple point clouds and the maximum recursion depth has not been reached, continue along... The midpoint of the axis is used to cut it into 8 equal-sized sub-cubes. The sub-cube to which each point belongs is determined based on the three-dimensional coordinates of the point cloud, thus completing the point cloud allocation.

[0049] Repeat the cube subdivision and point cloud allocation process until all sub-cubes contain only one point cloud, no point cloud, or reach the maximum recursion depth, thus completing the construction of the octree map.

[0050] Preferably, in step S4, the geometric constraint loop closure detection process includes:

[0051] In the corrected visual images of the inspection robot's current and historical positions, based on the inspection environment features extracted in step S1, the gradient descriptors of the feature points are calculated using a Gaussian function, where... The directional gradient is the difference between the Gaussian function values ​​of the pixels to the right and left of a feature point. The directional gradient is the difference between the Gaussian function values ​​of the pixels below and above a feature point;

[0052] Match feature point pairs between the current location and historical locations, calculate the comprehensive similarity of the feature point sets of the two locations, and determine whether there is a loop.

[0053] The geometric relationship between feature point pairs is verified by using a homography matrix. If the feature point pair satisfies the geometric relationship, a loop closure is confirmed, and the robot pose graph is optimized using the homography matrix.

[0054] The formula for verifying the geometric relationship of feature point pairs using the homography matrix is ​​as follows:

[0055] (twenty one)

[0056] in, , These are the corrected pixel coordinates of the current image and the historical image, respectively. Homography matrix Element.

[0057] Preferably, the calculation of the comprehensive similarity between the feature point sets of the two is specifically as follows:

[0058] The Euclidean distance between feature point descriptors is calculated using the following formula:

[0059] (19)

[0060] in, These are the i-th components of the two feature point descriptors, respectively. Let be the Euclidean distance between two feature point descriptors, and D be the dimension of the feature point descriptor;

[0061] The comprehensive similarity distance is obtained by combining geometric similarity and semantic similarity, and the formula is as follows:

[0062] (20)

[0063] in, The fusion coefficient is a combination of similarity distance. The average value of the Euclidean distance for the feature descriptors; The Jaccard distance is the semantic segmentation result of the two images.

[0064] Advantages of this invention:

[0065] (1) This invention integrates the distance information of the lidar, the attitude information of the IMU and the feature information of the visual camera through multi-sensor fusion, and processes it based on the particle filter algorithm. It can effectively overcome the limitations of a single sensor, accurately locate the inspection robot, reduce positioning errors, and achieve stable and reliable positioning even in complex environments, thus avoiding inspection task failure due to inaccurate positioning.

[0066] (2) This invention introduces static structure preservation loss into the YOLOv8n model. This effectively reduces the probability of misidentifying the edges of static objects (such as door frames and equipment) as dynamic targets, providing SLAM systems with more stable and reliable static environmental features;

[0067] (3) This invention utilizes a geometrically constrained loop closure detection algorithm combined with semantic information obtained from deep learning to optimize the map, and updates the position and state of dynamic targets in real time. This effectively avoids the problem of traditional SLAM systems misclassifying dynamic targets as static backgrounds, reduces map construction bias, and improves the accuracy and consistency of the map. Semantic consistency scoring is introduced into particle filtering. This enables the positioning system to correct deviations based on the spatial consistency of semantic information when visual features are missing or dynamic interference is severe, significantly improving the robustness of the system and providing a more reliable map basis for the navigation and environmental understanding of inspection robots.

[0068] (4) This invention integrates multi-sensor data and combines it with deep learning algorithms, enabling the system to better adapt to complex and ever-changing environments. Deep learning's powerful object detection and semantic understanding capabilities help the system accurately distinguish between dynamic and static objects. In loop closure detection, a comprehensive similarity distance that integrates geometric and semantic information is used. This significantly improves the recall and accuracy of loop closure detection in scenarios where visual appearance changes (such as changes in lighting or seasons). The multi-sensor fusion strategy fully leverages the complementarity of data, enhances the stability and reliability of the system in interference environments, and ensures that the inspection robot can work normally in various complex scenarios. Attached Figure Description

[0069] Figure 1 This is a flowchart of a SLAM localization and mapping method for an inspection robot based on deep learning multi-sensor fusion according to an embodiment of the present invention;

[0070] Figure 2 This is a flowchart of a lidar data processing method according to an embodiment of the present invention;

[0071] Figure 3 This is a flowchart of dynamic target detection in a YOLOv8n inspection environment according to an embodiment of the present invention;

[0072] Figure 4 This is a flowchart of semantic segmentation in a DeepLabv3+ network inspection environment according to an embodiment of the present invention.

[0073] Figure 5 This is a flowchart of a multi-sensor fusion positioning process based on a particle filter algorithm according to an embodiment of the present invention;

[0074] Figure 6This is a schematic diagram of a loop closure detection process according to an embodiment of the present invention;

[0075] Figure 7 The figure shows a comparison of trajectory estimation for a static workshop dataset using different SLAM methods according to an embodiment of the present invention. In the figure, (a) is a comparison of overall trajectory estimation in a static workshop environment, (b) is a comparison of trajectory details in a region with complex geometric structures, and (c) is a comparison of trajectory details in a region with dense equipment.

[0076] Figure 8 This is a comparison diagram of dynamic target processing for a dynamic workshop dataset according to an embodiment of the present invention; in the figure, (a) is the dynamic target trajectory diagram before processing by the method of the present invention, (b) is the dynamic target trajectory diagram after processing by the method of the present invention, and (c) is a comparison diagram of the trajectories of six SLAM methods in a dynamic target scene;

[0077] Figure 9 The following is a comparison of map building effects according to an embodiment of the present invention; in the figure, (a) is the map building effect of ORB-SLAM3, (b) is the map building effect of VINS-Mono, (c) is the map building effect of Cartographer, (d) is the map building effect of LOAM, (e) is the map building effect of LIO-SAM, and (f) is the map building effect of the method of the present invention.

[0078] Figure 10 The following is a bar chart of an ablation experiment according to an embodiment of the present invention; in the figure, (a) is a bar chart of positioning accuracy, (b) is a bar chart of loop closure detection recall rate, (c) is a bar chart of processing speed, (d) is a bar chart of dynamic target recognition accuracy, and (e) is a bar chart of comprehensive performance comparison. Detailed Implementation

[0079] An embodiment of the present invention will be further described below with reference to the accompanying drawings.

[0080] In this embodiment of the invention, the SLAM localization and mapping method for inspection robots based on deep learning multi-sensor fusion is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps:

[0081] S1: Collect data from LiDAR, IMU, and visual camera, and perform the following preprocessing respectively:

[0082] LiDAR data processing flowchart as follows Figure 2 As shown:

[0083] Outliers in LiDAR point cloud data are removed using statistical filtering, as detailed below:

[0084] (1) Calculation point To its neighborhood centroid distance The centroid of a neighborhood point is the average coordinate of all points within its preset neighborhood, as shown in the formula:

[0085] (1);

[0086] (2) Calculate the mean distance of all neighboring points of the point. and standard deviation ;

[0087] (3) Set the multiple threshold ,like Then the decision point Outliers are identified and removed from the point cloud dataset;

[0088] Voxel filtering is used to reduce data density. This is achieved by dividing the data into voxel grids and calculating the centroid of the point set within each voxel as a representative point, as detailed below:

[0089] In the entire point cloud dataset, based on the LiDAR point cloud voxel filtering algorithm, the first step is to calculate... Maximum value on the three coordinate axes and minimum value Dividing the point cloud space into basic units yields a minimum spatial rectangle; the side length of the minimum spatial rectangle is... :

[0090] (2);

[0091] Set voxel grid size ,Will Divide the raster according to its size and calculate the total number of voxel raster cells. The calculation formula is as follows:

[0092] (3)

[0093] in, They are respectively Number of voxels on the axis The number of voxel grids;

[0094] Then, determine the index number of all voxel graticles. The grid position of each data point is determined by rounding down. The calculation formula is as follows:

[0095] (4)

[0096] in, The first Data points at Coordinates on the axis;

[0097] Finally, we only need to simplify the point cloud data in the voxel raster, using the center of the point cloud in the voxel raster to represent all the point clouds in the voxel raster. The calculation formula is as follows:

[0098] (5)

[0099] in, The centroid of the point set within the voxel. The set of points contained in each voxel. For the first Point cloud data from a single LiDAR sensor;

[0100] Each frame of point cloud data is a local map data. The point cloud map is constructed using a full sampling method. After voxel filtering, the point cloud data removes redundant information and the point information is effectively compressed.

[0101] IMU Data Processing:

[0102] A stable attitude estimate is obtained by fusing accelerometer and gyroscope data from the IMU using a complementary filtering algorithm, as detailed below:

[0103] The initial attitude is determined based on accelerometer data. The component of gravitational acceleration in the carrier coordinate system is: The accelerometer measures the direction vector of gravity as follows: The angular velocity measured by the gyroscope is ;

[0104] Based on the above data, the initial pitch angle and roll angle The estimated value is calculated using the following formula:

[0105] (6)

[0106] (7)

[0107] At each sampling time, the attitude information is updated by weighted fusion of the attitude change calculated by the gyroscope integral and the attitude change calculated by the accelerometer.

[0108] The update formulas for pitch and roll angles are:

[0109] (8)

[0110] (9)

[0111] in, =0.95 is the weighting coefficient. These are the updated pitch and roll angles, These are the initial pitch angle and roll angle, respectively. These are the changes in pitch angle and roll angle, respectively. These are the pitch and roll angles at the previous moment, respectively.

[0112] Visual camera data processing:

[0113] Denoising of the visual camera images is achieved through bilateral filtering, followed by distortion correction and environmental feature extraction using Zhang's calibration method, as detailed below:

[0114] Bilateral filtering for noise reduction: When performing bilateral filtering on images from a visual camera, the pixel similarity in both the spatial and grayscale domains must be considered to effectively remove noise while preserving image edge features.

[0115] Zhang's calibration method for distortion correction and feature extraction: Camera distortion correction is performed based on Zhang's calibration method. The parameters are solved by constructing the coordinate mapping relationship of the checkerboard image, as detailed below:

[0116] (1) Prepare a checkerboard pattern of known size (the position of its interior corner points in the world coordinate system is known), and take multiple checkerboard images from different angles and positions using the camera to be calibrated;

[0117] (2) For each captured chessboard image, the Harris corner detection algorithm is used to extract the interior corners of the chessboard.

[0118] (3) Based on the pinhole camera model, establish the coordinates of the corner points of the chessboard grid in the world coordinate system. Image pixel coordinates The mapping relationship, for each corner point, satisfies:

[0119] (10)

[0120] in, As a scaling factor, For the camera , Focal length component of direction, The coordinates of the principal point of the image. Construct a rotation matrix, Image pixel coordinates, It is a translation vector;

[0121] (4) Solve the above overdetermined equations using the least squares method to obtain the camera's intrinsic parameter matrix, including the focal length. and principal point coordinates Then, the rotation matrix corresponding to each image is solved. and translation vector ;

[0122] (5) Based on the obtained intrinsic and extrinsic parameters, a distortion model containing radial and tangential distortion is established. The original image is then distorted using this model to obtain the corrected image and to provide accurate environmental features for subsequent steps (Formulas 17 and 18).

[0123] S2: Based on the inspection environment features extracted in step S1, dynamic target detection in the inspection environment is achieved using the YOLOv8n model. Simultaneously, the DeepLabv3+ network is used to complete semantic segmentation of the inspection environment, accurately identifying dynamic targets and static backgrounds within the inspection environment, as detailed below:

[0124] Inspection of dynamic target detection in the environment, flowchart as follows Figure 3 As shown: The lightweight YOLOv8n model was selected, balancing detection accuracy and real-time performance; training and detection were conducted for dynamic targets (such as personnel, mobile devices, etc.) in inspection scenarios, specifically as follows:

[0125] (1) The YOLOv8n model was selected as the detection model, and the inspection environment features were used as input;

[0126] (2) To adapt to model training, the dataset is preprocessed. All images in the training dataset, which includes the KITTI large-scale public dataset and the custom dataset of inspection scenarios (such as factory equipment, personnel, and handling inspection robots), are adjusted to 640×640 pixels. At the same time, the target annotation data is converted to the YOLO model standard training format.

[0127] (3) The model is trained using the stochastic gradient descent (SGD) algorithm:

[0128] Set the initial learning rate ,momentum During training, a learning rate decay strategy is used. Every 10 training epochs, the learning rate is decayed to 0.9 times the current value to ensure stable convergence of the model.

[0129] The loss function adopts a multi-task joint loss function. Taking into account classification, location, confidence level, and static structure protection, the formula is as follows:

[0130] (11)

[0131] in, These are the classification loss weight coefficients. To locate the loss weight coefficients, The confidence loss weighting coefficient is... The loss weight coefficients are retained for static structures. Cross-entropy classification loss, For CIoU positioning loss, For binary cross-entropy confidence loss, Loss is retained for static structures;

[0132] In this embodiment of the invention, a static structure preservation loss is introduced during the training process. This is specifically designed to penalize cases where static edges (such as equipment outlines or door frames) are misidentified as dynamic targets. The formula is as follows:

[0133] (12)

[0134] in, The total number of pixels involved in the calculation. This is a weight matrix generated based on Canny edge detection, with higher weights for image edge regions. The total number of target categories, and Pixels Category The true label and predicted probability.

[0135] Inspection environment semantic segmentation, flowchart as follows Figure 4 As shown: The DeepLabv3+ network is used to achieve semantic segmentation of the environment. Its encoder-decoder architecture accurately extracts deep features from the image and restores spatial resolution, ultimately distinguishing dynamic targets from static backgrounds (such as factory equipment, shelves, walls, etc.). Specifically:

[0136] (1) The DeepLabv3+ network with encoder-decoder architecture is used. The encoder extracts deep semantic features of the image (captures the overall structure of the environment), and the decoder restores pixel-level spatial resolution (ensuring the accuracy of segmentation details).

[0137] (2) The model is trained using the stochastic gradient descent (SGD) algorithm:

[0138] The learning rate is set to 0.0001, and the total number of training rounds is 30.

[0139] The loss function uses the total loss function. To balance semantic classification accuracy and boundary alignment, the formula is as follows:

[0140] (13)

[0141] in, For cross-entropy loss, For Dice's loss, =0.5 is the weight coefficient for semantic boundary consistency loss. This is a semantic boundary consistency loss;

[0142] In this embodiment of the invention, semantic boundary consistency loss is introduced during the training process. This ensures that the predicted semantic boundaries are aligned with the geometric edges of the image, improving segmentation accuracy. The formula is as follows:

[0143] (14)

[0144] in, This is the result of normalizing the grayscale gradient magnitude of the image. =0.5 is the geometric edge weight adjustment factor. For pixels Semantic prediction probability distribution at the location;

[0145] Through calculation and training, the model is able to accurately segment static backgrounds in the factory, such as equipment, shelves, and walls, providing semantic information for subsequent map building and environmental understanding.

[0146] S3: Based on the dynamic targets and static background identified in step S2, and using a particle filter algorithm, the robot integrates LiDAR distance information, IMU attitude information, and visual camera feature information to achieve precise positioning. The flowchart is as follows: Figure 5 As shown, the details are as follows:

[0147] (1) At the initial moment, initialize the robot according to the prior information of the robot's pose. particles, denoted as Each particle Representing a possible pose of the inspection robot, assigning initial weights to each particle. Ensure that the sum of the weights of all particles satisfies ;

[0148] (2) At each time step Based on the motion model of the inspection robot, the particle's pose at the previous moment... Based on this, the predicted pose of the particle at the current moment is calculated. This enables dynamic updating of the particle set as it moves with the inspection robot.

[0149] (3) Measure the angular velocity and acceleration of the inspection robot through IMU to obtain real-time attitude change data; collect distance information of the surrounding environment of the inspection robot through lidar to capture the spatial features of the static background; extract key feature points in the environment through visual camera to supplement the environmental information of the visual dimension;

[0150] (4) Introduce semantic consistency scoring to calculate the current weight of each particle. The formula is as follows:

[0151] (15)

[0152] in, For multi-sensor likelihood, it reflects the degree of matching between particle pose and actual observation. For semantic weight coefficients, Score semantic consistency;

[0153] After the weights are calculated, the weights of all particles are normalized to ensure... .

[0154] (5) Based on the normalized particle weights, re-select from the current particle set Of the particles, those with higher weights are retained first.

[0155] (6) Using a particle-weighted average method, the pose of each particle after resampling is used. Based on this, combined with its corresponding weights To obtain the current time step of the inspection robot Final estimated pose The formula is as follows:

[0156] (16);

[0157] (7) Set the time step Add 1, repeat steps (2)-(6) of particle prediction → data acquisition → weight update → resampling → pose estimation process, continuously perform real-time dynamic estimation of the pose of the inspection robot, so as to adapt to the movement state and environmental changes of the inspection robot and maintain stable positioning accuracy.

[0158] S4: Based on the precise localization results of the inspection robot in step S3, an octree map is constructed, geometric constraint loop closure detection is performed, and the map is optimized by combining the semantic information from step S2. This enables real-time updates of dynamic targets, ultimately improving the accuracy and consistency of the map, as detailed below:

[0159] Octree map construction:

[0160] Based on the positioning results output in step S3, an octree is constructed by iteratively subdividing the cube (balancing map storage efficiency and detail accuracy), as follows:

[0161] (1) The maximum recursion depth of the octree is preset to 3. The three-dimensional coordinate data of 10 evenly distributed seed points are selected as the basic point cloud elements for constructing the octree; the basic point cloud is calculated in The maximum and minimum values ​​along the three coordinate axes are used as boundaries to construct a large cube that completely encloses the entire point cloud, which serves as the root node of the octree. The center of the root node cube is the intersection of the midpoints of the maximum and minimum values ​​along the three coordinate axes, and the side length is taken as... The maximum value among the "maximum-minimum difference" in the three directions ensures that the cube can completely cover the point cloud;

[0162] (2) Substitute all point cloud data points into the root node cube, so that all point clouds are initially at the root node level of the octree; if the current cube contains multiple point clouds and has not reached the maximum recursion depth, proceed along... The midpoint of the coordinate axis divides it into 8 equal-sized sub-cubes, with the side length of each sub-cube being half that of the parent cube. Based on the 3D coordinates of each point cloud, the spatial range of the sub-cube to which it belongs is determined, and the point cloud is assigned to the corresponding sub-cube.

[0163] (3) Repeat step (2) to continuously subdivide and assign point clouds to all cubes that meet the subdivision conditions until all sub-cubes contain only 1 point cloud, no point cloud, or have reached the maximum recursion depth of 3, and the octree map is completed.

[0164] Geometric constraint loop closure detection:

[0165] (1) Extract environmental feature points from the visual images of the current and historical positions of the inspection robot, and use the gradient algorithm to calculate the feature point descriptors, as shown in the following formula:

[0166] (17)

[0167] (18)

[0168] in, The standard deviation is Two-dimensional Gaussian function, Corresponding feature points at direction and The gradient value in the direction;

[0169] (2) Match feature point pairs between the current location and historical locations, calculate the comprehensive similarity of the feature point sets of the two locations, and then determine whether there is a loop:

[0170] The Euclidean distance for feature point descriptors is calculated using the following formula:

[0171] (19)

[0172] in, These are the i-th components of the two feature point descriptors, respectively. Let be the Euclidean distance between two feature point descriptors, and D be the dimension of the feature point descriptor;

[0173] The comprehensive similarity distance between the feature point sets of the current location and historical locations is calculated by fusing geometric and semantic information. :

[0174] (20)

[0175] in, The fusion coefficient is a combination of similarity distance. The average value of the Euclidean distance for the feature descriptors; The Jaccard distance between the semantic segmentation results of the two images;

[0176] Set loop closure threshold ,like If so, it is determined that there is a loop relationship between the current position and the historical position;

[0177] (3) For the matched feature point pairs, the homography matrix is ​​used to... Calculate and verify its geometric consistency using the following formula:

[0178] (twenty one)

[0179] in, , These are the corrected pixel coordinates of the current image and the historical image, respectively. Homography matrix Element;

[0180] If the geometric relationship between feature point pairs is verified to be consistent through this matrix, then the existence of a loop is finally confirmed, and this is then utilized... Optimize the pose diagram of the inspection robot.

[0181] Based on the semantic information from step S2, the map structure is further optimized and dynamic targets are updated in real time:

[0182] Optimize map structure: Utilize the pose associations confirmed by loop closure detection, and combine them with the semantic labels from step S2 (such as static areas like equipment, shelves, and walls, and dynamic areas like personnel and mobile devices) to correct pose deviations in the octree map; for example, by "matching the positions of the same static device in the historical and current semantic maps", the pose error of the loop closure position is calibrated to improve the consistency of the overall map structure.

[0183] A schematic diagram of the loop closure detection process is shown below. Figure 6 As shown, the multimodal fusion detection process of historical and current images is demonstrated based on data combination, semantic segmentation, and feature matching. Finally, loop closure confirmation is completed through geometric constraints and semantic consistency verification.

[0184] Real-time updates of dynamic targets: Based on the dynamic targets identified in step S2 and their real-time locations, the current area of ​​the dynamic targets is marked in the octree map, and the residual point cloud of dynamic targets at historical moments is deleted; if a dynamic target remains stationary for a long time (exceeding a preset threshold, which is set according to the motion characteristics of the dynamic targets in the inspection scenario, such as personnel dwell time, equipment movement frequency, etc.), it is temporarily classified as a static background, otherwise its position and motion status are continuously updated.

[0185] This invention underwent comprehensive comparative experiments, ablation experiments, and system performance analysis on the publicly available KITTI dataset and self-built static and dynamic workshop datasets, comparing it with current mainstream SLAM schemes such as ORB-SLAM3, VINS-Mono, Cartographer, LOAM, and LIO-SAM. As shown in Table 1, the positioning accuracy of this invention is significantly superior in the comparison of absolute trajectory error (ATE) across different datasets. In the standard outdoor driving scenario of the KITTI dataset, the ATE of this invention is 0.072 meters, significantly lower than all compared methods. On the self-built static workshop dataset, the ATE of this invention is 0.067 meters, comparable to LIO-SAM (0.062 meters) and far superior to vision-based schemes. On the most challenging self-built dynamic workshop dataset, the advantage of this invention is most prominent, with an ATE of 0.088 meters, while ORB-SLAM3 fails to locate due to severe interference from dynamic targets, VINS-Mono has an error as high as 0.551 meters, and the errors of other laser schemes are also much higher than those of this invention.

[0186] Table 1. Absolute trajectory error (unit: meters)

[0187]

[0188] Figure 7 The results visually demonstrate the trajectory estimation results of this invention compared to other methods in a static workshop dataset, showing a higher degree of trajectory fit.

[0189] Figure 8 This paper demonstrates the effectiveness of processing dynamic targets in a dynamic workshop environment. Traditional methods retain the trajectories of moving pedestrians in the laser point cloud, while this method successfully identifies and filters out these dynamic point clouds through dynamic target detection and semantic segmentation, resulting in a clean static environment structure.

[0190] In terms of loop closure detection, this invention introduces a comprehensive similarity distance that integrates geometric and semantic information. Even in complex scenarios such as changes in lighting and viewing angle, it can still reliably identify loop closures, achieving a loop closure recall rate of up to 91.5% while maintaining a high accuracy of 96.8%, effectively avoiding pose jumps caused by erroneous loop closures.

[0191] Regarding map quality, as shown in Table 2, the global map quality of this invention... Figure 1 The system exhibits the lowest consistency error (MCE) at only 0.079 meters. By optimizing the pose graph using semantic information and combining it with an octree map to update dynamic targets in real time, the system can construct a more accurate and consistent environment model, such as... Figure 9 As shown, the map structure constructed by this invention is clear, and the ghosting and trailing phenomena caused by dynamic objects are effectively suppressed.

[0192] Table 2 Comparison of Loopback Detection and Global Consistency Performance

[0193]

[0194] In terms of real-time performance, as shown in Table 3, the average processing frame rate of this invention is 11.5 FPS. Although it is lower than some pure geometric methods, it can fully meet the real-time requirements of the inspection robot. In addition, the system requires GPU acceleration to run the deep learning model, and its CPU utilization rate is 78%, which is within a reasonable range.

[0195] Table 3 Comparison of System Resource Consumption and Real-Time Performance

[0196]

[0197] Further analysis of computational overhead, as shown in Table 4, reveals that visual image processing and semantic segmentation are the main time-consuming components. Dynamic target detection is highly efficient, with an average total processing time of 94.2ms per frame. This demonstrates that the system still possesses excellent real-time processing capabilities even after integrating multiple complex modules.

[0198] Table 4. Time consumed by each module of the system

[0199]

[0200] The present invention designed an ablation experiment, and the results are shown in Table 5. Figure 10The ablation experiment bar chart visually reflects the performance improvement process from the basic configuration to the complete solution. Configuration A (LiDAR + IMU) serves as the basic performance reference, achieving a positioning accuracy of 0.153 meters and a loop retrieval recall rate of 79.6%. Configuration B (A + dynamic target detection), after introducing YOLOv8n for dynamic target detection, improved the positioning accuracy to 0.125 meters, the loop retrieval recall rate to 82.4%, and the dynamic target recognition rate to 89.3%. Configuration C (B + semantic segmentation), after adding DeepLabV3+ for fine semantic segmentation, achieved a positioning accuracy of 0.098 meters and a loop retrieval recall rate of 86.7%. Configuration D (C + auxiliary particle filtering), after introducing semantic consistency scoring into particle filtering, significantly improved the positioning accuracy to 0.081 meters. Configuration E (D + semantic-assisted loop retrieval detection), after fusing semantic similarity distance in loop retrieval detection, achieved optimal system performance, with a positioning accuracy of 0.078 meters and a loop retrieval recall rate of 91.4%.

[0201] Table 5 Ablation Experiment Analysis

[0202]

[0203] The experimental results above demonstrate that the SLAM localization and mapping method for inspection robots based on deep learning multi-sensor fusion proposed in this invention exhibits significant advantages in core performance aspects such as dynamic environment adaptability, semantic perception, and localization accuracy fusion.

[0204] Experiments have demonstrated that this invention effectively solves the problems of positioning drift and mapping distortion in complex scenes using traditional methods through dynamic target filtering, semantic-assisted localization, and loop closure detection. Especially in environments with severe dynamic interference, its positioning robustness and... Figure 1 Significant improvements in consistency. Although there is room for optimization in some metrics under purely static environments or extreme computational efficiency requirements, this solution successfully establishes a technical path for the deep integration of multimodal perception and deep learning, providing a reliable foundation for the practical deployment of a highly robust inspection robot SLAM system, and pointing to the future direction of further improving overall performance through model lightweighting and computing power optimization.

[0205] The method provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A method for SLAM positioning and mapping of a patrol robot based on deep learning multi-sensor fusion, characterized in that: The method comprises the following steps: S1: Collecting and pre-processing the data of the laser radar, IMU and visual camera, and synchronously extracting the inspection environment features; S2: Based on the extracted inspection environment features, performing dynamic target detection and semantic segmentation to obtain dynamic target categories, positions and static background semantic information; S3: Based on the dynamic target and static background, fusing the laser radar distance information, IMU attitude information and visual camera feature information based on a particle filtering algorithm to realize the positioning of the inspection robot; S4: Constructing an octree map, combining geometric constraint loop detection and semantic information to optimize the map, and updating the dynamic target position and state in real time.

2. The deep learning-based multi-sensor fusion patrol robot SLAM positioning and mapping method of claim 1, characterized in that, In step S1, the laser radar data preprocessing process comprises: Outlier removal using statistical filtering: Calculate the value of each point in the lidar point cloud. Distance to its neighborhood centroid Obtain the mean and standard deviation of the neighborhood distance of that point, and set a multiplier threshold. If the value is greater than the sum of the mean and a set multiple threshold multiplied by the standard deviation, then the decision point is... Identify and remove outliers; Adopting voxel filtering to reduce data density: setting voxel grid size, according to laser radar point cloud in The maximum and minimum values of the axis divide the point cloud space into a voxel grid, the center of gravity of the point set in each voxel is calculated, and the data dimensionality reduction is completed by taking the center of gravity as the representative point of the voxel; the center of gravity is the average value of the coordinates of all points in the voxel.

3. The deep learning-based multi-sensor fusion patrol robot SLAM positioning and mapping method of claim 1, characterized in that, In step S1, the IMU data preprocessing process comprises: Determining the initial attitude of the inspection robot: after denoising the IMU accelerometer data, calculating the initial pitch angle and initial roll angle according to the gravity direction vector; Real-time updating of the attitude of the inspection robot: at each sampling time, combining the angular velocity measured by the gyroscope to calculate the attitude change, fusing the attitude at the last time and the current attitude change according to the preset weighting coefficient, and updating the attitude of the inspection robot.

4. The deep learning-based multi-sensor fusion inspection robot SLAM positioning and mapping method of claim 1, characterized in that, In step S1, the visual camera data preprocessing and inspection environment feature extraction process comprises: Denoising the visual camera image by bilateral filtering; Using Zhang's calibration method to complete distortion correction and environment feature extraction: based on the pinhole camera model, using the least squares method to solve the camera intrinsic parameter matrix, rotation matrix and translation vector, and combining the distortion model to complete image correction and inspection environment feature extraction; The pinhole camera model is: （10） wherein, is a scale factor, is a focal length component of the camera in the direction, is a principal point coordinate of the image, constitutes a rotation matrix, is an image pixel coordinate, is a translation vector.

5. The deep learning-based multi-sensor fusion inspection robot SLAM positioning and mapping method of claim 1, characterized in that, In step S2, the dynamic target detection process comprises: Using YOLOv8n model as the detection model, preprocessing the training data set to adapt to the model standard training format; Taking the inspection environment features as the input and the dynamic target categories and positions as the output, training the model using the stochastic gradient descent algorithm, specifically: Setting the initial learning rate, momentum and training rounds, gradually optimizing according to the training rounds, and decaying the learning rate to a preset proportion of the current value every preset number of training rounds; Designing a multi-task joint loss function as: （11） wherein, is a classification loss weight coefficient, is a positioning loss weight coefficient, is a confidence loss weight coefficient, is a static structure reservation loss weight coefficient, is a cross-entropy classification loss, is a CIoU positioning loss, is a binary cross-entropy confidence loss, is a static structure reservation loss, used to reduce the probability of misjudging the edge of a static object (such as a device outline, door frame) as a dynamic target.

6. The deep learning-based multi-sensor fusion inspection robot SLAM positioning and mapping method of claim 1, characterized in that, In step S2, the semantic segmentation process comprises: Using a DeepLabv3+ network with an encoder-decoder architecture as the segmentation model, the encoder is used to extract deep semantic features of the image, and the decoder is used to restore the pixel-level spatial resolution; Taking the inspection environment features as the input and the static background semantic information as the output, training the model using the stochastic gradient descent algorithm, specifically: Setting the learning rate and total training rounds, and gradually optimizing according to the total training rounds; Setting the loss function as: （13） wherein, is a cross-entropy loss, is a Dice loss, is a semantic boundary consistency loss weight coefficient, is a semantic boundary consistency loss; （14） wherein, is the total number of pixels participating in the computation, is the image gray level gradient magnitude normalization result, is the geometric edge weight adjustment factor, is the semantic prediction probability distribution at pixel .

7. The deep learning-based multi-sensor fusion inspection robot SLAM positioning and mapping method of claim 1, characterized in that, In step S3, the process of realizing the positioning of the inspection robot based on the particle filtering algorithm comprises: According to the robot pose prior information, initializing a preset number of particles, each particle representing a possible robot pose, assigning an initial weight to each particle, and the sum of the initial weights of all particles being 1; At each time step, according to the inspection robot motion model, the predicted pose of each particle at the current time is calculated based on the pose of each particle at the previous time; The laser radar data, IMU data, visual camera data after the preprocessing in the fusion step S1 and the semantic information in the step S2 are fused, a semantic consistency score is introduced to update the weight of each particle, and after the update, the weight of all particles is normalized, so that the sum of the weights of all particles is 1; The particles after the normalization are resampled, high-weight particles are retained, the estimated pose of the inspection robot at the current time is calculated through particle weighted average, and the positioning of the inspection robot is realized; The weight of each particle is updated as follows: （15） wherein, is a multi-sensor likelihood, reflecting the matching degree of the particle pose and the actual observation, is a semantic weight coefficient, is a semantic consistency score.

8. The deep learning-based multi-sensor fusion inspection robot SLAM positioning and mapping method of claim 1, characterized in that, In the step S4, the process of constructing the octree map comprises: A preset maximum recursion depth of octree is set, a preset number of three-dimensional coordinates of laser radar point clouds are selected as basic point cloud elements, maximum values and minimum values of the basic point cloud in three coordinate axis directions are calculated, and a root node cube completely wrapping all point clouds is constructed. a preset maximum recursion depth of octree is set, a preset number of three-dimensional coordinates of laser radar point clouds are selected as basic point cloud elements, maximum values and minimum values of the basic point cloud in three coordinate axis directions are calculated, and a root node cube completely wrapping all point clouds is constructed. All the lidar point clouds are classified into the root node cube. If the current cube contains multiple point clouds and the maximum recursion depth is not reached, it is cut into 8 equal sub-cubes along the midpoints of the axes, and the point clouds are assigned to the sub-cubes according to their three-dimensional coordinates. The midpoints of the axes cut it into 8 equal sub-cubes, and the point clouds are assigned to the sub-cubes according to their three-dimensional coordinates. The processes of repeating the cubic subdivision and the point cloud assignment are repeated until all subcubes contain only one point cloud, no point cloud or reach the maximum recursion depth, and the construction of the octree map is completed. 9.The deep learning based multi-sensor fusion inspection robot SLAM positioning and mapping method according to claim 1, characterized in that, In the step S4, the process of geometric constraint loop detection comprises: In the corrected visual image of the current position and the historical position of the inspection robot, based on the inspection environment features extracted in step S1, the gradient descriptor of the feature point is calculated by a Gaussian function, wherein The direction gradient is the difference between the Gaussian function values of the right and left pixels of the feature point, The direction gradient is the difference between the Gaussian function values of the lower and upper pixels of the feature point. The feature point pairs of the current position and the historical position are matched, the comprehensive similarity of the feature point sets is calculated, and it is judged whether there is a loop; The geometric relationship of the feature point pairs is verified through a homography matrix, if the feature point pairs satisfy the geometric relationship, the loop is confirmed, and the robot pose graph is optimized by using the homography matrix; The geometric relationship of the feature point pairs is verified through a homography matrix, and the formula is as follows: （21） in, , These are the corrected pixel coordinates of the current image and the historical image, respectively. Homography matrix Element.

10. The deep learning-based multi-sensor fusion patrol robot SLAM positioning and mapping method of claim 9, characterized in that, The comprehensive similarity of the feature point sets is calculated, and the specific process is as follows: The Euclidean distance between the feature point descriptors is calculated, and the formula is as follows: （19） wherein are the first and second components of the two feature point descriptors, respectively, is the Euclidean distance between the two feature point descriptors, and D is the dimension of the feature point descriptors.​ The comprehensive similarity distance is obtained by fusing the geometric similarity and the semantic similarity, and the formula is as follows: （20） wherein, is a fusion coefficient of the comprehensive similarity distance, is an average value of the Euclidean distance of the feature descriptor; is the Jaccard distance of the semantic segmentation results of the two images.

Images