Multi-level semantic perception map construction method for distributed slam system

Abstract

The application relates to a multi-level semantic perception map construction method for a distributed slam system, which comprises the following steps: collecting site information, obtaining a candidate matching frame based on the site information; extracting a characteristic value from the candidate matching frame, generating a vector as a segment description of the characteristic value, processing the segment description, and obtaining processed point cloud data; constructing a probability 3D occupancy grid map based on the processed point cloud data, performing plane segmentation on the probability 3D occupancy grid map, and constructing a 2D grid map; inputting the 2D grid map into a convolutional neural network model to perform image depth estimation and semantic segmentation, and obtaining depth information and semantic information; fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data; and obtaining a three-dimensional semantic map based on the three-dimensional semantic point cloud data and a camera pose and optimizing the three-dimensional semantic map. The application combines the advantages of geometric information of a traditional slam system and deep learning technology, and realizes efficient perception and modeling of an environment.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction methods for SLAM systems, and in particular to a multi-level semantic perception map construction method for distributed SLAM systems. Background Art

[0002] In recent years, with the rapid development of mobile robotics, distributed autonomous navigation and environmental perception, a key technology, has become a hot research topic in the field of autonomous mobile robots. Slam technology aims to achieve environmental perception and understanding by allowing robots to autonomously navigate unknown environments while simultaneously building maps and estimating their own positions in real time.

[0003] Traditional SLAM systems typically build maps and estimate the robot's pose based on geometric information, such as lidar or visual sensors. However, with the development of deep learning and computer vision, SLAM systems based on semantic information have gradually gained attention. Compared to traditional SLAM systems, semantic-based SLAM systems can leverage the semantic information in maps, providing a richer and deeper understanding of the robot's environment.

[0004] Aiming at the limitations of traditional SLAM systems, this paper proposes a multi-level semantic perception map construction method for distributed SLAM systems. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-level semantic perception map construction method for distributed SLAM systems, combining the advantages of geometric information and deep learning technology of traditional SLAM systems to achieve efficient perception and modeling of the environment.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multi-level semantic perception map construction method for a distributed SLAM system includes:

[0008] Collecting location information, and obtaining candidate matching frames based on the location information;

[0009] Extracting feature values ​​from the candidate matching frames, generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data;

[0010] constructing a probabilistic 3D occupancy grid map based on the processed point cloud data, performing plane segmentation on the probabilistic 3D occupancy grid map, and constructing a 2D grid map;

[0011] Inputting the 2D grid image into a convolutional neural network model to perform image depth estimation and semantic segmentation to obtain depth information and semantic information;

[0012] Fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data;

[0013] Based on the three-dimensional semantic point cloud data and the camera posture, a three-dimensional semantic map is obtained and optimized.

[0014] Optionally, obtaining a candidate matching frame based on the location information includes:

[0015] Acquiring location information, wherein the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR;

[0016] Establishing a point cloud coordinate system according to the posture of the sensor on each node, generating a histogram based on the reflection intensity and the point cloud coordinate system, and constructing a DELIGHT descriptor;

[0017] A similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

[0018] Optionally, obtaining processed point cloud data includes:

[0019] Segment matching using a fast nearest neighbor search in the space described by the segment using the number of neighbors and a distance threshold based on the FLANN tree;

[0020] Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, namely the processed point cloud data.

[0021] Optionally, constructing a probabilistic 3D occupancy grid map includes:

[0022] Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane;

[0023] The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct the probabilistic 3D occupancy grid map.

[0024] Optionally, constructing a 2D grid graph includes:

[0025] Filtering the probabilistic 3D occupancy grid map along the z-axis to remove points on the ceiling plane;

[0026] The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground, the ground is removed, and the remaining voxels are projected onto the ground to obtain the 2D grid map.

[0027] Optionally, inputting the 2D grid map into a convolutional neural network model for image depth estimation and semantic segmentation includes:

[0028] Inputting the 2D grid image into the input layer for preprocessing;

[0029] The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images;

[0030] Inputting the multi-scale feature image into a decoder layer, merging the multi-scale feature image with the upsampling to obtain a fused feature image;

[0031] Input the fused feature image into the discriminant layer for image depth estimation and semantic segmentation, obtain the fitting result and classification probability of each pixel, and then obtain the depth image and semantic image;

[0032] The depth image and the semantic image are input into an output layer, and the depth image and the semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and the semantic information.

[0033] Optionally, constructing the decoder layer includes: encapsulating a convolutional layer, BatchNorm, and an activation function into a convbnrelu module, wherein the convolutional layer performs a depthwise separable convolution operation;

[0034] The decoder layer is constructed based on several convbnrelu modules.

[0035] Optionally, obtaining the fitting result and classification probability for each pixel includes:

[0036] Performing semantic segmentation on the fused feature image several times using a convolution kernel of a preset size to predict the classification probability of each pixel;

[0037] The depth of field of each pixel of the fused feature image is regressively fitted using the convolution kernel of the preset size to obtain a fitting result for each pixel.

[0038] Optionally, obtaining and optimizing the three-dimensional semantic map includes:

[0039] fusing the depth information and the semantic information into three-dimensional semantic point cloud data, and mapping the point cloud data of each frame to a world coordinate system according to the camera posture to obtain the three-dimensional semantic map;

[0040] The robot's rotation and complete pose transformation are calculated to optimize the three-dimensional semantic graph.

[0041] Optionally, the robots further communicate with each other, including:

[0042] When robot α captures new LiDAR data, the location recognition module generates a DELIGHT descriptor to compress the original point cloud and restore it to the local descriptor tree, where the local descriptor tree is a hierarchical organization of DELIGHT descriptors in the local area;

[0043] The communication message of the descriptor sends data to the robot β in the form of R_MSG, and the position recognition module retrieves the most similar descriptor in the descriptor tree, wherein the descriptor tree is a hierarchical organization form of DELIGHT descriptors;

[0044] Robot β matches the segment description of the node's preset range feature value with the most similar descriptor in the descriptor tree to determine the node's position in the map, and generates relative pose information and sends it to robot α to establish a communication protocol between the robots.

[0045] The beneficial effects of the present invention are:

[0046] The present invention extracts feature values ​​from two different candidate matching frames through a relative pose estimation algorithm, can accurately estimate the camera pose, and apply the feature value matching pairs to map construction, thereby achieving accurate map modeling.

[0047] By adopting a deep semantic fusion convolutional neural network, the depth information and semantic information of the image are fused to generate three-dimensional semantic point cloud data, thereby realizing the construction of multi-level semantic perception maps.

[0048] By fusing depth information and semantic information into three-dimensional semantic point cloud data, it helps improve the map's ability to understand and express the environment, making the map richer and more accurate.

[0049] Implementing inter-robot communication protocols through the position recognition module facilitates information exchange and collaboration between multiple robots, thereby achieving more intelligent and efficient task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 Flowchart of a method for constructing a multi-level semantic perception map for a distributed SLAM system according to an embodiment of the present invention;

[0052] Figure 2 A flowchart for creating a new data association according to an embodiment of the present invention;

[0053] Figure 3 This is a flowchart of completing image depth estimation and semantic segmentation according to an embodiment of the present invention;

[0054] Figure 4 This is a flowchart of obtaining a three-dimensional semantic graph and performing accuracy comparison according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] This embodiment provides a multi-level semantic perception map construction method for a distributed SLAM system, including:

[0058] Collecting location information, and obtaining candidate matching frames based on the location information;

[0059] Extracting feature values ​​from candidate matching frames and generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data;

[0060] Based on the processed point cloud data, a probabilistic 3D occupancy grid map is constructed, and the probabilistic 3D occupancy grid map is plane segmented to construct a 2D grid map;

[0061] The 2D grid map is input into the convolutional neural network model for image depth estimation and semantic segmentation to obtain depth information and semantic information;

[0062] Fuse depth information and semantic information to obtain three-dimensional semantic point cloud data;

[0063] Based on 3D semantic point cloud data and camera pose, a 3D semantic map is obtained and optimized.

[0064] Specifically, this embodiment utilizes a deep semantic fusion convolutional neural network to fuse image depth and semantic information to generate three-dimensional semantic point cloud data, thereby achieving multi-layered semantic perception map construction. By fusing depth and semantic information into three-dimensional semantic point cloud data, the map's ability to understand and express the environment is improved, making the map richer and more accurate. The location recognition module implements inter-robot communication protocols, facilitating information exchange and collaboration between multiple robots, leading to more intelligent and efficient task execution.

[0065] Furthermore, obtaining candidate matching frames based on location information includes:

[0066] Obtaining location information, where the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR;

[0067] Establish a point cloud coordinate system based on the posture of the sensor on each node, generate a histogram based on the reflection intensity and the point cloud coordinate system, and construct a DELIGHT descriptor;

[0068] The similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

[0069] Specifically, the process of collecting location information and performing preprocessing includes: capturing the reflection intensity of each surface point on the map from the surrounding environment through 3D LiDAR, forming a DELIFHT descriptor from the intensity histograms in m non-overlapping bins, selecting points from the original point cloud by using the sensor distance or distance threshold, generating a histogram, and establishing a coordinate system of the point cloud according to the posture of the sensor at each node, and obtaining alternative matching frames by calculating the similarity value and selecting the smaller similarity value.

[0070] The process of constructing the DELIGHT descriptor involves partitioning the space into spherical partitions along the radial, elevation, and azimuth axes. A sphere can be divided into eight bins, depending on the orientation of the three axes. To obtain valid points, sensor distance r1 is used to select points from the original point cloud. Closer points can be selected as the inner sphere using a distance threshold r2. The rest of the valid point cloud is called the outer sphere.

[0071] Furthermore, obtaining processed point cloud data includes:

[0072] Segment matching is performed using a fast nearest neighbor search in the segment description space using the number of neighbors and distance thresholds based on the FLANN tree;

[0073] Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, namely the processed point cloud data.

[0074] Specifically, such as Figure 2 As shown, first, the distance image-based method segments the point cloud into several segments, followed by describing the eigenvalues ​​in the form of vectors. Then, the FLANN tree is used to perform segment matching in the space described by the eigenvalue-based segments using a selected number of neighbors and a distance threshold through a fast nearest neighbor search. Then, a RANSAC-based geometric verification process is used to obtain clusters that meet geometric consistency with a resolution threshold and a minimum cluster size.

[0075] Furthermore, constructing a probabilistic 3D occupancy grid map includes:

[0076] Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane;

[0077] The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct a probabilistic 3D occupancy grid map.

[0078] Specifically, the process of constructing a probabilistic 3D occupancy grid map to represent spatial occupancy includes: first, converting the point cloud image in LIDAR coordinates into camera coordinates and projecting it onto the image plane; then, defining the intrinsic parameters of the camera, mapping a 3D point in camera coordinates to the image plane, and assigning the intensity value of each 2D point on the image plane to the z-axis value of its corresponding 3D point, so that the rasterized image has physical meaning.

[0079] Furthermore, constructing a 2D grid graph includes:

[0080] Filter the probabilistic 3D occupancy grid along the z-axis to remove points on the ceiling plane;

[0081] The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground. The ground plane mirror is removed, and the remaining voxels are projected onto the ground to obtain a 2D grid map.

[0082] Specifically, the process of constructing a 2D grid map includes: first, applying a straight-through filter to the global point cloud map along the z-axis to remove points on the ceiling plane; then, using the RANSAC algorithm to iteratively fit the ground plane to obtain the normal vector and height of the ground, excluding the ground, and projecting the remaining voxels onto the ground to obtain a two-dimensional grid map.

[0083] Furthermore, the 2D grid map is fed into the convolutional neural network model for image depth estimation and semantic segmentation, including:

[0084] Input the 2D grid image into the input layer for preprocessing;

[0085] The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images;

[0086] The multi-scale feature image is input into the decoder layer, and the multi-scale feature image is merged with the upsampling to obtain a fused feature image;

[0087] The fused feature image is input into the discriminant layer for image depth estimation and semantic segmentation, and the fitting result and classification probability of each pixel are obtained, thereby obtaining a depth image and a semantic image;

[0088] The depth image and semantic image are input into the output layer, and the depth image and semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and semantic information.

[0089] Furthermore, constructing the decoder layer includes: encapsulating the convolutional layer, BatchNorm and activation function into a convbnrelu module, wherein the convolutional layer performs a depth-wise separable convolution operation;

[0090] Construct the decoder layer based on several convbnrelu modules.

[0091] Furthermore, obtaining the fitting results and classification probability of each pixel includes:

[0092] Use a convolution kernel of preset size to perform semantic segmentation on the fused feature image several times to predict the classification probability of each pixel;

[0093] The depth of field of each pixel of the fused feature image is regressed and fitted using a convolution kernel of a preset size to obtain the fitting result of each pixel.

[0094] Specifically, such as Figure 3 As shown in the figure, a deep semantic fusion convolutional neural network is designed to complete the two tasks of image depth estimation and semantic segmentation, including: preprocessing the original image in the input layer; the encoder layer mainly performs image downsampling based on the convolutional neural network, encapsulating the convolution operation, BatchNorm and activation function into a convbnrelu module, and the convolution layer uses 1×1 and 3×3 small convolutions for depth-separable convolution operations. Then, three convbnrelu modules are added to a block module in sequence. Each layer in the encoder network accumulates the output value of each block with the output value of the previous block through the residual connection to obtain the input value of the next layer; the decoder layer consists of four layers, which refines the feature image through convolution calculation and merges the multi-scale feature image with the upsampling; next, semantic segmentation and depth estimation are performed in the discriminant layer; semantic segmentation prediction uses a 1×1×n convolution kernel to calculate the output feature image of the decoder network and predict the classification probability of each pixel. Depth prediction uses a 1×1 convolution kernel to regress and fit the depth of field of each pixel and obtain the fitting result.

[0095] The preprocessing process of the original image in the input layer includes: standardizing the image according to the statistical characteristics of natural images, so that the samples of the feature image conform to the standard normal distribution with a mean of 0 and a standard deviation of 1, using the 0th layer in the input layer to reduce the resolution of the feature image, and reducing the number of parameters and shortening the time by linear interpolation of the preprocessed feature image.

[0096] Furthermore, obtaining and optimizing a three-dimensional semantic map includes:

[0097] The depth information and semantic information are fused into 3D semantic point cloud data, and the point cloud data of each frame is mapped to the world coordinate system according to the camera posture to obtain a 3D semantic map;

[0098] Compute the robot's rotation and complete pose transformation to optimize the 3D semantic graph.

[0099] Specifically, the depth information and semantic information are fused into three-dimensional semantic point cloud data. According to the camera posture, the point cloud data of each frame is mapped to the world coordinate system to obtain a three-dimensional semantic map. The MPA and MIoU are used as semantic image accuracy indicators for comparison. The process includes: the output layer restores the semantic image and depth image predicted by the discriminant layer to a 640×480 pixel image with the same resolution as the original image through bilinear interpolation. The image depth information and semantic information are obtained through the above process. The coordinate values ​​(X, Y, Z) of all pixels in the camera coordinate system, each semantic information corresponds to an RGB value, and the depth information and semantic information are integrated to generate point cloud data (X, Y, Z, R, B, G).

[0100] By continuously updating the camera's posture, all point cloud data of each frame image are converted into coordinates in a unified world coordinate system to obtain a three-dimensional semantic map.

[0101] MPA and MIoU are used as semantic image accuracy indicators for comparison: MPA represents the average proportion of correctly classified pixels in each category, and MIoU represents the ratio of the intersection and union of the true value and the predicted value, as shown in formulas (1) and (2).

[0102]

[0103] Among them, p ij represents the j-th pixel point predicted by category i, p ii Represents the i-th type of pixel predicted by the i-th type, p ji It represents the i-th pixel point predicted by the j-th category, and K represents the number of categories.

[0104] The two-stage strategy for distributed graph optimization involves solving the distributed graph optimization problem in two stages: the first stage calculates all the rotations of multiple robots, and the second stage calculates the complete pose transformation. The results of both stages can be shown as follows:

[0105]

[0106] In the first stage, Ar is the matrix associated with the rotations of multiple robots, representing the coefficient matrix associated with the rotation variables during graph optimization. r is the rotation vector, representing the variables (targets) for the multiple robot rotations, and is the rotation value to be estimated through optimization. br is the observation vector associated with the rotations, representing the observed or expected values ​​of the rotations in the input data. By optimizing r, the error between the rotation vector and the observation vector is minimized.

[0107] Phase 2: Ap: The matrix associated with the robot's pose, representing the relationship between the pose (translation and rotation) during the graph optimization process. p: The pose vector, containing the robot's pose information in the world coordinate system. This typically includes translation (x, y, z) and rotation (roll, pitch, yaw). bp: The observation vector associated with the pose, representing the pose information observed during the graph optimization process. By optimizing p, a pose solution matching the observations is obtained.

[0108] Furthermore, robots communicate with each other, including:

[0109] When a robot captures new LiDAR data, the location recognition module generates a global descriptor that compresses the original point cloud and restores it to a local descriptor tree;

[0110] The descriptor communication message sends data to another robot in the form of R_MSG. The position recognition module retrieves the most similar descriptor in the descriptor tree, where the descriptor tree is a hierarchical organization of descriptors.

[0111] Another robot root matches the segment description of the node's preset range feature value with the most similar descriptor in the descriptor tree, and corresponds to the most similar position in the map, and at the same time generates relative pose information and sends it to the robot that sends the description communication message, so that a communication protocol is established between the robots.

[0112] The present invention is further described in detail below:

[0113] like Figure 1 As shown, this embodiment provides a method for constructing a multi-level semantic perception map of a distributed SLAM system, which mainly includes the following steps:

[0114] Step 1: Collect location information and preprocess it:

[0115] The location recognition module explores alternative matches between each robot node, which involve common areas. Alternative matching refers to identifying matching relationships between them by finding alternative local nodes in a common area shared by two or more robots. Traditional location recognition methods attempt to align between all point clouds to find alternative local nodes in the common area of ​​two separate robots. However, these methods always consume a lot of time and computing resources. To implement a real-time SLAM system, the time consumption of the location recognition module should be reduced as much as possible. This embodiment applies an intensity-based global descriptor (called DELIGHT) to reduce time consumption.

[0116] The intensity of each point captured by 3D LiDAR from the surrounding environment is the reflection intensity from the surface. A DELIGHT descriptor is constructed based on the reflection intensity, and then a candidate matching frame is obtained. The location information is the reflection intensity of each surface point captured by LiDAR from the environment.

[0117] The coordinate system of the point cloud is established based on the pose of the sensor at each node. To reduce the estimation error caused by the sensor pose, the following two steps are applied to the entire point cloud: 1) the point cloud is normalized so that the origin of the point cloud is in the same position under the same environment. 2) the positive direction of the coordinate axis is artificially defined in the local coordinate system.

[0118] The establishment of point cloud coordinates provides a standard spatial framework that determines the spatial location of each point. Based on this spatial coordinate, points are assigned to different bins. Each bin collects point cloud data within a specific spatial region and generates a histogram based on the intensity values ​​within that region. In the original DELIGHT descriptor, a local reference frame is obtained through principal component analysis (PCA) of all surrounding points within the support, where axis orientation is ambiguous. There are four possible reference frames corresponding to four different bin sequences, resulting in four different versions of the descriptor. To eliminate ambiguity and define a unique local reference frame, the number of points in two spheres divided by the xz plane is counted, and the larger value is set as the positive y-axis. The positive z-axis can be determined based on the direction of gravity. Therefore, the orientation of each axis in the right-handed system can be determined. This can effectively reduce the estimation error caused by the sensor pose.

[0119] The DELIGHT descriptor consists of intensity histograms in m non-overlapping bins. It divides the space into spherical regions along the radial, elevation, and azimuth axes. A sphere can be divided into eight bins based on the orientation of the three axes. The orientation of the three axes is determined by the constructed point cloud coordinates.

[0120] To obtain valid points, we select points from the original point cloud using the sensor distance r1. Closer points can be selected using the distance threshold r2 as the inner sphere. The rest of the valid point cloud is called the outer sphere. Thus, the points can be divided into two spheres with m = 16 regions.

[0121] In each bin, the intensity value (0-255) is distributed to the slots with equal Ibins. "Equal Ibins" means that the pixel values ​​in the intensity value range (0-255) are evenly distributed to the specified number of bins (slots). The width of each bin (slot) is equal, and a histogram is generated.

[0122] The distance between the histograms of descriptor A and descriptor B of bin i is evaluated by the chi-square test as follows:

[0123]

[0124] in, is the distance between the histograms of descriptor A and descriptor B, Ibins is the number of descriptors, and also the number of columns in the histogram. The descriptor is the columnar part in the histogram, and each column represents an intensity value within a certain range.

[0125] In each bin, the intensity histogram from 1 to Ibins is calculated. The similarity measure of the DELIGHT descriptor is calculated by comparing the descriptors between two nodes. The similarity measure of two descriptors is equal to the average distance of all corresponding histograms. The similarity value can then be calculated. The candidate matching frame is obtained by selecting the smaller similarity value of two independent robot nodes. The similarity value is calculated as follows:

[0126]

[0127] Among them, S AB is the similarity value, and m is the number of spherical partitions.

[0128] Step 2: Extract feature values ​​from two different candidate matching frames based on the relative pose estimation algorithm, and process the feature values ​​based on the point cloud segment matching method based on FLANN nearest neighbor search and RANSAC geometric verification.

[0129] In the previous process, candidate matching frames were obtained. The transformations of these matching frames still need to be calculated to establish accurate associations between the data. Therefore, relative pose estimation is performed to determine the accurate transformation with less time and computational cost.

[0130] like Figure 2As shown in the figure, the point cloud acquired by 3D LiDAR in step 1 is divided into several segments using a range image-based method. These segments are part of the original point cloud and highlight the core information of the point cloud. The segment descriptions based on the eigenvalues ​​are presented in the form of vectors. These segment descriptions can not only represent the segments but also reduce the size of the frame.

[0131] To construct a probabilistic 3D occupancy grid and extract stable features from the rasterized image, it is necessary to remove noise and outliers from the original point cloud. The 3D point cloud recorded by multi-line LIDAR contains circular rings on the ground plane. If the rasterized image contains these rings, the accuracy of feature detection and motion estimation will be affected. Random Sample and Consensus (RANSAC) plane fitting is applied to detect and remove the ground plane, and the remaining point cloud is used for image generation. Specifically, it includes:

[0132] Based on the FLANN tree, segment descriptions are matched by fast nearest neighbor search number and distance threshold, so that the segments of two independent frames can correspond to each other, reducing the estimation error.

[0133] The RANSAC-based geometric verification process processes the segment descriptors after segment matching with a resolution threshold and a minimum cluster size to obtain clusters that meet geometric consistency, that is, point clouds with noise and outliers removed.

[0134] Step 3: Build a probabilistic 3D occupancy grid map:

[0135] By using the camera parameter matrix to project the point cloud with noise and outliers removed onto the image plane, the 3D points in the point cloud are represented by LIDAR coordinates. It is hoped that each 3D point can be projected onto a 2D pixel in the image coordinates with an intensity value. The pinhole camera model is used instead of the image formed by direct rasterization, providing basic data for further plane segmentation.

[0136] Based on the transformation matrix R(I|t), the point cloud image in the LIDAR coordinate is converted to the camera coordinate, which is P for the LIDAR coordinate l Point, use the formula P c =R(I|t)P l , convert it to camera coordinates P c , where I|t represents the rotation or posture relationship between the two coordinate systems, I represents the absolute coordinate system, and t represents time.

[0137] Using the formula P i =KR(I|t)P l =CP l , project the point cloud onto the image plane to form a grayscale image, where K defines the intrinsic parameters of the camera, and P irepresents the projection point on the image plane, and C represents the camera projection matrix that projects the 3D point onto the image plane.

[0138] The z-value of each 3D point is saved to give the rasterized image some physical meaning. To minimize information loss and eigenvalue reprojection errors, the intensity value of each 2D point on the image plane is assigned to the z-axis value of its corresponding 3D point. For a point cloud, there may be several points at the same (x, y) position with different z-axis values. Since the z-axis is used as the optical axis of projection, these points are projected to the same pixel. In this case, the maximum z-value of these points is used as the corresponding pixel intensity on the rasterized image. The reason for using the maximum z-value is that the maximum z-value represents the visible height of these points observed by the LIDAR.

[0139] Step 4: Perform plane segmentation on the 3D occupancy grid map and construct a 2D grid map:

[0140] The 3D occupancy grid constructed in step 3 stores the z-value information for each point in the point cloud. This height information allows accurate segmentation of the point cloud, identifying and removing the floor and ceiling. This point cloud is then projected onto the ground, forming a 2D grid. During this process, the z-values ​​in step 3 support segmentation, helping to ensure that the point cloud's height information accurately reflects the actual scene structure in the 2D projection, as detailed below.

[0141] Apply a pass filter to the global point cloud map along the z-axis to remove points on the ceiling plane.

[0142] Use the RANSAC algorithm to iteratively fit the ground plane, assuming that the ground can be represented as M ground : ax+by+ca+d=0, the initial values ​​of the iteration are a=0, b=0, c=1, d=0, and the normal vector and height of the ground are:

[0143]

[0144] The ground is excluded and the remaining voxels are projected onto the ground to obtain a 2D grid map.

[0145] Step 5: Based on the deep semantic fusion convolutional neural network, image depth estimation and semantic segmentation are completed, as shown in Figure 3 As shown:

[0146] A convolutional neural network model was designed. Its key improvements lie in depthwise separable convolutions and residual connections in the encoding and decoding layers, multi-scale feature fusion, improved upsampling methods, and a multi-task processing design in the discriminant layer. This network model, consisting of an input layer, an encoding layer, a decoding layer, a discriminant layer, and an output layer, employs a unified semantic downsampling and upsampling mechanism for feature extraction and recovery. The encoding layer uses depthwise separable convolutions and residual connections to enhance feature extraction while reducing computational costs. The decoding layer optimizes feature recovery through multi-scale feature fusion and bilinear interpolation upsampling. The discriminant layer performs feature classification, probability estimation, and deep regression based on different task requirements, making the model more accurate and efficient when handling multiple tasks.

[0147] Step 5-1, preprocess the two-dimensional grid image in the input layer:

[0148] According to the statistical characteristics of natural images, except for remote sensing images or medical images, the mean of each channel is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225) for each channel.

[0149] The image is standardized so that the samples of the feature image conform to the standard normal distribution with a mean of 0 and a standard deviation of 1.

[0150] Considering that the input image resolution is too high, which affects the number of inference calculations and the speed of the model, the input layer uses layer 0 to reduce the resolution of the feature image and linearly interpolate the preprocessed feature image to reduce the number of parameters and shorten the time.

[0151] In step 5-2, the encoder layer mainly performs image downsampling based on convolutional neural networks:

[0152] The operator fusion method is adopted to encapsulate the convolution operation, BatchNorm and activation function into a convbnrelu module.

[0153] In order to prevent gradient explosion and gradient disappearance in the network, the convolution layer uses 1×1 and 3×3 small convolutions for depth-separable convolution operations.

[0154] Then, three ConvBnReLu modules are added sequentially into one module.

[0155] To facilitate network optimization, each layer in the encoder network accumulates the output value of each block with the output value of the previous block through residual connections to obtain the input value of the next layer.

[0156] In step 5-3, the decoder layer consists of four layers, which refines the feature image through convolution calculation and merges the multi-scale feature image with upsampling.

[0157] The 8th layer uses convolution kernels of size 1 and maximum pooling of size 5 to perform operations alternately to obtain feature images.

[0158] Then, the feature image is restored to the same size as that of the 6th and 5th layers through bilinear interpolation upsampling calculation.

[0159] The 9th layer fuses the three feature images of the 5th, 6th and 8th layers by adding corresponding pixels.

[0160] The 10th layer has the same design concept as the 8th layer. First, the feature image is calculated by bilinear interpolation upsampling to obtain a feature image of the same size as the 4th layer.

[0161] Then, the feature images of the 10th layer and the 4th layer are fused by adding the corresponding pixels. Considering that the size of the feature image restored by the 11th layer is the same as that of the 3rd layer, that is, the scale is reduced by half compared with the original image, the maximum pooling calculation is too large and the time is too long. Therefore, a 1×1 convolution kernel is selected for convolution calculation in the 11th layer.

[0162] Then the feature images of the 3rd and 10th layers are fused by pixel addition.

[0163] Step 5-4, perform semantic segmentation and depth estimation in the discriminant layer:

[0164] Considering that semantic segmentation needs to be performed n times, semantic segmentation prediction uses a 1×1×n convolution kernel to calculate the output feature image of the decoder network and predict the classification probability of each pixel.

[0165] Depth prediction uses a 1×1 convolution kernel to perform regression fitting on the depth of field of each pixel and obtain the fitting result.

[0166] Step 6: Fusion the depth information and semantic information into 3D semantic point cloud data. According to the camera pose, the point cloud data of each frame is mapped to the world coordinate system to obtain a 3D semantic map. MPA and MIoU are used as semantic image accuracy indicators for comparison, as shown in the following example: Figure 4 As shown:

[0167] The output layer restores the semantic image and depth image predicted by the discriminant layer to an image of 640×480 pixels with the same resolution as the original image through bilinear interpolation.

[0168] Obtaining image depth information helps to obtain the coordinates (X, Y, Z) of all pixels in the camera coordinate system, as well as the semantic information of all pixels that can be obtained.

[0169] Each semantic information corresponds to an RGB value. The depth information and semantic information are integrated to generate point cloud data (X, Y, Z, R, B, G).

[0170] By continuously updating the camera's posture, all point cloud data of each frame image are converted into coordinates in a unified world coordinate system to obtain a three-dimensional semantic map.

[0171] MPA and MIoU are used as semantic image accuracy indicators for comparison.

[0172] Step 7: Optimize the 3D semantic map based on a two-stage strategy:

[0173] The first stage calculates the rotations of all multiple robots.

[0174] The second stage computes the complete pose transformation.

[0175] After calculating the rotation matrices and pose transformations of multiple robots, global optimization is used to further improve the accuracy of map construction.

[0176] First, the relative rotation angles between multiple robots are estimated through eigenvalue matching and reprojection methods, generating their respective rotation matrices. A rotation matrix is ​​a matrix used to represent the orientation change of a robot in three-dimensional space.

[0177] After calculating the rotation matrix, we further calculate the pose matrix. The pose matrix includes translation and rotation information, representing the robot's position (translation vector) and orientation (rotation matrix) in the world coordinate system. The pose matrix can fully describe the robot's posture in space.

[0178] After multiple robots have calculated their pose and rotation matrices, graph optimization techniques are used to treat the pose matrices of each frame as nodes in a graph, and the pose constraints between frames as edges. Graph optimization minimizes the error between all nodes and edges, reducing the impact of error accumulation on map construction.

[0179] To further improve the map's geometric accuracy, bundle adjustment jointly optimizes the pose matrices and 3D point coordinates for all frames. This optimization process simultaneously corrects the relative position and rotation of each frame, ensuring geometric consistency across multiple frames.

[0180] In addition, loop closure detection is used to identify when a robot returns to an area it has already visited, reducing posture drift and improving positioning accuracy. The optimization process also combines semantic information with geometric information, jointly optimizing through metrics such as MPA and MIoU to ensure that the map is not only geometrically accurate but also semantically accurate. Finally, local optimization is used to reduce the amount of computation in large-scale scenarios, and when positioning errors are detected, the posture is further adjusted through a relocalization algorithm to ensure real-time and robustness in multi-robot collaboration. Geometric information refers to the spatial structure of the environment, represented by point cloud data obtained by LiDAR. It is geometrically verified using algorithms such as RANSAC, from which geometrically consistent clusters are extracted.

[0181] Step 8: Implement the communication protocol between robots based on the position recognition module:

[0182] When a robot captures new LiDAR data, the location recognition module generates a Delight descriptor to compress the raw point cloud and restore it to a local descriptor tree. A local descriptor tree is a hierarchical organization of Delight descriptors within a local region, typically used to store and match information about a specific robot's local environment. The local descriptor tree is a portion of the descriptor tree used to efficiently search and match descriptors within a local area, enabling rapid localization and optimized mapping of the local environment.

[0183] The communication message of the DELIGHT descriptor is in the form of R_MSG to send data to another robot.

[0184] When another robot β receives a PR_MSG from its neighboring robot α, the PR (place recognition module) is triggered to retrieve the most similar descriptor in the descriptor tree. The descriptor tree is a hierarchical organization of DELIGHT descriptors. It uses the FLANN (Fast Nearest Neighbor Search) algorithm to arrange descriptors by distance or feature value and quickly search for matching descriptors. Global descriptors acquire and compress raw point cloud data using 3D LiDAR and then store it in the descriptor tree for matching and communication.

[0185] Another robot, β, matches the segment descriptions of the feature values ​​surrounding the node with the most similar descriptors in the descriptor tree using a similarity metric to determine the corresponding node's location in the map. Simultaneously, it encapsulates a relative pose message (RP_MSG) and sends it. When robot α receives the RP_MSG, the relative pose information in the RP_MSG is calculated based on feature value matching. Robot α still needs to perform a secondary calculation to ensure that the relative pose is consistent with its own environmental perception. Therefore, it extracts the corresponding feature value-based segment descriptions. The relative pose module (RP) uses the descriptors to calculate the relative pose and establish new inter-data associations. To generate a complete and accurate global map from multiple robots, the DGO module is used to obtain multiple trajectory estimates. Each robot estimates its local trajectory using only measurements involving the poses of boundary nodes shared with other robots. Once new inter-data associations are established, the distributed graph optimization (DGO) module is triggered to update the map by sharing the poses of boundary nodes multiple times, which are then propagated as DGO_MSGs. If multiple robots fail to capture data, the entire system sends a FIN message during this loop to terminate communication. At the end of each communication cycle, each robot maintains a globally consistent state. The node is the specific location in the map where the robot collects environmental information.

[0186] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A multi-level semantic perception map construction method for a distributed SLAM system, characterized in that: include: Collecting location information, and obtaining candidate matching frames based on the location information; Extracting feature values ​​from the candidate matching frames, generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data; constructing a probabilistic 3D occupancy grid map based on the processed point cloud data, performing plane segmentation on the probabilistic 3D occupancy grid map, and constructing a 2D grid map; Inputting the 2D grid image into a convolutional neural network model to perform image depth estimation and semantic segmentation to obtain depth information and semantic information; Fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data; Based on the three-dimensional semantic point cloud data and the camera posture, a three-dimensional semantic map is obtained and optimized; Obtaining the three-dimensional semantic graph and optimizing it includes: fusing the depth information and the semantic information into three-dimensional semantic point cloud data, and mapping the point cloud data of each frame to a world coordinate system according to the camera posture to obtain the three-dimensional semantic map; Calculating the rotation and complete pose transformation of the robot and optimizing the three-dimensional semantic graph; The robots also communicate with each other, including: When robot α captures new LiDAR data, the location recognition module generates a DELIGHT descriptor to compress the original point cloud and restore it to the local descriptor tree, where the local descriptor tree is a hierarchical organization of DELIGHT descriptors in the local area; The communication message of the descriptor sends data to the robot β in the form of R_MSG, and the position recognition module retrieves the most similar descriptor in the descriptor tree, wherein the descriptor tree is a hierarchical organization form of DELIGHT descriptors; Robot β matches the segment description of the node's preset range feature value with the most similar descriptor in the descriptor tree to determine the node's position in the map, and generates relative pose information and sends it to robot α to establish a communication protocol between the robots.

2. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Acquiring candidate matching frames based on the location information includes: Acquiring location information, wherein the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR; Establishing a point cloud coordinate system according to the posture of the sensor on each node, generating a histogram based on the reflection intensity and the point cloud coordinate system, and constructing a DELIGHT descriptor; A similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

3. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Obtaining processed point cloud data includes: Segment matching using a fast nearest neighbor search in the space described by the segment using the number of neighbors and a distance threshold based on the FLANN tree; Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, namely the processed point cloud data.

4. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Constructing a probabilistic 3D occupancy grid map involves: Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane; The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct the probabilistic 3D occupancy grid map.

5. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 4 is characterized in that: Constructing a 2D grid graph involves: Filtering the probabilistic 3D occupancy grid map along the z-axis to remove points on the ceiling plane; The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground, the ground is removed, and the remaining voxels are projected onto the ground to obtain the 2D grid map.

6. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Inputting the 2D grid map into a convolutional neural network model for image depth estimation and semantic segmentation includes: Inputting the 2D grid image into the input layer for preprocessing; The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images; Inputting the multi-scale feature image into a decoder layer, merging the multi-scale feature image with the upsampling to obtain a fused feature image; Input the fused feature image into the discriminant layer for image depth estimation and semantic segmentation, obtain the fitting result and classification probability of each pixel, and then obtain the depth image and semantic image; The depth image and the semantic image are input into an output layer, and the depth image and the semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and the semantic information.

7. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 6 is characterized in that: Constructing the decoder layer includes: encapsulating the convolution layer, BatchNorm and activation function into a convbnrelu module, wherein the convolution layer performs a depth-wise separable convolution operation; The decoder layer is constructed based on several convbnrelu modules.

8. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 6 is characterized in that: Obtaining the fitting results and classification probability for each pixel includes: Performing semantic segmentation on the fused feature image several times using a convolution kernel of a preset size to predict the classification probability of each pixel; The depth of field of each pixel of the fused feature image is regressively fitted using the convolution kernel of the preset size to obtain a fitting result for each pixel.

Images