Multi-level semantic perception map construction method for distributed slam system

Abstract

The invention relates to a multilevel semantic perception map construction method for a distributed slam system, and the method comprises the steps: collecting site information, and obtaining an alternative matching frame based on the site information; extracting a feature value from the alternative matching frame, generating a vector as a segment description of the feature value, processing the segment description, and obtaining processed point cloud data; based on the processed point cloud data, constructing a probabilistic 3D occupancy grid map, performing plane segmentation on the probabilistic 3D occupancy grid map, and constructing a 2D grid map; inputting the 2D grid graph into a convolutional neural network model for image depth estimation and semantic segmentation to obtain depth information and semantic information; fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data; and based on the three-dimensional semantic point cloud data and the camera attitude, obtaining and optimizing a three-dimensional semantic graph. According to the method, the advantages of geometric information of a traditional slam system and the deep learning technology are combined, and efficient sensing and modeling of the environment are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction methods for SLAM systems, and in particular to a multi-level semantic perception map construction method for distributed SLAM systems. Background Art

[0002] In recent years, with the rapid development of mobile robot technology, distributed as a key autonomous navigation and environmental perception technology has become one of the hot research directions in the field of autonomous mobile robots. Slam technology aims to achieve perception and understanding of the environment by autonomously moving the robot in an unknown environment while building a map and estimating the robot's own position in real time.

[0003] Traditional SLAM systems usually build maps and estimate the robot's posture based on geometric information, such as lidar or visual sensors. However, with the development of deep learning and computer vision, SLAM systems based on semantic information have gradually attracted attention. Compared with traditional SLAM systems, SLAM systems based on semantic information can utilize semantic information in maps to make the robot's understanding of the environment richer and deeper.

[0004] Aiming at the limitations of traditional SLAM systems, the present invention proposes a multi-level semantic perception map construction method for distributed SLAM systems. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-level semantic perception map construction method for a distributed SLAM system, combining the advantages of geometric information and deep learning technology of traditional SLAM systems to achieve efficient perception and modeling of the environment.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multi-level semantic perception map construction method for a distributed SLAM system includes:

[0008] Collecting location information, and obtaining candidate matching frames based on the location information;

[0009] Extracting feature values ​​from the candidate matching frames, generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data;

[0010] Based on the processed point cloud data, construct a probabilistic 3D occupancy grid map, perform plane segmentation on the probabilistic 3D occupancy grid map, and construct a 2D grid map;

[0011] Inputting the 2D grid image into a convolutional neural network model to perform image depth estimation and semantic segmentation to obtain depth information and semantic information;

[0012] Fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data;

[0013] Based on the three-dimensional semantic point cloud data and the camera posture, a three-dimensional semantic map is obtained and optimized.

[0014] Optionally, obtaining a candidate matching frame based on the location information includes:

[0015] Acquiring location information, wherein the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR;

[0016] Establishing a point cloud coordinate system according to the posture of the sensor on each node, generating a histogram based on the reflection intensity and the point cloud coordinate system, and constructing a DELIGHT descriptor;

[0017] The similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

[0018] Optionally, obtaining processed point cloud data includes:

[0019] Segment matching based on the FLANN tree by fast nearest neighbor search in the space described by the segment using the number of neighbors and distance threshold;

[0020] Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, that is, the processed point cloud data.

[0021] Optionally, constructing a probabilistic 3D occupancy grid map includes:

[0022] Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane;

[0023] The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct the probabilistic 3D occupancy grid map.

[0024] Optionally, constructing a 2D grid graph includes:

[0025] Filtering the probabilistic 3D occupancy grid map along the z-axis to remove points on the ceiling plane;

[0026] The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground, the ground is removed, and the remaining voxels are projected onto the ground to obtain the 2D grid map.

[0027] Optionally, inputting the 2D grid map into a convolutional neural network model for image depth estimation and semantic segmentation includes:

[0028] Input the 2D grid image into an input layer for preprocessing;

[0029] The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images;

[0030] Inputting the multi-scale feature image into a decoder layer, merging the multi-scale feature image with the upsampling to obtain a fused feature image;

[0031] Input the fused feature image into the discriminant layer for image depth estimation and semantic segmentation, obtain the fitting result and classification probability of each pixel, and then obtain the depth image and semantic image;

[0032] The depth image and the semantic image are input into an output layer, and the depth image and the semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and the semantic information.

[0033] Optionally, constructing the decoder layer includes: encapsulating a convolutional layer, BatchNorm, and an activation function into a convbnrelu module, wherein the convolutional layer performs a depth-separable convolution operation;

[0034] The decoder layer is built based on several convbnrelu modules.

[0035] Optionally, obtaining the fitting result and classification probability of each pixel includes:

[0036] Performing semantic segmentation on the fused feature image several times using a convolution kernel of a preset size to predict the classification probability of each pixel;

[0037] The depth of field of each pixel of the fused feature image is regressively fitted using the convolution kernel of the preset size to obtain a fitting result of each pixel.

[0038] Optionally, obtaining and optimizing the three-dimensional semantic map includes:

[0039] The depth information and the semantic information are merged into three-dimensional semantic point cloud data, and the point cloud data of each frame is mapped to a world coordinate system according to the camera posture to obtain the three-dimensional semantic map;

[0040] The rotation and complete posture transformation of the robot are calculated, and the three-dimensional semantic graph is optimized.

[0041] Optionally, the robots further communicate with each other, including:

[0042] When robot α captures new LiDAR data, the location recognition module generates a DELIGHT descriptor to compress the original point cloud and restore it to the local descriptor tree, where the local descriptor tree is a hierarchical organization of DELIGHT descriptors in the local area;

[0043] The communication message of the descriptor sends data to the robot β in the form of R_MSG, and the position recognition module retrieves the most similar descriptor in the descriptor tree, wherein the descriptor tree is a hierarchical organization form of DELIGHT descriptors;

[0044] Robot β matches the segment description of the node's preset range feature value with the most similar descriptor in the descriptor tree to determine the node's position in the map, and generates relative pose information and sends it to robot α to establish a communication protocol between the robots.

[0045] The beneficial effects of the present invention are:

[0046] The present invention extracts feature values ​​from two different candidate matching frames through a relative posture estimation algorithm, can accurately estimate the posture of the camera, and applies the feature value matching pairs to map construction, thereby achieving accurate map modeling.

[0047] By adopting a deep semantic fusion convolutional neural network, the depth information and semantic information of the image are fused to generate three-dimensional semantic point cloud data, thereby realizing the construction of a multi-level semantic perception map.

[0048] By fusing depth information and semantic information into three-dimensional semantic point cloud data, it helps to improve the map's ability to understand and express the environment, making the map richer and more accurate.

[0049] Implementing the communication protocol between robots through the position recognition module facilitates information exchange and cooperation between multiple robots, thereby achieving more intelligent and efficient task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for constructing a multi-level semantic perception map for a distributed SLAM system according to an embodiment of the present invention;

[0052] Figure 2 A flowchart for creating a new data association according to an embodiment of the present invention;

[0053] Figure 3 A flowchart of completing image depth estimation and semantic segmentation according to an embodiment of the present invention;

[0054] Figure 4 The present invention is a flowchart of obtaining a three-dimensional semantic graph and performing accuracy comparison according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] This embodiment provides a multi-level semantic perception map construction method for a distributed SLAM system, including:

[0058] Collecting location information, and obtaining candidate matching frames based on the location information;

[0059] Extracting feature values ​​from candidate matching frames, generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data;

[0060] Based on the processed point cloud data, a probabilistic 3D occupancy grid map is constructed, and the probabilistic 3D occupancy grid map is plane segmented to construct a 2D grid map;

[0061] The 2D grid image is input into the convolutional neural network model for image depth estimation and semantic segmentation to obtain depth information and semantic information;

[0062] Fuse depth information and semantic information to obtain three-dimensional semantic point cloud data;

[0063] Based on the 3D semantic point cloud data and camera pose, the 3D semantic map is obtained and optimized.

[0064] Specifically, this embodiment uses a deep semantic fusion convolutional neural network to fuse the depth information and semantic information of the image to generate three-dimensional semantic point cloud data, thereby realizing multi-level semantic perception map construction. By fusing the depth information and semantic information into three-dimensional semantic point cloud data, it helps to improve the map's understanding and expression of the environment, making the map richer and more accurate. The communication protocol between robots is realized through the location recognition module, which helps multiple robots to exchange information and cooperate, thereby realizing more intelligent and efficient task execution.

[0065] Furthermore, obtaining candidate matching frames based on location information includes:

[0066] Obtaining location information, wherein the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR;

[0067] Establish a point cloud coordinate system according to the posture of the sensor on each node, generate a histogram based on the reflection intensity and the point cloud coordinate system, and construct a DELIGHT descriptor;

[0068] The similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

[0069] Specifically, the process of collecting location information and preprocessing includes: capturing the reflection intensity of each surface point on the map from the surrounding environment through 3D LiDAR, forming a DELIFHT descriptor from the intensity histograms in m non-overlapping boxes, selecting points from the original point cloud by using the sensor distance or distance threshold, generating a histogram, and establishing a coordinate system of the point cloud according to the posture of the sensor at each node, and obtaining alternative matching frames by calculating the similarity value and selecting the smaller similarity value.

[0070] The process of constructing the DELIGHT descriptor includes: the space is divided by spherical partitions along the radial, elevation and azimuth axes. According to the directions of the three axes, a sphere can be divided into 8 bins. To obtain valid points, points are selected from the original point cloud using the sensor distance r1. Points with closer distances can be selected by the distance threshold r2 as the inner sphere. The rest of the valid point cloud is called the outer sphere.

[0071] Furthermore, obtaining the processed point cloud data includes:

[0072] Segment matching is performed based on the number of neighbors and distance thresholds in the space of segment descriptions through fast nearest neighbor search based on the FLANN tree;

[0073] Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, that is, the processed point cloud data.

[0074] Specifically, Figure 2 As shown, first, the distance image-based method segments the point cloud into several segments, next the eigenvalues ​​are described in the form of vectors, and then the FLANN tree is used to perform segment matching in the space described by the eigenvalue-based segments using the selected number of neighbors and distance threshold through a fast nearest neighbor search, and then the RANSAC-based geometric verification process obtains clusters that meet geometric consistency with a resolution threshold and a minimum cluster size.

[0075] Furthermore, constructing a probabilistic 3D occupancy grid map includes:

[0076] Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane;

[0077] The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct a probabilistic 3D occupancy grid map.

[0078] Specifically, the process of constructing a probabilistic 3D occupancy grid map to represent space occupancy includes: first, converting the point cloud image in LIDAR coordinates into camera coordinates and projecting it onto the image plane; then defining the intrinsic parameters of the camera, mapping a 3D point in the camera coordinates to the image plane, and assigning the intensity value of each 2D point on the image plane to the z-axis value of the corresponding 3D point, so that the rasterized image has physical meaning.

[0079] Furthermore, constructing a 2D grid graph includes:

[0080] Filter the probabilistic 3D occupancy grid map along the z-axis to remove points on the ceiling plane;

[0081] The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground, remove the ground plane mirror, and project the remaining voxels onto the ground to obtain a 2D grid map.

[0082] Specifically, the process of constructing a 2D grid map includes: first, applying a pass filter to the global point cloud map along the z-axis to remove points on the ceiling plane; then, using the RANSAC algorithm to iteratively fit the ground plane to obtain the normal vector and height of the ground, excluding the ground, and projecting the remaining voxels onto the ground to obtain a two-dimensional grid map.

[0083] Furthermore, the 2D grid map is input into the convolutional neural network model for image depth estimation and semantic segmentation including:

[0084] Input the 2D grid image into the input layer for preprocessing;

[0085] The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images;

[0086] Input the multi-scale feature image into the decoder layer, merge the multi-scale feature image with the upsampling to obtain a fused feature image;

[0087] The fused feature image is input into the discriminant layer for image depth estimation and semantic segmentation, and the fitting result and classification probability of each pixel are obtained, thereby obtaining a depth image and a semantic image;

[0088] The depth image and semantic image are input into the output layer, and the depth image and semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and semantic information.

[0089] Furthermore, constructing the decoder layer includes: encapsulating the convolutional layer, BatchNorm and the activation function into a convbnrelu module, wherein the convolutional layer performs a depth-wise separable convolution operation;

[0090] The decoder layer is built based on several convbnrelu modules.

[0091] Furthermore, obtaining the fitting result and classification probability of each pixel includes:

[0092] Use a convolution kernel of preset size to perform semantic segmentation on the fused feature image several times to predict the classification probability of each pixel;

[0093] The depth of field of each pixel of the fused feature image is regressed and fitted using a convolution kernel of a preset size to obtain a fitting result for each pixel.

[0094] Specifically, Figure 3 As shown in the figure, a deep semantic fusion convolutional neural network is designed to complete the two tasks of image depth estimation and semantic segmentation, including: preprocessing the original image in the input layer; the encoder layer mainly performs image downsampling based on the convolutional neural network, encapsulates the convolution operation, BatchNorm and activation function into a convbnrelu module, and the convolution layer uses 1×1 and 3×3 small convolutions for depth-separable convolution operations. Then, three convbnrelu modules are added to a block module in sequence. Each layer in the encoder network accumulates the output value of each block with the output value of the previous block through the residual connection to obtain the input value of the next layer; the decoder layer consists of four layers, which refines the feature image through convolution calculation and merges the multi-scale feature image with the upsampling; next, semantic segmentation and depth estimation are performed in the discriminant layer; the semantic segmentation prediction uses a 1×1×n convolution kernel to calculate the output feature image of the decoder network and predict the classification probability of each pixel. The depth prediction uses a 1×1 convolution kernel to regress and fit the depth of field of each pixel and obtain the fitting result.

[0095] The preprocessing process of the original image in the input layer includes: standardizing the image according to the statistical characteristics of natural images, the samples of the feature image conform to the standard normal distribution with a mean of 0 and a standard deviation of 1, using the 0th layer in the input layer to reduce the resolution of the feature image, and reducing the number of parameters and shortening the time by linear interpolation of the preprocessed feature image.

[0096] Furthermore, obtaining and optimizing the three-dimensional semantic graph includes:

[0097] The depth information and semantic information are fused into 3D semantic point cloud data, and the point cloud data of each frame is mapped to the world coordinate system according to the camera posture to obtain a 3D semantic map;

[0098] Calculate the robot's rotation and complete pose transformation to optimize the 3D semantic graph.

[0099] Specifically, the depth information and semantic information are fused into three-dimensional semantic point cloud data. According to the camera posture, the point cloud data of each frame is mapped to the world coordinate system to obtain a three-dimensional semantic map. The process of using MPA and MIoU as semantic image accuracy indicators for comparison includes: the output layer restores the semantic image and depth image predicted by the discriminant layer to an image of 640×480 pixels with the same resolution as the original image through bilinear interpolation. The image depth information and semantic information are obtained through the above process. The coordinate values ​​(X, Y, Z) of all pixels in the camera coordinate system, each semantic information corresponds to an RGB value, and the depth information and semantic information are integrated to generate point cloud data (X, Y, Z, R, B, G).

[0100] By continuously updating the camera's posture, all point cloud data of each frame image are converted into coordinates in a unified world coordinate system to obtain a three-dimensional semantic map.

[0101] MPA and MIoU are used as semantic image accuracy indicators for comparison: MPA represents the average proportion of correctly classified pixels in each category, and MIoU represents the ratio of the intersection and union of the true value and the predicted value, as shown in formulas (1) and (2).

[0102]

[0103] Among them, p ij represents the j-th pixel predicted by category i, p ii represents the i-th type of pixel predicted by the i-th type, p ji represents the i-th pixel of the j-th category prediction, and K represents the number of categories.

[0104] The two-stage strategy for distributed graph optimization includes: using a two-stage strategy to solve the distributed graph optimization problem, the first stage calculates the rotation of all multiple robots, and the second stage calculates the complete pose transformation. The results of both stages can be shown as follows:

[0105]

[0106] Among them, the first stage: Ar: the matrix related to the rotation of multiple robots, which represents the coefficient matrix related to the rotation variable in the graph optimization process. r: the rotation vector, which represents the variable (target) of the rotation of multiple robots, is the rotation value to be estimated through optimization. br: the observation vector related to the rotation, which represents the observation or expected value related to the rotation in the input data. By optimizing r, the error between the rotation vector and the observation vector is minimized.

[0107] Phase 2: Ap: A matrix related to the robot’s posture, which represents the relationship with the posture (translation and rotation) during the graph optimization process. p: A posture vector, which includes the posture information of the robot in the world coordinate system. It usually includes information such as translation (x, y, z) and rotation (roll, pitch, yaw). bp: An observation vector related to the posture, which represents the posture information observed during the graph optimization process. By optimizing p, a posture solution matching the observation is obtained.

[0108] Furthermore, the robots communicate with each other, including:

[0109] When a robot captures new LiDAR data, the location recognition module generates a global descriptor that compresses the raw point cloud and restores it to a local descriptor tree;

[0110] The communication message of the descriptor sends data to another robot in the form of R_MSG. The position recognition module retrieves the most similar descriptor in the descriptor tree, where the descriptor tree is a hierarchical organization of descriptors.

[0111] Another robot root matches the segment description of the node preset range feature value with the most similar descriptor in the descriptor tree, corresponding to the most similar position in the map, and generates relative pose information to send to the robot that sends the description communication message, so that a communication protocol is established between the robots.

[0112] The present invention is further described in detail below:

[0113] like Figure 1 As shown, this embodiment provides a method for constructing a multi-level semantic perception map of a distributed SLAM system, which mainly includes the following steps:

[0114] Step 1: Collect location information and preprocess it:

[0115] The location recognition module explores alternative matches between each robot node that involve common areas. Alternative matching refers to identifying matching relationships between two or more robots by finding alternative local nodes in a common area shared by them. Traditional location recognition methods attempt to align between all point clouds to find alternative local nodes in the common area of ​​two separate robots. However, these methods always consume a lot of time and computing resources. To implement a real-time SLAM system, the time consumption of the location recognition module should be reduced as much as possible. This embodiment applies an intensity-based global descriptor (called DELIGHT) to reduce time consumption.

[0116] The intensity of each point captured from the surrounding environment by 3D LiDAR is the reflection intensity from the surface. A DELIGHT descriptor is constructed based on the reflection intensity to obtain an alternative matching frame. Among them, the location information is the reflection intensity of each surface point captured by LiDAR from the environment.

[0117] The coordinate system of the point cloud is established according to the attitude of the sensor at each node. In order to reduce the estimation error caused by the sensor attitude, the following two steps are applied to the entire point cloud: 1) Normalize the point cloud so that the origin of the point cloud is in the same position in the same environment. 2) Artificially define the positive direction of the coordinate axis in the local coordinate system.

[0118] The establishment of point cloud coordinates provides a standard spatial frame that determines the spatial position of each point. Based on this spatial coordinate, points are assigned to different bins, each of which collects point cloud data in a specific spatial region and generates a histogram based on the intensity values ​​within the region. In the original DELIGHT descriptor, the local reference frame is obtained by principal component analysis (PCA) of all surrounding points within the support, where there is ambiguity in the axis direction. There are 4 possible reference frames corresponding to 4 different bin sequences, resulting in 4 different versions of the descriptor. In order to eliminate ambiguity and define a unique local reference frame, the number of points in the two spheres divided by the xz plane is calculated, and the larger value is set to the positive y axis. The positive z axis can be determined based on the direction of gravity. Therefore, the direction of each axis in the right-handed system can be determined. The estimation error caused by the sensor attitude can be effectively reduced.

[0119] The DELIGHT descriptor consists of intensity histograms in m non-overlapping bins. The space is divided into spherical regions along the radial, elevation and azimuth axes. A sphere can be divided into 8 bins according to the directions of the three axes. The directions of the three axes are determined by the constructed point cloud coordinates.

[0120] To obtain valid points, points are selected from the original point cloud using the sensor distance r1. Points with a closer distance can be selected by the distance threshold r2 as the inner sphere. The rest of the valid point cloud is called the outer sphere. Therefore, the points can be divided into two spheres with m = 16 regions.

[0121] In each bin, the intensity value (0-255) is distributed to the slots with equal Ibins. "Equal Ibins" means that the pixel values ​​in the intensity value range (0-255) are evenly distributed to the specified number of bins (slots), and the width of each bin (slot) is equal, and a histogram is generated.

[0122] The distance between the histograms of descriptor A and descriptor B of bin i is evaluated by the chi-square test as follows:

[0123]

[0124] in, is the distance between the histograms of descriptor A and descriptor B, Ibins is the number of descriptors, and also the number of columns in the histogram. The descriptor is the columnar part in the histogram, and each column represents an intensity value within a certain range.

[0125] In each bin, the intensity histogram from 1 to Ibins is calculated. The similarity measure of the DELIGHT descriptor is calculated by comparing the descriptors between two nodes. The similarity measure of two descriptors is equal to the average distance of all corresponding histograms. Then the similarity value can be calculated. The candidate matching frame is obtained by selecting the smaller similarity value of two independent robot nodes. The similarity value is calculated as follows:

[0126]

[0127] Among them, S AB is the similarity value, and m is the number of spherical partitions.

[0128] Step 2: Extract feature values ​​from two different candidate matching frames based on the relative pose estimation algorithm, and process the feature values ​​based on the point cloud segment matching method based on FLANN nearest neighbor search and RANSAC geometric verification.

[0129] In the previous process, candidate matching frames are obtained. The transformations of these matching frames still need to be calculated to establish accurate data associations. Therefore, relative pose estimation is performed to determine the accurate transformation with less time and computational cost.

[0130] like Figure 2As shown in the figure, the point cloud acquired by 3D LiDAR in step 1 is divided into several segments based on the distance image. These segments are part of the original point cloud and highlight the core information of the point cloud. The segment description based on the eigenvalue is presented in the form of vectors. These segment descriptions can not only represent the fragments but also reduce the size of the frame.

[0131] In order to construct a probabilistic 3D occupancy grid map and extract stable features from the rasterized image, it is necessary to remove noise and outliers from the original point cloud. The 3D point cloud recorded by the multi-line LIDAR contains circular rings on the ground plane. If the rasterized image contains these rings, it will affect the accuracy of feature detection and motion estimation. Random Samples and Consensus (RANSAC) plane fitting is applied to detect and remove the ground plane, and the remaining point cloud is used for image generation. Specifically, it includes:

[0132] Based on the FLANN tree, segment descriptions are matched by fast nearest neighbor search quantity and distance threshold, so that the segments of two independent frames can correspond to each other, reducing the estimation error.

[0133] The RANSAC-based geometric verification process processes the segment descriptors after segment matching with a resolution threshold and a minimum cluster size to obtain clusters that meet geometric consistency, that is, point clouds with noise and outliers removed.

[0134] Step 3: Build a probabilistic 3D occupancy grid map:

[0135] The point cloud with noise and outliers removed is projected onto the image plane using the camera parameter matrix. The 3D points in the point cloud are represented by LIDAR coordinates. It is hoped that each 3D point can be projected onto a 2D pixel in the image coordinates with an intensity value. The pinhole camera model is used instead of the image formed by direct rasterization, which provides basic data for further plane segmentation.

[0136] Based on the transformation matrix R(I|t), the point cloud image in the LIDAR coordinate is converted to the camera coordinate. For the LIDAR coordinate, it is P l Point, using the formula P c =R(I|t)P l , convert it to camera coordinates P c , where I|t represents the rotation or posture relationship between the two coordinate systems, I represents the absolute coordinate system, and t represents time.

[0137] Using the formula P i =KR(I|t)P l =CP l , projecting the point cloud onto the image plane to form a grayscale image, where K defines the intrinsic parameters of the camera, and P irepresents the projection point on the image plane, and C represents the camera projection matrix that projects the 3D point onto the image plane.

[0138] The z value of each 3D point is saved so that the rasterized image has some physical meaning. In order to minimize the loss of information and the reprojection error of the eigenvalues, the intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point; for a point cloud, there may be several points at the same (x, y) position with different z-axis values. Since the z-axis is used as the optical axis of the projection, these points are projected to the same pixel. In this case, the maximum z value of these points is used as the corresponding pixel intensity on the rasterized image. The reason for using the maximum z value is that the maximum z value represents the visible height of these points observed by the LIDAR.

[0139] Step 4: perform plane segmentation on the 3D occupancy grid map and construct a 2D grid map:

[0140] The 3D occupancy grid constructed in step 3 saves the z value information of each point in the point cloud. Through the height information, the point cloud can be accurately segmented, and the ground and ceiling can be identified and removed. These point clouds are projected onto the ground to form a two-dimensional grid map. In this process, the z value in step 3 provides support for segmentation, which helps to ensure that the height information of the point cloud can accurately reflect the structure of the actual scene in the two-dimensional projection, as follows.

[0141] Apply a pass filter to the global point cloud map along the z-axis to remove points on the ceiling plane.

[0142] Use the RANSAC algorithm to iteratively fit the ground plane, assuming that the ground can be represented as M ground : ax+by+ca+d=0, the initial values ​​of the iteration are a=0, b=0, c=1, d=0, and the normal vector and height of the ground are obtained as follows:

[0143]

[0144] The ground is excluded and the remaining voxels are projected onto the ground to obtain a 2D grid map.

[0145] Step 5: Based on the deep semantic fusion convolutional neural network, image depth estimation and semantic segmentation are completed, such as Figure 3 As shown:

[0146] A convolutional neural network model was designed. The main improvements of the convolutional neural network are reflected in the deep separable convolution and residual connection of the encoding and decoding layers, multi-scale feature fusion, improvement of upsampling methods, and multi-task processing design of the discriminant layer. The network model consists of an input layer, an encoding layer, a decoding layer, a discriminant layer, and an output layer. It adopts a unified semantic downsampling and upsampling mechanism for feature extraction and recovery. The encoding layer enhances the feature extraction capability and reduces the computational cost through deep separable convolution and residual connection; the decoding layer optimizes the feature recovery process through multi-scale feature fusion and bilinear interpolation upsampling. The discriminant layer classifies, estimates probability, and regresses features according to different task requirements, making the model more accurate and efficient when processing multiple tasks.

[0147] Step 5-1, preprocess the two-dimensional grid image in the input layer:

[0148] According to the statistical characteristics of natural images, except for remote sensing images or medical images, the mean of each channel is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225) for each channel.

[0149] The image is standardized so that the samples of the feature image conform to the standard normal distribution with a mean of 0 and a standard deviation of 1.

[0150] Considering that the resolution of the input image is too high, which affects the number of inference calculations and the speed of the model, the input layer uses layer 0 to reduce the resolution of the feature image and linearly interpolate the preprocessed feature image to reduce the number of parameters and shorten the time.

[0151] Step 5-2, the encoder layer mainly performs image downsampling based on convolutional neural network:

[0152] The operator fusion method is adopted to encapsulate the convolution operation, BatchNorm and activation function into a convbnrelu module.

[0153] In order to prevent gradient explosion and gradient disappearance in the network, the convolution layer uses 1×1 and 3×3 small convolutions for depth-separable convolution operations.

[0154] Then, three ConvBnReLu modules are added sequentially into one module.

[0155] To facilitate network optimization, each layer in the encoder network accumulates the output value of each block with the output value of the previous block through a residual connection to obtain the input value of the next layer.

[0156] In step 5-3, the decoder layer consists of four layers, which refines the feature image through convolution calculation and merges the multi-scale feature image with upsampling.

[0157] The 8th layer uses convolution kernels of size 1 and maximum pooling of size 5 to perform operations alternately to obtain feature images.

[0158] Then, the feature image is restored to the same size as that of the 6th and 5th layers through bilinear interpolation upsampling calculation.

[0159] The 9th layer fuses the three feature images of the 5th, 6th and 8th layers by adding corresponding pixels.

[0160] The 10th layer has the same design idea as the 8th layer. First, the feature image is calculated by bilinear interpolation upsampling to obtain a feature image of the same size as the 4th layer.

[0161] Then, the feature images of the 10th layer and the 4th layer are fused by adding the corresponding pixels. Considering that the size of the feature image restored by the 11th layer is the same as that of the 3rd layer, that is, the scale is reduced by half compared with the original image, the maximum pooling calculation is too large and the time is too long, so the 11th layer selects a 1×1 convolution kernel for convolution calculation.

[0162] Then the feature images of the 3rd and 10th layers are fused by pixel addition.

[0163] Step 5-4, perform semantic segmentation and depth estimation in the discriminative layer:

[0164] Considering that semantic segmentation needs to be performed n times, semantic segmentation prediction uses a 1×1×n convolution kernel to calculate the output feature image of the decoder network and predict the classification probability of each pixel.

[0165] Depth prediction uses a 1×1 convolution kernel to perform regression fitting on the depth of field of each pixel and obtain the fitting result.

[0166] Step 6: The depth information and semantic information are fused into 3D semantic point cloud data. According to the camera posture, the point cloud data of each frame is mapped to the world coordinate system to obtain a 3D semantic map. MPA and MIoU are used as semantic image accuracy indicators for comparison, as shown in Figure 4 As shown:

[0167] The output layer restores the semantic image and depth image predicted by the discriminant layer to an image of 640×480 pixels with the same resolution as the original image through bilinear interpolation.

[0168] Obtaining image depth information helps obtain the coordinates (X, Y, Z) of all pixels in the camera coordinate system, as well as the semantic information of all pixels that can be obtained.

[0169] Each semantic information corresponds to an RGB value. The depth information and semantic information are integrated to generate point cloud data (X, Y, Z, R, B, G).

[0170] By continuously updating the camera's posture, all point cloud data of each frame image are converted into coordinates in a unified world coordinate system to obtain a three-dimensional semantic map.

[0171] MPA and MIoU are used as semantic image accuracy indicators for comparison.

[0172] Step 7: Optimize the 3D semantic map based on a two-stage strategy:

[0173] The first stage calculates the rotations of all multiple robots.

[0174] The second stage computes the complete pose transformation.

[0175] After calculating the rotation matrices and pose transformations of multiple robots, the accuracy of map construction is further improved through global optimization.

[0176] First, the relative rotation angles between multiple robots are estimated through eigenvalue matching and reprojection methods to generate their respective rotation matrices. The rotation matrix is ​​a matrix used to represent the direction change of the robot in three-dimensional space.

[0177] After calculating the rotation matrix, we further calculate the pose matrix, which includes translation and rotation information, indicating the position (translation vector) and orientation (rotation matrix) of the robot in the world coordinate system. The pose matrix can fully describe the posture of the robot in space.

[0178] After multiple robots have completed the calculation of the pose and rotation matrix, the pose matrix of each frame is used as a node in the graph through graph optimization technology, and the pose constraints between frames are used as edges of the graph. Graph optimization reduces the impact of error accumulation on map construction by minimizing the errors between all nodes and edges.

[0179] To further improve the geometric accuracy of the map, bundle adjustment jointly optimizes the pose matrix and 3D point coordinates of all frames. The bundle adjustment optimization process can simultaneously correct the relative position and rotation of each frame to ensure geometric consistency between multiple frames.

[0180] In addition, loop detection is used to identify when the robot returns to the visited area, reduce posture drift, and improve positioning accuracy. The optimization process also combines semantic information with geometric information, and performs joint optimization through indicators such as MPA and MIoU to ensure that the map is not only geometrically accurate, but also semantically accurate. Finally, local optimization is used to reduce the amount of calculation in large-scale scenes, and the posture is further adjusted through the relocalization algorithm when positioning errors are detected to ensure real-time and robustness in multi-robot collaboration. Among them, geometric information refers to the spatial structure information of the environment, which is represented by point cloud data obtained by laser radar (LiDAR), and geometric verification is performed through algorithms such as RANSAC to extract clusters with geometric consistency.

[0181] Step 8: Implement the communication protocol between robots based on the position recognition module:

[0182] When a robot captures new LiDAR data, the location recognition module generates a DELIGHT descriptor to compress the original point cloud and restore it to a local descriptor tree. The local descriptor tree refers to the hierarchical organization of DELIGHT descriptors in a local area, which is usually used for local environment information storage and matching of a specific robot. The local descriptor tree is a part of the descriptor tree that is used to efficiently search and match descriptors in a local range in order to quickly locate and optimize the map construction of the local environment.

[0183] The communication message of the DELIGHT descriptor is in the form of R_MSG to send data to another robot.

[0184] When another robot β receives PR_MSG from its neighbor robot α, the PR (place recognition module) will be triggered to retrieve the most similar descriptor in the descriptor tree. The descriptor tree is a hierarchical organization of DELIGHT descriptors. The descriptors are arranged hierarchically by distance or feature value using the FLANN (fast nearest neighbor search) algorithm and used to quickly search for matching descriptors. The global descriptor obtains and compresses the raw point cloud data through 3D LiDAR, and then stores it in the descriptor tree for matching and communication.

[0185] Another robot β matches the segment description of the feature value around the node with the most similar descriptor in the descriptor tree through a similarity measure to determine the corresponding position of the node in the map. At the same time, the relative pose message is encapsulated as RP_MSG and sent. When robot α receives RP_MSG, the relative pose information in RP_MSG is calculated based on feature value matching. Robot α still needs to perform secondary calculations to ensure that the relative pose is consistent with its own environmental perception. Therefore, the corresponding feature value-based segment description is extracted. The RP (relative pose module) calculates the relative pose using the descriptor and establishes a new data-to-data association. In order to generate a complete and accurate global map from multiple robots, the DGO module is used to obtain multi-trajectory estimates. Each robot only uses measurements involving boundary node poses shared with other robots to estimate its local trajectory. Once a new data-to-data association is established, the DGO (Distributed Graph Optimization) distributed graph optimization module is triggered to update the map by sharing boundary node poses multiple times and propagating them in the form of DGO_MSG. If multiple robots do not capture data, the entire system will send a FIN message in this loop to terminate the communication. At the end of each communication cycle, each robot maintains a globally consistent state. The node is the specific location in the map where the robot collects environmental information.

[0186] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-level semantic perception map construction method for a distributed SLAM system, characterized in that: include: Collecting location information, and obtaining candidate matching frames based on the location information; Extracting feature values ​​from the candidate matching frames, generating vectors as segment descriptions of the feature values, processing the segment descriptions, and obtaining processed point cloud data; Based on the processed point cloud data, construct a probabilistic 3D occupancy grid map, perform plane segmentation on the probabilistic 3D occupancy grid map, and construct a 2D grid map; Inputting the 2D grid image into a convolutional neural network model to perform image depth estimation and semantic segmentation to obtain depth information and semantic information; Fusing the depth information and the semantic information to obtain three-dimensional semantic point cloud data; Based on the three-dimensional semantic point cloud data and the camera posture, a three-dimensional semantic map is obtained and optimized.

2. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Acquiring candidate matching frames based on the location information includes: Acquiring location information, wherein the location information is obtained by capturing the reflection intensity of each surface point on the map from the environment through 3D LiDAR; Establishing a point cloud coordinate system according to the posture of the sensor on each node, generating a histogram based on the reflection intensity and the point cloud coordinate system, and constructing a DELIGHT descriptor; The similarity value of the DELIGHT descriptor is calculated, and based on the similarity value, the candidate matching frame is obtained.

3. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Obtaining processed point cloud data includes: Segment matching based on the FLANN tree by fast nearest neighbor search in the space of segment descriptions using the number of neighbors and distance thresholds; Based on the geometric verification of RANSAC, the segment description after segment matching is processed with the resolution threshold and the minimum cluster size to obtain the clusters that meet the geometric consistency, that is, the processed point cloud data.

4. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Constructing a probabilistic 3D occupancy grid map includes: Convert the acquired and processed point cloud data into camera coordinates and project them onto the image plane; The intensity value of each 2D point on the image plane is assigned to the z-axis value of the corresponding 3D point to construct the probabilistic 3D occupancy grid map.

5. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 4 is characterized in that: Constructing a 2D grid graph includes: Filtering the probabilistic 3D occupancy grid map along the z-axis to remove points on the ceiling plane; The RANSAC algorithm is used to iteratively fit the ground plane to obtain the normal vector and height of the ground, the ground is removed, and the remaining voxels are projected onto the ground to obtain the 2D grid map.

6. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 1 is characterized in that: Inputting the 2D grid map into a convolutional neural network model for image depth estimation and semantic segmentation includes: Input the 2D grid image into an input layer for preprocessing; The preprocessed 2D grid image is input into the encoder layer for image downsampling to obtain multi-scale feature images; Inputting the multi-scale feature image into a decoder layer, merging the multi-scale feature image with the upsampling to obtain a fused feature image; Input the fused feature image into the discriminant layer for image depth estimation and semantic segmentation, obtain the fitting result and classification probability of each pixel, and then obtain the depth image and semantic image; The depth image and the semantic image are input into an output layer, and the depth image and the semantic image are restored to the same resolution as the 2D grid image through bilinear interpolation to obtain the depth information and the semantic information.

7. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 6 is characterized in that: Constructing the decoder layer includes: encapsulating the convolution layer, BatchNorm and activation function into a convbnrelu module, wherein the convolution layer performs a depth-separable convolution operation; The decoder layer is built based on several convbnrelu modules.

8. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 6 is characterized in that: Obtaining the fitting results and classification probability of each pixel includes: Performing semantic segmentation on the fused feature image several times using a convolution kernel of a preset size to predict the classification probability of each pixel; The depth of field of each pixel of the fused feature image is regressively fitted using the convolution kernel of the preset size to obtain a fitting result of each pixel.

9. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 3 is characterized in that: Obtaining the three-dimensional semantic graph and optimizing it includes: The depth information and the semantic information are merged into three-dimensional semantic point cloud data, and the point cloud data of each frame is mapped to a world coordinate system according to the camera posture to obtain the three-dimensional semantic map; The rotation and complete posture transformation of the robot are calculated, and the three-dimensional semantic graph is optimized.

10. The multi-level semantic perception map construction method for a distributed SLAM system according to claim 9, characterized in that: The robots also communicate with each other, including: When robot α captures new LiDAR data, the location recognition module generates a DELIGHT descriptor to compress the original point cloud and restore it to the local descriptor tree, where the local descriptor tree is a hierarchical organization of DELIGHT descriptors in the local area; The communication message of the descriptor sends data to the robot β in the form of R_MSG, and the position recognition module retrieves the most similar descriptor in the descriptor tree, wherein the descriptor tree is a hierarchical organization form of DELIGHT descriptors; Robot β matches the segment description of the node's preset range feature value with the most similar descriptor in the descriptor tree to determine the node's position in the map, and generates relative pose information and sends it to robot α to establish a communication protocol between the robots.

Images