Point cloud instance segmentation model based on dynamic convolution and SLAM map construction method

Abstract

The invention discloses a point cloud instance segmentation model based on dynamic convolution and an SLAM map construction method. The point cloud instance segmentation model comprises a sparse convolutional network, a mask head, an offset head, a segmentation head, a dynamic weight generator and an instance decoder. The sparse convolutional network extracts global features of the surrounding environment from the point cloud data. A mask head, an offset head and a segmentation head separate mask features, centroid offset features and semantic features from global features. The dynamic weight generator generates corresponding position information embedding, category masks and instance weights corresponding to different instances based on the centroid offset features, the semantic features and the global features. And the instance decoder generates a dynamic target mask for accurately describing the contour of the dynamic object by combining instance weights based on the mask features and instance features generated by position information embedding and category mask, so that diversified dynamic objects in the point cloud data are accurately segmented, the interference of the dynamic objects on static environment mapping is reduced, and the mapping efficiency is improved. And the accuracy of positioning and mapping is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of marking scene content, and in particular to a point cloud instance segmentation model based on dynamic convolution and a SLAM map construction method. Background Art

[0002] Traditional LiDAR SLAM algorithms often face a series of challenges when building maps in dynamic environments, including insufficient accuracy, poor robustness, and poor performance in complex environments. These limitations affect the reliability of LiDAR SLAM systems in rapidly changing scenes. Especially in dynamic environments with multiple dynamic objects of different shapes and sizes, static objects are prone to misjudgment and erroneous removal, resulting in reduced positioning and mapping accuracy, reducing the robustness and reliability of LiDAR SLAM systems in complex, rapidly changing dynamic environments. Summary of the invention

[0003] In view of the shortcomings of the existing technology, the present invention proposes a point cloud instance segmentation model and SLAM map construction method based on dynamic convolution, which can improve the recognition accuracy of dynamic objects in complex environments, thereby ensuring the robustness and reliability of the LiDAR SLAM system in complex and rapidly changing dynamic environments. The specific technical solution is as follows: In a first aspect, a point cloud instance segmentation model based on dynamic convolution is provided. In a first implementable manner of the first aspect, the model includes: A sparse convolutional network configured to acquire point cloud data and extract global features from the point cloud data; A mask head, an offset head and a segmentation head are respectively configured to separate mask features, centroid offset features and semantic features from the global features extracted by the sparse convolution; A dynamic weight generator configured to determine position information embedding, category mask and instance weight according to the centroid offset feature, semantic feature and the global feature separated by the offset head and the segmentation head; The instance decoder is configured to generate a dynamic target mask corresponding to the point cloud data according to the instance weight and the instance feature embedded by combining the mask feature, the category mask and the position information.

[0004] In combination with the first implementable manner of the first aspect, in a second implementable manner of the first aspect, the sparse convolutional network is a submanifold sparse convolutional network.

[0005] In combination with the first implementable manner of the first aspect, in a third implementable manner of the first aspect, a lightweight Transformer block is configured between the encoder and the decoder of the sparse convolutional network.

[0006] In combination with the first implementable method of the first aspect, in a fourth implementable method of the first aspect, the offset head obtains the centroid offset feature by moving points in the point cloud to the centroid of the corresponding instance, and clusters the particles with similar geometric centroids and the same category prediction results.

[0007] In combination with the first implementable manner of the first aspect, in a fifth implementable manner of the first aspect, the dynamic weight generator clusters particles with similar geometric centroids and the same category prediction results according to centroid offset features.

[0008] In combination with the first implementable manner of the first aspect, in a sixth implementable manner of the first aspect, by grouping homogeneous points within the cluster, a sub-network is used to aggregate large-scale contexts to generate instance-aware instance weights.

[0009] In a second aspect, a SLAM map construction method based on dynamic convolution is provided. In a first implementable manner of the second aspect, the method includes: Constructing the point cloud instance segmentation model, and training the point cloud instance segmentation model using a pre-constructed training set; Acquire point cloud image data of the environment to be constructed in real time, and generate a dynamic target mask corresponding to the point cloud image data through a trained cloud instance segmentation model; Separating the dynamic point cloud and the static point cloud in the point cloud image data by using the dynamic object mask; A static map is constructed based on the separated static point cloud, and a dynamic map is constructed based on the separated dynamic point cloud, and the dynamic map and the static map are aligned and combined to form a global map.

[0010] In combination with the first implementable manner of the second aspect, in a second implementable manner of the second aspect, acquiring the point cloud image data includes: The environmental scanning data collected by the laser radar is acquired, and denoising, ground segmentation and / or cluster segmentation are performed on the environmental scanning data to obtain the point cloud image data.

[0011] In combination with the first implementable manner of the second aspect, in a third implementable manner of the second aspect, constructing a static map based on the separated static point cloud includes: Perform feature extraction on the static point cloud to obtain edge feature points and plane feature points corresponding to the current frame; Constructing a local edge map and a local plane map according to the edge feature points and the plane feature points respectively, and constructing a local map corresponding to the current frame through the local edge map and the local plane map; Based on the extracted edge feature points, plane feature points and constructed local map, odometer estimation is performed to determine the key frames, and a static map is constructed based on the static point cloud corresponding to the key frames.

[0012] In combination with the third implementable manner of the second aspect, in a fourth implementable manner of the second aspect, performing odometer estimation based on edge feature points, plane feature points, and a local map includes: Searching for the nearest edge feature point from the local edge map, and selecting two nearest edge feature points for the coordinates of each edge feature point in the local map to construct a residual equation from the starting point to the edge; Selecting three closest plane feature points from the local plane map to construct a residual equation from the starting point to the surface; Based on the residual equation from point to edge and the residual equation from point to surface, the transformation relationship from the current frame to the local map is determined, and whether the current frame is a key frame is determined according to the transformation relationship.

[0013] Beneficial effect: The point cloud instance segmentation model and SLAM map construction method based on dynamic convolution of the present invention are adopted, and the corresponding instance weights are configured for instances of different shapes and sizes by setting a dynamic weight generator, so as to generate a dynamic target mask that can accurately describe the contour of the dynamic object, so as to accurately segment a variety of dynamic objects, reduce the interference of dynamic objects on static environment mapping, thereby ensuring the accuracy of positioning and mapping, and further improving the robustness and reliability of the lidar SLAM system in complex and rapidly changing dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific implementation of the present invention, the following will briefly introduce the drawings required for use in the specific implementation. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0015] Figure 1 A schematic diagram of the framework structure of a point cloud instance segmentation model based on dynamic convolution provided by an embodiment of the present invention; Figure 2 A flowchart of a SLAM map construction method based on dynamic convolution provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0017] like Figure 1 The schematic diagram of the framework structure of the point cloud instance segmentation model based on dynamic convolution is shown, and the segmentation model includes: A sparse convolutional network configured to acquire point cloud data and extract global features from the point cloud data; A mask head, an offset head and a segmentation head are respectively configured to separate mask features, centroid offset features and semantic features from the global features extracted by the sparse convolution; A dynamic weight generator configured to determine position information embedding, category mask and instance weight according to the centroid offset feature, semantic feature and the global feature separated by the offset head and the segmentation head; The instance decoder is configured to generate a dynamic target mask corresponding to the point cloud data according to the instance weight and the instance feature embedded by combining the mask feature, the category mask and the position information.

[0018] Specifically, the point cloud instance segmentation model includes a sparse convolutional network, a mask header, an offset head, a segmentation head, a dynamic weight generator and an instance decoder. Among them, the sparse convolutional network can receive the point cloud data collected by the laser radar scanning the surrounding environment, and extract the global features of the surrounding environment from the acquired point cloud data. The mask header, the offset head and the segmentation head can respectively separate the mask features, the centroid offset features and the semantic features from the global features extracted by the sparse convolutional network. The dynamic weight generator can generate the corresponding position information embedding, the category mask, and the instance weights for perceiving different instances in the point cloud data based on the centroid offset features, the semantic features separated by the offset head and the segmentation head and the global features extracted by the sparse convolutional network. The instance decoder can generate a dynamic target mask for representing dynamic targets based on the instance features generated after adding the position information embedding and the category mask to the mask features, and the instance weights corresponding to different instances determined by the dynamic weight generator.

[0019] The dynamic target mask can accurately describe the contours of dynamic objects to accurately segment diverse dynamic objects in point cloud data, reduce the interference of dynamic objects on static environment mapping, thereby ensuring the accuracy of positioning and mapping, and further improving the robustness and reliability of the lidar SLAM system in complex and rapidly changing dynamic environments.

[0020] In this embodiment, optionally, the sparse convolution network is a submanifold sparse convolution network. Specifically, a submanifold sparse convolution network can be used to improve the segmentation efficiency, thereby improving the real-time and efficiency of the LiDAR SLAM system in a dynamic environment. Specifically, the traditional sparse convolution allows the activation position to propagate to all adjacent positions during the convolution process, which may lead to the expansion of the calculation area, thereby increasing the amount of calculation. The submanifold sparse convolution limits the convolution operation so that calculations are only performed on activated positions, and new activation points are not generated in inactive areas, effectively reducing computational redundancy. The submanifold sparse convolution does not create new activation points in inactive areas, so deep feature extraction can be performed without destroying the sparse structure. Maintain the spatial consistency of the point cloud data and avoid introducing noise, thereby improving the accuracy of segmentation. The network can still maintain high-quality feature extraction with less computation.

[0021] In this embodiment, optionally, a lightweight Transformer block is configured between the encoder and the decoder of the sparse convolutional network.

[0022] Specifically, due to the small number of convolutional layers and channels, sparse convolutions are often limited by limited receptive fields and representation capabilities. To this end, lightweight Transformer blocks can be configured in sparse convolutional networks to enhance the long-range interaction between the encoder and decoder.

[0023] In this embodiment, optionally, the offset head obtains the centroid offset feature by moving points in the point cloud to the centroid of the corresponding instance.

[0024] Specifically, due to the sparsity and non-uniformity of 3D point clouds, the variability of the distribution of points across surfaces makes the aggregation of contextual information complex and unstable. To this end, the offset head can simplify the distance relationship between different surfaces by moving points in the global features to the centroid of the corresponding instance. In order to supervise the prediction of the centroid offset of the point, the following loss function can be used to minimize the distance between the predicted offset point and the true centroid: ; in, It is The coordinates of the points, Indicates the offset Items, represents the geometric centroid of the corresponding instance, Yes means is a valid point for centroid prediction, Indicates the total number of valid points.

[0025] In this embodiment, optionally, the dynamic weight generator clusters particles with similar geometric centroids and the same category prediction results according to centroid offset features.

[0026] Specifically, due to the differences in the morphology and features of different instances, directly using a fixed convolution kernel may lead to inaccurate instance segmentation. Therefore, the weight generator needs to adaptively generate different convolution weights for each instance.

[0027] At the instance level, a sub-network is applied to aggregate the features within the candidate instance so that the point cloud features within the instance can better capture the overall information of the instance. The instance-level dynamic weights are calculated through a lightweight MLP (multi-layer perceptron) network. The weight generator generates instance-level dynamic weights based on global features, offset predictions, and semantic segmentation results.

[0028] In this embodiment, optionally, by grouping particles in a cluster, a sub-network is used to aggregate large-scale contexts to generate instance-aware instance weights.

[0029] The dynamic weight generator can use the geometric centroid of each instance as a benchmark to calculate the position information embedding of each point in the global feature relative to the centroid, and concatenate it with the mask feature to enhance the instance differentiation ability, thereby effectively distinguishing different instances and improving the instance segmentation accuracy in complex environments. The position information embedding is expressed as follows: ; in, Indicates The geometric centroid of each instance.

[0030] Specifically, the generation process of the category mask depends on the input of the weight generator, including global features, centroid prediction features of the offset head, and semantic segmentation features of the segmentation head. Through the centroid prediction features, the geometric centroid offset of each point is calculated so that similar points can be clustered around the centroid of the same instance. Combined with the category prediction of the segmentation head, it ensures that the clustered points belong to the same semantic category, thus forming instance candidates.

[0031] Calculate the semantic category distribution of the points in each instance candidate and count the category with the most points. If the category label of the instance candidate meets certain confidence requirements, it is considered a valid instance. For each instance candidate, construct a category mask that records the category corresponding to the instance and is used for subsequent instance feature extraction and weight application.

[0032] In this embodiment, the example decoder can be composed of three convolution kernels with a size of The activation function of each convolutional layer is ReLU. The loss function of the entire instance segmentation model can be expressed as: ; in, is the total number of clusters, It is Semantic prediction of each point, It is The semantic labels of the clusters, is the binary cross entropy loss function, is an indicator function, indicating that the loss is only Calculate on points with the same semantic label. represents the real mask, Represents the prediction mask.

[0033] like Figure 2 The flowchart of the SLAM map construction method based on dynamic convolution shown in FIG. 1 includes: Step 1: construct the above-mentioned point cloud instance segmentation model, and train the point cloud instance segmentation model through a pre-constructed training set; Step 2: Acquire point cloud image data of the environment to be constructed in real time, and generate a dynamic target mask corresponding to the point cloud image data through the trained cloud instance segmentation model; Step 3: Separating the dynamic point cloud and the static point cloud in the point cloud image data by using the dynamic target mask; Step 4: construct a static map based on the separated static point cloud, and construct a dynamic map based on the separated dynamic point cloud, and align and combine the dynamic map and the static map to form a global map.

[0034] Specifically, first, the above-mentioned point cloud instance segmentation model can be constructed, and the point cloud instance segmentation model can be trained through a pre-constructed training set to obtain a trained point cloud instance segmentation model. Then, the point cloud data of the surrounding environment can be scanned in real time by a lidar, and a dynamic target mask corresponding to the point cloud image data can be generated by the trained point cloud instance segmentation model. After that, the dynamic point cloud and the static point cloud in the point cloud image data can be separated by the dynamic target mask. Finally, a static map can be constructed based on all the separated static point clouds. At the same time, a dynamic map can be constructed based on the separated dynamic point clouds, and the dynamic map and the static map can be aligned and combined to form a global map.

[0035] The dynamic target mask generated by the point cloud instance segmentation model can accurately describe the contours of dynamic objects, so as to accurately segment the diverse dynamic objects in the point cloud data, reduce the interference of dynamic objects in complex dynamic environments on static environment mapping, thereby ensuring the accuracy of positioning and mapping, and further improving the robustness and reliability of the lidar SLAM system in complex and rapidly changing dynamic environments.

[0036] In this embodiment, optionally, obtaining the point cloud image data includes: The environmental scanning data collected by the laser radar is acquired, and denoising, ground segmentation and / or cluster segmentation are performed on the environmental scanning data to obtain the point cloud image data.

[0037] Specifically, after the laser radar obtains the environmental scanning data of the surrounding environment, the point cloud image data of the surrounding environment can be obtained by pre-processing the environmental scanning data such as denoising, ground segmentation and / or cluster segmentation. The quality of the point cloud image data can be improved by denoising, ground segmentation and / or cluster segmentation, providing a solid foundation for subsequent analysis. Similarly, when constructing a training set, the obtained sample data can be pre-processed in the same way to improve the quality of the samples in the training set.

[0038] In this embodiment, optionally, constructing a static map based on the separated static point cloud includes: Perform feature extraction on the static point cloud to obtain edge feature points and plane feature points corresponding to the current frame; Constructing a local edge map and a local plane map according to the edge feature points and the plane feature points respectively, and constructing a local map corresponding to the current frame through the local edge map and the local plane map; Based on the extracted edge feature points, plane feature points and constructed local map, odometer estimation is performed to determine the key frames, and a static map is constructed based on the static point cloud corresponding to the key frames.

[0039] Specifically, when constructing a static map, first, feature extraction can be performed on all separated static point clouds to obtain edge feature points and plane feature points in the point cloud image data of the current frame. Specifically, edge feature points and plane feature points can be extracted from all static point clouds based on the set search radius and curvature threshold through the constructed KD tree. Specifically, for each point in the static point cloud, the adjacent points of each point are searched according to the search radius and the radius sphere, and the curvature between the point and the adjacent points is calculated. If the calculated curvature is greater than the set curvature threshold, the point is an edge feature point, otherwise the point is a plane feature point.

[0040] Then, a local edge map and a local plane map can be constructed according to the extracted edge feature points and plane feature points, respectively, and a local map can be constructed by combining the local edge map and the local plane map. Afterwards, the transformation relationship between the current frame and the local map can be determined based on the constructed local map and the extracted edge feature points and plane feature points through the odometer estimation method, and the transformation relationship includes the rotation change and translation change from the current frame to the local map. When the translation change or the rotation change is greater than a preset threshold, the current frame can be determined as a key frame, and the local map can be updated through a certain number of updated key frames. When a new key frame is added, the oldest key frame is removed to re-form the local map.

[0041] In this embodiment, optionally, odometer estimation is performed based on edge feature points, plane feature points, and a local map, including: Searching for the nearest edge feature point from the local edge map, and selecting two nearest edge feature points for the coordinates of each edge feature point in the local map to construct a residual equation from the starting point to the edge; Selecting three closest plane feature points from the local plane map to construct a residual equation from the starting point to the surface; Based on the residual equation from point to edge and the residual equation from point to surface, the transformation relationship from the current frame to the local map is determined, and whether the current frame is a key frame is determined according to the transformation relationship.

[0042] Specifically, the nearest edge feature point can be searched from the local edge map , and for each edge feature point searched out, the coordinates in the local map , select the two nearest edge feature points and , construct the residual equation from the starting point to the edge, specifically:

[0043] You can also select the three nearest plane feature points from the local plane map , , The residual equation from point to surface constructed in this way is:

[0044] The total residual is: After the residual is minimized, the transformation relationship from the point to the local map can be obtained. .

[0045] The construction method of dynamic map is the same as that of static map, so it will not be repeated here. Each frame of point cloud data contains a timestamp. The timestamp can be used to determine the static map that matches the dynamic map constructed by the dynamic point cloud segmented from the point cloud data. Finally, the matching dynamic map is added to the already constructed static map to reduce the interference of dynamic objects on the static environment mapping, thereby ensuring the accuracy of positioning and mapping, and then improving the robustness and reliability of the LiDAR SLAM system in complex and rapidly changing dynamic environments, and obtaining a static map containing dynamic objects.

[0046] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A point cloud instance segmentation model based on dynamic convolution, characterized in that: include: A sparse convolutional network configured to acquire point cloud data and extract global features from the point cloud data; A mask head, an offset head and a segmentation head are respectively configured to separate mask features, centroid offset features and semantic features from the global features extracted by the sparse convolution; A dynamic weight generator configured to determine position information embedding, category mask and instance weight according to the centroid offset feature, semantic feature and the global feature separated by the offset head and the segmentation head; The instance decoder is configured to generate a dynamic target mask corresponding to the point cloud data according to the instance weight and the instance feature embedded by combining the mask feature, the category mask and the position information.

2. The point cloud instance segmentation model according to claim 1, characterized in that: The sparse convolutional network is a submanifold sparse convolutional network.

3. The point cloud instance segmentation model according to claim 1, characterized in that: A lightweight Transformer block is configured between the encoder and the decoder of the sparse convolutional network.

4. The point cloud instance segmentation model according to claim 1, characterized in that: The offset head obtains the centroid offset feature by moving points in the point cloud to the centroid of the corresponding instance.

5. The point cloud instance segmentation model according to claim 1, characterized in that: The dynamic weight generator clusters particles with similar geometric centroids and the same category prediction results according to centroid offset characteristics.

6. The point cloud instance segmentation model according to claim 5, characterized in that: By grouping homogeneous points within clusters, a sub-network is utilized to aggregate large-scale contexts and generate instance-aware instance weights.

7. A SLAM map construction method based on dynamic convolution, characterized in that: include: Constructing a point cloud instance segmentation model as described in any one of claims 1 to 6, and training the point cloud instance segmentation model using a pre-constructed training set; Acquire point cloud image data of the environment to be constructed in real time, and generate a dynamic target mask corresponding to the point cloud image data through a trained cloud instance segmentation model; Separating the dynamic point cloud and the static point cloud in the point cloud image data by using the dynamic object mask; A static map is constructed based on the separated static point cloud, and a dynamic map is constructed based on the separated dynamic point cloud, and the dynamic map and the static map are aligned and combined to form a global map.

8. The SLAM map construction method according to claim 7, characterized in that: Acquiring the point cloud image data includes: The environmental scanning data collected by the laser radar is acquired, and denoising, ground segmentation and / or cluster segmentation are performed on the environmental scanning data to obtain the point cloud image data.

9. The SLAM map construction method according to claim 7, characterized in that: Build a static map based on the separated static point cloud, including: Perform feature extraction on the static point cloud to obtain edge feature points and plane feature points corresponding to the current frame; Constructing a local edge map and a local plane map according to the edge feature points and the plane feature points respectively, and constructing a local map corresponding to the current frame through the local edge map and the local plane map; Based on the extracted edge feature points, plane feature points and constructed local map, odometer estimation is performed to determine the key frames, and a static map is constructed based on the static point cloud corresponding to the key frames.

10. The SLAM map construction method according to claim 9, characterized in that: Odometry estimation based on edge feature points, plane feature points and local maps, including: Searching for the nearest edge feature point from the local edge map, and selecting two nearest edge feature points for the coordinates of each edge feature point in the local map to construct a residual equation from the starting point to the edge; Selecting three closest plane feature points from the local plane map to construct a residual equation from the starting point to the surface; Based on the residual equation from point to edge and the residual equation from point to surface, the transformation relationship from the current frame to the local map is determined, and whether the current frame is a key frame is determined according to the transformation relationship.

Images