A method for generating local semantic grid maps for intelligent vehicles

Abstract

The present invention provides a method for generating a local semantic grid map of an intelligent vehicle, comprising the steps of: S01, constructing a training data set for a semantic grid map generation task; S02, constructing a semantic grid map generation model; S03, training the semantic grid map generation model; S04, deploying the semantic grid map generation model. The present invention aims to construct a semantic grid map by using a single-frame laser radar point cloud, which inherits the advantages of grid maps in planning and decision-making and can meet the needs of complex driving environments. The method for constructing a training data set for a semantic grid map generation task proposed in the present invention can avoid the problem of low efficiency of manual labeling, and realize fast and efficient construction of training data. A large amount of training data is the basic guarantee of model performance. The semantic grid map generation model proposed in the present invention is a lightweight and efficient deep learning model that can construct a semantic grid map around a vehicle using a single-frame sparse laser radar point cloud.

Description

Technical Field

[0001] The invention relates to the field of grid maps, and in particular to a method for generating a local semantic grid map of an intelligent vehicle. Background Art

[0002] The grid map divides the surrounding environment into equally spaced areas, and uses the characteristics of each cell to represent environmental information. The occupancy grid map is represented by the probability value of the cell being occupied. The grid map is easy to create and has a detailed description of the environment. It is generally used in modules such as local path planning. The semantic grid map (Road Grid Map, RGM) is a local environment representation method that contains multiple categories such as roads, vehicles, pedestrians, and trees represented by a grid map. The value of each cell in the grid map represents the probability of it belonging to a certain semantic category, that is, Among them, H G and W G is the length and width of the grid map, and C is the number of semantic categories. The cells of the classic occupancy grid map have only two possible states: occupied or unoccupied, so there is not enough semantic information for autonomous driving in complex driving environments.

[0003] A full understanding of the driving scene and a reliable representation of the surrounding environment are crucial for the autonomous driving of intelligent vehicles. To achieve this goal, the vehicle must be equipped with a series of complementary sensor systems and sufficient computing resources to extract relevant information. Cameras and lidar sensors are widely used sensors. With the development of deep learning technology, semantic segmentation technology at the image and point cloud levels has achieved remarkable results. However, the representation of images and point clouds cannot be directly applied to autonomous driving planning and control systems. Grid maps, as a commonly used environmental representation method, are a dense environmental representation with an equidistant unit structure. Compared with unstructured point clouds and perspective images, the representation of grid maps is very suitable for sensor fusion applications and can be directly applied to subsequent autonomous driving planning and control subsystems.

[0004] Aiming at the problem of lack of semantic association information of environmental elements in complex driving scene modeling, the present invention studies a method for constructing a multi-category semantic grid map by fusing lidar and visual sensors, that is, estimating a semantic category for each cell in the local grid map. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention proposes a method for generating a local semantic grid map of an intelligent vehicle.

[0006] A method for generating a local semantic grid map of an intelligent vehicle comprises the following steps:

[0007] S01, constructing a training dataset for the semantic raster map generation task;

[0008] S02, constructing a semantic grid map generation model;

[0009] S03, training semantic raster map generation model;

[0010] S04. Deploy the semantic grid map generation model.

[0011] Based on the above, step S01 includes the following steps:

[0012] S011. Collect point cloud data and vehicle positioning data;

[0013] S012. Use a point cloud semantic segmentation algorithm to semantically annotate the collected point cloud data;

[0014] S013, aggregating the annotated point cloud data to obtain dense point cloud data at the current position;

[0015] S014. performing coordinate transformation on the semantic point cloud to obtain semantic annotation of the raster map;

[0016] S015. Optimize semantic grid map labels.

[0017] Based on the above, for step S02, a two-dimensional convolutional neural network is used to take the irregular lidar point cloud data as input data, and rasterization processing is performed to obtain multi-dimensional raster attribute features from a bird's-eye view. After that, the semantic raster map generation model is output after multiple downsampling, residual convolution and upsampling processes.

[0018] Based on the above, for step S03, the training task of the semantic grid map generation model is regarded as a multi-category pixel-level classification problem, the training hyperparameters are set, and then the adaptive distance estimation Adam optimizer is set, and the cross entropy loss function is used as the objective function to improve the convergence result of the model. The adaptive distance estimation optimizer is used to improve the loss descent path to improve the performance of the training model. Based on the above, the collected point cloud data is processed offline using the point cloud segmentation algorithm to obtain the semantic segmentation label of the single-frame point cloud, and the semantic point cloud P = {p i |i=1,,,N},where P i =(x i ,y i , z i , r i ,s i ) is one of the points, x i ,y i and z i Its coordinates, r i and i are their reflectivity and semantic labels respectively.

[0019] Based on the above, for step S013, according to the positioning information between adjacent frames, the spatial conversion relationship between adjacent frames is obtained, and the single-frame point clouds within a certain range before and after are aggregated to the current frame to obtain a dense semantic point cloud P k = {P k-n , ..., P k , ..., P k+n}.

[0020] Based on the above, the aggregated point cloud is rasterized to obtain the grid attribute map G k ={c ij |i=1,…,H;j=1,…,W}, and the number of points of each category contained in each cell C sd is the total category set, and the semantic label category of the cell is set according to the category of the points contained in the cell.

[0021] Based on the above, after clustering the dynamic category cells in the label through the DBSCAN algorithm to obtain the target instance, it is matched with the target instance under single-frame detection. By judging whether there is a matching instance in the single-frame point cloud, the annotation of the invisible target is identified and screened for optimization processing.

[0022] Based on the above, the objective function adopted is

[0023]

[0024] Where H and W are the length and width of the grid respectively; M is the number of semantic categories; y and p are the predicted value and the true value respectively;

[0025]

[0026] Among them, N c and N are the number of samples of category c and the number of samples of all categories respectively.

[0027] Based on the above, according to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model and calculate the gradient g at time step t. t

[0028]

[0029] First, calculate the exponential moving average m of the gradient t , and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, where m0 = 0, β1 is 0.9, and the more obvious formula is as follows

[0030] m t=β1m t-l +(1-β1)g t Second, calculate the exponential moving average v of the squared gradient t , v0 = 0, β2 = 0.999 is the influence of controlling the square of the gradient at the previous moment, and the formula is as follows

[0031]

[0032] Again, for m t Correct the deviation to reduce the impact of the deviation on the initial stage of training

[0033]

[0034] v t Correction is performed to reduce the impact of deviation on the initial stage of training. The formula is as follows

[0035]

[0036] Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 To avoid division by 0, the formula is as follows

[0037]

[0038] The present invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0039] 1. The present invention aims to construct a semantic grid map by using a single-frame LiDAR point cloud. The classic occupancy grid map does not have enough semantic information for autonomous driving in complex driving environments, and its cells have only two possible states: occupied or unoccupied. The semantic grid map constructed by this method has richer semantic features than the classic occupancy grid map. It inherits the advantages of the grid map in planning and decision-making, and can meet the needs of complex driving environments.

[0040] 2. The method for constructing a training data set for the semantic raster map generation task proposed in the present invention can avoid the problem of low efficiency of manual labeling and realize fast and efficient construction of training data. A large amount of training data is the basic guarantee of model performance.

[0041] 3. The semantic grid map generation model proposed in the present invention is a lightweight and efficient deep learning model that can construct a semantic grid map around the vehicle using a single-frame sparse lidar point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic block diagram of the process of the present invention.

[0043] Figure 2 It is a schematic diagram of the process of constructing a training data set for a semantic raster map generation task in the present invention.

[0044] Figure 3 It is a schematic block diagram of the road segmentation model architecture of the present invention.

[0045] Figure 4 Schematic diagram of the context convolution module of the present invention.

[0046] Figure 5 Schematic diagram of the residual convolution module of the present invention.

[0047] Figure 6 Schematic diagram of the upsampling module of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1 As shown, a method for generating a local semantic grid map of an intelligent vehicle comprises the following steps: S01, constructing a training data set for a semantic grid map generation task; S02, constructing a semantic grid map generation model; S03, training the semantic grid map generation model; S04, deploying the semantic grid map generation model.

[0050] Specific:

[0051] 1. Constructing a training dataset for semantic raster map generation tasks

[0052] like Figure 2 As shown, it is very difficult to construct a dense semantic grid map of the real driving environment, because various sensor information is a discrete detection and representation of the driving environment. Therefore, the present invention makes full use of the fusion between multiple frames and the existing efficient laser point cloud semantic segmentation algorithm to realize the automatic generation of dense semantic grid map training data.

[0053] On a vehicle equipped with a lidar sensor and a high-precision positioning sensor, the lidar point cloud data and vehicle positioning data are collected in various driving scenarios. Then, the collected lidar point cloud data is processed offline using a high-precision point cloud segmentation algorithm to obtain a single-frame lidar point cloud semantic segmentation label, and the semantic point cloud P = {p i |i=1,,,N},where p i =(x i ,yi , z i , r i ,s i ) is one of the points, x i ,y i and z i Its coordinates, r i and i Then, according to the positioning information between adjacent frames, the spatial transformation relationship between adjacent frames is obtained, and the single-frame lidar point cloud within a certain range before and after is aggregated to the current frame to obtain a dense semantic lidar point cloud P k = {P k-n , ..., P k , ..., P k+n}.

[0054] On this basis, the aggregated point cloud is rasterized to obtain the raster attribute map G k ={c ij |i=1,…,H;j=1,…,W} and the number of points of each category contained in each cell C sd is the total category set, which contains 12 categories in total, C sd ∈{road, sidewalk, parking area, building, railing, vegetation, terrain, pole, vehicle, two-wheeler, pedestrian, cyclist}. Specifically, if the cell contains dynamic categories such as vehicles, two-wheelers, pedestrians and cyclists, the semantic category of the cell is the corresponding dynamic category. If the cell does not contain a dynamic category and does not contain points of a static category, the semantic category of the cell is unknown. If the cell does not contain a dynamic category but contains a static category, the static category with the largest number of points is set as the semantic label category of the cell. The semantic label category setting of a cell is shown in the following formula:

[0055]

[0056] The dynamic semantic labels generated based on the aggregated semantic point cloud contain all dynamic targets in the region of interest. However, some of these targets are invisible from the perspective of the current frame due to occlusion by foreground objects. Therefore, in order to ensure more accurate model training using the dataset, it is necessary to remove the annotations of these invisible targets from the labels to optimize the semantic grid map labels.

[0057] In order to find the invisible target in the annotation, we first cluster the dynamic category cells in the label through the DBSCAN algorithm to obtain the target instance, and then match it with the target instance under the single-frame detection. If there is a matching instance in the single-frame laser point cloud, it means that the target is visible in the current single-frame perspective, otherwise it is invisible. Specifically, for the dynamic semantic label obtained by aggregating semantic point clouds, Gets a collection of cell indexes containing dynamic targets And dynamic semantic labels obtained through single-frame semantic point cloud and its cell index collection The clustering results are obtained by clustering with DBSCAN algorithm. and For D i ∈D 1 , if there is D j ∈D 2 Make Then D i It is a visible target under the current perspective, otherwise it is an invisible target.

[0058] 2. Construction of semantic grid map generation model

[0059] like Figure 3 As shown in the figure, the constructed semantic grid map generation model includes the current single-frame lidar point cloud as input, including point cloud rasterization, input layer, downsampling layer, residual convolution layer, upsampling layer and output layer. The input layer is a single-frame lidar point cloud. In order to construct a semantic grid map from a bird's-eye view, an efficient two-dimensional convolutional neural network is used to rasterize the irregular lidar point cloud to obtain a multi-dimensional grid attribute feature map from a bird's-eye view. Specifically, the point-by-point fusion features obtained in the above steps are first clustered according to the grid division to obtain a point set P of the cell dimension. j ={p i |0<i<N}, where N is the number of points in the cell. Due to the detection characteristics of the LiDAR sensor, the number of points in each cell is different, and there are a large number of empty cells. In order to standardize the data size of the cell and facilitate the subsequent use of convolutional neural network operations, the number of points in the cell needs to be unified. According to the characteristics of the LiDAR used, N is used. t As the threshold, for more than N t cells of points, randomly sample N t points; for less than N t Then, a multi-layer perceptron is used to extract features from all points in each column to obtain a multi-dimensional grid attribute feature map.

[0060] The downsampling layer downsamples the input features through a context convolution module with a step size of 2. The downsampling layer is as follows: Figure 4 As shown in Figure 1, a context branch is added in parallel with the main convolution. First, 1×1 convolution and Softmax function are used to obtain the attention weight, and then attention pooling is performed to obtain the global context features of the feature layer; then the global context features are transformed by 1×1 convolution and added to the feature map of the main convolution. Figure 5 As shown in , it reduces the gradient vanishing problem by adding skip connections at both ends of the convolution operation, which is conducive to extracting deeper features. Figure 6 As shown in the figure, it uses the context convolution module and bilinear upsampling operation to effectively combine the two feature maps. First, the two branches of the two feature maps are merged along the channel, and the merged feature map is processed by a context convolution module. In the whole model, the upsampling module is cascaded from deep features to shallow features. The current upsampling module uses the results of the previous level and the corresponding decoding features, making the upsampling operation more efficient.

[0061] 3. Training semantic raster map generation model

[0062] The constructed training set is used to train the constructed semantic grid map generation model. The algorithm regards this task as a multi-category pixel-level classification problem, sets the training hyperparameters, and then sets the adaptive distance estimation Adam optimizer. The cross entropy loss function is used as the objective function to improve the convergence result of the model. The adaptive distance estimation optimizer is used to improve the loss descent path and improve the performance of the training model. Due to the serious category imbalance problem of category samples in the semantic grid map construction task, a weight is assigned to each category to balance the loss values ​​of different categories. The weight of each category is calculated by counting the number of cells of all categories in the statistical data set. The objective function used is

[0063]

[0064] Among them, H and W are the length and width of the grid respectively; M is the number of semantic categories; y and p are the predicted value and the true value respectively.

[0065]

[0066] Among them, N c and N are the number of samples of category c and the number of samples of all categories respectively.

[0067] According to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model. The gradient of the t time step is calculated:

[0068]

[0069] First, calculate the exponential moving average m of the gradient t , m0 = 0, and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, β1 is set to 0.9, and the more explicit formula is as follows:

[0070] m t =β1m t-1 +(1-β1)g t

[0071] Second, calculate the exponential moving average v of the squared gradient t , v0 = 0, β2 = 0.999 is the influence of controlling the square of the gradient at the previous moment. The formula is as follows:

[0072] Again, for m t Correct the deviation to reduce the impact of the deviation on the initial stage of training

[0073]

[0074] Then, for v t Correction is performed to reduce the impact of deviation on the initial stage of training. The formula is as follows:

[0075]

[0076] Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 To avoid division by 0, the formula is as follows:

[0077]

[0078] 4. Deploy the semantic grid map generation model

[0079] A semantic grid map generation model is deployed on the vehicle controller. The point cloud data of the road ahead of the vehicle is collected by the vehicle-mounted lidar sensor and input into the deployed semantic grid map generation model to obtain the semantic category prediction value of each cell in the vehicle's local grid map, and construct a semantic grid map of the area around the vehicle in real time.

[0080] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for generating a local semantic grid map of an intelligent vehicle, characterized in that: Includes steps: S01, constructing a training dataset for the semantic raster map generation task; S02, constructing a semantic grid map generation model; S03, training semantic grid map generation model; S04, deploy semantic grid map generation model; Step S01 includes the following steps: S011. Collect point cloud data and vehicle positioning data; S012. Use a point cloud semantic segmentation algorithm to semantically annotate the collected point cloud data; S013, aggregating the annotated point cloud data to obtain dense point cloud data at the current position; S014. performing coordinate transformation on the semantic point cloud to obtain semantic annotation of the raster map; S015. Optimize semantic grid map labels; For step S03, the training task of the semantic raster map generation model is regarded as a multi-category pixel-level classification problem, the training hyperparameters are set, and then the adaptive moment estimation Adam optimizer is set, the cross entropy loss function is used as the objective function to improve the convergence result of the model, and the adaptive moment estimation optimizer is used to improve the loss descent path to improve the performance of the training model; The objective function used is, ， Where H and W are the height and width of the grid, M is the number of semantic categories, y and p are the predicted value and the true value, respectively; ， Among them, N c and N are the number of samples of category c and the number of samples of all categories respectively; According to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model and calculate the gradient g at time step t. t , ， First, calculate the exponential moving average m of the gradient t , and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, where m0 = 0, β1 is 0.9, and the update formula is as follows: m t =β1m t-1 +(1-β1)g t , Second, calculate the exponential moving average v of the squared gradient t , v0 = 0, β2 = 0.999 is the influence of controlling the square of the gradient at the previous moment, and the formula is as follows: ， Again, for m t Correct the deviation and reduce the impact of the deviation on the initial stage of training. ， v t Correction is performed to reduce the impact of deviation on the initial stage of training. The formula is as follows: ， Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 To avoid division by 0, the formula is as follows: ； Rasterize the aggregated point cloud to obtain the grid attribute map G k ={c ij |i=1,…,H;j=1,…,W}, and the number of points of each category contained in each cell , C sd is the total category set, and the semantic label category of the cell is set according to the category of the points contained in the cell.

2. The method for generating a local semantic grid map of an intelligent vehicle according to claim 1, wherein step S02 is characterized in that: Using a two-dimensional convolutional neural network, the irregular lidar point cloud data is taken as input data and rasterized to obtain multi-dimensional raster attribute features from a bird's-eye view. After that, the semantic raster map generation model is output after multiple downsampling, residual convolution and upsampling processes.

3. The method for generating a local semantic grid map of an intelligent vehicle according to claim 1, characterized in that: The point cloud segmentation algorithm is used to process the collected point cloud data offline, obtain the semantic segmentation label of the single-frame point cloud, and obtain the semantic point cloud P = {p i |i=1,,,N},where p i =(x i ,y i , z i , r i ,s i ) is one of the points, x i ,y i and z i Its coordinates, r i and i are their reflectivity and semantic labels respectively.

4. The method for generating a local semantic grid map of an intelligent vehicle according to claim 1, characterized in that: For step S013, based on the positioning information between adjacent frames, the spatial transformation relationship between adjacent frames is obtained, and the single-frame point clouds within a certain range before and after are aggregated to the current frame to obtain a dense semantic point cloud P k = {P k-n , ..., P k , ..., P k+n }.

5. The method for generating a local semantic grid map of an intelligent vehicle according to claim 2, characterized in that: After clustering the dynamic category cells in the label through the DBSCAN algorithm to obtain the target instance, it is matched with the target instance under single-frame detection. By judging whether there is a matching instance in the single-frame point cloud, the annotation of the invisible target is identified and screened for optimization processing.

Images