A Fast Semantic Map Construction Method for Unstructured Roads

Abstract

The present invention relates to a method for rapidly constructing a semantic map for unstructured roads, belonging to the technical field of autonomous driving technology, and solves the problems of low accuracy and slow speed in the construction of high-precision maps for existing mining area scenarios. The semantic map model of the present invention rasterizes the point cloud and extracts the cylinder features of the rasterized points for raster point cloud features; through downsampling and upsampling processing, RGB branches are added at the same time. During the sampling process, multi-modal focus convolution based on RGB information is used, and the importance of the point cloud around the point cloud raster is analyzed through the RGB color information output by the RGB branch to determine the shape of the multi-modal focus convolution. The finally output features are used again by MLP for point feature extraction, fused with the initial point features, and the points within the raster are refined, improving the accuracy of point cloud recognition, resulting in a high-precision map and a fast map construction speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving technology and relates to a method for quickly constructing a semantic map for unstructured roads. Background Art

[0002] In recent years, the rapid development of artificial intelligence and new generation information technology has promoted the further improvement of key technologies for autonomous driving. Due to the characteristics of road closure and relatively simple environment in open-pit mines, it has become one of the scenarios where autonomous driving can be quickly implemented.

[0003] The key technologies for autonomous driving are divided into three major technologies: environmental perception, planning and decision-making, and tracking control. The participation of high-precision maps is indispensable in these three major technologies. In terms of perception, high-precision maps play a crucial role in aspects such as ultra-long-range perception for autonomous driving, reducing the effective ROI area, and improving perception accuracy. At the same time, high-precision maps can provide lane-level information and are the basic guarantee for planning and decision-making and tracking control.

[0004] Chinese Patent Publication No. CN113665500A, with the invention title "Method for Constructing Semantic High-Precision Map Based on Point-Line Feature Fusion Laser", mainly provides a method for fusing two-dimensional image semantic segmentation information, mapping it onto a three-dimensional space, and endowing semantic and point-line features to radar point clouds. This method extracts label information of each pixel on a two-dimensional image through image semantic segmentation, projects the obtained pixel semantic information onto a three-dimensional point cloud for depth fitting to obtain a depth map, and finally performs operations such as map construction and dynamic obstacle filtering.

[0005] Chinese Patent Publication No. CN113532417A, with the invention title "A Method for Collecting High-Precision Map for Parking Lot", mainly provides a method for collecting and constructing a high-precision map for a parking lot. It constructs a road network through vehicle chassis information and position map, collects panoramic images for alignment and semantic segmentation, and vectorizes the results and maps them into a map coordinate system.

[0006] However, the method of mapping a three-dimensional space based on two-dimensional image semantic segmentation information endows semantic and point-line features to radar point clouds. The accuracy of semantic segmentation of this method is affected by pixel resolution, and at the same time, there will inevitably be accuracy loss during the conversion between two dimensions and three dimensions.

[0007] In addition, the method of constructing a map based on multi-dimensional information such as vehicle chassis, position map, image, and ultrasonic wave is only applicable to structured roads. The obstacles and edges around the road are fixed, but it is difficult to adapt to the situation of unstructured road scenes in mines with a single scenario and frequent changes in loading and unloading areas. Summary of the Invention

[0008] In view of the above analysis, the present invention provides a method for quickly constructing a semantic map for unstructured roads to solve the problems of low accuracy and slow speed in the construction of high-precision maps for mining area scenarios in the prior art.

[0009] A method for quickly constructing a semantic map for unstructured roads according to the present invention specifically includes the following steps:

[0010] Step 1, construct a semantic segmentation model;

[0011] Step 11, obtain image and point cloud data at the same time;

[0012] Step 12, rasterize the point cloud data in a cylinder to obtain point cloud raster cells and cylindrical features of points, and obtain a tensor of features of each point cloud raster cell based on the cylindrical features of points and an MLP multi-layer neural network;

[0013] Step 13, downsample the points in the point cloud raster cells;

[0014] Step 131, input the tensor of features of each point cloud raster cell into an asymmetric residual block; output a tensor of sparse convolution features;

[0015] Step 132, multi-modal focus convolution downsampling:

[0016] Perform sub-manifold hole convolution on each cylindrical raster to obtain the output shape at each cylindrical raster position;

[0017] Project the RGB pixel point information of the image onto the point cloud data in each point cloud raster cell, assign the RGB pixel point information to each point in each point cloud raster cell, and obtain an RGB point cloud raster cell;

[0018] Extract sparse convolution features from the points in the RGB point cloud raster cell and calculate the importance probability through the sigmoid function to obtain the third importance map I at position k in the convolution kernel k p and the center I0 of the third importance map p ;

[0019] Based on the third importance map I at position k in the convolution kernel k p and the center I of the third importance map p Select important rasters;

[0020] Generate the dynamic convolution output position p of the convolution kernel based on the output shape at each cylindrical raster position and the position of the convolution kernel of the important raster out ;

[0021] For the dynamic convolution output position p of the convolution kernel outPerform multi-modal focus convolution to obtain the tensor of the downsampled output features;

[0022] Step 14: Upsample the points in the point cloud grid cell;

[0023] Step 141: Multi-modal focus convolution upsampling:

[0024] Input the tensor of the downsampled output features, and perform multi-modal focus deconvolution based on the RGB point cloud grid cell to obtain the vector of the features after deconvolution;

[0025] Concatenate the tensor of the downsampled output features and the tensor of the features after deconvolution to obtain the tensor of the concatenated features; Input the tensor of the concatenated features into the asymmetric residual block; Obtain the upsampled output grid features;

[0026] Step 142: Upsample the obtained upsampled output grid features to obtain the tensor of the superimposed features of downsampling and upsampling for each point;

[0027] Step 15: Use the MLP point features of each point obtained in Step 12 to refine the tensor of the superimposed features of downsampling and upsampling for each point; Use the loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of that point;

[0028] Step 2: Construct a real-time semantic map;

[0029] Collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle;

[0030] Use the semantic segmentation model constructed in Step 1 to label each point in each frame of the obtained real-time point cloud data to form the key frame semantic point cloud;

[0031] Overlay all the key frame semantic point clouds to obtain the semantic map, and downsample the semantic map to generate the semantic RGB map.

[0032] Optionally, the point label includes an obstacle label, a ground label, and a retaining wall label.

[0033] Optionally, it further includes Step 3: The step of correcting the semantic RGB map, and the specific steps are as follows:

[0034] Step 31: Search for the labels of all points within a radius of R around the points classified as obstacle labels in the semantic RGB map through KDtree. Except for the points classified as ground labels, reset the other points to obstacle labels; Search for the original labels of N points around the points with the reset labels through KNN, and count the category of the label with the most categories of the original labels at this time as the updated label of the point reset to the obstacle label; Form the obstacle semantic map with all the points after the updated labels;

[0035] Step 32: Filter out dynamic obstacles to obtain an output semantic map;

[0036] Input the current frame obstacle semantic map and the previous frame obstacle semantic map;

[0037] If the labels of the pixels at the same position in the obstacle semantic maps of two adjacent frames are different, it is assumed that there is a dynamic obstacle in the obstacle semantic maps of the two adjacent frames;

[0038] Add a penalty term odds(p) to the stability term in the recursive Bayesian filter penalty ), obtain the stability probability l of the dynamic obstacle at the current moment s (t) :

[0039]

[0040] Among them, l s (t-1) is the stability probability of the dynamic obstacle at the previous moment; P stable and p prior are the probability of known stable map bins and the prior probability of known map bins, respectively; p penalty is the point cloud noise penalty term; α is the angle between the normal of the range image and the normal of the depth image, d is the difference between the obstacle semantic map and the range image; σ α 2 is the noise of α; σ d 2 is the noise of d;

[0041] According to the stability probability l of the dynamic obstacle at the current moment s (t) Determine whether there are dynamic obstacles in the current frame obstacle semantic map, l τ To set the stability threshold, if There are no dynamic obstacles, and the obstacle semantic map of the frame without dynamic obstacles constitutes the output semantic map.

[0042] Optionally, when constructing a real-time semantic map, the calibrated GNSS / IMU integrated navigation and positioning system and lidar of the unmanned vehicle are used to collect real-time GPS-IMU data and real-time radar point cloud data of the unmanned vehicle.

[0043] Optionally, the calibration method for the GNSS / IMU integrated navigation and positioning system and lidar of the unmanned vehicle is:

[0044] Obtain the initial rotation and translation matrix of the GNSS / IMU integrated navigation and positioning system and the lidar of the driverless vehicle; the rotation and translation matrix is the initial deviation and initial attitude angles in the XYZ directions of the lidar coordinate system and the GPS vehicle body coordinate system, where the attitude angles include yaw angle, roll angle, and pitch angle;

[0045] Install the lidar at the front and / or rear of the autonomous vehicle, and install the GNSS / IMU integrated navigation and positioning system on the vehicle body to measure the measurement deviations in the XYZ directions between the lidar position and the position of the vehicle body center; park the vehicle on the road surface, place a chess calibration board vertically in front of the center line in the vehicle body's forward direction, extract the point features of the calibration board corner points, fit the plane where the calibration board is located according to the extracted point features of the corner points, obtain the measurement attitude angles of the GPS vehicle body coordinate system and the lidar coordinate system, and determine the rotation and translation conversion relationship between the GPS vehicle body coordinate system and the lidar coordinate system of the driverless vehicle based on the measurement deviations in the XYZ directions at their positions and the measurement attitude angles;

[0046] Calibrate the initial rotation and translation matrix of the GNSS / IMU integrated navigation and positioning system and the lidar of the driverless vehicle based on the rotation and translation conversion relationship.

[0047] Optionally, extract the edge features and plane features of each frame of input real-time radar point cloud data; obtain the real-time position and pose information of the current driverless vehicle through real-time GPS-IMU data, and use the initial rotation and translation matrix to transform all key frame semantic point clouds into the world coordinate system; if the NDT matching score between two frames of semantic point clouds is less than the threshold, use the edge features and plane features to perform matching between multiple key frame semantic point clouds to optimize the parameters of the initial rotation and translation matrix.

[0048] Compared with the prior art, the present invention can at least achieve one of the following beneficial effects:

[0049] (1) The method of the present invention uses cylindrical grid sampling combined with point cloud to extract the features of the original data, overcomes the sparsity and uneven density of the point cloud, and balances the accuracy and speed of the feature extraction network;

[0050] (2) The method of the present invention inputs the three-dimensional data of the cylinder features ρ, θ, z into the BackBone for grid point cloud feature extraction. Through four downsamplings and three upsamplings, and adding an RGB branch at the same time, the features of each downsampling are fused with the corresponding features of the corresponding upsampling through a semantic understanding module based on multi-dimensional decomposition. During the sampling process, multi-modal focus convolution based on RGB information is used, and the RGB color information output through the RGB branch is used to analyze the importance of the point cloud around the point cloud grid to determine the shape of the multi-modal focus convolution. Finally, the output features are used again by the MLP for point feature extraction, which is fused with the initial point features to refine the points within the grid, improving the accuracy of point cloud recognition;

[0051] (3) The method of the present invention processes the abnormal points based on the boundary erosion algorithm of semantic segmentation. The model extracted by the semantic segmentation algorithm classifies the single-frame point cloud data, and boundary erosion is used to process the segmented abnormal points, optimizing the semantic segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention.

[0053] Figure 1 is a flowchart of the fast semantic map construction method of the present invention;

[0054] Figure 2 is a schematic diagram of the cylinder grid point cloud of the present invention;

[0055] Figure 3 is a flowchart of the downsampling of the present invention;

[0056] Figure 4 is an upsampling flowchart of the present invention;

[0057] Figure 5 is an asymmetric residual flowchart of the present invention;

[0058] Figure 6 is a multi-modal focus convolution flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0060] A specific embodiment of the present invention, as Figure 1-6 , discloses a fast semantic map construction method for unstructured roads, and the specific steps include:

[0061] Step 1, construct a semantic segmentation model;

[0062] Step 11: Make the output frequencies of the images and the point cloud data consistent to obtain the images and the point cloud data at the same moment, thereby achieving the time synchronization of the images and the point cloud data.

[0063] Preferably, the images are obtained by a camera, and the point cloud data is obtained by a lidar.

[0064] Step 12: Rasterize the point cloud data of the cylinder to obtain the point cloud raster cells and the cylinder features of the points. Based on the cylinder features of the points and the MLP multi-layer neural network, obtain the feature tensors of each point cloud raster cell.

[0065] As Figure 2 shown, divide the point cloud data from the Cartesian coordinate system according to the cylinder raster to obtain the cylinder raster; convert the points in the cylinder raster point cloud from the coordinates in the Cartesian coordinate system to the cylinder coordinates.

[0066] Optionally, extract the point-wise features of the point cloud within the cylinder raster through the MLP multi-layer neural network based on the point-wised object detection method. The specific steps are as follows:

[0067] Divide the point cloud data from the Cartesian coordinate system according to the cylinder raster to obtain the point cloud raster cells; obtain the cylinder features of the points within the point cloud raster cells. The expression is:

[0068]

[0069]

[0070] z’z’ = z.

[0071] Where x is the X-axis coordinate of the point in the Cartesian coordinate system; y is the Y-axis coordinate of the point in the Cartesian coordinate system; z is the Z-axis coordinate of the point in the Cartesian coordinate system; ρ is the horizontal radius of the point from the Z-axis in the Cartesian coordinate system; θ is the azimuth angle of the point relative to the X-axis - Y-axis plane; z’ is the height difference between the point and the origin of the Cartesian coordinate system; map all points into the cylinder raster.

[0072] At the same time, input the cylinder features ρ, θ, and z’ of all points into the MLP multi-layer neural network to perform point-based MLP feature extraction on the point cloud data, raise the 3D point features of the point cloud to C in dimensional point features, and perform max pooling on the points within the point cloud raster cells through the max pooling layer to obtain C in dimensional MLP point features, and output the tensor of feature C in , H, W, L, where H, W, and L are the radius, azimuth angle, and height of the point cloud raster cell respectively, and C in is the dimension of the point;

[0073] Step 13: Downsample the points of the point cloud raster cells.

[0074] Step 131: Input the tensors of feature C in , H, W, L into the asymmetric residual block; perform horizontal 3D convolution and vertical 3D convolution on feature C in , H, W, L tensors; where, for horizontal 3D convolution, perform feature extraction with a 3×1×3 convolution kernel and a 1×3×3 convolution kernel in sequence; for vertical 3D convolution, perform feature extraction with a 1×3×3 convolution kernel and a 3×1×3 convolution kernel in sequence; simultaneously increase the dimension of the input C in , and output a sparse convolution feature of 2*C in , H, W, L tensors; traverse each point in each point cloud grid cell.

[0075] Use the asymmetric residual block to extract features from the points within the cylinder grid. This enhances the ability to represent the points within the grid, thereby enhancing the robustness of the network. At the same time, compared with traditional 3D convolution, the asymmetric residual extraction of features reduces the computational cost.

[0076] Step 132: Multimodal focus convolution downsampling:

[0077] Perform submanifold hollow convolution on each cylinder grid, that is, when the center point position p of each convolution kernel overlaps with the input point position P of the non-empty cylinder grid in , perform a convolution operation on the center point of each convolution kernel; where, perform submanifold hollow convolution Y p on each cylinder grid, and the expression is:

[0078]

[0079] where, x p is the input sparse convolution feature at the center point position p of the convolution kernel, x p = (2C inp , H p , W p , L p ); w k is the weight at position k in the convolution kernel, d is the spatial dimension, preferably, the spatial dimension is 3; K d is the weight of the submanifold hollow convolution kernel, preferably, when K = 3, K d = 3 3 .

[0080] For the submanifold hollow convolution at each cylinder grid position, the output shape of each cylinder grid position is:

[0081]

[0082]

[0083] Among them, P(p,K d )={p+k|k∈K d}.

[0084] Neighboring point importance selection:

[0085] The image is converted to the coordinate system of the point cloud data through the conversion matrix, the RGB pixel information of the image is projected onto the point cloud data in each point cloud grid unit, and the RGB pixel information is assigned to each point in each point cloud grid unit to obtain the RGB point cloud grid unit; thereby, the image data and the point cloud data are spatially synchronized;

[0086] Importance Mapping I p : Perform sparse convolution feature extraction on the points of the RGB point cloud grid unit and calculate the importance probability through the sigmoid function to obtain the cubic importance map at position k in the convolution kernel and the center of the cubic importance map Among them, the sparse convolution kernel weight and the submanifold hole convolution kernel weight K d same.

[0087] Importance Mapping I p Involves the sparse convolution feature 2*C input at the convolution kernel center position p in ,H,W,L tensors, RGB image information, thereby obtaining the importance of the surrounding candidate grid output features. As a result, the regular sparse conv and submanifold sparse conv void convolution can be balanced, reducing the amount of calculation while ensuring that the connectivity information is not lost.

[0088] Select important grids:

[0089] When the center of the three-dimensional importance map output by the RGB point cloud grid unit When it is greater than or equal to the importance threshold τ, the RGB point cloud grid unit is an important grid, and the position of the convolution kernel of the important grid is obtained:

[0090]

[0091] Among them, P im is the position of the convolution kernel of the important grid; is the output shape of the convolution kernel of the important grid; Generate convolution kernel weights for important grids.

[0092] The grid importance includes important, relatively important, and unimportant grids; the unimportant grid has no point cloud data in the point cloud grid cell, the relatively important grid has point cloud data in the point cloud grid cell, but the RGB feature extraction of the image is background point cloud; the important grid has point cloud in the point cloud grid cell, and the RGB feature extraction is target point cloud.

[0093] Among them, the selected important grids generate the dynamic output shape of the convolution kernel from the three importance maps and is determined by the expression:

[0094]

[0095] Finally, based on the output shape at each cylinder grid position and the position of the convolution kernel of the important grid, the dynamic convolution output position p of the convolution kernel is generated out :

[0096]

[0097] Perform convolution with a stride of 2 on the dynamic convolution output position p of the convolution kernel out to obtain p out The output feature is a tensor of 2*C in , H / 2, W / 2, L / 2; perform four downsamplings to obtain the output feature of the first downsampling 2*C in , H / 2, W / 2, L / 2 tensor, the output feature of the second downsampling 4*C in , H / 4, W / 4, L / 4 tensor, the output feature of the third downsampling 8*C in , H / 8, W / 8, L / 8 tensor, and the output feature of the fourth downsampling 16*C in , H / 16, W / 16, L / 16 tensor.

[0098] Step 14: Upsample the points in the point cloud grid cell;

[0099] Step 141: Multimodal focus convolution upsampling:

[0100] Input the output feature of the fourth downsampling 16*C in , H / 16, W / 16, L / 16 tensor, and perform multimodal focus convolution upsampling. The specific process is the same as 132. First, perform submanifold dilated convolution, and obtain the final transposed convolution shape P based on the three importance maps obtained from the RGB point cloud grid cell o ‘ ut ; Based on this transposed convolution shape P o ‘ ut Perform transposed convolution with a stride of 2 to output the feature 8*C after transposed convolution inA tensor of H / 8, W / 8, L / 8;

[0101] Step 142: Concatenate the third downsampled output feature 8*C in the downsampled output features with the deconvolved feature 8*C in , a tensor of H / 8, W / 8, L / 8 to obtain a concatenated feature 16*C in , a tensor of H / 8, W / 8, L / 8; Input the concatenated feature 16*C in , a tensor of H / 8, W / 8, L / 8 into an asymmetric residual block; Horizontally and vertically three-dimensional convolve the concatenated feature 16*C in , a tensor of H / 8, W / 8, L / 8; Among them, for horizontal three-dimensional convolution, successively perform feature extraction with 3×1×3 and then 1×3×3 convolutional kernels; for vertical three-dimensional convolution, successively perform feature extraction with 1×3×3 and then 3×1×3 convolutional kernels; At the same time, reduce the input dimension and output an upsampled feature 8*C in , a tensor of H / 8, W / 8, L / 8 to obtain an upsampled output grid feature; in , a tensor of H / 8, W / 8, L / 8

[0102] Perform upsampling three more times on the obtained upsampled output grid feature to obtain the superimposed feature C of downsampling and upsampling for each point in , a tensor of H, W, L.

[0103] An asymmetric three-dimensional multi-modal focus convolution network is established by superimposing the downsampling block and the upsampling block.

[0104] The present invention uses an asymmetric residual block to strengthen the horizontal and vertical kernels, thereby enhancing the robustness of the cylinder grid.

[0105] Step 15: Use the MLP point features obtained in Step 12 to refine the superimposed feature of downsampling and upsampling for each point; Use a loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of that point; Use a Point-wised refinement module to reclassify the point labels, thereby reducing the problem of information loss caused by incorrect label division when dividing labels for the cylinder grid.

[0106] Specifically, the point labels include obstacle labels, ground labels, and retaining wall labels.

[0107] Step 2: Calibrate the GNSS / IMU integrated navigation and positioning system of the driverless vehicle and the lidar;

[0108] Obtain the initial rotation and translation matrix of the GNSS / IMU integrated navigation and positioning system and lidar of the driverless vehicle; the rotation and translation matrix is the initial deviation and initial attitude angles in the XYZ directions between the lidar coordinate system and the GPS vehicle body coordinate system, where the attitude angles include yaw angle, roll angle, and pitch angle;

[0109] Install the lidar at the front and / or rear of the autonomous vehicle, and install the GNSS / IMU integrated navigation and positioning system on the vehicle body to measure the measurement deviation in the XYZ directions between the lidar position and the position of the vehicle body center; park the vehicle on an open and flat road surface, place a checkerboard calibration board perpendicular to the center line 10 meters in front of the center line in the forward direction of the vehicle body, extract the point features of the checkerboard calibration board corner points, fit the plane where the checkerboard is located according to the extracted point features of the corner points, obtain the measurement attitude angles of the GPS vehicle body coordinate system and the lidar coordinate system, and determine the rotation and translation conversion relationship between the GPS vehicle body coordinate system and the lidar coordinate system of the driverless vehicle according to the measurement deviation in the XYZ directions of their positions and the measurement attitude angles;

[0110] Unify the coordinate systems of the GNSS / IMU integrated navigation and positioning system and lidar of the driverless vehicle based on the rotation and translation conversion relationship obtained by calibration.

[0111] Step 3: Construct a real-time semantic map

[0112] Use the calibrated GNSS / IMU integrated navigation and positioning system and lidar of the driverless vehicle to collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle respectively;

[0113] Extract the edge features and plane features of each frame of input real-time radar point cloud data according to roughness; save the edge features and plane features of each frame of real-time radar point cloud data.

[0114] Optionally, use radar odometry to extract features from each frame of real-time radar point cloud data.

[0115] Use the semantic segmentation model constructed in Step 1 to label each point in the obtained each frame of real-time point cloud data to form key-frame semantic point clouds; delete the other data between the key-frame semantic point clouds and use the key-frame semantic point clouds for estimation;

[0116] Obtain the real-time position and pose information of the current driverless vehicle through the real-time GPS-IMU data, and convert all key-frame semantic point clouds to the world coordinate system using the initial rotation and translation matrix; use the NDT feature matching effect to judge whether to optimize the initial rotation and translation matrix parameters. If the NDT matching score between two frames of semantic point clouds is less than the threshold, use the edge features and plane features to perform matching between multiple key-frame semantic point clouds and optimize the initial rotation and translation matrix parameters.

[0117] Overlay all key-frame semantic point clouds to obtain a semantic map, downsample the semantic map, and generate a semantic RGB map.

[0118] Step 4. Correct the semantic RGB map:

[0119] When obtaining the semantic RGB map using the above method, there may be cases of misdetection or small mIOU for dynamic obstacles. The boundary erosion algorithm is used to restore the dynamic obstacles in the optimized single-frame semantic RGB map. The specific steps are as follows:

[0120] Step 41. Optimization of obstacle labels in the semantic RGB map:

[0121] Search for the labels of all points within a radius of R around the points classified as obstacle labels in the semantic RGB map through KDtree. Except for the points classified as ground labels, set other points as obstacle labels; after successful setting, search for the original labels of N points around each point re-classified as an obstacle label through KNN, and count the category of the label with the most occurrences of the original labels at this time as the new label of the point re-classified; form an obstacle semantic map with all points whose labels are updated.

[0122] Step 42. Filter out dynamic obstacles to obtain the output semantic map

[0123] Input the current-frame obstacle semantic map and the previous-frame obstacle semantic map;

[0124] If the labels of the pixel points at the same position in the adjacent two-frame obstacle semantic maps are different, assume that there are dynamic obstacles in the adjacent two-frame obstacle semantic maps;

[0125] Judge the probability of the existence of dynamic obstacles in the assumed adjacent two-frame obstacle semantic maps through a recursive Bayesian filter. The specific judgment method is as follows:

[0126] Add a penalty term odds(p penalty ) to the stability term in the recursive Bayesian filter to obtain the stability probability l of dynamic obstacles at the current moment s (t) :

[0127]

[0128] where l s (t-1) is the stability probability of dynamic obstacles at the previous moment; P stable and p prior are the known stable map patch probability and the known map patch prior probability respectively; p penaltyis the point cloud noise penalty term; α is the angle between the distance image normal and the depth image normal, d is the difference between the obstacle semantic map and the distance image; σ α 2 is the noise of α; σ d 2 is the noise of d.

[0129] According to the stability probability l of the dynamic obstacle at the current moment s (t) judge whether there is a dynamic obstacle in the obstacle semantic map of the current frame, l τ is the set stability threshold, if then there is a dynamic obstacle, and the corresponding obstacle points are deleted; if then there is no dynamic obstacle, and the obstacle semantic map of the frame without dynamic obstacles constitutes the output semantic map.

[0130] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A rapid semantic map construction method for unstructured roads, characterized in that, The specific steps include: Step 1: Construct a semantic segmentation model; Step 11: Obtain image and point cloud data at the same time; Step 12: Rasterize the cylindrical point cloud data to obtain point cloud raster cells and the cylindrical features of points. Based on the cylindrical features of points and the MLP multi-layer neural network, obtain the tensor of the features of each point cloud raster cell; Step 13: Downsample the points in the point cloud raster cells; Step 131: Input the tensor of the features of each point cloud raster cell into the asymmetric residual block; output the tensor of the sparse convolution features; Step 132: Multimodal focus convolution downsampling: Perform submanifold hole convolution on each cylindrical raster to obtain the output shape at each cylindrical raster position; Project the RGB pixel point information of the image onto the point cloud data in each point cloud raster cell, assign the RGB pixel point information to each point in each point cloud raster cell, and obtain the RGB point cloud raster cell; Extract sparse convolution features from the points in the RGB point cloud raster cell and calculate the importance probability through the sigmoid function to obtain the three importance maps I at position k in the convolution kernel k p and the center I0 of the three importance maps p ; Based on the third importance map I at position k in the convolutional kernel k p and the center I0 of the third importance map p Select important grids; Generate the dynamic convolution output position p of the convolution kernel based on the output shape at each cylinder grid position and the position of the convolution kernel of the important grid out ; For the dynamic convolution output position p of the convolution kernel out perform multi-modal focus convolution to obtain the tensor of the downsampled output feature; Step 14: Upsample the points in the point cloud raster cells; Step 141: Multimodal focus convolution upsampling: Input the tensor of the downsampling output features, and perform multimodal focus deconvolution based on the RGB point cloud raster cell to obtain the vector of the features after deconvolution; Concatenate the tensor of the downsampling output features and the tensor of the features after deconvolution to obtain the tensor of the concatenated features; input the tensor of the concatenated features into the asymmetric residual block; obtain the upsampling output raster features; Step 142: Upsample the obtained upsampling output raster features to obtain the tensor of the superimposed features of downsampling and upsampling for each point; Step 15: Use the MLP point features of each point obtained in Step 12 to refine the tensor of the superimposed features of downsampling and upsampling for each point; use the loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of the point; Step 2: Construct a real-time semantic map; Collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle; Use the semantic segmentation model constructed in Step 1 to label each point in each frame of the obtained real-time point cloud data to form the key frame semantic point cloud; Superimpose all the key frame semantic point clouds to obtain the semantic map, and downsample the semantic map to generate the semantic RGB map.

2. The rapid semantic map construction method according to claim 1, wherein The point labels include obstacle labels, ground labels, and retaining wall labels.

3. The method for constructing a fast semantic map according to claim 2, wherein It also includes Step 3: The step of correcting the semantic RGB map, and the specific steps are as follows: Step 31: Search for the labels of all points within a radius of R around the points classified as obstacle labels in the semantic RGB map through KDtree. Except for the points classified as ground labels, reset the other points as obstacle labels; search for the original labels of N points around the points with the reset labels through KNN, and count the category of the label with the most categories of the original labels at this time as the updated label of the point reset as the obstacle label; form the obstacle semantic map with all the points after the updated labels; Step 32: Filter out the dynamic obstacles to obtain the output semantic map; Input the current frame obstacle semantic map and the previous frame obstacle semantic map; If the labels of the pixel points at the same position in the obstacle semantic maps of two adjacent frames are different, it is assumed that there are dynamic obstacles in these two adjacent frames of obstacle semantic maps. Add a penalty term odds(p penalty ) to the stability term in the recursive Bayesian filter to obtain the stability probability l of the dynamic obstacle at the current moment s (t) : where l s (t-1) is the stability probability of the dynamic obstacle at the previous moment; P stable and p prior are the known stable map element probability and the prior probability of the known map element respectively; p penalty is the point cloud noise penalty term; α is the angle between the normal of the distance image and the normal of the depth image, and d is the difference between the obstacle semantic map and the distance image; σ α 2 is the noise of α; σ d 2 is the noise of d; According to the stability probability l of the dynamic obstacle at the current moment s (t) Determine whether there is a dynamic obstacle in the obstacle semantic map of the current frame, l τ Let be the set stability threshold. If Then there is no dynamic obstacle, and the obstacle semantic map of the frame without dynamic obstacles constitutes the output semantic map.

4. The rapid semantic map construction method according to any one of claims 1-3, characterized in that When constructing a real-time semantic map, the calibrated GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle are used to collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle.

5. The rapid semantic map construction method according to claim 4, characterized in that The calibration method for the GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle is as follows: Obtain the initial rotation and translation matrix of the GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle; the rotation and translation matrix is the initial deviation and initial attitude angles of the XYZ directions between the lidar coordinate system and the GPS vehicle body coordinate system, where the attitude angles include yaw angle, roll angle, and pitch angle. Install the lidar at the front and / or rear of the autonomous vehicle, and install the GNSS / IMU combined navigation and positioning system on the vehicle body to measure the measurement deviation of the XYZ directions between the lidar position and the position of the vehicle body center; park the vehicle on the road surface, place a chess calibration board vertically in front of the center line in the vehicle body's forward direction, extract the point features of the calibration board corners, fit the plane where the calibration board is located according to the extracted point features of the corners, obtain the measurement attitude angles of the GPS vehicle body coordinate system and the lidar coordinate system, and determine the rotation and translation conversion relationship between the GPS vehicle body coordinate system and the lidar coordinate system of the driverless vehicle based on the measurement deviation of the XYZ directions at their positions and the measurement attitude angles. Calibrate the initial rotation and translation matrix of the GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle based on the rotation and translation conversion relationship.

6. The rapid semantic map construction method according to claim 5, characterized in that, Extract the edge features and plane features of each frame of the input real-time radar point cloud data. Obtain the real-time position and pose information of the current driverless vehicle through the real-time GPS-IMU data, and use the initial rotation and translation matrix to transform all key-frame semantic point clouds into the world coordinate system; if the NDT matching score between two frames of semantic point clouds is less than the threshold, then use the edge features and plane features to perform matching between multiple key-frame semantic point clouds to optimize the parameters of the initial rotation and translation matrix.

Images