A Method for Constructing Semantic Maps and Real-time Extracting Boundaries for Open-pit Mining Areas

Abstract

The present invention relates to a method for constructing a semantic map and real-time extraction of boundaries for open-pit mining areas, belonging to the field of autonomous driving technology, and is used to solve the problems in existing autonomous driving technologies that the extraction of driving area boundaries is inaccurate and slow, and the problem that the driving area boundaries cannot be extracted while performing semantic segmentation on images. The method of the present invention rasterizes point clouds, and extracts raster point cloud features from the cylindrical features of the rasterized points; through downsampling and upsampling processes, and adding an RGB branch at the same time to obtain a semantic RGB map. Based on the boundary extraction architecture of the semantic RGB map, when constructing the map, it analyzes the points and grid features within the grids divided in the semantic segmentation network, extracts boundaries through their features, and adopts control point fitting to smooth the boundaries, improving the accuracy and speed of unstructured road boundary extraction.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and relates to a method for constructing a semantic map and real-time extracting boundaries for open-pit mining areas. Background Art

[0002] In recent years, the rapid development of artificial intelligence and new-generation information technologies has promoted the further improvement of key autonomous driving technologies. Due to the characteristics of closed roads and relatively simple environments, open-pit mining areas have become one of the scenarios where autonomous driving can be quickly implemented.

[0003] The key technologies of autonomous driving can be divided into three major technologies: environmental perception, planning and decision-making, and tracking control. In terms of perception, road boundaries can narrow the effective ROI area. Therefore, the automatic and accurate extraction of road boundaries plays a crucial role in the rapid implementation of autonomous driving.

[0004] Chinese Patent Publication No. CN108320293A, with the invention title of "A Fast Point Cloud Boundary Extraction Method Combining an Improved Particle Swarm Optimization Algorithm", uses the particle swarm optimization algorithm to find all boundary feature points in the optimal model by selecting the uniformity of point distribution within the K-neighborhood to complete boundary extraction. However, this method searches for boundaries through the particle swarm optimization algorithm, and continuously searches for the optimal boundary feature points in a traversal manner, which is time-consuming and prone to falling into local optimal points; in addition, it is necessary to manually set the inertia weight to avoid the algorithm falling into local optimality, reducing the automation degree of boundary extraction.

[0005] Chinese Patent Publication No. CN114120260A, with the invention title of "Method and System for Identifying Driveable Areas, Computer Equipment, Storage Medium", the driveable area extraction method involved in this system mainly performs semantic segmentation on images through a semantic segmentation model to obtain the driveable area, and fuses the initial driveable area with the historical driveable area to obtain the current driveable area. The identification of the driveable area by this method is mainly through semantic segmentation, but it does not extract and optimize the road boundaries. The road driveable area obtained only from two-dimensional images has the problem of instability and is prone to segmentation errors. Summary of the Invention

[0006] In view of the above analysis, the present invention provides a method for constructing a semantic map and real-time extracting boundaries for open-pit mining areas, to solve the problems in existing autonomous driving technologies that the extraction of driving area boundaries is inaccurate and slow, and the problem that the driving area boundaries cannot be extracted while performing semantic segmentation on images.

[0007] A method for constructing a semantic map and real-time extracting boundaries for open-pit mining areas according to the present invention specifically includes the following steps:

[0008] Step 1: Construct a semantic segmentation model:

[0009] Step 11: Obtain the image and point cloud data at the same time;

[0010] Step 12: Rasterize the point cloud data of the cylinder to obtain the point cloud raster cells and the cylinder features of the points. Based on the cylinder features of the points and the MLP multi-layer neural network, obtain the tensor of the features of each point cloud raster cell;

[0011] Step 13: Downsample the points of the point cloud raster cells:

[0012] Step 131: Input the tensor of the features of each point cloud raster cell into the asymmetric residual block; output the tensor of the sparse convolution features;

[0013] Step 132: Project the RGB pixel point information of the image onto the point cloud data in each point cloud raster cell to obtain the RGB point cloud raster cells; perform multi-modal focus convolution downsampling based on the RGB point cloud raster cells:

[0014] Step 14: Upsample the points of the point cloud raster cells:

[0015] Step 141: Perform multi-modal focus convolution upsampling based on the RGB point cloud raster cells:

[0016] Step 142: Upsample the obtained upsampling output raster features to obtain the tensor of the downsampling and upsampling superposition features of each point;

[0017] Step 15: Use the MLP point features of each point obtained in Step 12 to refine the tensor of the downsampling and upsampling point superposition features of each point; use the loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of that point;

[0018] Step 2: Construct a real-time semantic map:

[0019] Collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle;

[0020] Use the semantic segmentation model constructed in Step 1 to label each point in the acquired real-time point cloud data per frame to form the key frame semantic point cloud;

[0021] Overlay all the key frame semantic point clouds to obtain the semantic map and generate the optimized single-frame semantic RGB map;

[0022] Step 3: Extract the boundary in real time:

[0023] Based on the semantic segmentation model in Step 1, construct a raster map when generating the optimized single-frame semantic RGB map, extract the boundary of the raster map, and use control points to fit and smooth the boundary to obtain the final boundary.

[0024] Optionally, before real-time boundary extraction, there is also a step of dynamic obstacle filtering, and the specific steps are as follows:

[0025] Judge the labels of each point in the optimized single-frame semantic RGB map, and exclude the non-ground and dynamic obstacle point clouds in the optimized single-frame semantic RGB map;

[0026] Obtain the superimposed semantic RGB local map by superimposing the weights of the optimized multi-frame semantic maps:

[0027] Traverse the superimposed semantic RGB local map, and finally obtain the corrected local map point cloud to form a corrected superimposed semantic map;

[0028] When real-time boundary extraction is performed, a grid map is constructed for the corrected superimposed semantic map, the boundary of the grid map is extracted, and control points are taken to fit and smooth the boundary to obtain the final boundary.

[0029] Optionally, the specific steps for obtaining the superimposed semantic RGB local map by superimposing the weights of the optimized multi-frame semantic maps are as follows: Save the vehicle GPS points of the current single-frame semantic RGB map and the point cloud data of the current single-frame semantic RGB map and the next four single-frame semantic RGB maps;

[0030] Taking the vehicle GPS point of the current single-frame semantic RGB map as the origin, convert the point cloud data of the aforementioned five frames of single-frame semantic RGB maps to the world coordinate system for NDT matching and then superimpose them; when superimposing, different weights are given to the point cloud data of the first to fifth frames of semantic RGB maps respectively;

[0031] Retain the superimposed point cloud within the threshold range in the Y-axis direction of the point cloud data of the aforementioned five frames of single-frame semantic RGB maps that are superimposed;

[0032] Perform KNN voting on the i-th point in the superimposed point cloud, search for N points corresponding to the i-th point at a preset distance through Kdtree, and superimpose the number Num of labels of the N points around the i-th point n，label and the weight w n , if the total value K i exceeds the threshold τ, then the current i-th point l 's label k i，labe is modified to the coefficient maxk n ;

[0033] k n =w n* Num label，n ;

[0034] K i =w 1* Num label,1 +w 2*Num label,2 +…+w n* Num label，n >τ;

[0035] k i，label =max{w 1* Num label,1 ,w 2* Num label,2 ,…,w n* Num label，n};

[0036] Traverse each point in the overlapped point cloud, and perform KNN voting on each point to obtain the semantic RGB local map after superposition.

[0037] Optionally, when converting the point cloud data of the single-frame semantic RGB maps of the aforementioned five frames to the world coordinate system for NDT matching and then superposing, the weight values of the point cloud data from the first frame to the fifth frame increase linearly in sequence from the first frame to the fifth frame.

[0038] Optionally, the specific steps for real-time extraction of the boundary are as follows:

[0039] Based on the corrected superposed semantic map, obtain the vehicle GPS points of the current corrected single-frame semantic map and the previous two corrected single-frame semantic maps, and convert them to the world map through the rotation and translation matrix; judge the road boundary on the opposite side of the driving area through the slope of the vehicle GPS points of the three corrected single-frame semantic maps.

[0040] Judge the point cloud grid positions in the three corrected single-frame semantic maps that have both road labels and retaining wall labels, search for the boundary points on the opposite side of the driving area that are farthest from the vehicle GPS point in the plane grid of the three corrected single-frame semantic maps, and save them to the left two boundary point memories and the right two boundary point memories respectively.

[0041] Remove the outlier points from the road boundary points in the multiple corrected single-frame semantic maps in the left and right two boundary point memories.

[0042] Convert the point cloud in the current corrected single-frame semantic map with left and right road boundary information after removing the discrete points to the world coordinate system through the rotation and translation matrix; in the single-frame RGB map after removing the discrete points in the world coordinate system, select boundary fitting control points at preset intervals for least squares curve fitting to obtain the final boundary.

[0043] Optionally, the specific steps for removing discrete points are as follows:

[0044] Set the initial distance threshold D, and initialize the left boundary point memory and the right boundary point memory.

[0045] Taking the vehicle GPS point that corrects the single-frame semantic map in the first frame as the origin, search for the left nearest point Point that is closest to the vehicle GPS point in terms of distance. left And the right nearest point Point right ; Taking the left nearest point Point left And the right nearest point Point right As two initial points, store them in the left and right boundary point memories respectively;

[0046] Taking the left nearest point Point left As the origin, search for the left nearest point Point that does not exceed the initial distance threshold D left And store it in the left boundary point memory; Taking the right nearest point Point right As the origin, search for the right nearest point that does not exceed the initial distance threshold D and store it in the right boundary point memory; If the left nearest point Point left Or the right nearest point Point right Is a point that has been traversed or a point exceeding the threshold, then this nearest point is a discrete point, and this discrete point is excluded;

[0047] After traversing all the road boundary points, end the search to obtain all the left and right road boundary points;

[0048] Based on all the obtained left and right road boundary points, obtain the final boundary.

[0049] Optionally, when constructing a real-time semantic map, use the calibrated GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle to collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle.

[0050] Compared with the prior art, the present invention can at least achieve one of the following beneficial effects:

[0051] (1) The method of the present invention performs semantic segmentation on unstructured roads, and at the same time uses the point cloud ρ, θ and z as inputs to extract features of the point cloud, improving the accuracy of unstructured road boundary extraction;

[0052] (2) The semantic map obtained by the method of the present invention enables semantic segmentation of images while also being able to extract the driving area boundary.

[0053] (3) The method of the present invention is based on the boundary extraction architecture of semantic segmentation. When building a map, it analyzes the points and grid features within the grids divided in the semantic segmentation network, extracts the boundary through its features, and adopts control point fitting to smooth the boundary, improving the boundary extraction speed. Description of the Drawings

[0054] The drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention.

[0055] Figure 1 This is the flowchart of the semantic map construction and boundary real-time extraction method for open-pit mining areas of the present invention;

[0056] Figure 2 This is the schematic diagram of the cylinder grid point cloud feature extraction of the present invention;

[0057] Figure 3 This is the flowchart of the downsampling of the present invention;

[0058] Figure 4 This is the flowchart of the upsampling of the present invention;

[0059] Figure 5 This is the flowchart of the asymmetric residual of the present invention;

[0060] Figure 6 This is the flowchart of the multi-modal focus convolution of the present invention;

[0061] Figure 7 This is the single-frame semantic point cloud of the present invention;

[0062] Figure 8 This is the semantic map of the present invention;

[0063] Figure 9 This is the effect diagram of the boundary extraction of the present invention. Detailed implementation manners

[0064] To make the objectives, technical solutions and advantages of the present invention clearer and more complete, the following further detailed description of the present invention is provided with reference to the accompanying drawings. It should be understood that the described implementation manners are only for illustration and not for limiting the present invention.

[0065] A specific embodiment of the present invention, as Figures 1-9 , discloses a semantic map construction and boundary real-time extraction method for open-pit mining areas, and the specific steps include:

[0066] Step 1, construct a semantic segmentation model;

[0067] Step 11, make the output frequencies of the pictures and the point cloud data consistent to obtain the pictures and the point cloud data at the same moment; thereby making the time synchronization of the pictures and the point cloud data;

[0068] Preferably, the pictures are obtained by a camera, and the point cloud data is obtained by a lidar.

[0069] Step 12, perform cylinder grid quantization on the point cloud data to obtain point cloud grid units and the cylinder features of the points, and obtain the feature tensors of each point cloud grid unit based on the cylinder features of the points and the MLP multi-layer neural network;

[0070] As Figure 2As shown, divide the point cloud data in the Cartesian coordinate system according to the cylindrical grid to obtain the cylindrical grid; convert the points in the cylindrical grid point cloud from the coordinates in the Cartesian coordinate system to the cylindrical coordinates;

[0071] Optionally, based on the point-wised object detection method, extract the point-wise features of the point cloud within the cylindrical grid through the MLP multi-layer neural network. The specific steps are as follows:

[0072] Divide the point cloud data in the Cartesian coordinate system according to the cylindrical grid to obtain the point cloud grid unit; obtain the cylindrical features of the points within the point cloud grid unit. The expression is:

[0073]

[0074]

[0075] z’ = z.

[0076] Where x is the X-axis coordinate of the point in the Cartesian coordinate system; y is the Y-axis coordinate of the point in the Cartesian coordinate system; z is the Z-axis coordinate of the point in the Cartesian coordinate system; ρ is the horizontal radius of the point from the Z-axis in the Cartesian coordinate system; θ is the azimuth angle of the point relative to the X-axis - Y-axis plane; z’ is the height difference between the point and the origin of the Cartesian coordinate system; map all points into the cylindrical grid.

[0077] At the same time, input the cylindrical features ρ, θ, and z’ of all points into the MLP multi-layer neural network to perform point-based MLP feature extraction on the point cloud data, raise the 3D point features of the point cloud to C in dimensional point features, and perform max pooling on the points within the point cloud grid unit through the max pooling layer to obtain C in dimensional MLP point features, and output the tensor of features C in , H, W, L, where H, W, and L are the radius, azimuth angle, and height of the point cloud grid unit respectively, and C in is the dimension of the point;

[0078] Step 13: Downsample the points of the point cloud grid unit;

[0079] Step 131: Input the tensor of features C in , H, W, L into the asymmetric residual block; perform horizontal three-dimensional convolution and vertical three-dimensional convolution on the tensor of features C in , H, W, L; where, for the horizontal three-dimensional convolution, perform feature extraction with a 3×1×3 convolution kernel and a 1×3×3 convolution kernel in sequence; for the vertical three-dimensional convolution, perform feature extraction with a 1×3×3 convolution kernel and a 3×1×3 convolution kernel in sequence; at the same time, raise the input dimension C in , and output the sparse convolution feature 2*C in, tensors of H, W, L; iterate through each point in each point cloud grid cell.

[0080] Use an asymmetric residual block to extract features from the points within the cylinder grid. This enhances the expressive ability of the points within the grid, thereby enhancing the robustness of the network. At the same time, compared with traditional 3D convolution, the asymmetric residual extraction of features reduces the computational cost.

[0081] Step 132, multi-modal focus convolution downsampling:

[0082] Perform submanifold hole convolution on each cylinder grid, that is, when the center point position p of each convolution kernel overlaps with the input point position P of the non-empty cylinder grid in perform a convolution operation on each convolution kernel center point; among them, perform submanifold hole convolution Y on each cylinder grid p , the expression is:

[0083]

[0084] where, x p is the input sparse convolution feature at the center point position p of the convolution kernel, w k is the weight at position k in the convolution kernel, d is the spatial dimension, preferably, the spatial dimension is 3; K d is the submanifold hole convolution kernel weight, preferably, when K = 3, K d = 3 3 .

[0085] For the submanifold hole convolution at each cylinder grid position, the output shape of each cylinder grid position is:

[0086]

[0087]

[0088] where, P(p, K d ) = (p + k|k ∈ K d}.

[0089] Adjacent point importance selection:

[0090] Convert the picture to the coordinate system of the point cloud data through the transformation matrix, project the RGB pixel point information of the picture onto the point cloud data in each point cloud grid cell, assign the RGB pixel point information to each point in each point cloud grid cell, and obtain the RGB point cloud grid cell; thereby synchronizing the image data and the point cloud data space;

[0091] Importance mapping I p: Extract sparse convolutional features from the points in the RGB point cloud raster cell and calculate the importance probability through the sigmoid function to obtain the three importance maps at position k in the convolutional kernel. And the center of the three importance maps Among them, the sparse convolutional kernel weight is the same as the submanifold hole convolutional kernel weight K d Same.

[0092] Importance mapping I p Involves the sparse convolutional feature 2*C input at the center point position p of the convolutional kernel in , the tensors of H, W, L, and RGB image information, so as to obtain the importance of the surrounding candidate raster output features. Thus, it can balance the regular sparse conv and submanifold sparse conv hole convolutions, while reducing the computational amount, ensuring that the connected information is not lost.

[0093] Select important rasters:

[0094] When the center of the three importance maps output by the RGB point cloud raster cell Is greater than or equal to the importance threshold τ, this RGB point cloud raster cell is an important raster, and obtain the position of the convolutional kernel of this important raster:

[0095]

[0096] Among them, P im Is the position of the convolutional kernel of the important raster; Is the output position of the convolutional kernel of the important raster; Is the convolutional kernel weight generated by the important raster.

[0097] Raster importance includes important, relatively important, and unimportant rasters; unimportant rasters have no point cloud data in the point cloud raster cell, relatively important rasters have point cloud data in the point cloud raster cell, but the image RGB feature extraction is background point cloud; important rasters have point cloud in the point cloud raster cell, and the RGB feature extraction is target point cloud.

[0098] Among them, the selected important rasters generate the dynamic output shape of the convolutional kernel Determined by the three importance maps The expression is:

[0099]

[0100] Finally, based on the output shape at each cylinder raster position and the position of the convolutional kernel of the important raster, generate the dynamic convolutional output position p of the convolutional kernel out :

[0101]

[0102] Perform convolution with a stride of 2 on the dynamic convolution output position p of the convolution kernel to obtain p out Output a tensor of features 2*C out ,H / 2, W / 2, L / 2; perform four downsamplings to sequentially obtain the output features of the first downsampling 2*C in ,H / 2, W / 2, L / 2 tensor, the output features of the second downsampling 4*C in ,H / 4, W / 4, L / 4 tensor, the output features of the third downsampling 8*C in ,H / 8, W / 8, L / 8 tensor, the output features of the fourth downsampling 16*C in ,H / 16, W / 16, L / 16 tensor. in

[0103] Step 14: Upsample the points in the point cloud grid cell;

[0104] Step 141: Multimodal focus convolution upsampling:

[0105] Input the output features of the fourth downsampling 16*C in ,H / 16, W / 16, L / 16 tensor, perform multimodal focus convolution upsampling. The specific process is the same as 132. First, perform submanifold atrous convolution and obtain the final transposed convolution shape P' based on the three importance maps obtained from the RGB point cloud grid cell out ; Based on this transposed convolution shape P' out Perform transposed convolution with a stride of 2 to output the transposed convolution features 8*C in ’, H / 8, W / 8, L / 8 tensor;

[0106] Step 142: Concatenate the output features of the third downsampling 8*C in ,H / 8, W / 8, L / 8 tensor in the downsampling output features with the transposed convolution features 8*C in ’,H / 8, W / 8, L / 8 tensor to obtain the concatenated features 16*C in ,H / 8, W / 8, L / 8 tensor; Input the concatenated features 16*C in ,H / 8, W / 8, L / 8 tensor into the asymmetric residual block; Horizontally and vertically three - dimensional convolve the concatenated features 16*C in ,H / 8, W / 8, L / 8 tensor; Among them, for horizontal three - dimensional convolution, sequentially perform feature extraction with 3×1×3 and 1×3×3 convolution kernels; for vertical three - dimensional convolution, sequentially perform feature extraction with 1×3×3 and 3×1×3 convolution kernels; At the same time, reduce the input dimension and output the upsampled features 8*C in , a tensor of H / 8, W / 8, L / 8, to obtain an upsampled output grid feature;

[0107] Perform upsampling three more times on the obtained upsampled output grid feature to obtain the downsampled and upsampled superimposed feature C of each point in , a tensor of H, W, L.

[0108] By superimposing the downsampling block and the upsampling block, an asymmetric three-dimensional multimodal focus convolution network is established.

[0109] The present invention uses an asymmetric residual block to strengthen the horizontal and vertical kernels, thereby enhancing the robustness of the cylinder grid.

[0110] Step 15: Use the MLP point features of each point obtained in Step 12 to refine the downsampled and upsampled superimposed features of each point; use the loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of that point; use the Point-wised refinement module to reclassify the point label, thereby reducing the problem of information loss caused by incorrect label division when dividing labels for the cylinder grid.

[0111] Specifically, the point labels include obstacle labels, ground labels, and retaining wall labels.

[0112] Step 2: Calibrate the GNSS / IMU integrated navigation and positioning system of the driverless vehicle and the lidar;

[0113] Obtain the initial rotation and translation matrix of the GNSS / IMU integrated navigation and positioning system of the driverless vehicle and the lidar; the rotation and translation matrix is the initial deviation and initial attitude angle of the XYZ directions of the lidar coordinate system and the GPS vehicle body coordinate system, where the attitude angle includes yaw angle, roll angle, and pitch angle;

[0114] Install the lidar at the front and / or rear of the autonomous vehicle, and install the GNSS / IMU integrated navigation and positioning system on the vehicle body to measure the measurement deviation of the XYZ directions of the lidar position and the position where the vehicle body center is located; park the vehicle on an open and flat road surface, place a checkerboard calibration board perpendicular to the center line 10 meters in front of the center line of the vehicle body's forward direction, extract the point features of the checkerboard calibration board corner points, fit the plane where the checkerboard is located according to the extracted point features of the corner points, obtain the measurement attitude angles of the GPS vehicle body coordinate system and the lidar coordinate system, and determine the rotation and translation conversion relationship between the GPS vehicle body coordinate system and the lidar coordinate system of the driverless vehicle according to the measurement deviation of the XYZ directions of their positions and the measurement attitude angles;

[0115] Unify the coordinate systems of the GNSS / IMU integrated navigation and positioning system of the driverless vehicle and the lidar based on the obtained rotation and translation conversion relationship.

[0116] Step 3: Construct a real-time semantic map

[0117] Use the GNSS / IMU integrated navigation and positioning system and lidar of the calibrated driverless vehicle to collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle respectively;

[0118] Extract the edge features and plane features of each frame of the input real-time radar point cloud data according to roughness; save the edge features and plane features of each frame of the real-time radar point cloud data.

[0119] Optionally, use radar odometry to extract features from each frame of the real-time radar point cloud data.

[0120] Use the semantic segmentation model constructed in Step 1 to label each point in the acquired real-time point cloud data of each frame to form a key-frame semantic point cloud; delete the other data between the key-frame semantic point clouds and use the key-frame semantic point cloud for estimation;

[0121] Obtain the real-time position and pose information of the current driverless vehicle through the real-time GPS-IMU data, and use the initial rotation and translation matrix to transform all the key-frame semantic point clouds into the world coordinate system; use the NDT feature matching effect to judge whether to optimize the parameters of the initial rotation and translation matrix. If the NDT matching score between two frames of semantic point clouds is less than the threshold, use the edge features and plane features to match between multiple key-frame semantic point clouds and optimize the parameters of the initial rotation and translation matrix.

[0122] Overlay all the key-frame semantic point clouds to obtain a semantic map, and downsample the semantic map to generate an optimized single-frame semantic RGB map.

[0123] Step 4: Dynamic obstacle filtering:

[0124] Step 41: Judge the label of each point in the optimized single-frame semantic RGB map, and exclude the non-ground and dynamic obstacle point clouds in the optimized single-frame semantic RGB map.

[0125] Step 42: Weight superposition of multi-frame semantic maps:

[0126] Save the vehicle GPS points of the current single-frame semantic RGB map, as well as the point cloud data of the current single-frame semantic RGB map and the next four single-frame semantic RGB maps;

[0127] Taking the vehicle GPS point of the current single-frame semantic RGB map as the origin, transform the point cloud data of the aforementioned five frames into the world coordinate system for NDT matching and then overlay; when overlaying, give different weights to the point cloud data from the first frame to the fifth frame, and the five weight values increase linearly in turn from the first frame to the fifth frame point cloud data;

[0128] Retain the overlapping point clouds within the threshold range in the Y-axis direction among the aforementioned five-frame point cloud data. For example, if the radar is installed on the vehicle roof, only some noise points can be collected from the point clouds around the vehicle body. The effective detection distance of a single-frame point cloud is 50 m, and the distance between key frames is 2 m. Then, delete the point cloud data within 10 m (i.e., noise points) and beyond 40 m (i.e., sparse point clouds) in the Y-axis direction to reduce the impact caused by abnormal semantic segmentation;

[0129] Perform KNN (K-Nearest Neighbor algorithm) voting on the i-th point in the overlapping point clouds. Search for N points corresponding to the i-th point within a preset distance through Kdtree, and sum up the number Num of labels of the N points around the i-th point n，label and the weight w n , if the total value K i exceeds the threshold τ, then modify the label k l of the current i-th point i，labe to the coefficient maxk n ;

[0130] k n = w n* Num label，n ;

[0131] K i = w 1* Num label,1 + w 2* Num label,2 +…+ w n* Num label，n > τ;

[0132] k i，label = max{w 1* Num label,1 , w 2* Num label,2 ,…, w n* Num label，n};

[0133] Traverse each point in the overlapping point clouds, and perform KNN (K-Nearest Neighbor algorithm) voting on each point to obtain the superimposed semantic RGB local map.

[0134] Step 43, traverse the superimposed semantic RGB local map, and finally obtain the corrected local map point cloud to form the corrected superimposed semantic map.

[0135] Step 5, extraction of the driving area boundary:

[0136] Based on the semantic segmentation model in Step 1, construct a grid map when establishing the corrected superimposed semantic map, extract the inner boundary of the grid, and adopt control point fitting to smooth the boundary to improve the boundary extraction speed;

[0137] 1) Based on the corrected superposed semantic map, obtain the vehicle GPS points of the current corrected single-frame semantic map and the previous two corrected single-frame semantic maps, and transform them into the world map through the rotation and translation matrix; judge the road boundary on the opposite side of the driving area through the slopes of the GPS points of the three frames of vehicles; the slopes of the GPS points of the three frames of vehicles are the slopes of the tangents of the GPS points of the vehicle in the above-mentioned three corrected single-frame semantic maps.

[0138] 2) Judge the point cloud grid positions with both road labels and retaining wall labels in the three corrected single-frame semantic maps, search for the boundary points on the opposite side of the driving area that are farthest from the vehicle GPS point in the plane grid of the three corrected single-frame semantic maps, and save them into two point containers on the left and right respectively;

[0139] 3) Remove the outlier points from the road boundary points in the multiple corrected single-frame semantic maps in the left and right containers to obtain the left and right road boundary points;

[0140] The specific steps for removing discrete points are as follows:

[0141] A: Set the initial distance threshold D, and initialize the left boundary point container and the right boundary point container;

[0142] B: Take the vehicle GPS point of the first frame as the origin, and search for the left nearest point Point left and the right nearest point Point right as two initial points and store them in the left and right point containers;

[0143] C: Take the left nearest point Point left as the origin, search for the left nearest point not exceeding the initial distance threshold D and store it in the left boundary point container; take the right nearest point Point right as the origin, search for the right nearest point not exceeding the initial distance threshold D and store it in the right boundary point container; if the left nearest point or the right nearest point is a point that has been traversed or a point exceeding the threshold, then the nearest point is a discrete point, and the discrete point is removed;

[0144] D: End the search after traversing all the road boundary points to obtain all the left and right road boundary points.

[0145] 4) In the single-frame RGB map after removing discrete points in the world coordinate system, select boundary fitting control points every 1m for least squares curve fitting to obtain the final boundary, convert the final boundary information back to the WGS84 coordinate system and upload it to the cloud database for storage of boundary information, where the WGS84 coordinate system is the coordinate system of the original GPS or the earth coordinate system or the world coordinate system.

[0146] It can be understood that if 5 points are selected as control points for the boundary of a single-frame point cloud, the least squares method is used to fit the boundary of the five points:

[0147] The world coordinate system coordinates of the five points: (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5);

[0148] Polynomial fitting is used to obtain the polynomial coefficient θ i Thereby determining the road boundary h θ (x):

[0149] h θ (x) = θ0 + θ1x1 + θ2x2 + θ3x3 + θ4x4 + θ5x5.

[0150]

[0151] Wherein, xi and y i , are respectively the coordinates of the control points in the world coordinate system coordinates.

[0152] As described above, it is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A semantic map construction and real-time boundary extraction method for an open-pit mine three-frame corrected single-frame semantic map area facing the exposed semantic map, characterized in that, The specific steps include: Step 1: Construct a semantic segmentation model: Step 11: Obtain images and point cloud data at the same time; Step 12: Rasterize the point cloud data of the cylinder to obtain point cloud raster cells and the cylinder features of points. Based on the cylinder features of points and the MLP multi-layer neural network, obtain the tensor of the features of each point cloud raster cell; Step 13: Downsample the points of the point cloud raster cells: Step 131: Input the tensor of the features of each point cloud raster cell into an asymmetric residual block; output the tensor of the sparse convolution features; Step 132: Project the RGB pixel point information of the image onto the point cloud data in each point cloud raster cell to obtain RGB point cloud raster cells; perform multi-modal focus convolution downsampling based on the RGB point cloud raster cells: Step 14: Upsample the points of the point cloud raster cells: Step 141: Perform multi-modal focus convolution upsampling based on the RGB point cloud raster cells: Step 142: Upsample the obtained upsampled output raster features to obtain the tensor of the superimposed features of the downsampling and upsampling of each point; Step 15: Use the MLP point features of each point obtained in Step 12 to refine the tensor of the superimposed features of the downsampling and upsampling points of each point; use the loss function to obtain the class probability of each refined point, and use the highest class probability of each point as the point label of that point; Step 2: Construct a real-time semantic map: Collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle; Use the semantic segmentation model constructed in Step 1 to label each point in the acquired real-time point cloud data of each frame to form key frame semantic point clouds; Superimpose all key frame semantic point clouds to obtain a semantic map, and generate an optimized single-frame semantic RGB map; Step 3: Extract the boundary in real time: Based on the semantic segmentation model in Step 1, construct a raster map when generating the optimized single-frame semantic RGB map, extract the boundary of the raster map, and use control points to fit and smooth the boundary to obtain the final boundary.

2. The semantic map construction and boundary real-time extraction method according to claim 1, wherein Before extracting the boundary in real time, there is also a step of filtering dynamic obstacles, and the specific steps are as follows: Judge the labels of each point in the optimized single-frame semantic RGB map, and exclude the non-ground and dynamic obstacle point clouds in the optimized single-frame semantic RGB map; Superimpose the weights of the optimized multi-frame semantic maps to obtain the superimposed semantic RGB local map: Traverse the superimposed semantic RGB local map, and finally obtain the corrected local map point cloud to form the corrected superimposed semantic map; When extracting the boundary in real time, construct a raster map for the corrected superimposed semantic map, extract the boundary of the raster map, and use control points to fit and smooth the boundary to obtain the final boundary.

3. The semantic map construction and boundary real-time extraction method according to claim 2, wherein The specific steps for superimposing the weights of the optimized multi-frame semantic maps to obtain the superimposed semantic RGB local map are: save the vehicle GPS points of the current single-frame semantic RGB map and the point cloud data of the current single-frame semantic RGB map and the next four single-frame semantic RGB maps; Taking the vehicle GPS point of the current single-frame semantic RGB map as the origin, transform the point cloud data of the aforementioned five frames of single-frame semantic RGB maps into the world coordinate system for NDT matching and then superimpose them; When superimposing, different weights are given to the semantic RGB map point cloud data from the first frame to the fifth frame respectively; The superimposed point cloud within the threshold range in the Y-axis direction of the point cloud data of the single-frame semantic RGB map of the aforementioned five frames is retained; Perform KNN voting on the $i$-th point in the overlapped point cloud. Search for $N$ points within a preset distance corresponding to the $i$-th point through Kdtree, and sum up the number of labels Num of the $N$ points around the $i$-th point n，label and the weights $w$ n . If the total value $K$ i exceeds the threshold $\tau$, then modify the label $k$ of the current $i$-th point l to the coefficient $maxk$ i，labe ; n ​ k n = w n* Num label，n ; K i = w 1* Num label,1 + w 2* Num label,2 + … + w n* Num label，n > τ; k i，label = max{w 1* Num label,1 , w 2* Num label,2 , …, w n* Num label，n}; Each point in the superimposed point cloud is traversed, and each point conducts KNN voting to obtain the superimposed semantic RGB local map.

4. The semantic map construction and boundary real-time extraction method according to claim 3, characterized in that When the point cloud data of the single-frame semantic RGB map of the aforementioned five frames is converted to the world coordinate system for NDT matching and then superimposed, the weight values of the point cloud data from the first frame to the fifth frame increase linearly in sequence from the first frame to the fifth frame.

5. The semantic map construction and boundary real-time extraction method according to any one of claims 2-4, characterized in that The specific steps for real-time boundary extraction are as follows: Based on the corrected superimposed semantic map, the vehicle GPS points of the current corrected single-frame semantic map and the previous two frames of corrected single-frame semantic maps are obtained and transformed into the world map through the rotation and translation matrix; The road boundaries on the opposite side of the driving area are judged by the slopes of the vehicle GPS points of the three frames of corrected single-frame semantic maps; Judge the point cloud grid positions with both road labels and retaining wall labels in the three frames of corrected single-frame semantic maps, search for the boundary points on the opposite side of the driving area farthest from the vehicle GPS point in the plane grid of the three frames of corrected single-frame semantic maps, and save them into the left two boundary point memories and the right two boundary point memories respectively; Outlier removal is performed on the road boundary points in the multiple corrected single-frame semantic maps in the left and right two boundary point memories; The point cloud in the current corrected single-frame semantic map with left and right road boundary information after removing discrete points is transformed into the world coordinate system through the rotation and translation matrix; in the single-frame RGB map after removing discrete points in the world coordinate system, boundary fitting control points are selected at preset intervals for least squares curve fitting to obtain the final boundary.

6. The semantic map construction and boundary real-time extraction method according to claim 5, characterized in that The specific steps for discrete point removal are as follows: Set the initial distance threshold D, and initialize the left boundary point memory and the right boundary point memory; Taking the vehicle GPS point that corrects the single-frame semantic map in the first frame as the origin, search for the left nearest point Point that is closest to the vehicle GPS point in terms of distance left and the right nearest point Point right ; Use the left nearest point Point left and the right nearest point Point right as two initial points and store them in the left and right boundary point memories respectively; Using the left nearest point Point left as the origin, search for the left nearest point Point that does not exceed the initial distance threshold D left and store it in the left boundary point memory; Using the rightmost nearest point Point right as the origin, search for the rightmost nearest points within a distance not exceeding the initial distance threshold D and store them in the right boundary point memory; If the left nearest point Point left or the right nearest point Point right is a point that has been traversed or a point exceeding the threshold, then this nearest point is a discrete point, and this discrete point is removed; After traversing all road boundary points, the search ends to obtain all left and right road boundary points; Based on all the obtained left and right road boundary points, the final boundary is obtained.

7. The semantic map construction and boundary real-time extraction method according to claim 1, wherein When constructing a real-time semantic map, the calibrated GNSS / IMU combined navigation and positioning system and lidar of the driverless vehicle are used to collect the real-time GPS-IMU data and real-time radar point cloud data of the driverless vehicle.

Images