A high-precision map multi-stage semantic feature extraction method

Abstract

The application discloses a high-precision map multi-stage semantic feature extraction method, mainly carries out accurate training on the recognition model of aerial targets, ground elements and backgrounds, and carries out targeted extraction and differentiation, and the application discloses a model reasoning structure from coarse to fine by stages, step-by-step disassembly, so that the segmentation effect of a smaller number of target categories is more obvious and efficient.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to a high-precision map multi-stage semantic feature extraction method. BACKGROUND

[0002] The high-precision map is an important part of the automatic driving field, and can help the car to pre-knowledge the complex information on the road surface, determine the position, and make the car make correct decisions. In the process of automatic vector construction of the high-precision map, semantic extraction is an important link directly affecting the accuracy of the subsequent results. However, the number of different label targets in the real scene is seriously unbalanced (data long tail problem), especially the proportion of the number of road related elements is very low and there is a few orders of magnitude difference between other elements, which leads to different effects of semantic extraction and great difference between different targets. Simple data enhancement and training skills are difficult to achieve good segmentation effect.

[0003] Chinese patent CN114037966A discloses a high-precision map feature extraction method, device, medium and electronic equipment. The application extracts the corresponding road surface identification geometric features by acquiring the road images collected twice in the same section, and fuses at least the two original geometric features to obtain the target geometric features of the road surface identification. The application adopts the road image information collected multiple times. Since the image collection is affected by the collection angle, distortion will occur at the edge of the image. At the same time, the image collected by the camera lacks depth information, which will also affect the position accuracy of the subsequent algorithm extraction. Therefore, relying only on image feature extraction will result in low accuracy. The application adopts the point cloud information collected by the laser radar, has accurate ranging capability, and has natural advantages in feature extraction on the point cloud. The position accuracy will be obviously higher than that of image feature extraction.

[0004] Chinese patent CN110610650A discloses a construction method and process of point cloud semantic map based on deep learning and depth camera. The application projects the point cloud map after calibration by using the depth camera, estimates the surface normal information of the point cloud, and realizes the scene perception of the point cloud map by combining the PointSIFT module with the point cloud convolution operator for pretreatment.

[0005] The application mainly uses the method of deep learning to extract features from the point cloud information collected by the depth camera. Firstly, the patent mainly aims at indoor scenes, and the number of point clouds is less than that of outdoor scenes, and the segmentation difficulty is low. The application does not design a processing method for the data imbalance problem.

[0006] The information disclosed in this BACKGROUND section is only intended to increase an understanding of the general background of the application and should not be construed as recognition of the information being the prior art to the application. SUMMARY

[0007] The purpose of the present application is that the semantic extraction of large scenes is very difficult, and the number of targets of different labels often differs greatly, which seriously affects the final extraction result in the process of training and reasoning using a deep learning model. Therefore, the present application designs multi-stage model training and reasoning to continuously improve the model extraction accuracy of small road targets.

[0008] To achieve the above purpose, the present application provides a multi-stage semantic feature extraction method for high-precision maps, characterized in that it comprises the following steps:

[0009] In the air target extraction, the SOTA point cloud convolution semantic segmentation model Conv Point is used for model training and reasoning of elements in the air target (both the air target and the road element refer to elements that need to be extracted in high-precision map construction, for example: the air target includes: guardrails, poles, road signs, etc. The road element includes: lane lines, arrows, road markings, etc.). The model in this stage only segments out one type of label of road elements as a whole to prepare for the subsequent semantic extraction. In the preprocessing stage (the stage of data processing before model training, for example, according to the semantic label, it is found that the number of point clouds of a certain category accounts for less than 5% of the total number of point clouds, and the point cloud data containing this category needs to be duplicated), the targets are classified according to their categories, and the sample size of the training data of the semantic label less than 5% is subjected to corresponding data augmentation when training the model; due to the huge number of point clouds in outdoor scenes (the huge number refers to: the number of single-frame point clouds at a resolution (density) of 0.1 meters within a range of 100*100 meters usually exceeds one million, generally in the range of 3-5 million), in the data reasoning stage, the sliding window traversal method can be used, and the size of the window should not exceed 10*10 meters under the condition that the resolution of single-frame point clouds is not less than 0.1 meters; both the air target and the road element refer to elements that need to be extracted in high-precision map construction, for example: the air target includes: guardrails, poles, road signs, etc. The road element includes: lane lines, arrows, road markings, etc.

[0010] The road surface elements and the background are distinguished, and the model adopts the same network structure (Conv Point model. For example, the first stage adopts the Conv Point model, and the second stage also adopts the Conv Point). The main purpose of this stage is to separate the road surface elements and other areas of the road surface. The main purpose of this stage is two-fold: on the one hand, the noise of the road surface is filtered out to improve the segmentation result of the subsequent stage; on the other hand, through the separation of the road surface elements and other areas of the road surface, the geometric information of the road surface elements is enhanced, and the segmentation effect of the subsequent stage is improved; the model inference of the second stage also adopts the sliding window traversal method. Due to the elimination of the aerial target, the number of point clouds is significantly reduced, and therefore the size of the window is increased by 1 times compared with the first stage model. In order to avoid the lack of point clouds of the road surface elements caused by the model segmentation, it is necessary to traverse each point cloud of the road surface elements after the model inference of this stage. The knn nearest neighbor searching method is used to find the nearest point cloud and make its label as the road surface element, which has a similar effect to the dilation in image morphological processing, and increases the range of the road surface element point cloud.

[0011] The road surface element fine-grained segmentation adopts a fusion model to perform fine-grained segmentation on the remaining road surface elements. This stage model needs to train two models with and without intensity value input. Then, by freezing the parameters of the two models, the feature values output by the models are used to perform feature fusion through a multi-layer MLP structure to improve the model segmentation effect. This structure can fully utilize the geometric information obtained by separating other areas of the road surface in the second stage, and can also facilitate data augmentation of the road surface elements. The model inference adopts the sliding window traversal method.

[0012] Preferably, in the above technical solution, the point cloud information is obtained by a laser radar.

[0013] The present application innovatively proposes a multi-stage model inference structure from coarse to fine and step-by-step disassembly, which makes the segmentation effect of a small number of target categories more obvious.

[0014] Preferably, in the above technical solution, in the aerial target extraction, the size of the sliding window in the xy plane can be selected as 8m*8m.

[0015] Preferably, in the above technical solution, in the separation of road surface elements and background, the size of the sliding window in the xy plane can be selected as 16m*16m.

[0016] Preferably, in the above technical solution, in the fine-grained segmentation of road surface elements, the size of the sliding window in the xy plane is selected as 100m*100m.

[0017] Preferably, in the above technical solution, the method of model fusion is added in the fine-grained segmentation model, and the inputs of the two branch models are: model one is original point cloud information including position information and intensity value information, and model two only contains position information, compared with the single model structure commonly used in the prior art, the learning of geometric information can be paid more attention to, so that the segmentation effect is more accurate.

[0018] Preferably, in the above technical solution, the Conv Point network can also be replaced by other point cloud convolution models, point cloud mlp related models, graph network models and voxel models.

[0019] Preferably, in the above technical solution, the point cloud information is obtained by laser radar collection.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] The multi-stage semantic processing method proposed by the present application by using the method of deep learning can process the missegmentation caused by large-scale point clouds. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart schematic diagram of the present application is shown in the figure;

[0023] Figure 2 The fusion model schematic diagram of the present application is shown in the figure;

[0024] Figure 3 The first stage segmentation effect schematic diagram is shown in the figure;

[0025] Figure 4 The second stage segmentation effect schematic diagram is shown in the figure;

[0026] Figure 5 The third stage segmentation effect schematic diagram is shown in the figure. DETAILED DESCRIPTION

[0027] The specific embodiments of the present application are described in detail below, but it should be understood that the protection scope of the present application is not limited by the specific embodiments.

[0028] Unless otherwise clearly indicated, in the entire specification and claims, the term "comprise" or its variants such as "contain" or "include" and the like will be understood to include the stated element or component, but not exclude other elements or components.

[0029] The semantic extraction of large scenes is very difficult, and the number of targets of different labels often differs greatly, which seriously affects the final extraction result in the training and inference process of the deep learning model. Therefore, the present application designs a multi-stage model training and inference to continuously improve the model extraction accuracy of small targets on the road surface. The steps of the present scheme mainly have the following 3 modules:

[0030] First stage model (air target extraction)

[0031] The first stage model adopts the SOTA point cloud convolution semantic segmentation model Conv Point to train and infer the elements in the air target. The road surface elements in this stage are only one type of label segmented as a whole to prepare for the semantic extraction of the subsequent stage. In the preprocessing stage, the targets are classified according to their categories, and the sample size of the training data of the labels with less quantity is appropriately data-augmented. Due to the huge number of point clouds in outdoor scenes, the sliding window traversal method can be used in the data inference stage, and the window size cannot be too large, for example, an 8m*8m window size can be selected in the xy plane.

[0032] Second stage model (road surface elements and background are distinguished)

[0033] The second stage model adopts the same network structure, and the main purpose of this stage is to separate the road surface elements and the other areas of the road surface. The main purposes of this stage segmentation are two aspects: (1) filtering out the noise of the road surface to improve the segmentation result of the subsequent stage (2) through the separation of road surface elements and other areas of the road surface, the geometric information of the road surface elements is enhanced, and the segmentation effect of the subsequent stage is improved. The second stage model inference also adopts the sliding window traversal method, and due to the elimination of air targets, the number of point clouds is significantly reduced, so the window size can be increased compared with the first stage model, for example, a 16m*16m size can be selected in the xy plane. In order to avoid the lack of road surface element point clouds caused by model segmentation as much as possible, it is necessary to traverse each point cloud of the road surface elements after the inference of this stage model, find the nearest point cloud to the point cloud by using the knn nearest neighbor search method, and make its label as a road surface element, which has a similar effect to the dilation in image morphology processing, and increases the range of road surface element point clouds.

[0034] Third stage model (fine-grained segmentation of road surface elements)

[0035] The third stage model adopts a fusion model to perform fine-grained segmentation on the remaining pavement elements. The third stage model needs to train two models, one with intensity value input and one without intensity value input. Then, by freezing the parameters of the two models, the feature values output by the models are used to perform feature fusion using a multi-layer MLP structure to improve the segmentation effect of the model. This structure can fully utilize the geometric information obtained by stripping other areas of the road surface in the second stage, and can also facilitate data augmentation of the pavement elements. The model inference of the third stage is still performed in a sliding window traversal manner. Since the third stage model further reduces the number of point clouds, the window size of this stage can be further increased. For example, a 100m*100m size can be selected in the xy plane.

[0036] The training strategy of the three-stage model is the same, with an initial learning rate of 0.001, an adamW optimizer, a batchsize of 16, 100 epochs of training in each stage, 1000 iterations per epoch, and a cross-entropy loss function.

[0037] The three-stage model of the present patent has a precision (miou) index improvement of 7 percentage points compared to a single model with a running time difference of less than 5%. The semantic label recognition accuracy (iou) of road surface elements with a proportion less than 5% is improved by more than 20 percentage points.

[0038] The foregoing description of specific exemplary embodiments of the application is intended to be illustrative only and is not intended to limit the application to the precise forms described. Many modifications and variations are possible in light of the above teachings without departing from the spirit or essential characteristics of the application. The exemplary embodiments are chosen and described in order to explain the principles of the application and its practical application and to allow others skilled in the art to understand the application for various exemplary embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the application be defined by the claims and their equivalents.

Claims

1. A high-precision map multi-stage semantic feature extraction method, characterized by: The steps include: For aerial target extraction, the model uses the state-of-the-art point cloud convolutional semantic segmentation model (ConvPoint) to perform model training and inference on the elements of aerial targets. In this stage, the model segments the road surface elements as a class of labels as a whole to prepare for semantic extraction in the subsequent stage. In the preprocessing stage, the targets are classified according to their categories. When training the model, data augmentation is performed on samples with less than 5% semantic label training data. Due to the large number of point clouds in outdoor scenes, a sliding window traversal method is used in the data inference stage. The window size does not exceed 10*10 meters when the resolution of a single frame point cloud is not less than 0.1 meters. Both aerial targets and road features refer to the elements that need to be extracted in high-precision map construction: aerial targets include guardrails, poles, and road signs; road features include lane lines, arrows, and road markings; The road surface elements and background are distinguished, and the model adopts the same network structure. The main purpose of this stage is to separate the road surface elements from other road surface areas. The main purposes of segmentation in this stage are twofold: on the one hand, the noise of the road surface is filtered out to improve the segmentation results of the subsequent stages; on the other hand, by separating the road surface elements from other road surface areas, the geometric information of the road surface elements is enhanced, and the segmentation effect of the subsequent stages is improved; the model reasoning in the second stage is carried out in a sliding window traversal manner, and the size of the window is doubled compared with the first stage model; in order to avoid the point cloud of the road surface elements caused by the model segmentation, it is necessary to traverse each point cloud of the road surface elements after the model reasoning in this stage, and use the knn nearest neighbor search method to find the point cloud closest to the point cloud and label it as a road surface element; For fine-grained segmentation of pavement elements, a fusion model is used to perform fine-grained segmentation on the retained pavement elements. At this stage, the model needs to train two models with and without intensity value inputs. Then, by freezing the parameters of the two models and using the eigenvalues ​​of the model outputs, a multi-layer MLP structure is used to perform feature fusion to improve the model segmentation effect; model inference uses a sliding window traversal method.

2. The high-precision map multi-stage semantic feature extraction method according to claim 1 is characterized by: In the aerial target extraction, the sliding window size is selected as 8m*8m on the xy plane.

3. The high-precision map multi-stage semantic feature extraction method according to claim 1 is characterized by: In distinguishing between road features and background, the size of the sliding window is selected as 16 meters * 16 meters on the xy plane.

4. The high-precision map multi-stage semantic feature extraction method according to claim 1 is characterized by: In the fine-grained segmentation of road features, the size of the sliding window is selected as 100 meters * 100 meters on the xy plane.

5. The high-precision map multi-stage semantic feature extraction method according to claim 1 is characterized by: A model fusion method is added to the fine-grained segmentation model. The inputs of the two branch models are: Model 1 is the original point cloud information including position information and intensity value information, and Model 2 is the position information. Compared with the single model structure, it pays more attention to the learning of geometric information, making the segmentation effect more accurate.

6. The high-precision map multi-stage semantic feature extraction method according to claim 1, characterized in that: The Conv Point network is replaced by the point cloud convolution model, point cloud MLP model, graph network model, and voxel model.

7. The high-precision map multi-stage semantic feature extraction method according to claim 1, characterized in that: Point cloud information is collected by lidar.

Images