Road boundary detection method, and classification model generation method and device

Abstract

The present disclosure provides a road boundary detection method, a classification model generation method and device, and relates to the technical field of intelligent driving. First, a first point cloud frame currently collected by a millimeter wave radar is obtained, and then feature points in the first point cloud frame are preprocessed to obtain point cloud data to be identified. Then, the point cloud data to be identified is input into a preset classification model to obtain a classification label corresponding to each feature point output by the classification model. Finally, the current road boundary is determined according to the feature points with the corresponding classification label for the road boundary. Thus, the influence of the road environment on the road boundary detection is effectively avoided, and the accuracy and reliability of the road boundary detection method are improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and in particular to a road boundary detection method, a classification model generation method and apparatus. Background Technology

[0002] Road boundary detection is crucial for determining drivable areas on roads. However, it often faces challenges due to various factors such as occlusion, lighting conditions, and varying road boundary appearances. Reliable road boundary detection provides the accurate information needed for navigation and path planning, thereby improving the safety and efficiency of autonomous driving. Improving the accuracy and reliability of road boundary detection is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] The first aspect of this disclosure proposes a road boundary detection method, including:

[0005] Acquire the first point cloud frame currently collected by the millimeter-wave radar;

[0006] The feature points in the first point cloud frame are preprocessed to obtain the point cloud data to be identified.

[0007] The point cloud data to be identified is input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model, wherein the preset classification model is obtained based on the method described in the second aspect of the present disclosure;

[0008] The current road boundary is determined based on the feature points whose corresponding classification labels are road boundaries.

[0009] A second aspect of this disclosure provides a method for generating a classification model, comprising:

[0010] Obtain the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence;

[0011] Based on the laser point cloud segments corresponding to the times from the ijth time to the (i+jth time)th time in the laser point cloud sequence, determine the ith road boundary line segment, where i and j are natural numbers;

[0012] Based on the i-th road boundary line segment, the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud is determined.

[0013] Based on the labeled second millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model, which is used to classify each feature point in the millimeter-wave point cloud.

[0014] A third aspect of this disclosure provides a road boundary detection device, comprising:

[0015] The first acquisition module is used to acquire the first point cloud frame currently collected by the millimeter-wave radar.

[0016] The processing module is used to preprocess the feature points in the first point cloud frame to obtain the point cloud data to be identified.

[0017] An input module is used to input the point cloud data to be identified into a preset classification model to obtain a classification label corresponding to each feature point output by the classification model, wherein the preset classification model is obtained based on the method described in the second aspect of the present disclosure;

[0018] The first determination module is used to determine the current road boundary based on feature points whose corresponding classification labels are road boundaries.

[0019] A fourth aspect of this disclosure provides an apparatus for generating a classification model, comprising:

[0020] The second acquisition module is used to acquire the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence;

[0021] The second determining module is used to determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers.

[0022] The annotation module is used to annotate the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence based on the i-th road boundary line segment, and to determine the annotation label corresponding to each feature point in the i-th frame of millimeter-wave point cloud.

[0023] The training module is used to train the initial model based on the labeled second millimeter-wave point cloud sequence to obtain a classification model, wherein the classification model is used to classify each feature point in the millimeter-wave point cloud.

[0024] A fifth aspect of this disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the road boundary detection method proposed in the first aspect of this disclosure and the classification model generation method proposed in the second aspect of this disclosure.

[0025] A sixth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the road boundary detection method proposed in the first aspect of this disclosure and the classification model generation method proposed in the second aspect of this disclosure.

[0026] The road boundary detection method, classification model generation method, and apparatus disclosed herein have the following beneficial effects:

[0027] In this embodiment, the first point cloud frame currently acquired by the millimeter-wave radar is first obtained. Then, the feature points in the first point cloud frame are preprocessed to obtain point cloud data to be identified. Next, the point cloud data to be identified is input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model. Finally, based on the feature points whose classification labels correspond to road boundaries, the current road boundary is determined. Therefore, by acquiring the point cloud data currently acquired by the millimeter-wave radar, preprocessing it, and inputting it into a classification model, the classification label of each feature point in the point cloud data is obtained. Based on the feature points whose classification labels correspond to road boundaries, the current road boundary is determined, thereby effectively avoiding the influence of the road environment on road boundary detection and improving the accuracy and reliability of the road boundary detection method.

[0028] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic flowchart of a road boundary detection method provided in an embodiment of this disclosure;

[0031] Figure 2 This is a schematic flowchart of a road boundary detection method provided in another embodiment of the present disclosure;

[0032] Figure 3 This is a schematic diagram showing the effect comparison before and after preprocessing of the first point cloud frame provided in this disclosure;

[0033] Figure 4 This is a schematic flowchart of a road boundary detection method provided in another embodiment of the present disclosure;

[0034] Figure 5 A flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure;

[0035] Figure 6A schematic diagram illustrating the feature point annotations of the millimeter-wave point cloud provided in this disclosure;

[0036] Figure 7 A flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure;

[0037] Figure 8 A flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure;

[0038] Figure 9 A flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure;

[0039] Figure 10 This is a schematic diagram illustrating the effect of the classification model for road boundary detection provided in this disclosure.

[0040] Figure 11 This is a schematic diagram of the structure of a road boundary detection device provided in another embodiment of the present disclosure;

[0041] Figure 12 A schematic diagram of the structure of a classification model generation apparatus provided in another embodiment of this disclosure;

[0042] Figure 13 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0043] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0044] The road boundary detection method, classification model generation method, and apparatus of this disclosure are described below with reference to the accompanying drawings.

[0045] Figure 1 This is a schematic flowchart of a road boundary detection method provided in an embodiment of the present disclosure.

[0046] like Figure 1 As shown, the road boundary detection method may include the following steps:

[0047] Step 101: Obtain the first point cloud frame currently collected by the millimeter-wave radar.

[0048] It should be noted that the specific type of millimeter-wave radar can be determined according to actual needs. For example, millimeter-wave radar can be a 4D millimeter-wave radar, which can measure data in four dimensions: velocity, distance, horizontal azimuth, and vertical height. Here, 4D stands for 4-dimensional, and this disclosure does not limit it.

[0049] The first point cloud frame can be the road point cloud frame corresponding to the vehicle at the current moment.

[0050] It should be noted that the number of point cloud points contained in the first point cloud frame can be preset or determined according to actual needs. For example, the number of point cloud points in the first point cloud frame can be 512 points, and this disclosure does not limit this.

[0051] In this disclosure, the first point cloud frame currently collected by millimeter-wave radar is used to provide a data foundation for road boundary detection.

[0052] Step 102: Preprocess the feature points in the first point cloud frame to obtain the point cloud data to be identified.

[0053] It should be noted that each feature point in the first point cloud frame may have multiple features, such as three-dimensional spatial coordinates, distance, signal-to-noise ratio, Doppler velocity, vehicle speed, vehicle yaw rate, and the point cloud frame to which the feature point belongs. This disclosure does not limit these features.

[0054] The point cloud data to be identified can be point cloud data used for road boundary detection.

[0055] In this disclosure, after obtaining the first point cloud frame currently acquired by the millimeter-wave radar, in order to reduce the noise feature points in the first point cloud frame and improve the accuracy and efficiency of road boundary detection, the feature points in the first point cloud frame can be preprocessed to obtain the point cloud data to be identified, thereby improving the reliability and accuracy of the point cloud data.

[0056] Optionally, since the first point cloud frame may include feature points of non-road boundaries such as overpasses, trees, and other obstacles with excessively high heights, or because millimeter-wave radar may have errors during acquisition, the first point cloud frame may also include feature points with lower heights (such as below -1.5 meters). In this case, in order to improve the reliability of the first point cloud frame, when preprocessing the feature points in the first point cloud frame, feature points whose corresponding height values ​​are not within the preset height range can be discarded. This disclosure does not limit this.

[0057] The height range can be used to determine whether a feature point in a point cloud frame is a road boundary point. It can be preset or determined based on the actual situation. For example, the height range can be from -1.5 meters to 3 meters, and this disclosure does not limit it.

[0058] Step 103: Input the point cloud data to be identified into the preset classification model to obtain the classification label corresponding to each feature point output by the classification model.

[0059] The preset classification model is obtained based on the classification model generation method provided in the embodiments of this disclosure.

[0060] The classification model can be a model used to perform binary classification of feature points in point cloud data, or it can be used to determine whether a feature point is a road boundary point. It can be a model of any pre-defined type and structure. For example, the classification model can be a PointNet++ model, and this disclosure does not limit it.

[0061] PointNet++ is a hierarchical neural network model that can recursively process point cloud data. It can divide point cloud data into overlapping local regions, thereby capturing fine geometric structures within small neighborhoods and aggregating these structures into higher-level features to achieve multi-scale feature extraction. Its point-based processing method is suitable for sparse millimeter-scale radar-acquired point cloud data, but this disclosure does not limit it.

[0062] The classification label can be used to characterize whether a feature point in the point cloud data is a feature point of the road boundary. It can be any pre-set label. For example, the classification label can be the number "1" or "0". When the classification label is "1", it can indicate that the corresponding feature point is a feature point of the road boundary. When the classification label is "0", it can indicate that the corresponding feature point is not a feature point of the road boundary. This disclosure does not limit it.

[0063] It should be noted that after the point cloud data is input into the classification model, the classification model can determine the probability that the feature point is a feature point of the road boundary and the probability that it is a feature point of the non-road boundary based on the characteristics of each feature point. The feature point with the higher probability is identified as the classification result of the feature point, and the classification label corresponding to the feature point is output. This disclosure does not limit this.

[0064] In this disclosure, after obtaining the point cloud data to be identified, the point cloud data can be input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model, thereby improving the efficiency and reliability of road detection.

[0065] Step 104: Determine the current road boundary based on the feature points whose corresponding classification labels are road boundaries.

[0066] It should be noted that the feature points corresponding to the classification label of road boundary can be curb points. For example, they can be feature points such as the curb or guardrail of the road boundary, and this disclosure does not limit them.

[0067] In this disclosure, after obtaining the classification label corresponding to each feature point output by the classification model, the current road boundary can be determined based on the feature points whose corresponding classification labels are road boundaries. This effectively avoids the influence of the road environment on road boundary detection, improves the accuracy and reliability of road boundary detection, and enhances the safety of autonomous driving.

[0068] It should be noted that when determining the current road boundary based on feature points with corresponding classification labels as road boundaries, clustering algorithms (such as the DBSCAN algorithm) can be used to cluster the detected feature points of the road boundary, grouping feature points belonging to the same road boundary line into a group. Then, for each group, Gaussian process regression is used to model the road boundary line. The kernel function used for Gaussian process regression can be a combination of a constant kernel and an RBF kernel, and this disclosure does not limit it.

[0069] DBSCAN is short for Density-Based Spatial Clustering of Applications with Noise.

[0070] RBF is short for Radial Basis Function.

[0071] In this embodiment, the first point cloud frame currently acquired by the millimeter-wave radar is first obtained. Then, the feature points in the first point cloud frame are preprocessed to obtain point cloud data to be identified. Next, the point cloud data to be identified is input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model. Finally, based on the feature points whose classification labels correspond to road boundaries, the current road boundary is determined. Therefore, by acquiring the point cloud data currently acquired by the millimeter-wave radar, preprocessing it, and inputting it into a classification model, the classification label of each feature point in the point cloud data is obtained. Based on the feature points whose classification labels correspond to road boundaries, the current road boundary is determined, thereby effectively avoiding the influence of the road environment on road boundary detection and improving the accuracy and reliability of the road boundary detection method.

[0072] Figure 2 This is a schematic flowchart of a road boundary detection method provided in another embodiment of the present disclosure.

[0073] like Figure 2As shown, the road boundary detection method may include the following steps:

[0074] Step 201: Obtain the first point cloud frame currently collected by the millimeter-wave radar.

[0075] The specific implementation of step 201 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0076] Step 202: Determine the reference Doppler velocity for each feature point based on the vehicle's current moving speed and the position of each feature point.

[0077] It should be noted that Doppler velocity refers to the relative velocity between a target object and a vehicle as measured by millimeter-wave radar. Reference Doppler velocity, on the other hand, refers to the Doppler velocity of a stationary point (or static point) at the feature point location.

[0078] In this embodiment of the disclosure, when determining the reference Doppler velocity corresponding to each feature point based on the current moving speed of the vehicle and the position of each feature point, the longitudinal distance (distance in the vehicle's driving direction) and the lateral distance (distance in the direction perpendicular to the driving direction) between the static point at the feature point position and the vehicle can be determined first. Then, the lateral relative velocity of the static point relative to the vehicle is determined by multiplying the longitudinal distance by the vehicle's own steering rate. The longitudinal relative velocity of the static point relative to the vehicle is determined by adding the current moving speed of the vehicle to the lateral distance multiplied by the vehicle's own steering rate and then taking the negative. Then, the components of the lateral relative velocity and the longitudinal relative velocity along the orientation direction of the static point are obtained and summed to obtain the Doppler velocity of the static point at the feature point position, which is the reference Doppler velocity corresponding to the feature point at the feature point position.

[0079] It should be noted that the reference Doppler velocity corresponding to the feature point is different depending on the location of the feature point.

[0080] In this disclosure, after obtaining the first point cloud frame currently acquired by the millimeter-wave radar, before preprocessing the first point cloud frame, the reference Doppler velocity corresponding to each feature point can be determined first based on the current moving speed of the vehicle and the position of each feature point, thereby providing conditions for preprocessing the first point cloud frame.

[0081] Step 203: The feature points that match the corresponding Doppler velocity with the reference Doppler velocity are determined as static feature points.

[0082] In this disclosure, after determining the reference Doppler velocity, since road boundary points are usually static and non-moving points, when the Doppler velocity corresponding to a feature point matches the reference Doppler velocity, the feature point can be determined to be a static feature point, that is, a non-moving feature point, thereby providing conditions for improving the accuracy of road detection.

[0083] Step 204: Discard feature points in the first point cloud frame whose difference between the first Doppler velocity value and the second Doppler velocity value is greater than the difference threshold, to obtain the point cloud data to be identified, wherein the second Doppler velocity value is the velocity value corresponding to the static feature point.

[0084] The first Doppler velocity value can be the Doppler velocity value corresponding to the feature point in the first point cloud frame.

[0085] The difference threshold can be a critical value used to determine whether a feature point within the first point cloud frame is a static feature point. It can be preset or determined based on actual conditions. For example, the difference threshold can be 1 meter per second, and this disclosure does not limit it.

[0086] In this disclosure, after determining static feature points, when the difference between the first Doppler velocity value corresponding to a feature point within the first point cloud frame and the second Doppler velocity value corresponding to a static feature point at the same location is greater than a difference threshold, the corresponding feature point can be determined to be a moving feature point, such as a feature point of a moving vehicle or person. In this case, the corresponding feature point can be discarded, thereby effectively reducing noisy feature points in the first point cloud frame and improving the reliability of the first point cloud frame. Figure 3 As shown, Figure 3 This is a schematic diagram showing the effect comparison before and after the first point cloud frame preprocessing provided in this disclosure. Figure 3 Including Figure 3 a and Figure 3 b, of which Figure 3 a is a schematic diagram of the first point cloud frame before preprocessing provided in this disclosure. Figure 3 b is a schematic diagram of the first point cloud frame after preprocessing provided in this disclosure. Figure 3 a, Figure 3 In point b, the x, y, and z axes represent the positions of the feature points, and the origin of the coordinate system represents the position of the vehicle itself. Figure 3 In frame a, the blue point cloud represents the first point cloud frame before preprocessing. Figure 3 In frame b, the red point cloud is the first point cloud frame after preprocessing. Figure 3 Therefore, compared to Figure 3 The first point cloud frame shown in a, Figure 3 In the first cloud frame of b, the noise feature points are significantly reduced, and the road boundaries are clearer. Figure 3 This is merely an example and is not intended to limit the scope of this disclosure.

[0087] Step 205: Input the point cloud data to be identified into the preset classification model to obtain the classification label corresponding to each feature point output by the classification model.

[0088] The preset classification model is obtained based on the classification model generation method provided in the embodiments of this disclosure.

[0089] Step 206: Determine the current road boundary based on the feature points whose corresponding classification labels are road boundaries.

[0090] The specific implementation of steps 205 to 206 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0091] In this embodiment, the first point cloud frame currently collected by the millimeter-wave radar is first acquired. Then, based on the vehicle's current moving speed and the position of each feature point, the reference Doppler velocity corresponding to each feature point is determined. Feature points whose corresponding Doppler velocities match the reference Doppler velocities are identified as static feature points. Subsequently, feature points in the first point cloud frame whose difference between the corresponding first Doppler velocity value and the second Doppler velocity value is greater than the difference threshold are discarded to obtain the point cloud data to be identified. Finally, the point cloud data to be identified is input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model. Based on the feature points whose corresponding classification labels are road boundaries, the current road boundary is determined. Therefore, after obtaining point cloud data collected by millimeter-wave radar, the reference Doppler velocity corresponding to the feature point is determined based on the vehicle's current moving speed and the position of the feature point. Based on the reference Doppler velocity, static feature points are determined. Feature points whose difference between the Doppler velocity corresponding to the point cloud data and the Doppler velocity corresponding to the static feature point is greater than the difference threshold are discarded, thus obtaining the point cloud data to be identified. Based on the classification model, the classification label of each feature point in the point cloud data to be identified is determined. Based on the feature points whose classification label is the road boundary, the road boundary is determined. This effectively reduces noise in the point cloud data, improves the reliability of the point cloud data, and improves the efficiency and accuracy of the road detection method.

[0092] Figure 4 This is a schematic flowchart of a road boundary detection method provided in another embodiment of the present disclosure.

[0093] like Figure 4 As shown, the road boundary detection method may include the following steps:

[0094] Step 401: Obtain the first point cloud frame currently collected by the millimeter-wave radar.

[0095] The specific implementation of step 401 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0096] Step 402: Fuse the feature points in the first point cloud frame with at least one feature point in the second point cloud frame to obtain the point cloud data to be identified.

[0097] The second point cloud frame is obtained by preprocessing the third point cloud frame, which was collected before the first point cloud frame and is adjacent to the first point cloud frame.

[0098] It should be noted that the preprocessing of the third point cloud frame can be the same as that of the first point cloud frame. For example, feature points whose corresponding height values ​​are not within the preset height range in the third point cloud frame can be discarded, and / or feature points whose Doppler velocity values ​​corresponding to feature points in the third point cloud frame have a difference greater than a difference threshold with the Doppler velocity values ​​corresponding to static feature points can be discarded, etc. This disclosure does not limit this.

[0099] In this embodiment of the present disclosure, after obtaining the first point cloud frame currently collected by the millimeter-wave radar, in order to improve the reliability and accuracy of the first point cloud frame, the feature points in the first point cloud frame can be fused with the feature points in at least one second point cloud frame to obtain the point cloud data to be identified.

[0100] Optionally, when fusing feature points in the first point cloud frame with feature points in at least one second point cloud frame, since the single-frame point cloud data acquired by millimeter-wave radar is usually sparse point cloud data, in order to improve the reliability of the single-frame point cloud data and reduce the sparsity of the point cloud frame, a preset number of feature points in the first point cloud frame can be fused with a preset number of feature points in at least one second point cloud frame, thereby enhancing the continuity of the point cloud data, compensating for missing points in the single-frame point cloud frame acquired by millimeter-wave radar, and improving the accuracy of the point cloud data.

[0101] The preset quantity can be any pre-set quantity, or it can be a quantity determined according to the actual situation. For example, the preset quantity can be 512, and this disclosure does not limit it.

[0102] For example, when fusing feature points in a first point cloud frame with feature points in at least one second point cloud frame, after preprocessing the third point cloud frames of the two preceding frames adjacent to the first point cloud frame, corresponding second point cloud frames can be obtained respectively. After fusing the 512 feature points in the first point cloud frame with the 512 feature points in each of the two second point cloud frames, the point cloud data to be identified can contain 1536 feature points. This disclosure does not limit this.

[0103] Optionally, when fusing a preset number of feature points in the first point cloud frame with a preset number of feature points in at least one second point cloud frame, a preset number of feature points may be randomly selected from the first point cloud frame, or the interference of noisy feature points in the first point cloud frame may be reduced. Alternatively, a preset number of feature points may be randomly selected from the region in the first point cloud frame where the feature point density is greater than the density threshold. This disclosure does not limit this.

[0104] The density threshold can be used to determine the critical value of feature point density in the road boundary region of the first point cloud frame. The region with feature point density greater than the density threshold can be identified as the road boundary region. The value can be preset, and this disclosure does not limit it.

[0105] Step 403: Input the point cloud data to be identified into the preset classification model to obtain the classification label corresponding to each feature point output by the classification model.

[0106] The preset classification model is obtained based on the classification model generation method provided in the embodiments of this disclosure.

[0107] Step 404: Determine the current road boundary based on the feature points whose corresponding classification labels are road boundaries.

[0108] The specific implementation of steps 403 to 404 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0109] In this embodiment, the first point cloud frame currently acquired by the millimeter-wave radar is first obtained. The feature points within the first point cloud frame are then fused with feature points from at least one second point cloud frame to obtain point cloud data to be identified. This point cloud data is then input into a preset classification model to obtain the classification label corresponding to each feature point output by the model. Finally, based on feature points whose classification label corresponds to road boundaries, the current road boundary is determined. Thus, after obtaining the point cloud frame currently acquired by the millimeter-wave radar, the feature points within the point cloud frame are fused with feature points from at least one adjacent preprocessed point cloud frame to obtain point cloud data to be identified. Based on the classification model, the classification label for each feature point in the point cloud data is determined. The road boundary is then determined based on feature points whose classification label corresponds to road boundaries, thereby improving the accuracy and reliability of road boundary detection.

[0110] Figure 5 This is a flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure.

[0111] like Figure 5 As shown, the method for generating this classification model may include the following steps:

[0112] Step 501: Obtain the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence.

[0113] The first millimeter-wave point cloud sequence can be the point cloud sequence currently collected by the millimeter-wave radar.

[0114] Among them, the laser point cloud sequence can be the point cloud sequence collected by lidar.

[0115] In this disclosure, when training a classification model, in order to improve the accuracy of the classification model training, a first millimeter-wave point cloud sequence and a corresponding laser point cloud sequence can be obtained first. The laser point cloud sequence can be used to annotate the first millimeter-wave point cloud sequence, thereby improving the accuracy and efficiency of the classification model training.

[0116] Step 502: Determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers.

[0117] Where i can be the i-th time, ij and i+j can be the j-th time before and j-th time after the i-th time respectively, and the value of j can be preset or determined according to the actual situation. This disclosure does not limit the value of j.

[0118] In this disclosure, after obtaining the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence, before labeling the first millimeter-wave point cloud sequence using the laser point cloud sequence, although the lidar can generate dense point cloud feature points, there may still be a small number of frames in the point cloud sequence that have missed road boundary detection. Moreover, the detection range of lidar is small, while the detection range of millimeter-wave radar is large. In order to solve the difference in road boundary detection and reduce missed labeling, the i-th road boundary line segment can be determined based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence. For example, the road boundary line segment at the current time can be determined based on the laser point cloud segments adjacent to the current time for 5 seconds before and after the current time in the laser point cloud sequence, thereby improving the accuracy of the classification model.

[0119] Step 503: Based on the i-th road boundary line segment, label the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence, and determine the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud.

[0120] Among them, the label can be used to characterize whether a feature point is a feature point of the road boundary.

[0121] In this disclosure, after determining the i-th road boundary line segment, feature points in the i-th frame of the millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence can be labeled based on the i-th road boundary line segment. This determines the label corresponding to each feature point in the i-th frame of the millimeter-wave point cloud, thereby improving the accuracy and efficiency of feature point labeling in the millimeter-wave point cloud. Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the feature point annotation of the millimeter-wave point cloud provided in this disclosure. Figure 6In the diagram, the x, y, and z axes represent the positions of feature points, the origin represents the vehicle's position, green segments represent road boundary segments, red feature points are those marked as road boundaries in the millimeter-wave point cloud data, and blue feature points are those marked as non-road boundaries in the millimeter-wave point cloud data. Figure 6 It can be seen that by labeling feature points in millimeter-wave point clouds based on road boundary line segments, the accuracy and efficiency of labeling can be improved. Figure 6 This is merely an example and is not intended to limit the scope of this disclosure.

[0122] Step 504: Based on the labeled second millimeter-wave point cloud sequence, train the initial model to obtain a classification model, which is used to classify each feature point in the millimeter-wave point cloud.

[0123] The initial model can be any pre-set model of any type and structure, or it can be a model determined according to the actual situation. For example, the initial model can be a PointNet++ model, and this disclosure does not limit it.

[0124] In this disclosure, after labeling the feature points of each frame of the millimeter-wave point cloud in the first millimeter-wave point cloud sequence, the initial model can be trained based on the second millimeter-wave point cloud sequence obtained after labeling to obtain a classification model that classifies the feature points, thereby improving the classification effect and quality of the obtained classification model.

[0125] It should be noted that when training the initial model based on the labeled second millimeter-wave point cloud sequence, in order to increase the amount of training data, and because of the symmetry of millimeter-wave radar point clouds (the point clouds on the left and right sides of the vehicle can be identical), the point cloud data can be flipped along the vehicle's driving direction to enhance the training set. Taking the vehicle as the origin, the vehicle's driving direction as the y-axis, and the direction perpendicular to the driving direction as the x-axis, the x-coordinate of each feature point in the second millimeter-wave point cloud sequence is symmetrically flipped left and right along the y-axis. For example, feature points located on the left side of the vehicle are flipped left and right with feature points located on the right side of the vehicle, or feature points with x-coordinates of -1 and 1 are flipped left and right, etc. By using the flipped millimeter-wave point cloud sequence to increase the amount of training data during model training, the generalization and accuracy of model training can be improved. This disclosure does not limit this.

[0126] In this embodiment, a first millimeter-wave point cloud sequence and its corresponding laser point cloud sequence are first acquired. Then, based on the laser point cloud segments from time ij to time i+j in the laser point cloud sequence, the i-th road boundary line segment is determined. Next, based on the i-th road boundary line segment, feature points in the i-th frame of millimeter-wave point cloud corresponding to time i in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud is determined. Finally, based on the labeled second millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model. Thus, by using the laser point cloud segments corresponding to time i and the j times before and after in the acquired laser point cloud sequence, the i-th road boundary line segment is determined. Based on the i-th road boundary line segment, the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud corresponding to time i in the acquired millimeter-wave point cloud sequence is determined. Based on the labeled millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model, thereby improving the classification effect and quality of the classification model and enhancing the accuracy and reliability of the generated classification model.

[0127] Figure 7 This is a flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure.

[0128] like Figure 7 As shown, the method for generating this classification model may include the following steps:

[0129] Step 701: Obtain the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence.

[0130] Step 702: Determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers.

[0131] The specific implementation of steps 701 to 702 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0132] Step 703: Determine the reference area corresponding to the i-th road boundary line segment based on the relative position between the i-th road boundary line segment and the vehicle.

[0133] In this disclosure, after determining the i-th road boundary line segment, since the i-th road boundary line segment is determined based on the laser point cloud sequence, and laser radar usually defines the road boundary based on changes in ground height, while the point cloud data collected by millimeter-wave radar can usually represent fences or other obstacles located further away from the road boundary, in order to improve the accuracy and reliability of the annotation of the millimeter-wave point cloud sequence based on the road boundary line segment, the reference area corresponding to the i-th road boundary line segment can be determined according to the relative position between the i-th road boundary line segment and the vehicle.

[0134] Optionally, if the i-th road boundary line segment is located on the first side of the vehicle, the distance between the first boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the second boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value.

[0135] The first side can be either the left side or the right side of the vehicle; this disclosure does not limit it to either.

[0136] The first boundary line can be the left boundary line of the reference area, and the second boundary line can be the right boundary line of the reference area.

[0137] The values ​​of the first and second values ​​can be determined according to the actual situation. For example, the first value can be 1 and the second value can be 0.5. This disclosure does not limit this.

[0138] It should be noted that the lengths of the first boundary line and the second boundary line can be the same as the i-th road boundary line segment, and this disclosure does not impose any restrictions on this.

[0139] It should be noted that, when the first side is the left side of the vehicle, since the road boundary detected by millimeter-wave radar is usually farther than that detected by lidar, and the first boundary line of the reference area is located to the left of the second boundary line, the extension distance of the first boundary line (extending to the left) is greater than the extension distance of the second boundary line. In other words, the distance between the first boundary line and the road boundary line segment is farther than the distance between the second boundary line and the road boundary line segment. The first value is greater than the second value, and this disclosure does not limit this.

[0140] It should be noted that, when the first side is the right side of the vehicle, since the road boundary detected by millimeter-wave radar is usually farther than that detected by lidar, and the first boundary line of the reference area is located to the right of the second boundary line, the extension distance of the first boundary line (extending to the right) is greater than the extension distance of the second boundary line. The first value is greater than the second value, and this disclosure does not limit this.

[0141] Optionally, if the i-th road boundary line segment is located on the second side of the vehicle, the distance between the second boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the first boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value.

[0142] The second side can be either the right side or the left side of the vehicle; this disclosure does not limit it to either.

[0143] It should be noted that when the second side is the right side of the vehicle, the first boundary line is located to the left of the second boundary line, similar to the case where the first side is the right side of the vehicle. In this case, the distance between the second boundary line located to the right of the first boundary line and the road boundary line segment is greater than the distance between the first boundary line and the road boundary line segment. In other words, the first value is greater than the second value, and this disclosure does not limit this.

[0144] It should be noted that when the second side is the left side of the vehicle, the first boundary line is located to the right of the second boundary line, similar to the case where the first side is the left side of the vehicle. In this case, the distance between the second boundary line located to the left of the first boundary line and the road boundary line segment is greater than the distance between the first boundary line and the road boundary line segment. In other words, the first value is greater than the second value, and this disclosure does not limit this.

[0145] Step 704: Determine the label corresponding to the feature point located within the reference area in the i-th frame millimeter-wave point cloud as the first label, and the label corresponding to the feature point not located within the reference area as the second label.

[0146] The first label can be used to represent a road boundary point, and the second label can be used to represent a non-road boundary point. Both the first and second labels can be pre-set labels of any form. For example, the first label can be the number "1" to represent a road boundary point, and the second label can be the number "0" to represent a non-road boundary point. This disclosure does not impose any limitations on this.

[0147] In this disclosure, after determining the reference region corresponding to the i-th road boundary line segment, the label corresponding to the feature point in the i-th frame millimeter-wave point cloud located within the reference region can be determined as the first label, and the label corresponding to the feature point not located within the reference region can be determined as the second label, thereby improving the accuracy of millimeter-wave point cloud sequence labeling.

[0148] Step 705: Based on the labeled second millimeter-wave point cloud sequence, train the initial model to obtain a classification model, which is used to classify each feature point in the millimeter-wave point cloud.

[0149] The specific implementation of step 705 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0150] In this embodiment, a first millimeter-wave point cloud sequence and its corresponding laser point cloud sequence are first acquired. Then, based on the laser point cloud segments corresponding to times ij to ix (i+j) in the laser point cloud sequence, the ith road boundary line segment is determined. Based on the relative position between the ith road boundary line segment and the vehicle, a reference region corresponding to the ith road boundary line segment is determined. Next, the labels corresponding to feature points within the reference region in the ith frame of the millimeter-wave point cloud are designated as first labels, and the labels corresponding to feature points not within the reference region are designated as second labels. Finally, based on the labeled second millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model. Thus, after acquiring the millimeter-wave point cloud sequence and its corresponding laser point cloud sequence, road boundary line segments are determined based on the laser point cloud segments corresponding to multiple times in the laser point cloud sequence. Based on the relative position between the road boundary line segments and the vehicle, a reference region corresponding to the road boundary line segment is determined. Based on the reference region, the millimeter-wave point cloud sequence is labeled. Based on the labeled millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model, thereby improving the accuracy and reliability of the generated classification model.

[0151] Figure 8 This is a flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure.

[0152] like Figure 8 As shown, the method for generating this classification model may include the following steps:

[0153] Step 801: Obtain the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence.

[0154] Step 802: Determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers.

[0155] Step 803: Based on the i-th road boundary line segment, the feature points in the i-th frame of the millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of the millimeter-wave point cloud is determined.

[0156] The specific implementation of steps 801 to 803 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0157] Step 804: Input the labeled second millimeter-wave point cloud sequence into the initial model to obtain the predicted label corresponding to each feature point output by the initial model.

[0158] Among them, the predicted label can represent the predicted classification result corresponding to each feature point.

[0159] In this disclosure, after labeling each feature point in the first millimeter-wave point cloud sequence, in order to reliably and efficiently train the initial model, the second millimeter-wave point cloud sequence obtained after labeling can be input into the initial model. The initial model predicts whether each feature point is a road boundary point and outputs the predicted label corresponding to each feature point.

[0160] Step 805: Determine the first loss value based on the difference between the predicted label and the labeled label corresponding to each feature point.

[0161] The first loss value can be a loss value calculated based on any computable classification loss function. For example, the first loss value can be a value calculated based on the cross-entropy loss function.

[0162] In this disclosure, after obtaining the predicted label corresponding to each feature point, a first loss value can be determined based on the difference between the predicted label and the labeled label corresponding to each feature point, thereby providing conditions for improving the performance and quality of the training-obtained classification model.

[0163] Step 806: Determine the minimum distance between the feature point with the predicted label as the first label and the i-th road boundary line segment.

[0164] In this disclosure, to avoid predicting feature points of non-road boundaries such as overpasses, trees, or cars as feature points of road boundaries, after obtaining the prediction label corresponding to each feature point, the minimum distance between the feature point with the first prediction label and the i-th road boundary line segment can be determined.

[0165] Step 807: Determine the second loss value based on the mean of the minimum distances corresponding to all feature points of the first label.

[0166] The second loss value can be the distance loss value during the model training process.

[0167] In this disclosure, after determining the minimum distance between each feature point corresponding to the first label and the i-th eastern boundary line segment, the mean of the minimum distances corresponding to all feature points of the first label can be determined and used as the second loss value, thereby improving the reliability of the second loss value.

[0168] Step 808: Based on the first loss value and the second loss value, the initial model is reverse-corrected until a classification model is obtained.

[0169] It should be noted that when performing reverse correction on the initial model based on the first loss value and the second loss value, the first loss value and the second loss value can be first weighted and fused, and then the initial model can be reverse corrected based on the fused loss value. The weights of the weighted fusion can be preset or determined according to actual needs, and this disclosure does not limit them.

[0170] It should be noted that when back-correcting the initial model based on the first and second loss values, in order to optimize model performance, a network search method can also be used to select key hyperparameters, which may include the number of network layers, learning rate, optimizer type, batch size and number of training rounds, etc. The model with the minimum loss is determined as the classification model. This disclosure does not limit this.

[0171] In this disclosure, after determining the first loss value and the second loss value, the initial model can be reverse-corrected based on the first loss value and the second loss value until the model with the minimum loss is obtained, and it is determined as the classification model, thereby improving the road boundary detection effect of the model. This disclosure does not limit this.

[0172] In this embodiment, a first millimeter-wave point cloud sequence and a corresponding laser point cloud sequence are first acquired. Based on the laser point cloud segments from time ij to time i+j in the laser point cloud sequence, the i-th road boundary line segment is determined. Then, based on the i-th road boundary line segment, feature points in the i-th frame of millimeter-wave point cloud corresponding to time i in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud is determined. The labeled second millimeter-wave point cloud sequence is then input into the initial model to obtain the predicted label corresponding to each feature point output by the initial model. Subsequently, based on the difference between the predicted label and the labeled label corresponding to each feature point, a first loss value is determined, and the minimum distance between the feature point with the predicted label as the first label and the i-th road boundary line segment is determined. Finally, based on the average of the minimum distances corresponding to all feature points with the first label, a second loss value is determined. Based on the first loss value and the second loss value, the initial model is reverse-corrected until a classification model is obtained. Therefore, after labeling the millimeter-wave point cloud sequence based on road boundary line segments to obtain the labeled millimeter-wave point cloud sequence, the labeled millimeter-wave point cloud sequence is input into the initial model to obtain the predicted label corresponding to each feature point. Based on the predicted label and the labeled label of each feature point, the first loss value is determined. Then, all predicted labels are determined as the average of the minimum distances between the feature points of the road boundary and the road boundary line segments, and this is determined as the second loss value. Based on the first loss value and the second loss value, the initial model is reverse-corrected until a classification model is obtained, thereby improving the road boundary detection effect and quality of the classification model, and improving the reliability and efficiency of the classification model generation.

[0173] Figure 9 This is a flowchart illustrating a method for generating a classification model according to another embodiment of this disclosure.

[0174] like Figure 9 As shown, the method for generating this classification model may include the following steps:

[0175] Step 901: Obtain the third millimeter-wave point cloud sequence and the laser point cloud sequence collected by the millimeter-wave radar and lidar, respectively.

[0176] The third millimeter-wave point cloud sequence is an unprocessed millimeter-wave point cloud sequence acquired by millimeter-wave radar.

[0177] In this disclosure, the third millimeter-wave point cloud sequence and the laser point cloud sequence collected by millimeter-wave radar and lidar, respectively, are used to provide a data foundation for obtaining training data.

[0178] Step 902: Preprocess the third millimeter-wave point cloud sequence to obtain the first millimeter-wave point cloud sequence.

[0179] In this disclosure, after obtaining the third millimeter-wave point cloud sequence and the laser point cloud sequence, since the model is trained using the millimeter-wave point cloud sequence, in order to improve the reliability of the third millimeter-wave point cloud sequence and reduce the noise feature points in the third millimeter-wave point cloud sequence, the third millimeter-wave point cloud sequence can be preprocessed to obtain the first millimeter-wave point cloud sequence.

[0180] Optionally, when preprocessing the third millimeter-wave point cloud sequence, feature points whose corresponding height values ​​are not within a preset height range in each point cloud frame of the third millimeter-wave point cloud sequence can be discarded. This disclosure does not limit this.

[0181] Optionally, when preprocessing the third millimeter-wave point cloud sequence, feature points in each point cloud frame of the third millimeter-wave point cloud sequence whose difference between the first Doppler velocity value and the second Doppler velocity value is greater than the difference threshold can be discarded. The second Doppler velocity value is the velocity value corresponding to the static point, and this disclosure does not limit it.

[0182] Optionally, when preprocessing the third millimeter-wave point cloud sequence, the feature points in each point cloud frame of the third millimeter-wave point cloud sequence can be fused with the feature points in at least one reference point cloud frame. The reference point cloud frame is obtained by preprocessing at least one point cloud frame that was acquired before the point cloud frame and is adjacent to the point cloud frame. This disclosure does not limit this.

[0183] It should be noted that the preprocessing process for the third millimeter-wave point cloud sequence can be the same as the preprocessing process for the first point cloud frame mentioned in the embodiments of this disclosure, and will not be described in detail here.

[0184] Step 903: Determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers.

[0185] Step 904: Based on the i-th road boundary line segment, label the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence, and determine the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud.

[0186] Step 905: Based on the labeled second millimeter-wave point cloud sequence, train the initial model to obtain a classification model, which is used to classify each feature point in the millimeter-wave point cloud.

[0187] The specific implementation of steps 903 to 905 can be found in the detailed description of other embodiments of this disclosure, and will not be repeated here.

[0188] In this embodiment, firstly, third millimeter-wave point cloud sequences and laser point cloud sequences acquired by millimeter-wave radar and lidar, respectively, are obtained. Then, the third millimeter-wave point cloud sequence is preprocessed to obtain a first millimeter-wave point cloud sequence. Next, based on the laser point cloud segments corresponding to times ij to i+j in the laser point cloud sequence, the i-th road boundary line segment is determined. Based on the i-th road boundary line segment, feature points in the i-th frame of the millimeter-wave point cloud corresponding to time i in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of the millimeter-wave point cloud is determined. Finally, based on the labeled second millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model. Therefore, after obtaining the millimeter-wave point cloud sequences and laser point cloud sequences acquired by millimeter-wave radar and lidar, respectively, the millimeter-wave point cloud sequences are preprocessed, road boundary line segments are determined based on the laser point cloud sequence, and the preprocessed millimeter-wave point cloud sequences are labeled based on the road boundary line segments. Based on the labeled millimeter-wave point cloud sequences, the initial model is trained to obtain a classification model, thereby improving the accuracy and reliability of road boundary detection in the generated classification model.

[0189] The following is combined with Figure 10 The following example illustrates the road boundary detection performance of the classification model obtained using the classification model generation method provided in this disclosure. Figure 10 This is a schematic diagram illustrating the effect of the classification model for road boundary detection provided in this disclosure. Figure 10 Including Figure 10 a and Figure 10 b.

[0190] Figure 10 a, Figure 10 The left-hand images in b are the output results of the classification model, the middle images are the annotation images of millimeter-wave radar point cloud data based on LiDAR point cloud data, and the right-hand images are the corresponding real-world images. Figure 10 a, Figure 10 In the left-hand image of diagram b, the red feature points represent road boundary features from the millimeter-wave point cloud data output by the classification model, while the blue feature points represent non-road boundary features. The 95% confidence interval can be used as the confidence interval for the road boundary lines output when modeling road boundary lines using Gaussian process regression. In the middle image, the green feature points represent road boundary features from millimeter-wave radar point cloud data annotated with LiDAR point cloud data, while the black feature points represent non-road boundary features from millimeter-wave radar point cloud data annotated with LiDAR point cloud data. Figure 10 It can be seen that, Figure 10 The output of the classification model in a, and Figure 10 The annotation results in b are very close, which confirms that the road boundary detection results of the classification model provided in this disclosure have high quality and reliability, and effectively avoids the influence of obstacles such as overpasses, trees, and cars on road boundary detection, thereby improving the accuracy of road boundary detection.

[0191] To achieve the above embodiments, this disclosure also proposes a road boundary detection device.

[0192] Figure 11 This is a schematic diagram of the structure of a road boundary detection device provided in another embodiment of the present disclosure.

[0193] like Figure 11 As shown, the road boundary detection device 1100 may include: a first acquisition module 1101, a processing module 1102, an input module 1103, and a first determination module 1104.

[0194] The first acquisition module 1101 is used to acquire the first point cloud frame currently collected by the millimeter-wave radar.

[0195] The processing module 1102 is used to preprocess the feature points in the first point cloud frame to obtain the point cloud data to be identified.

[0196] The input module 1103 is used to input the point cloud data to be identified into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model. The preset classification model is obtained based on the classification model generation method described in the embodiments of this disclosure.

[0197] The first determining module 1104 is used to determine the current road boundary based on feature points whose corresponding classification labels are road boundaries.

[0198] Optionally, the above-mentioned processing module 1102 is specifically used for at least one of the following:

[0199] Discard feature points whose corresponding height values ​​are not within the preset height range in the first point cloud frame;

[0200] Feature points within the first point cloud frame whose difference between the first Doppler velocity value and the second Doppler velocity value is greater than the difference threshold are discarded. The second Doppler velocity value is the velocity value corresponding to the static feature point.

[0201] The feature points in the first point cloud frame are fused with the feature points in at least one second point cloud frame, wherein the second point cloud frame is obtained by preprocessing a third point cloud frame that was acquired before the first point cloud frame and is adjacent to the first point cloud frame.

[0202] Optionally, the above-mentioned processing module 1102 is further configured to:

[0203] Based on the vehicle's current speed and the position of each feature point, determine the reference Doppler velocity corresponding to each feature point;

[0204] The feature points whose corresponding Doppler velocities match the reference Doppler velocities are determined as static feature points.

[0205] Optionally, the above-mentioned processing module 1102 is further configured to:

[0206] A predetermined number of feature points in the first point cloud frame are fused with a predetermined number of feature points in at least one second point cloud frame.

[0207] Optionally, the above-mentioned processing module 1102 is further used for any of the following:

[0208] A preset number of feature points are randomly selected from the first point cloud frame;

[0209] From the region in the first point cloud frame where the feature point density is greater than the density threshold, a preset number of feature points are randomly selected.

[0210] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0211] The road boundary detection device of this embodiment first acquires the first point cloud frame currently collected by millimeter-wave radar. Then, it preprocesses the feature points in the first point cloud frame to obtain point cloud data to be identified. Next, it inputs the point cloud data to be identified into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model. Finally, based on the feature points whose classification labels correspond to road boundaries, the current road boundary is determined. Therefore, by acquiring the point cloud data currently collected by millimeter-wave radar, preprocessing it, and inputting it into a classification model to obtain the classification label for each feature point in the point cloud data, and determining the current road boundary based on the feature points whose classification labels correspond to road boundaries, the influence of the road environment on road boundary detection is effectively avoided, improving the accuracy and reliability of the road boundary detection method.

[0212] Figure 12 This is a schematic diagram of the structure of a classification model generation apparatus provided in another embodiment of the present disclosure.

[0213] like Figure 12 As shown, the classification model generation device 1200 may include: a second acquisition module 1201, a second determination module 1202, a labeling module 1203, and a training module 1204.

[0214] The second acquisition module 1201 is used to acquire the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence;

[0215] The second determining module 1202 is used to determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers respectively;

[0216] The annotation module 1203 is used to annotate the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence based on the i-th road boundary line segment, and to determine the annotation label corresponding to each feature point in the i-th frame of millimeter-wave point cloud.

[0217] Training module 1204 is used to train the initial model based on the labeled second millimeter-wave point cloud sequence to obtain a classification model, wherein the classification model is used to classify each feature point in the millimeter-wave point cloud.

[0218] Optionally, the above-mentioned annotation module 1203 is specifically used for:

[0219] Based on the relative position between the i-th road boundary line segment and the vehicle, determine the reference area corresponding to the i-th road boundary line segment;

[0220] In the i-th frame of millimeter-wave point cloud, the label corresponding to the feature point located within the reference area is determined as the first label, and the label corresponding to the feature point not located within the reference area is determined as the second label.

[0221] Optionally, the above-mentioned annotation module 1203 is also used for any of the following:

[0222] When the i-th road boundary line segment is located on the first side of the vehicle, the distance between the first boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the second boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value;

[0223] When the i-th road boundary line segment is located on the second side of the vehicle, the distance between the second boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the first boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value.

[0224] Optionally, the above training module 1204 is specifically used for:

[0225] Input the second millimeter-wave point cloud sequence into the initial model to obtain the predicted label corresponding to each feature point output by the initial model;

[0226] The first loss value is determined based on the difference between the predicted label and the labeled label for each feature point;

[0227] Determine the minimum distance between the feature point whose predicted label is the first label and the i-th road boundary line segment;

[0228] The second loss value is determined based on the mean of the minimum distances corresponding to all feature points of the first label.

[0229] Based on the first and second loss values, the initial model is reverse-corrected until a classification model is obtained.

[0230] Optionally, the second acquisition module described above is specifically used for:

[0231] Acquire the third millimeter-wave point cloud sequence and the laser point cloud sequence collected by the millimeter-wave radar and lidar, respectively;

[0232] The third millimeter-wave point cloud sequence was preprocessed to obtain the first millimeter-wave point cloud sequence.

[0233] Optionally, the second acquisition module described above is further used for at least one of the following:

[0234] Discard feature points whose corresponding height values ​​are not within the preset height range in each point cloud frame of the third millimeter-wave point cloud sequence.

[0235] In the third millimeter-wave point cloud sequence, feature points whose difference between the first Doppler velocity value and the second Doppler velocity value in each point cloud frame is greater than the difference threshold are discarded. The second Doppler velocity value is the velocity value corresponding to the static point.

[0236] The feature points in each point cloud frame of the third millimeter-wave point cloud sequence are fused with the feature points in at least one reference point cloud frame. The reference point cloud frame is obtained by preprocessing at least one point cloud frame that was acquired before the point cloud frame and is adjacent to the point cloud frame.

[0237] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0238] The classification model generation apparatus of this embodiment first acquires a first millimeter-wave point cloud sequence and a corresponding laser point cloud sequence. Then, based on the laser point cloud segments corresponding to times ij to ix (i+j) in the laser point cloud sequence, it determines the ith road boundary line segment. Next, based on the ith road boundary line segment, it labels the feature points in the ith frame of the millimeter-wave point cloud corresponding to time ix in the first millimeter-wave point cloud sequence, determining the label corresponding to each feature point in the ith frame. Finally, based on the labeled second millimeter-wave point cloud sequence, it trains the initial model to obtain a classification model. Thus, by using the laser point cloud segments corresponding to time ix and j times before and after the acquired laser point cloud sequence, the ith road boundary line segment is determined. Based on the ith road boundary line segment, the label corresponding to each feature point in the ith millimeter-wave point cloud corresponding to time ix in the acquired millimeter-wave point cloud sequence is determined. Based on the labeled millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model, thereby improving the classification effect and quality of the classification model and enhancing the accuracy and reliability of the generated classification model.

[0239] To implement the above embodiments, this disclosure also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the road boundary detection method and the classification model generation method proposed in the foregoing embodiments of this disclosure.

[0240] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the road boundary detection method and the classification model generation method proposed in the foregoing embodiments of this disclosure.

[0241] Figure 13 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 13The electronic device 1300 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0242] like Figure 13 As shown, the electronic device 1300 is presented in the form of a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: one or more processors or processing units 1316, system memory 1328, and bus 1318 connecting different system components (including system memory 1328 and processing unit 1316).

[0243] Bus 1318 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. Examples of these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0244] Electronic device 1300 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 1300, including volatile and non-volatile media, removable and non-removable media.

[0245] Memory 1328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1330 and / or cache memory 1332. Electronic device 1300 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1334 may be used to read and write non-removable, non-volatile magnetic media (… Figure 13 Not shown; usually referred to as a "hard drive"). Although Figure 13Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media). In these cases, each drive may be connected to bus 1318 via one or more data media interfaces. Memory 1328 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0246] A program / utility 1340 having a set (at least one) of program modules 1342 may be stored, for example, in memory 1328. Such program modules 1342 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1342 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0247] Electronic device 1300 can also communicate with one or more external devices 1314 (e.g., keyboard, pointing device, display 1324, etc.), and with one or more devices that enable a user to interact with electronic device 1300, and / or with any device that enables electronic device 1300 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1322. Furthermore, electronic device 1300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1320. As shown, network adapter 1320 communicates with other modules of electronic device 1300 via bus 1318. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0248] The processing unit 1316 executes various functional applications and data processing by running programs stored in the system memory 1328, such as implementing the methods mentioned in the foregoing embodiments.

[0249] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0250] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0251] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0252] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0253] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0254] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0255] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0256] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A road boundary detection method, characterized in that, include: Acquire the first point cloud frame currently collected by the millimeter-wave radar; The feature points in the first point cloud frame are preprocessed to obtain the point cloud data to be identified. The point cloud data to be identified is input into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model, wherein the preset classification model is obtained based on the method described in any one of claims 6-11; The current road boundary is determined based on the feature points whose corresponding classification labels are road boundaries.

2. The method as described in claim 1, characterized in that, The preprocessing of feature points in the first point cloud frame includes at least one of the following: Discard feature points whose corresponding height values ​​are not within the preset height range within the first point cloud frame; Feature points in the first point cloud frame whose difference between the first Doppler velocity value and the second Doppler velocity value is greater than the difference threshold are discarded, wherein the second Doppler velocity value is the velocity value corresponding to the static feature point; The feature points in the first point cloud frame are fused with the feature points in at least one second point cloud frame, wherein the second point cloud frame is obtained by preprocessing a third point cloud frame that was collected before the first point cloud frame and is adjacent to the first point cloud frame.

3. The method as described in claim 2, characterized in that, The method further includes: Based on the vehicle's current speed and the position of each feature point, determine the reference Doppler velocity corresponding to each feature point; The feature points whose corresponding Doppler velocities match the reference Doppler velocity are determined as the static feature points.

4. The method as described in claim 2, characterized in that, The step of fusing feature points in the first point cloud frame with feature points in at least one second point cloud frame includes: A predetermined number of feature points in the first point cloud frame are fused with a predetermined number of feature points in at least one second point cloud frame.

5. The method as described in claim 4, characterized in that, The method further includes any one of the following: Randomly select the preset number of feature points from the first point cloud frame; From the region in the first point cloud frame where the feature point density is greater than the density threshold, a preset number of feature points are randomly selected.

6. A method for generating a classification model, characterized in that, include: Obtain the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence; Based on the laser point cloud segments corresponding to the times from the ijth time to the (i+jth time)th time in the laser point cloud sequence, determine the ith road boundary line segment, where i and j are natural numbers; Based on the i-th road boundary line segment, the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence are labeled, and the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud is determined. Based on the labeled second millimeter-wave point cloud sequence, the initial model is trained to obtain a classification model, which is used to classify each feature point in the millimeter-wave point cloud.

7. The method as described in claim 6, characterized in that, The step involves labeling feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the millimeter-wave point cloud sequence based on the i-th road boundary line segment, and determining the label corresponding to each feature point in the i-th frame of millimeter-wave point cloud, including: Based on the relative position between the i-th road boundary line segment and the vehicle, the reference area corresponding to the i-th road boundary line segment is determined; The label corresponding to the feature point in the i-th frame millimeter-wave point cloud located within the reference area is determined as the first label, and the label corresponding to the feature point not located within the reference area is determined as the second label.

8. The method as described in claim 7, characterized in that, The step of determining the reference area corresponding to the i-th road boundary line segment based on the relative position between the i-th road boundary line segment and the vehicle includes any one of the following: When the i-th road boundary line segment is located on the first side of the vehicle, the distance between the first boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the second boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value; When the i-th road boundary line segment is located on the second side of the vehicle, the distance between the second boundary line of the reference area and the i-th road boundary line segment is determined as a first value, and the distance between the first boundary line and the i-th road boundary line segment is determined as a second value, wherein the first boundary line is located on the first side of the second boundary line, and the first value is greater than the second value.

9. The method as described in any one of claims 6-8, characterized in that, The initial model is trained based on the labeled second millimeter-wave point cloud sequence to obtain a classification model, including: The second millimeter-wave point cloud sequence is input into the initial model to obtain the predicted label corresponding to each feature point output by the initial model. The first loss value is determined based on the difference between the predicted label and the labeled label corresponding to each feature point; Determine the minimum distance between the feature point whose predicted label is the first label and the i-th road boundary line segment; The second loss value is determined based on the mean of the minimum distances corresponding to all feature points of the first label. Based on the first loss value and the second loss value, the initial model is reverse-corrected until the classification model is obtained.

10. The method according to any one of claims 6-8, characterized in that, The acquisition of the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence includes: Acquire the third millimeter-wave point cloud sequence and the laser point cloud sequence collected by the millimeter-wave radar and lidar, respectively; The third millimeter-wave point cloud sequence is preprocessed to obtain the first millimeter-wave point cloud sequence.

11. The method as described in claim 10, characterized in that, The preprocessing of the third millimeter-wave point cloud sequence includes at least one of the following: Discard feature points whose corresponding height values ​​are not within the preset height range in each point cloud frame of the third millimeter-wave point cloud sequence. Feature points in each point cloud frame of the third millimeter-wave point cloud sequence whose difference between the first Doppler velocity value and the second Doppler velocity value is greater than a difference threshold are discarded, wherein the second Doppler velocity value is the velocity value corresponding to the static point; The feature points in each point cloud frame of the third millimeter-wave point cloud sequence are fused with the feature points in at least one reference point cloud frame. The reference point cloud frame is obtained by preprocessing at least one point cloud frame that was acquired before the point cloud frame and is adjacent to the point cloud frame.

12. A road boundary detection device, characterized in that, The device includes: The first acquisition module is used to acquire the first point cloud frame currently collected by the millimeter-wave radar. The processing module is used to preprocess the feature points in the first point cloud frame to obtain the point cloud data to be identified. The input module is used to input the point cloud data to be identified into a preset classification model to obtain the classification label corresponding to each feature point output by the classification model, wherein the preset classification model is obtained based on the method described in any one of claims 6-11; The first determination module is used to determine the current road boundary based on feature points whose corresponding classification labels are road boundaries.

13. A device for generating a classification model, characterized in that, The device includes: The second acquisition module is used to acquire the first millimeter-wave point cloud sequence and the corresponding laser point cloud sequence; The second determining module is used to determine the i-th road boundary line segment based on the laser point cloud segments corresponding to the i-th to i+j-th times in the laser point cloud sequence, where i and j are natural numbers. The annotation module is used to annotate the feature points in the i-th frame of millimeter-wave point cloud corresponding to the i-th time in the first millimeter-wave point cloud sequence based on the i-th road boundary line segment, and to determine the annotation label corresponding to each feature point in the i-th frame of millimeter-wave point cloud. The training module is used to train the initial model based on the labeled second millimeter-wave point cloud sequence to obtain a classification model, wherein the classification model is used to classify each feature point in the millimeter-wave point cloud.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the road boundary detection method according to any one of claims 1-5, or the classification model generation method according to any one of claims 6-11.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the road boundary detection method according to any one of claims 1-5, or the classification model generation method according to any one of claims 6-11.

Images