A 3D lane line detection model training method and a 3D lane line detection method

Abstract

The application discloses a 3D lane line detection model training method and a 3D lane line detection method. The training method comprises the following steps: step 1, constructing a 3D lane line detection model, wherein the 3D lane line detection model comprises at least a 3D lane line branch and a 2D lane line branch; step 2, constructing a network loss function according to a first perception result of the 3D lane line branch and a second perception result of the 2D lane line branch respectively, and iteratively training the 3D lane line detection model based on the network loss function; and step 3, if it is determined that a preset early stopping condition is met, stopping the iterative training of the 2D lane line branch, continuing the iterative training of the 3D lane line branch until a preset termination condition is met, and saving the model parameters of the last iterative training of the 3D lane line branch. Through the technical scheme in the application, the training effect of the three-dimensional lane line detection model under a complex scene can be improved, and the detection accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a 3D lane line detection model training method and a 3D lane line detection method. BACKGROUND

[0002] With the rapid development of automatic driving technology, higher requirements are put forward for the accuracy and real-time performance of lane line detection. Traditional 2D lane line methods are usually based on the assumption of planar roads, ignoring the actual three-dimensional structure of the lane, and are difficult to cope with complex road environments such as lane lines being blocked by vehicles or overlapping; lane lines of different lengths, different curvatures, and complex backgrounds, etc.

[0003] Therefore, 3D lane line detection technology based on 3D anchor lines has emerged as the times require. This 3D lane line detection method adopts an anchor line mechanism, uses 3D anchor lines as predefined 3D lane lines, and realizes more accurate and efficient detection through deep learning, thereby improving detection speed and achieving accurate 3D positioning.

[0004] However, in the prior art, although existing three-dimensional lane line detection methods have achieved good results, there are still some limitations in three-dimensional lane line detection in some complex scenarios. For example:

[0005] 1. When the lane line is blocked by a vehicle or overlaps;

[0006] 2. For lane lines of different lengths, different curvatures, and complex backgrounds.

[0007] For such complex scenarios, the detection accuracy of existing three-dimensional lane line detection methods is low, and there is false detection. Even if a sufficient number of training data is added, the final detection effect is not ideal. SUMMARY

[0008] The purpose of the present application is to train a three-dimensional lane line detection model in a complex scenario and improve the accuracy of three-dimensional lane line detection.

[0009] The technical scheme of the first aspect of the application is: a 3D lane line detection model training method is provided, which comprises the following steps: step 1: constructing a 3D lane line detection model, wherein the 3D lane line detection model comprises at least a 3D lane line branch and a 2D lane line branch; step 2: constructing a network loss function according to a first perception result of the 3D lane line branch and a second perception result of the 2D lane line branch, respectively, and iteratively training the 3D lane line detection model based on the network loss function; and step 3: if it is determined that a preset early stopping condition is met, stopping the iterative training of the 2D lane line branch, continuing the iterative training of the 3D lane line branch until a preset termination condition is met, and saving the model parameters of the last iteration of the 3D lane line branch, wherein the 3D lane line branch is used to output a 3D lane line candidate anchor line, and the 3D lane line candidate anchor line is used to determine a lane line.

[0010] In any of the above technical solutions, further, the network loss function comprises at least a 3D loss function, a 2D loss function and a category divergence loss function, in step 2, the 3D lane line detection model is iteratively trained based on the network loss function, specifically comprising the following steps: step 21: inputting sample data into the 3D lane line detection model to output the first perception result and the second perception result, wherein the first perception result is predicted by the 3D lane line branch, and the second perception result is predicted by the 2D lane line branch; step 22: substituting the first perception result and first label data into the 3D loss function to calculate a first loss value, and substituting the second perception result and second label data into the 2D loss function to calculate a second loss value; step 23: substituting the first perception result and the second perception result into the category divergence loss function to calculate a category divergence loss value; and step 24: weighting the first loss value, the second loss value and the category divergence loss value to obtain a weighted loss value, and determining the model parameters in the 3D lane line branch and the 2D lane line branch in the next iteration training by minimizing the weighted loss value to iteratively train the 3D lane line detection model.

[0011] In any of the above technical solutions, further, the 2D loss function Loss 2D is calculated according to the following formula:

[0012]

[0013] In the formula, a 2d_cls is a 2D classification loss weight, m is a label of a current anchor line, M is a total number of anchor lines in the second label data, L is a total number of lane line categories, l is a category number of the current anchor line, cls' is a probability value (with a value range of 0-1) that the anchor line m is predicted as a category 1 in the second perception result, cls is a true value (with a value of 0 or 1) that the anchor line m is marked as the category 1 in the second label data, and a 2d_coordis a 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u mn ,v′ mn ) is the image coordinates of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinates of the corresponding nth point on the mth anchor line in the second label data.

[0014] In any of the technical solutions above, further, in step 3, it is determined that the preset early stopping condition is met, specifically comprising: step 31: in the process of each iteration training, the loss value of the network loss function is calculated, when it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current iteration number is labeled; step 32: after the 3D lane line detection model is iteratively trained for the first training number of times, the 3D lane line detection model is iteratively trained for the second training number of times, and in the process of the second training number of iterations, the trained 3D lane line detection model is selected multiple times, and the detection indicators of the selected multiple 3D lane line detection models on the verification data are calculated in turn; step 33: it is determined whether any of the multiple detection indicators is greater than the reference indicator, if yes, the reference indicator is updated based on the maximum value of the multiple detection indicators, the iteration training of the 3D lane line detection model for the second training number of times is continued, the loss value of the network loss function and the detection indicator are calculated, if no, when it is determined that the labeled iteration number does not belong to the second training number, it is determined that the preset early stopping condition is met, and when it is determined that the labeled iteration number belongs to the second training number, after the 3D lane line detection model is iteratively trained for a third training number of times, it is determined that the preset early stopping condition is met.

[0015] In any of the technical solutions above, further, the detection indicator at least includes one of F1 score, precision, and recall.

[0016] In any of the technical solutions above, further, the reference indicator is the detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model is iteratively trained for the first training number of times.

[0017] The technical solution of the second aspect of the present application is: a 3D lane line detection method is provided, which comprises: step 100: based on the collected road surface image, a trained 3D lane line detection model is used to output 3D lane line candidate anchor lines, wherein the 3D lane line detection model is trained by using the 3D lane line detection model training method of any of the first aspect technical solutions; step 200: the 3D lane line candidate anchor lines are screened, and a 3D lane line detection result is generated based on the screened 3D lane line candidate anchor lines in a curve fitting manner.

[0018] In any of the technical solutions above, further, the 3D lane line candidate anchor lines are screened, specifically including: step 201: based on the confidence score of the 3D lane line candidate anchor lines, the 3D lane line candidate anchor lines are sorted, and the 3D lane line candidate anchor line corresponding to the maximum confidence score is recorded as a first candidate anchor line; step 202: the three-dimensional anchor line information of the sorted 3D lane line candidate anchor lines is sequentially obtained, and the anchor point distance mean value is calculated with the first candidate anchor line respectively; step 203: based on the anchor point distance mean value, the sorted 3D lane line candidate anchor lines are screened.

[0019] In any of the technical solutions above, further, the three-dimensional anchor line information includes the visibility attribute of each anchor point, and the method further includes: step 211: based on the visibility attribute of each anchor point in the 3D lane line candidate anchor line, an initial truth matrix is generated with each 3D lane line candidate anchor line as a column and in the first anchor line order; step 212: from top to bottom, the elements of each column in the initial truth matrix are sequentially added to generate a first matrix; step 213: based on the visibility attribute of each anchor point in the 3D lane line candidate anchor line, an intermediate truth matrix is generated with each 3D lane line candidate anchor line as a column and in the second anchor line order, wherein the second anchor line order is symmetric to the first anchor line order; step 214: from top to bottom, the elements of each column in the intermediate truth matrix are sequentially added to generate a second matrix; step 215: the values of the elements in the same position in the first matrix and the second matrix are sequentially compared in the element position, and if the values are all greater than 0, the value of the element at the corresponding position in the initial truth matrix is set to 1, otherwise, it is set to 0 to generate a visibility attribute matrix; step 216: based on the visibility attribute matrix, the visibility attribute of each anchor point in the three-dimensional anchor line information is corrected.

[0020] In any of the technical solutions above, further, the method further includes: step 204: when it is determined that the anchor point distance mean value of any 3D lane line candidate anchor line belongs to a first preset interval, a near-distance same-class suppression value between the 3D lane line candidate anchor line and the first candidate anchor line is calculated; step 205: it is judged whether the near-distance same-class suppression value belongs to a second preset interval, and if yes, the 3D lane line candidate anchor line is retained, and if not, the 3D lane line candidate anchor line is deleted.

[0021] The beneficial effects of the present application are:

[0022] The technical solutions in the application are configured with a 2D lane line branch in the 3D lane line detection model to better understand and process the boundary and shape information of the lane line in the training process of the model, classify each pixel of the lane line, and enable the model to learn more accurate target representation. The loss function is modified based on the 3D perception result and the 2D perception result, thereby improving the accuracy and robustness of 3D lane line detection.

[0023] In addition, the application considers that the 2D lane line branch is mainly trained for 2D images and 2D prediction results, and in order to avoid over-reliance of the training result on the 2D lane line branch, an early stopping condition is introduced. In the process of iterative training, if the early stopping condition is met, the training of the 2D lane line branch is stopped, and the training of the 3D lane line branch is continued until the preset termination condition is met, the model training is completed, and only the model parameters of the 3D lane line branch are saved as the trained model. This strategy not only effectively prevents the model from over-relying on 2D information, but also promotes the overall training convergence of the model, further improving the practical value and performance of the model.

[0024] In addition, in the detection process, the 3D lane line candidate anchor lines are filtered based on the visibility of the anchor points in the lane line, and the visibility of the anchor points is corrected based on the symmetry of the lane line, to ensure the accuracy of the basis for 3D lane line detection and further improve the accuracy of model detection. BRIEF DESCRIPTION OF DRAWINGS

[0025] The advantages of the above and / or additional aspects of the application will become apparent and easy to understand in connection with the description of embodiments, which will be described in conjunction with the following drawings, in which:

[0026] Figure 1 is a schematic flow chart of a 3D lane line detection model training method according to an embodiment of the application;

[0027] Figure 2 is a schematic block diagram of data flow in the model training process according to an embodiment of the application;

[0028] Figure 3 is a schematic diagram of anchor line information in a 3D lane line candidate anchor line according to an embodiment of the application;

[0029] Figure 4 is a partial schematic diagram of an initial true value matrix according to an embodiment of the application;

[0030] Figure 5 is a partial schematic diagram of a first matrix according to an embodiment of the application;

[0031] Figure 6is a partial schematic view of a second matrix according to an embodiment of the present application;

[0032] Figure 7 is a partial schematic view of a visible attribute matrix according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable every person skilled in the art to better understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0035] Embodiment 1

[0036] As shown in Figure 1 and Figure 2 The present embodiment provides a 3D lane line detection model training method, which comprises the following steps:

[0037] Step 1: Constructing a 3D lane line detection model, wherein the 3D lane line detection model at least comprises a 3D lane line branch and a 2D lane line branch; wherein the 3D lane line branch is used to output a 3D lane line candidate anchor line, the 3D lane line candidate anchor line is used to determine a lane line, and the 2D lane line branch is used to assist the 3D lane line branch, so as to improve the accuracy and robustness of the overall lane line detection of the model by better understanding and processing the boundary and shape information of the lane line.

[0038] Step 2: Constructing a network loss function according to a first perception result of the 3D lane line branch and a second perception result of the 2D lane line branch respectively, and iteratively training the 3D lane line detection model based on the network loss function;

[0039] Step 3: If it is determined that the preset early stopping condition is met, the iterative training of the 2D lane line branch is stopped, the iterative training of the 3D lane line branch is continued until the preset termination condition is met, and the model parameters of the last iteration training of the 3D lane line branch are saved, the training of the 3D lane line detection model is completed, and the predicted 3D lane line candidate anchor line is output.

[0040] Specifically, in the training process of the 3D lane line detection model, the sample data (RGB image and first and second label data) are input into the backbone network in the 3D lane line detection model, and then two-dimensional image encoded image features are obtained. The image features are then transmitted to two branches, i.e., the 3D lane line branch and the 2D lane line branch.

[0041] The 3D lane line branch, as the main branch of the model, generates a first perception result and outputs a 3D lane line candidate anchor line, which contains predicted three-dimensional lane line information, such as left and right offset coordinates x of an anchor point, height offset coordinates z, and visibility of the anchor point. The 2D lane line branch, as an auxiliary training branch of the model, obtains a second perception result during training, which contains segmented 2D lane line information, and assists in training, supervision, and correction of the main branch to improve the prediction effect of the model.

[0042] Then, the first perception result and the second perception result in each iteration training process are used to calculate the corresponding loss value by combining the set network loss function, and the iteration training is performed.

[0043] However, considering that the 2D lane line branch is mainly trained for 2D images and 2D prediction results, and the embodiment is to obtain a 3D lane line candidate anchor line, an early stopping condition is introduced to avoid excessive dependence of the training result on the 2D lane line branch. During the iteration training, if the early stopping condition is met, the training of the 2D lane line branch is stopped, and the training of the 3D lane line branch is continued until the preset termination condition is met, and only the model parameters of the 3D lane line branch are saved as the trained model. The early stopping condition can be set according to actual needs, for example, the iteration threshold (first threshold) of the iteration training and the convergence threshold of the loss function. The termination condition can be the iteration threshold (second threshold, greater than the first threshold) of the iteration training.

[0044] When the training process meets the termination condition and stops, the final model parameters obtained during training are saved as the 3D lane line detection model, and the 3D lane line candidate anchor line is output by using the optimized model.

[0045] It should be noted that the 2D lane line branch only assists the 3D lane line branch in training during the training process and does not participate in 3D lane line detection after model training, so the model parameters of the 2D lane line branch do not need to be saved.

[0046] In any of the above embodiments, further, the network loss function at least includes a 3D loss function, a 2D loss function, and a category divergence loss function. In step 2, the 3D lane line detection model is iteratively trained based on the network loss function, specifically including:

[0047] Step 21: Input the sample data into the 3D lane detection model and output a first perception result and a second perception result. The first perception result is obtained by the 3D lane branch prediction, and the second perception result is obtained by the 2D lane branch prediction.

[0048] Step 22: Substitute the first perception result and the first label data into the 3D loss function to calculate the first loss value, and substitute the second perception result and the second label data into the 2D loss function to calculate the second loss value;

[0049] Step 23: Substitute the first perception result and the second perception result into the category divergence loss function to calculate the category divergence loss value;

[0050] Step 24: Weight the first loss value, the second loss value, and the category divergence loss value to obtain a weighted loss value. By minimizing the weighted loss value, the model parameters in the 3D lane line branch and the 2D lane line branch in the next iterative training are determined to iteratively train the 3D lane line detection model.

[0051] Specifically, the network loss function set in this embodiment is obtained by weighted summing the 3D loss function, the 2D loss function, and the category divergence loss function according to the set weights.

[0052] During the training process, Figure 2 As shown, the sample data (RGB image data and corresponding label data) can be used as input to the 3D lane detection model. The training process is roughly as follows:

[0053] 1. Randomly extract a batch from the sample data as the data used in this iteration.

[0054] 2. The batch of sample data is forward propagated through the network, and based on the current model parameters, the prediction results, namely the first perception results and the second perception results, are obtained.

[0055] 3. Substitute the first perception result and the corresponding first label data into the 3D loss function to calculate the first loss value; at the same time, substitute the second perception result and the corresponding second label data into the 2D loss function to calculate the second loss value.

[0056] 4. Substitute the first perception result and the second perception result into the category divergence loss function for calculation to obtain the category divergence loss value between the first perception result and the second perception result.

[0057] 5. According to the set weights, the first loss value, the second loss value and the category divergence loss value are weighted and summed to obtain the loss value of the total network loss function, which is used as the basis for judging whether the termination condition is met.

[0058] 6. If the termination condition is reached, stop the iterative training, save the model parameters of this iteration training; if the termination condition is not reached, calculate the gradient of the loss of the two branches with respect to the model parameters of this iteration training by using the back propagation algorithm, to provide the basis for the model parameter update of the next iteration training.

[0059] 7. According to the gradient, the model parameters of the two branches are optimized and adjusted, and a batch of sample data is randomly extracted again for the next iteration training.

[0060] In any of the above embodiments, further, the 2D loss function Loss 2D is calculated as follows:

[0061]

[0062] In the formula, a 2d_cls is a 2D classification loss weight, m is the label of the current anchor line, M is the total number of anchor lines in the second label data, L is the total number of lane line categories, l is the category number of the current anchor line, cls′ ml is the probability value of anchor line m being predicted as category l in the second perception result (the value range is 0-1), cls ml is the true value of anchor line m being marked as category 1 in the second label data (the value is 0 or 1), a 2d_coord is a 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u′ mn ,v′ mn ) is the image coordinates of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinates of the corresponding nth point on the mth anchor line in the second label data.

[0063] In this embodiment, the calculation formula of the 3D loss function Loss 3D is as follows:

[0064]

[0065] In the formula, a 3d_cls is a 3D classification loss weight, m is the label of the current anchor line, M is the total number of anchor lines in the first label data (which is the same as the total number of anchor lines in the second label data), C is the total number of lane line categories, c is the category number of the current anchor line, p mc is the probability value of the mth anchor line being predicted as category c in the first perception result (the value range is 0-1), q mc represents the true value of the category of the mth anchor line in the first label data, if it is category c, then q mc =1, if it is not category c, then qmc = 0.

[0066] α 3d_coord is a 3D point regression loss weight, M1 represents the total number of three-dimensional anchor lines, N1 represents the number of points contained in each three-dimensional anchor line, respectively, the x coordinate value and the z coordinate value of the n1th point of the m1th three-dimensional anchor line in the first perception result, respectively, the x coordinate value and the z coordinate value of the n1th point of the m1th three-dimensional anchor line in the first label data, is the visibility attribute of the n1th point of the m1th three-dimensional anchor line in the first perception result, is the visibility attribute of the n1th point of the m1th three-dimensional anchor line in the first label data.

[0067] It should be noted that the 3D loss function Loss 3D in the embodiment can be set according to actual needs.

[0068] The weight parameters described above can be adjusted according to the actual training effect of the model, or can be set as fixed values, such as α 2d_cls = 0.15, α 2d_coord = 0.15, α 3d_cls = 0.35, and α 3d_coord = 0.35.

[0069] Considering that the 2D lane line branch and the 3D lane line branch have different feature extraction methods and different perspectives and dimensions of the same object, in order to make the detection results of the 3D lane line branch and the 2D lane line branch more dispersed in the category space, thereby improving the discrimination degree between different categories, and at the same time promoting the model to consider two-dimensional and three-dimensional information in the optimization process, realizing the complementation of information, and improving the accuracy and robustness of the detection result. The embodiment also constructs a category dispersion loss to modify the total loss function, so the total loss function of the embodiment is:

[0070] Loss(line) = Loss 2D + Loss 3D + Loss div

[0071]

[0072] In the formula, Loss div is a category dispersion loss function, m is the label of the current anchor line, M is the total number of anchor lines in the first label data (which is the same value as the total number of anchor lines in the second label data), C is the total number of lane line categories, c is the category number of the current anchor line, is the probability value of predicting that the mth anchor line in the first perception result is of category c. is a probability value of predicting the mth anchor line class as c in the second perception result.

[0073] It should be noted that when it is determined that the preset early stopping condition is met, the above-mentioned category divergence loss also no longer participates in the training of the model, that is, after the early stopping condition is met, the total loss function is Loss 3D .

[0074] In any of the above embodiments, further, in order to improve the training effect of the 2D lane line branch and the overall 3D lane line detection model, the early stopping condition is set in this embodiment, which includes a convergence threshold and an iteration number threshold, and the number of iteration training is adjusted in real time. Therefore, in step 3, it is determined that the preset early stopping condition is met, which specifically includes:

[0075] Step 31: In the process of any iteration training, the loss value of the network loss function is calculated, when it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current iteration number is labeled;

[0076] Step 32: After the 3D lane line detection model is iteratively trained for the first training number of times, the 3D lane line detection model is iteratively trained for the second training number of times, and in the process of the second training number of iterations, the trained 3D lane line detection model is selected multiple times, and the detection indexes of the selected multiple 3D lane line detection models on the verification data are calculated in turn;

[0077] Step 33: whether any of the multiple detection indexes is greater than the reference index,

[0078] If yes, update the reference index based on the maximum value of the multiple detection indexes, continue to iteratively train the 3D lane line detection model for the second training number of times, calculate the loss value of the network loss function and the detection index,

[0079] If not, when it is determined that the labeled iteration number does not belong to the second training number, it is determined that the preset early stopping condition is met,

[0080] When it is determined that the labeled iteration number belongs to the second training number, after the 3D lane line detection model is iteratively trained for the third training number of times, it is determined that the preset early stopping condition is met.

[0081] Specifically, in the model training process, the first training number can be set to 300, the initial convergence threshold is 0.6, and then in the process of 300 iteration training, the loss value of the above network loss function is calculated every time, and the loss value is compared with the initial convergence threshold 0.6, if the loss value corresponding to the iteration training is 0.58, the convergence threshold is updated to 0.58, and the iteration training number is labeled and marked.

[0082] When the iteration training meets 300 times, the second training number of iteration training is performed, and the second training number is less than the first training number, for example, it is set to 50 times.

[0083] In the process of 50 iteration training, 3D lane line detection model after multiple iteration training can be randomly selected or equally selected, and the verification data is input into the model after this training, and the corresponding detection index is calculated, wherein the detection index can be set according to actual demand, for example, F1 score.

[0084] Now this embodiment takes calculating 5 detection indexes F1 score as an example for illustration.

[0085] After obtaining 5 F1 scores, the 5 F1 scores are compared with the reference index to determine whether there is a score greater than the reference index in the 5 F1 scores. For example, if the reference index is set to 80, and if there is an F1 score of 81 in the 5 F1 scores, the reference index is updated to 81, and the above model is trained for 50 times again, and the above process is repeated to obtain 5 F1 scores again.

[0086] If the 5 F1 scores obtained again are all less than or equal to 81, and 400 iteration training has been performed, it is determined whether the training number of the latest updated convergence threshold, i.e. the iteration number of the labeled iteration, is in the 351th-400th iteration training number in the process of 400 iteration training. If the training number of the updated convergence threshold is the 200th, it is determined that the early stopping condition is met, the training of the 2D lane line branch is stopped, and only the iteration training of the 3D lane line branch is performed until the termination condition is met.

[0087] If the training number of the updated convergence threshold is the 360th, the third training number (such as 30 times, 50 times) of iteration training of the overall 3D lane line detection model is performed, and it is determined that the early stopping condition is met, the training of the 2D lane line branch is stopped, and the F1 score does not need to be calculated in the process of the third training number of iteration training.

[0088] It should be noted that you can also set a termination training number in the early stopping condition, such as 500 times. When the iterative training reaches 500 times, even if the above-set conditions are not met, it is directly determined that the iterative training meets the early stopping condition and the training of the 2D lane line branch is stopped.

[0089] In any of the above embodiments, further, the detection indicator includes at least one of F1 score, precision, and recall rate.

[0090] In any of the above embodiments, further, the benchmark indicator is: a detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model performs iterative training for the first number of training times.

[0091] Example 2:

[0092] This embodiment provides a 3D lane line detection method, which includes:

[0093] Step 100: Based on the collected road surface image, using a trained 3D lane detection model, outputting 3D lane line candidate anchor lines, wherein the 3D lane detection model is trained using the 3D lane detection model training method of any one of the embodiments 1;

[0094] Step 200: Screen candidate 3D lane line anchor lines, and generate 3D lane line detection results based on the screened candidate 3D lane line anchor lines using a curve fitting method.

[0095] Specifically, such as Figure 3 As shown in the figure, the model outputs a total of N 3D lane line candidate anchor lines. Each 3D lane line candidate anchor line includes information such as the confidence score (of the entire anchor line), the left and right offset coordinates x of the anchor point, the height offset coordinate z, the visible attribute vis of the anchor point, and the category probability value cls. The length of each 3D lane line candidate anchor line is set to 100 meters, and a point (i.e., anchor point) is sampled every 5 meters, forming a sequence of 20 anchor points.

[0096] Each anchor point has three attributes: the left and right offset coordinates x predicted by the model, the height offset coordinate z, and the visible attribute vis. When the visible attribute vis is "true", it means that the anchor point is on the lane line, and when it is "false", it means that it is not.

[0097] Therefore, after obtaining the 3D lane line candidate anchor lines output by the model, the set rules can be used to filter out the 3D lane line candidate anchor lines, and then the retained, more accurate 3D lane line candidate anchor lines can be used to generate the final 3D lane line detection results. Among them, the rules can be set according to actual needs.

[0098] For example, based on the confidence score of each 3D lane line candidate anchor line, the 3D lane line candidate anchor line for curve fitting is screened, the 3D lane line candidate anchor line with a confidence score lower than a certain threshold is filtered out, and the 3D lane line candidate anchor line with a higher confidence score is retained.

[0099] For another example, the number of anchor points with a visible attribute vis continuously being "true" in each 3D lane line candidate anchor line is counted, if the number of anchor points with a visible attribute vis continuously being "true" is greater than a certain threshold, it is considered that the anchor line is long enough, and it is retained, otherwise, it is filtered out.

[0100] The curvature offset of the corresponding 3D lane line candidate anchor line can also be calculated based on the left and right offset coordinates x of each anchor point, and by judging whether the difference value of the left and right offset coordinates x of each 3D lane line candidate anchor line in the transverse direction exceeds a certain threshold, the 3D lane line candidate anchor line with a larger curvature is removed, and the 3D lane line candidate anchor line with a smaller coverage area is retained.

[0101] In the process of fitting the lane line, the curve fitting method used in this embodiment can be a cubic parametric spline curve fitting algorithm, or other curve fitting algorithms, and the specific fitting process will not be described again.

[0102] In any of the above embodiments, further, the 3D lane line candidate anchor line is screened, specifically including:

[0103] Step 201: Based on the confidence score of the 3D lane line candidate anchor line, the 3D lane line candidate anchor line is sorted, and the 3D lane line candidate anchor line corresponding to the maximum confidence score is recorded as the first candidate anchor line;

[0104] Step 202: The three-dimensional anchor line information of each anchor point in the sorted 3D lane line candidate anchor line is obtained in turn, and the anchor points in the sorted 3D lane line candidate anchor line selected in turn are one-to-one corresponding to the anchor points in the first candidate anchor line, and the average distance between the two anchor lines is calculated.

[0105] Step 203: Based on the average distance between the anchor points, the sorted 3D lane line candidate anchor line is screened.

[0106] Specifically, first, the multiple 3D lane line candidate anchor lines predicted by the model are sorted in order of confidence score from high to low, and the 3D lane line candidate anchor line with the highest score is saved to the list keep, and the 3D lane line candidate anchor line is recorded as the first candidate anchor line.

[0107] The three-dimensional anchor line information of the first candidate anchor line is extracted, including at least the left and right offset coordinates x of the anchor point, the height offset coordinate z, and the visible attribute vis of the anchor point, and the category probability value cls is obtained by using functions such as argmax.

[0108] Select any of the sorted 3D lane line candidate anchor lines in turn, divide it by the 3D lane line candidate anchor lines saved in the keep list, and use it as the second candidate anchor line. Repeat the above process to obtain the 3D anchor line information of the second candidate anchor line and calculate the mean anchor point distance. The corresponding calculation formula is:

[0109]

[0110] Where, dis A_p is the mean distance between anchor points, I represents the total number of anchor points on each candidate anchor line of 3D lane line (set to 20 in this embodiment), i represents the number of the current anchor point, (X i ,Z i )、(x i ,z i are the left and right offset coordinates x and height offset coordinate z of the i-th anchor point in the first candidate anchor line and the second candidate anchor line, respectively. Ai 、vis pi are the visible attributes of the i-th anchor point in the first candidate anchor line and the second candidate anchor line respectively.

[0111] If the calculated mean anchor point distance is greater than or equal to a set threshold (e.g., 2, 2.5, etc.), the second candidate anchor line is retained and saved to the keep list. Otherwise, the second candidate anchor line is deleted. The next 3D lane line candidate anchor line is then selected until all 3D lane line candidate anchor lines are screened. Finally, curve fitting is performed on the 3D lane line candidate anchor lines saved in the keep list to generate the 3D lane line detection result.

[0112] In any of the above embodiments, further, the three-dimensional anchor line information includes a visibility attribute of each anchor point, and the method further includes:

[0113] Step 211: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, generate an initial truth matrix in the order of the first anchor line, with each 3D lane line candidate anchor line as a column;

[0114] Specifically, such as Figure 4 As shown in the figure, the trained model will predict multiple 3D lane line candidate anchor lines in the current road image. Each 3D lane line candidate anchor line will contain multiple (e.g., 20) anchor points. Each anchor point has a visible attribute. For example, "anchor point visible" can be set to "true" and the corresponding value is set to "1"; otherwise, it is set to "false" and the corresponding value is "0".

[0115] The order of the road image from left to right is set as the first anchor line order, and the visible attributes of each anchor point in each 3D lane line candidate anchor line are counted in turn to generate the initial truth value matrix.

[0116] Step 212: from top to bottom, the elements of each column in the initial true value matrix are sequentially accumulated to generate a first matrix;

[0117] Specifically, as shown in Figure 5 , considering that the bottom of the road image is closer to the vehicle, the reliability of the anchor points in the predicted 3D lane line candidate anchor line is higher, therefore, in the order from top to bottom, each 3D lane line candidate anchor line is sequentially accumulated and calculated, and the accumulated value of each element in the corresponding column in the initial true value matrix is saved to generate a first matrix.

[0118] Step 213: based on the visibility attribute of each anchor point in the 3D lane line candidate anchor line, taking each 3D lane line candidate anchor line as a column, a second anchor line order is used to generate an intermediate true value matrix, wherein the second anchor line order is symmetrical to the first anchor line order left and right.

[0119] Specifically, the order of the road image from right to left is set as the second anchor line order, the process of step 211 is repeated, or the initial true value matrix is directly flipped left and right to generate the intermediate true value matrix, and the specific process is not repeated here.

[0120] Step 214: from top to bottom, the elements of each column in the intermediate true value matrix are sequentially accumulated to generate a second matrix;

[0121] Step 215: comparing the values of the elements at the same position in the first matrix and the second matrix in order of element position, if the values of the elements at the same position in the first matrix and the second matrix are both greater than 0, the value of the element at the corresponding position in the initial true value matrix is set to 1, otherwise, it is set to 0, to generate a visibility attribute matrix.

[0122] Step 216, based on the visibility attribute matrix, the visibility attribute of each anchor point in the three-dimensional anchor line information is corrected.

[0123] Specifically, as shown in Figure 6 and Figure 7 , the elements in the intermediate true value matrix are accumulated in the same way as step 212 to generate a second matrix.

[0124] Comparing the values of the elements at the same position in the first matrix and the second matrix, when the values of the elements at the same position in the first matrix and the second matrix are both greater than 0, the value of the element at the corresponding position in the initial true value matrix is set to "1", indicating that the visibility attribute of the anchor point corresponding to the position is "true", otherwise, it is set to "0" to generate a visibility attribute matrix, to correct the visibility attribute of the multiple 3D lane line candidate anchor lines predicted by the model in the current road image, to ensure the accuracy of 3D lane line detection.

[0125] In any of the above embodiments, further, the method further comprises:

[0126] Step 204: When it is determined that the mean of the anchor point distances of any 3D lane line candidate anchor line belongs to the first preset interval, a near-distance same-class suppression value between the 3D lane line candidate anchor line and the first candidate anchor line is calculated.

[0127] Step 205: It is determined whether the near-distance same-class suppression value belongs to the second preset interval, if yes, the 3D lane line candidate anchor line is retained, if not, the 3D lane line candidate anchor line is deleted.

[0128] Specifically, considering that lane lines with close distances but different classes, such as turning lines and bus turning sign lines, will be filtered out in the above steps 201-203, therefore, the embodiment further introduces a near-distance same-class suppression strategy, and a near-distance same-class suppression value is calculated based on a set near-distance same-class suppression formula, and the corresponding calculation formula can be:

[0129] dis_near = dis A_p *diff_flag

[0130] In the formula, dis_near is the near-distance same-class suppression value, dis A_p is the mean of the anchor point distances, and diff_flag is a same-class flag, which takes a value of 0 or 1, that is, when the first candidate anchor line and the second candidate anchor line are of the same class, the value of diff_flag is 0, and when they are of different classes, the value of diff_flag is 1. Therefore, when the calculated near-distance same-class suppression value dis_near belongs to the second preset interval (such as [0.2-0.5]), the second candidate anchor line is retained, otherwise, it is filtered out.

[0131] So far, the embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0132] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, and are not intended to limit the scope of the present application.

[0133] The steps in the present application can be adjusted in sequence, combined and reduced according to actual needs.

[0134] Although the present application is disclosed in detail through examples with reference to the accompanying drawings, it should be understood that the description is only exemplary, and is not intended to limit the application of the present application. The protection scope of the present application is defined by the appended claims, and can include various modifications, improvements and equivalent solutions made to the invention without departing from the protection scope and spirit of the present application.

Claims

1. A 3D lane detection model training method, characterized in that: The method comprises: Step 1: Construct a 3D lane line detection model, wherein the 3D lane line detection model includes at least a 3D lane line branch and a 2D lane line branch; Step 2: Constructing a network loss function based on the first perception result of the 3D lane line branch and the second perception result of the 2D lane line branch, and iteratively training the 3D lane line detection model based on the network loss function; Step 3: If it is determined that the preset early stopping condition is met, the iterative training of the 2D lane line branch is stopped, and the iterative training of the 3D lane line branch is continued until the preset termination condition is met, and the model parameters of the last iterative training of the 3D lane line branch are saved, where: The 3D lane line branch is used to output a 3D lane line candidate anchor line, and the 3D lane line candidate anchor line is used to determine the lane line; The determination of whether the preset early stopping conditions are met specifically includes: Step 31: During any iterative training process, the loss value of the network loss function is calculated. When it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current iteration number is labeled. Step 32: After iteratively training the 3D lane detection model for a first number of training times, iteratively training the 3D lane detection model for a second number of training times. During the second number of training times of iterative training, the trained 3D lane detection models are selected multiple times, and detection indicators of the selected 3D lane detection models on the validation data are sequentially calculated. Step 33: Determine whether any one of the plurality of detection indicators is greater than a reference indicator. If yes, based on the maximum value among the multiple detection indices, update the benchmark indicator, continue to iterate the 3D lane line detection model for the second number of training times, calculate the loss value of the network loss function and the detection indicator, If not, when it is determined that the number of labeling iterations does not belong to the second training number, it is determined that the preset early stopping condition is met. When it is determined that the number of labeling iterations falls within the second training number, continuing to iterate the 3D lane line detection model for a third training number of training times, and determining that the preset early stopping condition is satisfied; The network loss function includes at least a 3D loss function, a 2D loss function and a category divergence loss function. In step 2, iteratively training the 3D lane detection model based on the network loss function specifically includes: Step 21: Inputting sample data into the 3D lane detection model and outputting a first perception result and a second perception result, wherein the first perception result is obtained by the 3D lane branch prediction and the second perception result is obtained by the 2D lane branch prediction; Step 22: Substitute the first perception result and the first label data into the 3D loss function to calculate a first loss value, and substitute the second perception result and the second label data into the 2D loss function to calculate a second loss value; Step 23: Substitute the first perception result and the second perception result into the category divergence loss function to calculate the category divergence loss value; Step 24: Weight the first loss value, the second loss value, and the category divergence loss value to obtain a weighted loss value, and determine the model parameters of the 3D lane line branch and the 2D lane line branch in the next iterative training by minimizing the weighted loss value to iteratively train the 3D lane line detection model.

2. The 3D lane detection model training method according to claim 1, wherein: The 2D loss function Loss 2D The calculation formula is: Where, α 2d_cls is the 2D classification loss weight, m ​​is the label of the current anchor line, M is the total number of anchor lines in the second label data, L is the total number of lane line categories, l is the category number of the current anchor line, cls' is the probability value of anchor line m being predicted as category l in the second perception result (the value range is 0-1), cls is the true value of anchor line m marked as category l in the second label data (the value is 0 or 1), α 2d_coord is the 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u' mn ,v' mn ) is the image coordinate of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinate of the nth point on the corresponding mth anchor line in the second label data.

3. The 3D lane detection model training method according to claim 1, wherein: The detection index includes at least one of F1 score, precision, and recall rate.

4. The 3D lane detection model training method according to claim 1, wherein: The benchmark indicator is: the detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model performs iterative training for the first number of training times.

5. A 3D lane line detection method, characterized in that: The method comprises: Step 100: Based on the collected road image, use the trained 3D lane line detection model to output 3D lane line candidate anchor lines. The 3D lane line detection model is trained using the 3D lane line detection model training method according to any one of claims 1 to 4; Step 200: Screen the candidate 3D lane line anchor lines, and generate a 3D lane line detection result based on the screened candidate 3D lane line anchor lines using a curve fitting method.

6. The 3D lane line detection method according to claim 5, wherein: Screening the candidate 3D lane anchor lines includes: Step 201: Sort the 3D lane line candidate anchor lines based on their confidence scores, and record the 3D lane line candidate anchor line with the maximum confidence score as the first candidate anchor line. Step 202: sequentially obtain the 3D anchor line information of the sorted 3D lane line candidate anchor lines, and calculate the average anchor point distance between each candidate anchor line and the first candidate anchor line. Step 203: Screening the sorted 3D lane line candidate anchor lines based on the mean anchor point distance.

7. The 3D lane detection method according to claim 6, wherein: The three-dimensional anchor line information includes the visible attributes of each anchor point, and the method further includes: Step 211: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, generate an initial truth matrix in the order of the first anchor line, with each 3D lane line candidate anchor line as a column; Step 212: Accumulate the elements of each column of the initial truth value matrix from top to bottom to generate a first matrix; Step 213: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, generate an intermediate truth matrix in a second anchor line order, with each 3D lane line candidate anchor line as a column. The second anchor line order is bilaterally symmetric to the first anchor line order. Step 214: Accumulate the elements of each column in the intermediate truth matrix from top to bottom to generate a second matrix; Step 215: Compare the values ​​of the elements at the same position in the first matrix and the second matrix in sequence according to the element position to see if they are all greater than 0. If so, set the value of the element at the corresponding position in the initial truth value matrix to 1; otherwise, set it to 0, thereby generating a visible attribute matrix. Step 216: Modify the visible attribute of each anchor point in the three-dimensional anchor line information based on the visible attribute matrix.

8. The 3D lane line detection method according to claim 6, wherein: The method further comprises: Step 204: When it is determined that the mean anchor point distance of any 3D lane line candidate anchor line falls within a first preset interval, a close distance same-category suppression value between the 3D lane line candidate anchor line and the first candidate anchor line is calculated; Step 205: Determine whether the close-range same-category suppression value falls within a second preset interval. If so, retain the 3D lane line candidate anchor line; if not, delete the 3D lane line candidate anchor line.

Images