3D lane line detection model training method and 3D lane line detection method

Abstract

The invention discloses a 3D lane line detection model training method and a 3D lane line detection method.The training method comprises the steps that 1, a 3D lane line detection model is constructed, and the 3D lane line detection model at least comprises a 3D lane line branch and a 2D lane line branch; step 2, constructing a network loss function according to the first sensing result of the 3D lane line branch and the second sensing result of the 2D lane line branch, and performing iterative training on the 3D lane line detection model based on the network loss function; 3, if it is judged that the preset early stop condition is met, iteration training of the 2D lane line branch is stopped, iteration training of the 3D lane line branch continues until the preset stop condition is met, and model parameters of last iteration training of the 3D lane line branch are stored. According to the technical scheme of the invention, the training effect of the three-dimensional lane line detection model in a complex scene can be improved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and more specifically, to a 3D lane line detection model training method and a 3D lane line detection method. Background Art

[0002] With the rapid development of autonomous driving technology, higher requirements are placed on the accuracy and real-time performance of lane detection. Traditional 2D lane detection methods are usually based on the assumption of a planar road, ignoring the actual three-dimensional structure of the lane, and are difficult to cope with complex road environments, such as lanes blocked by vehicles or overlapping; lanes of different lengths, different curvatures, and complex backgrounds.

[0003] Therefore, 3D lane line detection technology based on 3D anchor lines has emerged. This 3D lane line detection method adopts the anchor line mechanism, uses the 3D anchor line as the predefined 3D lane line, and achieves more accurate and efficient detection through deep learning, improves the detection speed, and realizes accurate 3D positioning.

[0004] In the prior art, although the existing 3D lane line detection methods have achieved good results, there are still some limitations in 3D lane line detection in some complex scenes. For example:

[0005] 1. When the lane lines are blocked by vehicles or overlap;

[0006] 2. For lane lines of different lengths, different curvatures and complex backgrounds.

[0007] For such complex scenes, the existing three-dimensional lane line detection methods have low detection accuracy and are prone to false detections. Even if a sufficient amount of training data is added, the final detection effect is not ideal. Summary of the invention

[0008] The purpose of this application is to train a 3D lane detection model in complex scenarios and improve the accuracy of 3D lane detection.

[0009] The technical solution of the first aspect of the present application is: a 3D lane line detection model training method is provided, the method comprising: step 1: constructing a 3D lane line detection model, wherein the 3D lane line detection model includes at least a 3D lane line branch and a 2D lane line branch; step 2: constructing a network loss function according to a first perception result of the 3D lane line branch and a second perception result of the 2D lane line branch, respectively, and iteratively training the 3D lane line detection model based on the network loss function; step 3: if it is determined that a preset early stopping condition is met, then stopping the iterative training of the 2D lane line branch, continuing to iteratively train the 3D lane line branch until a preset termination condition is met, and saving the model parameters of the last iterative training of the 3D lane line branch, wherein the 3D lane line branch is used to output a 3D lane line candidate anchor line, and the 3D lane line candidate anchor line is used to determine the lane line.

[0010] In any of the above technical solutions, further, the network loss function at least includes a 3D loss function, a 2D loss function and a category divergence loss function. In step 2, the 3D lane line detection model is iteratively trained based on the network loss function, specifically including: Step 21: inputting sample data into the 3D lane line detection model, and outputting a first perception result and a second perception result, wherein the first perception result is obtained by the 3D lane line branch prediction, and the second perception result is obtained by the 2D lane line branch prediction; Step 22: substituting the first perception result and the first label data into the 3D loss function, calculating the first loss value, and substituting the second perception result and the second label data into the 2D loss function, calculating the second loss value; Step 23: bringing the first perception result and the second perception result into the category divergence loss function, and calculating the category divergence loss value; Step 24: weighting the first loss value, the second loss value, and the category divergence loss value to obtain a weighted loss value, and by minimizing the weighted loss value, respectively determining the model parameters in the 3D lane line branch and the 2D lane line branch in the next iterative training, so as to iteratively train the 3D lane line detection model.

[0011] In any of the above technical solutions, further, the 2D loss function Loss 2D The calculation formula is:

[0012]

[0013] In the formula, α 2d_cls is the 2D classification loss weight, m ​​is the label of the current anchor line, M is the total number of anchor lines in the second label data, L is the total number of lane line categories, l is the category number of the current anchor line, cls′ is the probability value of anchor line m being predicted as category 1 in the second perception result (the value range is 0-1), cls is the true value of anchor line m marked as category 1 in the second label data (the value is 0 or 1), α 2d_coordis the 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u′ mn ,v′ mn ) is the image coordinate of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinate of the nth point on the corresponding mth anchor line in the second label data.

[0014] In any of the above technical solutions, further, in step 3, it is determined that the preset early stopping condition is met, specifically including: step 31: in any iterative training process, the loss value of the network loss function is calculated, and when it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current number of iterations is labeled; step 32: after the 3D lane line detection model is iteratively trained for the first number of training times, the 3D lane line detection model is iteratively trained for the second number of training times, and in the iterative training process of the second number of training times, the trained 3D lane line detection model is selected multiple times, and the 3D lane line detection model is calculated in sequence. The detection indicators of the selected multiple 3D lane line detection models on the verification data; Step 33: Determine whether any one of the multiple detection indicators is greater than the benchmark indicator. If so, based on the maximum value of the multiple detection indicators, update the benchmark indicator, continue to iterate the 3D lane line detection model for the second training number of times, calculate the loss value of the network loss function and the detection indicator, if not, when it is determined that the number of labeling iterations does not belong to the second training number of times, it is determined that the preset early stopping condition is met. When it is determined that the number of labeling iterations belongs to the second training number of times, continue to iterate the 3D lane line detection model for the third training number of times, and determine that the preset early stopping condition is met.

[0015] In any of the above technical solutions, further, the detection index includes at least one of F1 score, precision, and recall rate.

[0016] In any of the above technical solutions, further, the benchmark indicator is: a detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model performs iterative training for the first number of training times.

[0017] The technical solution of the second aspect of the present application is: a 3D lane line detection method is provided, the method comprising: step 100: based on the collected road surface image, using the trained 3D lane line detection model, outputting 3D lane line candidate anchor lines, wherein the 3D lane line detection model is trained using the 3D lane line detection model training method of any one of the technical solutions of the first aspect; step 200: screening the 3D lane line candidate anchor lines, and using curve fitting to generate 3D lane line detection results based on the screened 3D lane line candidate anchor lines.

[0018] In any of the above technical solutions, further, the 3D lane line candidate anchor lines are screened, specifically including: step 201: based on the confidence scores of the 3D lane line candidate anchor lines, the 3D lane line candidate anchor lines are sorted, and the 3D lane line candidate anchor line corresponding to the maximum confidence score is recorded as the first candidate anchor line; step 202: the three-dimensional anchor line information of the sorted 3D lane line candidate anchor lines is obtained in sequence, and the mean anchor point distance between the candidate anchor lines and the first candidate anchor line is calculated respectively; step 203: based on the mean anchor point distance, the sorted 3D lane line candidate anchor lines are screened.

[0019] In any of the above technical solutions, further, the three-dimensional anchor line information includes the visible attributes of each anchor point, and the method further includes: step 211: based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, each 3D lane line candidate anchor line is a column, and an initial truth value matrix is ​​generated in the first anchor line order; step 212: from top to bottom, the elements of each column in the initial truth value matrix are accumulated in turn to generate a first matrix; step 213: based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, each 3D lane line candidate anchor line is a column, and an initial truth value matrix is ​​generated in the second anchor line order. , generate an intermediate truth matrix, wherein the second anchor line sequence is symmetrical with the first anchor line sequence; step 214: from top to bottom, accumulate the elements of each column in the intermediate truth matrix in turn to generate a second matrix; step 215: according to the element position, compare in turn whether the values ​​of the elements at the same position in the first matrix and the second matrix are all greater than 0, if so, set the element value at the corresponding position in the initial truth matrix to 1, otherwise, set it to 0, and generate a visible attribute matrix; step 216, based on the visible attribute matrix, correct the visible attributes of each anchor point in the three-dimensional anchor line information.

[0020] In any of the above technical solutions, further, the method also includes: Step 204: when it is determined that the mean anchor point distance of any 3D lane line candidate anchor line belongs to the first preset interval, calculating the close-distance same-category suppression value between the 3D lane line candidate anchor line and the first candidate anchor line; Step 205: determining whether the close-distance same-category suppression value belongs to the second preset interval, if so, retaining the 3D lane line candidate anchor line, if not, deleting the 3D lane line candidate anchor line.

[0021] The beneficial effects of this application are:

[0022] The technical solution in this application sets a 2D lane line branch in the 3D lane line detection model to classify each pixel of the lane line so that the model can better understand and process the boundary and shape information of the lane line during the training process. This enables the model to learn a more accurate target representation. The loss function is modified based on the 3D perception results and the 2D perception results, thereby improving the accuracy and robustness of 3D lane line detection.

[0023] In addition, this application considers that the 2D lane line branch is mainly trained for 2D images and 2D prediction results. In order to avoid excessive dependence of the training results on the 2D lane line branch, an early stopping condition is introduced. During the iterative training process, if the early stopping condition is met, the training of the 2D lane line branch is stopped, and the training of the 3D lane line branch continues until the preset termination condition is met. The model training is completed and only the model parameters of the 3D lane line branch are saved as the trained model. This strategy not only effectively prevents the model from being overly dependent on 2D information, but also promotes the overall training convergence of the model, further improving the practical value and performance of the model.

[0024] In addition, during the detection process, the present application also screened and filtered out candidate 3D lane line anchor lines based on the visible attributes of the anchor points in the lane lines, and corrected the visible attributes of the anchor points based on the symmetry of the lane lines to ensure the accuracy of the 3D lane line detection basis and further improve the accuracy of the model detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0026] Figure 1 is a schematic flow chart of a 3D lane detection model training method according to an embodiment of the present application;

[0027] Figure 2 is a schematic block diagram of data flow during a model training process according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of anchor line information in a 3D lane line candidate anchor line according to an embodiment of the present application;

[0029] Figure 4 is a partial schematic diagram of an initial truth value matrix according to an embodiment of the present application;

[0030] Figure 5 is a partial schematic diagram of a first matrix according to an embodiment of the present application;

[0031] Figure 6is a partial schematic diagram of a second matrix according to an embodiment of the present application;

[0032] Figure 7 is a partial schematic diagram of a visible attribute matrix according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0035] Embodiment 1:

[0036] like Figure 1 and Figure 2 As shown, this embodiment provides a 3D lane line detection model training method, the method comprising:

[0037] Step 1: Construct a 3D lane line detection model, wherein the 3D lane line detection model includes at least a 3D lane line branch and a 2D lane line branch; wherein the 3D lane line branch is used to output 3D lane line candidate anchor lines, the 3D lane line candidate anchor lines are used to determine lane lines, and the 2D lane line branch is used to assist the 3D lane line branch, so as to improve the accuracy and robustness of the overall lane line detection of the model by better understanding and processing the boundary and shape information of the lane lines.

[0038] Step 2: Construct a network loss function based on the first perception result of the 3D lane line branch and the second perception result of the 2D lane line branch, and iteratively train the 3D lane line detection model based on the network loss function;

[0039] Step 3: If it is determined that the preset early stopping condition is met, the iterative training of the 2D lane line branch is stopped, and the iterative training of the 3D lane line branch continues until the preset termination condition is met. The model parameters of the last iterative training of the 3D lane line branch are saved, and the training of the 3D lane line detection model is completed to output the predicted 3D lane line candidate anchor line.

[0040] Specifically, during the training process of the above-mentioned 3D lane line detection model, the sample data (RGB image and the first and second label data) is taken as input and input into the backbone network in the 3D lane line detection model to obtain the image features of the two-dimensional image encoding. This image feature is then passed into two branches, namely the 3D lane line branch and the 2D lane line branch.

[0041] The 3D lane line branch is the main branch of the model, generating the first perception result and outputting the 3D lane line candidate anchor line, including the predicted three-dimensional lane line information, such as the left and right offset coordinates x, height offset coordinates z, and visible attributes vis of the anchor point. The 2D lane line branch is the auxiliary training branch of the model, obtaining the second perception result during training, including the segmented 2D lane line information, to assist in training, supervision, and correction of the main branch, and improve the prediction effect of the model.

[0042] Afterwards, combined with the set network loss function, the first perception results and the second perception results in each iterative training process are used to calculate the corresponding loss value and perform iterative training.

[0043] However, considering that the 2D lane line branch is mainly trained for 2D images and 2D prediction results, and the present embodiment is to obtain 3D lane line candidate anchor lines, therefore, in order to avoid the training results from being overly dependent on the 2D lane line branch, an early stopping condition is also introduced. During the iterative training process, if the early stopping condition is met, the training of the 2D lane line branch is stopped, and the training of the 3D lane line branch continues until the preset termination condition is met, and only the model parameters of the 3D lane line branch are saved as the trained model. Among them, the early stopping condition can be set according to actual needs, such as setting a threshold for the number of iterations of iterative training (first threshold) and a convergence threshold of the loss function; the termination condition can be to set a threshold for the number of iterations of iterative training (second threshold, and greater than the first threshold).

[0044] When the training process meets the termination condition and stops, the final model parameters obtained in the training are saved as the 3D lane line detection model, and the optimized model is used to output the 3D lane line candidate anchor lines.

[0045] It should be noted that the 2D lane line branch only assists the 3D lane line branch in training during the training process and does not participate in the 3D lane line detection after the model training. Therefore, there is no need to save the model parameters of the 2D lane line branch.

[0046] In any of the above embodiments, further, the network loss function includes at least a 3D loss function, a 2D loss function and a category divergence loss function. In step 2, the 3D lane line detection model is iteratively trained based on the network loss function, specifically including:

[0047] Step 21: input the sample data into the 3D lane line detection model, and output a first perception result and a second perception result, wherein the first perception result is obtained by the 3D lane line branch prediction, and the second perception result is obtained by the 2D lane line branch prediction;

[0048] Step 22: Substitute the first perception result and the first label data into the 3D loss function to calculate the first loss value, and substitute the second perception result and the second label data into the 2D loss function to calculate the second loss value;

[0049] Step 23: Substitute the first perception result and the second perception result into the category divergence loss function to calculate the category divergence loss value;

[0050] Step 24: Weight the first loss value, the second loss value, and the category divergence loss value to obtain a weighted loss value, and determine the model parameters in the 3D lane line branch and the 2D lane line branch in the next iterative training by minimizing the weighted loss value, so as to iteratively train the 3D lane line detection model.

[0051] Specifically, the network loss function set in this embodiment is obtained by weighted summing the 3D loss function, the 2D loss function, and the category divergence loss function according to the set weights.

[0052] During the training process, Figure 2 As shown, the sample data (RGB image data and corresponding label data) can be used as input to the 3D lane detection model. The training process is roughly as follows:

[0053] 1. Randomly select a batch from the sample data as the data used in this iteration.

[0054] 2. The batch of sample data is forward propagated through the network, and based on the current model parameters, the prediction results, i.e., the first perception results and the second perception results, are obtained.

[0055] 3. Substitute the first perception result and the corresponding first label data into the 3D loss function to calculate the first loss value; at the same time, substitute the second perception result and the corresponding second label data into the 2D loss function to calculate the second loss value.

[0056] 4. Substitute the first perception result and the second perception result into the category divergence loss function for calculation to obtain the category divergence loss value between the first perception result and the second perception result.

[0057] 5. According to the set weights, the first loss value, the second loss value and the category divergence loss value are weighted and summed to obtain the loss value of the total network loss function, which is used as the basis for judging whether the termination condition is met.

[0058] 6. If the termination condition is reached, the iterative training is stopped and the model parameters of this iterative training are saved; if the termination condition is not reached, the back propagation algorithm is used to calculate the gradient of the loss of the above two branches with respect to the model parameters of this iterative training, providing a basis for updating the model parameters of the next iterative training.

[0059] 7. According to the above gradient, the model parameters of the two branches are optimized and adjusted, and a batch is randomly selected from the sample data again for the next iterative training.

[0060] In any of the above embodiments, further, the 2D loss function Loss 2D The calculation formula is:

[0061]

[0062] In the formula, α 2d_cls is the 2D classification loss weight, m ​​is the label of the current anchor line, M is the total number of anchor lines in the second label data, L is the total number of lane line categories, l is the category number of the current anchor line, cls′ ml is the probability value (ranging from 0 to 1) that the anchor line m is predicted to be of category l in the second perception result, cls ml is the true value (valued as 0 or 1) of the anchor line m marked as category 1 in the second label data, α 2d_coord is the 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u′ mn ,v′ mn ) is the image coordinate of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinate of the nth point on the mth anchor line corresponding to the second label data;

[0063] In this embodiment, the 3D loss function Loss 3D The calculation formula is:

[0064]

[0065] In the formula, α 3d_cls is the 3D classification loss weight, m ​​is the label of the current anchor line, M is the total number of anchor lines in the first label data (which is the same as the total number of anchor lines in the second label data), C is the total number of lane line categories, c is the category number of the current anchor line, and p mc is the probability value (ranging from 0 to 1) that the mth anchor line in the first perception result is predicted to be of category c, q mc Indicates the true value of the category of the mth anchor line in the first label data is c. If it is category c, then q mc =1, if it is not category c, then qmc =0.

[0066] α 3d_coord is the 3D point regression loss weight, M1 represents the total number of 3D anchor lines, N1 represents the number of points contained in each 3D anchor line, are respectively the x-coordinate value and the z-coordinate value of the n1-th point of the m1-th 3D anchor line in the first perception result, are the x-coordinate value and z-coordinate value of the n1-th point of the m1-th 3D anchor line in the first label data, respectively. is the visible attribute of the n1th point of the m1th 3D anchor line in the first perception result, It is the visible attribute of the n1th point of the m1th 3D anchor line in the first label data.

[0067] It should be noted that the 3D loss function Loss in this embodiment 3D It can be set according to actual needs.

[0068] The above weight parameters can be adjusted according to the actual training effect of the model, or they can be set to fixed values, such as α 2d_cls =0.15, α 2d_coord =0.15, α 3d_cls =0.35, α 3d_coord =0.35.

[0069] Taking into account the differences in the feature extraction methods of the 2D lane line branch and the 3D lane line branch and the perspective and dimension of the same object, in order to make the detection results of the 3D lane line branch and the 2D lane line branch more dispersed in the category space, thereby improving the distinction between different categories, and promoting the model to consider two-dimensional and three-dimensional information at the same time during the optimization process, to achieve information complementarity and improve the accuracy and robustness of the detection results. This embodiment also constructs a category divergence loss to correct the total loss function. Therefore, the total loss function of this embodiment is:

[0070] Loss(line)=Loss 2D +Loss 3D +Loss div

[0071]

[0072] In the formula, Loss div is the category divergence loss function, m is the label of the current anchor line, M is the total number of anchor lines in the first label data (which is the same as the total number of anchor lines in the second label data), C is the total number of lane line categories, and c is the category number of the current anchor line. is the probability value of predicting the mth anchor line category as c in the first perception result, is the probability value of predicting the mth anchor line category as c in the second perception result.

[0073] It should be noted that when the preset early stopping condition is met, the above-mentioned category divergence loss will no longer participate in the training of the model. That is, after the early stopping condition is met, the total loss function is Loss 3D .

[0074] In any of the above embodiments, further, in order to improve the training effect of the 2D lane line branch and the overall 3D lane line detection model, this embodiment also sets an early stopping condition, which includes both a convergence threshold and an iteration number threshold, and adjusts the number of iterative training in real time. Therefore, in step 3, determining whether the preset early stopping condition is met specifically includes:

[0075] Step 31: During any iterative training process, the loss value of the network loss function is calculated. When it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current number of iterations is labeled;

[0076] Step 32: After performing iterative training on the 3D lane line detection model for a first number of training times, the 3D lane line detection model is then iteratively trained for a second number of training times, and during the iterative training process for the second number of training times, the trained 3D lane line detection model is selected multiple times, and detection indicators of the selected multiple 3D lane line detection models on the verification data are calculated in sequence;

[0077] Step 33: Determine whether any one of the multiple detection indicators is greater than the benchmark indicator.

[0078] If yes, based on the maximum value among multiple detection indicators, update the benchmark indicator, continue to iterate the 3D lane line detection model for the second training number of times, calculate the loss value of the network loss function and the detection indicator,

[0079] If not, when it is determined that the number of labeling iterations does not belong to the second training number, it is determined that the preset early stopping condition is met.

[0080] When it is determined that the number of iterations for labeling belongs to the second training number, the 3D lane line detection model is continuously iterated for the third training number, and it is determined that the preset early stopping condition is met.

[0081] Specifically, during the model training process, the first training times can be set to 300, and the initial convergence threshold can be set to 0.6. After that, during the 300 iterative training times, the loss value of the above network loss function is calculated each time, and the loss value is compared with the initial convergence threshold of 0.6. If the loss value corresponding to a certain iterative training is 0.58, the convergence threshold is updated to 0.58, and the number of iterative training times is labeled and marked.

[0082] When the iterative training reaches 300 times, the iterative training of the second training times is performed, and the second training times is less than the first training times, for example, it is set to 50 times.

[0083] During the 50 iterations of training, the 3D lane line detection model after multiple iterations of training can be selected randomly or at equal intervals, and the verification data can be input into the trained model to calculate the corresponding detection index, where the detection index can be set according to actual needs, such as the F1 score.

[0084] This embodiment is now described by taking the calculation of the F1 score of the detection index 5 times as an example.

[0085] After obtaining 5 F1 scores, compare these 5 F1 scores with the benchmark index to determine whether there is a score greater than the benchmark index. For example, if the benchmark index is set to 80, if there is an F1 score of 81 among the 5 F1 scores, then update the benchmark index to 81, and train the above model again for 50 iterations, repeat the above process, and obtain 5 F1 scores again.

[0086] If the five F1 scores obtained again are all less than or equal to 81, 400 iterative trainings have been performed. Then, it is determined whether the number of trainings for the most recent update of the convergence threshold during the 400 iterative trainings, that is, the number of labeling iterations, is between the 351st and 400th iterative trainings. If the number of trainings for updating the convergence threshold is the 200th, it is determined that the preset early stopping condition is met, and the training of the 2D lane line branch is stopped. Only the 3D lane line branch is iteratively trained until the termination condition is met.

[0087] If the convergence threshold is updated for the 360th training run, then the overall 3D lane line detection model is iteratively trained for the third number of training runs (e.g., 30 or 50 times), and it is determined that the preset early stopping condition is met, and the training of the 2D lane line branch is stopped. During the third number of iterative training runs, there is no need to calculate the F1 score.

[0088] It should be noted that a termination training number, such as 500, can also be set in the early stopping condition. When the iterative training reaches 500 times, even if the above-set conditions are not met, the iterative training is directly determined to meet the early stopping condition, and the training of the 2D lane line branch is stopped.

[0089] In any of the above embodiments, further, the detection index includes at least one of F1 score, precision, and recall rate.

[0090] In any of the above embodiments, further, the benchmark indicator is: a detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model performs iterative training for the first number of training times.

[0091] Embodiment 2:

[0092] This embodiment provides a 3D lane line detection method, the method comprising:

[0093] Step 100: Based on the collected road surface image, using the trained 3D lane line detection model, outputting a 3D lane line candidate anchor line, wherein the 3D lane line detection model is trained using the 3D lane line detection model training method as described in any one of Embodiment 1;

[0094] Step 200: Screening the 3D lane line candidate anchor lines, and using curve fitting to generate 3D lane line detection results based on the screened 3D lane line candidate anchor lines.

[0095] Specifically, Figure 3 As shown in the figure, there are N 3D lane line candidate anchor lines output by the model. Each 3D lane line candidate anchor line contains information such as the confidence score (of the entire anchor line), the left and right offset coordinates x of the anchor point, the height offset coordinate z, the visible attribute vis of the anchor point, and the category probability value cls. The length of each 3D lane line candidate anchor line is set to 100 meters, and a point (i.e., anchor point) is sampled every 5 meters to form a sequence of 20 anchor points.

[0096] Each anchor point has three attributes: the left and right offset coordinates x predicted by the model, the height offset coordinate z, and the visible attribute vis. When the visible attribute vis is "true", it means that the anchor point is on the lane line, and when it is "false", it means that it is not.

[0097] Therefore, after obtaining the 3D lane line candidate anchor lines output by the model, the set rules can be used to filter out the 3D lane line candidate anchor lines, and then the retained and more accurate 3D lane line candidate anchor lines can be used to generate the final 3D lane line detection results, where the rules can be set according to actual needs.

[0098] For example, based on the confidence score of each 3D lane line candidate anchor line, the 3D lane line candidate anchor lines used for curve fitting are screened, the 3D lane line candidate anchor lines with confidence scores lower than a certain threshold are filtered out, and the 3D lane line candidate anchor lines with higher confidence scores are retained.

[0099] For example, the number of anchor points whose visible attribute vis is continuously "true" in each 3D lane line candidate anchor line is counted. If the number of anchor points that are continuously "true" is greater than a certain threshold, the anchor line is considered to be "long" enough and is retained. Otherwise, it is filtered out.

[0100] The curvature offset of the corresponding 3D lane line candidate anchor line can also be calculated based on the left and right offset coordinates x of each anchor point. By judging whether the difference coverage area of ​​the lateral left and right offset coordinates x of each 3D lane line candidate anchor line exceeds a set threshold, the falsely detected lane lines with larger curvature can be eliminated, and the 3D lane line candidate anchor lines with smaller coverage areas can be retained.

[0101] In the process of fitting the lane lines in this embodiment, the curve fitting method adopted may be a cubic parametric spline curve fitting algorithm or other curve fitting algorithms, and the specific fitting process will not be described in detail.

[0102] In any of the above embodiments, further, screening the candidate anchor lines of the 3D lane line includes:

[0103] Step 201: sorting the 3D lane line candidate anchor lines based on their confidence scores, and recording the 3D lane line candidate anchor line corresponding to the maximum confidence score as the first candidate anchor line;

[0104] Step 202: sequentially obtain the 3D anchor line information of each anchor point in the sorted 3D lane line candidate anchor lines, and respectively correspond the anchor points in the sequentially selected and sorted 3D lane line candidate anchor lines to the anchor points in the first candidate anchor line, and calculate the average anchor point distance between the two anchor lines;

[0105] Step 203: Screening the sorted 3D lane line candidate anchor lines based on the mean anchor point distance.

[0106] Specifically, first, multiple 3D lane line candidate anchor lines predicted by the model are sorted from high to low according to the confidence scores, and the 3D lane line candidate anchor line with the highest score is saved in the list keep, and the 3D lane line candidate anchor line is recorded as the first candidate anchor line.

[0107] The three-dimensional anchor line information of the first candidate anchor line is extracted, including at least the left and right offset coordinates x, the height offset coordinate z, and the visible attribute vis of the anchor point, and the corresponding category probability value cls is obtained by using functions such as argmax.

[0108] Select any of the sorted 3D lane line candidate anchor lines in turn, divide it by the 3D lane line candidate anchor line saved in the keep list, and use it as the second candidate anchor line. Repeat the above process to obtain the 3D anchor line information of the second candidate anchor line and calculate the mean anchor point distance. The corresponding calculation formula is:

[0109]

[0110] In the formula, dis A_p is the mean anchor point distance, I represents the total number of anchor points on each 3D lane line candidate anchor line (set to 20 in this embodiment), i represents the number of the current anchor point, (X i ,Z i )、(x i ,z i are the left and right offset coordinates x and height offset coordinates z of the i-th anchor point in the first candidate anchor line and the second candidate anchor line, respectively. Ai 、vis pi are the visible attributes of the i-th anchor point in the first candidate anchor line and the second candidate anchor line respectively.

[0111] When the calculated anchor point distance mean is greater than or equal to the set threshold (such as 2, 2.5, etc.), the second candidate anchor line is retained and saved in the list keep, otherwise, the second candidate anchor line is deleted. After that, the next 3D lane line candidate anchor line is reselected until all 3D lane line candidate anchor lines are screened. Finally, curve fitting is performed on the 3D lane line candidate anchor lines saved in the list keep to generate the 3D lane line detection result.

[0112] In any of the above embodiments, further, the three-dimensional anchor line information includes a visible attribute of each anchor point, and the method further includes:

[0113] Step 211: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, an initial truth matrix is ​​generated in the order of the first anchor line, with each 3D lane line candidate anchor line as a column;

[0114] Specifically, Figure 4 As shown, the trained model will predict multiple 3D lane line candidate anchor lines in the current road surface image. Each 3D lane line candidate anchor line will contain multiple (e.g., 20) anchor points. Each anchor point has a visible attribute. For example, "anchor point visible" can be set to "true" and the corresponding value is set to "1"; otherwise, it is set to "false" and the corresponding value is "0".

[0115] The order of the road surface image from left to right is set as the first anchor line order, and the visible attributes of each anchor point in each 3D lane line candidate anchor line are counted in turn to generate an initial truth matrix.

[0116] Step 212: From top to bottom, sequentially add the elements of each column in the initial truth value matrix to generate a first matrix;

[0117] Specifically, Figure 5 As shown in FIG. 1 , considering that the bottom of the road surface image is close to the vehicle, the reliability of the anchor points in the predicted 3D lane line candidate anchor lines is high. Therefore, each 3D lane line candidate anchor line is cumulatively calculated in order from top to bottom, and the accumulated values ​​of each element in the corresponding column in the initial truth matrix are saved to generate the first matrix.

[0118] Step 213: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, an intermediate truth value matrix is ​​generated according to the second anchor line sequence, with each 3D lane line candidate anchor line as a column, wherein the second anchor line sequence is bilaterally symmetric to the first anchor line sequence;

[0119] Specifically, the order of the road surface image from right to left is set as the second anchor line order, and the process of step 211 is repeated, or the initial truth value matrix is ​​directly flipped left to right to generate an intermediate truth value matrix. The specific process is not repeated here.

[0120] Step 214: From top to bottom, accumulate the elements of each column in the intermediate truth value matrix in turn to generate a second matrix;

[0121] Step 215: Compare the values ​​of the elements at the same position in the first matrix and the second matrix in turn according to the element position to see if they are all greater than 0. If so, set the value of the element at the corresponding position in the initial truth value matrix to 1, otherwise, set it to 0, and generate a visible attribute matrix;

[0122] Step 216: based on the visible attribute matrix, modify the visible attribute of each anchor point in the three-dimensional anchor line information.

[0123] Specifically, Figure 6 and Figure 7 As shown, in the same manner as step 212, the elements in the intermediate truth value matrix are accumulated to generate a second matrix.

[0124] Compare the values ​​of the elements at the same position in the first matrix and the second matrix. When the values ​​at the same position in the first matrix and the second matrix are both greater than 0, the value of the element at the corresponding position in the initial truth matrix is ​​set to "1", indicating that the visible attribute of the anchor point corresponding to the position is "true". Otherwise, its value is set to "0" to generate a visible attribute matrix to correct the visible attributes of multiple 3D lane line candidate anchor lines predicted by the model in the current road surface image to ensure the accuracy of 3D lane line detection.

[0125] In any of the above embodiments, the method further includes:

[0126] Step 204: when it is determined that the mean anchor point distance of any 3D lane line candidate anchor line belongs to the first preset interval, calculating the short-distance same-category suppression value between the 3D lane line candidate anchor line and the first candidate anchor line;

[0127] Step 205: Determine whether the short-distance same-category suppression value belongs to the second preset interval. If so, retain the 3D lane line candidate anchor line; if not, delete the 3D lane line candidate anchor line.

[0128] Specifically, considering that lane lines with close distances but different categories, such as turning lines and bus transfer marking lines, are filtered out in the above steps 201-203, this embodiment also introduces a close-distance same-category suppression strategy. Based on the set close-distance same-category suppression formula, the close-distance same-category suppression value is calculated. The corresponding calculation formula can be:

[0129] dis_near=dis A_p *diff_flag

[0130] Where dis_near is the suppression value of the same category at a short distance, dis A_p is the mean of the anchor point distances, and diff_flag is the flag of the same category, which takes a value of 0 or 1, that is, when the first candidate anchor line and the second candidate anchor line are of the same category, the value is 0, and when they are of different categories, the value is 1. Therefore, when the calculated near-distance same-category suppression value dis_near belongs to the second preset interval (such as [0.2-0.5]), the second candidate anchor line is retained, otherwise, it is filtered out.

[0131] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0132] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0133] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0134] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A 3D lane detection model training method, characterized in that: The method comprises: Step 1: construct a 3D lane line detection model, wherein the 3D lane line detection model includes at least a 3D lane line branch and a 2D lane line branch; Step 2: constructing a network loss function according to the first perception result of the 3D lane line branch and the second perception result of the 2D lane line branch, and iteratively training the 3D lane line detection model based on the network loss function; Step 3: If it is determined that the preset early stopping condition is met, the iterative training of the 2D lane line branch is stopped, and the iterative training of the 3D lane line branch is continued until the preset termination condition is met, and the model parameters of the last iterative training of the 3D lane line branch are saved, wherein: The 3D lane line branch is used to output a 3D lane line candidate anchor line, and the 3D lane line candidate anchor line is used to determine the lane line.

2. The 3D lane detection model training method according to claim 1, characterized in that: The network loss function at least includes a 3D loss function, a 2D loss function and a category divergence loss function. In the step 2, iterative training of the 3D lane line detection model is performed based on the network loss function, specifically including: Step 21: Input sample data into the 3D lane line detection model, and output a first perception result and a second perception result, wherein the first perception result is obtained by the 3D lane line branch prediction, and the second perception result is obtained by the 2D lane line branch prediction; Step 22: Substitute the first perception result and the first label data into the 3D loss function to calculate a first loss value, and substitute the second perception result and the second label data into the 2D loss function to calculate a second loss value; Step 23: Substitute the first perception result and the second perception result into the category divergence loss function to calculate the category divergence loss value; Step 24: Weight the first loss value, the second loss value, and the category divergence loss value to obtain a weighted loss value, and determine the model parameters in the 3D lane line branch and the 2D lane line branch in the next iterative training by minimizing the weighted loss value to iteratively train the 3D lane line detection model.

3. The 3D lane detection model training method according to claim 2, characterized in that: The 2D loss function Loss 2D The calculation formula is: In the formula, α 2d_cls is the 2D classification loss weight, m ​​is the number of the current anchor line, M is the total number of anchor lines in the second label data, L is the total number of lane line categories, l is the category number of the current anchor line, cls′ is the probability value of anchor line m being predicted as category l in the second perception result (the value range is 0-1), cls is the true value of anchor line m marked as category l in the second label data (the value is 0 or 1), α 2d_coord is the 2D point regression loss weight, N is the number of points in each anchor line, n is the label of the current point in each anchor line, (u ′ mn ,v ′ mn ) is the image coordinate of the nth point on the mth anchor line in the second perception result, (u mn ,v mn ) is the image coordinate of the nth point on the corresponding mth anchor line in the second label data.

4. The 3D lane detection model training method according to any one of claims 1 to 3, characterized in that: In step 3, determining whether the preset early stopping condition is met specifically includes: Step 31: During any iterative training process, the loss value of the network loss function is calculated. When it is determined that the loss value of the network loss function is less than the convergence threshold, the convergence threshold is updated according to the loss value of the network loss function, and the current number of iterations is labeled; Step 32: after performing iterative training on the 3D lane line detection model for a first number of training times, performing iterative training on the 3D lane line detection model for a second number of training times, and during the iterative training process for the second number of training times, selecting the trained 3D lane line detection model for multiple times, and calculating detection indicators of the selected multiple 3D lane line detection models on the verification data in sequence; Step 33: Determine whether any one of the plurality of detection indicators is greater than a reference indicator. If yes, based on the maximum value of the plurality of detection indices, update the benchmark indicator, continue to iterate the 3D lane line detection model for the second number of training times, calculate the loss value of the network loss function and the detection indicator, If not, when it is determined that the number of labeling iterations does not belong to the second training number, it is determined that the preset early stopping condition is met. When it is determined that the number of labeling iterations belongs to the second training number, the 3D lane line detection model is continuously iterated for a third training number of times, and it is determined that the preset early stopping condition is satisfied.

5. The 3D lane detection model training method according to claim 4, characterized in that: The detection index includes at least one of F1 score, precision and recall rate.

6. The 3D lane detection model training method according to claim 4, characterized in that: The benchmark indicator is: the detection indicator of the 3D lane line detection model on the verification data after the 3D lane line detection model performs iterative training for the first number of training times.

7. A 3D lane line detection method, characterized in that: The method comprises: Step 100: Based on the collected road surface image, the trained 3D lane line detection model is used to output the 3D lane line candidate anchor line. The 3D lane line detection model is trained using the 3D lane line detection model training method according to any one of claims 1 to 6; Step 200: Screening the 3D lane line candidate anchor lines, and using a curve fitting method to generate a 3D lane line detection result based on the screened 3D lane line candidate anchor lines.

8. The 3D lane line detection method according to claim 7, characterized in that: The candidate anchor lines of the 3D lane lines are screened, specifically including: Step 201: sorting the 3D lane line candidate anchor lines based on the confidence scores of the 3D lane line candidate anchor lines, and recording the 3D lane line candidate anchor line corresponding to the maximum confidence score as the first candidate anchor line; Step 202: sequentially obtain the three-dimensional anchor line information of the sorted 3D lane line candidate anchor lines, and calculate the average anchor point distance with the first candidate anchor lines respectively; Step 203: Screening the sorted 3D lane line candidate anchor lines based on the anchor point distance mean.

9. The 3D lane line detection method according to claim 8, characterized in that: The three-dimensional anchor line information includes the visible attributes of each anchor point, and the method further includes: Step 211: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, an initial truth matrix is ​​generated in the order of the first anchor line, with each 3D lane line candidate anchor line as a column; Step 212: From top to bottom, sequentially add the elements of each column in the initial truth value matrix to generate a first matrix; Step 213: Based on the visible attributes of each anchor point in the 3D lane line candidate anchor line, an intermediate truth value matrix is ​​generated according to a second anchor line sequence with each 3D lane line candidate anchor line as a column, wherein the second anchor line sequence is bilaterally symmetric to the first anchor line sequence; Step 214: From top to bottom, sequentially add the elements of each column in the intermediate truth value matrix to generate a second matrix; Step 215: Compare the values ​​of the elements at the same position in the first matrix and the second matrix in turn according to the element position to see if they are all greater than 0. If so, set the value of the element at the corresponding position in the initial truth value matrix to 1, otherwise, set it to 0, and generate a visible attribute matrix; Step 216: based on the visible attribute matrix, modify the visible attribute of each anchor point in the three-dimensional anchor line information.

10. The 3D lane line detection method according to claim 8, characterized in that: The method further comprises: Step 204: when it is determined that the mean anchor point distance of any 3D lane line candidate anchor line belongs to the first preset interval, calculating the short-distance same-category suppression value between the 3D lane line candidate anchor line and the first candidate anchor line; Step 205: Determine whether the short-distance same-category suppression value belongs to a second preset interval. If so, retain the 3D lane line candidate anchor line; if not, delete the 3D lane line candidate anchor line.

Images