Lane detection method, device, electronic device, storage medium and vehicle

Abstract

The present application relates to a lane line detection method, device, electronic device, storage medium and vehicle. The lane line detection method comprises: acquiring or generating multiple anchor points in a road image to be tested that are identified as belonging to the same identified lane line; forming at least one anchor point line according to the distance between the multiple anchor points of the same identified lane line; generating a fitting lane line based on the anchor point line according to the distance between the anchor points other than the anchor point line and the anchor point line. The present application is beneficial in that it provides a lane line detection method, device, electronic device, storage medium and vehicle that take into account both recognition efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a lane line detection method, device, electronic equipment, storage medium and vehicle. Background Art

[0002] With the development of vehicle intelligence technology, lane detection has become an important part of technical fields such as assisted driving and autonomous driving. For example, in assisted driving, lane keeping and lane departure warning based on lane detection can effectively share the driver's work and reduce the driver's workload, thereby effectively reducing the occurrence of traffic accidents.

[0003] In related technologies, a neural network model is used to process images collected by vehicles to obtain lane line data in the images, thereby providing "virtual lane lines" required for computer applications; however, current technical solutions have the problem of not being able to balance recognition efficiency and accuracy. Summary of the invention

[0004] The embodiments of the present application provide a lane detection method, device, electronic device, storage medium and vehicle to at least partially solve the above-mentioned technical problems.

[0005] To achieve the aforementioned purpose, according to the first aspect of the present application, a lane line detection method is provided, which includes: acquiring or generating multiple anchor points identified as belonging to the same identified lane line in a road image to be tested; forming at least one anchor point line according to the distance between the multiple anchor points of the same identified lane line; and generating a fitting lane line based on the anchor point line according to the distance between the anchor points other than the anchor point line and the anchor point line.

[0006] Optionally, in some embodiments of the present application, wherein at least one anchor point line is formed based on the distance between multiple anchor points of the same identified lane line, including: when the distance between two anchor points is less than a preset line distance threshold, the line between the two anchor points is taken as a part of the anchor point line.

[0007] Optionally, in some embodiments of the present application, generating a fitted lane line based on the anchor point line according to the distance between the anchor point other than the anchor point line and the anchor point line includes: when the distance between the anchor point other than the anchor point line and the anchor point line is less than a preset fitting distance threshold, fitting the anchor point other than the anchor point line to the anchor point line.

[0008] Optionally, in some embodiments of the present application, generating a fitted lane line based on the anchor point line according to the distance between the anchor points other than the anchor point line and the anchor point line also includes: when there is no anchor point of the same identified lane line other than the anchor point line, using the anchor point line connecting the most anchor points as the fitted lane line.

[0009] Optionally, in some embodiments of the present application, generating a fitted lane line based on the anchor point line according to the distance between the anchor point other than the anchor point line and the anchor point line also includes: when the distance between two anchor points other than the anchor point line is less than a preset anchor point distance threshold, deleting the two anchor points.

[0010] Optionally, in some embodiments of the present application, when the distance between the anchor points other than the anchor point line and the anchor point line is greater than or equal to a preset fitting distance threshold, the anchor points other than the anchor point line are fitted to other anchor point lines.

[0011] Optionally, in some embodiments of the present application, the acquiring or generating of multiple anchor points identified as belonging to the same identified lane line in a road image to be tested includes: inputting the road image to be tested into a lane line detection model so that the lane line detection model outputs the anchor point and its anchor point detection data and identification detection data; wherein the anchor point detection data is used to indicate the position of the anchor point; and the identification detection data is used to indicate the identified lane line to which the anchor point belongs.

[0012] Optionally, in some embodiments of the present application, the acquiring or generating of multiple anchor points identified as belonging to the same identified lane line in a road image to be tested includes: inputting the road image to be tested into the lane line detection model so that the lane line detection model outputs category detection data; wherein the category detection data is used to indicate the lane line type of the identified lane line.

[0013] Optionally, in some embodiments of the present application, the lane line detection method further includes: training or providing the lane line detection model; wherein the training or providing the lane line detection model includes:

[0014] Each historical road image in the training data set is processed by the lane line detection model to obtain lane line detection data of each historical road image; the detection loss value of the lane line detection model is determined according to the lane line detection data of multiple historical road images and the lane line annotation data of multiple historical road images in the training data set; the parameters of the lane line detection model are adjusted to make the detection loss value converge, so as to obtain the lane line detection model that has completed training.

[0015] Optionally, in some embodiments of the present application, the training or providing of the lane line detection model also includes: acquiring at least one initial road image and initial annotation data of each initial road image; performing data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain multiple historical road images and the lane line annotation data of each historical road image; and constructing the training data set based on the multiple historical road images and the lane line annotation data of each historical road image.

[0016] Optionally, in some embodiments of the present application, the data enhancement processing is performed on the initial road image and the initial annotation data of the initial road image to obtain the multiple historical road images and the lane line annotation data of each historical road image, including: rotating the initial road image according to a preset angle to obtain the historical road image; performing coordinate rotation processing on the initial point annotation data in the initial annotation data of the initial road image according to the rotation matrix corresponding to the preset angle; generating anchor point annotation data of the historical road image based on the processed initial point annotation data; generating the lane line annotation data of the historical road image based on the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0017] Optionally, in some embodiments of the present application, the data enhancement processing is performed on the initial road image and the initial annotation data of the initial road image to obtain the multiple historical road images and the lane line annotation data of each historical road image, including: cropping the initial road image according to a preset sliding window to obtain the historical road image; performing coordinate conversion processing on the initial point annotation data in the initial annotation data of the initial road image according to the sliding window coordinate system corresponding to the preset sliding window; generating anchor point annotation data of the historical road image based on the processed initial point annotation data; generating the lane line annotation data of the historical road image based on the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0018] Optionally, in some embodiments of the present application, generating the anchor point annotation data of the historical road image based on the processed initial point annotation data includes: mapping each lane line annotation point indicated by the processed initial point annotation data to a road grid of the historical road image; for each of the road grids, when at least one lane line annotation point is mapped to the road grid, generating the anchor point in the road grid based on at least one lane line annotation point mapped to the road grid; generating the anchor point annotation data of the historical road image based on the anchor point in at least one road grid of the historical road image.

[0019] According to a second aspect of the present application, a lane line detection device is provided, which includes: a memory on which a computer program is stored; and a processor for executing the computer program in the memory to implement the aforementioned lane line detection method.

[0020] According to a third aspect of the present application, an electronic device is provided, which is the aforementioned lane line detection device.

[0021] According to a fourth aspect of the present application, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the aforementioned lane line detection method is implemented.

[0022] According to a fifth aspect of the present application, the vehicle includes the aforementioned electronic device, or includes the aforementioned non-temporary computer-readable storage medium.

[0023] The benefit of the present application lies in providing a lane line detection method, device, electronic device, storage medium and vehicle that take into account both recognition efficiency and accuracy.

[0024] More specifically, some embodiments of the present application may produce the following specific beneficial effects:

[0025] The method of first identifying and then correcting is adopted to improve the recognition efficiency while ensuring the accuracy of the recognition result; the lane line detection method provided in the embodiment of the present application processes the anchor points detected by the lane line detection model through the region growing algorithm to filter out the wrongly detected anchor points, and then fits the lane line through the retained anchor points. Since the anchor points used in lane line fitting have been filtered by the region growing algorithm, the retained anchor points belong to the lane line and the probability that the lane line identification is accurate is higher, so the fitted lane line is more accurate, which effectively improves the accuracy of lane line detection.

[0026] The road grid method is adopted to further improve the efficiency of lane line recognition using a neural network model. The lane line detection method provided in the embodiment of the present application performs lane line detection on the road image to be tested through the lane line detection model to output lane line detection data, and the lane line detection data includes anchor point detection data, identification detection data and category detection data. According to the lane line detection data, at least one lane line in the road image to be tested, as well as the lane line identification and lane line type of each lane line can be determined. The embodiment of the present application performs lane line detection based on the lane line detection model, and the detection process is simple and the detection speed is fast, which can ensure the accuracy and efficiency of lane line detection for various road scenes. In addition, in the embodiment of the present application, lane line detection is implemented based on anchor point detection, and dense anchor points help to fit the parameters of lane lines with a large degree of curvature, thereby improving the accuracy of lane line detection with large curvature. In addition, when performing lane line detection, the lane line detection model of the embodiment of the present application divides the road image to be tested into multiple road grids, and detects at most one anchor point for each road grid, which simplifies the detection content of the lane line detection model and reduces the calculation amount of the lane line detection model, so as to facilitate the deployment of the lane line detection model on the edge computing device.

[0027] The lane line detection method provided in the embodiment of the present application performs data enhancement processing on the initial road image and its lane line annotation data when constructing the training data set of the lane line detection model. Among them, the data enhancement processing may include rotation enhancement processing and / or cropping enhancement processing. The rotation enhancement processing can enrich the angle between the lane line and the road image, and the cropping enhancement processing can enrich the different positions of the lane line in the road image. Therefore, the embodiment of the present application can effectively expand the number of samples in the training data set through data enhancement processing, improve the diversity of training samples in the training data set, and effectively improve the robustness and generalization ability of the lane line detection model.

[0028] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0030] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0031] Figure 1It is a schematic diagram of the main steps of a lane line detection method provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of an anchor point detection result provided in an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of anchor point filtering provided in an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of a road image processing provided by an embodiment of the present application;

[0035] Figure 5 is a schematic diagram of a road grid provided in an embodiment of the present application;

[0036] Figure 6 is a schematic diagram of a lane detection model provided in an embodiment of the present application;

[0037] Figure 7 is a block diagram of a lane detection device provided in an embodiment of the present application;

[0038] Figure 8 is a block diagram of another lane detection device provided in an embodiment of the present application;

[0039] Fig. 9 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0041] With the development of vehicle intelligence technology, lane detection has become an important part of technical fields such as assisted driving and autonomous driving. For example, in assisted driving, lane keeping and lane departure warning based on lane detection can effectively share the driver's work and reduce the driver's workload, thereby effectively reducing the occurrence of traffic accidents. In addition, efficient and stable lane detection is also a basic requirement for the implementation of autonomous driving products and safe and stable operation.

[0042] The current methods for lane line detection mainly include the following.

[0043] Semantic segmentation-based method: In this method, the road image is first input into the deep learning semantic segmentation model to obtain a binary image, and then the expression of each lane line in the road image is obtained through clustering and polynomial fitting.

[0044] Row classification-based method: In this method, the road image is first input into the deep learning network to extract features to obtain a feature map. A channel of the feature map represents a lane line, and then the horizontal coordinate of the lane line on each row is extracted from the channel.

[0045] Anchor point based method: In this method, anchor points are set at different positions on the feature map so that lane lines can be identified through the anchor points.

[0046] Function fitting based method: In this method, a specific function is used to represent the lane lines, and then the various parameters in the function are learned through deep learning methods.

[0047] However, the aforementioned method based on semantic segmentation classifies each pixel point of the road image, and the data annotation for training the segmentation model is difficult. The data post-processing process is complex and time-consuming. The aforementioned method based on row classification has poor point positioning error on the lane line and is not suitable for road scenes with a large angle between the lane line and the direction of movement. The aforementioned method based on anchor points is computationally complex, and it is difficult to ensure the accuracy and efficiency of lane line detection, especially in complex road scenes. The aforementioned method based on function fitting is difficult to train, and it is difficult to ensure the accuracy and stability of lane line detection, especially in complex road scenes.

[0048] In view of this, the embodiments of the present application provide a lane line detection method, device, electronic device, storage medium and vehicle, which can achieve high-precision and high-efficiency detection of lane lines, with a simple data post-processing process and low computational complexity, and are easy to deploy on edge computing devices; moreover, they can also handle road scenes where the angle between the lane line and the direction of vehicle movement is large; in addition, dense anchor point output also helps to fit lane lines with larger curvatures, thereby achieving large curvature lane line detection.

[0049] According to the first aspect of the present application, an embodiment of the present application provides a lane line detection method.

[0050] Reference Figure 1 As shown, Figure 1 Schematic diagram of the main steps of a lane line detection method provided in an embodiment of the present application. The lane line detection method may include the following steps:

[0051] S100: Acquire or generate a plurality of anchor points in a road image to be tested that are identified as belonging to the same identified lane line.

[0052] S200: Form at least one anchor point line according to the distances between the multiple anchor points of the same identified lane line.

[0053] S300: Generate a fitted lane line based on the anchor point line according to the distance between the anchor points other than the anchor point line and the anchor point line.

[0054] The anchor point in step S100 may be generated by a neural network model.

[0055] In some embodiments of the present application, step S100 includes the following sub-steps:

[0056] S101: Inputting a road image to be tested into the lane line detection model so that the lane line detection model outputs the anchor point and its anchor point detection data and identification detection data; wherein the anchor point detection data is used to indicate the position of the anchor point; and the identification detection data is used to indicate the identified lane line to which the anchor point belongs;

[0057] S102: Inputting the road image to be tested into the lane line detection model so that the lane line detection model outputs category detection data; wherein the category detection data is used to indicate the lane line type of the identified lane line.

[0058] As an optional solution, step S101 and step S102 may be performed in the same step.

[0059] In some embodiments of the present application, step S200 includes the following sub-steps:

[0060] S201: When the distance between the two anchor points is less than a preset line distance threshold, taking the line between the two anchor points as a part of the anchor point line.

[0061] In some embodiments of the present application, step S300 includes the following sub-steps:

[0062] S301: When the distance between the anchor points other than the anchor point line and the anchor point line is less than a preset fitting distance threshold, fit the anchor points other than the anchor point line to the anchor point line; or, when the distance between the anchor points other than the anchor point line and the anchor point line is greater than or equal to a preset fitting distance threshold, fit the anchor points other than the anchor point line to other anchor point lines.

[0063] S302: When there is no anchor point of the same identified lane line outside the anchor point line, use the anchor point line connecting the most anchor points as the fitted lane line.

[0064] S303: When the distance between two anchor points other than the anchor point connection line is less than a preset anchor point distance threshold, delete the two anchor points.

[0065] As a specific solution of this application, from the perspective of data processing, the technical solution of this application mainly includes the following steps:

[0066] Step S110: Processing the road image to be tested by the lane line detection model to obtain lane line detection data;

[0067] Step S120: determining at least one lane line and a lane line mark of each lane line in the road image to be tested according to the anchor point detection data and the mark detection data in the lane line detection data;

[0068] Step S130: Determine the lane line type of each lane line according to the category detection data in the lane line detection data.

[0069] The lane line detection data is obtained by the lane line detection model performing lane line detection on the road image to be tested. The lane line detection model is a neural network model based on deep learning. For the training process of the lane line detection model, please refer to the following embodiments, which will not be elaborated here. The road image to be tested can be a road image taken by a camera installed on the front side of the vehicle, that is, a road image under the front-view perspective, such as a frame of road image extracted from the driving video taken by the vehicle's driving recorder; it can also be other road images, such as road images under a bird's-eye view, etc. The embodiments of the present application are not limited to this. In actual applications, the road image to be tested and the historical road images used in the training of the lane line detection model correspond to the same shooting perspective, such as both are road images under the front-view perspective, which can ensure the detection accuracy of the lane line detection model.

[0070] In the embodiment of the present application, the lane line detection model performs lane line detection in at least three dimensions on the road image to be tested, namely: anchor points on the lane line, lane line markings, and lane line types. Therefore, the lane line detection data at least includes anchor point detection data, marking detection data, and category detection data.

[0071] The anchor point detection data is used to indicate an anchor point in at least one road grid of the road image to be tested, such as coordinate data of the anchor point. The lane line detection model of the embodiment of the present application divides the road image to be tested into multiple road grids during lane line detection on the road image to be tested.

[0072] Optionally, at most one anchor point is detected for each road grid. Thus, the anchor point detection data is used to indicate an anchor point corresponding to at least one of the multiple road grids divided, that is, for the multiple road grids divided by the road image to be tested, one or more road grids respectively detect an anchor point, and there may be one or more other road grids in which the anchor point is not detected. For the way of dividing the road grids, please refer to the following embodiment, which will not be described in detail here.

[0073] The identification detection data is used to indicate the lane identification of at least one anchor point, that is, the lane line to which the anchor point belongs (refer to Figure 4 As shown in the figure, this refers to the identified lane lines recognized by the model).

[0074] The lane line identification is used to indicate the position of the lane line in the road image to be tested. For example, the embodiment of the present application can sort the lane lines in the road image from left to right, and the lane line identification can indicate the order of the lane lines in the arrangement order from left to right, but this does not constitute a limitation of the present application, and the lane line identification can also have other sorting methods. The embodiment of the present application detects the lane line identification of each anchor point detected. Thus, in step S120, at least one lane line and the lane line identification of each lane line can be determined according to the anchor point detection data and the identification detection data. For example, the detected anchor points can be divided into multiple groups according to the identification detection data, and each group of anchor points has the same lane line identification. The anchor points with the same lane line identification are clustered into one lane line, which is defined as an identified lane line. In this way, at least one identified lane line and the lane line identification of each identified lane line can be generated for one image to be tested.

[0075] The category detection data is used to indicate the lane line type corresponding to the lane line mark. The embodiment of the present application does not limit the specific division method of the lane line type.

[0076] Exemplarily, lane line types include, but are not limited to, solid white lines, dashed white lines, solid yellow lines, dashed yellow lines, etc. It should be understood that each lane line may correspond to only one lane line type, and each lane line has only one lane line identifier, so that one lane line identifier also corresponds to only one lane line type. It should also be understood that in the lane line detection results obtained by the lane line detection model performing lane line detection on different road images to be tested, the same lane line identifier may correspond to different lane line types in different road images to be tested. In step S130, based on the category detection data in the lane line detection data, the lane line type corresponding to each lane line identifier can be obtained, and combined with the lane line identifier of each lane line determined in step S120, the lane line type of each lane line can be determined.

[0077] In summary, the lane line detection method provided in the embodiment of the present application performs lane line detection on the road image to be tested through the lane line detection model to output lane line detection data, and the lane line detection data includes anchor point detection data, identification detection data and category detection data. According to the lane line detection data, at least one identified lane line in the road image to be tested, as well as the lane line identification and lane line type of each identified lane line can be determined. The embodiment of the present application performs lane line detection based on the lane line detection model, and the detection process is simple and the detection speed is fast, which can ensure the accuracy and efficiency of lane line detection for various road scenes.

[0078] Furthermore, in the embodiment of the present application, lane line detection is implemented based on anchor point detection, and dense anchor points are helpful for parameter processing of lane lines with greater curvature, thereby improving the accuracy of lane line detection with large curvature.

[0079] In addition, when performing lane line detection, the lane line detection model of the embodiment of the present application divides the road image to be tested into multiple road grids, and detects at most one anchor point for each road grid, which simplifies the detection content of the lane line detection model and reduces the computational complexity of the lane line detection model, so as to facilitate the deployment of the lane line detection model on the edge computing device.

[0080] In some embodiments, the aforementioned step S120 may include the following steps:

[0081] Step S121: determining at least one first anchor point according to the anchor point detection data;

[0082] Step S122: determining the lane line mark of each first anchor point according to the mark detection data;

[0083] Step S123: fitting at least one first anchor point according to the lane line identifier of at least one first anchor point to obtain at least one lane line and the lane line identifier of each lane line.

[0084] The anchor point detection data is used to indicate at least one anchor point, and the at least one anchor point can be used as the first anchor point, or one or more anchor points can be selected from the at least one anchor point as the first anchor point. Optionally, the anchor point detection data also includes the confidence of at least one anchor point, and the first anchor point is the anchor point whose confidence is greater than the confidence threshold among the at least one anchor point. It should be understood that the larger the confidence threshold is set, the fewer first anchor points are extracted from the at least one anchor point, and the greater the probability that the first anchor point belongs to the lane line; the smaller the confidence threshold is set, the more first anchor points are extracted from the at least one anchor point, and the first anchor point may include some non-lane line points (such as speed bumps, etc.).

[0085] The lane line identification of each first anchor point can be further parsed from the identification detection data, and then the first anchor point is fitted according to the lane line identifications of all the first anchor points to obtain at least one identified lane line and the lane line identification of each lane line.

[0086] When at least one identified lane line is obtained by fitting, all first anchor points screened in step S121 may participate in the fitting process, or these first anchor points may be further screened and then fitted to further improve the accuracy of lane line detection.

[0087] Based on this, in some embodiments, the aforementioned step S123 may include the following steps:

[0088] Step S1231: extracting at least one second anchor point having the same lane line identifier from at least one first anchor point according to the lane line identifier of at least one first anchor point;

[0089] Step S1232: determining at least one anchor point subset according to at least one second anchor point;

[0090] Step S1233: Generate an identified lane line corresponding to the lane line identification according to at least one anchor point subset.

[0091] According to the lane line mark, at least one first anchor point can be divided into multiple groups of second anchor points, each group of second anchor points has the same lane line mark. Then, each group of second anchor points is divided into at least one anchor point subset, and then the lane line corresponding to the lane line mark of the group of second anchor points is obtained by fitting according to the at least one anchor point subset.

[0092] For each group of second anchor points, the total number of second anchor points in at least one anchor point subset may be equal to or less than the number of second anchor points in the group. That is, when dividing at least one anchor point subset for each group of second anchor points, the group of second anchor points may be screened to filter out the wrongly detected anchor points. Figure 2 As shown, according to the lane line markings, two groups of second anchor points can be obtained, wherein the lane line markings of one group of second anchor points are all L0, that is, they belong to one identified lane line, and the lane line markings of the other group of second anchor points are all L1, that is, they belong to another identified lane line. Among the second anchor points with the lane line marking L0, two second anchor points P0 and P1 are detected incorrectly. The actual lane line markings of these two second anchor points should be L1, but they are mistakenly detected as L0. The embodiment of the present application filters these misdetected anchor points to improve the accuracy of lane line fitting.

[0093] In some embodiments, a region growing algorithm may be used to screen each group of second anchor points. Based on this, taking one group of second anchor points as an example, the aforementioned step S1232 may include the following steps:

[0094] Step S12321: for each initial growth point in at least one second anchor point, extract multiple groups of iterative growth points of the initial growth point from at least one second anchor point;

[0095] Step S12322: Generate an anchor point subset corresponding to the initial growth point based on the initial growth point and multiple groups of iterative growth points.

[0096] It should be understood that steps S12321 and S12322 can be executed in a loop multiple times to traverse all anchor points in a group of second anchor points. In each loop execution process, an initial growth point needs to be selected from at least one second anchor point included in the group, and the initial growth point can be randomly selected; or, the initial growth point based on the current loop execution process can be selected from the second anchor points that have not been traversed in the previous loop execution process.

[0097] Taking one of the loop execution processes as an example, after determining the initial growth point, multiple groups of iterative growth points are obtained through multiple iterations according to the regional growth algorithm. Among them, the distance between the first group of iterative growth points and the initial growth point in the multiple groups of iterative growth points, and the distance between two adjacent groups of iterative growth points in the multiple groups of iterative growth points, are all less than a preset distance threshold, for example, the distance threshold is 30 pixels. That is to say, when obtaining the iterative growth points in each iteration, they are screened according to the preset distance threshold.

[0098] For example, Figure 3 As shown, for a group of second anchor points whose lane line marks are all L0, an initial growth point Q0 can be randomly selected; then, second anchor points whose distance from the initial growth point Q0 is less than a preset distance threshold are extracted from other second anchor points as the first group of iterative growth points.

[0099] Reference Figure 3 As shown, the first group of iterative growth points includes Q1 and Q2; then for each iterative growth point in the first group of iterative growth points, a second anchor point whose distance to the iterative growth point is less than a preset distance threshold is extracted from the remaining second anchor points as the second group of iterative growth points, such as Figure 3 As shown, the second set of iterative growth points includes Q3 and Q4; and by analogy, the anchor point subset Lane0 corresponding to the initial growth point Q0 can be obtained.

[0100] like Figure 3 As shown, with one or more new initial growth points, the aforementioned steps S12321 and S12322 are respectively executed cyclically, so that a group of second anchor points with the lane line identified as L0 can be divided into two anchor point subsets Lane0 and Lane1, that is, two different anchor point lines.

[0101] Among them, a group of second anchor points with the lane line identifier L0 also includes two second anchor points P0 and P1. The two second anchor points do not form an anchor point subset with other second anchor points. The two second anchor points can be respectively regarded as an anchor point subset, or the two second anchor points can be directly filtered out for efficiency considerations, and there is no need to consider whether the two second anchor points belong to other lane lines.

[0102] Based on the aforementioned steps S12321 and S12322, at least one anchor point subset corresponding to a group of second anchor points can be obtained. In the aforementioned step S1233, based on at least one anchor point subset, a lane line corresponding to the lane line identification of the group of second anchor points is generated. The embodiment of the present application does not limit the specific way of fitting the lane line in step S1233. Optionally, for at least one anchor point subset corresponding to a group of second anchor points, a lane line can be fitted according to the second anchor points in all anchor point subsets; or, a fitted lane line can be fitted according to the anchor point subset with the largest number of anchor points.

[0103] When a lane line is fitted according to the anchor point subset with the largest number of anchor points, the second anchor point in other anchor point subsets can be directly filtered out, or a distance comparison can be performed with the fitted lane line to determine whether to retain the second anchor point.

[0104] In summary, the lane line detection method provided in the embodiment of the present application processes the anchor points detected by the lane line detection model through the region growing algorithm to filter out the erroneously detected anchor points, and then obtains the fitted lane line through the retained anchor points.

[0105] Since the anchor points used in fitting the lane lines have been filtered by the region growing algorithm, the retained anchor points are more likely to belong to the lane lines and the lane line markings are accurate, so the fitted lane lines are more accurate, effectively improving the accuracy of lane line detection.

[0106] As a solution to the problem, in some embodiments, the lane line detection method provided in the embodiments of the present application may also include a part of pre-training the lane line detection model, specifically including the following steps:

[0107] Step 010: Process each historical road image in the training data set through the lane line detection model to obtain lane line detection data of each historical road image;

[0108] Step 020: determining a detection loss value of a lane line detection model according to lane line detection data of a plurality of historical road images and lane line annotation data of a plurality of historical road images in a training data set;

[0109] Step 030: Adjust the parameters of the lane line detection model to converge the detection loss value and obtain a trained lane line detection model.

[0110] The training data set includes multiple historical road images and lane line annotation data for each historical road image. The embodiment of the present application processes each historical road image through a lane line detection model to obtain lane line detection data for each historical road image. Then, based on the lane line detection data and lane line annotation data of multiple historical road images, the detection loss value of the lane line detection model can be determined. By adjusting the parameters of the lane line detection model so that the detection loss value converges, such as making the detection loss value tend to a stable value, a lane line detection model that has completed training can be obtained.

[0111] In the embodiment of the present application, the lane line detection model performs lane line detection in at least three dimensions: anchor points on the lane line, lane line markings, and lane line types. Therefore, the detection loss value of the lane line detection model can be constructed based on the losses in these three dimensions.

[0112] Exemplarily, the detection loss value includes negative sample confidence loss Loss_neg, positive sample confidence loss Loss_pos, x-coordinate loss Loss_x, y-coordinate loss Loss_y, clustering loss Loss_cluster, and classification loss Loss_cls. The calculation formula of the detection loss value Loss is as follows:

[0113] Loss = Loss_neg + Loss_pos + Loss_x + Loss_y + Loss_cluster + Loss_cls

[0114] Optionally, in actual application, a weight value may be set for each loss of the detection loss value, and the sum of the weight values ​​of all losses is equal to one.

[0115] Among them, a positive sample refers to a road grid in which lane lines exist in a historical road image, and a negative sample refers to a road grid in which no lane lines exist in a historical road image. The embodiment of the present application pre-annotates the historical road image, annotates multiple road grids in the historical road image, and annotates whether there are lane lines in each road grid, so that positive samples and negative samples can be determined based on the lane line annotation data of the historical road image.

[0116] Exemplarily, the anchor point detection data output by the lane line detection model for historical road images includes pred_points, which includes three dimensions, namely the probability that the detected anchor point belongs to the lane line (i.e., confidence), the x-axis coordinate in the road grid, and the y-axis coordinate in the road grid. These three dimensions are represented by pred_points[:,:,0], pred_points[:,:,1], and pred_points[:,:,2], respectively.

[0117] Exemplarily, the lane line marking data includes label_points and label_cls. The label_cls indicates the lane line type corresponding to the marked lane line mark; the label_points includes four dimensions, namely, whether there is a lane line in the road grid (1 if it exists, 0 if it does not exist), the x-axis coordinate of the lane line marking point in the road grid, the y-axis coordinate of the lane line marking point in the road grid, and the lane line mark of the lane line in the road grid. These four dimensions are represented by label_points[:,:,0], label_points[:,:,1], label_points[:,:,2], and label_points[:,:,3], respectively.

[0118] Exemplarily, the identification detection data is represented by pred_id.

[0119] Exemplarily, the class detection data is represented by pred_cls.

[0120] Based on this, the calculation formulas for negative sample confidence loss Loss_neg, positive sample confidence loss Loss_pos, x-coordinate loss Loss_x, y-coordinate loss Loss_y, clustering loss Loss_cluster, and classification loss Loss_cls are as follows:

[0121] Loss_neg=0.05×∑|(pred_points[:,:,0]- label_points[:,:,0])×(1-label_points[:,:,0])|

[0122] Loss_pos=∑|(pred_points[:,:,0]- label_points[:,:,0]) × label_points[:,:,0]|

[0123] Loss_x=∑|(pred_points[:,:,1]- label_points[:,:,1]) × label_points[:,:,0]|

[0124] Loss_y=2×∑|(pred_points[:,:,2]- label_points[:,:,2]) × label_points[:,:,0]|

[0125] Loss_cluster=focalloss(label_points[:,:,2], pred_id, label_points[:,:,0])

[0126] Loss_cls=focalloss(label_cls, pred_cls)

[0127] Among them, focalloss is a loss function based on binary cross entropy. Both clustering loss and classification loss are calculated based on focalloss. Only road grids with lane lines are calculated, that is, road grids with label_points[:,:,0] being 1.

[0128] The historical road image is obtained based on the road image taken by the camera installed on the front side of the vehicle. The historical road image can be the initial camera-collected image, or it can be obtained after data preprocessing and / or data enhancement processing of the camera-collected image. The embodiment of the present application constructs a training data set based on multiple historical road images and the lane line annotation data of each historical road image.

[0129] In some embodiments, the lane line detection method provided in the embodiments of the present application may further include the following steps:

[0130] Step 040: Acquire at least one initial road image and initial annotation data of each initial road image;

[0131] Step 050: performing data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain a plurality of historical road images and lane line annotation data of each historical road image;

[0132] Step 060: Construct a training data set based on multiple historical road images and lane line annotation data of each historical road image.

[0133] In the embodiment of the present application, multiple frames of images can be extracted at a certain interval from the driving video captured by the camera installed on the front side of the vehicle, and at least one initial road image can be extracted from the multiple frames of images. Among them, an image containing lane lines can be extracted from the multiple frames of images as the initial road image. After that, it can be annotated manually or based on an open source annotation tool to obtain initial annotation data of the initial road image.

[0134] For example, for each image, N annotation points including two endpoints are marked on the axis of each lane line. The size of N is determined according to the actual situation. When the lane line is more curved, the number of annotation points can be increased to ensure that the center of each line segment coincides with the axis of the lane line. The marked lane lines need to be extended to the bottom of the image and ensure that the far ends of each lane line are aligned, such as Figure 4As shown. If the lane line is completely blocked, no annotation is required. Otherwise, annotation is required. Then, the lane line type of each lane line is marked. Finally, the annotation data can be saved in json format.

[0135] The embodiment of the present application can perform interpolation processing on the annotated points in the initial annotated data to achieve the expansion of the annotated points. For example, for any initial road image, all the annotated points on each lane line can be extracted from the initial annotated data, and then each lane line can be interpolated according to the annotated points to ensure that the distance between adjacent points after interpolation is less than 1 pixel, but it is not limited to this. The smaller the distance, the higher the accuracy of the anchor point generated by the lane line detection model. After that, the lane lines in the initial road image can be sorted in order from left to right to obtain the lane line identification of each lane line.

[0136] In order to expand the diversity of training samples in the training data set and ensure that the lane line detection model can achieve high-precision positioning, the embodiment of the present application, after obtaining at least one initial road image and initial annotation data of each initial road image, also performs data enhancement processing on the initial road image and its initial annotation data to obtain historical road images and their lane line annotation data for constructing the training data set.

[0137] In some embodiments, the data enhancement process includes rotation enhancement process; taking one of the initial road images as an example, the aforementioned step S050 may include the following steps:

[0138] Step S051: rotating the initial road image according to a preset angle to obtain a historical road image;

[0139] Step S052: performing coordinate rotation processing on the initial point annotation data in the initial annotation data of the initial road image according to the rotation matrix corresponding to the preset angle;

[0140] Step S053: generating anchor point annotation data of the historical road image according to the processed initial point annotation data;

[0141] Step S054: Generate lane line annotation data of the historical road image according to the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0142] The preset angle refers to a preset rotation angle. For example, to ensure that the lane lines still comply with driving rules after the road image is rotated, the preset angle can be any rotation angle within ±30°; or, the preset angle is the rotation angle corresponding to each certain angle interval within ±30°, such as the rotation angle corresponding to each rotation interval of 5°. Of course, the preset angle can also be set in other ways, and the embodiment of the present application is not limited to this. In the aforementioned step S051, when rotating the initial road image, the vertical line passing through the center of the initial road image can be used as the rotation axis, but it is not limited to this. It should be understood that rotating the initial road image according to a preset angle can obtain a historical road image, and rotating the initial road image according to multiple different preset angles can obtain multiple historical road images, that is, the embodiment of the present application can repeat the aforementioned steps S051 to S054 at different preset angles to obtain multiple historical road images and their lane line annotation data.

[0143] A rotation matrix can be calculated according to a preset angle, and all the annotation points in the initial road image can be rotated according to the rotation matrix, that is, the initial point annotation data. The initial point annotation data includes the coordinates of each annotation point in the initial road image. Optionally, in an embodiment of the present application, the processed initial point annotation data obtained by performing coordinate rotation processing on the initial point annotation data includes the annotation point data within the range of the historical road image after the coordinate rotation processing.

[0144] Since the lane line detection model in the embodiment of the present application divides the road image into multiple road grids when performing lane line detection, and predicts anchor points in each road grid, the embodiment of the present application further generates anchor point annotation data based on the processed initial point annotation data.

[0145] In some embodiments, the aforementioned step S053 may include the following steps:

[0146] Step S05A: mapping each lane line annotation point indicated by the processed initial point annotation data to a road grid of the historical road image;

[0147] Step S05B: for each road grid, when at least one lane line marking point is mapped to the road grid, an anchor point is generated in the road grid according to the at least one lane line marking point mapped to the road grid;

[0148] Step S05C: Generate anchor point annotation data of the historical road image according to at least one anchor point in the road grid of the historical road image.

[0149] The embodiment of the present application divides the historical road image into multiple road grids, wherein the size of each road grid can be the same as the size of the feature map output by the lane detection model. For example, if the size of the feature map output by the lane detection model is 60×30, then the size of each road grid is 60×30; if the size of the feature map output by the lane detection model is 120×60, then the size of each road grid is 120×60. Optionally, the size of the road grid can differ from the size of the historical road image by a target multiple, such as 16 times. For example, the embodiment of the present application sets the size of the historical road image to 960×480, then the size of each road grid is 60×30.

[0150] Optionally, since the lane line detection model detects at most one anchor point for a road grid, the embodiment of the present application first maps each lane line annotation point to a road grid, and then generates an anchor point based on all lane line annotation points mapped to a road grid, so as to ensure that the anchor point annotation data is used to indicate an anchor point that corresponds one-to-one to at least one road grid.

[0151] For example, the anchor point annotation data can be expressed as [x, y, dx, dy, pred_id], where (x, y) is used to indicate the position of the road grid in the historical road image, (dx, dy) is used to indicate the position of the anchor point annotated in the road grid, and pred_id is used to indicate the lane line identifier of the annotated anchor point. Figure 5 As shown in , the size of the historical road image is 16 times the size of the road grid (the width of the historical road image is 16 times the width of the road grid, and the height of the historical road image is 16 times the height of the road grid). The lane lines in the historical road image are sorted from left to right, then Figure 5 The position of the road grid shown in the black bold grid can be expressed as (5,3); an anchor point is determined for the road grid, and the lane line of the anchor point is marked as L0.

[0152] Among them, for each road grid, the coordinates (xi, yi) mapped to each lane line annotation point in the road grid can be obtained based on the initial point annotation data. The coordinates are used to indicate the position of the lane line annotation point in the historical road image. According to the coordinates (x, y) of the road grid in the historical road image, the size of the road grid (wsize, hsize) and the coordinates (xi, yi) of all lane line annotation points in the road grid in the historical road image, the coordinates (dx, dy) of the anchor point in the road grid can be determined. Taking the size of the historical road image as 960×480 and the size of each road grid as 60×30 as an example, the calculation formula for the coordinates (dx, dy) of the anchor point in the road grid is as follows:

[0153] dx=(x1-x×wsize) / wsize

[0154] dy=(y1-y×hsize) / hsize

[0155] wsize=960 / 60=16

[0156] hsize=460 / 30=16

[0157] x1=sum(xi) / N

[0158] y1=sum(xi) / N

[0159] Wherein, N refers to the number of lane marking points in a road grid, N is a positive integer, and i is a positive integer less than or equal to N.

[0160] In some embodiments, the data enhancement process includes a cropping enhancement process; taking one of the initial road images as an example, the aforementioned step S050 may include the following steps:

[0161] Step S055: cropping the initial road image according to a preset sliding window to obtain a historical road image;

[0162] Step S056: performing coordinate conversion processing on the initial point annotation data in the initial annotation data of the initial road image according to the sliding window coordinate system corresponding to the preset sliding window;

[0163] Step S057: generating anchor point annotation data of the historical road image according to the processed initial point annotation data;

[0164] Step S058: Generate lane line annotation data of the historical road image according to the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0165] The preset sliding window refers to a preset cropping range. Optionally, the preset sliding window has the same aspect ratio as the initial road image. Optionally, the area of ​​the preset sliding window is greater than half of the area of ​​the initial road image. According to different positions of the preset sliding window in the initial road image, multiple different historical road images can be cropped. In the embodiment of the present application, a certain step length can be set for the preset sliding window, and the preset sliding window can be controlled to slide in the initial road image according to the step length to obtain multiple historical road images.

[0166] According to the sliding window coordinate system corresponding to the preset sliding window, the coordinates of each annotation point can be converted from the initial road image coordinate system to the sliding window coordinate system to update the coordinates of each annotation point. Optionally, in an embodiment of the present application, only the annotation points that are still within the preset sliding window range after cropping can be subjected to coordinate conversion processing to obtain the processed initial point annotation data. Optionally, in an embodiment of the present application, a uniform size can be set for the road image input to the lane line detection model, such as 960×480, so that after the historical road image is obtained by cropping according to the preset sliding window, the size of the historical road image can be adjusted to a preset uniform size, such as 960×480. Correspondingly, the coordinates of the annotation points should also be adjusted to the coordinates in the preset uniform size after the coordinate conversion processing.

[0167] Optionally, since in the embodiment of the present application, the lane line detection model divides the road image into multiple road grids when performing lane line detection, and predicts at most one anchor point in each road grid. Therefore, the embodiment of the present application further generates anchor point annotation data based on the processed initial point annotation data. In some embodiments, the aforementioned step S057 may include the above steps S05A to S05C. Please refer to the aforementioned embodiment for the introduction and description, which will not be repeated here.

[0168] In order to improve the diversity of training samples and the generalization ability of the lane line detection model, the embodiment of the present application can also perform flip enhancement processing. For example, after step S057, the historical road image and the anchor point annotation data and the identifier annotation data can be flipped with a preset probability, and then the lane line annotation data of the flipped historical road image is generated according to the flipped anchor point annotation data and the identifier annotation data, as well as the category annotation data. The preset probability can be set in combination with actual needs, for example, set to 50%. Among them, in order to ensure that the flipped road image conforms to the driving rules, the embodiment of the present application can perform left-right flipping, so that the lane line marking after flipping processing is opposite to the left-right order of the original lane line marking, and the sum of the coordinates of the flipped anchor point and the original anchor point on the image width is equal to the image width. For example, the width of the road grid is wsize, and the coordinate of the anchor point in the road grid of the original historical road image corresponding to the image width is dx, then the coordinate of the anchor point in the road grid of the flipped historical road image corresponding to the image width is wsize-dx.

[0169] In summary, the lane line detection method provided in the embodiment of the present application performs data enhancement processing on the initial road image and its lane line annotation data when constructing the training data set of the lane line detection model. Among them, the data enhancement processing may include rotation enhancement processing and / or cropping enhancement processing. The rotation enhancement processing can enrich the angle between the lane line and the road image, and the cropping enhancement processing can enrich the different positions of the lane line in the road image. Therefore, the embodiment of the present application can effectively expand the number of samples in the training data set through data enhancement processing, improve the diversity of training samples in the training data set, and effectively improve the robustness and generalization ability of the lane line detection model.

[0170] Below, the lane line detection model, the training method of the lane line detection model, and the method of using the lane line detection model provided in the embodiments of the present application are introduced and explained with a specific example.

[0171] Reference Figure 6 , Figure 6 It is a schematic diagram of a lane detection model provided in an embodiment of the present application.

[0172] The embodiment of the present application can use pytorch to build a lane line model, but is not limited to this. Figure 6 As shown in the figure, the backbone of the model uses the deep learning model STDCNet813 for feature extraction, and takes the output stride=16 and stride=32 feature maps for subsequent calculations. On the stride=32 feature map extracted by the backbone, the attention mechanism can be used to extract features, then upsampled using interpolation, and finally a conv2d+bn+relu is used to obtain the feature map; the feature map is added to the stride=16 feature map to fuse the features of the previous and next layers; after the fusion of features, several conv2d+bn+relu are connected to obtain the head of the model. The size of the head is 1 / 16 of the input road image. For example, the size of the input road image is 960×480, and the size of the feature map is 60×30. Connecting three branches in parallel after the feature map can realize anchor point detection, lane line clustering function and lane line classification function.

[0173] The anchor detection branch can use a conv2d+sigmoid on the head feature map, and the output feature map size can be 60×30×3, where the three channels are a confidence score, a coordinate value dx, and a coordinate value dy.

[0174] The lane line clustering branch connects conv2d+20×sigmoid+softmax in sequence on the head feature map, and the final feature map size can be 60×30×8. After the anchor point detection branch extracts the anchor point, the 8-dimensional feature vector in the lane line clustering branch can be used to parse which lane line the anchor point belongs to, that is, the lane line identification of the anchor point. To ensure clustering convergence and model training, the lane lines can be arranged in a fixed order, such as from left to right.

[0175] The lane classification branch uses 1x1 convolution instead of full connection to reduce the number of parameters. After the head feature map passes through avgpool or maxpool in sequence, it is connected to multiple conv2d+bn+relu layers, where conv2d is 1x1 convolution, and finally connected to conv2d+20×sigmoid, and the output dimension is 104. The output dimension is 104 because each road image is preset to predict a maximum of 8 lane lines, and 13 lane line types are preset. After that, the 104-dimensional feature vector is adjusted to 8×13 and connected to the softmax function as the output feature map of the lane classification branch. The lane line type can be extracted based on the output feature map.

[0176] The embodiment of the present application performs data enhancement processing such as rotation, cropping and / or flipping on the initial road image and its lane line annotation data to obtain multiple historical road images and their lane line annotation data, and constructs a training data set based on this, effectively expanding the sample diversity in the training data set.

[0177] Optionally, the embodiment of the present application uses RGB for model training, normalizes the historical road image, and then adds data enhancement methods such as changing the image brightness, chromaticity, and contrast. Among them, image normalization includes dividing each pixel value in the image by 255.

[0178] Optionally, the embodiment of the present application uses a training data set to Figure 6 Before training the lane detection model shown, the lane detection model can be trained using the imagenet dataset to obtain a pre-trained lane detection model, and then the pre-trained lane detection model can be trained using the training dataset. Optionally, a gradient descent algorithm can be used during model training, with an initial learning rate of 0.0001, and the learning rate is reduced to 1 / 10 after several epochs.

[0179] Optionally, the embodiment of the present application may convert the lane line annotation data including the anchor point annotation data, the identifier annotation data and the class annotation data into model training labels label_points[x,y] and label_class[i] in the following manner:

[0180] label_points=zeros(60,30,4)

[0181] label_points[x,y]=[1,dx,dy,lane_id]

[0182] label_ class[i]=class_index

[0183] Multiple historical road images in the training data set are input into the lane line detection model to obtain the lane line detection data of each historical road image; then, the detection loss value of the lane line detection model is determined according to the lane line detection data of the multiple historical road images and the model training label; the parameters of the lane line detection model are adjusted so that the detection loss value converges, and a lane line detection model that has completed training can be obtained.

[0184] When using the lane line detection model to perform lane line detection, the embodiment of the present application can first adjust the size of the road image to be tested, and adjust the size of the road image to be tested to a preset uniform size. For example, the original size of the road image to be tested is 1920×1080, and it is adjusted to 960×480. Then, the road image to be tested is normalized and then input into the lane line detection model to obtain the lane line detection result of the road image to be tested.

[0185] in, Figure 6 The three branches shown can output feature maps pred_points, pred_id and pred_cls respectively. The embodiment of the present application sets a confidence threshold, extracts anchor points with confidence greater than the confidence threshold from the pred_points feature map, and maps the coordinates of the anchor points in the road grid to the image; then parses the lane line mark corresponding to the anchor point in the pred_id feature map; and finally parses the lane line type from the pred_cls feature map.

[0186] According to the second aspect of the present application, an embodiment of the present application also provides a lane line detection device.

[0187] Reference Figure 7 , Figure 7 is a block diagram of a lane line detection device provided in an embodiment of the present application. The lane line detection device may be an electronic device, or may be arranged in an electronic device. Figure 7 As shown, the lane line detection device 700 includes: a detection module 710 and a determination module 720.

[0188] A detection module 710 is used to process the road image to be detected by using a lane line detection model to obtain lane line detection data;

[0189] The determination module 720 is used to determine at least one lane line in the road image to be tested and the lane line identification of each lane line according to the anchor point detection data and the identification detection data in the lane line detection data; and determine the lane line type of each lane line according to the category detection data in the lane line detection data; wherein the anchor point detection data is used to indicate an anchor point in at least one road grid of the road image to be tested, the identification detection data is used to indicate the lane line identification of at least one anchor point, and the category detection data is used to indicate the lane line type corresponding to the lane line identification.

[0190] In some embodiments, the determination module 720 is further used to: determine at least one first anchor point based on the anchor point detection data; determine the lane line identification of each first anchor point based on the identification detection data; fit at least one first anchor point according to the lane line identification of at least one first anchor point to obtain at least one lane line and the lane line identification of each lane line.

[0191] In some embodiments, the determination module 720 is further used to: extract at least one second anchor point with the same lane line identification from at least one of the first anchor points according to the lane line identification of at least one of the first anchor points; determine at least one anchor point subset based on at least one of the second anchor points; and generate the lane line corresponding to the lane line identification based on at least one of the anchor point subsets.

[0192] In some embodiments, the determination module 720 is also used to: for each initial growth point in at least one of the second anchor points, extract multiple groups of iterative growth points of the initial growth point from at least one of the second anchor points; generate the anchor point subset corresponding to the initial growth point based on the initial growth point and the multiple groups of iterative growth points; wherein the distance between the first group of iterative growth points in the multiple groups of iterative growth points and the initial growth point, and the distance between two adjacent groups of iterative growth points in the multiple groups of iterative growth points, are both less than a preset distance threshold.

[0193] In some embodiments, Figure 8 As shown, the lane line detection device 700 also includes a training module 730, which is used to: process each historical road image in the training data set through the lane line detection model to obtain the lane line detection data of each historical road image; determine the detection loss value of the lane line detection model based on the lane line detection data of multiple historical road images and the lane line annotation data of multiple historical road images in the training data set; adjust the parameters of the lane line detection model to converge the detection loss value, and obtain the lane line detection model that has completed training.

[0194] In some embodiments, Figure 8 As shown, the lane line detection device 700 also includes a construction module 740, which is used to: obtain at least one initial road image and initial annotation data of each initial road image; perform data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain multiple historical road images and the lane line annotation data of each historical road image; and construct the training data set based on the multiple historical road images and the lane line annotation data of each historical road image.

[0195] In some embodiments, Figure 8 As shown, the construction module 740 is also used to: rotate the initial road image according to a preset angle to obtain the historical road image; perform coordinate rotation processing on the initial point annotation data in the initial annotation data of the initial road image according to the rotation matrix corresponding to the preset angle; generate anchor point annotation data of the historical road image based on the processed initial point annotation data; generate the lane line annotation data of the historical road image based on the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0196] In some embodiments, Figure 8 As shown, the construction module 740 is further used to: crop the initial road image according to a preset sliding window to obtain the historical road image; perform coordinate conversion processing on the initial point annotation data in the initial annotation data of the initial road image according to the sliding window coordinate system corresponding to the preset sliding window; generate anchor point annotation data of the historical road image according to the processed initial point annotation data; generate the lane line annotation data of the historical road image according to the anchor point annotation data and the identification annotation data and category annotation data in the initial annotation data.

[0197] In some embodiments, Figure 8 As shown, the construction module 740 is also used to: map each lane line annotation point indicated by the processed initial point annotation data to one of the road grids of the historical road image; for each of the road grids, when at least one lane line annotation point is mapped to the road grid, generate the anchor point in the road grid according to at least one lane line annotation point mapped to the road grid; and generate the anchor point annotation data of the historical road image according to the anchor point in at least one of the road grids of the historical road image.

[0198] It should be understood that, in specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. In addition, the specific implementation methods and corresponding beneficial effects of the above modules can be referred to the above method embodiments, which will not be described in detail here.

[0199] According to the third aspect of the present application, the embodiment of the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the aforementioned lane line detection method are implemented. The non-transitory computer-readable storage medium has all the beneficial effects of the aforementioned lane line detection method, and the present application will not repeat them here.

[0200] According to the fourth aspect of the present application, an embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein a computer program is stored in the memory; and the processor is used to execute the computer program in the memory to implement the steps of the aforementioned lane line detection method. The electronic device has all the beneficial effects of the aforementioned lane line detection method, which will not be described in detail in this application.

[0201] Reference Fig. 9 , the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0202] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Fig. 9 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0203] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the aforementioned functions defined in the method of some embodiments of the present application are executed.

[0204] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the foregoing two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above, which is not specifically limited in the present application. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0206] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0207] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently without being installed in the electronic device. The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device: processes the road image to be tested through the lane line detection model to obtain lane line detection data; determines at least one lane line in the road image to be tested and the lane line identification of each lane line according to the anchor point detection data and the identification detection data in the lane line detection data; determines the lane line type of each lane line according to the category detection data in the lane line detection data; wherein the anchor point detection data is used to indicate an anchor point in at least one road grid of the road image to be tested, the identification detection data is used to indicate the lane line identification of at least one anchor point, and the category detection data is used to indicate the lane line type corresponding to the lane line identification.

[0208] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function.

[0210] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.

[0211] For example, two boxes shown in succession may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0212] The units described in some embodiments of the present application may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a detection module and a determination module. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the detection module may also be described as "a unit for processing the road image to be tested by means of a lane line detection model to obtain lane line detection data".

[0213] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0214] According to a fifth aspect of the present application, an embodiment of the present application further provides a vehicle, including a lane line detection device or an electronic device, which can be used to perform the aforementioned lane line detection method. The vehicle has all the beneficial effects of the aforementioned lane line detection device or electronic device, which will not be described in detail in this application.

[0215] The vehicle may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific limitation on this.

[0216] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0217] In the aforementioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0218] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0219] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of various embodiments in the embodiments of the present application have different focuses, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A lane line detection method, comprising: Acquire or generate multiple anchor points in a road image to be tested that are identified as belonging to the same identified lane line; wherein, the acquisition or generation of multiple anchor points in a road image to be tested that are identified as belonging to the same identified lane line includes: inputting the road image to be tested into a lane line detection model so that the lane line detection model outputs the anchor point and its anchor point detection data, identification detection data and category detection data; the anchor point detection data is used to indicate the position of the anchor point; the identification detection data is used to indicate the identified lane line to which the anchor point belongs; the category detection data is used to indicate the lane line type of the identified lane line; Forming at least one anchor point line according to the distances between the multiple anchor points of the same identified lane line; wherein, forming at least one anchor point line according to the distances between the multiple anchor points of the same identified lane line comprises: when the distance between two anchor points is less than a preset line distance threshold, taking the line between the two anchor points as a part of the anchor point line; Generate a fitted lane line based on the anchor point line according to the distance between the anchor point line and the anchor point line other than the anchor point line; wherein, generating a fitted lane line based on the anchor point line according to the distance between the anchor point line and the anchor point line other than the anchor point line comprises: when the distance between the anchor point line other than the anchor point line and the anchor point line is less than a preset fitting distance threshold, fitting the anchor point line other than the anchor point line to the anchor point line; Wherein, the lane line detection method further includes: Training or providing the lane detection model; The training or providing of the lane line detection model further includes: Acquire at least one initial road image and initial annotation data of each of the initial road images; Performing data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain a plurality of historical road images and lane line annotation data of each of the historical road images; Constructing a training data set according to the plurality of historical road images and the lane line annotation data of each of the historical road images; The step of performing data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain a plurality of historical road images and lane line annotation data of each of the historical road images includes: Rotating the initial road image according to a preset angle to obtain the historical road image; performing coordinate rotation processing on the initial point annotation data in the initial annotation data of the initial road image according to the rotation matrix corresponding to the preset angle; generating anchor point annotation data of the historical road image according to the processed initial point annotation data; Generate the lane line annotation data of the historical road image according to the anchor point annotation data and the identification annotation data and the category annotation data in the initial annotation data; Alternatively, the performing data enhancement processing on the initial road image and the initial annotation data of the initial road image to obtain a plurality of historical road images and lane line annotation data of each of the historical road images includes: Cropping the initial road image according to a preset sliding window to obtain the historical road image; performing coordinate conversion processing on the initial point annotation data in the initial annotation data of the initial road image according to the sliding window coordinate system corresponding to the preset sliding window; generating anchor point annotation data of the historical road image according to the processed initial point annotation data; The lane line annotation data of the historical road image is generated according to the anchor point annotation data and the identification annotation data and the category annotation data in the initial annotation data.

2. The lane line detection method according to claim 1, characterized in that: in, The generating, according to the distance between the anchor points other than the anchor point line and the anchor point line, a fitted lane line based on the anchor point line also includes: When there is no anchor point of the same identified lane line outside the anchor point line, the anchor point line connecting the most anchor points is used as the fitted lane line.

3. The lane line detection method according to claim 2, characterized in that: in, The generating, according to the distance between the anchor points other than the anchor point line and the anchor point line, a fitted lane line based on the anchor point line also includes: When the distance between two anchor points other than the anchor point connection line is less than a preset anchor point distance threshold, the two anchor points are deleted.

4. The lane line detection method according to claim 2, characterized in that: When the distance between the anchor points other than the anchor point line and the anchor point line is greater than or equal to a preset fitting distance threshold, the anchor points other than the anchor point line are fitted to the other anchor point lines.

5. The lane line detection method according to claim 1, characterized in that: The training or providing the lane line detection model includes: Processing each historical road image in the training data set by the lane line detection model to obtain lane line detection data of each historical road image; Determining a detection loss value of the lane line detection model according to the lane line detection data of the plurality of historical road images and the lane line annotation data of the plurality of historical road images in the training data set; The parameters of the lane line detection model are adjusted to converge the detection loss value, thereby obtaining the trained lane line detection model.

6. The lane line detection method according to claim 1, characterized in that: The generating anchor point annotation data of the historical road image according to the processed initial point annotation data comprises: Mapping each lane line marking point indicated by the processed initial point marking data to a road grid in the historical road image; For each of the road grids, when at least one lane line marking point is mapped to the road grid, generating the anchor point in the road grid according to the at least one lane line marking point mapped to the road grid; The anchor point annotation data of the historical road image is generated according to the anchor point in at least one of the road grids of the historical road image.

7. A lane line detection device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the lane line detection method as described in any one of claims 1 to 6.

8. An electronic device comprising the lane line detection device according to claim 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lane line detection method described in any one of claims 1 to 6 is implemented.

10. A vehicle, characterized in that: The vehicle includes the electronic device of claim 8 , or includes the non-transitory computer-readable storage medium of claim 9 .

Images