Lane line detection method, storage medium, electronic equipment and vehicle

Abstract

The invention relates to a lane line detection method, a storage medium, electronic equipment and a vehicle. The lane line detection method comprises the steps of inputting a to-be-detected road image to a lane line detection model in response to input of the to-be-detected road image, so that the lane line detection model outputs lane line detection information of the to-be-detected road image; wherein the lane line detection model is obtained by training according to the first visual angle feature and the second visual angle feature. According to the method, the lane line detection model is constructed by combining the two visual angles, and the two visual angles complement each other to ensure that the lane line detection model achieves higher detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a lane line detection method, a storage medium, an electronic device and a vehicle. Background Art

[0002] With the development of intelligent vehicle technology, lane detection has become an important part of technical fields such as assisted driving and autonomous driving. For example, in assisted driving, lane keeping and lane deviation warning based on lane detection can effectively share the driver's work and reduce the driver's workload, thereby effectively reducing the occurrence of traffic accidents. At present, traditional pixel features such as edge detection are used to extract lane features, detect the position of lanes and perform curve fitting, but this method has low detection accuracy. Summary of the invention

[0003] The embodiments of the present application provide a lane line detection method, a storage medium, an electronic device and a vehicle, which use two perspectives in combination to construct a lane line detection model, which complement each other to ensure that the lane line detection model achieves higher detection accuracy, so as to at least partially solve the above-mentioned technical problems.

[0004] To achieve the above-mentioned purpose, according to the first aspect of the present application, a lane line detection method is provided, comprising: in response to the input of a road image to be tested, inputting the road image to be tested into a lane line detection model, so that the lane line detection model outputs lane line detection information of the road image to be tested; wherein, the lane line detection model is trained based on a first perspective feature and a second perspective feature.

[0005] Optionally, the method further includes: providing the lane line detection model; wherein, providing the lane line detection model includes: acquiring a convergence condition of the lane line detection model based on the first viewing angle feature and the second viewing angle feature.

[0006] Optionally, obtaining the convergence condition of the lane line detection model based on the first perspective feature and the second perspective feature includes: processing the front road image through the first detection branch in the lane line detection model to obtain first lane line detection information of the front road image; wherein the first detection branch is used to obtain the first perspective feature from the front road image, and perform lane line detection based on the first perspective feature; processing the front road image through the second detection branch in the lane line detection model to obtain second lane line detection information of the front road image; wherein the second detection branch is used to obtain the second perspective feature from the front road image, and perform lane line detection based on the second perspective feature; determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the front road image.

[0007] Optionally, the processing of the forward road image by the first detection branch in the lane line detection model to obtain first lane line detection information of the forward road image includes: extracting the first viewing angle feature in the forward road image by the feature extraction module in the first detection branch; processing the first viewing angle feature by the first detection module in the first detection branch to obtain the first lane line detection information.

[0008] Optionally, the forward road image is processed by the second detection branch in the lane line detection model to obtain second lane line detection information of the forward road image, including: processing the first viewing angle feature by the spatial conversion module in the second detection branch to obtain the second viewing angle feature; determining the values ​​of each parameter in the lane line representation preset under the second viewing angle according to the second viewing angle feature by the parameter estimation module in the second detection branch; processing the second viewing angle annotation information according to the lane line representation by the second detection module in the second detection branch to obtain third lane line detection information; processing the third lane line detection information by the spatial conversion module in the second detection branch to obtain the second lane line detection information.

[0009] Optionally, determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the forward road image includes: determining the intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model based on the first lane line detection information and the lane line annotation information; determining the detection loss value of the lane line detection model based on the intersection loss value and the regression loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0010] Optionally, the first lane line detection information includes at least one lane line and a lane line prediction probability for each of the lane lines; determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line labeling information of the forward road image includes: determining an intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining a regression loss value of the lane line detection model based on the first lane line detection information and the lane line labeling information; determining a classification loss value of the lane line detection model based on the lane line prediction probability of each of the lane lines in at least one of the lane lines and the lane line label of each of the lane lines; determining a detection loss value of the lane line detection model based on the intersection loss value, the regression loss value and the classification loss value; wherein the convergence condition includes convergence of the detection loss value of the lane line detection model.

[0011] Optionally, determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the forward road image includes: determining the intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model based on the first lane line detection information and the lane line annotation information; processing the forward road image through the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the forward road image; determining the segmentation loss value of the lane line detection model based on the lane line detection mask and the lane line annotation mask of the forward road image; determining the detection loss value of the lane line detection model based on the intersection loss value, the regression loss value and the segmentation loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0012] Optionally, the first lane line detection information includes at least one lane line and a lane line prediction probability of each lane line; the determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information and the lane line annotation information of the front view road image includes: determining a crossing loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; determining a regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; determining a lane line prediction probability of each lane line in at least one lane line according to the lane line detection information of each lane line in the front view road image; determining a lane line prediction probability of each lane line in the front view road image according to the first lane line detection information and the lane line annotation information ... The lane line prediction probability and the lane line label of each lane line are used to determine the classification loss value of the lane line detection model; the front view road image is processed by the feature enhancement module of the lane line detection model to obtain the lane line detection mask of the front view road image; the segmentation loss value of the lane line detection model is determined according to the lane line detection mask and the lane line annotation mask of the front view road image; the detection loss value of the lane line detection model is determined according to the intersection loss value, the regression loss value, the classification loss value and the segmentation loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0013] Optionally, providing the lane line detection model includes: performing data preprocessing on an initial forward-view road image to obtain the preprocessed forward-view road image; wherein the data preprocessing includes resolution adjustment processing and / or maximum value normalization processing; processing the preprocessed forward-view road image through the lane line detection model to obtain the first viewing angle feature and the second viewing angle feature; and acquiring the convergence condition of the lane line detection model based on the first viewing angle feature and the second viewing angle feature.

[0014] According to a second aspect of the present application, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the lane line detection method described in the first aspect is implemented.

[0015] According to a third aspect of the present application, an electronic device is provided, comprising: a memory on which a computer program is stored; and a processor for executing the computer program in the memory to implement the lane line detection method described in the first aspect above.

[0016] According to a fourth aspect of the present application, a vehicle is provided, comprising the electronic device as described above.

[0017] Some embodiments of the present application may produce the following specific beneficial effects:

[0018] The lane line detection method provided in the embodiment of the present application obtains lane line detection information of the road image to be tested through a lane line detection model, and the lane line detection model is constructed by combining two perspectives, and the features under the two perspectives complement each other, which can ensure that the lane line detection model achieves higher detection accuracy. For example, the two perspectives include a front view perspective and a top view perspective. Since the road image under the front view perspective directly reflects the actual scene of the road in front of the vehicle and usually has a high resolution, the lane line features under the front view perspective have directness and high resolution; and since the lane lines in the road image under the top view perspective are usually parallel or crossed lines, they can clearly reflect the topological structure of the road, so that the lane line features under the top view perspective have geometric characteristics. When constructing the loss function of the lane line detection model, the embodiment of the present application integrates the detection results under the front view perspective and the detection results under the top view perspective. By cross-validating the detection results under the front view perspective and the top view perspective, the detection advantages under the two perspectives complement each other. During model training, joint training is achieved through the constructed loss function, which greatly improves the detection accuracy of the lane line detection model.

[0019] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0021] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0022] Figure 1 is a flow chart of a lane line detection method provided in an embodiment of the present application;

[0023] Figure 2 is a flow chart of a lane detection model training method provided in an embodiment of the present application;

[0024] Figure 3 It is a structural schematic diagram of a lane detection model provided in an embodiment of the present application;

[0025] Figure 4 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0027] At present, most lane line detection methods simply select the front-view road image, or convert it to the top-view road image, use traditional pixel features such as edge detection, or learnable neural network models to extract features, detect the position of the lane line and perform curve fitting, and finally use the curve parameters in the subsequent planning decision module.

[0028] However, the same lane line is viewed from different spaces (perspective space and Euclidean space) based on the front view perspective or the top view perspective, so the features of the lane line will be inconsistent, and the corresponding detection methods also have their own advantages and disadvantages.

[0029] As an introduction to the embodiments of this application, the following two lane detection methods and their advantages and disadvantages are introduced:

[0030] Lane line detection method based on the front view perspective: Since the lane line coordinate points are identified based on the acquired precise image information by directly using the front high-resolution camera to collect data, it can often achieve high-accuracy spatial positioning; at the same time, the perception scheme based on the front view can obtain the road condition information ahead in real time without the need for space conversion and other operations, which can meet higher performance requirements. However, the lane line in the perspective space cannot intuitively represent the spatial relationship and relative position between the current vehicle and the surrounding lane lines.

[0031] Lane line detection method based on top view perspective: It is often necessary to project the front view into the BEV (Bird-eye-view) view first. At this time, the lane lines are presented in a state closer to the real scene (most lane lines are approximately straight lines, and adjacent lane lines are approximately parallel to each other). At this time, a second-order or third-order polynomial can be used to characterize the lane lines well.

[0032] The IPM (Inverse perspective mapping) operation is usually used to complete the projection from the front view to the BEV view. However, the execution of the IPM operation needs to meet two inherent assumptions: 1. It is on the ground plane; 2. There is no relative motion between the camera and the ground, that is, there are stable camera parameters. In actual driving scenarios, due to uphill and downhill or bumpy conditions, the vehicle is not in an absolute plane, and the camera may cause unstable parameters as the vehicle moves, which will not meet the ideal assumptions of the IPM operation. Therefore, the use of a fixed transformation matrix may lead to inaccurate subsequent detection. At the same time, this approach is usually used as a post-processing part of the lane detection model output, and cannot provide a good guide to improve the problem of abnormal points in the lane detection model.

[0033] In view of this, the embodiments of the present application provide a lane line detection method, a storage medium, an electronic device and a vehicle, which can effectively alleviate the problems existing in the lane line detection methods under the above two perspectives. The embodiments of the present application use a combination of two perspectives to construct a lane line detection model, and cross-validate the detection results under the two perspectives, complementing each other to ensure that the lane line detection model achieves higher detection accuracy. In addition, in order to better capture the spatial details and semantic structure of the lane line at the same time, the embodiments of the present application use point detection and curve fitting methods in parallel to detect lanes. In addition, in order to avoid the errors caused by using a fixed spatial transformation matrix, the embodiments of the present application use a network model to predict the spatial transformation matrix to replace the traditional calibration calculation spatial transformation matrix.

[0034] According to the first aspect of the present application, an embodiment of the present application provides a lane line detection method.

[0035] See also Figure 1 , Figure 1 : is a flow chart of a lane line detection method provided in an embodiment of the present application. The lane line detection method may include the following steps:

[0036] Step S001: In response to input of a road image to be tested, the road image to be tested is input into a lane line detection model, so that the lane line detection model outputs lane line detection information of the road image to be tested; wherein the lane line detection model is trained based on a first perspective feature and a second perspective feature. wherein the lane line detection information includes a set of coordinate points of at least one lane line.

[0037] Optionally, the lane line detection method further includes step S002: providing a lane line detection model. Step S002 includes: acquiring a convergence condition of the lane line detection model according to the first viewing angle feature and the second viewing angle feature.

[0038] Optionally, the above-mentioned acquisition of the convergence condition of the lane line detection model based on the first perspective feature and the second perspective feature includes: processing the front road image through the first detection branch in the lane line detection model to obtain first lane line detection information of the front road image; wherein the first detection branch is used to obtain the first perspective feature from the front road image, and perform lane line detection based on the first perspective feature; processing the front road image through the second detection branch in the lane line detection model to obtain second lane line detection information of the front road image; wherein the second detection branch is used to obtain the second perspective feature from the front road image, and perform lane line detection based on the second perspective feature; and determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the front road image.

[0039] Optionally, the above-mentioned processing of the forward road image by the first detection branch in the lane line detection model to obtain first lane line detection information of the forward road image includes: extracting a first viewing angle feature in the forward road image by a feature extraction module in the first detection branch; processing the first viewing angle feature by the first detection module in the first detection branch to obtain the first lane line detection information.

[0040] Optionally, the above-mentioned processing of the forward road image by the second detection branch in the lane line detection model to obtain second lane line detection information of the forward road image includes: processing the first perspective feature by the spatial conversion module in the second detection branch to obtain the second perspective feature; determining the values ​​of each parameter in the preset lane line representation under the second perspective according to the second perspective feature by the parameter estimation module in the second detection branch; processing the second perspective annotation information according to the lane line representation by the second detection module in the second detection branch to obtain third lane line detection information; processing the third lane line detection information by the spatial conversion module in the second detection branch to obtain the second lane line detection information.

[0041] Optionally, the above-mentioned determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the forward road image includes: determining the intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model based on the first lane line detection information and the lane line annotation information; determining the detection loss value of the lane line detection model based on the intersection loss value and the regression loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0042] Optionally, the first lane line detection information includes at least one lane line and a lane line prediction probability for each lane line; the above-mentioned determination of the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the forward road image includes: determining the intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model based on the first lane line detection information and the lane line annotation information; determining the classification loss value of the lane line detection model based on the lane line prediction probability of each lane line in at least one lane line and the lane line label of each lane line; determining the detection loss value of the lane line detection model based on the intersection loss value, the regression loss value and the classification loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0043] Optionally, the above-mentioned determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information and the lane line annotation information of the front road image includes: determining the intersection loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; processing the front road image by a feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front road image; determining the segmentation loss value of the lane line detection model according to the lane line detection mask and the lane line annotation mask of the front road image; determining the detection loss value of the lane line detection model according to the intersection loss value, the regression loss value and the segmentation loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

[0044] Optionally, the first lane line detection information includes at least one lane line and a lane line prediction probability for each lane line; the above-mentioned determining the convergence condition of the lane line detection model based on the first lane line detection information, the second lane line detection information and the lane line annotation information of the front road image includes: determining the intersection loss value of the lane line detection model based on the first lane line detection information and the second lane line detection information; determining the regression loss value of the lane line detection model based on the first lane line detection information and the lane line annotation information; determining the classification loss value of the lane line detection model based on the lane line prediction probability of each lane line in at least one lane line and the lane line label of each lane line; processing the front road image through the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front road image; determining the segmentation loss value of the lane line detection model based on the lane line detection mask and the lane line annotation mask of the front road image; determining the detection loss value of the lane line detection model based on the intersection loss value, the regression loss value, the classification loss value and the segmentation loss value.

[0045] Optionally, the above step S002 includes: performing data preprocessing on the initial forward road image to obtain a preprocessed forward road image; wherein the data preprocessing includes resolution adjustment processing and / or maximum value normalization processing; processing the preprocessed forward road image through a lane line detection model to obtain a first perspective feature and a second perspective feature; and obtaining a convergence condition of the lane line detection model based on the first perspective feature and the second perspective feature.

[0046] In summary, the lane line detection method provided by the embodiment of the present application obtains the lane line detection information of the road image to be tested through the lane line detection model, and the lane line detection model is constructed by combining two perspectives, and the features under the two perspectives complement each other, which can ensure that the lane line detection model achieves higher detection accuracy. For example, the two perspectives include the front view perspective and the top view perspective. Since the road image under the front view perspective directly reflects the actual scene of the road in front of the vehicle and usually has a high resolution, the lane line features under the front view perspective are direct and high resolution; and since the lane lines in the road image under the top view perspective are usually parallel or crossed lines, they can clearly reflect the topological structure of the road, so that the lane line features under the top view perspective have geometric characteristics. When constructing the loss function of the lane line detection model, the embodiment of the present application integrates the detection results under the front view perspective and the detection results under the top view perspective. By cross-validating the detection results under the front view perspective and the top view perspective, the detection advantages under the two perspectives complement each other. When training the model, joint training is achieved through the constructed loss function, which greatly improves the detection accuracy of the lane line detection model.

[0047] Below, the lane line detection method provided in the embodiment of the present application is introduced and explained by taking two perspectives as a front view perspective and a top view perspective, that is, the first perspective feature is a front view feature, the second perspective feature is a top view feature, the second perspective annotation information is the top view annotation information, and the third lane line detection information is the top view lane line detection information as an example. In the following embodiments, the training process and application process of the lane line detection model are introduced and explained respectively.

[0048] See also Figure 2 , Figure 2 : is a flow chart of a lane line detection model training method provided in an embodiment of the present application. The lane line detection model training method may include the following steps:

[0049] Step S110: Processing the front view road image through the first detection branch in the lane line detection model to obtain first lane line detection information of the front view road image; the first detection branch is used to perform lane line detection based on the front view features obtained from the front view road image;

[0050] Step S120: Processing the front view road image through the second detection branch in the lane line detection model to obtain second lane line detection information of the front view road image; the second detection branch is used to perform lane line detection based on the top view features obtained from the front view road image;

[0051] Step S130: determining a detection loss value of a lane line detection model according to the first lane line detection information, the second lane line detection information, and the lane line annotation information of the front view road image;

[0052] Step S140: Adjust the parameters of the lane line detection model so that the detection loss value converges to obtain a trained lane line detection model.

[0053] The front road image is a road image captured by a camera installed on the front side of the vehicle. The camera can be installed in the center of the front side of the vehicle, or it can be installed in a position slightly to the left or right, which is not limited in the embodiments of the present application. In the embodiments of the present application, at least one front road image can be used to train the lane line detection model, such as one, fifty, one hundred, two hundred, five hundred, one thousand, etc. The number of front road images used for training can be determined in combination with actual conditions. In the following embodiments, the lane line detection model training method is introduced by taking the processing of one of the front road images by the lane line detection model as an example. It should be understood that this does not constitute a limitation on the embodiments of the present application.

[0054] The first detection branch is the detection part of the lane line detection model corresponding to the front view perspective, which is used to perform lane line detection based on the front view features obtained from the front view road image. The front view road image is input into the first detection branch to obtain the first lane line detection information. Optionally, the first detection branch can first extract the front view features from the front view road image, and then perform lane line detection based on the front view features to obtain the first lane line detection information. For the specific content and execution steps of the first detection branch in the lane line detection model, please refer to the following embodiments, which will not be elaborated here.

[0055] The second detection branch is the detection part of the lane line detection model corresponding to the top view perspective, which is used to perform lane line detection based on the top view features obtained from the front road image. The front road image is input into the second detection branch to obtain the second lane line detection information. Optionally, the second detection branch may first convert the front road image into a top view road image and then extract the top view features from the top view road image, or may first extract the front view features from the front road image and then convert the front view features into top view features, and then perform lane line detection based on the top view features to obtain the top view lane line detection information, and then convert the top view lane line detection information into the second lane line detection information. For the specific content and execution steps of the second detection branch in the lane line detection model, please refer to the following embodiments, which will not be elaborated here.

[0056] In steps S130 and S140, a loss function of a lane line detection model is constructed based on the first lane line detection information obtained from the front view perspective, the second lane line detection information obtained from the top view perspective, and the lane line annotation information, and the parameters of the lane line detection model are adjusted according to the detection loss value of the loss function to achieve cross-validation of the detection results from the front view perspective and the top view perspective. Since the road image from the front view perspective directly reflects the actual scene of the road in front of the vehicle and usually has a high resolution, the lane line features from the front view perspective are direct and high resolution. And since the lane lines in the road image from the top view perspective are usually parallel or crossed lines, the topological structure of the road can be clearly reflected, so that the lane line features from the top view perspective have geometric characteristics. In the embodiment of the present application, the detection results from the front view perspective and the detection results from the top view perspective are integrated when constructing the loss function, so that the detection advantages from these two perspectives complement each other, and joint training is achieved through the constructed loss function during model training, which greatly improves the detection accuracy of the lane line detection model. For the construction and determination process of the loss function and its detection loss value, please refer to the following embodiments, which will not be elaborated here.

[0057] In some embodiments, the lane detection model training method provided in the embodiments of the present application may further include the following steps:

[0058] Step S010: performing data preprocessing on the initial forward-view road image to obtain a preprocessed forward-view road image.

[0059] Based on the data preprocessing of the forward road image, the embodiment of the present application trains the lane detection model according to the preprocessed forward road image, that is, the preprocessed forward road image may be used in steps S110 and S120.

[0060] The embodiment of the present application does not limit the specific method of data preprocessing, and optionally, the data preprocessing includes resolution adjustment processing and / or maximum value normalization processing. The resolution adjustment processing includes changing the resolution of the front view road image through an interpolation operation; the maximum value normalization processing includes dividing the pixel value of each pixel in the front view road image by the maximum value of the pixel values ​​of all pixels.

[0061] For example, data preprocessing includes resolution adjustment and maximum value normalization. The front road image is collected by the camera installed on the front side of the vehicle. Here, R represents the real number set, which includes all rational and irrational numbers. Here, it means that the front view road image belongs to the real number set and its size is h. org × org After obtaining the initial forward-looking road image After that, the resolution adjustment process can be performed first to crop the front view road image into Such as high h in is 360, width w in is 640; then, a maximum value normalization process is performed on the forward-view road image after the resolution adjustment process to obtain a processed forward-view road image.

[0062] Optionally, when performing data preprocessing on the front road image, or after performing data preprocessing on the front road image, data annotation processing may also be performed on the front road image to obtain lane line annotation information of the front road image. Optionally, the lane line annotation information may include a coordinate point set, and the coordinate point set is used to describe the location of the lane line in the front road image. For example, the lane line annotation information includes a coordinate point set in, Refers to the coordinates of the i-th marked coordinate point in the k-th lane line; refers to the x-axis coordinate of the marked coordinate point. Refers to the y-axis coordinate of the marked coordinate point; k is used to identify each lane line and k is a positive integer, and i is used to identify each marked coordinate point in the kth lane line and i is a positive integer. Optionally, the embodiment of the present application can construct a coordinate system with the upper left corner of the front-view road image as the coordinate origin, the width as the x-axis and the height as the y-axis to achieve the description of each marked coordinate point, but this does not constitute a limitation of the present application, and there may be other coordinate system construction methods in actual applications. Optionally, the lane line annotation information may also include a lane line annotation mask, that is, a lane line mask that is pre-annotated (such as manual annotation, etc.) before model training. Among them, the lane line mask can be a size consistent with the front-view road image, with a black background (pixel value of 0) and a white position where the lane line is located (pixel value of 255). Based on this, when the front-view road image is In the case of label The size is h in × in .

[0063] In summary, the lane line detection model training method provided in the embodiment of the present application performs lane line detection from the front view perspective through the first detection branch of the lane line detection model, and performs lane line detection from the top view perspective through the second detection branch of the lane line detection model; then, based on the first lane line detection information obtained from the front view perspective, the second lane line detection information obtained from the top view perspective, and the lane line annotation information, a loss function of the lane line detection model is constructed to adjust the parameters of the lane line detection model according to the detection loss value of the loss function. Since the road image from the front view perspective directly reflects the actual scene of the road in front of the vehicle and is usually at a high resolution, the lane line features from the front view perspective are direct and high resolution; and since the lane lines in the road image from the top view perspective are usually shown as parallel or intersecting lines, they can clearly reflect the topological structure of the road, so that the lane line features from the top view perspective have geometric characteristics. The embodiment of the present application integrates the detection results from the front view perspective and the detection results from the top view perspective when constructing the loss function. By cross-validating the detection results from the front view perspective and the top view perspective, the detection advantages of these two perspectives complement each other. During model training, joint training is achieved through the constructed loss function, which greatly improves the detection accuracy of the lane line detection model.

[0064] Furthermore, the lane line detection model training method provided in the embodiment of the present application performs data preprocessing such as resolution adjustment and / or maximum value normalization on the forward road image after acquiring the initial forward road image, and trains the lane line detection model based on the processed forward road image. Among them, the resolution adjustment process can improve the image resolution, and more accurate lane line features can be extracted from the forward road image after the resolution adjustment process; the maximum value normalization process can narrow the value range of the pixel value, which is more conducive to the calculation of the lane line detection model. The embodiment of the present application can improve the training accuracy and training efficiency of the lane line detection model through data preprocessing.

[0065] In some embodiments, the above step S110 may include the following steps:

[0066] Step S111: extracting front-view features in the front-view road image by using a feature extraction module in the first detection branch;

[0067] Step S112: Processing the front-view features by the first detection module in the first detection branch to obtain first lane line detection information.

[0068] That is, the first detection branch of the lane detection model includes a feature extraction module and a first detection module. Figure 3As shown, the feature extraction module 211 and the first detection module 212 in the lane detection model 200 form a first detection branch to perform lane detection in a front view perspective.

[0069] The front view road image is input into the feature extraction module, and the feature extraction module extracts the front view features; then the front view features output by the feature extraction module are input into the first detection module, and the first detection module performs detection processing and outputs the first lane line detection information.

[0070] Optionally, the feature extraction module may use a residual backbone neural network, so that the extracted forward-looking features may be represented in the form of a forward-looking feature graph F. The forward-looking features include low-level image features (such as edges / coordinates, colors, textures, etc.) and / or high-level image features (such as shapes, semantic features, etc.); the objects of the forward-looking features include lane lines, roads, various obstacles, etc.

[0071] Optionally, the first detection module may include multiple convolutional layers, such as 5 convolutional layers, and the first detection module may detect the coordinate point set of the kth lane line in the front view and the lane line prediction probability of each detected coordinate point. For example, the first lane line detection information includes the coordinate point set detected by the kth lane line N is a positive integer; (x k，i ,y k，i ) refers to the coordinates of the i-th detection coordinate point in the k-th lane line. Among them, the lane line prediction probability P of each detection coordinate point in the k-th lane line k，i It refers to the probability that the detected coordinate point belongs to the kth lane line.

[0072] In some embodiments, the above step S120 may include the following steps:

[0073] Step S121: Processing the front view feature by the space conversion module in the second detection branch to obtain the top view feature;

[0074] Step S122: determining the values ​​of various parameters in the preset lane line representation under the top-down perspective according to the top-down features through the parameter estimation module in the second detection branch;

[0075] Step S123: Processing the overhead marking information according to the lane line representation by the second detection module in the second detection branch to obtain overhead lane line detection information;

[0076] Step S124: Process the overhead lane line detection information through the space conversion module in the second detection branch to obtain second lane line detection information.

[0077] That is, the second detection branch of the lane line detection model includes a space conversion module, a parameter estimation module and a second detection module. Figure 3 As shown, the space conversion module 222, the parameter estimation module 223 and the second detection module 224 in the lane line detection model 200 constitute a second detection branch to perform lane line detection from a top view perspective.

[0078] The intermediate processing result (forward-view feature) of the first detection branch is input into the space conversion module, and the space conversion module converts the forward-view feature into the overhead view feature. The overhead view feature is then input into the lane line representation under the overhead view perspective preset in the parameter estimation module to determine the values ​​of each parameter in the lane line representation. The second detection module then processes the pre-annotated overhead annotation information according to the lane line representation to obtain the overhead lane line detection information. Finally, the space conversion module converts the overhead lane line detection information into the second lane line detection information (lane line detection information under the front view perspective).

[0079] Optionally, the spatial transformation module can obtain a spatial transformation matrix, and apply the spatial transformation matrix to the front-view feature to obtain the top-view feature. For example, the spatial transformation matrix is ​​H, the front-view feature is represented by the front-view feature map F, and the top-view feature is represented by the top-view feature map B, then B=HF. In the embodiment of the present application, the top-view feature can also be called a bird's-eye view feature, etc. The spatial transformation matrix can be estimated by a multi-scale fusion network, so that the second detection branch also includes a matrix estimation module. For example, Figure 3 As shown, the matrix estimation module 221 and the spatial conversion module 222, the parameter estimation module 223 and the second detection module 224 together constitute the second detection branch. The front view road image is input into the matrix estimation module, which estimates the spatial conversion matrix using the constructed multi-scale feature fusion network.

[0080] Optionally, the parameter estimation module is implemented by convolution and fully connected neural network. The parameter estimation module includes a preset lane line representation, which can be a third-order polynomial, such as f(x)=ax 2 +bx+c. In the parameter estimation module, the top-view feature is input into the lane line representation to determine the values ​​of each parameter in the lane line representation under each lane line. For example, the values ​​of each parameter under the kth lane line are polynomial parameters (a k , b k , c k ).

[0081] The second detection module processes the overhead annotation information according to the lane line representation output by the parameter estimation module to obtain the overhead lane line detection information. The overhead annotation information includes the coordinates of each annotation coordinate point on each lane line in the overhead road image. The overhead annotation information can be obtained by manual annotation before model training. The overhead road image corresponds to the front road image input by the lane line detection model, which is the road image of the same road scene under the overhead perspective and the front perspective respectively. The overhead annotation information is input into the lane line representation to obtain the overhead lane line detection information. For example, the overhead annotation information includes the y-axis coordinates of each annotation coordinate point, then the overhead annotation information includes Input it into the lane line representation f(x) = a k x 2 +b k x+c k , get the overhead lane line detection information

[0082] Then, the spatial transformation module is used again to perform an inverse coordinate transformation from the top-view lane line detection information to the second lane line detection information under the front-view perspective. That is, the inverse matrix of the above spatial transformation matrix is ​​applied to the top-view lane line detection information to obtain the second lane line detection information. For example, the top-view lane line detection information includes The second lane line detection information includes

[0083] To sum up, the lane line detection model training method provided in the embodiment of the present application combines the method of point detection and curve parameter estimation and fitting to realize lane line detection. Compared with a single lane line detection method, the embodiment of the present application takes into account both the spatial detail information (coordinate points) and the overall semantic structure of the lane line (lane line representation).

[0084] In addition, the lane line detection model training method provided in the embodiment of the present application adaptively learns the transformation matrix of the road image corresponding to different perspectives through a neural network, and is adapted to various road scenes. It solves the problem of incorrect lane line detection in a distorted space due to the inherent assumption that it is difficult to meet the requirements of IPM operation during actual vehicle driving.

[0085] In some embodiments, the above step S130 may include the following steps:

[0086] Step S13A1: determining a crossing loss value of a lane line detection model according to the first lane line detection information and the second lane line detection information;

[0087] Step S13A2: determining a regression loss value of a lane line detection model according to the first lane line detection information and the lane line marking information;

[0088] Step S13A3: Determine the detection loss value according to the cross loss value and the regression loss value.

[0089] In some embodiments, the above step S130 may include the following steps:

[0090] Step S13B1: determining a crossing loss value of a lane line detection model according to the first lane line detection information and the second lane line detection information;

[0091] Step S13B2: determining a regression loss value of a lane line detection model according to the first lane line detection information and the lane line marking information;

[0092] Step S13B3: determining a classification loss value of a lane line detection model according to a lane line prediction probability of each lane line in at least one lane line and a lane line label of each lane line;

[0093] Step S13B4: Determine the detection loss value according to the cross loss value, the regression loss value and the classification loss value.

[0094] In some embodiments, the above step S130 may include the following steps:

[0095] Step S13C1: determining a crossing loss value of a lane line detection model according to the first lane line detection information and the second lane line detection information;

[0096] Step S13C2: determining a regression loss value of a lane line detection model according to the first lane line detection information and the lane line marking information;

[0097] Step S13C3: Processing the front view road image by the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front view road image;

[0098] Step S13C4: determining a segmentation loss value of a lane detection model according to the lane detection mask and the lane annotation mask of the front view road image;

[0099] Step S13C5: Determine the detection loss value according to the cross loss value, the regression loss value and the segmentation loss value.

[0100] In some embodiments, the above step S130 may include the following steps:

[0101] Step S13D1: determining a crossing loss value of a lane line detection model according to the first lane line detection information and the second lane line detection information;

[0102] Step S13D2: determining a regression loss value of a lane line detection model according to the first lane line detection information and the lane line marking information;

[0103] Step S13D3: determining a classification loss value of a lane line detection model according to a lane line prediction probability of each lane line in at least one lane line and a lane line label of each lane line;

[0104] Step S13D4: Processing the front view road image through the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front view road image;

[0105] Step S13D5: determining a segmentation loss value of a lane detection model according to the lane detection mask and the lane annotation mask of the front view road image;

[0106] Step S13D6: Determine the detection loss value according to the cross loss value, regression loss value, classification loss value and segmentation loss value.

[0107] like Figure 3 As shown, the lane detection model 200 also includes a cross-validation module 240 , and the above steps S13A1 to S13A3 , the above steps S13B1 to S13B4 , the above steps S13C1 to S13C5 , and the above steps S13D1 to S13D5 can be performed by the cross-validation module 240 .

[0108] For example, the cross loss value is expressed as Loss bev , the regression loss value is expressed as Loss reg , the classification loss value is expressed as Loss cls , segmentation loss is expressed as Loss set , then based on the above steps S13D1 to S13D6, the detection loss value Loss can be determined total The calculation formula of Loss can be shown as follows: total =α1Loss reg +α2Loss bev +α3Loss cls +α4Loss seg

[0109] Among them, α1, α2, α3 and α4 are the weights of the regression loss value, the cross loss value, the classification loss value and the segmentation loss value respectively. The weights can be preset, and the sum of the weights of all loss values ​​in the calculation formula is equal to one.

[0110] Similarly, based on the above steps S13A1 to S13A3, the detection loss value Loss can be determined. total The calculation formula can be shown as follows:

[0111] Loss total =α1Loss reg +α2Lossbev

[0112] Among them, α1 and α2 are the weights of the regression loss value and the cross loss value respectively, the weights can be preset, and the sum of the weights of all loss values ​​in the calculation formula is equal to one.

[0113] Similarly, based on the above steps S13B1 to S13B4, the detection loss value Loss can be determined. total The calculation formula can be shown as follows:

[0114] Loss total =α1Loss reg +α2Loss bev +α3Loss cls

[0115] Among them, α1, α2 and α3 are the weights of the regression loss value, the cross loss value and the classification loss value respectively. The weights can be preset, and the sum of the weights of all loss values ​​in the calculation formula is equal to one.

[0116] Similarly, based on the above steps S13C1 to S13C5, the detection loss value Loss can be determined. total The calculation formula can be shown as follows:

[0117] Loss total =α1Loss reg ++α2Loss bev +α4Loss seg

[0118] Among them, α1, α2 and α4 are the weights of the regression loss value, the cross loss value and the segmentation loss value respectively. The weights can be preset, and the sum of the weights of all loss values ​​in the calculation formula is equal to one.

[0119] In the above steps S13A1, S13B1, S13C1 and S13D1, the first lane line detection information may include a set of coordinate points on at least one lane line detected by the lane line detection model in a front view perspective, and the second lane line detection information may include a set of coordinate points on at least one lane line detected by the lane line detection model in a top view perspective. Exemplarily, the first lane line detection information includes The second lane line detection information includes Then, the intersection loss value of the kth lane line Loss bev The calculation formula can be shown as follows:

[0120]

[0121] Wherein, ‖·‖1 represents the L1 norm, and N represents the number of detection coordinate points on the kth lane line. It should be understood that the intersection loss value of all lane lines in the front view road image is used in the calculation of the detection loss value. Optionally, the average of the intersection loss values ​​of all lane lines is used as the final intersection loss value.

[0122] In the above steps S13A2, S13B2, S13C2 and S13D2, the first lane line detection information may include a set of coordinate points on at least one lane line detected by the lane line detection model in the front view perspective, and the lane line annotation information may include a set of coordinate points on at least one lane line pre-annotated in the front view perspective. Exemplarily, the first lane line detection information includes Lane marking information includes Then, the regression loss value Loss of the kth lane line reg The calculation formula can be shown as follows:

[0123]

[0124] Wherein, ||·||1 represents the L1 norm, and N represents the number of detection coordinate points on the kth lane line. It should be understood that the regression loss values ​​of all lane lines in the front view road image are used in the calculation of the detection loss value. Optionally, the average of the regression loss values ​​of all lane lines is used as the final regression loss value.

[0125] In the above steps S13B3 and S13D3, the first lane line detection information also includes the lane line prediction probability of each lane line in at least one lane line, and the lane line label of each lane line can be manually marked. Among them, for each lane line, optionally, the average of the lane line prediction probabilities of all detection coordinate points on the lane line in the above embodiment can be used as the lane line prediction probability of the lane line. The lane line label can be a manually marked one-hot label, and for each lane line detected by the lane line detection model, it is marked as 1 when the lane line is a lane line, otherwise it is marked as 0. Optionally, the classification loss value can be calculated by a cross entropy function (Cross Entropy). Exemplarily, s k represents the one-hot encoding label of the k-th lane line, p k represents the lane line prediction probability of the kth lane line, M represents the number of lane lines, and the classification loss value Loss cls The calculation formula can be shown as follows:

[0126]

[0127] In the above steps S13C3 and S13D4, the lane line detection model may further include a feature enhancement module, such as Figure 3As shown, the lane line detection model 200 also includes a feature enhancement module 230. Optionally, the feature enhancement module can use a lightweight ENet segmentation network, and the front view road image is input into the feature enhancement module to obtain a lane line detection mask, which refers to the lane line mask obtained by the lane line detection model. Among them, the feature enhancement module can also output the lane line prediction probability of each pixel in the lane line detection mask, and the lane line prediction probability of each pixel refers to the probability that the pixel belongs to the lane line.

[0128] In the above steps S13C4 and S13D5, the segmentation loss value is determined according to the lane detection mask and the lane annotation mask. Optionally, the segmentation loss value can be calculated by a cross entropy function. Exemplarily, Z represents the number of pixels in the lane detection mask, t e represents the one-hot encoding label of the e-th pixel (the pixel is 1 if it is a lane line, and 0 if it is not a lane line), p e represents the lane line prediction probability of the e-th pixel, then the segmentation loss value Loss seg The calculation formula can be shown as follows:

[0129]

[0130] In the embodiment of the present application, the detection loss value obtained according to any of the above embodiments can be back-propagated using an Adam (Adaptive moment estimation, adaptive learning rate) optimizer to adjust the parameters of the lane line detection model to obtain a trained lane line detection model. When adjusting the parameters of the lane line detection model, it is necessary to make the detection loss value converge, for example, to make the detection loss value tend to a stable value.

[0131] In some embodiments, the embodiments of the present application also provide a method for using a lane line detection model that is compatible with the lane line detection model training method. The lane line detection method based on the above lane line detection model may include the following steps:

[0132] Step S150: performing data preprocessing on the road image to be tested to obtain a preprocessed road image to be tested;

[0133] Step S160: Processing the preprocessed road image to be tested by using the trained lane line detection model to obtain lane line detection information of the road image to be tested.

[0134] Before training the lane line detection model, if the forward road image is preprocessed, and then the lane line detection model is trained based on the preprocessed forward road image, when using the lane line detection model, the road image to be tested also needs to be preprocessed first. Among them, the road image to be tested is also a road image from the perspective of the front view, and the data preprocessing method for the road image to be tested needs to be consistent with the data preprocessing method for the forward road image. For an introduction to the data preprocessing method, please refer to the above embodiment, which will not be elaborated here.

[0135] The lane line detection information of the road image to be tested can be obtained by inputting the preprocessed road image to be tested into the trained lane line detection model. The lane line detection information includes a set of coordinate points of at least one lane line. In the embodiment of the present application, the lane line detection model includes a first detection branch under the perspective of the front view and a second detection branch under the perspective of the top view, and both detection branches output the lane line detection information of the front view road image. Therefore, in the use of the lane line detection model, the road image to be tested can be processed only based on the first detection branch under the perspective of the front view to obtain the lane line detection information of the road image to be tested (i.e., the first lane line detection information); the road image to be tested can also be processed only based on the second detection branch under the perspective of the top view to obtain the lane line detection information of the road image to be tested (i.e., the second lane line detection information); the road image to be tested can also be processed based on the first detection branch and the second detection branch at the same time, and then the detection results of the two detection branches are combined to obtain the lane line detection information of the road image to be tested, and the embodiment of the present application does not make specific limitations on this.

[0136] In summary, the lane line detection method provided in the embodiment of the present application processes the road image to be detected by the trained lane line detection model to obtain lane line detection information. Since the lane line detection model integrates the directness and high resolution of the lane line features under the front view perspective and the geometric features of the lane line features under the top view perspective during the training process, the detection advantages under the two perspectives complement each other, so that the trained lane line detection model can achieve more accurate lane line detection of the road image.

[0137] According to the second aspect of the present application, the embodiment of the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the lane line detection method described above are implemented. The non-transitory computer-readable storage medium has all the beneficial effects of the lane line detection method described above, and the present application will not repeat them here.

[0138] According to the third aspect of the present application, an embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein a computer program is stored in the memory; and the processor is used to execute the computer program in the memory to implement the steps of the above lane line detection method. The electronic device has all the beneficial effects of the above lane line detection method, which will not be described in detail in this application.

[0139] See also Figure 4 , the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0140] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0141] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present application are executed.

[0142] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above, which is not specifically limited in the present application. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to the input of the road image to be tested, inputs the road image to be tested into the lane line detection model, so that the lane line detection model outputs the lane line detection information of the road image to be tested; wherein the lane line detection model is trained based on the first perspective feature and the second perspective feature.

[0146] Computer program code for performing the operations of some embodiments of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function.

[0148] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.

[0149] For example, two boxes shown in succession may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] The units described in some embodiments of the present application may be implemented by software or by hardware.

[0151] In some embodiments, the described units may also be set in a processor, for example, it may be described as: a processor includes a detection module. The names of these units do not constitute limitations on the units themselves in some cases, for example, the detection module may also be described as "a unit for responding to an input of a road image to be tested, inputting the road image to be tested into a lane line detection model, so that the lane line detection model outputs lane line detection information of the road image to be tested".

[0152] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0153] According to a fourth aspect of the present application, an embodiment of the present application further provides a vehicle, including an electronic device, which can be used to perform the above lane line detection method. The vehicle has all the beneficial effects of the above electronic device, which will not be described in detail in this application.

[0154] The vehicle may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific limitation on this.

[0155] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0158] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of various embodiments in the embodiments of the present application have different focuses, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A lane line detection method, characterized in that: include: In response to input of a road image to be tested, inputting the road image to be tested into a lane line detection model, so that the lane line detection model outputs lane line detection information of the road image to be tested; The lane line detection model is trained based on the first perspective feature and the second perspective feature.

2. The lane line detection method according to claim 1, characterized in that: The method further comprises: Providing the lane detection model; Wherein, providing the lane line detection model includes: acquiring a convergence condition of the lane line detection model according to the first viewing angle feature and the second viewing angle feature.

3. The lane line detection method according to claim 2, characterized in that: The acquiring the convergence condition of the lane line detection model according to the first viewing angle feature and the second viewing angle feature includes: Processing the front road image through the first detection branch in the lane line detection model to obtain first lane line detection information of the front road image; wherein the first detection branch is used to obtain the first viewing angle feature from the front road image, and perform lane line detection according to the first viewing angle feature; Processing the front road image through the second detection branch in the lane line detection model to obtain second lane line detection information of the front road image; wherein the second detection branch is used to obtain the second viewing angle feature from the front road image, and perform lane line detection according to the second viewing angle feature; The convergence condition of the lane line detection model is determined according to the first lane line detection information, the second lane line detection information and the lane line annotation information of the front view road image.

4. The lane line detection method according to claim 3, characterized in that: The processing of the front view road image by the first detection branch in the lane line detection model to obtain first lane line detection information of the front view road image includes: Extracting the first viewing angle feature in the front road image by a feature extraction module in the first detection branch; The first viewing angle feature is processed by the first detection module in the first detection branch to obtain the first lane line detection information.

5. The lane line detection method according to claim 3, characterized in that: The processing of the front view road image by the second detection branch in the lane line detection model to obtain second lane line detection information of the front view road image includes: Processing the first viewing angle feature by a spatial conversion module in the second detection branch to obtain the second viewing angle feature; Determining, by the parameter estimation module in the second detection branch, the values ​​of various parameters in the preset lane line representation under the second viewing angle according to the second viewing angle feature; Processing the second perspective annotation information according to the lane line representation by the second detection module in the second detection branch to obtain third lane line detection information; The third lane line detection information is processed by the space conversion module in the second detection branch to obtain the second lane line detection information.

6. The lane line detection method according to claim 3, characterized in that: The determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information, and the lane line annotation information of the front road image includes: Determining a crossing loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; Determining a regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; The detection loss value of the lane line detection model is determined according to the intersection loss value and the regression loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

7. The lane line detection method according to claim 3, characterized in that: The first lane line detection information includes at least one lane line and a lane line prediction probability of each lane line; and determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information, and the lane line annotation information of the front road image includes: Determining a crossing loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; Determining a regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; Determining a classification loss value of the lane line detection model according to the lane line prediction probability of each of the lane lines in at least one of the lane lines and the lane line label of each of the lane lines; The detection loss value of the lane line detection model is determined according to the intersection loss value, the regression loss value and the classification loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

8. The lane line detection method according to claim 3, characterized in that: The determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information, and the lane line annotation information of the front road image includes: Determining a crossing loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; Determining a regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; Processing the front view road image by means of the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front view road image; Determining a segmentation loss value of the lane detection model according to the lane detection mask and the lane annotation mask of the front view road image; The detection loss value of the lane line detection model is determined according to the intersection loss value, the regression loss value and the segmentation loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

9. The lane line detection method according to claim 3, characterized in that: The first lane line detection information includes at least one lane line and a lane line prediction probability of each lane line; and determining the convergence condition of the lane line detection model according to the first lane line detection information, the second lane line detection information, and the lane line annotation information of the front road image includes: Determining a crossing loss value of the lane line detection model according to the first lane line detection information and the second lane line detection information; Determining a regression loss value of the lane line detection model according to the first lane line detection information and the lane line annotation information; Determining a classification loss value of the lane line detection model according to the lane line prediction probability of each of the lane lines in at least one of the lane lines and the lane line label of each of the lane lines; Processing the front view road image by means of the feature enhancement module of the lane line detection model to obtain a lane line detection mask of the front view road image; Determining a segmentation loss value of the lane detection model according to the lane detection mask and the lane annotation mask of the front view road image; The detection loss value of the lane line detection model is determined according to the intersection loss value, the regression loss value, the classification loss value and the segmentation loss value; wherein the convergence condition includes the convergence of the detection loss value of the lane line detection model.

10. The lane line detection method according to claim 2, characterized in that: Providing the lane line detection model includes: Performing data preprocessing on the initial forward-view road image to obtain the preprocessed forward-view road image; wherein the data preprocessing includes resolution adjustment processing and / or maximum value normalization processing; Processing the preprocessed front view road image by using the lane line detection model to obtain the first viewing angle feature and the second viewing angle feature; A convergence condition of the lane line detection model is obtained according to the first viewing angle feature and the second viewing angle feature.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lane line detection method described in any one of claims 1 to 10 is implemented.

12. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the lane line detection method described in any one of claims 1 to 10.

13. A vehicle, characterized in that: The vehicle includes the electronic device as claimed in claim 12.

Images