Lane detection method, device, electronic device and storage medium

Abstract

The present application relates to a lane line detection method, device, electronic device and storage medium, the method comprising: obtaining an image to be detected, the image to be detected including an area where the lane line is blocked and / or an area where the lane line is not blocked; inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, the first semantic prediction result including two-dimensional lane line information corresponding to the area where the lane line is blocked and / or two-dimensional lane line information corresponding to the area where the lane line is not blocked; inputting the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, the first three-dimensional lane line information including three-dimensional lane line information corresponding to the area where the lane line is blocked and / or three-dimensional lane line information corresponding to the area where the lane line is not blocked; determining the first three-dimensional lane line information as the lane line detection result of the image to be detected. In this way, complete three-dimensional lane line information can be obtained, which improves the lane line detection effect in the scene where the lane line is blocked.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a lane line detection method, device, electronic device and storage medium. Background Art

[0002] In the vehicle autonomous driving system, lane line detection helps the autonomous driving system better understand the surrounding environment, including the type, width, curvature of the road, etc., so as to make safer driving decisions and improve driving safety.

[0003] Lane line detection methods in related technologies include lane line detection methods at the image level and lane line detection methods at the bird's-eye view level. However, when these methods are used for lane line detection, for scenes where lane lines are blocked, the blocked lane lines cannot be detected or the detection effect is poor. Therefore, how to improve the detection effect of lane lines has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a lane line detection method, device, electronic device and storage medium to solve the problem in the prior art that in the scenario where the lane line is blocked, the blocked lane line cannot be detected or the detection effect is poor.

[0005] In a first aspect, the present application provides a lane line detection method, the method comprising:

[0006] Acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked;

[0007] Inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used to extract features and restore semantic information of the lane line obscured area and the lane line unobscured area, and the first semantic prediction result includes two-dimensional lane line information corresponding to the lane line obscured area and / or two-dimensional lane line information corresponding to the lane line unobscured area;

[0008] Inputting the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the first three-dimensional lane line information includes three-dimensional lane line information corresponding to an occluded area of ​​the lane line and / or three-dimensional lane line information corresponding to an unoccluded area of ​​the lane line;

[0009] The first three-dimensional lane line information is determined as the lane line detection result of the image to be detected.

[0010] Optionally, the semantic segmentation model includes an encoder and a decoder;

[0011] The step of inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result includes:

[0012] Inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected, wherein the feature information of the image to be detected includes feature information corresponding to the blocked area of ​​the lane line and / or feature information corresponding to the unblocked area of ​​the lane line;

[0013] The feature information of the image to be detected is input into the decoder for semantic prediction to obtain the first semantic prediction result.

[0014] Optionally, the encoder includes a convolution module and an aggregator module;

[0015] The step of inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected includes:

[0016] Inputting the image to be detected into the convolution module for performing a convolution operation to obtain intermediate feature information of the image to be detected;

[0017] The intermediate feature information of the image to be detected is input into the aggregator module for feature fusion operation to obtain the global lane line feature of the image to be detected, and the global lane line feature of the image to be detected is determined as the feature information of the image to be detected.

[0018] Optionally, the aggregator module includes a first aggregator submodule and a second aggregator submodule;

[0019] The step of inputting the intermediate feature information of the image to be detected into the aggregator module for performing feature fusion operation to obtain the global lane line feature of the image to be detected includes:

[0020] Inputting the intermediate feature information of the image to be detected into the first aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in a vertical direction to obtain a first feature fusion result, and inputting the first feature fusion result into the second aggregator submodule to perform a feature fusion operation on the first feature fusion result in a horizontal direction to obtain a global lane line feature of the image to be detected; or,

[0021] The intermediate feature information of the image to be detected is input into the second aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction to obtain a second feature fusion result, and the second feature fusion result is input into the first aggregator submodule to perform a feature fusion operation on the second feature fusion result in the vertical direction to obtain the global lane line feature of the image to be detected.

[0022] Optionally, before inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, the method further includes:

[0023] Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of which is a single-view image carrying a two-dimensional lane line true value label;

[0024] Inputting the first training samples in the first training sample set into the first model to be trained in sequence for semantic segmentation to obtain a segmentation prediction result and a second semantic prediction result, and inputting the first training samples in the first training sample set into the first model to be trained in sequence for occlusion probability estimation to obtain an occlusion probability estimation result, wherein the second semantic prediction result is obtained by normalizing the segmentation prediction result;

[0025] Based on the segmentation prediction result, the second semantic prediction result, and the occlusion probability estimation result, sequentially calculating a first loss value corresponding to each first training sample, wherein the first loss value is the sum of a second loss value and a third loss value, the second loss value is calculated based on the segmentation prediction result and the lane line true value label, and the third loss value is calculated based on the second semantic prediction result, the occlusion probability estimation result, and the lane line true value label;

[0026] When the first loss value is greater than the first preset threshold, continue to input the first training sample in the first training sample set into the first model to be trained for training until the calculated first loss value is less than or equal to the first preset threshold, stop training the first model to be trained, and obtain the semantic segmentation model.

[0027] Optionally, a calculation formula for the first loss value is as follows:

[0028] + ;

[0029] ;

[0030] + ;

[0031] = ;

[0032] = ;

[0033] in, represents the first loss value, represents the second loss value, represents the third loss value, and represents the weight coefficient, represents the segmentation prediction result, represents the lane line true value label, represents the cross entropy loss function, represents the weighted semantic prediction result, represents the weighted lane line true value label, represents the mean square error loss function, represents the second semantic prediction result, represents the occlusion probability estimation result, represents the mean of the weighted semantic prediction results, represents the mean of the lane line true value label, and Represents the weight coefficient.

[0034] Optionally, before inputting the first semantic prediction result into the three-dimensional occupancy grid model to obtain the first three-dimensional lane line information, the method further includes:

[0035] Acquire a second training sample set, wherein the second training sample set includes a plurality of second training samples, each of which is a multi-view image carrying a two-dimensional lane line true value label and a three-dimensional lane line true value label, and the three-dimensional lane line label is obtained by collecting and mapping point cloud data of the same target road section multiple times and then annotating;

[0036] Inputting the second training samples in the second training sample set into the second to-be-trained model in sequence to recover the three-dimensional lane line information, thereby obtaining second three-dimensional lane line information, wherein the second three-dimensional lane line information is used to represent information of a grid occupied by the lane line in three-dimensional space;

[0037] Based on the three-dimensional lane line true value label and the second three-dimensional lane line information, sequentially calculating a fourth loss value corresponding to each second training sample;

[0038] When the fourth loss value is greater than the second preset threshold, continue to input the second training samples in the second training sample set into the second model to be trained until the calculated fourth loss value is less than or equal to the second preset threshold, stop training the second model to be trained, and obtain the three-dimensional occupancy grid model.

[0039] In a second aspect, the present application further provides a lane line detection device, the device comprising:

[0040] A first acquisition module, used to acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked;

[0041] A semantic segmentation module, used for inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used for performing feature extraction and semantic information recovery on the lane line obscured area and the lane line unobscured area, and the first semantic prediction result includes two-dimensional lane line information corresponding to the lane line obscured area and / or two-dimensional lane line information corresponding to the lane line unobscured area;

[0042] an information recovery module, configured to input the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the first three-dimensional lane line information includes three-dimensional lane line information corresponding to an occluded area of ​​the lane line and / or three-dimensional lane line information corresponding to an unoccluded area of ​​the lane line;

[0043] A determination module is used to determine the first three-dimensional lane line information as the lane line detection result of the image to be detected.

[0044] In a third aspect, the present application further provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0045] Memory, used to store computer programs;

[0046] The processor is used to implement the lane line detection method described in any one of the first aspects when executing the program stored in the memory.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the lane line detection method described in any one of the first aspects is implemented.

[0048] The above-mentioned technical solution provided by the present application has the following advantages over the prior art: the method provided by the present application, by acquiring an image to be detected, wherein the image to be detected includes an area where the lane line is obscured and / or an area where the lane line is not obscured; inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used to perform feature extraction and semantic information recovery on the area where the lane line is obscured and the area where the lane line is not obscured, and the first semantic prediction result includes two-dimensional lane line information corresponding to the area where the lane line is obscured and / or two-dimensional lane line information corresponding to the area where the lane line is not obscured; inputting the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the first three-dimensional lane line information includes three-dimensional lane line information corresponding to the area where the lane line is obscured and / or three-dimensional lane line information corresponding to the area where the lane line is not obscured; and determining the first three-dimensional lane line information as the lane line detection result of the image to be detected. Through the above method, the semantic segmentation model can be used to first obtain the two-dimensional lane line information corresponding to the lane line obscured area and / or the two-dimensional lane line information corresponding to the lane line unobstructed area, and then the three-dimensional occupancy grid model can be used to restore the two-dimensional lane line information corresponding to the lane line obscured area and / or the two-dimensional lane line information corresponding to the lane line unobstructed area into the three-dimensional lane line information corresponding to the lane line obscured area and / or the three-dimensional lane line information corresponding to the lane line unobstructed area, so that accurate and complete three-dimensional lane line information can be obtained, thereby improving the lane line detection effect in the lane line obscured scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0052] Figure 1 A schematic diagram of a process flow of a lane line detection method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a prediction process of a semantic segmentation model provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of feature fusion of an aggregator module provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a training process of a semantic segmentation model provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a training process of a three-dimensional occupancy grid model provided in an embodiment of the present application;

[0057] Figure 6 A flowchart of another lane detection method provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a lane detection device provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0061] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0062] In order to solve the problem in the prior art that the obscured lane lines cannot be detected or the detection effect is poor in the scenario where the lane lines are obscured, the present application provides a lane line detection method, device, electronic device and storage medium, which can improve the lane line detection effect.

[0063] See also Figure 1 , Figure 1A schematic diagram of a lane line detection method provided in an embodiment of the present application. Figure 1 As shown, the lane line detection method may include the following steps:

[0064] Step 101: Acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked.

[0065] Specifically, the above-mentioned image to be detected refers to any single-view image that needs to perform lane line detection, and the image to be detected may include an area where the lane line is blocked and / or an area where the lane line is not blocked. It should be noted that when the image to be detected includes an area where the lane line is blocked, the two-dimensional lane line information of the blocked area can be first predicted to obtain the complete two-dimensional lane line information of the image to be detected, and then the complete three-dimensional lane line information is obtained based on the complete two-dimensional lane line information; when the image to be detected does not include an area where the lane line is blocked (that is, it already has complete two-dimensional lane line information), the complete three-dimensional lane line information can be directly obtained based on the complete two-dimensional lane line information. When acquiring the image to be detected, it can be directly acquired through a vehicle-mounted camera, or it can be acquired by other means, and the embodiments of the present application do not make specific limitations.

[0066] Step 102: input the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used to extract features and restore semantic information of lane line occluded areas and lane line unobstructed areas, and the first semantic prediction result includes two-dimensional lane line information corresponding to lane line occluded areas and / or two-dimensional lane line information corresponding to lane line unobstructed areas.

[0067] Specifically, the above-mentioned semantic segmentation model can be a fully convolutional network (FCN) model, a U-Net (i.e., an image semantic segmentation network based on a convolutional neural network) model, a DeepLab series model, a short-term dense concatenation (STDC) network model, or other models that can realize semantic segmentation.

[0068] In this step, after acquiring the image to be detected, the image to be detected can be input into the semantic segmentation model to obtain the two-dimensional lane line information corresponding to the occluded area of ​​the lane line and / or the two-dimensional lane line information corresponding to the unobstructed area of ​​the lane line, that is, the first semantic prediction result. It should be noted that since the semantic segmentation model can not only extract features and restore semantic information for the unobstructed area of ​​the lane line, but also extract features and restore semantic information for the occluded area of ​​the lane line. In other words, the semantic segmentation model can pay attention to the occluded lane line information, and thus can improve the detection rate of the two-dimensional lane line in the occluded scene.

[0069] Step 103: input the first semantic prediction result into the three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the first three-dimensional lane line information includes three-dimensional lane line information corresponding to the lane line obscured area and / or three-dimensional lane line information corresponding to the lane line unobstructed area.

[0070] Specifically, the three-dimensional occupancy grid model can be formed by dividing the environment space into a series of regular three-dimensional grids, each grid storing probability information about the position being occupied, thereby forming a probabilistic occupancy map. The three-dimensional occupancy grid model can be pre-trained.

[0071] In this step, after obtaining the first semantic prediction result, the first semantic prediction result can be input into the three-dimensional occupancy grid model to obtain the three-dimensional lane line information corresponding to the lane line obscured area and / or the three-dimensional lane line information corresponding to the lane line unobscured area, that is, the first three-dimensional lane line information. It should be noted that, since the first semantic prediction result includes the two-dimensional lane line information corresponding to the lane line obscured area and / or the two-dimensional lane line information corresponding to the lane line unobscured area, when the three-dimensional lane line information is restored using the three-dimensional occupancy grid model, the three-dimensional lane line information of the lane line obscured area and / or the lane line unobscured area can be restored at the same time, so as to obtain the three-dimensional lane line information of the lane line obscured area and / or the lane line unobscured area.

[0072] Step 104: determine the first three-dimensional lane line information as the lane line detection result of the image to be detected.

[0073] In this step, the first three-dimensional lane line information can be determined as the lane line detection result of the image to be detected. Compared with the two-dimensional lane line information, the three-dimensional lane line information here can effectively avoid the detection error caused by uneven terrain in driving scenes such as uphill and downhill driving, making the lane line detection result more in line with the actual situation, thereby improving the accuracy of the lane line detection result.

[0074] In this embodiment, the semantic segmentation model can be used to first obtain the two-dimensional lane line information corresponding to the lane line obscured area and / or the two-dimensional lane line information corresponding to the lane line unobstructed area, and then the three-dimensional occupancy grid model can be used to restore the two-dimensional lane line information corresponding to the lane line obscured area and / or the two-dimensional lane line information corresponding to the lane line unobstructed area into the three-dimensional lane line information corresponding to the lane line obscured area and / or the three-dimensional lane line information corresponding to the lane line unobstructed area. In this way, accurate and complete three-dimensional lane line information can be obtained, thereby improving the lane line detection effect in the lane line obscured scene.

[0075] Furthermore, the semantic segmentation model includes an encoder and a decoder; the above step 102, inputting the image to be detected into the semantic segmentation model to obtain a first semantic prediction result, includes:

[0076] Inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected, wherein the feature information of the image to be detected includes feature information corresponding to the area where the lane line is blocked and / or feature information corresponding to the area where the lane line is not blocked;

[0077] The feature information of the image to be detected is input into the decoder for semantic prediction to obtain a first semantic prediction result.

[0078] In one embodiment, the model structure of the semantic segmentation model may include an encoder and a decoder, wherein the encoder is mainly used to extract features of the lane line obscured area and / or the lane line unobscured area to obtain feature information corresponding to the lane line obscured area and / or feature information corresponding to the lane line unobscured area. The decoder is mainly used to recover and predict feature information corresponding to the lane line obscured area and / or feature information corresponding to the lane line unobscured area to obtain two-dimensional lane line information corresponding to the lane line obscured area and / or two-dimensional lane line information corresponding to the lane line unobscured area, that is, the first semantic prediction result. It should be noted that the internal structure of the above encoder and decoder can be set according to actual needs and is not specifically limited here.

[0079] In this embodiment, the semantic segmentation model can be used to accurately obtain the two-dimensional lane line information corresponding to the occluded area of ​​the lane line and / or the two-dimensional lane line information corresponding to the unobstructed area of ​​the lane line, that is, the global features of the lane line can be used to restore the two-dimensional lane line information of the occluded area.

[0080] Further, the encoder includes a convolution module and an aggregator module;

[0081] The image to be detected is input into the encoder for feature extraction to obtain feature information of the image to be detected, including:

[0082] Input the image to be detected into the convolution module for convolution operation to obtain the intermediate feature information of the image to be detected;

[0083] The intermediate feature information of the image to be detected is input into the aggregator module for feature fusion operation to obtain the global lane line features of the image to be detected, and the global lane line features of the image to be detected are determined as the feature information of the image to be detected.

[0084] In one embodiment, the encoder in the semantic segmentation model may include a convolution module and an aggregator module. In the process of inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected, the image to be detected may be first input into the convolution module for convolution operation to obtain multi-scale features, i.e., intermediate feature information of the image to be detected. Then, the intermediate feature information of the image to be detected is Input to the improved aggregator module (Recurrent Feature-Shift Aggregator for Lane Detection, referred to as RESA) for feature fusion operation to obtain the global lane line features of the image to be detected , and the global lane line features of the image to be detected It is determined as the feature information of the image to be detected. It should be noted that the aggregator module here can perform feature fusion operation, that is, traverse and fuse the features of each slice according to different intervals (also called spans). Specifically, the aggregator module can cyclically move the intermediate feature information of the image to be detected. The corresponding feature map slices are used to aggregate information vertically and horizontally, so that the features of each slice receive the features of another slice at a certain interval, and finally each position can collect information in the entire space. In addition, the decoder in the semantic segmentation model may include a 1×1 convolution module, a bilateral linear interpolation module, a first ReLU activation module, a transposed convolution module, a dimensionality reduction module, and a second ReLU activation module. The decoder can convert the global lane line features of the image to be detected obtained by the encoder into Perform 1x1 convolution, bilateral linear interpolation and ReLU activation operations to obtain the features , and the global lane line features of the image to be detected Perform transposed convolution, dimensionality reduction (i.e. Non-bottleNeck) and ReLU activation operations to obtain the features , then, the feature and Features By superimposing and performing SoftMax operation, we can get the first semantic prediction result. , the process is as follows Figure 2 shown.

[0085] Through the above method, the aggregator module can be used to extract the global features of the lane line, and then the global features of the lane line can be used to restore the lane line information of the occluded area to obtain the semantic prediction results of the occluded area and the unoccluded area of ​​the lane line.

[0086] Further, the aggregator module includes a first aggregator submodule and a second aggregator submodule;

[0087] The intermediate feature information of the image to be detected is input into the aggregator module for feature fusion operation to obtain the global lane line features of the image to be detected, including:

[0088] Inputting the intermediate feature information of the image to be detected into the first aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the vertical direction to obtain a first feature fusion result, and inputting the first feature fusion result into the second aggregator submodule to perform a feature fusion operation on the first feature fusion result in the horizontal direction to obtain the global lane line feature of the image to be detected; or,

[0089] The intermediate feature information of the image to be detected is input into the second aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction to obtain a second feature fusion result, and the second feature fusion result is input into the first aggregator submodule to perform a feature fusion operation on the second feature fusion result in the vertical direction to obtain the global lane line feature of the image to be detected.

[0090] In one embodiment, the aggregator module may include a first aggregator submodule RESA_U and a second aggregator submodule RESA_R, wherein the first aggregator submodule RESA_U may perform feature traversal and fusion in a vertical direction, and the second aggregator submodule RESA_R may perform feature traversal and fusion in a horizontal direction, such as Figure 3 shown.

[0091] As an optional implementation, when using the aggregator module to perform feature fusion operations to obtain the global lane line features of the image to be detected, the intermediate feature information of the image to be detected can be first input into the first aggregator submodule RESA_U to perform feature fusion operations on the intermediate feature information of the image to be detected in the vertical direction to obtain a first feature fusion result, and then the first feature fusion result is input into the second aggregator submodule RESA_U to perform feature fusion operations on the first feature fusion result in the horizontal direction to obtain the global lane line features of the image to be detected. Specifically, assuming that the intermediate feature information of the image to be detected Size , Taking the first aggregator submodule RESA_U as an example, the feature fusion operation from top to bottom can be Slice the image in the H direction to get H The feature slices of , assuming H is 24, the initial order of H slices from top to bottom is [0,1,2,3,...23], at this time, the convolution kernel of [1,H / 3] can be used for convolution. Different from the improved RESA module, considering the curvature and possible occlusion of the lane line, deformable convolution can be used here to extract more information. The first aggregator submodule RESA_U can iterate 3 times per interval. Suppose the feature to be processed for , then the steps of the first aggregator submodule RESA_U to perform feature traversal and fusion from top to bottom are as follows:

[0092] right For the first iteration with an interval of 1, The slice order is [1,2,3,...23,0] and each slice extracts features and adds them to The slice order is [0,1,2,3,...23], and the feature is obtained .

[0093] right Perform the second iteration with an interval of 1 and The slice order is [2,3,...23,0,1] and each slice extracts features and adds them to On each slice of the slice order [0,1,2,3,...23], we get the feature .

[0094] right Perform the third iteration with an interval of 1 and The slice order is [3,...23,0,1,2] and each slice extracts features and adds them to On each slice of the slice order [0,1,2,3,...23], we get the feature .

[0095] right Perform the first iteration with an interval of 2 and The slice order is [2,3,4,,...23,0,1], and the slice features are extracted and added to The slice order is [0,1,2,3,...23], and the feature is obtained .

[0096] right Perform the second iteration with an interval of 2 and The slice order is [4,...23,0,1,2,3] and the slice features are extracted and added to The slice order is [0,1,2,3,...23], and the feature is obtained .

[0097] right Perform the third iteration with an interval of 2 and The slice order is [6,...23,0,1,2,3,4,5] and the features are extracted and added to The slice order is [0,1,2,3,...23], and the feature is obtained .

[0098] In getting features After that, it can continue to be used as the input of the second aggregator submodule RESA_U to continue the feature fusion operation in the horizontal direction until the feature to be processed is The entire horizontal loop is traversed.

[0099] As another optional implementation, when using the aggregator module to perform a variable convolution operation to obtain the global lane line features of the image to be detected, the intermediate feature information of the image to be detected can be first input into the second aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction to obtain a second feature fusion result, and then the second feature fusion result is input into the first aggregator submodule to perform a feature fusion operation on the second feature fusion result in the vertical direction to obtain the global lane line features of the image to be detected. It should be noted that the difference between this implementation and the previous implementation is that this implementation first performs a feature fusion operation on the intermediate feature information of the image to be detected in the vertical direction, and then performs a feature fusion operation on the first feature fusion result in the horizontal direction; while the previous implementation first performs a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction, and then performs a feature fusion operation on the second feature fusion result in the vertical direction. The other implementation details of the two are basically the same and will not be repeated here.

[0100] In this embodiment, the first aggregator sub-module and the second aggregator sub-module can be used to extract the global features of the lane line from the vertical direction and the horizontal direction respectively, so as to use the global features of the lane line to restore the lane line information of the occluded area to obtain the semantic prediction results of the occluded area and the unoccluded area of ​​the lane line.

[0101] Furthermore, before the above step 102, inputting the image to be detected into the semantic segmentation model to obtain the first semantic prediction result, the method further includes:

[0102] Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of which is a single-view image carrying a two-dimensional lane line true value label;

[0103] Inputting the first training samples in the first training sample set into the first model to be trained in sequence for semantic segmentation to obtain a segmentation prediction result and a second semantic prediction result, and inputting the first training samples in the first training sample set into the first model to be trained in sequence for occlusion probability estimation to obtain an occlusion probability estimation result, wherein the second semantic prediction result is obtained by normalizing the segmentation prediction result;

[0104] Based on the segmentation prediction result, the second semantic prediction result and the occlusion probability estimation result, the first loss value corresponding to each first training sample is calculated in sequence, wherein the first loss value is the sum of the second loss value and the third loss value, the second loss value is calculated based on the segmentation prediction result and the lane line true value label, and the third loss value is calculated based on the second semantic prediction result, the occlusion probability estimation result and the lane line true value label;

[0105] When the first loss value is greater than the first preset threshold, continue to input the first training sample in the first training sample set into the first model to be trained until the calculated first loss value is less than or equal to the first preset threshold, stop training the first model to be trained, and obtain a semantic segmentation model.

[0106] In one embodiment, the first model to be trained includes a semantic branch and an occlusion probability estimation branch, wherein the semantic branch adopts an encoder-decoder structural design, and the specific structure of the encoder and decoder may be the same as the structure of the encoder and decoder in the semantic segmentation model introduced above, which will not be repeated here. The structure of the occlusion probability estimation branch may include a multi-layer convolution module, a batch normalization module, a maximum pooling module, a fully connected module, a ReLU activation module, a fully connected module and a Sigmoid activation module, etc., which mainly utilizes the multi-scale features during semantic branch encoding to calculate the probability of each lane line being occluded. Specifically, the multi-scale features can be scaled and superimposed, and then the superimposed features are sequentially subjected to multi-layer convolution, batch normalization, maximum pooling, full connection, ReLU activation, full connection and Sigmoid activation operations, and finally the occlusion probability estimation result is obtained, that is, the probability of each lane line being occluded in the H direction of the image. The process is as follows Figure 4 shown.

[0107] In the process of training the first model to be trained to obtain a semantic segmentation model, the first training sample set can be first obtained, and then the first training samples in the first training sample set are sequentially input into the first model to be trained for semantic segmentation to obtain segmentation prediction results and second semantic prediction results, and the first training samples in the first training sample set are sequentially input into the first model to be trained for occlusion probability estimation to obtain occlusion probability estimation results. Then, based on the segmentation prediction results, the second semantic prediction results and the occlusion probability estimation results, the first loss values ​​corresponding to each first training sample are calculated in turn. When the first loss value is greater than the first preset threshold, the first training samples in the first training sample set are continuously input into the first model to be trained for training until the calculated first loss value is less than or equal to the first preset threshold, and the first model to be trained is stopped to obtain a semantic segmentation model.

[0108] It should be noted that the first preset threshold here can be set according to the actual situation and is not specifically limited here. Since the first loss value is the sum of the second loss value and the third loss value, and the third loss value is mainly affected by the occlusion probability estimation result output by the occlusion probability estimation branch, when the actual occlusion probability is high and the prediction result of the first model to be trained does not correctly predict the lane line of the occluded part, the third loss value will be very high, which will guide the model to pay more attention to the occluded part during training, thereby improving the lane line detection effect in the occluded area.

[0109] Through the above method, the trained semantic segmentation model can pay more attention to the occluded part of the lane line, thereby improving the lane line detection effect in the occluded area.

[0110] Furthermore, the calculation formula of the first loss value is as follows:

[0111] + ;

[0112] ;

[0113] + ;

[0114] = ;

[0115] = ;

[0116] in, represents the first loss value, represents the second loss value, represents the third loss value, and represents the weight coefficient, represents the segmentation prediction result, represents the true value label of the lane line, represents the cross entropy loss function, represents the weighted semantic prediction result, represents the weighted lane line true value label, represents the mean square error loss function, represents the second semantic prediction result, represents the occlusion probability estimation result, represents the mean of the weighted semantic prediction results, represents the mean of the lane line true value label, and Represents the weight coefficient.

[0117] In one embodiment, according to the above formula, the second loss value The cross entropy loss between the segmentation prediction result and the lane line true value label is calculated. and It is used for balance and Parameters, the third loss value The calculation is the mean square error loss between the weighted semantic prediction result and the weighted lane line true value label, as well as a regularization term. In addition to calculating the error between the weighted semantic prediction result and the weighted lane line true value label, there is also a regularization term, which calculates the difference between the mean of the weighted semantic prediction result and the mean of the lane line true value label. It makes the output of the occlusion probability estimation branch tend to a percentage of the mean of the lane line true value label, making the output of the occlusion probability estimation branch not 0, but also plays a certain guiding role in training. The third loss value The loss is mainly affected by the occlusion probability estimation result output by the occlusion probability estimation branch When the actual occlusion probability is high and the prediction result of the first model to be trained does not correctly predict the lane line of the occluded part, the third loss value will be very high, which will guide the model to pay more attention to the occluded part during training, thereby improving the lane line detection effect in the occluded area.

[0118] Furthermore, before the above step 103, inputting the first semantic prediction result into the three-dimensional occupancy grid model to obtain the first three-dimensional lane line information, the method further includes:

[0119] Acquire a second training sample set, wherein the second training sample set includes a plurality of second training samples, each of which is a multi-view image carrying a two-dimensional lane line true value label and a three-dimensional lane line true value label, and the three-dimensional lane line label is obtained by collecting and mapping point cloud data of the same target road section multiple times and then annotating;

[0120] Inputting the second training samples in the second training sample set into the second to-be-trained model in sequence to perform three-dimensional lane line information recovery, thereby obtaining second three-dimensional lane line information, wherein the second three-dimensional lane line information is used to characterize information of a grid occupied by the lane line in three-dimensional space;

[0121] Based on the three-dimensional lane line true value label and the second three-dimensional lane line information, calculate the fourth loss value corresponding to each second training sample in sequence;

[0122] When the fourth loss value is greater than the second preset threshold, continue to input the second training sample in the second training sample set into the second model to be trained until the calculated fourth loss value is less than or equal to the second preset threshold, stop training the second model to be trained, and obtain a three-dimensional occupancy grid model.

[0123] In one embodiment, in the process of training the second model to be trained to obtain a three-dimensional occupancy grid model, a second training sample set can be first obtained, and then the second training samples in the second training sample set are sequentially input into the second model to be trained to recover the three-dimensional lane line information to obtain the second three-dimensional lane line information, and then based on the three-dimensional lane line true value label and the second three-dimensional lane line information, the fourth loss value corresponding to each second training sample is calculated in turn. When the fourth loss value is greater than the second preset threshold, the second training samples in the second training sample set continue to be input into the second model to be trained for training until the calculated fourth loss value is less than or equal to the second preset threshold, and the training of the second model to be trained is stopped to obtain the three-dimensional occupancy grid model.

[0124] It should be noted that when obtaining the second training sample set, it is necessary to obtain the three-dimensional lane line label of each second training sample. Since the lane line is occluded, both the image and the point cloud will be occluded. Therefore, a single acquisition cannot obtain the three-dimensional true value data of the lane line. For this, the same target road section can be collected and mapped multiple times to obtain the true value point cloud label of the three-dimensional lane line. For the road section where occlusion occurs, multiple data collections are performed to ensure that there are unobstructed moments during the collection process. The point cloud data collected multiple times are mapped, and the dense point cloud data in the corresponding mapping results is obtained according to the positioning of the vehicle at the moment of occlusion. Then, it is annotated and produced into label data according to the actual situation. Since the three-dimensional occupancy grid model only predicts three-dimensional lane line information, there are only lane line and non-lane line categories when performing three-dimensional grid prediction, and other categories do not need to be considered. Different from the original three-dimensional occupancy grid model (such as the SurroundOcc algorithm model, etc.), in the training stage, the input is a single camera image, a two-dimensional lane line label (including occlusion scenes) and a produced three-dimensional lane line label, and the output three-dimensional lane line occupancy grid information, such as Figure 5 As shown. In the model inference stage, the input is a single camera image and the predicted semantic prediction result, and the output is three-dimensional lane line information. The above second preset threshold can be set according to actual conditions and is not specifically limited here.

[0125] Through the above method, the trained three-dimensional occupancy grid model can quickly predict the occupancy information of the three-dimensional lane line from the two-dimensional lane line prediction results of the occlusion scene, thereby obtaining the three-dimensional lane line information under the occlusion scene.

[0126] In one embodiment, the lane line detection process provided in the embodiment of the present application is as follows: Figure 6 As shown, the following steps may be specifically included:

[0127] Step 601: Obtain single-view image data.

[0128] Step 602: Obtain the semantic prediction result of the lane line using the semantic segmentation model. The semantic segmentation model as a whole consists of a semantic branch and an occlusion probability estimation branch. In the model training phase, both the semantic branch and the occlusion probability estimation branch are used; in the model reasoning phase, the occlusion probability estimation branch does not need to be used, and only the semantic branch is used for result prediction, because the occlusion probability estimation branch has already affected the parameters of the model in the training phase, making the model pay more attention to the occluded area, and this can also ensure the accuracy and efficiency of model reasoning.

[0129] Step 603: Utilize multiple point cloud aggregation and annotation to obtain the true value labels of the three-dimensional lane lines, and train to obtain the three-dimensional occupancy grid model.

[0130] Step 604: Restore the three-dimensional lane line information using the three-dimensional occupancy grid model.

[0131] Step 605: Obtain three-dimensional lane line information in the occlusion scene.

[0132] It can be seen that the lane line detection method provided in the embodiment of the present application has the following beneficial effects:

[0133] Improve accuracy and stability. The proposed two-dimensional lane line detection method based on the improved RESA module and probability estimation branch can improve the detection rate of two-dimensional lane lines in occluded scenes, and further improve the detection accuracy and stability of two-dimensional lane lines.

[0134] Improve generalization and reliability. Based on the proposed two-dimensional lane line detection method and three-dimensional lane line information recovery method in occlusion scenes, it can be applied to a variety of scenarios and has good effects on both occlusion scenes and non-occlusion scenes.

[0135] Improve the quality of regulation, control and decision-making. Three-dimensional lane line recognition in occluded scenes can provide effective environmental information for regulation and control, which can improve the quality of regulation, control and decision-making.

[0136] See also Figure 7 , Figure 7 A schematic diagram of the structure of a lane line detection device provided in an embodiment of the present application. Figure 7 As shown, the lane line detection device 700 includes:

[0137] A first acquisition module 701 is used to acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked;

[0138] A semantic segmentation module 702 is used to input the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used to extract features and restore semantic information of the lane line occluded area and the lane line unoccluded area, and the first semantic prediction result includes two-dimensional lane line information corresponding to the lane line occluded area and / or two-dimensional lane line information corresponding to the lane line unoccluded area;

[0139] An information recovery module 703 is used to input the first semantic prediction result into the three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the first three-dimensional lane line information includes three-dimensional lane line information corresponding to the lane line blocked area and / or three-dimensional lane line information corresponding to the lane line unblocked area;

[0140] The determination module 704 is used to determine the first three-dimensional lane line information as the lane line detection result of the image to be detected.

[0141] Furthermore, the semantic segmentation model includes an encoder and a decoder; the semantic segmentation module 702 includes:

[0142] A feature extraction submodule is used to input the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected, wherein the feature information of the image to be detected includes feature information corresponding to the area where the lane line is blocked and / or feature information corresponding to the area where the lane line is not blocked;

[0143] The semantic prediction submodule is used to input feature information of the image to be detected into the decoder for semantic prediction to obtain a first semantic prediction result.

[0144] Further, the encoder includes a convolution module and an aggregator module; the feature extraction submodule includes:

[0145] The feature extraction unit is used to input the image to be detected into the convolution module for convolution operation to obtain the intermediate feature information of the image to be detected;

[0146] The convolution operation unit is used to input the intermediate feature information of the image to be detected into the aggregator module for feature fusion operation, obtain the global lane line feature of the image to be detected, and determine the global lane line feature of the image to be detected as the feature information of the image to be detected.

[0147] Further, the aggregator module includes a first aggregator submodule and a second aggregator submodule; and the convolution operation unit is specifically used for:

[0148] Inputting the intermediate feature information of the image to be detected into the first aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the vertical direction to obtain a first feature fusion result, and inputting the first feature fusion result into the second aggregator submodule to perform a feature fusion operation on the first feature fusion result in the horizontal direction to obtain the global lane line feature of the image to be detected; or,

[0149] The intermediate feature information of the image to be detected is input into the second aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction to obtain a second feature fusion result, and the second feature fusion result is input into the first aggregator submodule to perform a feature fusion operation on the second feature fusion result in the vertical direction to obtain the global lane line feature of the image to be detected.

[0150] Furthermore, the lane line detection device 700 further includes:

[0151] A second acquisition module is used to acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of which is a single-view image carrying a two-dimensional lane line true value label;

[0152] A first input module is used to sequentially input the first training samples in the first training sample set into the first model to be trained for semantic segmentation to obtain a segmentation prediction result and a second semantic prediction result, and sequentially input the first training samples in the first training sample set into the first model to be trained for occlusion probability estimation to obtain an occlusion probability estimation result, wherein the second semantic prediction result is obtained by normalizing the segmentation prediction result;

[0153] A first calculation module is used to calculate the first loss value corresponding to each first training sample in sequence based on the segmentation prediction result, the second semantic prediction result and the occlusion probability estimation result, wherein the first loss value is the sum of the second loss value and the third loss value, the second loss value is calculated based on the segmentation prediction result and the lane line true value label, and the third loss value is calculated based on the second semantic prediction result, the occlusion probability estimation result and the lane line true value label;

[0154] The first training module is used to continue inputting the first training sample in the first training sample set into the first model to be trained for training when the first loss value is greater than the first preset threshold, until the calculated first loss value is less than or equal to the first preset threshold, stop training the first model to be trained, and obtain a semantic segmentation model.

[0155] Furthermore, the calculation formula of the first loss value is as follows:

[0156] + ;

[0157] ;

[0158] + ;

[0159] = ;

[0160] = ;

[0161] in, represents the first loss value, represents the second loss value, represents the third loss value, and represents the weight coefficient, represents the segmentation prediction result, represents the true value label of the lane line, represents the cross entropy loss function, represents the weighted semantic prediction result, represents the weighted lane line true value label, represents the mean square error loss function, represents the second semantic prediction result, represents the occlusion probability estimation result, represents the mean of the weighted semantic prediction results, represents the mean of the lane line true value label, and Represents the weight coefficient.

[0162] Furthermore, the lane line detection device 700 further includes:

[0163] A third acquisition module is used to acquire a second training sample set, wherein the second training sample set includes a plurality of second training samples, each of which is a multi-view image carrying a two-dimensional lane line true value label and a three-dimensional lane line true value label, and the three-dimensional lane line label is obtained by collecting and mapping point cloud data of the same target road section multiple times and then annotating;

[0164] A second input module is used to sequentially input the second training samples in the second training sample set into the second to-be-trained model to perform three-dimensional lane line information recovery, thereby obtaining second three-dimensional lane line information, wherein the second three-dimensional lane line information is used to characterize information of a grid occupied by the lane line in three-dimensional space;

[0165] A second calculation module, used to calculate the fourth loss value corresponding to each second training sample in sequence based on the three-dimensional lane line true value label and the second three-dimensional lane line information;

[0166] The second training module is used to continue inputting the second training samples in the second training sample set into the second model to be trained for training when the fourth loss value is greater than the second preset threshold, until the calculated fourth loss value is less than or equal to the second preset threshold, stop training the second model to be trained, and obtain a three-dimensional occupancy grid model.

[0167] It should be noted that the lane line detection device 700 can implement the lane line detection method provided by any of the aforementioned method embodiments and can achieve the same technical effects, which will not be described one by one here.

[0168] like Figure 8 As shown, the embodiment of the present application further provides an electronic device, including a processor 811, a communication interface 812, a memory 813 and a communication bus 814, wherein the processor 811, the communication interface 812, and the memory 813 communicate with each other through the communication bus 814.

[0169] Memory 813, used for storing computer programs;

[0170] In one embodiment of the present application, the processor 811 is used to implement the lane line detection method provided by any of the aforementioned method embodiments when executing the program stored in the memory 813.

[0171] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the lane line detection method provided by any of the aforementioned method embodiments is implemented.

[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0174] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0175] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A lane line detection method, characterized in that: The method comprises: Acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked; Inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used to extract features and restore semantic information for the lane line obscured area and the lane line unobscured area, the semantic segmentation model includes an encoder and a decoder, the encoder includes a convolution module and an aggregator module, the aggregator module is used to perform a feature fusion operation to obtain the global lane line feature of the image to be detected, and the first semantic prediction result includes two-dimensional lane line information corresponding to the lane line obscured area and / or two-dimensional lane line information corresponding to the lane line unobscured area; Inputting the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the three-dimensional occupancy grid model is a probability occupancy map formed by dividing the environment space into a series of regular three-dimensional grids and using the probability information of each grid being occupied, and the first three-dimensional lane line information includes the three-dimensional lane line information corresponding to the blocked area of ​​the lane line and / or the three-dimensional lane line information corresponding to the unblocked area of ​​the lane line; The first three-dimensional lane line information is determined as the lane line detection result of the image to be detected.

2. The lane line detection method according to claim 1, characterized in that: The step of inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result includes: Inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected, wherein the aggregator module is used to perform feature fusion operation to obtain global lane line features of the image to be detected, and determine the global lane line features of the image to be detected as the feature information of the image to be detected, and the feature information of the image to be detected includes feature information corresponding to the blocked area of ​​the lane line and / or feature information corresponding to the unblocked area of ​​the lane line; The feature information of the image to be detected is input into the decoder for semantic prediction to obtain the first semantic prediction result.

3. The lane line detection method according to claim 2, characterized in that: The step of inputting the image to be detected into the encoder for feature extraction to obtain feature information of the image to be detected includes: Inputting the image to be detected into the convolution module for performing a convolution operation to obtain intermediate feature information of the image to be detected; The intermediate feature information of the image to be detected is input into the aggregator module for feature fusion operation to obtain the global lane line feature of the image to be detected, and the global lane line feature of the image to be detected is determined as the feature information of the image to be detected.

4. The lane line detection method according to claim 3, characterized in that: The aggregator module includes a first aggregator submodule and a second aggregator submodule; The step of inputting the intermediate feature information of the image to be detected into the aggregator module for performing feature fusion operation to obtain the global lane line feature of the image to be detected includes: Inputting the intermediate feature information of the image to be detected into the first aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in a vertical direction to obtain a first feature fusion result, and inputting the first feature fusion result into the second aggregator submodule to perform a feature fusion operation on the first feature fusion result in a horizontal direction to obtain a global lane line feature of the image to be detected; or, The intermediate feature information of the image to be detected is input into the second aggregator submodule to perform a feature fusion operation on the intermediate feature information of the image to be detected in the horizontal direction to obtain a second feature fusion result, and the second feature fusion result is input into the first aggregator submodule to perform a feature fusion operation on the second feature fusion result in the vertical direction to obtain the global lane line feature of the image to be detected.

5. The lane line detection method according to claim 1, characterized in that: Before inputting the image to be detected into the semantic segmentation model to obtain the first semantic prediction result, the method further includes: Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of which is a single-view image carrying a two-dimensional lane line true value label; Inputting the first training samples in the first training sample set into the first model to be trained in sequence for semantic segmentation to obtain a segmentation prediction result and a second semantic prediction result, and inputting the first training samples in the first training sample set into the first model to be trained in sequence for occlusion probability estimation to obtain an occlusion probability estimation result, wherein the second semantic prediction result is obtained by normalizing the segmentation prediction result; Based on the segmentation prediction result, the second semantic prediction result, and the occlusion probability estimation result, sequentially calculating a first loss value corresponding to each first training sample, wherein the first loss value is the sum of a second loss value and a third loss value, the second loss value is calculated based on the segmentation prediction result and the lane line true value label, and the third loss value is calculated based on the second semantic prediction result, the occlusion probability estimation result, and the lane line true value label; When the first loss value is greater than the first preset threshold, continue to input the first training sample in the first training sample set into the first model to be trained for training until the calculated first loss value is less than or equal to the first preset threshold, stop training the first model to be trained, and obtain the semantic segmentation model.

6. The lane line detection method according to claim 5, characterized in that: The calculation formula of the first loss value is as follows: + ； ； + ； = ； = ； in, represents the first loss value, represents the second loss value, represents the third loss value, and represents the weight coefficient, represents the segmentation prediction result, represents the lane line true value label, represents the cross entropy loss function, represents the weighted semantic prediction result, represents the weighted lane line true value label, represents the mean square error loss function, represents the second semantic prediction result, represents the occlusion probability estimation result, represents the mean of the weighted semantic prediction results, represents the mean of the lane line true value label, and Represents the weight coefficient.

7. The lane line detection method according to claim 1, characterized in that: Before inputting the first semantic prediction result into the three-dimensional occupancy grid model to obtain the first three-dimensional lane line information, the method further includes: Acquire a second training sample set, wherein the second training sample set includes a plurality of second training samples, each of which is a multi-view image carrying a two-dimensional lane line true value label and a three-dimensional lane line true value label, and the three-dimensional lane line true value label is obtained by collecting and mapping point cloud data of the same target road section multiple times and then annotating; Inputting the second training samples in the second training sample set into the second to-be-trained model in sequence to recover the three-dimensional lane line information, thereby obtaining second three-dimensional lane line information, wherein the second three-dimensional lane line information is used to represent information of a grid occupied by the lane line in three-dimensional space; Based on the three-dimensional lane line true value label and the second three-dimensional lane line information, sequentially calculating a fourth loss value corresponding to each second training sample; When the fourth loss value is greater than the second preset threshold, continue to input the second training samples in the second training sample set into the second model to be trained until the calculated fourth loss value is less than or equal to the second preset threshold, stop training the second model to be trained, and obtain the three-dimensional occupancy grid model.

8. A lane line detection device, characterized in that: The lane line detection device comprises: A first acquisition module, used to acquire an image to be detected, wherein the image to be detected includes an area where lane lines are blocked and / or an area where lane lines are not blocked; A semantic segmentation module, used for inputting the image to be detected into a semantic segmentation model to obtain a first semantic prediction result, wherein the semantic segmentation model is used for performing feature extraction and semantic information recovery on the lane line obscured area and the lane line unobscured area, the semantic segmentation model includes an encoder and a decoder, the encoder includes a convolution module and an aggregator module, the aggregator module is used for performing a feature fusion operation to obtain the global lane line feature of the image to be detected, and the first semantic prediction result includes two-dimensional lane line information corresponding to the lane line obscured area and / or two-dimensional lane line information corresponding to the lane line unobscured area; an information recovery module, configured to input the first semantic prediction result into a three-dimensional occupancy grid model to obtain first three-dimensional lane line information, wherein the three-dimensional occupancy grid model is a probability occupancy map formed by dividing the environment space into a series of regular three-dimensional grids and using the probability information of each grid being occupied, wherein the first three-dimensional lane line information includes the three-dimensional lane line information corresponding to the blocked area of ​​the lane line and / or the three-dimensional lane line information corresponding to the unblocked area of ​​the lane line; A determination module is used to determine the first three-dimensional lane line information as the lane line detection result of the image to be detected.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the lane line detection method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lane line detection method described in any one of claims 1 to 7 is implemented.

Images