Lane line detection method and device, vehicle and storage medium

Abstract

The invention relates to a lane line detection method and device, a vehicle and a storage medium. The method comprises the steps of obtaining a road image in front of a vehicle; lane line detection is conducted on the road image through a pre-trained lane line detection model, lane line information in the road image is obtained, and the lane line information comprises N lane lines and the prediction category of each lane line; the lane line detection model comprises a first task head used for generating a lane line starting point heat map and determining a prediction category of a lane line to which each lane line starting point belongs in the lane line starting point heat map, and a category output module in the first task head comprises a plurality of channels in one-to-one correspondence with a plurality of preset lane line categories; a parameter output module in the first task head comprises a plurality of channel groups in one-to-one correspondence with the plurality of lane line categories. While lane line detection in the road image is realized, the lane line type of each lane line is identified, so that different requirements of downstream tasks are met, and the automatic driving safety is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a lane line detection method, device, vehicle and storage medium. Background Art

[0002] Lane detection technology is a key technology in autonomous driving perception. In order to meet the requirements of downstream tasks such as rule control (for example, driving route planning, lane keeping, etc.) and high-precision positioning, lane detection technology often needs to identify all lane lines in the input image as much as possible. At present, lane detection technology can usually only identify lane lines in the input image, and cannot predict the lane line category. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a lane line detection method, device, vehicle and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a lane line detection method is provided, including:

[0005] Acquire a road image in front of the vehicle;

[0006] Performing lane line detection on the road image by using a pre-trained lane line detection model to obtain lane line information in the road image, wherein the lane line information includes N lane lines and a predicted category of each lane line, where N≥1;

[0007] The lane line detection model includes a feature extraction network, a first task head and a second task head;

[0008] The feature extraction network is used to extract features from the road image to obtain an intermediate feature map and a multi-scale feature map;

[0009] The first task head is connected to the feature extraction network, and is used to generate a lane line starting point heat map and dynamic convolution parameters according to the intermediate feature map, and determine the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map, wherein the category output module includes a plurality of channels corresponding to a plurality of preset lane line categories one by one, and the parameter output module includes a plurality of channel groups corresponding to the plurality of lane line categories one by one, the category output module is a module in the first task head for outputting the lane line starting point heat map, and the parameter output module is a module in the first task head for outputting the dynamic convolution parameters;

[0010] The second task head is connected to the feature extraction network and the first task head respectively, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

[0011] Optionally, the first task head is used to determine, for each lane line starting point in the lane line starting point heat map, a lane line category corresponding to a target channel as a predicted category of the lane line to which the lane line starting point belongs, wherein the target channel is the channel in the lane line starting point heat map where the lane line starting point is located.

[0012] Optionally, the second task head is used to dynamically convolve the target convolution parameter with the multi-scale feature map for each lane line starting point in the lane line starting point heat map to obtain the lane line where the lane line starting point is located, wherein the target convolution parameter is a dynamic convolution sub-parameter in the dynamic convolution parameter corresponding to the lane line starting point.

[0013] Optionally, when N>1, the method further includes:

[0014] The N lane lines are overlapped and eliminated to obtain a target lane line.

[0015] Optionally, eliminating overlaps of the N lane lines to obtain a target lane line includes:

[0016] Determine the lane lines to be traversed from the detection results, wherein the detection results are initially the N lane lines;

[0017] For each lane line other than the lane line to be traversed in the detection result, determining whether the other lane line overlaps with the lane line to be traversed; if the other lane line overlaps with the lane line to be traversed, removing the other lane line from the detection result;

[0018] Eliminate the lane line to be traversed from the detection result obtained after the elimination, and return to the step of determining the lane line to be traversed from the detection result until the number of lane lines included in the detection result obtained after the elimination is less than or equal to 2;

[0019] If the number of lane lines included in the detection result is less than or equal to 2, the detection result obtained after elimination and the lane line to be traversed determined in each cycle are determined as the target lane line.

[0020] Optionally, the determining whether the other lane line overlaps with the lane line to be traversed includes:

[0021] Determine the line distance between the other lane line and the lane line to be traversed;

[0022] If the line distance is less than a preset distance threshold, it is determined that the other lane line overlaps with the lane line to be traversed.

[0023] Optionally, each of the lane lines includes a plurality of feature points;

[0024] The determining the line distance between the other lane line and the lane line to be traversed includes:

[0025] For each target feature point in the lane line to be traversed, determine the point distance between the target feature point and a corresponding feature point in the other lane line, wherein the target feature point and the corresponding feature point in the other lane line are arranged in a horizontal direction;

[0026] The average value of each of the point distances is determined as the line distance between the other lane line and the lane line to be traversed.

[0027] Optionally, determining the lane line to be traversed from the detection result includes:

[0028] The lane line with the highest confidence in the detection result is determined as the lane line to be traversed.

[0029] According to a second aspect of an embodiment of the present disclosure, a lane line detection device is provided, comprising:

[0030] An image acquisition module is configured to acquire an image of a road in front of the vehicle;

[0031] The detection module is configured to perform lane line detection on the road image through a pre-trained lane line detection model to obtain lane line information in the road image, wherein the lane line information includes N lane lines and a predicted category of each lane line, where N≥1; wherein the lane line detection model includes a feature extraction network, a first task head, and a second task head; the feature extraction network is used to perform feature extraction on the road image to obtain an intermediate feature map and a multi-scale feature map; the first task head is connected to the feature extraction network to generate a lane line starting point heat map and dynamic convolution parameters according to the intermediate feature map, and determine the lane line starting point heat map and the ... obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate The predicted category of the lane line to which the starting point of each lane line belongs in the lane starting point heat map, wherein the category output module includes a plurality of channels corresponding one-to-one to a plurality of preset lane line categories, and the parameter output module includes a plurality of channel groups corresponding one-to-one to the plurality of lane line categories, the category output module is a module in the first task head for outputting the lane line starting point heat map, and the parameter output module is a module in the first task head for outputting the dynamic convolution parameters; the second task head is respectively connected to the feature extraction network and the first task head, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

[0032] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] Among them, the processor is configured to run the executable instructions to implement the steps of the lane line detection method provided in the first aspect of the present disclosure.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the lane line detection method provided in the first aspect of the present disclosure are implemented.

[0037] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: after acquiring the road image in front of the vehicle, firstly, the feature extraction network in the lane line detection model is used to extract the features of the road image to obtain the intermediate feature map and the multi-scale feature map; then, the first task head of the lane line detection model generates the lane line starting point heat map and the dynamic convolution parameter according to the intermediate feature map, and determines the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map; finally, the second task head of the lane line detection model generates N lane lines according to the dynamic convolution parameter and the multi-scale feature map. Among them, the category output module for outputting the lane line starting point heat map in the first task head includes a plurality of channels corresponding to a plurality of preset lane line categories, so that the category output module can output the lane line starting point sub-heat map corresponding to each channel, that is, the lane line starting point sub-heat map under each lane line category can be obtained, and then the lane line category of the lane line to which each lane line starting point in the lane line starting point sub-heat map belongs can be determined according to the channel corresponding to the lane line starting point sub-heat map. In this way, while realizing lane line detection in road images, the lane line category of each lane line can be identified, thereby meeting the different requirements of downstream tasks such as driving rule control and high-precision positioning, and improving the safety of autonomous driving.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0040] Figure 1 The figure is a flow chart of a lane line detection method according to an exemplary embodiment.

[0041] Figure 2 is a schematic diagram of a lane detection model according to an exemplary embodiment.

[0042] Figure 3 is a schematic diagram of a first task head according to an exemplary embodiment.

[0043] Figure 4 is a distribution diagram of target feature points and corresponding feature points according to an exemplary embodiment;

[0044] Figure 5 The figure is a block diagram of a lane line detection device according to an exemplary embodiment.

[0045] Figure 6 is a block diagram of a vehicle according to an exemplary embodiment.

[0046] Figure 7 The invention is a block diagram of a device for lane line detection according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0049] Figure 1 is a flow chart of a lane line detection method according to an exemplary embodiment, wherein the lane line detection method can be applied to a terminal, which can be, for example, a vehicle terminal (for example, a vehicle controller), a mobile device, etc. The lane line detection method can also be applied to a server, which can be, for example, a server, a server cluster composed of multiple servers, and a cloud computing service center, etc. Figure 1 As shown, the lane line detection method may include the following steps.

[0050] S101: Acquire a road image in front of the vehicle.

[0051] In one embodiment of the present disclosure, the image to be subjected to lane line detection is an image of the road in front of the vehicle, which can generally be a driving image of the vehicle, collected by devices such as sensors, cameras, and driving recorders.

[0052] S102: Perform lane line detection on the road image using a pre-trained lane line detection model to obtain lane line information in the road image, where the lane line information includes N lane lines and a predicted category for each lane line, where N≥1.

[0053] In the present disclosure, in order to improve the accuracy of lane line detection, after the original road image in front of the vehicle is collected, the original road image can be pre-processed such as denoising and equalization to obtain the road image to be detected.

[0054] The road image to be detected is input into the trained lane line detection model, and the output includes lane line information of multiple lane lines and the predicted category corresponding to each lane line, so as to complete the lane line detection of the road image.

[0055] Among them, Figure 2 As shown, the lane line detection model includes a feature extraction network, a first task head (Proposal head) and a second task head (Conditional shape head). The feature extraction network is used to extract features from the road image to obtain an intermediate feature map and a multi-scale feature map.

[0056] like Figure 2 As shown, the feature extraction network may specifically include a backbone network and a decoder. The backbone network is used to upsample the road image to obtain an intermediate feature map, and the backbone network may be, for example, a residual network (Residual Network, ResNet), a densely connected convolutional network (Densely Connected Convolutional Networks, DenseNet), a high-resolution network (High-Resolution Network, HRNet), a shifted window self-attention network (Shifted window Transformer, Swin transformer), etc.; the decoder is used to downsample the intermediate feature map to obtain a multi-scale feature map, and the decoder may be, for example, a feature pyramid network (feature pyramid networks, FPN), a feature pyramid network (FPN-LSS) using an LSS (Lift, Splat, Shoot) architecture, an improved feature pyramid network (PAFPN-LSS) using an LSS architecture, etc.

[0057] For example, the dimension of the road image is H×W×3, where H is the height and W is the width. The road image is input into the feature extraction network to obtain an intermediate feature map and a multi-scale feature map with dimensions of H×W×D, where D is the number of channels.

[0058] like Figure 2 , Figure 3As shown, the first task head is connected to the feature extraction network, and is used to generate a lane line starting point heatmap and a dynamic convolution parameter according to the intermediate feature map, and determine the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heatmap, wherein the category output module includes a plurality of channels corresponding to a plurality of preset lane line categories, and the category output module is a module in the first task head for outputting the lane line starting point heatmap.

[0059] The channel of the category output module in the first task head is set according to the lane line category to be predicted. If the preset number of lane line categories is C, the number of convolutional layer channels of the category output module is also set to C. For example, the lane line categories to be predicted include dotted lines and solid lines, that is, the preset multiple lane line categories include dotted lines and solid lines, then the category output module includes 2 channels, namely, a channel corresponding to the dotted line and a channel corresponding to the solid line.

[0060] The category output module for outputting the lane line starting point heat map in the first task head includes multiple channels corresponding to multiple preset lane line categories. In this way, the category output module can output the lane line starting point sub-heat map corresponding to each channel, that is, the lane line starting point sub-heat map under each lane line category can be obtained, and then the lane line category of the lane line to which each lane line starting point in the lane line starting point sub-heat map belongs can be determined according to the channel corresponding to the lane line starting point sub-heat map.

[0061] For example, Figure 3 As shown in FIG. 1 , the dimension of the intermediate feature map input to the first task head is H×W×D, and the lane line starting point heat map of dimension H×W×C (where C=2) is output through the category output module. The lane line starting point heat map of H×W×2 includes two lane line starting point sub-heat maps of H×W. Among them, the lane line sub-heat map where the lane line starting points n and o are located corresponds to the channel corresponding to the dotted line, and the lane line sub-heat map where the lane line starting points m and p are located corresponds to the channel corresponding to the solid line, that is, the lane line starting point sub-heat map under the dotted line category is the sub-heat map where the lane line starting points n and o are located, and the lane line starting point sub-heat map under the solid line category is the sub-heat map where the lane line starting points m and p are located. Therefore, the category of the lane line to which the lane line starting point n belongs and the category of the lane line to which the lane line starting point o belongs are both dotted lines, and the category of the lane line to which the lane line starting point m belongs and the category of the lane line to which the lane line starting point p belongs are both solid lines.

[0062] At the same time, the first task head also includes a parameter output module corresponding to the category output module, which is a module in the first task head for outputting dynamic convolution parameters (i.e., lane line starting point parameter map). The parameter output module includes multiple channel groups corresponding to multiple preset lane line categories, wherein the number of channels in each channel group is D, and the parameter output module includes D×C channels.

[0063] For example, Figure 3 As shown, the dimension of the intermediate feature map input to the first task head is H×W×D, and the parameter output module outputs a lane line starting point parameter map with a dimension of H×W×(D×C). The parameter output module includes a channel group corresponding to the dotted line and a channel group corresponding to the solid line.

[0064] The second task head is connected to the feature extraction network and the first task head respectively, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

[0065] The second task head uses the above dynamic convolution parameters and the multi-scale feature map to perform dynamic convolution (i.e., shape prediction) to generate N lane lines.

[0066] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: after acquiring the road image in front of the vehicle, firstly, the feature extraction network in the lane line detection model is used to extract the features of the road image to obtain the intermediate feature map and the multi-scale feature map; then, the first task head of the lane line detection model generates the lane line starting point heat map and the dynamic convolution parameter according to the intermediate feature map, and determines the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map; finally, the second task head of the lane line detection model generates N lane lines according to the dynamic convolution parameter and the multi-scale feature map. Among them, the category output module for outputting the lane line starting point heat map in the first task head includes a plurality of channels corresponding to a plurality of preset lane line categories, so that the category output module can output the lane line starting point sub-heat map corresponding to each channel, that is, the lane line starting point sub-heat map under each lane line category can be obtained, and then the lane line category of the lane line to which each lane line starting point in the lane line starting point sub-heat map belongs can be determined according to the channel corresponding to the lane line starting point sub-heat map. In this way, while realizing lane line detection in road images, the lane line category of each lane line can be identified, thereby meeting the different requirements of downstream tasks such as driving rule control and high-precision positioning, and improving the safety of autonomous driving.

[0067] In the present disclosure, before determining the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map, the first task head needs to identify the lane line starting point from the lane line starting point heat map. Specifically, the first task head can identify the lane line starting point in the following manner:

[0068] The lane line starting point heat map output by the category output module contains multiple pixel points, which are potential lane line starting points. According to a preset threshold θ of a certain dimension, the pixel points in the lane line starting point heat map whose dimension value is greater than the threshold θ are obtained. These pixel points greater than the threshold θ are the predicted lane line starting points, which can also be called recommended targets, where the dimension can be grayscale, etc.

[0069] As an optional implementation, the first task head is used to determine, for each lane line starting point in the lane line starting point heat map, the lane line category corresponding to the target channel as the predicted category of the lane line to which the lane line starting point belongs, wherein the target channel is the channel in the lane line starting point heat map where the lane line starting point is located.

[0070] In one embodiment of the present disclosure, the category output module for outputting the lane line starting point heat map in the first task head includes multiple channels corresponding to multiple preset lane line categories, so that the category output module can output the lane line starting point sub-heat map corresponding to each channel. The channel where the lane line starting point is located (i.e., the target channel) is the channel corresponding to the lane line starting point sub-heat map where the lane line is located. For each lane line starting point in the lane line starting point heat map, the lane line category corresponding to the target channel can be determined based on the correspondence between the channel and the lane line category, that is, the predicted category of the lane line to which the lane line starting point belongs is obtained.

[0071] For example, Figure 3 As shown, the lane line starting point heat map includes a lane line starting point sub-heat map output by a channel corresponding to a dotted line in a category output module, and a lane line starting point sub-heat map output by a channel corresponding to a solid line in a category output module. The target channels of lane line starting point n and lane line starting point o are channels corresponding to dotted lines, that is, the predicted category of lane lines to which lane line starting point n and lane line starting point o belong is a dotted line; the target channels of lane line starting point m and lane line starting point p are channels corresponding to a solid line, that is, the predicted category of lane lines to which lane line starting point m and lane line starting point p belong is a solid line.

[0072] In the present disclosure, the second task head can generate N lane lines in a variety of ways based on dynamic convolution parameters and multi-scale feature maps.

[0073] As an optional implementation, the second task head can directly perform dynamic convolution on the entire dynamic convolution parameters and the multi-scale feature map to obtain the N lane lines.

[0074] As an optional implementation, the second task head is used to dynamically convolve the target convolution parameter with the multi-scale feature map for each lane line starting point in the lane line starting point heat map to obtain the lane line where the lane line starting point is located, wherein the target convolution parameter is a dynamic convolution sub-parameter in the dynamic convolution parameter corresponding to the lane line starting point.

[0075] In one embodiment of the present disclosure, the second task head may dynamically convolve the parameter vectors (dynamic convolution sub-parameters) corresponding to the starting points of each lane line with the multi-scale feature map to generate lane lines corresponding to the starting points of each lane line.

[0076] For example, Figure 3 As shown, the dynamic convolution parameters include a parameter vector of the lane line starting point m, a parameter vector of the lane line starting point n, a parameter vector of the lane line starting point o, and a parameter vector of the lane line starting point p. The second task head can dynamically convolve the parameter vector of the lane line starting point m with the multi-scale feature map to obtain the lane line where the lane line starting point m is located. At the same time, the second task head can dynamically convolve the parameter vector of the lane line starting point n with the multi-scale feature map to obtain the lane line where the lane line starting point n is located, dynamically convolve the parameter vector of the lane line starting point o with the multi-scale feature map to obtain the lane line where the lane line starting point o is located, and dynamically convolve the parameter vector of the lane line starting point p with the multi-scale feature map to obtain the lane line where the lane line starting point p is located.

[0077] As an optional implementation, when N>1, the method further includes: eliminating overlaps of the N lane lines to obtain a target lane line.

[0078] In one embodiment of the present disclosure, when more than one lane line is obtained, the obtained lane lines may be subjected to overlap elimination, that is, overlapping lane lines may be removed to obtain a more accurate lane line detection result.

[0079] As an optional implementation manner, the step of eliminating overlaps of the N lane lines to obtain a target lane line may include the following steps:

[0080] (1) Determine the lane line to be traversed from the detection results.

[0081] The detection result is initially the N lane lines detected above, among which one lane line is determined as the lane line to be traversed. At this time, the lane lines in the detection result except the lane line to be traversed are other lane lines.

[0082] (2) For each lane line other than the lane line to be traversed in the detection result, determine whether the other lane line overlaps with the lane line to be traversed; if the other lane line overlaps with the lane line to be traversed, remove the other lane line from the detection result.

[0083] (3) Determine whether the number of lane lines contained in the detection result obtained after elimination is less than or equal to 2.

[0084] If the number of lane lines contained in the detection result obtained after elimination is greater than 2, it indicates that there are multiple lane lines other than the lane line to be traversed in the detection result after elimination. After that, it is necessary to eliminate the overlap of the multiple lane lines. At this time, the following step (4) can be performed, and then return to the above step (1); if the number of lane lines contained in the detection result obtained after elimination is less than or equal to 2, it indicates that there are 0 or one lane lines other than the lane line to be traversed in the detection result after elimination. At this time, there is no need to eliminate the overlap of the lane lines, and the following step (5) can be performed.

[0085] (4) Eliminate the lane line to be traversed from the detection results obtained after elimination.

[0086] (5) The detection result obtained after the elimination and the lane line to be traversed determined in each cycle are determined as the target lane line.

[0087] For example, the N lane lines include lane lines A, B, J, and K, that is, the detection results are initially lane lines A, B, J, and K.

[0088] The lane line to be traversed determined from the initial detection result is lane line A; then, among the three other lane lines B, J, and K, lane line B overlaps with lane line A to be traversed, and lane line J and lane line K do not overlap with lane line A to be traversed. Therefore, lane line B is removed from the detection result, that is, the detection result obtained after removal includes lane line A, lane line J, lane line K, and lane line K; at this time, the number of lane lines included in the detection result obtained after removal is 3, that is, greater than 2. At this time, lane line A to be traversed can be removed from the detection result obtained after removal to obtain a new detection result including lane lines J and K; then, from the new detection result, The re-determined lane line to be traversed is lane line J. At this time, it is only necessary to determine whether other lane lines K overlap with lane line J to be traversed; if other lane lines K do not overlap with lane line J to be traversed, other lane lines K are not eliminated. In this way, the detection result obtained after elimination includes lane line J and lane line K; at this time, the number of lane lines included in the detection result obtained after elimination is 2, that is, equal to 2. At this time, the detection result obtained after elimination (including lane lines J and K) and the lane lines to be traversed determined in each cycle (lane lines A and J) can be determined as the target lane lines, that is, the lane line detection results after overlapping elimination are lane lines A, J and K.

[0089] If other lane lines K overlap with the lane line J to be traversed, then other lane lines K are removed from the new detection result, so that the detection result obtained after removal includes lane line J; at this time, the number of lane lines included in the detection result obtained after removal is 1, that is, less than 2. At this time, the detection result obtained after removal (lane line J) and the lane lines to be traversed determined in each cycle (lane lines A and J) can be determined as the target lane lines, that is, the lane line detection result after overlapping elimination is lane lines A and J.

[0090] In the present disclosure, the lane line to be traversed may be determined from the detection results in a variety of ways.

[0091] As an optional implementation, determining the lane line to be traversed from the detection result includes: determining any lane line in the detection result as the lane line to be traversed.

[0092] As an optional implementation, determining the lane line to be traversed from the detection results includes: determining the lane line with the highest confidence in the detection results as the lane line to be traversed.

[0093] In one embodiment of the present disclosure, when lane line detection is performed using a lane line detection model, the confidence of each lane line can be obtained at the same time, and the lane line with the highest confidence can be determined as the lane line to be traversed. In this way, when lane line overlap elimination is performed, other lane lines with lower confidence that overlap with it are eliminated to retain the lane lines with higher confidence to be traversed, thereby improving the detection accuracy of the lane lines.

[0094] As an optional implementation manner, determining whether the other lane line overlaps with the lane line to be traversed includes:

[0095] Determine the line distance between the other lane line and the lane line to be traversed;

[0096] If the line distance is less than a preset distance threshold, it is determined that the other lane line overlaps with the lane line to be traversed.

[0097] In one embodiment of the present disclosure, first, the line distance between the other lane line and the lane line to be traversed is determined, and then the line distance is compared with a preset distance threshold to determine whether the other lane line overlaps with the lane line to be traversed. If the line distance between the other lane line and the lane line to be traversed is less than the preset distance threshold, for example, 20 mm, it indicates that the other lane line is close to the lane line to be traversed. At this time, it can be determined that the other lane line overlaps with the lane line to be traversed, that is, the other lane line is a falsely detected lane line; if the line distance between the other lane line and the lane line to be traversed is greater than or equal to the preset distance threshold, it indicates that the other lane line is far away from the lane line to be traversed, there is no false detection, and there is no need to eliminate overlap.

[0098] As an optional implementation, each of the lane lines includes a plurality of feature points;

[0099] At this time, determining the line distance between the other lane line and the lane line to be traversed includes:

[0100] For each target feature point in the lane line to be traversed, determine the point distance between the target feature point and a corresponding feature point in the other lane line, wherein the target feature point and the corresponding feature point in the other lane line are arranged in a horizontal direction;

[0101] The average value of each of the point distances is determined as the line distance between the other lane line and the lane line to be traversed.

[0102] In one embodiment of the present disclosure, each lane line includes a plurality of feature points. Since the lane lines in the road image in front of the vehicle are usually distributed longitudinally, the longitudinal intervals between two adjacent feature points on the same lane line are the same, and the number of feature points contained in lane lines of the same length is also the same. Each target feature point in the lane line to be traversed and the corresponding feature point in the other lane line are arranged in the horizontal direction, which means that the ordinate of a target feature point is the same as the ordinate of the corresponding feature point.

[0103] like Figure 4 As shown, the lane line A to be traversed includes feature points a1, a2, a3, a4, a5, a6, a7, a8, a9 and a 10, other lane lines B include feature points b1, b2, b3, b4, b5, b6, b7, b8, b9 and b 10 , where the feature point a3 has the same ordinate as the feature point b1 (i.e., the feature point a3 and the feature point b1 are arranged in the horizontal direction), the feature point a4 has the same ordinate as the feature point b2, the feature point a5 has the same ordinate as the feature point b3, the feature point a6 has the same ordinate as the feature point b4, the feature point a7 has the same ordinate as the feature point b5, the feature point a8 has the same ordinate as the feature point b6, the feature point a9 has the same ordinate as the feature point b7, and the feature point a 10 The ordinate is the same as that of feature point b8. Therefore, the feature points a3, a4, a5, a6, a7, a8, a9 and a 10 The feature point b1 of the other lane line B is the feature point corresponding to the target feature point a3, that is, the feature point b1 is the corresponding feature point of the target feature point a3, the feature point b2 is the corresponding feature point of the target feature point a4, the feature point b3 is the corresponding feature point of the target feature point a5, the feature point b4 is the corresponding feature point of the target feature point a6, the feature point b5 is the corresponding feature point of the target feature point a7, the feature point b6 is the corresponding feature point of the target feature point a8, the feature point b7 is the corresponding feature point of the target feature point a9, and the feature point b8 is the corresponding feature point of the target feature point a10. 10 Since there is no feature point with the same ordinate as feature points a1 and a2 among the feature points of other lane lines B, feature points a1 and a2 are non-target feature points, and feature points b9 and b 10 are non-corresponding feature points.

[0104] The point distance between each target feature point and its corresponding feature point is obtained, and the line distance between the lane line to be traversed and other lane lines is determined according to the average value of the distances between each point. Among them, since the ordinate of the target feature point and its corresponding feature point is the same, the point distance between the target feature point and its corresponding feature point is equal to the absolute value of the difference between the abscissa of the target feature point and its corresponding feature point.

[0105] For example, the target feature points in lane line A to be traversed are a1, a2, ..., a n , the coordinates of each target feature point are where i = 1, 2, ..., n; the corresponding feature points in other lane lines B are b1, b2, ..., b n , the coordinates of each corresponding feature point are At this time, the target feature point a i Its corresponding feature point b i The distance between the points is a i With b i The absolute value of the difference between the horizontal coordinates of It can be obtained that the line distance between the lane line to be traversed and other lane lines is the average value of the distances of each point, that is, the line distance between the lane line to be traversed A and other lane line B

[0106] Figure 5 is a block diagram of a lane line detection device according to an exemplary embodiment. Figure 5 The lane line detection device 200 includes an image acquisition module 201 and a detection module 202.

[0107] The image acquisition module 201 is configured to acquire a road image in front of the vehicle;

[0108] The detection module 202 is configured to perform lane line detection on the road image through a pre-trained lane line detection model to obtain lane line information in the road image, wherein the lane line information includes N lane lines and a predicted category of each lane line, where N≥1; wherein the lane line detection model includes a feature extraction network, a first task head, and a second task head; the feature extraction network is used to perform feature extraction on the road image to obtain an intermediate feature map and a multi-scale feature map; the first task head is connected to the feature extraction network to generate a lane line starting point heat map and dynamic convolution parameters according to the intermediate feature map, and determine the lane line starting point heat map and the ... obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters The predicted category of the lane line to which the starting point of each lane line belongs in the lane line starting point heat map, wherein the category output module includes a plurality of channels corresponding one-to-one to a plurality of preset lane line categories, and the parameter output module includes a plurality of channel groups corresponding one-to-one to the plurality of lane line categories, the category output module is a module in the first task head for outputting the lane line starting point heat map, and the parameter output module is a module in the first task head for outputting the dynamic convolution parameters; the second task head is respectively connected to the feature extraction network and the first task head, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

[0109] It is not difficult to find from the above description that after acquiring the road image in front of the vehicle, the present embodiment first extracts features from the road image through the feature extraction network in the lane line detection model to obtain an intermediate feature map and a multi-scale feature map; then, the first task head of the lane line detection model generates a lane line starting point heat map and a dynamic convolution parameter according to the intermediate feature map, and determines the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map; finally, the second task head of the lane line detection model generates N lane lines according to the dynamic convolution parameter and the multi-scale feature map. Among them, the category output module for outputting the lane line starting point heat map in the first task head includes a plurality of channels corresponding to a plurality of preset lane line categories, so that the category output module can output the lane line starting point sub-heat map corresponding to each channel, that is, the lane line starting point sub-heat map under each lane line category can be obtained, and then the lane line category of the lane line to which each lane line starting point in the lane line starting point sub-heat map belongs can be determined according to the channel corresponding to the lane line starting point sub-heat map. In this way, while realizing lane line detection in road images, the lane line category of each lane line can be identified, thereby meeting the different requirements of downstream tasks such as driving rule control and high-precision positioning, and improving the safety of autonomous driving.

[0110] Optionally, the first task head is used to determine, for each lane line starting point in the lane line starting point heat map, a lane line category corresponding to a target channel as a predicted category of the lane line to which the lane line starting point belongs, wherein the target channel is the channel in the lane line starting point heat map where the lane line starting point is located.

[0111] Optionally, the second task head is used to dynamically convolve the target convolution parameter with the multi-scale feature map for each lane line starting point in the lane line starting point heat map to obtain the lane line where the lane line starting point is located, wherein the target convolution parameter is a dynamic convolution sub-parameter in the dynamic convolution parameter corresponding to the lane line starting point.

[0112] Optionally, when N>1, the lane line detection device 200 further includes:

[0113] The overlap elimination module is configured to eliminate overlaps of the N lane lines to obtain a target lane line.

[0114] Optionally, the overlap elimination module includes:

[0115] A lane line to be traversed determining submodule is configured to determine the lane line to be traversed from the detection result, wherein the detection result is initially the N lane lines;

[0116] an overlap determination submodule, configured to determine, for each lane line other than the lane line to be traversed in the detection result, whether the other lane line overlaps with the lane line to be traversed; if the other lane line overlaps with the lane line to be traversed, remove the other lane line from the detection result;

[0117] A traversal submodule is configured to remove the lane line to be traversed from the detection result obtained after the removal, and return to the step of determining the lane line to be traversed from the detection result until the number of lane lines included in the detection result obtained after the removal is less than or equal to 2;

[0118] The target lane line determination submodule is configured to determine the detection result obtained after elimination and the lane line to be traversed determined in each cycle as the target lane line if the number of lane lines included in the detection result is less than or equal to 2.

[0119] Optionally, the overlap determination submodule includes:

[0120] A line distance determination submodule, configured to determine a line distance between the other lane line and the lane line to be traversed;

[0121] The line distance comparison submodule is configured to determine that the other lane line overlaps with the lane line to be traversed if the line distance is less than a preset distance threshold.

[0122] Optionally, each of the lane lines includes a plurality of feature points;

[0123] The line distance determination submodule includes:

[0124] A point distance determination submodule is configured to determine, for each target feature point in the lane line to be traversed, a point distance between the target feature point and a corresponding feature point in the other lane line, wherein the target feature point and the corresponding feature point in the other lane line are arranged in a horizontal direction;

[0125] The point distance averaging submodule is configured to determine the average value of each of the point distances as the line distance between the other lane line and the lane line to be traversed.

[0126] Optionally, the lane line to be traversed determining submodule is configured to determine the lane line with the highest confidence in the detection result as the lane line to be traversed.

[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] The present disclosure also provides a vehicle, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to run the executable instructions to implement the steps of the above lane line detection method provided by the present disclosure.

[0129] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the lane line detection method provided by the present disclosure are implemented.

[0130] Figure 6 6 is a block diagram of a vehicle 600 according to an exemplary embodiment. For example, the vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0131] Reference Figure 6 , the vehicle 600 may include various subsystems, for example, an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. The vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 600 may be interconnected by wire or wireless means.

[0132] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, among others.

[0133] The perception system 620 may include several sensors for sensing information about the environment around the vehicle 600. For example, the perception system 620 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0134] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0135] The drive system 640 may include components that provide powered motion for the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0136] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one first processor 651 and a first memory 652, and the first processor 651 may execute instructions 653 stored in the first memory 652.

[0137] The first processor 651 may be any conventional processor, such as a commercially available CPU. The first processor 651 may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0138] The first memory 652 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0139] In addition to the instructions 653 , the first memory 652 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in the first memory 652 may be used by the computing platform 650 .

[0140] In the embodiment of the present disclosure, the first processor 651 may execute instruction 653 to complete all or part of the steps of the above lane line detection method.

[0141] Figure 7 1 is a block diagram of a device 1900 for lane line detection according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more second processors, and a memory resource represented by a second memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the second memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform all or part of the steps of the lane detection method described above.

[0142] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958. The device 1900 may operate based on an operating system stored in the second memory 1932.

[0143] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above lane line detection method when executed by the programmable device.

[0144] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0145] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A lane line detection method, characterized in that: include: Acquire a road image in front of the vehicle; Performing lane line detection on the road image by using a pre-trained lane line detection model to obtain lane line information in the road image, wherein the lane line information includes N lane lines and a predicted category of each lane line, where N≥1; The lane line detection model includes a feature extraction network, a first task head and a second task head; The feature extraction network is used to extract features from the road image to obtain an intermediate feature map and a multi-scale feature map; The first task head is connected to the feature extraction network, and is used to generate a lane line starting point heat map and dynamic convolution parameters according to the intermediate feature map, and determine the predicted category of the lane line to which each lane line starting point belongs in the lane line starting point heat map, wherein the category output module includes a plurality of channels corresponding to a plurality of preset lane line categories one by one, and the parameter output module includes a plurality of channel groups corresponding to the plurality of lane line categories one by one, the category output module is a module in the first task head for outputting the lane line starting point heat map, and the parameter output module is a module in the first task head for outputting the dynamic convolution parameters; The second task head is connected to the feature extraction network and the first task head respectively, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

2. The method according to claim 1, characterized in that The first task head is used to determine, for each lane line starting point in the lane line starting point heat map, a lane line category corresponding to a target channel as a predicted category of the lane line to which the lane line starting point belongs, wherein the target channel is the channel in the lane line starting point heat map where the lane line starting point is located.

3. The method according to claim 1, characterized in that The second task head is used to dynamically convolve the target convolution parameter with the multi-scale feature map for each lane line starting point in the lane line starting point heat map to obtain the lane line where the lane line starting point is located, wherein the target convolution parameter is a dynamic convolution sub-parameter in the dynamic convolution parameter corresponding to the lane line starting point.

4. The method according to claim 1, characterized in that: When N>1, the method further includes: The N lane lines are overlapped and eliminated to obtain a target lane line.

5. The method according to claim 4, characterized in that Eliminating overlap of the N lane lines to obtain a target lane line includes: Determine the lane lines to be traversed from the detection results, wherein the detection results are initially the N lane lines; For each lane line other than the lane line to be traversed in the detection result, determining whether the other lane line overlaps with the lane line to be traversed; if the other lane line overlaps with the lane line to be traversed, removing the other lane line from the detection result; Eliminate the lane line to be traversed from the detection result obtained after the elimination, and return to the step of determining the lane line to be traversed from the detection result until the number of lane lines included in the detection result obtained after the elimination is less than or equal to 2; If the number of lane lines included in the detection result is less than or equal to 2, the detection result obtained after elimination and the lane line to be traversed determined in each cycle are determined as the target lane line.

6. The method according to claim 5, characterized in that The determining whether the other lane line overlaps with the lane line to be traversed includes: Determine the line distance between the other lane line and the lane line to be traversed; If the line distance is less than a preset distance threshold, it is determined that the other lane line overlaps with the lane line to be traversed.

7. The method according to claim 6, characterized in that Each of the lane lines includes a plurality of feature points; The determining the line distance between the other lane line and the lane line to be traversed includes: For each target feature point in the lane line to be traversed, determine the point distance between the target feature point and a corresponding feature point in the other lane line, wherein the target feature point and the corresponding feature point in the other lane line are arranged in a horizontal direction; The average value of each of the point distances is determined as the line distance between the other lane line and the lane line to be traversed.

8. The method according to claim 5, characterized in that The determining of the lane line to be traversed from the detection result includes: The lane line with the highest confidence in the detection result is determined as the lane line to be traversed.

9. A lane line detection device, characterized in that: include: An image acquisition module is configured to acquire an image of a road in front of the vehicle; The detection module is configured to perform lane line detection on the road image through a pre-trained lane line detection model to obtain lane line information in the road image, wherein the lane line information includes N lane lines and a predicted category of each lane line, where N≥1; wherein the lane line detection model includes a feature extraction network, a first task head, and a second task head; the feature extraction network is used to perform feature extraction on the road image to obtain an intermediate feature map and a multi-scale feature map; the first task head is connected to the feature extraction network to generate a lane line starting point heat map and dynamic convolution parameters according to the intermediate feature map, and determine the lane line starting point heat map and the ... obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate feature map, and obtain the lane line starting point heat map and the dynamic convolution parameters according to the intermediate The predicted category of the lane line to which the starting point of each lane line belongs in the lane starting point heat map, wherein the category output module includes a plurality of channels corresponding one-to-one to a plurality of preset lane line categories, and the parameter output module includes a plurality of channel groups corresponding one-to-one to the plurality of lane line categories, the category output module is a module in the first task head for outputting the lane line starting point heat map, and the parameter output module is a module in the first task head for outputting the dynamic convolution parameters; the second task head is respectively connected to the feature extraction network and the first task head, and is used to generate the N lane lines according to the dynamic convolution parameters and the multi-scale feature map.

10. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to run the executable instructions to implement the steps of the lane line detection method in any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the lane line detection method described in any one of claims 1 to 8 are implemented.

Images