Lane line detection method and device

Abstract

The invention relates to a lane line detection method. The method comprises the following steps: receiving an externally input picture; predicting a feature map of the picture pixel by pixel by using a deep learning network, wherein the feature map comprises a first feature map reflecting angle information from a current pixel point to a key point; and decoding the feature map so as to obtain the position information of the lane line in the picture. The invention also relates to a lane line detection device, a computer storage medium, a computer program product and a camera for automatic driving.

Description

Technical Field

[0001] The present application relates to the field of lane line detection, and more specifically, to a lane line detection method and device, a computer storage medium, a computer program product, and a camera for autonomous driving. Background Art

[0002] Lane detection plays an important role in automated driving systems (ADS). On the one hand, the position of the host vehicle and other traffic participants on the lane forms the basis for automated driving decisions. On the other hand, the geometry of lane markings can be considered as important landmarks of the environment and aligned with high-resolution or vector maps for high-precision positioning. At the same time, lane detection (or lane line detection) is widely used in advanced driver assistance systems (ADAS) and is the basis for some common functions such as lane keeping assist (LKA) and adaptive cruise control (ACC).

[0003] Lane (line) detection generally uses complex models and large receptive fields to establish global structural information in order to solve the "no visual clues" problem. However, large models and big data will inevitably bring problems such as high cost, high computational cost, and high latency.

[0004] There are related literatures that propose to abstract lane line detection into discrete key point detection and association problems to avoid using complex models, such as the Fololane lane line detection solution. However, this key point-based lane line detection solution is prone to deviations at curves and causes problems such as "lane line merging". Summary of the invention

[0005] According to one aspect of the present application, a lane line detection method is provided, the method comprising: receiving an external input picture; using a deep learning network to predict a feature map of the picture pixel by pixel, wherein the feature map comprises a first feature map reflecting the angle information from the current pixel point to the key point; and decoding the feature map to obtain the position information of the lane line in the picture.

[0006] As a supplement or alternative to the above solution, in the above method, the deep learning network adopts a Unet architecture, and the output size of the deep learning network is 1 / 4 of the size of the external input image.

[0007] As a supplement or replacement for the above-mentioned scheme, in the above-mentioned method, the feature map also includes: a heat map generated by using a Gaussian kernel to perform Gaussian distribution on each key point, a second feature map reflecting the error caused by downsampling, a third feature map reflecting the horizontal coordinate deviation of the current pixel point to the previous row of intersection key points, a fourth feature map reflecting the horizontal coordinate deviation of the current pixel point to the next row of intersection key points, a fifth feature map reflecting the horizontal coordinate deviation of the current pixel point to the current intersection key point, and a sixth feature map reflecting the lane line category.

[0008] As a supplement or alternative to the above solution, in the above method, the Gaussian radius of the Gaussian distribution performed on each key point is set to gradually decrease from near to far.

[0009] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, the Gaussian radius is also set to be adjusted based on the distance between the lane lines, wherein when half of the distance between the lane lines is less than a preset Gaussian radius, the Gaussian radius is adjusted to half of the distance between the lane lines.

[0010] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, decoding the feature map includes: (step A) in the heat map, finding the peak point Pi in each row as the key point, and taking the key point of the bottom row as the starting point; (step B) using the second feature map and the fifth feature map to correct the starting point and use it as the current key point; (step C) starting from the current key point, exploring the corresponding pixel points of the previous row based on the first feature map and the third feature map; (step D) when the value of the corresponding pixel point in the heat map is greater than a first threshold, connecting the corresponding pixel point to the current key point and determining the corresponding pixel point as the updated current key point; and (step E) repeating the above steps C and D until all rows are traversed or the value of the corresponding pixel point in the heat map is less than or equal to the first threshold.

[0011] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, decoding the feature map also includes: (step F) calculating the horizontal distance D between the corresponding pixel point and the peak point Pi+1 in the same row; (step G) when the distance D is greater than a second threshold, taking the peak point Pi+1 in the same row as the starting point of the new lane line; and (step H) repeating the above-mentioned steps B to G until all rows are traversed or the value of the corresponding pixel point in the heat map is less than or equal to the first threshold.

[0012] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, decoding the feature map also includes: (step I) starting from the current key point, exploring the corresponding pixel points of the next row based on the first feature map and the fourth feature map; and (step J) repeating the above-mentioned steps D to I until all rows are traversed or the value of the corresponding pixel point in the heat map is less than or equal to the first threshold.

[0013] As a supplement or alternative to the above solution, the above method also includes: decoding the sixth feature map to obtain the type of lane lines in the picture.

[0014] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, the deep learning network is trained through the following steps: determining training samples; preparing a lane line labeling matrix based on the training samples, wherein in each row of the lane line labeling matrix, only one pixel for each lane is marked as a key point; based on the lane line labeling matrix, generating a feature map true value; and inputting the feature map true value into the deep learning network for deep learning.

[0015] According to another aspect of the present application, a lane line detection device is provided, which includes: a receiving device for receiving an external input picture; a prediction device for using a deep learning network to predict a feature map of the picture pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel point to the key point; and a decoding device for decoding the feature map to obtain the position information of the lane line in the picture.

[0016] As a supplement or alternative to the above solution, in the above device, the deep learning network adopts a Unet architecture, and the output size of the deep learning network is 1 / 4 of the size of the external input image.

[0017] According to another aspect of the present application, a computer storage medium is provided, wherein the medium includes instructions, and the instructions execute the above method when executed.

[0018] According to another aspect of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0019] According to another aspect of the present application, a camera for autonomous driving is provided, wherein the camera includes the lane line detection device as described above.

[0020] The lane line detection scheme of the embodiment of the present application uses a deep learning network to predict the feature map of an external input image pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel point to the key point, and then the feature map is decoded to obtain the position information of the lane line in the image. Since the above lane line detection scheme predicts the angle information from the current pixel point to the key point (for example, the key point of the previous row of intersection points or the key point of the next row of intersection points), the local curve finally estimated is very accurate even in the curved road scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects and advantages of the present application will become more fully apparent from the following detailed description in conjunction with the accompanying drawings, wherein the same or similar elements are represented by the same reference numerals.

[0022] Figure 1 A schematic diagram of a process of a lane line detection method according to an embodiment of the present application is shown;

[0023] Figure 2 A schematic structural diagram of a lane line detection device according to an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram showing the horizontal coordinate deviation from the current pixel point to the key point of the intersection of the previous row according to an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram showing the horizontal coordinate deviation from the current pixel point to the next row intersection key point according to an embodiment of the present application is shown;

[0026] Figure 5 A schematic diagram showing a horizontal coordinate deviation from a current pixel point to a current intersection key point according to an embodiment of the present application; and

[0027] Figure 6 A schematic diagram showing angle information from a current pixel point to a key point of intersection in a previous row according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] Hereinafter, a lane line detection scheme according to various exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0029] The inventor of the present application realizes that, by combining the angle information from the current pixel to the key point, complex lane lines can be accurately fitted with polylines. Moreover, the local modeling method can improve the generalization of the system due to its limited receptive field and low model complexity.

[0030] In addition, in order to make the generated lane heatmap closer to reality, the embodiment of the present application proposes to set the Gaussian radius to gradually decrease from near to far (rather than using a fixed Gaussian radius), which can solve the problem of unclear lane lines in the distance. In one embodiment, the lane detection scheme of the present application also determines the distance between two lane lines: if half of the distance is less than the set Gaussian radius, the Gaussian radius is set to half of the distance between the lane lines, thereby reducing the false detection rate and missed detection rate of the double-line (approaching) lane line scene.

[0031] Figure 1 FIG. 1 is a schematic diagram of a lane line detection method 1000 according to an embodiment of the present application. Figure 1 As shown, the lane line detection method 1000 includes:

[0032] In step S110, receiving an external input picture;

[0033] In step S120, a deep learning network is used to predict a feature map of the image pixel by pixel, wherein the feature map includes a first feature map reflecting angle information from a current pixel to a key point; and

[0034] In step S130, the feature map is decoded to obtain the position information of the lane lines in the image.

[0035] In the context of this application, an "external input image" may be a driving scene image sensed by a vehicle-mounted sensor (such as a camera). Based on analyzing the driving scene image, driving environment information around the vehicle may be obtained, including but not limited to lane (line) information.

[0036] The term "deep learning network" may also be referred to as a neural network for deep learning. Deep learning is the inherent law and representation level of learning sample data. The information obtained in these learning processes is of great help to the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. In one or more embodiments of the present application, the deep learning network is mainly based on a neural network system of convolutional calculation, i.e., a convolutional neural network (CNN). It adopts, for example, a Unet architecture, and the (feature map) output size of the deep learning network is 1 / 4 of the size of the external input picture (for example, by performing 4 downsampling and 2 upsampling in the network), which can reduce the required storage space and the amount of calculation. In addition, in order to further reduce the amount of calculation and memory, addition is used instead of cascading (concat) for the intermediate amount during the network calculation process. Moreover, in one or more embodiments, when training a deep neural network, regression is performed using a feature pyramid form to obtain more deep and shallow features.

[0037] As described in step S120, a deep learning network is used to predict the feature map of the image pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel to the key point. In one embodiment, the feature map also includes: a heat map generated by performing Gaussian distribution for each key point using a Gaussian kernel, a second feature map reflecting the error caused by downsampling, a third feature map reflecting the horizontal coordinate deviation from the current pixel to the key point of the previous row of intersections, a fourth feature map reflecting the horizontal coordinate deviation from the current pixel to the key point of the next row of intersections, a fifth feature map reflecting the horizontal coordinate deviation from the current pixel to the key point of the current intersection, and a sixth feature map reflecting the lane line category.

[0038] Here, the heat map (i.e., heatmap) represents the possibility of the occurrence of key points and solves the existence problem of local curves. For example, the heat map is based on the segmentation lane line segmentation head, and its output is 1 / 4 of the original image resolution. During the training stage, in one embodiment, the manually annotated key points can be interpolated to obtain a continuous curve with a curve width of about 1 pixel. The pixels on the curve are the key-points set. Draw lines along the image at equal distances along the height, with a fixed distance of Δy. In this way, the intersection of the bisector and the lane line curve is a subset of the key point set, which is referred to as "intersection key points" in the following text for convenience.

[0039] In one embodiment, the Gaussian radius of the Gaussian distribution for each key point is set to gradually decrease from near to far (rather than using a fixed Gaussian radius), which can solve the problem of unclear lane lines in the distance. In one embodiment, the Gaussian radius is also set to be adjusted based on the distance between the lane lines, where when half of the distance between the lane lines is less than the preset Gaussian radius, the Gaussian radius is adjusted to half of the distance between the lane lines. This can reduce the false detection rate and missed detection rate in the double-line (approaching) lane line scene.

[0040] Figure 3 A schematic diagram showing the horizontal coordinate deviation from the current pixel point to the key point of the intersection of the previous row according to an embodiment of the present application is shown, which is, for example, a part of the third feature map. Figure 3 In the example, each rectangular grid represents a pixel, and there is only one pixel between adjacent rows (i.e., the row spacing is 1). Figure 3It is only a schematic diagram. In other embodiments, multiple pixels can be spaced between adjacent rows, and are not limited to 1 pixel. Pixels 310, 320 and 330 represent the key points in different rows (i.e., "intersection key points") respectively. Wherein, pixel 310 is assigned a value of 2, because the deviation of the current pixel 310 to the previous row intersection key point 320 is 2 (i.e., 2 pixels) in the horizontal direction. Similarly, pixel 320 is assigned a value of 3, because the deviation of this pixel 320 and the previous row intersection key point 330 in the horizontal direction is 3 pixels. By analogy, since the deviation of pixel 312 and the previous row intersection key point 320 in the horizontal direction is 3 pixels, the deviation of pixel 314 and the previous row intersection key point 320 in the horizontal direction is 1 pixel, the deviation of pixel 322 and the previous row intersection key point 330 in the horizontal direction is 4 pixels, and the deviation of pixel 324 and the previous row intersection key point 330 in the horizontal direction is 2 pixels, so they are respectively assigned a value of 3, 1, 4, 2 accordingly.

[0041] Figure 4 A schematic diagram showing the horizontal coordinate deviation from the current pixel point to the next row intersection key point according to an embodiment of the present application is shown, which is, for example, a part of the fourth feature map. Figure 4 In the example, each rectangular grid represents a pixel, and there is only one pixel between adjacent rows (i.e., the row spacing is 1). Figure 4 It is only a schematic diagram. In other embodiments, multiple pixels may be spaced between adjacent rows, and are not limited to 1 pixel. Pixels 410, 420, and 430 represent key points in different rows (i.e., "intersection key points"), respectively. Among them, pixel 420 is assigned a value of -3, because the horizontal deviation between the current pixel 420 and the next row intersection key point 410 is 3, and the current pixel 420 is on the right side of the next row intersection key point 410 (i.e., the right side is indicated by a minus sign). Similarly, pixel 410 is assigned a value of -2, because the pixel 410 is located on the right side of the next row intersection key point 430, and the horizontal deviation is 2 pixels. By analogy, since pixel 422 is to the right of the next row intersection key point 410 and has a horizontal deviation of 2 pixels, pixel 424 is to the right of the next row intersection key point 410 and has a horizontal deviation of 4 pixels, pixel 412 is to the right of the next row intersection key point 430 and has a horizontal deviation of 1 pixel, and pixel 414 is to the right of the next row intersection key point 430 and has a horizontal deviation of 3 pixels, they are assigned values ​​of -2, -4, -1, and -3, respectively.

[0042] Figure 5 A schematic diagram showing the horizontal coordinate deviation from the current pixel point to the current intersection key point according to an embodiment of the present application is shown, which is, for example, a part of the fifth feature map. Figure 5, each rectangular grid represents a pixel. Pixels 510, 520 and 530 represent key points in different rows (i.e., "intersection key points"), respectively. Among them, pixels 510, 520 and 530 are all assigned a value of 0, because they themselves are intersection key points, so the horizontal coordinate deviation from the current intersection key point is 0. Pixels 512, 522 and 532 are respectively on the left side of intersection key points 510, 520 and 530, and are separated from them by 1 pixel, so they are all assigned a value of 1. In addition, pixels 514, 524 and 534 are respectively on the right side of intersection key points 510, 520 and 530, and are separated from them by 1 pixel, so they are all assigned a value of -1 (positive on the left and negative on the right).

[0043] Figure 6 A schematic diagram showing angle information from a current pixel to a key point of a previous row of intersections according to an embodiment of the present application is shown, which is, for example, part of a first feature map. Figure 6 In the example, each rectangular grid represents a pixel, and there is only one pixel between adjacent rows (i.e., the row spacing is 1). Figure 6 It is only a schematic diagram. In other embodiments, multiple pixels may be spaced between adjacent rows, not limited to 1 pixel. Pixels 610, 620, and 630 represent key points in different rows (i.e., "intersection key points"), respectively. Among them, pixel 610 is assigned a value of -0.447, because the sine value of the angle between the line formed by the current pixel 610 and the previous row intersection key point 620 and the pixel row is 0.447, and the "minus sign" indicates that the previous row intersection key point 620 is on the right side of the current pixel 610. Similarly, pixel 620 is assigned a value of -0.316, because the sine value of the angle between the line formed by the current pixel 620 and the previous row intersection key point 630 and the pixel row is 0.316, and the "minus sign" indicates that the previous row intersection key point 630 is on the right side of the current pixel 620. According to the same principle, the angle information of other points (such as 622, 624, 612 and 614) can be obtained, which will not be repeated here.

[0044] Based on these feature maps, accurate geometric information (including angle information) of local curves near key points can be obtained. In one or more embodiments, since a deep learning network is used to predict the feature map of a picture pixel by pixel, the feature map includes a first feature map that reflects the angle information from the current pixel to the key point. In essence, by adding the angle information represented by the first feature map, it is helpful to cluster the relationship between points. For example, if two points are on the same straight line, the angle information of the two points (such as the sine of the angle between the straight line between the two points and the vertical line) will also be regressed accordingly while regressing the horizontal distance between the two points. This is particularly suitable for multi-task learning, so that related tasks can promote each other.

[0045] In one or more embodiments of the present application, by adopting a suitable decoding method to decode these feature maps, the position and category information of the lane line in the picture can be obtained. For example, in step S130, the feature map is decoded to obtain the position information of the lane line in the picture. In one embodiment, step S130 includes the following steps:

[0046] Step A: In the heat map, find the peak point Pi (for example, P is the coordinate, i represents the i-th line) in each row as the key point (for example, by using tools such as find peaks or maxpooling), and take the key point of the bottom row as the starting point.

[0047] Step B: Use the second feature map and the fifth feature map to correct the starting point and use it as the current key point. For example, for point p, the correction can be performed by the following formula: ^p=p+[Δx → (p); 0] Τ , where ^p represents the corrected point p, Δx → (p) represents the horizontal coordinate deviation between the current pixel and the key point of the intersection of the row where the current pixel is located, that is, it is represented by the fifth feature map.

[0048] In addition, the second feature map can be used to compensate for the error caused by downsampling. Due to the two downsampling, the coordinate value of each pixel in the pixel matrix needs to be divided by 4, which will result in a loss of precision. For example, if the coordinate value of a pixel is 127 (p0), then its coordinate value after downsampling is 31.75, and after rounding to 32 (p2), there will be an error of 0.25. In order to compensate or smooth out the error and normalize all error values ​​between -0.5 and 0.5, the error Offset can be calculated by the following formula:

[0049] Offset=(p2*4–p0) / 4=(32*4-127) / 4=1 / 4.

[0050] For example, if the coordinate value of a pixel is 125 (p0), then after downsampling, its coordinate value is 31.25, and after rounding, it is 31 (p2), so there will be an error of -0.25. In order to compensate or smooth out the error and normalize all error values ​​between -0.5 and 0.5, the error Offset can be calculated by the following formula:

[0051] Offset=(p2*4–p0) / 4=(31*4-125) / 4=-0.25.

[0052] Step C: Starting from the current key point, explore the corresponding pixel points in the previous row based on the first feature map and the third feature map. This can be done, for example, by the following formula:

[0053]

[0054] Among them, p ↑ is the corresponding pixel point in the previous row explored, Δx ↑ (p) is the horizontal coordinate deviation from the current pixel to the key point of the previous row intersection, which is represented by the third feature map. Δanglex ↑ (p) is the angle information from the current pixel point to the key point of the intersection of the previous row (for example, the sine value of the angle between the straight line from the current pixel point to the key point of the intersection of the previous row and the vertical line), represented in the first feature map, and -Δy represents the vertical offset value between the current row and the previous row.

[0055] Step D: When the value of the corresponding pixel in the heat map is greater than a first threshold (i.e., hm[P ↑ ,i+1]>threshold_hm), connect the corresponding pixel point with the current key point, and determine the corresponding pixel point as the updated current key point.

[0056] Step E: Repeat the above steps C and D until all rows are traversed, or the value of the corresponding pixel point in the heat map is less than or equal to the first threshold (ie, the line ends).

[0057] In one embodiment, step S130 further includes the following steps:

[0058] Step F: Calculate the horizontal distance (minimum value) D between the corresponding pixel point and the peak point Pi+1 in the same row;

[0059] Step G: When the distance D is greater than the second threshold, the peak point Pi+1 in the same row is used as the starting point of the new lane line. In other words, the corresponding pixel point is considered to be the starting point of a new lane line.

[0060] Step H: Repeat the above steps B to G until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

[0061] In one embodiment, step S130 further includes the following steps:

[0062] Step I: Starting from the current key point, explore the corresponding pixel points of the next row based on the first feature map and the fourth feature map, which can be accomplished, for example, by the following formula:

[0063]

[0064] Among them, p ↓ is the pixel corresponding to the next row to be explored, Δx ↓ (p) is the horizontal coordinate deviation from the current pixel to the next row intersection key point, which is represented by the fourth feature map. Δanglex ↓ (p) is the angle information from the current pixel point to the next row intersection key point (for example, the sine value of the angle between the straight line from the current pixel point to the next row intersection key point and the vertical line), which can be represented by the first feature map, and +Δy represents the vertical offset value between this row and the next row.

[0065] Step J: Repeat the above steps D to I until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

[0066] although Figure 1 Not shown, the method 1000 further includes: decoding the sixth feature map to obtain the type of lane line in the image. For example, the sixth feature map may represent the corresponding classification attribute of each pixel by a number, such as a dashed line or a solid line, where 0 represents a dashed line and 1 represents a solid line.

[0067] In one embodiment, a deep learning network is trained by the following steps: determining training samples; preparing a lane line labeling matrix based on the training samples, wherein in each row of the lane line labeling matrix, only one pixel for each lane is marked as a key point; generating a feature map true value based on the lane line labeling matrix; and inputting the feature map true value into the deep learning network for deep learning.

[0068] For example, based on the 2D coordinates in the labeled json file, it is easy to connect all the points together. It is necessary to ensure that only one pixel per row and lane is labeled as a keypoint (value 1) to avoid confusion during training. This lane label matrix should be the basis matrix for all other ground truth matrices to be generated later. The following points need to be noted: (1) Each lane has its unique lane label; (2) Each row has one and only one lane number corresponding to the lane label.

[0069] In addition, it is easy for a person skilled in the art to understand that the lane line detection method 1000 provided in the above one or more embodiments of the present application can be implemented by a computer program. For example, the computer program is included in a computer program product, and when the computer program is executed by a processor, the lane line detection method 1000 of one or more embodiments of the present application is implemented. For another example, when a computer storage medium (such as a USB flash drive) storing the computer program is connected to a computer, the lane line detection method 1000 of one or more embodiments of the present application can be executed by running the computer program.

[0070] Figure 2 FIG. 2 shows a schematic diagram of the structure of a lane line detection device 2000 according to an embodiment of the present application. Figure 2 As shown, the lane line detection device 2000 includes: a receiving device 210, a prediction device 220 and a decoding device 230. The receiving device 210 is used to receive an external input picture; the prediction device 220 is used to use a deep learning network to predict the feature map of the picture pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel point to the key point; and the decoding device 230 is used to decode the feature map to obtain the position information of the lane line in the picture.

[0071] In the context of this application, an "external input picture" may be a driving scene picture sensed by an on-board sensor (such as a camera). Based on the analysis of the driving scene picture, the driving environment information around the vehicle may be obtained, including but not limited to lane (line) information. It should be noted that the "external input" here is relative to the "receiving device 210" itself, so when the lane line detection device 2000 is integrated in a multi-purpose camera (such as mpc3), the driving scene picture sensed by the camera should also be regarded as an "external input picture".

[0072] The term "deep learning network" may also be referred to as a neural network for deep learning. Deep learning is the inherent law and representation level of learning sample data. The information obtained in these learning processes is of great help to the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. In one or more embodiments of the present application, the deep learning network is mainly based on a neural network system of convolutional calculation, i.e., a convolutional neural network (CNN). It adopts, for example, a Unet architecture, and the (feature map) output size of the deep learning network is 1 / 4 of the size of the external input picture (for example, by performing 4 downsampling and 2 upsampling in the network), which can reduce the required storage space and the amount of calculation. In addition, in order to further reduce the amount of calculation and memory, addition is used instead of cascading (concat) for the intermediate amount during the network calculation process. Moreover, in one or more embodiments, when training a deep neural network, regression is performed using a feature pyramid form to obtain more deep and shallow features.

[0073] The prediction device 220 is configured to use a deep learning network to predict the feature map of the picture pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel to the key point. In one embodiment, the feature map also includes: a heat map generated by using a Gaussian kernel to perform Gaussian distribution for each key point, a second feature map reflecting the error caused by downsampling, a third feature map reflecting the horizontal coordinate deviation from the current pixel to the previous row of intersection key points, a fourth feature map reflecting the horizontal coordinate deviation from the current pixel to the next row of intersection key points, a fifth feature map reflecting the horizontal coordinate deviation from the current pixel to the current intersection key point, and a sixth feature map reflecting the lane line category.

[0074] Here, the heat map (i.e., heatmap) represents the possibility of the occurrence of key points and solves the existence problem of local curves. For example, the heat map is based on the segmentation lane line segmentation head, and its output is 1 / 4 of the original image resolution. During the training stage, in one embodiment, the manually annotated key points can be interpolated to obtain a continuous curve with a curve width of about 1 pixel. The pixels on the curve are the key-points set. Draw lines along the image at equal distances along the height, with a fixed distance of Δy. In this way, the intersection of the bisector and the lane line curve is a subset of the key point set, which is referred to as "intersection key points" in the following text for convenience.

[0075] In one embodiment, the Gaussian radius of the Gaussian distribution for each key point is set to gradually decrease from near to far (rather than using a fixed Gaussian radius), which can solve the problem of unclear lane lines in the distance. In one embodiment, the Gaussian radius is also set to be adjusted based on the distance between the lane lines, where when half of the distance between the lane lines is less than the preset Gaussian radius, the Gaussian radius is adjusted to half of the distance between the lane lines. This can reduce the false detection rate and missed detection rate in the double-line (approaching) lane line scene.

[0076] In one or more embodiments of the present application, by adopting a suitable decoding method to decode these feature maps, the position and category information of the lane line in the picture can be obtained. For example, the decoding device 230 is configured to decode the feature map to obtain the position information of the lane line in the picture. In one embodiment, the decoding device 230 is configured to perform the following steps:

[0077] Step A: In the heat map, find the peak point Pi (for example, P is the coordinate, i represents the i-th line) in each row as the key point (for example, by using tools such as find peaks or maxpooling), and take the key point of the bottom row as the starting point.

[0078] Step B: Use the second feature map and the fifth feature map to correct the starting point and use it as the current key point. For example, for point p, the correction can be performed by the following formula: ^p=p+[Δx → (p); 0] Τ , where ^p represents the corrected point p, Δx → (p) represents the horizontal coordinate deviation between the current pixel and the key point of the intersection of the row where the current pixel is located, that is, it is represented by the fifth feature map.

[0079] In addition, the second feature map can be used to compensate for the error caused by downsampling. Due to the two downsampling, the coordinate value of each pixel in the pixel matrix needs to be divided by 4, which will result in a loss of precision. For example, if the coordinate value of a pixel is 127 (p0), then its coordinate value after downsampling is 31.75, and after rounding to 32 (p2), there will be an error of 0.25. In order to compensate or smooth out the error and normalize all error values ​​between -0.5 and 0.5, the error Offset can be calculated by the following formula:

[0080] Offset=(p2*4–p0) / 4=(32*4-127) / 4=1 / 4.

[0081] For example, if the coordinate value of a pixel is 125 (p0), then after downsampling, its coordinate value is 31.25, and after rounding, it is 31 (p2), so there will be an error of -0.25. In order to compensate or smooth out the error and normalize all error values ​​between -0.5 and 0.5, the error Offset can be calculated by the following formula:

[0082] Offset=(p2*4–p0) / 4=(31*4-125) / 4=-0.25.

[0083] Step C: Starting from the current key point, explore the corresponding pixel points in the previous row based on the first feature map and the third feature map. This can be done, for example, by the following formula:

[0084]

[0085] Among them, p ↑ is the corresponding pixel point in the previous row explored, Δx ↑ (p) is the horizontal coordinate deviation from the current pixel to the key point of the previous row intersection, which is represented by the third feature map. Δanglex ↑ (p) is the angle information from the current pixel point to the key point of the intersection of the previous row (for example, the sine value of the angle between the straight line from the current pixel point to the key point of the intersection of the previous row and the vertical line), represented in the first feature map, and -Δy represents the vertical offset value between the current row and the previous row.

[0086] Step D: When the value of the corresponding pixel in the heat map is greater than a first threshold (i.e., hm[P ↑ ,i+1]>threshold_hm), connect the corresponding pixel point with the current key point, and determine the corresponding pixel point as the updated current key point.

[0087] Step E: Repeat the above steps C and D until all rows are traversed, or the value of the corresponding pixel point in the heat map is less than or equal to the first threshold (ie, the line ends).

[0088] In one embodiment, the decoding device 230 is further configured to perform the following steps:

[0089] Step F: Calculate the horizontal distance (minimum value) D between the corresponding pixel point and the peak point Pi+1 in the same row;

[0090] Step G: When the distance D is greater than the second threshold, the peak point Pi+1 in the same row is used as the starting point of the new lane line. In other words, the corresponding pixel point is considered to be the starting point of a new lane line.

[0091] Step H: Repeat the above steps B to G until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

[0092] In one embodiment, the decoding device 230 is further configured to perform the following steps:

[0093] Step I: Starting from the current key point, explore the corresponding pixel points of the next row based on the first feature map and the fourth feature map, which can be accomplished, for example, by the following formula:

[0094]

[0095] Among them, p ↓ is the pixel corresponding to the next row to be explored, Δx ↓ (p) is the horizontal coordinate deviation from the current pixel to the next row intersection key point, which is represented by the fourth feature map. Δanglex ↓ (p) is the angle information from the current pixel point to the next row intersection key point (for example, the sine value of the angle between the straight line from the current pixel point to the next row intersection key point and the vertical line), which can be represented by the first feature map, and +Δy represents the vertical offset value between this row and the next row.

[0096] Step J: Repeat the above steps D to I until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

[0097] The decoding device 230 may also be configured to decode the sixth feature map to obtain the type of lane line in the image. For example, the sixth feature map may represent the corresponding classification attribute of each pixel by a number, such as a dashed line or a solid line, where 0 represents a dashed line and 1 represents a solid line.

[0098] In one or more embodiments, the lane detection device 2000 may be included in various cameras for autonomous driving (such as mpc3).

[0099] In summary, the lane line detection scheme of the embodiment of the present application uses a deep learning network to predict the feature map of an external input image pixel by pixel, wherein the feature map includes a first feature map reflecting the angle information from the current pixel point to the key point, and then the feature map is decoded to obtain the position information of the lane line in the image. Since the above lane line detection scheme predicts the angle information from the current pixel point to the key point (for example, the key point of the previous row of intersection points or the key point of the next row of intersection points), the local curve finally estimated is very accurate even in the curved road scene.

[0100] The above examples mainly illustrate the lane detection scheme of the embodiments of the present application. Although only some of the implementation methods of the present application are described, it should be understood by those skilled in the art that the present application can be implemented in many other forms without departing from its subject matter and scope. Therefore, the examples and implementation methods shown are regarded as illustrative rather than restrictive, and the present application may cover various modifications and substitutions without departing from the spirit and scope of the present application as defined in the claims.

Claims

1. A lane line detection method, characterized in that: The method comprises: Receive external input pictures; Using a deep learning network to predict a feature map of the image pixel by pixel, wherein the feature map includes a first feature map reflecting angle information from a current pixel to a key point; and The feature map is decoded to obtain the position information of the lane lines in the image.

2. The method of claim 1, wherein: The deep learning network adopts a Unet architecture, and the output size of the deep learning network is 1 / 4 of the size of the external input image.

3. The method according to claim 1 or 2, wherein: The feature map also includes: a heat map generated by using a Gaussian kernel to perform Gaussian distribution on each key point, a second feature map reflecting the error caused by downsampling, a third feature map reflecting the horizontal coordinate deviation from the current pixel point to the previous row of intersection key points, a fourth feature map reflecting the horizontal coordinate deviation from the current pixel point to the next row of intersection key points, a fifth feature map reflecting the horizontal coordinate deviation from the current pixel point to the current intersection key point, and a sixth feature map reflecting the lane line category.

4. The method of claim 3, wherein: The Gaussian radius of the Gaussian distribution performed on each key point is set to gradually decrease from near to far.

5. The method of claim 4, wherein: The Gaussian radius is also set to be adjusted based on the distance between the lane lines, wherein when half of the distance between the lane lines is less than a preset Gaussian radius, the Gaussian radius is adjusted to half of the distance between the lane lines.

6. The method of claim 3, wherein: Decoding the feature map includes: Step A: In the heat map, find the peak point P in each row i as key points, and use the key points of the bottom row as the starting point; Step B: using the second feature map and the fifth feature map to correct the starting point and use it as the current key point; Step C: starting from the current key point, exploring corresponding pixel points in the previous row based on the first feature map and the third feature map; Step D: When the value of the corresponding pixel point in the heat map is greater than a first threshold, connecting the corresponding pixel point with the current key point, and determining the corresponding pixel point as an updated current key point; and Step E: Repeat the above steps C and D until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

7. The method of claim 6, wherein: Decoding the feature map further includes: Step F: Calculate the peak point P of the corresponding pixel point and the same row i+1 The horizontal distance D; Step G: When the distance D is greater than the second threshold, the peak point P in the same row is i+1 As the starting point for new lane lines; and Step H: Repeat the above steps B to G until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

8. The method of claim 7, wherein: Decoding the feature map further includes: Step I: starting from the current key point, exploring corresponding pixel points in the next row based on the first feature map and the fourth feature map; and Step J: Repeat the above steps D to I until all rows are traversed or the value of the corresponding pixel in the heat map is less than or equal to the first threshold.

9. The method of claim 3, further comprising: The sixth feature map is decoded to obtain the type of lane line in the image.

10. The method of claim 1, wherein: The deep learning network is trained by the following steps: Determine the training samples; Based on the training samples, preparing a lane line label matrix, wherein in each row of the lane line label matrix, only one pixel for each lane is marked as a key point; Based on the lane line label matrix, generating a true value of a feature map; as well as The true value of the feature map is input into the deep learning network for deep learning.

11. A lane line detection device, characterized in that: The device comprises: A receiving device, used for receiving an external input picture; A prediction device, configured to predict a feature map of the image pixel by pixel using a deep learning network, wherein the feature map includes a first feature map reflecting angle information from a current pixel to a key point; and A decoding device is used to decode the feature map to obtain the position information of the lane lines in the image.

12. The device of claim 11, wherein: The deep learning network adopts a Unet architecture, and the output size of the deep learning network is 1 / 4 of the size of the external input image.

13. A computer storage medium, characterized in that: The medium includes instructions which, when executed, perform the method of any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

15. A camera for autonomous driving, characterized in that: The camera includes the lane line detection device as described in claim 11 or 12.

Images